A picture can be a program
A red circle beside a blue rectangle can be stored as thousands of pixel values. It can also be described as two objects: a circle with a center and radius, and a rectangle with a position, width, and height. Both descriptions can produce a similar image. They offer very different ways to understand and change it.
In the second description, moving the circle means changing its coordinates. Counting circles means inspecting objects. Changing the blue rectangle to green means editing one attribute. A useful vector representation exposes these decisions directly, so the same document can serve as a drawing, an editable program, and an input to reasoning.
Beyond Pixels investigates this representation problem through S²VG², Simple Semantic Vector Graphics Generation. It is the first study to adapt a pretrained autoregressive vision-language model for image-to-SVG code generation, with simple geometric images as its experimental setting. The target is a compact program whose objects remain recognizable to people and language models.
Rendering the right pixels is only part of the task
SVG is an XML-based graphics language. Its vocabulary includes named primitives such as circles and rectangles, transformations, colors, and arbitrary paths. This flexibility creates an important ambiguity: an SVG file can be visually accurate while revealing little of the scene’s structure.
A circle may be encoded as a circle element, or approximated with a collection of Bézier curves. A raster comparison can judge both by their rendered appearance. A person reading the source encounters a much simpler object in the first case. A language model answering a question about shapes faces the same distinction.
The research question therefore connects three objectives: recover the image, preserve useful semantic structure, and keep the resulting description readable. More geometric detail does not automatically improve all three. Conversely, a short SVG that omits an object has compressed away information the task needs.
Generate structure, then refine geometry
The model treats vectorization as image-conditioned code generation. A Vision Transformer encodes global and patch-level visual features. A causal text decoder attends to those features and predicts an SVG token sequence. The implementation described in the manuscript initializes the visual-language components from BLIP, using a BERT-initialized decoder with causal attention.
The training objective is the conditional negative log-likelihood of the reference SVG sequence. At each position, the model predicts the next token from the image and the preceding SVG tokens. This places named graphical objects in the same autoregressive framework used for other structured text outputs.
There is a useful division of labor at inference. The discrete prediction chooses shape types and program structure. A differentiable renderer then adjusts continuous attributes such as position, scale, rotation, and color against the input image. The manuscript describes a refinement objective combining pixelwise L1 and squared L2 errors while keeping the predicted shape types fixed.
This division also explains a failure boundary. Refining the coordinates of a correctly predicted circle can improve alignment. If an object is missing or has the wrong type, changing its continuous parameters does not by itself repair the structural decision.
Build tests around what the representation should preserve
SVG-SHAPE places colored geometric primitives on a grid. Questions test object presence and relative position. These scenes let an evaluation separate recognizing an object from preserving where it sits relative to another object.
SVG-Transform adds translation, scaling, skew, and rotation. Its purpose is to test geometric transformations rather than only a fixed arrangement of axis-aligned shapes. The revised manuscript describes training sets of 50,000 image–SVG pairs for SVG-SHAPE and 500,000 pairs for SVG-Transform.
The experimental setting remains deliberately geometric. It offers explicit objects and known structure, making errors easier to identify. Results in this setting do not establish reliable vectorization of photographs, intricate typography, or arbitrary illustrations.
Look at the reconstruction and its source together
The qualitative comparisons below preserve the original manuscript images. They make it possible to inspect the input, S²VG² output, and alternative vectorizations together. Shape identity, overlap, layout, and color are all relevant; a visually close reconstruction can still have an opaque or unnecessarily long description.
The revised manuscript reports a LIVE configuration of 10 paths and 50 iterations and a DiffVG configuration of 50 paths and 500 iterations. Those settings matter when interpreting the comparison. The figures represent these particular evaluations, rather than the best achievable result under every possible optimization budget.
Readability is a measurable property
Visual quality, code complexity, and usefulness for reasoning answer different questions. The manuscript evaluates rendered image quality, asks people to interpret generated SVG, and examines whether language models can answer questions from the vector description.
In the revised manuscript’s file-characteristics tables, S²VG² has a mean length of 186.72 tokens, compared with 3,624.951 for LIVE and 15,736.102 for DiffVG. These are manuscript-reported measurements for its evaluated configurations. A shorter description reduces the material a reader must inspect, but token count alone cannot certify correct semantics.
The human study reports that 77.8% of participants could interpret S²VG² output and 85.8% preferred its semantic SVG output. The original result panels below retain the paper’s tables and figures, allowing the readability evidence to be considered alongside reconstruction quality.
When the vector representation becomes the reasoning input
The later manuscript adds CLEVR-SVG experiments. A scene is represented with explicit objects, colors, and spatial attributes, and a text-only reasoning model answers questions from that representation. Counting, identifying attributes, and comparing spatial positions probe different parts of the recovered structure.
The interactive results retain two archived batches separately. The 101-scene batch contains 92 correct answers. The 200-scene batch contains 164. They have different recorded sample sets and are not a controlled comparison between model configurations. Switching batches changes which archived outcomes are displayed; it does not run a model.
The two actual scene files below are useful small examples of the interface. A question about yellow triangles requires identifying both a shape and a color. A leftmost-object question also requires comparing locations. Success in one category does not establish that every relation in the scene is faithfully represented.
