⟵ Back to the journalImagine receiving a mechanical part with no design history. You can rotate it, measure it and inspect its global shape. But the cylindrical surface in front of you does not say whether it came from a hole, an extrusion, a revolution or a Boolean operation. It does not tell you which entity to edit, or what sequence produced it.
This is the ordinary condition of imported CAD. The geometry is present; the explanation is not. A human engineer can make progress because the CAD interface lets perception and interrogation happen in the same place: point at a face, select it, inspect its properties, trace its neighbors, find the feature that owns it. An agent usually gets the same evidence, but with the links between the pieces cut.
Three partial descriptions of one part
A CAD part typically reaches an agent through three representations: an image, a boundary representation and a construction history. A screenshot shows appearance. A boundary representation, or B-rep, stores the exact boundary of the solid as faces, edges and vertices, together with the curves and surfaces that define them. A feature tree or CAD program records the construction history: sketches, extrusions, cuts, patterns, fillets and the references between them.
Each representation answers a different question, and none is sufficient in isolation. The screenshot may make a hole visually unmistakable but says nothing stable about its identity. The B-rep can say that a face is cylindrical, give its radius and list adjacent edges, yet still not reveal which modeling operation created it. The program can preserve intent and dependencies, but only if the agent can connect its operations back to the geometry visible now. Systems such as BRepNet and BrepGen make this structure explicit: a solid is organized through faces, edges and vertices whose geometry and topology must be understood together.
CAD makes the perception problem deeper
Ordinary 3D perception is already underdetermined. Projection removes depth, while occlusion hides surfaces and relations. Many different 3D scenes can produce similar pixels. Rotating the camera adds evidence, but it does not make a single view complete.
CAD adds more layers to the hidden state. Besides geometry, there are analytic surfaces and curves, discrete topology, dimensions, constraints, features, persistent references and dependency chains. A mesh is often more direct: triangles approximate the surface that is seen. CAD also represents design intent — what should remain concentric, what was patterned, what dimension is meant to drive the family of parts. Two objects can render identically and still be structurally different designs.
The agent lives in a POMDP
A useful (although intimidating) formal frame is a partially observable Markov decision process, or POMDP. The complete CAD document — geometry, topology, construction history, parameters, references, selections and application state — is the state. The agent does not receive that state directly. It receives observations: screenshots, renders, text summaries, code, tool results and perhaps a current selection.
That distinction matters. Giving the agent a B-rep does not suddenly turn the CAD world into a fully observable MDP. The B-rep is another observation, a rich one, but construction history or application context may still be absent. From those partial observations, the agent maintains a belief state: a probability distribution over the complete states that could still be true. It changes after every new observation or tool result. In CAD, it might express that a selected cylindrical region is probably Face_12, while keeping other candidate faces possible.
The classic formulation by Kaelbling, Littman and Cassandra makes this belief state the sufficient basis for choosing an action. The engineering objective is therefore not to claim full observability, but to create better observations and actions so the agent can refine that belief. Tool-augmented systems make the loop concrete in CAD. CAD-Assistant, for example, uses a vision-language planner with CAD-specific tools, executes commands through FreeCAD’s Python API, then adapts its next action to the changed design. The important behavior is not a single prediction. It is the repeated reduction of uncertainty.
A POMDP loop for CAD
The hidden CAD state produces a partial observation. The agent updates its belief, chooses an action, and receives a new state.
The missing correspondence
The central failure is rarely that one modality contains no useful information. It is that the agent cannot follow a visible region across representations. Which pixels belong to the cylindrical wall? Which B-rep face produced those pixels? What structural description names that face and its neighbors? Which feature or line of code consumes the corresponding entity? What edit is valid there?
Humans close this loop almost without noticing. We click the region, watch it highlight, read “cylindrical face,” check its radius, then open the feature tree to find the hole that created it. Agents commonly receive a screenshot in one message, a flat list of faces in another and code in a third. Even when every item is correct, the action remains ambiguous because identity did not survive the handoff.
Make the object discoverable
The first practical step is modest: expose the CAD or STEP object as a discoverable Python object. A useful initial description need not serialize the entire document. It can report the global bounding-box dimensions, counts and types of faces and edges, basic topology, and stable handles the agent can use in later calls.
Filter geometry, then select the top face
- User
Select the top face of the plate.
- Agent · inspect
candidates = [f for f in part.faces() if f.surface.kind == "plane" and f.normal.z > 0.99] top_face = max(candidates, key=lambda f: f.center.z) - Tool result
1 upward planar face z = 8.00 mm · area = 2286.9 mm² - Agent · act
✓ upper planar face selectedui.select(top_face)
A static agent trace for a rectangular plate with one twelve millimetre through-hole. Without relying on entity identifiers, the agent filters for upward planar surfaces, keeps the highest candidate, and selects the top face in the CAD interface.
From there, inspection becomes active. The agent can ask for the measure and analytic surface behind a face, enumerate its neighbors, render a view in which it is highlighted, compare candidate selections, or inspect structural properties. Each call is an information-gathering action. Instead of forcing one enormous context dump, the interface lets the agent spend attention where uncertainty is highest.
This is not only a data-format choice. It is an agent-interface choice. The best representation is one that supports a sequence of cheap, grounded questions and returns answers tied to persistent entities.
From boundary geometry back to operations
For an imported B-rep, the larger ambition is to reconstruct an executable model. Reverse-engineering research increasingly treats geometry not as the final answer but as evidence for a CAD program. CAD-Recode, by Danila Rukhovich and colleagues at ICCV 2025, translates point clouds into Python CadQuery programs. Inferring CAD Modeling Sequences Using Zone Graphs searches for plausible sketch, extrusion and Boolean sequences that reconstruct a B-rep.
Other work attacks the bridge directly from B-rep structure. eCAD-Net predicts feature-based modeling sequences from a UV-graph representation of a dumb B-rep. CADCL aligns B-rep and parametric-sequence representations through contrastive learning. Earlier, UV-Net showed how continuous face and edge geometry can be learned together with discrete topological adjacency.
If reconstruction yields CadQuery or another executable CAD language, the result creates a navigable bridge from geometry and topology to operations. It is not necessarily a simple mapping of one face to one code line: operations create, split, merge and consume multiple entities, and later features alter earlier boundaries. But code gives the agent named operations, parameters and dependencies it can inspect, rerun and test.
Faber is interested in this as an agent-computer interface problem. Reconstruction quality matters, but so does whether the recovered structure helps an agent answer: what am I looking at, which operation governs it, and what should I change?
Evaluate the edit, not only the shape
A plausible final solid is not enough. A model can reproduce the silhouette while selecting the wrong face, confusing two structurally different targets, or applying an edit to a downstream patch instead of the operation that owns the intent. Comparing final shapes will not catch any of this: the output looks right, and the model is wrong.
A stronger evaluation asks whether the agent can identify the exact face shown, distinguish targets that look alike but occupy different structural roles, link entities to the operations that use them, and explain why a proposed edit applies. The answer should survive another camera angle, another export and a replay of the model.
A CAD agent that only sees the shape is a spectator. The one we want can put its finger on the face, name the operation behind it, and change the right thing.
References and further reading
- Leslie Pack Kaelbling, Michael L. Littman and Anthony R. Cassandra, Planning and Acting in Partially Observable Stochastic Domains, Artificial Intelligence, 1998.
- Danila Rukhovich et al., CAD-Recode: Reverse Engineering CAD Code from Point Clouds, ICCV 2025.
- Xianghao Xu et al., Inferring CAD Modeling Sequences Using Zone Graphs, CVPR 2021.
- Chao Zhang et al., eCAD-Net: Editable Parametric CAD Models Reconstruction from Dumb B-Rep Models Using Deep Neural Networks, Computer-Aided Design, 2025.
- JianFei Liang et al., CADCL: Reconstruct Parametric CAD Models from B-rep via Contrastive Learning, Journal of Computational Design and Engineering, 2025.
- Pradeep Kumar Jayaraman et al., UV-Net: Learning from Boundary Representations, 2020.
- Dimitrios Mallis et al., CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers, ICCV 2025.
- Joseph G. Lambourne et al., BRepNet: A Topological Message Passing System for Solid Models, CVPR 2021.
- Xiang Xu et al., BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry, ACM Transactions on Graphics, 2024.