CADMate

Generating CAD Assembly Plan with Geometric Chain-of-Thought and Spatial Physical Rewards

ACL 2026 Main 🌟
Jiali Chen1,2,3,4, Dingba Fu1,2,3, Xusen Hei1,2,3, Yuhang Liu1,2,3, Yiyang Chen1, Jiayuan Xie4, Wenqi Fan4, Yi Cai2,1,3*,
1School of Software Engineering, South China University of Technology
2Key Laboratory of Big Data and Intelligent Robot, Ministry of Education
3Joint Guangdong-Hong Kong-Macao Research Laboratory of Big Data and Robotic Intelligence, Ministry of Education
4The Hong Kong Polytechnic University
*Corresponding author
Paper arXiv Code

🤗

 Dataset

TLDR: CADMate enables MLLMs to assemble CAD objects from reference images and disassembled parts.

Introduction Case

Overview of different ways (i.e., designer and our CADMate) to CAD assembly.

Abstract

Computer-aided design (CAD) is crucial in prototyping complex 3D objects through precise geometric modeling. In practical design workflows, designers manually define assembly sequences for individual CAD parts, a process that is both time-consuming and expertise-intensive. To address this challenge, we formulate CAD assembly as a parametric action prediction task: given a reference design image and disassembled parts, the model predicts 6-DoF transformations (\ie, actions) to progressively assemble each part. This paradigm enables multimodal large language models (MLLMs) to solve the task through autoregressive action generation. While recent MLLMs demonstrate promising spatial reasoning, they struggle with fine-grained geometric structure understanding and physical collision avoidance during assembly. In this paper, we propose CADMate, an MLLM-based framework for sequential CAD assembly action generation. Our training strategy comprises three stages: (i) CAD domain adaptation for spatial geometry and position understanding, (ii) supervised fine-tuning with geometric chain-of-thought (CoT) reasoning for action generation, and (iii) reinforcement learning with spatial-physical rewards jointly optimize spatial accuracy and collision avoidance. Additionally, we also construct CADBuilder dataset, comprising over 45K CAD assemblies with annotated action sequences. Our experiments demonstrate that CADMate significantly outperforms existing prominent MLLMs (e.g., GPT-5), showing great potential in design applications.

CADBuilder Dataset

The CADBuilder dataset is introduced to support the task of sequential CAD assembly action prediction, where models reconstruct complete 3D objects from disassembled parts by predicting step-by-step 6-DoF transformations. This task is challenging for multimodal large language models (MLLMs) due to:

  • Fine-grained geometric understanding: Accurate interpretation of part shapes and spatial constraints.
  • Sequential assembly reasoning: Multi-step, state-dependent decision making.
  • Physical feasibility: Ensuring collision-free and executable assembly trajectories.

CADBuilder is constructed as a large-scale parametric dataset with the following characteristics:

  • Large-scale and diverse: Over 45K CAD assemblies collected from real-world datasets, covering diverse object structures.
  • Multi-level assembly complexity: Each object contains 2 to 15 parts, including:
    • 2–5 parts: 35,483 samples
    • 6–10 parts: 7,302 samples
    • 11–15 parts: 2,276 samples
  • Rich multimodal representation: Each sample includes a reference image, disassembled parts, and initial 6-DoF poses.
  • Action-centric annotation: Provides assembly action sequences as relative 6-DoF transformations for each part.
  • Physically grounded trajectories: Assembly sequences are generated via physics-based simulation and verified for collision-free feasibility.
  • High-quality rendering: Reference images are rendered with carefully selected viewpoints to preserve structural details.

Overall, CADBuilder serves as a comprehensive benchmark for evaluating multimodal models on geometric reasoning, sequential planning, and physically feasible CAD assembly.



CADBuilder Annotation Pipeline

Annotation Process:
  • Collecting Reference Images for an Assembly
  • Constructing an Assembly Trajectory
  • Building a representation of each part
CADReview Annotation

CADMate Framework

To address the challenging task of sequential CAD assembly, we propose CADMate, a multimodal large language model (MLLM)-based framework that formulates CAD assembly as autoregressive action generation. Given a reference design image and disassembled parts, CADMate predicts step-by-step 6-DoF transformations to progressively assemble the complete object.

CADMate integrates a vision encoder, a large language model, and a vision-language projector, enabling joint reasoning over visual inputs and geometric representations. The model operates in a multi-turn interactive manner, where each step observes the current assembly state and generates the next action for a selected CAD part.

A key component of CADMate is the incorporation of Geometric Chain-of-Thought (Geo-CoT) reasoning, which explicitly models:

  • Part-level geometric descriptions: capturing shape, size, and structural attributes of each component.
  • Assembly constraints: reasoning about how parts fit together based on geometric compatibility.

This structured reasoning enables the model to perform fine-grained spatial understanding and generate physically plausible assembly actions in a sequential decision-making process.

To further ensure realistic assembly behavior, CADMate incorporates a simulation-in-the-loop mechanism, where predicted actions are executed to update the assembly state, and collision detection is performed to enforce physical feasibility.


CADMate Framework Overview


To effectively train the model, we adopt a three-stage training strategy: (i) Spatial Perception Adaptation to enhance geometric understanding and 6-DoF pose awareness in the CAD domain; (ii) Supervised Fine-Tuning with Geometric Chain-of-Thought to learn multi-turn assembly reasoning and action generation; and (iii) Reinforcement Learning with Spatial-Physical Rewards to jointly optimize spatial accuracy and collision-free feasibility.

Extensive experiments demonstrate that CADMate significantly improves geometric reasoning, sequential decision-making, and physical plausibility in CAD assembly, outperforming existing multimodal large language models across varying levels of assembly complexity.

Experiment Results

Some CADMate Case Studies

BibTeX