The best way to convert 2D designs into 3D models for rapid prototyping | MIT News

Engineers often use visual language models to generate new designs, such as aircraft or car parts. To simulate how those parts will perform in real-world situations, they’ll use tried-and-true computer-aided design (CAD) software to generate 3D models of those designs, which they can include in virtual crash or durability tests.

Researchers from MIT and elsewhere have now developed a program that can teach a visual language model to automatically convert 2D designs into CAD programs that are more accurate and efficient compared to other methods, while using only a fraction of the computations.

By improving the performance and efficiency of AI-driven CAD generation, this process can streamline the rapid prototyping process and reduce costs. It can also help developers identify beneficial design choices they might otherwise overlook.

The program generates new data based on the model’s capabilities as it tries to convert the 2D image into a CAD system. The framework corrects the model’s failures and integrates it with a dataset with its successful solutions.

It uses this data to teach the model how to fix certain errors and tackle tricky problems that it would struggle with on its own.

“We want engineers to be able to point our framework to an inefficient CAD model, set a computer budget, and let the system take over – turning model errors into better training data,” said lead author Giorgio Giannone, a research associate in the Design Computation and Digital Engineering (DeCoDE) Lab at MIT and chief scientist of the Red Innovation Research Team at Red Innovation Research.

He is joined on the paper by Anna Claire Doris, a mechanical engineering graduate student at MIT; Amin Heyrani Nobari, MIT postdoc; Kai Xu of RedHat; and senior authors Akash Srivastava, director of Core AI at IBM and principal investigator at the MIT-IBM Computing Research Lab; and Faez Ahmed, associate professor of mechanical engineering at MIT, leader of the DeCoDE Lab, and principal investigator at the MIT-IBM Computing Research Lab. The research was recently presented at the International Conference on Machine Learning.

“Almost every physical product around us, from airplanes to electronics, begins its life as a CAD model. Industry groups are eager for AI to help speed up the creation of these designs, but today’s models often produce simple shapes that are not good enough to do. What I like about this work is that it gives many photo-to-CAD-code models a way to improve themselves – bringing their human errors and delivering their reliable data rather than delivering their ideal data. AI design tools are very close in everyday engineering,” Ahmed said.

Model information data

Researchers are working to develop visual language models (VLMs) for CAD production. These VLMs take a 2D image and some descriptive text, and output Python code that can be executed in CAD software to produce a 3D model of the physical object.

They researched the challenges of deploying existing VLMs in this field and determined the biggest barrier limiting their capabilities was the lack of a diverse, high-quality CAD dataset to train them on.

To fix this, they want to create new data to teach the model how to do CAD generation, using a process known as data augmentation.

In data augmentation, scientists often create new data by randomly adjusting existing data to generate more samples, often by adjusting the color, size, and shape of objects in images.

Instead, MIT researchers developed a data augmentation system called GIFT (which stands for Geometric Inference Feedback Tuning) that generates data designed to improve the performance of a single VLM for a given task.

GIFT improves understanding of a model’s strengths and weaknesses by testing it. It then uses this information to generate data that can improve the model’s performance on problems CAD generations struggle to solve.

“We want to get more data informed by the model itself,” Giannone said.

Learning from mistakes

To do this, GIFT asks the model to generate code that solves the CAD production problem multiple times in parallel. It tests the validity of these assumptions to understand how the model can solve the problem.

“For a model, generating CAD query code is approx the correct one is not that difficult, but generating correct and executable code is a major challenge in standard VLM,” Giannone said.

With near-perfect guesses, GIFT adjusts itself into effective solutions. It saves these “close” and successful solutions to a new dataset that can teach the model how to overcome problems that would normally cripple it.

“If we sample the model 10 times and it produces 10 correct answers to the same problem, then there is not much to learn. We care about the intermediate cases, where the model may solve the problem only 50 percent,” he says.

Using these intermediate conditions allows GIFT to generate data augmentation that is both model-aware and task-aware. In addition, by combining multiple optimal solutions for the same problem, the new data increases the general knowledge of the CAD code generation model.

This automatic system does not require human intervention to correct model errors.

GIFT generates data enhancements from a pre-trained VLM using a technique known as time scale reasoning. This technique allows a static model, which is already trained, to produce better results without the high computational cost of retraining the entire model.

By using the inference-time scale, the user can determine how much computing power he wants to use for GIFT, within his time and budget constraints.

GIFT outperformed several competing techniques, producing CAD programs that were more accurate while using only about 20 percent more calculations. CAD models produced by VLMs using GIFT are better aligned to the conditions of ground truth models.

“With GIFT, we started with geometry because with engineering problems, if the geometry of a 3D shape is wrong, nothing else will be right, but there are many factors to consider,” said Giannone.

In the future, researchers want to extend GIFT so that the framework can teach modelers to produce CAD programs that improve the performance and production of 3D models. They also want to use the system for large models and various CAD production jobs.

This research was funded, in part, by the MIT-IBM Computing Research Lab.