AI and text inputs are used in LegoGPT to create Lego designs

A research team from Carnegie Mellon University has developed an innovative AI model called LegoGPT, designed to generate fully functional LEGO structures from simple text descriptions. As outlined in their research published on GitHub, LegoGPT is an autoregressive language model trained to predict and place the next LEGO brick based on prior placements. The standout feature is that it builds unique LEGO creations entirely from scratch, using only a text prompt.

The model was trained on a large dataset comprising over 47,000 LEGO builds that together form more than 28,000 distinct 3D models-ranging from furniture like bookshelves and chairs to vehicles, ships, and musical instruments. This vast dataset equips LegoGPT with the ability to produce creative, never-before-seen LEGO designs in response to user inputs.

LegoGPT is open-source and available on GitHub, making it accessible for public experimentation. You can even integrate it with image recognition software to further enhance its capabilities-for example, by taking a photo of your existing LEGO pieces and receiving custom design suggestions based on what you already own.

To ensure the outputs are always practical, the team implemented a validity checker and physics-aware rollback system. This means the AI checks each step during the building process to ensure bricks do not overlap and that the structure remains physically stable. If a part of the structure is deemed unstable, the model reverts to the last known good state and continues building from there. A robust, constructable LEGO model that can be used by both humans and robots is always the end product.

The StableText2Lego dataset, which drives LegoGPT, was also produced by the team. A text input is first turned into a 3D mesh using ShapeNetCore, which is then fit into a 20x20x20 voxel grid to lay out the initial brick structure. Multiple variations are produced while preserving the core shape, and only stable designs are retained. These models are rendered from 24 angles, and GPT-4o is used to generate detailed text descriptions.

The text input is converted by LegoGPT into tokenized brick-building instructions from the bottom up during generation. It evaluates each brick for validity, checks its place within the available parts library, and ensures there’s no structural conflict. The building continues until a full, stable design is reached.

For developers and hobbyists, the research team has shared the full dataset, code, and models-opening the door to further customization, including the possibility of developing a standalone app with personalized LEGO brick libraries.