In a groundbreaking move that blends creativity, engineering, and AI, researchers from Carnegie Mellon University have unveiled LegoGPT — a powerful large language model (LLM) that can generate complete LEGO designs from simple text prompts. And the best part? It’s now available for free on GitHub.

Build with Words: The Power of LegoGPT

Imagine typing “build a LEGO spaceship with four engines and a cockpit” — and watching an AI instantly design a valid, stable, and buildable model. That’s what LegoGPT offers.

Trained on a massive dataset of over 47,000 LEGO structures and more than 28,000 unique 3D models, the AI understands how to predict and place the next brick using autoregressive modeling, the same concept behind how ChatGPT predicts words.

From guitars and cars to bookshelves and chairs, LegoGPT can build almost anything using text alone. The model was designed to always output physically stable and non-overlapping structures, thanks to built-in validity checks and a rollback feature for unstable designs.

A Smart LEGO Companion — Built for Everyone

But this is more than just a toy for AI geeks. With LegoGPT, you can:

• Use your own bricks: Take a photo of your existing LEGO collection and let a paired vision model design something new from what you already have.

• Get full instructions: The AI pairs structured LEGO designs with annotations and step-by-step logic, making it possible for both humans and robots to follow.

• Fork and experiment: The StableText2Lego dataset, the codebase, and the models are all open-source — giving hobbyists and developers a chance to build on it.

How Does LegoGPT Work?

1. A text prompt is entered (e.g., “a colorful LEGO bridge”).

2. The prompt is translated into a 3D mesh using ShapeNetCore.

3. The mesh is voxelized and used to generate a starting LEGO layout.

4. Designs are tested for physical feasibility and filtered.

5. GPT-4o helps generate natural language descriptions.

6. The AI predicts and places each brick while checking for stability.

If the build becomes unstable mid-way, LegoGPT rolls back to the last stable state and resumes until a final design is completed.

What’s Next?

This research unlocks a world where anyone can design and build with LEGO—no CAD skills required. While currently hosted on GitHub, developers are already eyeing possibilities like a drag-and-drop app, AI-powered instruction manuals, or even robotics integration.

Whether you're a LEGO hobbyist, a teacher introducing STEM, or an AI enthusiast, LegoGPT is a brilliant example of how language models and physical creativity can come together.