Benchmark for LLM-based Text-to-BIM systems accepted at EC3 2026

A paper by the research group of Prof. Christian Bartelt, in cooperation with the research group of Prof. Stefan Lüdtke at the University of Rostock, has been accepted for presentation at the European Conference on Computing in Construction, EC3 2026, in Corfu, Greece.

The paper, titled “BIBIMBAP: A Benchmark for Instructional BIM-Based Automated Programming,” introduces a new benchmark for evaluating Large Language Model-based Text-to-BIM systems. The benchmark focuses on how reliably such systems can edit and reason about Building Information Modeling, or BIM, models represented using Industry Foundation Classes, or IFC.

Large Language Models are increasingly being explored as natural-language interfaces for engineering tools. In the context of BIM, this means that users may describe desired changes to a building model in ordinary language, while an AI system translates these instructions into concrete operations on the underlying digital model. However, evaluating whether such systems perform these operations correctly and reliably remains a major challenge.

BIBIMBAP addresses this gap by providing a benchmark with 100 curated tasks covering the full spectrum of Create, Read, Update, and Delete operations. These tasks include spatial, geometric, topological, numeric, and conceptual reasoning challenges. The benchmark also provides reference IFC models, structured expected outputs, and automated evaluation scripts, enabling reproducible assessment of Text-to-BIM systems.

Baseline experiments with several state-of-the-art Large Language Models show that current systems still face substantial challenges in robust BIM manipulation. In particular, the results highlight difficulties with spatial reasoning and with preserving IFC integrity constraints during model editing. By making these challenges measurable, BIBIMBAP aims to support future research on more reliable AI-assisted BIM workflows.

The work contributes to the broader goal of developing AI systems that can interact with complex engineering data in a dependable and verifiable way. It also reflects the vision behind the new AI Engineering bachelor’s program at TU Clausthal: combining artificial intelligence with engineering expertise to develop reliable, practical, and critically evaluated AI-supported systems for real-world applications.

More information about the AI Engineering bachelor’s program is available here.

The paper website is available here.