Erfolgreiche Promotion von Dr. Kristian Kolthoff

Wir freuen uns, bekanntzugeben, dass Kristian Kolthoff seine Promotion erfolgreich verteidigt hat. Seine Dissertation trägt den Titel: 

„From Natural Language Requirements to Graphical User Interfaces: Automated Prototyping and Verification with Pretrained Language Models"

In seiner Arbeit untersuchte er, wie vortrainierte Sprachmodelle genutzt werden können, um aus natürlichsprachlichen Anforderungen automatisch GUI-Prototypen zu generieren und zu verifizieren – ein bedeutender Beitrag zur Automatisierung der Softwareentwicklung.

Wir freuen uns besonders, dass Dr. Kolthoff dem Lehrstuhl von Prof. Christian Bartelt als Postdoktorand erhalten bleibt und seine Forschung hier weiter vorantreiben wird. Das gesamte Team gratuliert herzlich und freut sich auf die weitere Zusammenarbeit!

English Version

The dissertation addresses a central challenge in requirements engineering and software development: how natural language requirements can be translated into graphical user interfaces and how these interfaces can be verified automatically. The full English abstract of the thesis is included below:

Requirements elicitation is a crucial activity before and during the development of interactive software systems, ensuring that the resulting product meets the needs of the stakeholders. Typically, the process of requirements elicitation relies heavily on natural language (NL) communication between requirements analysts and stakeholders, being prone to misunderstandings due to the inherent ambiguity of NL. While formal software specifications serve as ambiguity mitigation, they necessitate technical knowledge for proper interpretation. Therefore, Graphical User Interface (GUI) prototyping has emerged as an effective requirements elicitation and validation technique, providing tangible, visual artifacts representing the implemented requirements and acting as a common language between analysts and stakeholders. In particular, employing GUI prototyping improves the elicitation and validation of requirements by enabling fruitful discussions and integrating stakeholders more closely into the development process. Although GUI prototyping offers these advantages in comparison to other elicitation and validation approaches, it simultaneously necessitates increased manual effort, which is time-consuming and costly, especially for high-fidelity GUI prototypes that enable stakeholders to provide feedback of higher quality. While requirements validation ensures that the software system aligns with the actual needs of the stakeholders, verification of requirements is equally important, ensuring that the implementation of requirements in the software system and prototypes aligns with the specification. However, similar to validation, verification is still often a manual and effort-demanding task, while existing solutions offer mostly merely rule-based, static techniques and approaches.

In this work, we propose several approaches to address the mentioned challenges, namely (C1) reducing the effort required to transform NL requirements into GUI prototypes to facilitate requirements elicitation and validation, and (C2) reducing the effort required for conducting requirements verification in GUI applications and prototypes. Specifically, we tackle the first challenge (C1) by initially presenting novel NL-based GUI retrieval and reranking approaches, enabling to rapidly map NL requirements to tangible GUI prototypes, while creating new benchmarks and achieving state-of-the-art performance. Furthermore, we propose several methods for efficiently adapting LLMs to GUI generation and devise novel techniques to efficiently enable LLM-based generation for proprietary GUI representations, allowing to quickly translate NL requirements into customizable GUI prototypes. We evaluated these techniques on a novel benchmark encompassing an extensive collection of human annotations created via crowdsourcing, showing their high effectiveness for GUI generation. To tackle the second challenge (C2) we initially present novel LLM-based methods enabling the verification of semantically complex NL requirements on static GUI prototypes. To assess their performance, we created a high-quality benchmark, which links NL requirements to GUI prototypes, indicating high effectiveness for requirements verification on GUI prototypes. Moreover, we propose a multi-modal (M)LLM-based agent approach that enables verification of complex functional and non-functional NL requirements of fully-fledged, highly dynamic GUI prototypes and applications by automatically creating and evaluating meaningful interaction trajectories, showing high effectiveness on a novel benchmark of human-annotated NL requirements and GUI applications. To summarize, by providing effective methods for rapidly mapping NL requirements to GUI prototypes, we facilitate automated GUI prototyping and substantially increase prototyping productivity, positively impacting elicitation and validation activities. Moreover, with our novel and effective MLLM-based agent approach, we substantially improve and facilitate the automated verification of complex NL requirements in GUIs.