BARISTA Benchmark Accepted at ICML 2026 Workshop for Advancing Procedural Video Understanding

How well do today’s vision-language models really understand what is happening in a video — not just which objects are visible, but how they are used, how they interact, and how a physical process unfolds over time?

This question is at the center of BARISTA, a new benchmark for compositional visual understanding in procedural videos. The paper, “BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding,” by Patrick Knab, Orgest Xhelili, Inis Buzi, Drago Andres Guggiana Nilo, Mohd Saquib Khan, Lorenz Kolb, Manuel Scherzer, Kerem Yildirir, Christian Bartelt, and Philipp Johannes Schubert, has been accepted at the ICML 2026 Workshop on Combining Theory and Benchmarks: Towards a Virtuous Cycle to Understand and Guarantee Foundation Model Performance.

The workshop focuses on strengthening the connection between theoretical understanding and empirical benchmarking in order to better evaluate, explain, and guarantee the performance of foundation models. BARISTA fits naturally into this theme: rather than offering only a single performance score, it is designed to help identify which parts of procedural video understanding remain difficult for current models.

Abstract:

Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both the state and temporal dynamics of a scene. Yet understanding physical processes remains difficult, as models must combine object localization, hand-object interactions, relational parsing, temporal reasoning, and step-level procedural inference. Existing benchmarks usually evaluate these capabilities separately, limiting diagnosis of why models fail on procedural tasks.

We introduce BARISTA, a densely annotated egocentric dataset and benchmark of 185 real-world coffee-preparation videos covering fully automatic, portafilter-based, and capsule-based workflows. BARISTA provides verified per-frame scene graphs linking persistent object identities to masks, tracks, boxes, attributes, typed relations, hand-object interactions, activities, and process steps.

From these graphs, we derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering. Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding.

Code and dataset are available here.

The full paper can be read here. 

ICML 2026 will take place in Seoul, South Korea, in July 2026.