Abstract
Augmented reality (AR) instructions help people understand how to operate or assemble objects by overlaying visual guidance directly onto the physical environment. However, repeatedly presenting detailed instructions can negatively impact performance. We introduce an interactive AR instruction system that groups instructions based on recurring action patterns, providing detailed guidance on the first occurrence of actions and abstracting repeated sequences. We introduce a novel workflow that automatically organizes narrated expert demonstrations into instruction structures using a multimodal large language model, improving the scalability of AR guidance systems. Our user study shows that grouped instruction visualization is clearly preferred.
Citation
@proceedings{ribeiroskreinig2026ActionSummaries,
author={Skreinig, Lucchas Ribeiro and Stanescu, Ana and Mohr, Peter and Ebner, Christoph and Tatzgern, Markus and Schmalstieg, Dieter and Kalkofen, Denis},
title={Augmented Reality Action Summaries for Situated Instructions of Repetitive Tasks},
booktitle={IEEE Int. Symp. on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
doi={10.1109/ISMAR-Adjunct73196.2026.00190}
}