Multimodal Foundation Models for Zero-Shot User Activity Classification: An Exploratory Analysis of Trace Modalities
†Senior author
Abstract
Task mining involves the reconstruction of business processes through the analysis of interaction logs between users and software applications. Identifying and classifying specific user actions from multi-modal telemetry can be challenging, and the performance characteristics of Multimodal Large Language Models (MLLMs) across different trace modalities, specifically structured logs versus visual screenshots, merit further investigation. This study explores five MLLMs (GPT-5, Gemini 3-Pro, Gemini 2.5-Flash, Grok-4-Fast, and Llama-4-Maverick) on their ability to identify user actions within a limited sample of interaction traces. We observed performance across three configurations: structured input logs, visual screenshots, and combined multimodal traces. In this sample, structured input logs appeared to provide more consistent signals for classification than screenshots, which exhibited greater sensitivity to model architecture and higher levels of ambiguity. Notably, models frequently reported high confidence even when producing incorrect predictions, suggesting a potential calibration gap. We characterised observed failure modes, such as premature action inference and the misinterpretation of visual evidence. These results highlight the role of modality selection and the need for confidence validation and cost-accuracy considerations in the development of MLLM-based task mining systems.