Abstract

Conversational agents with GUIs (CAGs) are emerging systems that leverage LLMs to translate user intents into GUI interactions, enabling natural language-driven task automation. However, existing CAGs often fail to capture and leverage evolving user preferences, resulting in repeated misalignments and inefficient navigation. We introduce MAESTRO, a CAG framework that maintains a preference memory that extracts preferences from natural-language utterances with their strength. Based on the preference memory, MAESTRO employs two mechanisms: GUI adaptation that reorganizes interface elements to match user preferences, and workflow navigation that identifies conflicts with preference memory and proposes backtracking to avoid repeating erroneous paths. We evaluated MAESTRO through a user study with 33 participants in movie booking scenarios, demonstrating that preference-aware GUI adaptation and workflow navigation significantly improved task efficiency and user satisfaction compared to existing approaches.