GRILLBot: An Assistant for Real-World Tasks with Neural Semantic Parsing and Graph-Based Representations

We present GRILLBot, a task-oriented multimodal conversational assistant developed during the 2021/2022 Alexa Prize TaskBot Challenge Gemmell et al. (2022). GRILLBot aims to be an open research platform for complex tasks and supports flexible graph-based task representations, contextual semantic parsing, and incorporates image and video content for clarity and instruction. We release the core components of the system as OAT¹¹1https://github.com/grill-lab/OAT (Open Assistant Toolkit).

GRILLBot is still deployed throughout the United States with users able to invoke the bot by issuing the command “Hey Alexa, Assist me” to their voice-only or screened Alexa device. Our system provides open-ended assistance focusing in the domains of cooking and home improvement. It guides the user through all phases of the task, from performing preference elicitation to guiding a user to a relevant task from large task corpora, (i.e. “making a New York-style pizza" or "how to paint a wall”) and then proceeds to assist in executing the task in an engaging way. Its capabilities include question answering, task-oriented chit-chat, and instructional video content.

GRILLBot is part of the first generation of assistants Ipek et al. that leverage screen-enabled conversational devices for complex real-world tasks. These tasks are extensive, with some taking over an hour. As a result, a performant system requires long-term state tracking with capabilities to adapt to a changing environment. It achieves this by introducing a new novel task structure, a TaskGraph, that captures the actions and information dependencies to guide the user through a complex task. TaskGraphs are enriched offline with content from information extraction, knowledge-based content, and multimedia images and videos.

Traditional task-oriented dialogue systems Young et al. (2013) take a slot-filling approach to deriving system actions. Academic datasets such as MultiWOZ Budzianowski et al. (2018) capture slot-value pairs from the user utterances within a constrained set of domains enabling data-driven neural models. Andreas et al. (2020) extend this traditional representation towards semantic parsing with dataflow graphs while constrained to the domain of events booking in the SMCalFlow dataset. The neural decision parser in GRILLBot similarly generates code but focused on all aspects of a conversation from navigation to task search and question answering. Other challenges such as DSTC11 Kottur et al. (2021) attempt this fully featured task-oriented experience, yet only do so in a virtual setting. GRILLBot stands apart as a system required to engage with real-world users in their environment and assist in complex tasks for cooking and home improvement.

The system uses a micro-service architecture with a centralized Orchestrator that defines the system behavior. We use a phase-based policy to transition from searching for a suitable task (i.e. planning phase) to guiding users in performing a task (i.e. execution phase).

The Orchestrator is the central process that directs and receives all the information from other microservices. Specifically, these child components are called functionalities and provide the necessary tools required by policies during conversations. The main components inside functionalities are: Neural Decision Parser, Task Searcher and Question Answering. We discuss the Neural Decision Parser in Section 4.

The task searcher leverages our collection of TaskGraphs to find candidate tasks. Our approach is based on a combination of traditional sparse & dense retrieval and neural re-ranking Gemmell et al. (2022). The question answering system provides extra task information, handles user questions, and provides chit-chat elements. It uses a collection of QA systems across six categories.

A TaskGraph is a new graph-based representation based on a directed acyclic graph that encodes the actions and information dependencies that the system needs to enable complex dialogue flows. Information is represented with heterogeneous nodes, each with a specific role:

• Steps: Represent a task instruction for the user, including visual information and textual descriptions.

• Requirements: Represent tools and ingredients that are needed to perform the task and can be grounded to specific steps.

• Conditions: Represents yes/no gates that require external information to resolve during execution dynamically.

• Logic: Represents logical operations. These can be used in conjunction with other nodes to enable compact dependencies and smoother execution flows. We currently support $\wedge$, $\vee$ and $\neg$ operations.

• Actions: Represents actions taken by the system. This could be operations like setting a timer or adding items to a list.

• Extra Information: Represents domain/task-specific knowledge like tips or fun facts that can enrich the user experience during the execution of the task.

Combining all these nodes, we can obtain adaptive interactions where system-initiative allows the system to adapt to the user’s needs. Figure 3 shows how TaskGraphs can be leveraged using a task-oriented conversation helping a user cooking “creamy zucchini pasta.”

TaskGraphs allows complex representations of real-world tasks. Due to the complex conversational dialogue required, traditional non-contextual intent classifiers struggle to manage stateful transitions. For this reason, we develop a Neural Decision Parser that leverages both TaskGraphs and user history for contextualized semantic parsing. Specifically, the model takes in natural language representations of a TaskGraph and prior conversational context to generate actions in the form of the custom GRILLBot Domain Specific Language (DSL). These generated arguments supply the task sub-components with parsed knowledge relating to the conversation.

For example, if a user asks “Can you go back to the first step?”, the Neural Decision Parser would generate a parameterized parse step_select(1).

This demo presents GRILLBot, a newly developed Alexa Prize Taskbot system for complex real-world tasks with rich multimodal capability. It demonstrates multiple novel components including TaskGraphs to manage long complex tasks that are automatically enriched with offline process to add multimodal image and instructional video content. It also shows key elements of the system that make it engaging, including its flexible Neural Decision Parser that performs contextual semantic parsing as parametrized code generation. The result is a demonstration of a new research platform designed from the ground-up around around flexible cloud micro-services and large-scale neural language models.