在本文中，我们提出了双工对话，这是一种多型，多模式的口语对话系统，使基于电话的代理能够与人类这样的客户互动。我们在电信中使用全双工的概念来证明人类般的互动体验应该是什么以及如何通过三个子任务实现平稳的转弯：用户状态检测，后拨频选择和驳船检测。此外，我们建议使用多模式数据增强的半监督学习，以利用未标记的数据来增加模型的概括。三个子任务的实验结果表明，与基准相比，所提出的方法可实现一致的改进。我们将双工对话部署到阿里巴巴智能客户服务，并在生产中分享经验教训。在线A/B实验表明，所提出的系统可以将响应潜伏期显着降低50％。

Any organization needs to improve their products, services, and processes. In this context, engaging with customers and understanding their journey is essential. Organizations have leveraged various techniques and technologies to support customer engagement, from call centres to chatbots and virtual agents. Recently, these systems have used Machine Learning (ML) and Natural Language Processing (NLP) to analyze large volumes of customer feedback and engagement data. The goal is to understand customers in context and provide meaningful answers across various channels. Despite multiple advances in Conversational Artificial Intelligence (AI) and Recommender Systems (RS), it is still challenging to understand the intent behind customer questions during the customer journey. To address this challenge, in this paper, we study and analyze the recent work in Conversational Recommender Systems (CRS) in general and, more specifically, in chatbot-based CRS. We introduce a pipeline to contextualize the input utterances in conversations. We then take the next step towards leveraging reverse feature engineering to link the contextualized input and learning model to support intent recognition. Since performance evaluation is achieved based on different ML models, we use transformer base models to evaluate the proposed approach using a labelled dialogue dataset (MSDialogue) of question-answering interactions between information seekers and answer providers.

The goal of building dialogue agents that can converse with humans naturally has been a long-standing dream of researchers since the early days of artificial intelligence. The well-known Turing Test proposed to judge the ultimate validity of an artificial intelligence agent on the indistinguishability of its dialogues from humans'. It should come as no surprise that human-level dialogue systems are very challenging to build. But, while early effort on rule-based systems found limited success, the emergence of deep learning enabled great advance on this topic. In this thesis, we focus on methods that address the numerous issues that have been imposing the gap between artificial conversational agents and human-level interlocutors. These methods were proposed and experimented with in ways that were inspired by general state-of-the-art AI methodologies. But they also targeted the characteristics that dialogue systems possess.

随着经济和社会的增长，企业，尤其是在金融科技行业中，对客户收集，市场营销，反欺诈电话等对客户的需求不断增加。但是，大部分重复性和机械工作都占据了人类代理商的大部分时间，因此企业的设备和劳动力成本正在增加。同时，随着过去几十年来人工智能技术的发展，公司使用大数据和人工智能等新技术来增强呼叫业务的能力已变得非常普遍。智能出站机器人是人工智能技术在出站呼叫业务领域的典型应用。它主要用于与客户交流以实现某个目标。它具有低成本，高额重用和易于合规性的特征，这引起了行业的更多关注。目前，该行业有两种智能出站机器人，但他们俩仍然为改进留下了巨大的空间。其中一种是基于有限状态机，该机器依赖于跳跃条件和基于手动体验的相应节点的配置。这种智能出站机器人也称为基于流的机器人。例如，图\ ref {图：标签}中显示了基于流的机器人的工作模型的示意图。在每个回合中，机器人将用与每个节点相对应的单词回复用户。

Empathy is a vital factor that contributes to mutual understanding, and joint problem-solving. In recent years, a growing number of studies have recognized the benefits of empathy and started to incorporate empathy in conversational systems. We refer to this topic as empathetic conversational systems. To identify the critical gaps and future opportunities in this topic, this paper examines this rapidly growing field using five review dimensions: (i) conceptual empathy models and frameworks, (ii) adopted empathy-related concepts, (iii) datasets and algorithmic techniques developed, (iv) evaluation strategies, and (v) state-of-the-art approaches. The findings show that most studies have centered on the use of the EMPATHETICDIALOGUES dataset, and the text-based modality dominates research in this field. Studies mainly focused on extracting features from the messages of the users and the conversational systems, with minimal emphasis on user modeling and profiling. Notably, studies that have incorporated emotion causes, external knowledge, and affect matching in the response generation models, have obtained significantly better results. For implementation in diverse real-world settings, we recommend that future studies should address key gaps in areas of detecting and authenticating emotions at the entity level, handling multimodal inputs, displaying more nuanced empathetic behaviors, and encompassing additional dialogue system features.

我们提出了Tacobot，这是为首届Alexa Prive Taskbot Challenge构建的面向任务的对话系统，该系统可帮助用户完成多步骤烹饪和家庭装修任务。Tacobot的设计采用以用户为中心的原则，并渴望提供协作且易于访问的对话体验。为此，它具有准确的语言理解，灵活的对话管理和引人入胜的响应生成。此外，Tacobot还以强大的搜索引擎和自动化的端到端测试套件为支持。在引导Tacobot的开发中，我们探索了一系列数据增强策略，以训练先进的神经语言处理模型，并通过收集的真实对话不断改善对话经验。在半决赛结束时，Tacobot的平均评分为3.55/5.0。

Recent advances in neural approaches greatly improve task-oriented dialogue (TOD) systems which assist users to accomplish their goals. However, such systems rely on costly manually labeled dialogs which are not available in practical scenarios. In this paper, we present our models for Track 2 of the SereTOD 2022 challenge, which is the first challenge of building semi-supervised and reinforced TOD systems on a large-scale real-world Chinese TOD dataset MobileCS. We build a knowledge-grounded dialog model to formulate dialog history and local KB as input and predict the system response. And we perform semi-supervised pre-training both on the labeled and unlabeled data. Our system achieves the first place both in the automatic evaluation and human interaction, especially with higher BLEU (+7.64) and Success (+13.6\%) than the second place.

多模式意图识别是理解现实世界中人类语言的重要任务。大多数现有意图识别方法在利用基准数据集的限制中利用多模式信息的局限性，仅使用文本信息。本文介绍了一个用于多模式意图识别（MinTreec）的新型数据集，以解决此问题。它根据电视系列超市收集的数据制定了粗粒和细粒度的分类法。该数据集由2,224个具有文本，视频和音频模式的高质量样本组成，并在二十个意图类别中具有多模式注释。此外，我们在每个视频段中提供带注释的扬声器框架框，并实现扬声器注释的自动过程。 MinTrec对研究人员有助于挖掘不同方式之间的关系，以增强意图识别的能力。我们通过适应三种强大的多模式融合方法来构建基准，从每种模式和模型跨模式相互作用中提取特征。广泛的实验表明，采用非语言方式与仅文本模式相比，实现了实质性改进，这表明使用多模式信息进行意图识别的有效性。表现最佳的方法与人类之间的差距表明了这项任务对社区的挑战和重要性。完整的数据集和代码可在https://github.com/thuiar/mintrec上使用。

人类通过不同的渠道表达感受或情绪。以语言为例，它在不同的视觉声学上下文下需要不同的情绪。为了精确了解人类意图，并减少歧义和讽刺引起的误解，我们应该考虑多式联路信号，包括文本，视觉和声学信号。至关重要的挑战是融合不同的特征模式以进行情绪分析。为了有效地融合不同的方式携带的信息，更好地预测情绪，我们设计了一种基于新的多主题的融合网络，这是由任何两个对方式之间的相互作用不同的观察来启发，它们是不同的，并且它们不同样有助于最终的情绪预测。通过分配具有合理关注和利用残余结构的声学 - 视觉，声学 - 文本和视觉文本特征，我们参加了重要的特征。我们对四个公共多模式数据集进行了广泛的实验，包括中文和三种英文中的一个。结果表明，我们的方法优于现有的方法，并可以解释双模相互作用在多种模式中的贡献。

Recently, spoken dialogue systems have been widely deployed in a variety of applications, serving a huge number of end-users. A common issue is that the errors resulting from noisy utterances, semantic misunderstandings, or lack of knowledge make it hard for a real system to respond properly, possibly leading to an unsatisfactory user experience. To avoid such a case, we consider a proactive interaction mechanism where the system predicts the user satisfaction with the candidate response before giving it to the user. If the user is not likely to be satisfied according to the prediction, the system will ask the user a suitable question to determine the real intent of the user instead of providing the response directly. With such an interaction with the user, the system can give a better response to the user. Previous models that predict the user satisfaction are not applicable to DuerOS which is a large-scale commercial dialogue system. They are based on hand-crafted features and thus can hardly learn the complex patterns lying behind millions of conversations and temporal dependency in multiple turns of the conversation. Moreover, they are trained and evaluated on the benchmark datasets with adequate labels, which are expensive to obtain in a commercial dialogue system. To face these challenges, we propose a pipeline to predict the user satisfaction to help DuerOS decide whether to ask for clarification in each turn. Specifically, we propose to first generate a large number of weak labels and then train a transformer-based model to predict the user satisfaction with these weak labels. Empirically, we deploy and evaluate our model on DuerOS, and observe a 19% relative improvement on the accuracy of user satisfaction prediction and 2.3% relative improvement on user experience.

Customers are rapidly turning to social media for customer support. While brand agents on these platforms are motivated and well-intentioned to help and engage with customers, their efforts are often ignored if their initial response to the customer does not match a specific tone, style, or topic the customer is aiming to receive. The length of a conversation can reflect the effort and quality of the initial response made by a brand toward collaborating and helping consumers, even when the overall sentiment of the conversation might not be very positive. Thus, through this study, we aim to bridge this critical gap in the existing literature by analyzing language's content and stylistic aspects such as expressed empathy, psycho-linguistic features, dialogue tags, and metrics for quantifying personalization of the utterances that can influence the engagement of an interaction. This paper demonstrates that we can predict engagement using initial customer and brand posts.

In this paper, we perform an exhaustive evaluation of different representations to address the intent classification problem in a Spoken Language Understanding (SLU) setup. We benchmark three types of systems to perform the SLU intent detection task: 1) text-based, 2) lattice-based, and a novel 3) multimodal approach. Our work provides a comprehensive analysis of what could be the achievable performance of different state-of-the-art SLU systems under different circumstances, e.g., automatically- vs. manually-generated transcripts. We evaluate the systems on the publicly available SLURP spoken language resource corpus. Our results indicate that using richer forms of Automatic Speech Recognition (ASR) outputs allows SLU systems to improve in comparison to the 1-best setup (4% relative improvement). However, crossmodal approaches, i.e., learning from acoustic and text embeddings, obtains performance similar to the oracle setup, and a relative improvement of 18% over the 1-best configuration. Thus, crossmodal architectures represent a good alternative to overcome the limitations of working purely automatically generated textual data.

在与用户进行交流时，以任务为导向的对话系统必须根据对话历史记录在每个回合时跟踪用户的需求。这个称为对话状态跟踪（DST）的过程至关重要，因为它直接告知下游对话政策。近年来，DST引起了很大的兴趣，文本到文本范式作为受欢迎的方法。在本评论论文中，我们首先介绍任务及其相关的数据集。然后，考虑到最近出版的大量出版物，我们确定了2021 - 2022年研究的重点和研究进展。尽管神经方法已经取得了重大进展，但我们认为对话系统（例如概括性）的某些关键方面仍未得到充实。为了激励未来的研究，我们提出了几种研究途径。

Artificial intelligence and natural language processing (NLP) are increasingly being used in customer service to interact with users and answer their questions. The goal of this systematic review is to examine existing research on the use of NLP technology in customer service, including the research domain, applications, datasets used, and evaluation methods. The review also looks at the future direction of the field and any significant limitations. The review covers the time period from 2015 to 2022 and includes papers from five major scientific databases. Chatbots and question-answering systems were found to be used in 10 main fields, with the most common use in general, social networking, and e-commerce areas. Twitter was the second most commonly used dataset, with most research also using their own original datasets. Accuracy, precision, recall, and F1 were the most common evaluation methods. Future work aims to improve the performance and understanding of user behavior and emotions, and address limitations such as the volume, diversity, and quality of datasets. This review includes research on different spoken languages and models and techniques.

通过整合人类的知识和经验，人在循环旨在以最低成本培训准确的预测模型。人类可以为机器学习应用提供培训数据，并直接完成在基于机器的方法中对管道中计算机中的难以实现的任务。在本文中，我们从数据的角度调查了人类循环的现有工作，并将它们分为三类具有渐进关系：（1）从数据处理中提高模型性能的工作，（2）通过介入模型培训提高模型性能，（3）系统的设计独立于循环的设计。使用上述分类，我们总结了该领域的主要方法;随着他们的技术优势/弱点以及自然语言处理，计算机愿景等的简单分类和讨论。此外，我们提供了一些开放的挑战和机遇。本调查打算为人类循环提供高级别的摘要，并激励有兴趣的读者，以考虑设计有效的循环解决方案的方法。

生成的开放域对话系统可以从外部知识中受益，但是缺乏外部知识资源和寻找相关知识的困难限制了该技术的发展。为此，我们使用动态服务信息提出了一个知识驱动的对话任务。具体而言，我们使用大量的服务API，可以作为外部知识来源提供高覆盖范围和时空敏感性。对话系统生成查询以请求外部服务以及用户信息，获取相关知识，并基于此知识生成响应。为了实现此方法，我们收集并发布了第一个开放式域中国服务知识对话数据集Dusinc。同时，我们构建了一个基线模型柏拉图 - 线，该模型实现了对话的自动利用。自动评估和人类评估都表明，我们提出的新方法可以显着改善开放域对话的效果，并且与对话预培训模型Plato-2相比，人类评估中的会话级总数提高了59.29％。数据集和基准模型将被开源。

在口头对话系统中，我们的目标是部署人工智能，以建立可以与人类交流的自动化对话剂。对话系统越来越多地旨在超越仅仅模仿对话，而且随着时间的推移，这些交互也会改善。在本次调查中，我们概述了多年来制定对话系统的方法的广泛概述。对话系统的不同用例范围从基于任务的系统到开放域聊天动机和需要特定的系统。从简单的规则的系统开始，研究已经朝着越来越复杂的建筑培训，这些建筑在大规模的数据集语料库中培训，如深度学习系统。激进了类似人类对话的直觉，通过加强学习将情绪纳入自然语言发生器的进展。虽然我们看到对某些指标的高度边际改善的趋势，但我们发现指标存在有限的理由，评估实践并不统一。要得出结论，我们标志着这些问题并突出了可能的研究方向。

在口语对话中构建强大的对话系统比书面对话更具挑战。在这方面，提出了DSTC10-TRACK2-TASK2，旨在构建以任务为导向的对话（TOD）系统，该系统将非结构化的外部知识结合在口语对话中，从而扩展了DSTC9-TRACK1。本文介绍了我们的系统，其中包含四种高级方法：数据构建，负面抽样，训练后和样式转移。我们首先自动构建大型培训数据，因为DSTC10-TRACK2未发布官方培训集。对于知识选择任务，我们提出了加权负抽样，以更加细粒度训练模型。我们还采用后培训和样式转移来制作响应生成任务，以生成具有与目标响应类似样式的适当响应。在实验中，我们研究了加权负抽样，训练后和样式转移的效果。我们的模型在客观评估中排名16个团队中的7个，在人类评估中排名6。

意图检测是对话助手的任何自然语言理解（NLU）系统的关键部分。对于存在多个指令和意图的电子邮件对话，检测正确的意图是必不可少的，但很难。在这种设置中，对话上下文可以成为检测助手的用户请求的关键歧义因素。合并上下文的一种突出方法是建模过去的对话历史，例如以任务为导向的对话模型。但是，电子邮件对话的性质（长形式）限制了直接使用面向任务的对话模型中最新进展。因此，在本文中，我们提供了一个有效的转移学习框架（EMTOD），该框架允许对话模型中的最新开发方式用于长形式的对话。我们表明，提出的EMTOD框架将预训练的语言模型的意图检测性能提高了45％，而预先培训的对话模型则提高了30％，以实现任务为导向的电子邮件对话。此外，提出的框架的模块化性质允许在预训练的语言和面向任务的对话模型中为未来的任何发展提供插件。

旨在用自然语言和谐地与人类交流的智能对话体系对于促进人工智能时代的人机互动的发展非常出色。有了逐渐复杂的人类计算机交互要求（例如，多模式输入，时间敏感性），传统的基于文本的对话系统很难满足对更加生动和方便的交互的需求。因此，视觉背景增强对话系统（VAD）有可能通过感知和理解多模式信息（即图像或视频中的视觉上下文，文本对话历史记录）与人类进行交流，已成为主要的研究范式。 VAD受益于视觉和文本上下文之间的一致性和互补性，具有产生引人入胜和背景感知响应的潜力。为了描述VAD的开发，我们首先表征VAD的概念和独特功能，然后介绍其通用系统体系结构以说明系统工作流程。随后，对一些研究挑战和代表性作品进行了详细研究，然后进行了权威基准摘要。我们通过提出一些开放问题和有前途的VAD研究趋势来结束本文，例如，在跨模式对话环境下，人机对话的认知机制以及知识增强的跨模式语义互动。