Abstractive dialogue summarization has long been viewed as an important standalone task in natural language processing, but no previous work has explored the possibility of whether abstractive dialogue summarization can also be used as a means to boost an NLP system's performance on other important dialogue comprehension tasks. In this paper, we propose a novel type of dialogue summarization task - STRUctured DiaLoguE Summarization - that can help pre-trained language models to better understand dialogues and improve their performance on important dialogue comprehension tasks. We further collect human annotations of STRUDEL summaries over 400 dialogues and introduce a new STRUDEL dialogue comprehension modeling framework that integrates STRUDEL into a graph-neural-network-based dialogue reasoning module over transformer encoder language models to improve their dialogue comprehension abilities. In our empirical experiments on two important downstream dialogue comprehension tasks - dialogue question answering and dialogue response prediction - we show that our STRUDEL dialogue comprehension model can significantly improve the dialogue comprehension performance of transformer encoder language models.

The advance of computer-aided detection systems using deep learning opened a new scope in endoscopic image analysis. However, the learning-based models developed on closed datasets are susceptible to unknown anomalies in complex clinical environments. In particular, the high false positive rate of polyp detection remains a major challenge in clinical practice. In this work, we release the FPPD-13 dataset, which provides a taxonomy and real-world cases of typical false positives during computer-aided polyp detection in real-world colonoscopy. We further propose a post-hoc module EndoBoost, which can be plugged into generic polyp detection models to filter out false positive predictions. This is realized by generative learning of the polyp manifold with normalizing flows and rejecting false positives through density estimation. Compared to supervised classification, this anomaly detection paradigm achieves better data efficiency and robustness in open-world settings. Extensive experiments demonstrate a promising false positive suppression in both retrospective and prospective validation. In addition, the released dataset can be used to perform 'stress' tests on established detection systems and encourages further research toward robust and reliable computer-aided endoscopic image analysis. The dataset and code will be publicly available at http://endoboost.miccai.cloud.

Lack of factual correctness is an issue that still plagues state-of-the-art summarization systems despite their impressive progress on generating seemingly fluent summaries. In this paper, we show that factual inconsistency can be caused by irrelevant parts of the input text, which act as confounders. To that end, we leverage information-theoretic measures of causal effects to quantify the amount of confounding and precisely quantify how they affect the summarization performance. Based on insights derived from our theoretical results, we design a simple multi-task model to control such confounding by leveraging human-annotated relevant sentences when available. Crucially, we give a principled characterization of data distributions where such confounding can be large thereby necessitating the use of human annotated relevant sentences to generate factual summaries. Our approach improves faithfulness scores by 20\% over strong baselines on AnswerSumm \citep{fabbri2021answersumm}, a conversation summarization dataset where lack of faithfulness is a significant issue due to the subjective nature of the task. Our best method achieves the highest faithfulness score while also achieving state-of-the-art results on standard metrics like ROUGE and METEOR. We corroborate these improvements through human evaluation.

We propose an interactive editing method that allows humans to help deep neural networks (DNNs) learn a latent space more consistent with human knowledge, thereby improving classification accuracy on indistinguishable ambiguous data. Firstly, we visualize high-dimensional data features through dimensionality reduction methods and design an interactive system \textit{SpaceEditing} to display the visualized data. \textit{SpaceEditing} provides a 2D workspace based on the idea of spatial layout. In this workspace, the user can move the projection data in it according to the system guidance. Then, \textit{SpaceEditing} will find the corresponding high-dimensional features according to the projection data moved by the user, and feed the high-dimensional features back to the network for retraining, therefore achieving the purpose of interactively modifying the high-dimensional latent space for the user. Secondly, to more rationally incorporate human knowledge into the training process of neural networks, we design a new loss function that enables the network to learn user-modified information. Finally, We demonstrate how \textit{SpaceEditing} meets user needs through three case studies while evaluating our proposed new method, and the results confirm the effectiveness of our method.

The security of artificial intelligence (AI) is an important research area towards safe, reliable, and trustworthy AI systems. To accelerate the research on AI security, the Artificial Intelligence Security Competition (AISC) was organized by the Zhongguancun Laboratory, China Industrial Control Systems Cyber Emergency Response Team, Institute for Artificial Intelligence, Tsinghua University, and RealAI as part of the Zhongguancun International Frontier Technology Innovation Competition (https://www.zgc-aisc.com/en). The competition consists of three tracks, including Deepfake Security Competition, Autonomous Driving Security Competition, and Face Recognition Security Competition. This report will introduce the competition rules of these three tracks and the solutions of top-ranking teams in each track.

Perceiving and manipulating objects in a generalizable way has been actively studied by the computer vision and robotics communities, where cross-category generalizable manipulation skills are highly desired yet underexplored. In this work, we propose to learn such generalizable perception and manipulation via Generalizable and Actionable Parts (GAParts). By identifying and defining 9 GAPart classes (e.g. buttons, handles, etc), we show that our part-centric approach allows our method to learn object perception and manipulation skills from seen object categories and directly generalize to unseen categories. Following the GAPart definition, we construct a large-scale part-centric interactive dataset, GAPartNet, where rich, part-level annotations (semantics, poses) are provided for 1166 objects and 8489 part instances. Based on GAPartNet, we investigate three cross-category tasks: part segmentation, part pose estimation, and part-based object manipulation. Given the large domain gaps between seen and unseen object categories, we propose a strong 3D segmentation method from the perspective of domain generalization by integrating adversarial learning techniques. Our method outperforms all existing methods by a large margin, no matter on seen or unseen categories. Furthermore, with part segmentation and pose estimation results, we leverage the GAPart pose definition to design part-based manipulation heuristics that can generalize well to unseen object categories in both simulation and real world. The dataset and code will be released.

Vision Transformers (ViTs) have achieved state-of-the-art performance on various vision tasks. However, ViTs' self-attention module is still arguably a major bottleneck, limiting their achievable hardware efficiency. Meanwhile, existing accelerators dedicated to NLP Transformers are not optimal for ViTs. This is because there is a large difference between ViTs and NLP Transformers: ViTs have a relatively fixed number of input tokens, whose attention maps can be pruned by up to 90% even with fixed sparse patterns; while NLP Transformers need to handle input sequences of varying numbers of tokens and rely on on-the-fly predictions of dynamic sparse attention patterns for each input to achieve a decent sparsity (e.g., >=50%). To this end, we propose a dedicated algorithm and accelerator co-design framework dubbed ViTCoD for accelerating ViTs. Specifically, on the algorithm level, ViTCoD prunes and polarizes the attention maps to have either denser or sparser fixed patterns for regularizing two levels of workloads without hurting the accuracy, largely reducing the attention computations while leaving room for alleviating the remaining dominant data movements; on top of that, we further integrate a lightweight and learnable auto-encoder module to enable trading the dominant high-cost data movements for lower-cost computations. On the hardware level, we develop a dedicated accelerator to simultaneously coordinate the enforced denser/sparser workloads and encoder/decoder engines for boosted hardware utilization. Extensive experiments and ablation studies validate that ViTCoD largely reduces the dominant data movement costs, achieving speedups of up to 235.3x, 142.9x, 86.0x, 10.1x, and 6.8x over general computing platforms CPUs, EdgeGPUs, GPUs, and prior-art Transformer accelerators SpAtten and Sanger under an attention sparsity of 90%, respectively.

开放设定的半监督学习（OSSL）引起了人们日益增长的兴趣，该学习调查了一个更实用的情况，在该情况下，仅在未标记的数据中包含了分布式（OOD）样本。现有的OSSL方法（例如OpenMatch）学习一个OOD检测器以识别离群值，该检测器通常会更新所有模态参数（即完整的微调），以从标记的数据传播类信息到未标记的数据。当前，已经开发了及时的学习来弥合预训练和微调之间的差距，这在几个下游任务中显示出较高的计算效率。在本文中，我们提出了一个迅速驱动的有效OSSL框架，称为OpenPrompt，该框架可以将类别的类信息传播到标记到未标记数据的类信息，只有少数可训练的参数。我们提出了一种迅速驱动的关节空间学习机制来检测OOD数据，通过在未标记的数据中最大化ID和OOD样本之间的分布差距，从而使我们的方法可以以新的方式检测到异常值。三个公共数据集的实验结果表明，OpenPrompt优于不到1％可训练参数的最先进方法。更重要的是，OpenPrompt在CIFAR10上完全监督模型的AUROC检测方面取得了4％的改善。

图像文本检索（ITR）在桥接视觉和舌形式方面具有挑战性。对比度学习已被大多数先前的艺术所采用。除了有限的负面图像文本对外，约束学习的能力受到手动加权负对以及对外部知识的不认识的限制。在本文中，我们提出了新型耦合多样性敏感的动量约束学习（编码器），以改善跨模式表示。首先，发明了一种新颖的多样性对比度学习（DCL）体系结构。我们引入了两种模式的动态词典，以扩大图像文本对的比例，并且通过自适应负面对加权实现多样性敏感性。此外，编码器设计了两个分支。一个人从图像/文本中学习实例级的嵌入式，它还基于其嵌入为其输入图像/文本生成伪在线聚类标签。同时，另一个分支学会从常识知识图中查询以形成两种模式的概念级描述符。之后，两个分支都利用DCL来对齐跨模式嵌入空间，而额外的伪聚类标签预测损失则用于促进第二个分支的概念级表示学习。在两个流行的基准测试（即Mscoco和Flicker30k）上进行的广泛实验，验证编码器的表现明显优于最先进的方法。

Video-Text检索（VTR）是多模式理解的一项有吸引力但具有挑战性的任务，该任务旨在在给定查询（视频）的情况下搜索相关的视频（文本）。现有方法通常采用完全异构的视觉文本信息来对齐视频和文本，同时缺乏对这两种模式中均匀的高级语义信息的认识。为了填补这一差距，在这项工作中，我们提出了一个新颖的视觉语言对准模型，名为VTR Hise，该模型通过合并显式高级语义来改善跨模式的表示。首先，我们探讨了显式高级语义的层次结构属性，并将其进一步分为两个级别，即离散的语义和整体语义。具体来说，对于视觉分支，我们利用了现成的语义实体预测器来生成离散的高级语义。同时，采用训练有素的视频字幕模型来输出整体高级语义。至于文本方式，我们将文本分为三个部分，包括发生，动作和实体。特别是，这种情况对应于整体高级语义，同时动作和实体代表离散的语义。然后，利用不同的图推理技术来促进整体和离散的高级语义之间的相互作用。广泛的实验表明，借助明确的高级语义，我们的方法在包括MSR-VTT，MSVD和DIDEMO在内的三个基准数据集上实现了优于最先进方法的卓越性能。