Prompting large language models has enabled significant recent progress in multi-step reasoning over text. However, when applied to text generation from semi-structured data (e.g., graphs or tables), these methods typically suffer from low semantic coverage, hallucination, and logical inconsistency. We propose MURMUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning. MURMUR is a best-first search method that generates reasoning paths using: (1) neural and symbolic modules with specific linguistic and logical skills, (2) a grammar whose production rules define valid compositions of modules, and (3) value functions that assess the quality of each reasoning step. We conduct experiments on two diverse data-to-text generation tasks like WebNLG and LogicNLG. These tasks differ in their data representations (graphs and tables) and span multiple linguistic and logical skills. MURMUR obtains significant improvements over recent few-shot baselines like direct prompting and chain-of-thought prompting, while also achieving comparable performance to fine-tuned GPT-2 on out-of-domain data. Moreover, human evaluation shows that MURMUR generates highly faithful and correct reasoning paths that lead to 26% more logically consistent summaries on LogicNLG, compared to direct prompting.

Vision transformers (ViTs) have achieved impressive results on various computer vision tasks in the last several years. In this work, we study the capability of frozen ViTs, pretrained only on visual data, to generalize to audio-visual data without finetuning any of its original parameters. To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT. To efficiently fuse visual and audio cues, our LAVISH adapter uses a small set of latent tokens, which form an attention bottleneck, thus, eliminating the quadratic cost of standard cross-attention. Compared to the existing modality-specific audio-visual methods, our approach achieves competitive or even better performance on various audio-visual tasks while using fewer tunable parameters and without relying on costly audio pretraining or external audio encoders. Our code is available at https://genjib.github.io/project_page/LAVISH/

The last several years have witnessed remarkable progress in video-and-language (VidL) understanding. However, most modern VidL approaches use complex and specialized model architectures and sophisticated pretraining protocols, making the reproducibility, analysis and comparisons of these frameworks difficult. Hence, instead of proposing yet another new VidL model, this paper conducts a thorough empirical study demystifying the most important factors in the VidL model design. Among the factors that we investigate are (i) the spatiotemporal architecture design, (ii) the multimodal fusion schemes, (iii) the pretraining objectives, (iv) the choice of pretraining data, (v) pretraining and finetuning protocols, and (vi) dataset and model scaling. Our empirical study reveals that the most important design factors include: temporal modeling, video-to-text multimodal fusion, masked modeling objectives, and joint training on images and videos. Using these empirical insights, we then develop a step-by-step recipe, dubbed VindLU, for effective VidL pretraining. Our final model trained using our recipe achieves comparable or better than state-of-the-art results on several VidL tasks without relying on external CLIP pretraining. In particular, on the text-to-video retrieval task, our approach obtains 61.2% on DiDeMo, and 55.0% on ActivityNet, outperforming current SOTA by 7.8% and 6.1% respectively. Furthermore, our model also obtains state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC and TVQA. Our code and pretrained models are publicly available at: https://github.com/klauscc/VindLU.

We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark (DUE).

Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. In this work, we analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias (i.e., a source sequence already mapped to a target sequence is less likely to be mapped to other target sequences), and the tendency to memorize whole examples rather than separating structures from contents. We propose two techniques to address these two issues respectively: Mutual Exclusivity Training that prevents the model from producing seen generations when facing novel, unseen examples via an unlikelihood-based loss; and prim2primX data augmentation that automatically diversifies the arguments of every syntactic function to prevent memorizing and provide a compositional inductive bias without exposing test-set data. Combining these two techniques, we show substantial empirical improvements using standard sequence-to-sequence models (LSTMs and Transformers) on two widely-used compositionality datasets: SCAN and COGS. Finally, we provide analysis characterizing the improvements as well as the remaining challenges, and provide detailed ablations of our method. Our code is available at https://github.com/owenzx/met-primaug

Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for vision-and-language summarization. We propose CLIPBERTScore, a simple weighted combination of CLIPScore and BERTScore to leverage the robustness and strong factuality detection performance between image-summary and document-summary, respectively. Next, due to the lack of meta-evaluation benchmarks to evaluate the quality of multimodal factuality metrics, we collect human judgments of factuality with respect to documents and images. We show that this simple combination of two metrics in the zero-shot setting achieves higher correlations than existing factuality metrics for document summarization, outperforms an existing multimodal summarization metric, and performs competitively with strong multimodal factuality metrics specifically fine-tuned for the task. Our thorough analysis demonstrates the robustness and high correlation of CLIPBERTScore and its components on four factuality metric-evaluation benchmarks. Finally, we demonstrate two practical downstream applications of our CLIPBERTScore metric: for selecting important images to focus on during training, and as a reward for reinforcement learning to improve factuality of multimodal summary generation w.r.t automatic and human evaluation. Our data and code are publicly available at https://github.com/meetdavidwan/faithful-multimodal-summ

在这项工作中，我们介绍了无文本视觉语言变压器（TVLT），其中均匀的变压器块使用最小的模态设计进行视觉和语言表示的原始视觉和音频输入，并且不使用特定于文本的模块，例如作为令牌化或自动语音识别（ASR）。 TVLT通过重建连续视频帧和音频谱图（蒙版自动编码）和对比度建模以使视频和音频对比度建模进行训练。 TVLT在各种多模式任务上的性能与其基于文本的对应物相当，例如视觉询问，图像检索，视频检索和多模式情感分析，具有28倍的推理速度和仅1/3参数。我们的发现表明，从低级视觉和音频信号中学习紧凑，有效的视觉语言表示的可能性，而无需假设文本的先前存在。我们的代码和检查点可在以下网址找到：https：//github.com/zinengtang/tvlt

当前的抽象摘要模型要么仅通过突出源文档的一部分而缺乏明显的解释性或提供不完整的理由。为此，我们提出了摘要程序（SP），这是一个由二进制树的（有序）列表组成的可解释的模块化框架，每个框架都编码来自源文档的抽象摘要句子的分步生成过程。一个摘要程序每个摘要句子包含一个根节点，一棵不同的树将每个摘要句子（根节点）连接到派生的文档句子（叶节点），其中包含中间生成的句子的连接节点。边缘代表涉及摘要的不同模块化操作，例如句子融合，压缩和释义。我们首先建议通过神经模块提出有效的最佳搜索方法，SP搜索通过直接优化Rouge分数来识别人类摘要的SP搜索。接下来，使用这些程序作为自动监督，我们建议使用生成摘要程序的SEQ2SEQ模型，然后执行以获取最终摘要。我们证明，SP搜索有效地代表了使用通常忠于其预期行为的模块的人类摘要背后的生成过程。我们还进行了一项仿真研究，以表明汇总计划通过允许人类更好地模拟模型推理来改善摘要模型的解释性。汇总计划构成了朝着可解释和模块化的抽象摘要迈出的有希望的步骤，这是先前主要通过黑框端到端神经系统解决的复杂任务。我们的代码可从https://github.com/swarnahub/summarization Programs获得

文本到图像合成的最新进展导致了较大的经过验证的变压器，具有出色的能力，可以从给定文本产生可视化。但是，这些模型不适合专门的任务，例如故事可视化，该任务要求代理商制作一系列图像，给定相应的字幕序列，形成叙述。此外，我们发现故事可视化任务无法适应新叙事中看不见的情节和角色的概括。因此，我们首先提出了故事延续的任务，其中生成的视觉故事是在源图像上进行的，从而可以更好地对具有新角色的叙述进行更好的概括。然后，我们使用特定于（a）顺序图像生成的任务特定模块和（b）从初始帧复制相关元素的任务特定模块来增强或“复古”文本对图像合成模型。然后，我们探讨了预训练模型的全模型芬太尼以及对参数适应的及时调整。我们在两个现有数据集（PororoSV和FlintStonessV）上评估了我们的方法storydall-e，并介绍了从视频吸引数据集收集的新数据集DIDEMOSV。我们还基于生成的对抗网络（GAN）开发了一个模型故事游戏，以进行故事的延续，并将其与StoryDall-E模型进行比较，以展示我们方法的优势。我们表明，我们的复古拟合方法优于基于GAN的模型，用于故事延续，并促进从源图像中复制视觉元素，从而改善了生成的视觉故事中的连续性。最后，我们的分析表明，经过审计的变压器努力理解包含几个角色的叙述。总体而言，我们的工作表明，可以验证的文本对图像合成模型可以适应复杂和低资源的任务，例如故事延续。

在抽象性摘要的背景下，已广泛讨论了不忠摘要的问题。尽管提取性摘要不太容易出现抽象性摘要的普遍不忠问题，但这是否意味着提取性等于忠实？原来答案是否定的。在这项工作中，我们定义了一种类型学，具有五种类型的广泛的不忠问题（包括和超越未登录），这些问题可能出现在提取性摘要中，包括不正确的核心，不完整的核心，不正确的话语，不完整的话语，不完整的话语以及其他误导性信息。我们要求人类在1500个由15种不同的提取系统产生的英语摘要中标记这些问题。我们发现，其中33％的摘要至少有五个问题中的一个。为了自动检测这些问题，我们发现5个现有的忠诚评估指标与人类判断力的相关性很差。为了解决这个问题，我们提出了一种新的度量标准，该指标旨在检测不忠的提取性摘要，并显示出最佳性能。我们希望我们的工作能够提高对提取性总结中不忠问题的认识，并帮助将来的工作评估和解决这些问题。我们的数据和代码可在https://github.com/zhangshiyue/extractive_is_not_faithful上公开获取