Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, {\textit UnitY}, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.

直接语音到语音翻译（S2ST）模型与传统级联系统可用的数据量相比，几乎没有平行的S2ST数据遇到数据稀缺问题，该数据包括自动语音识别（ASR），机器翻译（MT）和文本到语音（TTS）合成。在这项工作中，我们使用未标记的语音数据和数据扩展来探索自我监督的预训练，以解决此问题。我们利用了最近提出的语音到单位翻译（S2UT）框架，该框架将目标语音编码为离散表示形式，并转移前训练前和有效的部分填充技术，可很好地适用于语音到文本翻译（S2T）通过研究语音编码器和离散单位解码器预训练，S2UT域。我们在西班牙语 - 英语翻译上进行的实验表明，与多任务学习相比，自我监督的预训练始终如一地提高模型性能，平均为6.6-12.1 BLEU增长，并且可以与数据增强技术相结合，以应用MT来创建弱监督监督的培训数据。音频样本可在以下网址获得：https：//facebookresearch.github.io/speech_translation/enhanced_direct_s2st_units/index.html。

我们介绍了一种无线文字语音转换（S2ST）系统，可以将来自一种语言的语音转换为另一种语言，并且可以在不需要任何文本数据的情况下构建。与文献中的现有工作不同，我们解决了模拟多扬声器目标语音的挑战，并用现实世界的S2ST数据训练系统。我们方法的关键是一种自我监督的单位语音标准化技术，该标准化技术将预先训练的语音编码器具有来自多个扬声器的配对声音，以及单个参考扬声器，以减少由于复印件引起的变化，同时保留词汇内容。只有10分钟的语音标准化的配对数据，我们在培训\ vp〜s2st数据集上的S2ST模型时获得平均3.2 BLEU增益，而不是在未标准化的语音目标上培训的基线。我们还将自动开采的S2ST数据纳入并显示额外的2.0 BLEU增益。据我们所知，我们是第一个建立无线的S2ST技术，可以用真实世界的数据培训，并为多种语言配对工作。

本文介绍了基于Wav2VEC 2.0的跨语言语音表示学习的大规模模型。我们在128种语言中培训最多2B个公共讲话音频的近半小时的型号的模型，比公共数据的数量级比最大的已知事先工作。我们的评估涵盖了广泛的任务，域，数据制度和语言，都是高低资源。在Covost-2语音翻译基准测试中，我们将先前的最先进的状态平均为7.4 BLEU超过21个翻译方向进入英语。对于语音识别，XLS-R在Babel，MLS，CommonVoice以及Voxpopuli上的最佳已知工作中提高，降低了相对的误差率14-34％。 XLS-R还在Voxlingua107语言识别上设置了新的技术状态。此外，我们表明，具有足够的模型规模，交叉思维预先预测可以在将英语演讲翻译成其他语言时才能优于英语撇印，这是一个有利于单晶的预借预制的设置。我们希望XLS-R可以帮助改善世界上更多语言的语音处理任务。

语音到语音翻译（S2ST）将输入语音转换为另一种语言。实时交付S2ST的挑战是翻译和语音合成模块之间的累积延迟。尽管最近增量的文本到语音（ITTS）模型已显示出巨大的质量改进，但它们通常需要其他未来的文本输入才能达到最佳性能。在这项工作中，我们通过调整上游语音翻译器来为语音合成器生成高质量的伪lookahead来最大程度地减少ITT的最初等待时间。缓解初始延迟后，我们证明了合成语音的持续时间在延迟中也起着至关重要的作用。我们将其形式化为延迟度量，然后提出一种简单而有效的持续时间缩放方法，以减少延迟。我们的方法始终将延迟减少0.2-0.5秒，而无需牺牲语音翻译质量。

我们介绍了Fairseq S2T，这是语音到文本（S2T）建模任务的Fairseq扩展，例如端到端语音识别和语音到文本翻译。它遵循Fairseq的仔细设计，以实现可扩展性和可扩展性。我们提供从数据预处理，模型培训到离线推理的端到端工作流程。我们实施了基于最新的RNN，基于变压器以及基于构象的模型和开源详细培训配方。Fairseq的机器翻译模型和语言模型可以无缝集成到S2T工作流中，以进行多任务学习或转移学习。Fairseq S2T文档和示例可在https://github.com/pytorch/fairseq/tree/master/master/examples/speech_to_text上获得。

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.

Knowledge graphs (KG) have served as the key component of various natural language processing applications. Commonsense knowledge graphs (CKG) are a special type of KG, where entities and relations are composed of free-form text. However, previous works in KG completion and CKG completion suffer from long-tail relations and newly-added relations which do not have many know triples for training. In light of this, few-shot KG completion (FKGC), which requires the strengths of graph representation learning and few-shot learning, has been proposed to challenge the problem of limited annotated data. In this paper, we comprehensively survey previous attempts on such tasks in the form of a series of methods and applications. Specifically, we first introduce FKGC challenges, commonly used KGs, and CKGs. Then we systematically categorize and summarize existing works in terms of the type of KGs and the methods. Finally, we present applications of FKGC models on prediction tasks in different areas and share our thoughts on future research directions of FKGC.

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.