虽然以完全可差异的模型的端到端学习在自然语言过程（NLP）和机器学习中取得了巨大的成功，但最近的近期兴趣与潜在的离散结构一起学习以改善最新的最终任务性能和更好的归纳偏差更好的解释性。然而，该范例并不直接地适应主流梯度的优化方法。这项工作调查了三个主要的方法来学习此类模型：通过采样，替代梯度，连续放松和边缘似然最大化。我们结束了对这些方法的应用以及检查他们诱导的学习潜在结构的检查。

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the inability to backpropagate through samples. In this work, we present an efficient gradient estimator that replaces the non-differentiable sample from a categorical distribution with a differentiable sample from a novel Gumbel-Softmax distribution. This distribution has the essential property that it can be smoothly annealed into a categorical distribution. We show that our Gumbel-Softmax estimator outperforms state-of-the-art gradient estimators on structured output prediction and unsupervised generative modeling tasks with categorical latent variables, and enables large speedups on semi-supervised classification. * Work done during an internship at Google Brain.

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.

当前独立于域的经典计划者需要问题域和实例作为输入的符号模型，从而导致知识采集瓶颈。同时，尽管深度学习在许多领域都取得了重大成功，但知识是在与符号系统（例如计划者）不兼容的亚符号表示中编码的。我们提出了Latplan，这是一种无监督的建筑，结合了深度学习和经典计划。只有一组未标记的图像对，显示了环境中允许的过渡子集（训练输入），Latplan学习了环境的完整命题PDDL动作模型。稍后，当给出代表初始状态和目标状态（计划输入）的一对图像时，Latplan在符号潜在空间中找到了目标状态的计划，并返回可视化的计划执行。我们使用6个计划域的基于图像的版本来评估LATPLAN：8个插头，15个式嘴，Blockworld，Sokoban和两个LightsOut的变体。

变异推理（VI）的核心原理是将计算复杂后概率密度计算的统计推断问题转换为可拖动的优化问题。该属性使VI比几种基于采样的技术更快。但是，传统的VI算法无法扩展到大型数据集，并且无法轻易推断出越野数据点，而无需重新运行优化过程。该领域的最新发展，例如随机，黑框和摊销VI，已帮助解决了这些问题。如今，生成的建模任务广泛利用摊销VI来实现其效率和可扩展性，因为它利用参数化函数来学习近似的后验密度参数。在本文中，我们回顾了各种VI技术的数学基础，以构成理解摊销VI的基础。此外，我们还概述了最近解决摊销VI问题的趋势，例如摊销差距，泛化问题，不一致的表示学习和后验崩溃。最后，我们分析了改善VI优化的替代差异度量。

众所周知，端到端的神经NLP体系结构很难理解，这引起了近年来为解释性建模的许多努力。模型解释的基本原则是忠诚，即，解释应准确地代表模型预测背后的推理过程。这项调查首先讨论了忠诚的定义和评估及其对解释性的意义。然后，我们通过将方法分为五类来介绍忠实解释的最新进展：相似性方法，模型内部结构的分析，基于反向传播的方法，反事实干预和自我解释模型。每个类别将通过其代表性研究，优势和缺点来说明。最后，我们从它们的共同美德和局限性方面讨论了上述所有方法，并反思未来的工作方向忠实的解释性。对于有兴趣研究可解释性的研究人员，这项调查将为该领域提供可访问且全面的概述，为进一步探索提供基础。对于希望更好地了解自己的模型的用户，该调查将是一项介绍性手册，帮助选择最合适的解释方法。

超过三十年，研究人员已经开发和分析了潜伏树诱导的方法作为无监督句法解析的方法。尽管如此，与其监督的对应物相比，现代系统仍然不足以使其具有任何实际用途作为文本的结构注释。在这项工作中，我们提出了一种技术，该技术以跨度约束（即短语包围）的形式使用远端监督，以提高在无监督选项解析中的性能。使用相对少量的跨度约束，我们可以大大提高Diora的输出，这是一个已经竞争的无监督解析系统。与完整的解析树注释相比，可以通过最小的努力来获取跨度约束，例如使用从维基百科派生的词典，以查找确切的文本匹配。我们的实验显示了基于实体的跨度约束，提高了英语WSJ Penn TreeBank的选区分析超过5 F1。此外，我们的方法延伸到跨度约束易于实现的任何域，以及作为一个案例研究，我们通过从工艺数据集解析生物医学文本来证明其有效性。

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

组合优化是运营研究和计算机科学领域的一个公认领域。直到最近，它的方法一直集中在孤立地解决问题实例，而忽略了它们通常源于实践中的相关数据分布。但是，近年来，人们对使用机器学习，尤其是图形神经网络（GNN）的兴趣激增，作为组合任务的关键构件，直接作为求解器或通过增强确切的求解器。GNN的电感偏差有效地编码了组合和关系输入，因为它们对排列和对输入稀疏性的意识的不变性。本文介绍了对这个新兴领域的最新主要进步的概念回顾，旨在优化和机器学习研究人员。

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propagated by the chain rule through the graph are low variance unbiased estimators of the gradients of the expected loss. While many continuous random variables have such reparameterizations, discrete random variables lack useful reparameterizations due to the discontinuous nature of discrete states. In this work we introduce CONCRETE random variables-CONtinuous relaxations of disCRETE random variables. The Concrete distribution is a new family of distributions with closed form densities and a simple reparameterization. Whenever a discrete stochastic node of a computation graph can be refactored into a one-hot bit representation that is treated continuously, Concrete stochastic nodes can be used with automatic differentiation to produce low-variance biased gradients of objectives (including objectives that depend on the log-probability of latent stochastic nodes) on the corresponding discrete graph. We demonstrate the effectiveness of Concrete relaxations on density estimation and structured prediction tasks using neural networks.

尽管深层生成模型在图像处理，自然语言处理和强化学习方面已经成功，但由于其梯度估计过程的较高差异，涉及离散随机变量的培训仍然具有挑战性。蒙特卡洛是大多数降低方法中使用的常见解决方案。但是，这涉及耗时的重采样和多功能评估。我们提出了一个张开的直通（GST）估计器，以减少方差，而不会产生重新采样开销。该估计器的灵感来自直通牙龈 - 软胶的基本属性。我们确定这些特性，并通过消融研究表明它们是必不可少的。实验表明，与在两个离散的深层生成建模任务：MNIST-VAE和LISTOPS上相比，所提出的GST估计器与强基础相比具有更好的性能。

我用Hunglish2语料库训练神经电脑翻译任务的模型。这项工作的主要贡献在培训NMT模型期间评估不同的数据增强方法。我提出了5种不同的增强方法，这些方法是结构感知的，这意味着而不是随机选择用于消隐或替换的单词，句子的依赖树用作增强的基础。我首先关于神经网络的详细文献综述，顺序建模，神经机翻译，依赖解析和数据增强。经过详细的探索性数据分析和Hunglish2语料库的预处理之后，我使用所提出的数据增强技术进行实验。匈牙利语的最佳型号达到了33.9的BLEU得分，而英国匈牙利最好的模型达到了28.6的BLEU得分。

Pre-publication draft of a book to be published byMorgan & Claypool publishers. Unedited version released with permission. All relevant copyrights held by the author and publisher extend to this pre-publication draft.

Outstanding achievements of graph neural networks for spatiotemporal time series analysis show that relational constraints introduce an effective inductive bias into neural forecasting architectures. Often, however, the relational information characterizing the underlying data-generating process is unavailable and the practitioner is left with the problem of inferring from data which relational graph to use in the subsequent processing stages. We propose novel, principled - yet practical - probabilistic score-based methods that learn the relational dependencies as distributions over graphs while maximizing end-to-end the performance at task. The proposed graph learning framework is based on consolidated variance reduction techniques for Monte Carlo score-based gradient estimation, is theoretically grounded, and, as we show, effective in practice. In this paper, we focus on the time series forecasting problem and show that, by tailoring the gradient estimators to the graph learning problem, we are able to achieve state-of-the-art performance while controlling the sparsity of the learned graph and the computational scalability. We empirically assess the effectiveness of the proposed method on synthetic and real-world benchmarks, showing that the proposed solution can be used as a stand-alone graph identification procedure as well as a graph learning component of an end-to-end forecasting architecture.

Gumbel-Softmax是对单纯形的连续分布，通常用作离散分布的放松。因为它可以轻松解释和容易重新聚集，所以它可以广泛使用。我们提出了一个模块化，更灵活的可重新聚集分布家族，其中高斯噪声通过可逆函数转化为单热近似。这种可逆函数由修改后的软磁性组成，可以结合具有不同特定目的的各种转换。例如，破坏过程使我们能够将重新聚集技巧扩展到具有无数支持的分布，从而使我们沿非参数模型的分布使用或归一化流量使我们提高了分布的灵活性。我们的构造具有比Gumbel-Softmax（例如封闭形式KL）的理论优势，并且在各种实验中都显着优于它。我们的代码可在https://github.com/cunningham-lab/igr上获得。

本文对过去二十年来对自然语言生成（NLG）的研究提供了全面的审查，特别是与数据到文本生成和文本到文本生成深度学习方法有关，以及NLG的新应用技术。该调查旨在（a）给出关于NLG核心任务的最新综合，以及该领域采用的建筑;（b）详细介绍各种NLG任务和数据集，并提请注意NLG评估中的挑战，专注于不同的评估方法及其关系;（c）强调一些未来的强调和相对近期的研究问题，因为NLG和其他人工智能领域的协同作用而增加，例如计算机视觉，文本和计算创造力。

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

神经网络是强大的功能估计器，导致其作为建模结构化数据的首选范式的地位。但是，与其他强调问题模块化的结构化表示不同，例如因子图 - 神经网络通常是从输入到输出的单片映射，并具有固定的计算顺序。这种限制阻止他们捕获模型变量之间的不同计算方向和相互作用。在本文中，我们结合了因子图和神经网络的代表性强度，提出了无向神经网络（UNNS）：一个灵活的框架，用于指定可以按任何顺序执行的计算。对于特定的选择，我们提出的模型集成并扩展了许多现有的架构：带有隐式层的馈电，经常性，自我发项网络，自动编码器和网络。我们在一系列任务上展示了无方向性的神经体系结构的有效性：受树约束依赖性解析，卷积图像分类和序列完成。通过改变计算顺序，我们展示了如何同时将单个UNN用作分类器和原型发生器，以及它如何填充输入序列的缺失部分，从而使它们成为进一步研究的有希望的领域。