Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher). Although KD methods achieve state-of-the-art performance in numerous settings, they suffer from several problems limiting their performance. It is shown in the literature that the capacity gap between the teacher and the student networks can make KD ineffective. Additionally, existing KD techniques do not mitigate the noise in the teacher's output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. We propose a new KD method that addresses these problems and facilitates the training compared to previous techniques. Inspired by continuation optimization, we design a training procedure that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. Our method (Continuation-KD) achieves state-of-the-art performance across various compact architectures on NLU (GLUE benchmark) and computer vision tasks (CIFAR-10 and CIFAR-100).

Knowledge Distillation (KD) is a commonly used technique for improving the generalization of compact Pre-trained Language Models (PLMs) on downstream tasks. However, such methods impose the additional burden of training a separate teacher model for every new dataset. Alternatively, one may directly work on the improvement of the optimization procedure of the compact model toward better generalization. Recent works observe that the flatness of the local minimum correlates well with better generalization. In this work, we adapt Stochastic Weight Averaging (SWA), a method encouraging convergence to a flatter minimum, to fine-tuning PLMs. We conduct extensive experiments on various NLP tasks (text classification, question answering, and generation) and different model architectures and demonstrate that our adaptation improves the generalization without extra computation cost. Moreover, we observe that this simple optimization technique is able to outperform the state-of-the-art KD methods for compact models.

局部线性嵌入（LLE）是一种非线性光谱维度降低和多种学习方法。它有两个主要步骤，分别是线性重建和分别在输入空间和嵌入空间中的点的线性嵌入。在这项工作中，我们从随机的角度看线性重建步骤，其中假定每个数据点都以其线性重建权重为潜在因素。 LLE的随机线性重建是使用预期最大化解决的。我们表明，三种基本维度降低方法（即LLE，因子分析和概率主体组件分析（PCA））之间存在理论上的联系。 LLE的随机线性重建与因子分析和概率PCA相似。这也解释了为什么因子分析和概率PCA是线性的，而LLE是一种非线性方法。这项工作结合了两种降低维度的广泛方法，即光谱和概率算法。

这是关于生成对抗性网络（GaN），对抗性自身额外的教程和调查纸张及其变体。我们开始解释对抗性学习和香草甘。然后，我们解释了条件GaN和DCGAN。介绍了模式崩溃问题，介绍了各种方法，包括小纤维GaN，展开GaN，Bourgan，混合GaN，D2Gan和Wasserstein GaN，用于解决这个问题。然后，GaN中的最大似然估计与F-GaN，对抗性变分贝叶斯和贝叶斯甘甘相同。然后，我们涵盖了GaN，Infogan，Gran，Lsgan，Enfogan，Gran，Lsgan，Catgan，MMD Gan，Lapgan，Progressive Gan，Triple Gan，Lag，Gman，Adagan，Cogan，逆甘，Bigan，Ali，Sagan，Sagan，Sagan，Sagan，甘肃，甘肃，甘河的插值和评估。然后，我们介绍了GaN的一些应用，例如图像到图像转换（包括Pacchgan，Cyclegan，Deepfacedrawing，模拟GaN，Interactive GaN），文本到图像转换（包括Stackgan）和混合图像特征（包括罚球和mixnmatch）。最后，我们解释了基于对冲学习的AutoEncoders，包括对手AutoEncoder，Pixelgan和隐式AutoEncoder。

这是关于Boltzmann机器（BM），受限玻尔兹曼机器（RBM）和Deep信念网络（DBN）的教程和调查论文。我们从概率图形模型，Markov随机字段，Gibbs采样，统计物理学，ISING模型和Hopfield网络的必需背景开始。然后，我们介绍BM和RBM的结构。解释了可见变量和隐藏变量的条件分布，RBM中的GIBBS采样以生成变量，通过最大似然估计训练BM和RBM以及对比度差异。然后，我们讨论变量的不同可能的离散和连续分布。我们介绍有条件的RBM及其训练方式。最后，我们将深度信念网络解释为RBM模型的一堆。本文有关玻尔兹曼机器的论文在包括数据科学，统计，神经计算和统计物理学在内的各个领域都有用。

这是一份有关降低光谱维度降低方法统一的教程和调查论文，通过半决赛编程（SDP）学习内核学习，最大方差展开（MVU）或半芬特嵌入（SDE）及其变体。我们首先解释了如何将频谱降低方法降低方法统一为具有不同内核的内核主成分分析（PCA）。在距离矩阵方面，该统一可以解释为内核的本本函数学习或表示。然后，由于光谱方法被统一为内核PCA，因此我们说，让我们学习将数据的歧管展开至最大方差的最佳内核。我们首先简要介绍了SDP的内核学习来进行转导任务。然后，我们详细解释MVU。解释了使用最近的邻居图，通过课堂展开，Fisher Criterion和通过彩色MVU进行的各种监督MVU。我们还使用本征函数和内核映射解释了MVU的样本外扩展。最后，我们介绍了MVU的其他变体，包括尊重嵌入，放松的MVU和Landmark MVU的动作，以获取大数据。

这是针对非线性维度和特征提取方法的教程和调查论文，该方法基于数据图的拉普拉斯语。我们首先介绍邻接矩阵，拉普拉斯矩阵的定义和拉普拉斯主义的解释。然后，我们涵盖图形和光谱聚类的切割，该谱图应用于数据子空间。解释了Laplacian征收及其样本外扩展的不同优化变体。此后，我们将保留投影的局部性及其内核变体作为拉普拉斯征本征的线性特殊案例。然后解释了图嵌入的版本，这些版本是Laplacian eigenmap和局部保留投影的广义版本。最后，引入了扩散图，这是基于数据图和随机步行的方法。

随机邻居嵌入（SNE）是一种具有概率方法的多种学习和降低方法。在SNE中，每个点都被认为是所有其他点的邻居，并试图将这种概率保存在嵌入空间中。SNE认为在输入空间和嵌入空间中的概率都认为高斯分布。但是，T-SNE分别在这些空间中使用了Student-T和高斯分布。在本教程和调查论文中，我们解释了SNE，对称SNE，T-SNE（或Cauchy-Sne）和T-SNE具有一般自由度。我们还涵盖了这些方法的样本外扩展和加速度。

Existing automated techniques for software documentation typically attempt to reason between two main sources of information: code and natural language. However, this reasoning process is often complicated by the lexical gap between more abstract natural language and more structured programming languages. One potential bridge for this gap is the Graphical User Interface (GUI), as GUIs inherently encode salient information about underlying program functionality into rich, pixel-based data representations. This paper offers one of the first comprehensive empirical investigations into the connection between GUIs and functional, natural language descriptions of software. First, we collect, analyze, and open source a large dataset of functional GUI descriptions consisting of 45,998 descriptions for 10,204 screenshots from popular Android applications. The descriptions were obtained from human labelers and underwent several quality control mechanisms. To gain insight into the representational potential of GUIs, we investigate the ability of four Neural Image Captioning models to predict natural language descriptions of varying granularity when provided a screenshot as input. We evaluate these models quantitatively, using common machine translation metrics, and qualitatively through a large-scale user study. Finally, we offer learned lessons and a discussion of the potential shown by multimodal models to enhance future techniques for automated software documentation.

In this paper, we reduce the complexity of approximating the correlation clustering problem from $O(m\times\left( 2+ \alpha (G) \right)+n)$ to $O(m+n)$ for any given value of $\varepsilon$ for a complete signed graph with $n$ vertices and $m$ positive edges where $\alpha(G)$ is the arboricity of the graph. Our approach gives the same output as the original algorithm and makes it possible to implement the algorithm in a full dynamic setting where edge sign flipping and vertex addition/removal are allowed. Constructing this index costs $O(m)$ memory and $O(m\times\alpha(G))$ time. We also studied the structural properties of the non-agreement measure used in the approximation algorithm. The theoretical results are accompanied by a full set of experiments concerning seven real-world graphs. These results shows superiority of our index-based algorithm to the non-index one by a decrease of %34 in time on average.