Training accurate deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Though a number of approaches have been proposed for learning with noisy labels, many open issues remain. In this paper, we show that DNN learning with Cross Entropy (CE) exhibits overfitting to noisy labels on some classes ("easy" classes), but more surprisingly, it also suffers from significant under learning on some other classes ("hard" classes). Intuitively, CE requires an extra term to facilitate learning of hard classes, and more importantly, this term should be noise tolerant, so as to avoid overfitting to noisy labels. Inspired by the symmetric KL-divergence, we propose the approach of Symmetric cross entropy Learning (SL), boosting CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). Our proposed SL approach simultaneously addresses both the under learning and overfitting problem of CE in the presence of noisy labels. We provide a theoretical analysis of SL and also empirically show, on a range of benchmark and real-world datasets, that SL outperforms state-of-the-art methods. We also show that SL can be easily incorporated into existing methods in order to further enhance their performance.

Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100 and FASHION-MNIST datasets and synthetically generated noisy labels.

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.

Deep neural networks (DNNs) trained on large-scale datasets have exhibited significant performance in image classification. Many large-scale datasets are collected from websites, however they tend to contain inaccurate labels that are termed as noisy labels. Training on such noisy labeled datasets causes performance degradation because DNNs easily overfit to noisy labels. To overcome this problem, we propose a joint optimization framework of learning DNN parameters and estimating true labels. Our framework can correct labels during training by alternating update of network parameters and labels. We conduct experiments on the noisy CIFAR-10 datasets and the Clothing1M dataset.The results indicate that our approach significantly outperforms other state-of-the-art methods.

深层神经网络能够轻松地使用软磁横层（CE）丢失来记住嘈杂的标签。先前的研究试图解决此问题的重点是将噪声损失函数纳入CE损失。但是，记忆问题得到了缓解，但仍然由于非持鲁棒的损失而造成的。为了解决这个问题，我们专注于学习可靠的对比度表示数据，分类器很难记住CE损失下的标签噪声。我们提出了一种新颖的对比正则化函数，以通过标签噪声不主导表示表示的嘈杂数据来学习此类表示。通过理论上研究由提议的正则化功能引起的表示形式，我们揭示了学识渊博的表示形式将信息保留与真实标签和丢弃与损坏标签相关的信息有关的信息。此外，我们的理论结果还表明，学到的表示形式对标签噪声是可靠的。通过基准数据集的实验证明了该方法的有效性。

作为标签噪声，最受欢迎的分布变化之一，严重降低了深度神经网络的概括性能，具有嘈杂标签的强大训练正在成为现代深度学习中的重要任务。在本文中，我们提出了我们的框架，在子分类器（ALASCA）上创造了自适应标签平滑，该框架提供了具有理论保证和可忽略的其他计算的可靠特征提取器。首先，我们得出标签平滑（LS）会产生隐式Lipschitz正则化（LR）。此外，基于这些推导，我们将自适应LS（ALS）应用于子分类器架构上，以在中间层上的自适应LR的实际应用。我们对ALASCA进行了广泛的实验，并将其与以前的几个数据集上的噪声燃烧方法相结合，并显示我们的框架始终优于相应的基线。

深度学习在大量大数据的帮助下取得了众多域中的显着成功。然而，由于许多真实情景中缺乏高质量标签，数据标签的质量是一个问题。由于嘈杂的标签严重降低了深度神经网络的泛化表现，从嘈杂的标签（强大的培训）学习是在现代深度学习应用中成为一项重要任务。在本调查中，我们首先从监督的学习角度描述了与标签噪声学习的问题。接下来，我们提供62项最先进的培训方法的全面审查，所有这些培训方法都按照其方法论差异分为五个群体，其次是用于评估其优越性的六种性质的系统比较。随后，我们对噪声速率估计进行深入分析，并总结了通常使用的评估方法，包括公共噪声数据集和评估度量。最后，我们提出了几个有前途的研究方向，可以作为未来研究的指导。所有内容将在https://github.com/songhwanjun/awesome-noisy-labels提供。

最近，与培训样本相比，具有越来越多的网络参数的过度参数深度网络主导了现代机器学习的性能。但是，当培训数据被损坏时，众所周知，过度参数化的网络往往会过度合适并且不会概括。在这项工作中，我们提出了一种有原则的方法，用于在分类任务中对过度参数的深层网络进行强有力的培训，其中一部分培训标签被损坏。主要想法还很简单：标签噪声与从干净的数据中学到的网络稀疏且不一致，因此我们对噪声进行建模并学会将其与数据分开。具体而言，我们通过另一个稀疏的过度参数术语对标签噪声进行建模，并利用隐式算法正规化来恢复和分离基础损坏。值得注意的是，当在实践中使用如此简单的方法培训时，我们证明了针对各种真实数据集上标签噪声的最新测试精度。此外，我们的实验结果通过理论在简化的线性模型上证实，表明在不连贯的条件下稀疏噪声和低级别数据之间的精确分离。这项工作打开了许多有趣的方向，可以使用稀疏的过度参数化和隐式正则化来改善过度参数化模型。

使用嘈杂的标签学习是一场实际上有挑战性的弱势监督。在现有文献中，开放式噪声总是被认为是有毒的泛化，类似于封闭式噪音。在本文中，我们经验证明，开放式嘈杂标签可能是无毒的，甚至有利于对固有的嘈杂标签的鲁棒性。灵感来自观察，我们提出了一种简单而有效的正则化，通过将具有动态噪声标签（ODNL）引入培训的开放式样本。使用ODNL，神经网络的额外容量可以在很大程度上以不干扰来自清洁数据的学习模式的方式消耗。通过SGD噪声的镜头，我们表明我们的方法引起的噪音是随机方向，无偏向，这可能有助于模型收敛到最小的最小值，具有卓越的稳定性，并强制执行模型以产生保守预测-of-分配实例。具有各种类型噪声标签的基准数据集的广泛实验结果表明，所提出的方法不仅提高了许多现有的强大算法的性能，而且即使在标签噪声设置中也能实现分配异点检测任务的显着改进。

标签噪声显着降低了应用中深度模型的泛化能力。有效的策略和方法，\ Texit {例如}重新加权或损失校正，旨在在训练神经网络时缓解标签噪声的负面影响。这些现有的工作通常依赖于预指定的架构并手动调整附加的超参数。在本文中，我们提出了翘曲的概率推断（WARPI），以便在元学习情景中自适应地整理分类网络的培训程序。与确定性模型相比，WARPI通过学习摊销元网络来制定为分层概率模型，这可以解决样本模糊性，因此对严格的标签噪声更加坚固。与直接生成损耗的重量值的现有近似加权功能不同，我们的元网络被学习以估计从登录和标签的输入来估计整流向量，这具有利用躺在它们中的足够信息的能力。这提供了纠正分类网络的学习过程的有效方法，证明了泛化能力的显着提高。此外，可以将整流载体建模为潜在变量并学习元网络，可以无缝地集成到分类网络的SGD优化中。我们在嘈杂的标签上评估了四个强大学习基准的Warpi，并在变体噪声类型下实现了新的最先进的。广泛的研究和分析还展示了我们模型的有效性。

深神经网络（DNN）的记忆效果在许多最先进的标签噪声学习方法中起着枢轴作用。为了利用这一财产，通常采用早期停止训练早期优化的伎俩。目前的方法通常通过考虑整个DNN来决定早期停止点。然而，DNN可以被认为是一系列层的组成，并且发现DNN中的后一个层对标签噪声更敏感，而其前同行是非常稳健的。因此，选择整个网络的停止点可以使不同的DNN层对抗彼此影响，从而降低最终性能。在本文中，我们建议将DNN分离为不同的部位，逐步培训它们以解决这个问题。而不是早期停止，它一次列举一个整体DNN，我们最初通过用相对大量的时期优化DNN来训练前DNN层。在培训期间，我们通过使用较少数量的时期使用较少的地层来逐步培训后者DNN层，以抵消嘈杂标签的影响。我们将所提出的方法术语作为渐进式早期停止（PES）。尽管其简单性，与早期停止相比，PES可以帮助获得更有前景和稳定的结果。此外，通过将PE与现有的嘈杂标签培训相结合，我们在图像分类基准上实现了最先进的性能。

深度神经网络的成功在很大程度上取决于大量高质量注释的数据的可用性，但是这些数据很难或昂贵。由此产生的标签可能是类别不平衡，嘈杂或人类偏见。从不完美注释的数据集中学习无偏分类模型是一项挑战，我们通常会遭受过度拟合或不足的折磨。在这项工作中，我们彻底研究了流行的软马克斯损失和基于保证金的损失，并提供了一种可行的方法来加强通过最大化最小样本余量来限制的概括误差。我们为此目的进一步得出了最佳条件，该条件指示了类原型应锚定的方式。通过理论分析的激励，我们提出了一种简单但有效的方法，即原型锚定学习（PAL），可以轻松地将其纳入各种基于学习的分类方案中以处理不完美的注释。我们通过对合成和现实世界数据集进行广泛的实验来验证PAL对班级不平衡学习和降低噪声学习的有效性。

带有嘈杂标签的训练深神经网络（DNN）实际上是具有挑战性的，因为不准确的标签严重降低了DNN的概括能力。以前的努力倾向于通过识别带有粗糙的小损失标准来减轻嘈杂标签的干扰的嘈杂数据来处理统一的denoising流中的零件或完整数据，而忽略了嘈杂样本的困难是不同的，因此是刚性和统一的。数据选择管道无法很好地解决此问题。在本文中，我们首先提出了一种称为CREMA的粗到精细的稳健学习方法，以分裂和串扰的方式处理嘈杂的数据。在粗糙水平中，干净和嘈杂的集合首先从统计意义上就可信度分开。由于实际上不可能正确对所有嘈杂样本进行分类，因此我们通过对每个样本的可信度进行建模来进一步处理它们。具体而言，对于清洁集，我们故意设计了一种基于内存的调制方案，以动态调整每个样本在训练过程中的历史可信度顺序方面的贡献，从而减轻了错误地分组为清洁集中的嘈杂样本的效果。同时，对于分类为嘈杂集的样品，提出了选择性标签更新策略，以纠正嘈杂的标签，同时减轻校正错误的问题。广泛的实验是基于不同方式的基准，包括图像分类（CIFAR，Clothing1M等）和文本识别（IMDB），具有合成或自然语义噪声，表明CREMA的优势和普遍性。

We present a theoretically grounded approach to train deep neural networks, including recurrent networks, subject to class-dependent label noise. We propose two procedures for loss correction that are agnostic to both application domain and network architecture. They simply amount to at most a matrix inversion and multiplication, provided that we know the probability of each class being corrupted into another. We further show how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and thus providing an end-to-end framework. Extensive experiments on MNIST, IMDB, CIFAR-10, CIFAR-100 and a large scale dataset of clothing images employing a diversity of architectures -stacking dense, convolutional, pooling, dropout, batch normalization, word embedding, LSTM and residual layers -demonstrate the noise robustness of our proposals. Incidentally, we also prove that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise.

Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Code is available at https://github.com/LiJunnan1992/DivideMix.

标签平滑（LS）是一种出现的学习范式，它使用硬训练标签和均匀分布的软标签的正加权平均值。结果表明，LS是带有硬标签的训练数据的常规器，因此改善了模型的概括。后来，据报道，LS甚至有助于用嘈杂的标签学习时改善鲁棒性。但是，我们观察到，当我们以高标签噪声状态运行时，LS的优势就会消失。从直觉上讲，这是由于$ \ mathbb {p}的熵增加（\ text {noisy label} | x）$当噪声速率很高时，在这种情况下，进一步应用LS会倾向于“超平滑”估计后部。我们开始发现，文献中的几种学习与噪声标签的解决方案相反，与负面/不标签平滑（NLS）更紧密地关联，它们与LS相反，并将其定义为使用负重量来结合硬和软标签呢我们在使用嘈杂标签学习时对LS和NLS的性质提供理解。在其他已建立的属性中，我们从理论上表明，当标签噪声速率高时，NLS被认为更有益。我们在多个基准测试中提供了广泛的实验结果，以支持我们的发现。代码可在https://github.com/ucsc-real/negative-label-smooth上公开获取。

The existence of label noise imposes significant challenges (e.g., poor generalization) on the training process of deep neural networks (DNN). As a remedy, this paper introduces a permutation layer learning approach termed PermLL to dynamically calibrate the training process of the DNN subject to instance-dependent and instance-independent label noise. The proposed method augments the architecture of a conventional DNN by an instance-dependent permutation layer. This layer is essentially a convex combination of permutation matrices that is dynamically calibrated for each sample. The primary objective of the permutation layer is to correct the loss of noisy samples mitigating the effect of label noise. We provide two variants of PermLL in this paper: one applies the permutation layer to the model's prediction, while the other applies it directly to the given noisy label. In addition, we provide a theoretical comparison between the two variants and show that previous methods can be seen as one of the variants. Finally, we validate PermLL experimentally and show that it achieves state-of-the-art performance on both real and synthetic datasets.

为了训练强大的深神经网络（DNNS），我们系统地研究了几种目标修饰方法，其中包括输出正则化，自我和非自动标签校正（LC）。发现了三个关键问题：（1）自我LC是最吸引人的，因为它利用了自己的知识，不需要额外的模型。但是，在文献中，如何自动确定学习者的信任程度并没有很好地回答。 （2）一些方法会受到惩罚，而另一些方法奖励低渗透预测，促使我们询问哪一种更好。 （3）使用标准训练设置，当存在严重的噪音时，受过训练的网络的信心较低，因此很难利用其高渗透自我知识。为了解决问题（1），采取两个良好接受的命题 - 深度神经网络在拟合噪声和最小熵正则原理之前学习有意义的模式 - 我们提出了一种名为Proselflc的新颖的端到端方法，该方法是根据根据学习时间和熵。具体而言，给定数据点，如果对模型进行了足够的时间训练，并且预测的熵较低（置信度很高），则我们逐渐增加对预测标签分布的信任与其注释的信任。根据ProSelfLC的说法，对于（2），我们从经验上证明，最好重新定义有意义的低渗透状态并优化学习者对其进行优化。这是防御熵最小化的防御。为了解决该问题（3），我们在利用低温以纠正标签之前使用低温降低了自我知识的熵，因此修订后的标签重新定义了低渗透目标状态。我们通过在清洁和嘈杂的环境以及图像和蛋白质数据集中进行广泛的实验来证明ProSelfLC的有效性。此外，我们的源代码可在https://github.com/xinshaoamoswang/proselflc-at上获得。

Deep Learning with noisy labels is a practically challenging problem in weakly supervised learning. The stateof-the-art approaches "Decoupling" and "Co-teaching+" claim that the "disagreement" strategy is crucial for alleviating the problem of learning with noisy labels. In this paper, we start from a different perspective and propose a robust learning paradigm called JoCoR, which aims to reduce the diversity of two networks during training. Specifically, we first use two networks to make predictions on the same mini-batch data and calculate a joint loss with Co-Regularization for each training example. Then we select small-loss examples to update the parameters of both two networks simultaneously. Trained by the joint loss, these two networks would be more and more similar due to the effect of Co-Regularization. Extensive experimental results on corrupted data from benchmark datasets including MNIST, CIFAR-10, CIFAR-100 and Clothing1M demonstrate that JoCoR is superior to many state-of-the-art approaches for learning with noisy labels.

Modeling noise transition matrix is a kind of promising method for learning with label noise. Based on the estimated noise transition matrix and the noisy posterior probabilities, the clean posterior probabilities, which are jointly called Label Distribution (LD) in this paper, can be calculated as the supervision. To reliably estimate the noise transition matrix, some methods assume that anchor points are available during training. Nonetheless, if anchor points are invalid, the noise transition matrix might be poorly learned, resulting in poor performance. Consequently, other methods treat reliable data points, extracted from training data, as pseudo anchor points. However, from a statistical point of view, the noise transition matrix can be inferred from data with noisy labels under the clean-label-domination assumption. Therefore, we aim to estimate the noise transition matrix without (pseudo) anchor points. There is evidence showing that samples are more likely to be mislabeled as other similar class labels, which means the mislabeling probability is highly correlated with the inter-class correlation. Inspired by this observation, we propose an instance-specific Label Distribution Regularization (LDR), in which the instance-specific LD is estimated as the supervision, to prevent DCNNs from memorizing noisy labels. Specifically, we estimate the noisy posterior under the supervision of noisy labels, and approximate the batch-level noise transition matrix by estimating the inter-class correlation matrix with neither anchor points nor pseudo anchor points. Experimental results on two synthetic noisy datasets and two real-world noisy datasets demonstrate that our LDR outperforms existing methods.