高质量数据对于现代机器学习是必需的。但是，由于人类的嘈杂和模棱两可的注释，难以获取此类数据。确定图像标签的这种注释的聚合导致数据质量较低。我们提出了一个以数据为中心的图像分类基准，该基准具有9个现实世界数据集和每个图像的多次注释，以调查和量化此类数据质量问题的影响。我们通过询问如何提高数据质量来关注以数据为中心的观点。在数千个实验中，我们表明多个注释可以更好地近似实际的基础类别分布。我们确定硬标签无法捕获数据的歧义，这可能会导致过度自信模型的常见问题。根据呈现的数据集，基准基准和分析，我们为未来创造了多个研究机会。

一贯的高数据质量对于深度学习领域的新型损失功能和体系结构的发展至关重要。通常假定存在此类数据和标签的存在，而在许多情况下，获取高质量数据集仍然是一个主要问题。在现实世界数据集中，由于注释者的主观注释，我们经常遇到模棱两可的标签。在我们以数据为中心的方法中，我们提出了一种重新标记标签的方法，而不是在神经网络中实施此问题的处理。根据定义，硬分类不足以捕获数据的现实歧义。因此，我们提出了方法“以数据为中心的分类和聚类（DC3）”，该方法结合了半监督分类和聚类。它会自动估计图像的歧义，并根据歧义进行分类或聚类。 DC3本质上是普遍的，因此除了许多半监督学习（SSL）算法外，还可以使用它。平均而言，这会导致分类的F1得分高7.6％，而在多个评估的SSL算法和数据集中，簇的内距离降低了7.9％。最重要的是，我们给出了概念验证，即DC3的分类和聚类是对此类模棱两可标签的手动完善的建议。总体而言，SSL与我们的方法DC3的组合可以在注释过程中更好地处理模棱两可的标签。

高质量数据是现代机器学习的关键方面。但是，人类产生的标签遭受了标签噪声和阶级歧义等问题。我们提出了一个问题，即硬标签是否足以在存在这些固有的不精确的情况下代表基本的地面真相分布。因此，我们将学习的差异与硬和软标签进行定量和定性，以获取合成和现实世界数据集。我们表明，软标签的应用可改善性能，并产生内部特征空间的更常规结构。

从大型预训练模型转移学习对于许多计算机视觉任务来说都是至关重要的。最近的研究表明，由于存在存在的多个对象类的图像被分配单个标签，所以类似于想象成的数据集弱标记。这种模糊的偏置模型朝向单一预测，这可能导致抑制数据中倾向于共同发生的类。灵感来自语言出现文学，我们提出了多标签迭代学习（英里）来利用迭代学习框架从单个标签中融入多标签学习的归纳偏见。英里是一种简单而有效的过程，通过通过与学习瓶颈的连续几代教师和学生网络传播二进制预测来构建图像的多标签描述。实验表明，我们的方法对Imagenet的准确性以及真正的F1分数表现出系统的益处，这表明英里与标签歧义更好地优于标准训练程序，即使在自我监督权重的微调时也会比标准训练程序更好。我们还表明英里有效地减少标签噪音，实现了最先进的性能，如WebVision等现实大规模嘈杂的数据。此外，英里提高了类增量设置中的性能，例如IIRC，它是强大的分发班次。代码：https://github.com/rajeswar18/mile.

This work tackles the problem of semi-supervised learning of image classifiers. Our main insight is that the field of semi-supervised learning can benefit from the quickly advancing field of self-supervised visual representation learning. Unifying these two approaches, we propose the framework of self-supervised semi-supervised learning (S 4 L) and use it to derive two novel semi-supervised image classification methods. We demonstrate the effectiveness of these methods in comparison to both carefully tuned baselines, and existing semi-supervised learning methods. We then show that S 4 L and existing semi-supervised methods can be jointly trained, yielding a new state-of-the-art result on semi-supervised ILSVRC-2012 with 10% of labels.

We show that large pre-trained language models are inherently highly capable of identifying label errors in natural language datasets: simply examining out-of-sample data points in descending order of fine-tuned task loss significantly outperforms more complex error-detection mechanisms proposed in previous work. To this end, we contribute a novel method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP. We show that this noise has similar properties to real, hand-verified label errors, and is harder to detect than existing synthetic noise, creating challenges for model robustness. We argue that human-originated noise is a better standard for evaluation than synthetic noise. Finally, we use crowdsourced verification to evaluate the detection of real errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than existing models.

As an important data selection schema, active learning emerges as the essential component when iterating an Artificial Intelligence (AI) model. It becomes even more critical given the dominance of deep neural network based models, which are composed of a large number of parameters and data hungry, in application. Despite its indispensable role for developing AI models, research on active learning is not as intensive as other research directions. In this paper, we present a review of active learning through deep active learning approaches from the following perspectives: 1) technical advancements in active learning, 2) applications of active learning in computer vision, 3) industrial systems leveraging or with potential to leverage active learning for data iteration, 4) current limitations and future research directions. We expect this paper to clarify the significance of active learning in a modern AI model manufacturing process and to bring additional research attention to active learning. By addressing data automation challenges and coping with automated machine learning systems, active learning will facilitate democratization of AI technologies by boosting model production at scale.

由于筛选乳房X线照片的假阴性评估，通常在晚期检测到与其他癌症更差的间隔和大型侵入性乳腺癌。错过的筛选时间检测通常由其周围乳腺组织模糊的肿瘤引起的，这是一种称为掩蔽的现象。为了研究和基准爆发癌症的乳房Xmmpare掩蔽，在这项工作中，我们引入CSAW-M，最大的公共乳房数据集，从10,000多个人收集并用潜在的掩蔽注释。与以前的方法对比测量乳房图像密度作为代理的方法，我们的数据集直接提供了五个专家屏蔽潜在评估的注释。我们还培训了CSAW-M的深入学习模型来估计掩蔽水平，并显示估计的掩蔽更加预测筛查患有间隔和大型侵入性癌症的参与者 - 而不是明确培训这些任务 - 而不是其乳房密度同行。

We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% -15% on CIFAR-10 and 11% -14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that SSL algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and performance can degrade substantially when the unlabeled dataset contains out-ofdistribution examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available. 2 * Equal contribution 2 https://github.com/brain-research/realistic-ssl-evaluation 32nd Conference on Neural Information Processing Systems (NeurIPS 2018),

Labeling a module defective or non-defective is an expensive task. Hence, there are often limits on how much-labeled data is available for training. Semi-supervised classifiers use far fewer labels for training models, but there are numerous semi-supervised methods, including self-labeling, co-training, maximal-margin, and graph-based methods, to name a few. Only a handful of these methods have been tested in SE for (e.g.) predicting defects and even that, those tests have been on just a handful of projects. This paper takes a wide range of 55 semi-supervised learners and applies these to over 714 projects. We find that semi-supervised "co-training methods" work significantly better than other approaches. However, co-training needs to be used with caution since the specific choice of co-training methods needs to be carefully selected based on a user's specific goals. Also, we warn that a commonly-used co-training method ("multi-view"-- where different learners get different sets of columns) does not improve predictions (while adding too much to the run time costs 11 hours vs. 1.8 hours). Those cautions stated, we find using these "co-trainers," we can label just 2.5% of data, then make predictions that are competitive to those using 100% of the data. It is an open question worthy of future work to test if these reductions can be seen in other areas of software analytics. All the codes used and datasets analyzed during the current study are available in the https://GitHub.com/Suvodeep90/Semi_Supervised_Methods.

注释数据是用于培训和评估机器学习模型的自然语言处理中的重要成分。因此，注释具有高质量是非常理想的。但是，最近的工作表明，几个流行的数据集包含令人惊讶的注释错误或不一致之处。为了减轻此问题，多年来已经设计了许多注释错误检测方法。尽管研究人员表明他们的方法在新介绍的数据集上效果很好，但他们很少将其方法与以前的工作或同一数据集进行比较。这引起了人们对方法的一般表现的强烈关注，并且使他们的优势和劣势很难解决。因此，我们重新实现18种检测潜在注释错误的方法，并在9个英语数据集上对其进行评估，以进行文本分类以及令牌和跨度标签。此外，我们定义了统一的评估设置，包括注释错误检测任务，评估协议和一般最佳实践的新形式化。为了促进未来的研究和可重复性，我们将数据集和实施释放到易于使用和开源软件包中。

主动学习（AL）算法旨在识别注释的最佳数据子集，使得深神经网络（DNN）在此标记子集上培训时可以实现更好的性能。 AL特别有影响的工业规模设置，其中数据标签成本高，从业者使用各种工具来处理，以提高模型性能。最近自我监督预测（SSP）的成功突出了利用丰富的未标记数据促进模型性能的重要性。通过将AL与SSP结合起来，我们可以使用未标记的数据，同时标记和培训特别是信息样本。在这项工作中，我们研究了Imagenet上的AL和SSP的组合。我们发现小型玩具数据集上的性能 - 文献中的典型基准设置 - 由于活动学习者选择的类不平衡样本，而不是想象中的性能。在我们测试的现有基线中，各种小型和大规​​模设置的流行AL算法未能以随机抽样优于差异。为了解决类别不平衡问题，我们提出了平衡选择（基础），这是一种简单，可伸缩的AL算法，通过选择比现有方法更加平衡样本来始终如一地始终采样。我们的代码可用于：https://github.com/zeyademam/active_learning。

The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music. These are the proceedings of the 3rd International Workshop on Reading Music Systems, held in Alicante on the 23rd of July 2021.

部署在野外的机器学习系统通常在源分布上培训，但部署在不同的目标分布上。未标记的数据可以是用于缓解这些分布班次的强大的利用点，因为它通常比标记数据更具可用。然而，未标记数据的现有分配转换基准不反映现实世界应用中出现的方案的广度。在这项工作中，我们介绍了Wilds 2.0更新，该更新在分发转移的野外基准中扩展了10个数据集中的8个，以包括将在部署中逼真获得的策划未标记数据。为了保持一致性，标记的培训，验证和测试集以及评估度量与原始野外基准中的标记与评估度量完全相同。这些数据集涵盖了广泛的应用程序（从组织学到野生动物保护），任务（分类，回归和检测）和方式（照片，卫星图像，显微镜载玻片，文本，分子图）。我们系统地基准测试最先进的方法，可以利用未标记的数据，包括域不变，自我培训和自我监督方法，并表明他们在野外的成功2.0是有限的。为了方便方法开发和评估，我们提供了一个自动化数据加载的开源包，并包含本文中使用的所有模型架构和方法。代码和排行榜可在https://wilds.stanford.edu获得。

Lifelong learners must recognize concept vocabularies that evolve over time. A common yet underexplored scenario is learning with class labels over time that refine/expand old classes. For example, humans learn to recognize ${\tt dog}$ before dog breeds. In practical settings, dataset $\textit{versioning}$ often introduces refinement to ontologies, such as autonomous vehicle benchmarks that refine a previous ${\tt vehicle}$ class into ${\tt school-bus}$ as autonomous operations expand to new cities. This paper formalizes a protocol for studying the problem of $\textit{Learning with Evolving Class Ontology}$ (LECO). LECO requires learning classifiers in distinct time periods (TPs); each TP introduces a new ontology of "fine" labels that refines old ontologies of "coarse" labels (e.g., dog breeds that refine the previous ${\tt dog}$). LECO explores such questions as whether to annotate new data or relabel the old, how to leverage coarse labels, and whether to finetune the previous TP's model or train from scratch. To answer these questions, we leverage insights from related problems such as class-incremental learning. We validate them under the LECO protocol through the lens of image classification (CIFAR and iNaturalist) and semantic segmentation (Mapillary). Our experiments lead to surprising conclusions; while the current status quo is to relabel existing datasets with new ontologies (such as COCO-to-LVIS or Mapillary1.2-to-2.0), LECO demonstrates that a far better strategy is to annotate $\textit{new}$ data with the new ontology. However, this produces an aggregate dataset with inconsistent old-vs-new labels, complicating learning. To address this challenge, we adopt methods from semi-supervised and partial-label learning. Such strategies can surprisingly be made near-optimal, approaching an "oracle" that learns on the aggregate dataset exhaustively labeled with the newest ontology.

咳嗽音频信号分类是筛查呼吸道疾病（例如COVID-19）的潜在有用工具。由于从这种传染性疾病的患者那里收集数据是危险的，因此许多研究团队已转向众包来迅速收集咳嗽声数据，因为它是为了生成咳嗽数据集的工作。 Coughvid数据集邀请专家医生诊断有限数量上传的记录中存在的潜在疾病。但是，这种方法遭受了咳嗽的潜在标签，以及专家之间的显着分歧。在这项工作中，我们使用半监督的学习（SSL）方法来提高咳嗽数据集的标签一致性以及COVID-19的鲁棒性与健康的咳嗽声音分类。首先，我们利用现有的SSL专家知识聚合技术来克服数据集中的标签不一致和稀疏性。接下来，我们的SSL方法用于识别可用于训练或增加未来咳嗽分类模型的重新标记咳嗽音频样本的子样本。证明了重新标记的数据的一致性，因为它表现出高度的类可分离性，尽管原始数据集中存在专家标签不一致，但它比用户标记的数据高3倍。此外，在重新标记的数据中放大了用户标记的音频段的频谱差异，从而导致健康和COVID-19咳嗽之间的功率频谱密度显着不同，这既证明了新数据集的一致性及其与新数据的一致性及其与新数据的一致性的提高，其解释性与其与其解释性的一致性相同。声学的观点。最后，我们演示了如何使用重新标记的数据集来训练咳嗽分类器。这种SSL方法可用于结合几位专家的医学知识，以提高任何诊断分类任务的数据库一致性。

我们研究了用于半监控学习（SSL）的无监督数据选择，其中可以提供大规模的未标记数据集，并且为标签采集预算小额数据子集。现有的SSL方法专注于学习一个有效地集成了来自给定小标记数据和大型未标记数据的信息的模型，而我们专注于选择正确的数据以用于SSL的注释，而无需任何标签或任务信息。直观地，要标记的实例应统称为下游任务的最大多样性和覆盖范围，并且单独具有用于SSL的最大信息传播实用程序。我们以三步数据为中心的SSL方法形式化这些概念，使稳定性和精度的纤维液改善8％的CiFar-10（标记为0.08％）和14％的Imagenet -1k（标记为0.2％）。它也是一种具有各种SSL方法的通用框架，提供一致的性能增益。我们的工作表明，在仔细选择注释数据上花费的小计算带来了大注释效率和模型性能增益，而无需改变学习管道。我们完全无监督的数据选择可以轻松扩展到其他弱监督的学习设置。

Jitendra Malik once said, "Supervision is the opium of the AI researcher". Most deep learning techniques heavily rely on extreme amounts of human labels to work effectively. In today's world, the rate of data creation greatly surpasses the rate of data annotation. Full reliance on human annotations is just a temporary means to solve current closed problems in AI. In reality, only a tiny fraction of data is annotated. Annotation Efficient Learning (AEL) is a study of algorithms to train models effectively with fewer annotations. To thrive in AEL environments, we need deep learning techniques that rely less on manual annotations (e.g., image, bounding-box, and per-pixel labels), but learn useful information from unlabeled data. In this thesis, we explore five different techniques for handling AEL.

在过去的十年中，计算机愿景，旨在了解视觉世界的人工智能分支，从简单地识别图像中的物体来描述图片，回答有关图像的问题，以及围绕物理空间的机器人操纵甚至产生新的视觉内容。随着这些任务和应用程序的现代化，因此依赖更多数据，用于模型培训或评估。在本章中，我们展示了新颖的互动策略可以为计算机愿景提供新的数据收集和评估。首先，我们提出了一种众群界面，以通过数量级加速付费数据收集，喂养现代视觉模型的数据饥饿性质。其次，我们探索使用自动社交干预措施增加志愿者贡献的方法。第三，我们开发一个系统，以确保人类对生成视觉模型的评估是可靠的，实惠和接地在心理物理学理论中。我们结束了人机互动的未来机会，以帮助计算机愿景。