动机：近年来，基于形象的生物测定稳步成为高吞吐量，引发了快速自动化方法，以提取来自数百种图像的生物学有意义的信息。从想象成的成功取得灵感，我们驯服细胞造就花，一个公开源和弱标记的显微镜图像的大规模数据集（890K图像，894级）。预先训练的细胞造黄养箱产生了对上游显微镜分类任务的想象成特征具有竞争力的功能。我们展示了CytoImAgenet的证据表明，CytoImAgenet在想象中训练有素的功能中捕获信息不可用。数据集是在https://www.kaggle.com/stanleyhua/cyaagenet中提供的。

我们将WS-DINO作为一种新型框架，以从细胞的高内感荧光图像学习表型表示中使用弱标记信息。我们的模型基于具有视觉变压器骨干（Dino）的知识蒸馏方法，我们将其用作研究的基准模型。使用WS-DINO，我们对高含量显微镜屏幕（处理和化合物）中可用的弱标签信息进行了微调，并在BBBC021数据集的非同样化合物的动作预测中实现了最先进的性能（98％），并使用该化合物作为弱标签，而非类型的化合物和批处理性能（96％）。我们的方法绕过单细胞种植作为预处理步骤，并使用自发图表表明该模型学习结构上有意义的表型曲线。

分析显微镜图像中细胞的形态可以为化合物或基因的功能提供洞察。解决此任务需要不仅可以从图像中提取生物信息的方法，而且还忽略了技术变异，即，用于收集显微镜图像的设备之间的实验过程或差异的变化。我们提出了与专家混合（团队）的嵌入学习方法提出了治疗计划，该方法学习了一组专家，专门专门捕获我们的培训集中的技术变异，然后在测试时间汇总专家的预测。因此，通过最大限度地减少每个专家的噪声，团队可以通过更少的技术变化偏差来学习强大的嵌入。要培训我们的模型，我们利用了处理样本，使我们的方法能够在每个小靶中捕获整个数据集的分布，同时仍然适用于GPU存储器。我们在三个数据集中评估了我们的方法，如药物发现，促进了识别细胞治疗的真实作用机制的表现，通过最先进的5.5-11％。

State-of-the-art visual perception models for a wide range of tasks rely on supervised pretraining. ImageNet classification is the de facto pretraining task for these models. Yet, ImageNet is now nearly ten years old and is by modern standards "small". Even so, relatively little is known about the behavior of pretraining with datasets that are multiple orders of magnitude larger. The reasons are obvious: such datasets are difficult to collect and annotate. In this paper, we present a unique study of transfer learning with large convolutional networks trained to predict hashtags on billions of social media images. Our experiments demonstrate that training for large-scale hashtag prediction leads to excellent results. We show improvements on several image classification and object detection tasks, and report the highest ImageNet-1k single-crop, top-1 accuracy to date: 85.4% (97.6% top-5). We also perform extensive experiments that provide novel empirical data on the relationship between large-scale pretraining and transfer learning performance. Name template Description train-IG-I-1.5k Instagram training set of I images and ∼1.5k hashtags from ImageNet-1k. train-IG-I-8.5k Instagram training set of I images and ∼8.5k hashtags from WordNet. train-IG-I-17k Instagram training set of I images and ∼17k hashtags from WordNet. train-IN-1M-1k The standard ImageNet-1k ILSVRC training set with 1.28M images. val-IN-50k-1k The standard ImageNet-1k ILSVRC validation set with 50k images. train-IN-I-L Extended ImageNet training set of I images and L ∈ {5k, 9k} labels. val-IN-I-L Extended ImageNet validation set of I images and L ∈ {5k, 9k} labels. train-CUB-6k-200 The Caltech-UCSD Birds-200-2011 training set. val-CUB-6k-200 The Caltech-UCSD Birds-200-2011 validation set. train-Places-1.8M-365 The Places365-Standard training set (high-resolution version). val-Places-37k-365 The Places365-Standard validation set (high-resolution version). train-COCO-135k-80 The standard COCO detection training set (2017 version). val-COCO-5k-80 The standard COCO detection validation set (2017 version). test-COCO-20k-80 The standard COCO detection test-dev set (2017 version).Table 1: Summary of image classification datasets. Each dataset is named with a template, role-source-I-L, that indicates its role (training, validation, testing), source, number of images I, and number of labels L.

High content imaging assays can capture rich phenotypic response data for large sets of compound treatments, aiding in the characterization and discovery of novel drugs. However, extracting representative features from high content images that can capture subtle nuances in phenotypes remains challenging. The lack of high-quality labels makes it difficult to achieve satisfactory results with supervised deep learning. Self-Supervised learning methods, which learn from automatically generated labels has shown great success on natural images, offer an attractive alternative also to microscopy images. However, we find that self-supervised learning techniques underperform on high content imaging assays. One challenge is the undesirable domain shifts present in the data known as batch effects, which may be caused by biological noise or uncontrolled experimental conditions. To this end, we introduce Cross-Domain Consistency Learning (CDCL), a novel approach that is able to learn in the presence of batch effects. CDCL enforces the learning of biological similarities while disregarding undesirable batch-specific signals, which leads to more useful and versatile representations. These features are organised according to their morphological changes and are more useful for downstream tasks - such as distinguishing treatments and mode of action.

自动显微镜和定量图像分析的进展已促进了高含量筛查（HCS）作为有效的药物发现和研究工具。尽管HCS提供了高吞吐量图像的复杂细胞表型，但该过程可能会受到图像畸变的阻碍，例如异常图像模糊，荧光团饱和度，碎屑，高噪声，高水平的噪声，意外的自动荧光或空的图像。尽管此问题在文献中受到了温和的关注，但忽略这些人工制品会严重阻碍下游图像处理任务，并阻碍对细微表型的发现。因此，在HCS中使用质量控制是主要问题，也是先决条件。在这项工作中，我们评估了不需要大量图像注释的深度学习选项，即可为此问题提供直接且易于使用的半监督学习解决方案。具体而言，我们比较了最近的自我监督和转移学习方法的功效，以提供高吞吐量伪像图像检测器的基础编码器。这项研究的结果表明，对于此任务，应首选转移学习方法，因为它们不仅在这里表现出色，而且具有不需要敏感的超参数设置或大量额外培训的优势。

Computational pathology can lead to saving human lives, but models are annotation hungry and pathology images are notoriously expensive to annotate. Self-supervised learning has shown to be an effective method for utilizing unlabeled data, and its application to pathology could greatly benefit its downstream tasks. Yet, there are no principled studies that compare SSL methods and discuss how to adapt them for pathology. To address this need, we execute the largest-scale study of SSL pre-training on pathology image data, to date. Our study is conducted using 4 representative SSL methods on diverse downstream tasks. We establish that large-scale domain-aligned pre-training in pathology consistently out-performs ImageNet pre-training in standard SSL settings such as linear and fine-tuning evaluations, as well as in low-label regimes. Moreover, we propose a set of domain-specific techniques that we experimentally show leads to a performance boost. Lastly, for the first time, we apply SSL to the challenging task of nuclei instance segmentation and show large and consistent performance improvements under diverse settings.

胸部射线照相是一种相对便宜，广泛的医疗程序，可传达用于进行诊断决策的关键信息。胸部X射线几乎总是用于诊断呼吸系统疾病，如肺炎或最近的Covid-19。在本文中，我们提出了一个自我监督的深神经网络，其在未标记的胸部X射线数据集上掠夺。学习的陈述转移到下游任务 - 呼吸系统疾病的分类。在四个公共数据集获得的结果表明，我们的方法在不需要大量标记的培训数据的情况下产生竞争力。

药物重新利用可以加速鉴定有效化合物用于针对SARS-COV-2的临床使用，并具有先前存在的临床安全数据和已建立的供应链的优势。 RNA病毒（例如SARS-COV-2）操纵细胞途径并诱导亚细胞结构的重组以支持其生命周期。可以使用生物成像技术来量化这些形态学的变化。在这项工作中，我们开发了DEEMD：使用深层神经网络模型在多个实例学习框架内的计算管道，以基于对公开可用RXRX19A数据集的形态分析来确定针对SARS-COV-2有效的推定治疗方法。该数据集由SARS-COV-2未感染的细胞和受感染细胞的荧光显微镜图像组成，有或没有药物治疗。 Deemd首先提取歧视性形态学特征，以产生来自未感染和感染细胞的细胞形态特征。然后在统计模型中使用这些形态学特征，以根据与未感染细胞的相似性估算受感染细胞的应用治疗疗效。 DEEMD能够通过弱监督定位受感染的细胞，而无需任何昂贵的像素级注释。 DEEMD确定已知的SARS-COV-2抑制剂，例如Remdesivir和Aloxistatin，支持我们方法的有效性。可以在其他新兴病毒和数据集上探索DEEMD，以便将来快速识别候选抗病毒药治疗}。我们的实施可在线网络https://www.github.com/sadegh-saberian/deemd

组织病理学仍然是各种癌症诊断的黄金标准。计算机视觉的最新进展，特别是深度学习，促进了针对各种任务的组织病理学图像的分析，包括免疫细胞检测和微卫星不稳定性分类。每个任务的最新工作通常采用鉴定的基础体系结构，这些体系结构已鉴定为图像分类。开发组织病理学分类器的标准方法倾向于将重点放在优化单个任务的模型上，而不是考虑建模创新的各个方面，从而改善了跨任务的概括。在这里，我们提出了Champkit（模型预测工具包的全面组织病理学评估）：可扩展的，完全可重现的基准测试工具包，由大量的斑点级图像分类任务组成，跨不同的癌症。 Champkit能够系统地记录模型和方法中提议改进的性能影响的一种方法。 Champkit源代码和数据可在https://github.com/kaczmarj/champkit上自由访问。

Molecular and genomic properties are critical in selecting cancer treatments to target individual tumors, particularly for immunotherapy. However, the methods to assess such properties are expensive, time-consuming, and often not routinely performed. Applying machine learning to H&E images can provide a more cost-effective screening method. Dozens of studies over the last few years have demonstrated that a variety of molecular biomarkers can be predicted from H&E alone using the advancements of deep learning: molecular alterations, genomic subtypes, protein biomarkers, and even the presence of viruses. This article reviews the diverse applications across cancer types and the methodology to train and validate these models on whole slide images. From bottom-up to pathologist-driven to hybrid approaches, the leading trends include a variety of weakly supervised deep learning-based approaches, as well as mechanisms for training strongly supervised models in select situations. While results of these algorithms look promising, some challenges still persist, including small training sets, rigorous validation, and model explainability. Biomarker prediction models may yield a screening method to determine when to run molecular tests or an alternative when molecular tests are not possible. They also create new opportunities in quantifying intratumoral heterogeneity and predicting patient outcomes.

转移学习已成为减轻医疗分类任务中缺乏标记数据的标准做法。虽然FineEning使用受监督的想象佩尔预押的下游任务预磨损的功能是简单的，并且在许多作品中进行了广泛的调查，但对自我监督预测的有用性很少有研究。在本文中，我们评估了通过从三种自我监督技术（SIMCLR，SWAV和DINO）对所选医疗分类任务的三种自我监控技术（SIMCLRR，SWAV和DINO）初始化的模型的性能来评估想象成自我监督的可转换性。所选择的任务涵盖Sentinel腋窝淋巴结图像中的肿瘤检测，眼底图像中的糖尿病视网膜病变分类以及胸部X射线图像中的多种病理条件分类。我们展示了自我监督的佩戴模型产生比其监督对应物更丰富的嵌入式，这鉴于线性评估和FineTuning均有益处下游任务。例如，考虑到在织物上的数据的线性评估，我们在糖尿病视网膜病变分类任务中看到高达14.79％的提高，肿瘤分类任务中的5.4％，肺炎中的7.03％AUC检测和9.4％的AUC在胸部X射线的病理条件下检测。此外，我们将动态视觉元嵌入（DVME）引入端到端的转移学习方法，融合来自多种型号的佩尔净化的嵌入物。我们表明，与使用单个掠过的模型方法相比，DVME获得的集体表示导致所选任务的性能的显着改进，并且可以推广到预磨料模型的任何组合。

在材料科学领域，显微镜是结构表征的第一个且通常仅可访问的方法。对可以自动化显微镜图像的分析和解释的机器学习方法的开发越来越感兴趣。通常，对机器学习模型进行培训需要大量具有相关结构标签的图像，但是，手动标记图像需要域知识，并且容易受到人为错误和主观性的影响。为了克服这些局限性，我们提出了一种半监督的转移学习方法，该方法使用少数标记的显微镜图像进行训练，并像在明显更大的图像数据集中训练的方法一样有效地执行。具体而言，我们使用自我监督的学习方法训练图像编码器，并使用编码器来传输不同下游图像任务（分类和细分），并使用最少数量的标记图像进行培训来传输该编码器。我们测试了两种自我监督学习方法的转移学习能力：传输电子显微镜（TEM）图像的SIMCLR和Barlow-Twins。我们详细说明了该机器学习工作流程如何应用于蛋白质纳米线的TEM图像如何实现纳米线形态的自动分类（例如，单纳米线，纳米线，纳米线捆绑包，相位分离）以及可以用作量化纳米域域的基础的分段任务和形状分析。我们还将机器学习工作流程的应用扩展到纳米颗粒形态的分类以及从TEM图像中鉴定不同类型病毒的分类。

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

Jitendra Malik once said, "Supervision is the opium of the AI researcher". Most deep learning techniques heavily rely on extreme amounts of human labels to work effectively. In today's world, the rate of data creation greatly surpasses the rate of data annotation. Full reliance on human annotations is just a temporary means to solve current closed problems in AI. In reality, only a tiny fraction of data is annotated. Annotation Efficient Learning (AEL) is a study of algorithms to train models effectively with fewer annotations. To thrive in AEL environments, we need deep learning techniques that rely less on manual annotations (e.g., image, bounding-box, and per-pixel labels), but learn useful information from unlabeled data. In this thesis, we explore five different techniques for handling AEL.

亚细胞蛋白的自动视觉定位可以加速我们对健康和疾病中细胞功能的理解。尽管机器学习最近取得了进步（ML），但人类仍然通过使用各种视觉提示获得了卓越的准确性。我们通过解决三个关键方面可以缩小这一差距：（i）单元注释质量的自动改善，（ii）支持不平衡和嘈杂数据的新的深神经网络（DNN）体系结构，以及（iii）知情的选择和融合。多种机器学习模型。我们介绍了一种新的``Ai-Trains-ai''方法，用于提高弱标签的质量，并提出了利用小波过滤器和Weibull激活的新型DNN体系结构。我们还通过分析图像级和细胞级预测之间的相关性来探索多-DNN结合过程中的关键因素。最后，在人类蛋白质地图集的背景下，我们证明了我们的系统在多标签的单细胞单细胞分类中实现了蛋白质定位模式的最新性能，同时增强了概括能力。

Pre-training general-purpose visual features with convolutional neural networks without relying on annotations is a challenging and important task. Most recent efforts in unsupervised feature learning have focused on either small or highly curated datasets like ImageNet, whereas using non-curated raw datasets was found to decrease the feature quality when evaluated on a transfer task. Our goal is to bridge the performance gap between unsupervised methods trained on curated data, which are costly to obtain, and massive raw datasets that are easily available. To that effect, we propose a new unsupervised approach which leverages self-supervision and clustering to capture complementary statistics from large-scale data. We validate our approach on 96 million images from YFCC100M [42], achieving state-of-the-art results among unsupervised methods on standard benchmarks, which confirms the potential of unsupervised learning when only non-curated raw data are available. We also show that pre-training a supervised VGG-16 with our method achieves 74.9% top-1 classification accuracy on the validation set of ImageNet, which is an improvement of +0.8% over the same network trained from scratch. Our code is available at https://github. com/facebookresearch/DeeperCluster.

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -from 1 example per class to 1 M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.

自我监督学习的最新发展使我们有可能进一步减少人类干预的多步管道中的干预，其中重点围绕着特定感兴趣的对象而发展。在本文中，焦点在组织病理学图像中的细胞核中放置。特别是，我们旨在以无监督的方式提取蜂窝信息，以完成下游任务。随着核以各种尺寸表现出来，我们提出了一个新的依赖量表卷积层来绕过调整核时尺寸的问题。在三个核数据集上，我们基准了以下方法：手工制作的，预先训练的重新系统，有监督的重新系统和自我监督的特征。我们表明，所提出的卷积层提高了性能，并且与Barlows-Twins结合使用，与低样本设置中的监督范式相比，该层可以更好地编码核编码，并且胜过所有其他建议的无监督方法。此外，我们将现有的TNBC数据集扩展到合并核类别的注释，以丰富和公开释放一个小样本设置数据集以进行核分割和分类。

恶意软件检测在网络安全中起着至关重要的作用，随着恶意软件增长的增加和网络攻击的进步。以前看不见的恶意软件不是由安全供应商确定的，这些恶意软件通常在这些攻击中使用，并且不可避免地要找到可以从未标记的样本数据中自学习的解决方案。本文介绍了Sherlock，这是一种基于自学的深度学习模型，可根据视觉变压器（VIT）体系结构检测恶意软件。 Sherlock是一种新颖的恶意软件检测方法，它可以通过使用基于图像的二进制表示形式来学习独特的功能，以区分恶意软件和良性程序。在47种类型和696个家庭的层次结构中使用120万个Android应用的实验结果表明，自我监督的学习可以达到97％的恶意软件分类，而恶意软件的二进制分类比现有的最新技术更高。我们提出的模型还能够胜过针对多级恶意软件类型和家庭的最先进技术，分别为.497和.491。