We discover a robust self-supervised strategy tailored towards molecular representations for generative masked language models through a series of tailored, in-depth ablations. Using this pre-training strategy, we train BARTSmiles, a BART-like model with an order of magnitude more compute than previous self-supervised molecular representations. In-depth evaluations show that BARTSmiles consistently outperforms other self-supervised representations across classification, regression, and generation tasks setting a new state-of-the-art on 11 tasks. We then quantitatively show that when applied to the molecular domain, the BART objective learns representations that implicitly encode our downstream tasks of interest. For example, by selecting seven neurons from a frozen BARTSmiles, we can obtain a model having performance within two percentage points of the full fine-tuned model on task Clintox. Lastly, we show that standard attribution interpretability methods, when applied to BARTSmiles, highlight certain substructures that chemists use to explain specific properties of molecules. The code and the pretrained model are publicly available.

Models based on machine learning can enable accurate and fast molecular property predictions, which is of interest in drug discovery and material design. Various supervised machine learning models have demonstrated promising performance, but the vast chemical space and the limited availability of property labels make supervised learning challenging. Recently, unsupervised transformer-based language models pretrained on a large unlabelled corpus have produced state-of-the-art results in many downstream natural language processing tasks. Inspired by this development, we present molecular embeddings obtained by training an efficient transformer encoder model, MoLFormer, which uses rotary positional embeddings. This model employs a linear attention mechanism, coupled with highly distributed training, on SMILES sequences of 1.1 billion unlabelled molecules from the PubChem and ZINC datasets. We show that the learned molecular representation outperforms existing baselines, including supervised and self-supervised graph neural networks and language models, on several downstream tasks from ten benchmark datasets. They perform competitively on two others. Further analyses, specifically through the lens of attention, demonstrate that MoLFormer trained on chemical SMILES indeed learns the spatial relationships between atoms within a molecule. These results provide encouraging evidence that large-scale molecular language models can capture sufficient chemical and structural information to predict various distinct molecular properties, including quantum-chemical properties.

Models that accurately predict properties based on chemical structure are valuable tools in drug discovery. However, for many properties, public and private training sets are typically small, and it is difficult for the models to generalize well outside of the training data. Recently, large language models have addressed this problem by using self-supervised pretraining on large unlabeled datasets, followed by fine-tuning on smaller, labeled datasets. In this paper, we report MolE, a molecular foundation model that adapts the DeBERTa architecture to be used on molecular graphs together with a two-step pretraining strategy. The first step of pretraining is a self-supervised approach focused on learning chemical structures, and the second step is a massive multi-task approach to learn biological information. We show that fine-tuning pretrained MolE achieves state-of-the-art results on 9 of the 22 ADMET tasks included in the Therapeutic Data Commons.

分子性质预测在化学中至关重要，特别是对于药物发现应用。但是，可用的分子属性数据通常受到限制，鼓励信息从相关数据传输。转移学习对计算机视觉和自然语言处理信号等领域产生了巨大影响，以实现其在分子财产预测中的潜力。我们提出了使用反应数据进行分子表示学习的预训练程序，并将其用于预训练微笑变压器。我们对从物理化学，生物物理学和生理学中的分子的12个分子性质预测任务进行微调和评估预先训练的模型，并与非预先训练的基线模型相比，对12个任务中的5个任务显示出统计学上的显着积极作用。

Recently, deep learning approaches have been extensively studied for various problems in chemistry, such as property prediction, virtual screening, de novo molecule design, etc. Despite the impressive successes, separately designed networks for specific tasks are usually required for end-to-end training, so it is often difficult to acquire a unified principle to synergistically combine existing models and training datasets for novel tasks. To address this, here we present a novel multimodal chemical foundation model that can be used for various downstream tasks that require a simultaneous understanding of structure and property. Specifically, inspired by recent advances in pre-trained multi-modal foundation models such as Vision-Language Pretrained models (VLP), we proposed a novel structure-property multi-modal (SPMM) foundation model using the dual-stream transformer with X-shape attention, so that it can align the molecule structure and the chemical properties in a common embedding space. Thanks to the outstanding structure-property unimodal representation, experimental results confirm that SPMM can simultaneously perform molecule generation, property prediction, classification, reaction prediction, etc., which was previously not possible with a single architecture.

使用图神经网络（GNN）提取分子的信息表示，对于AI驱动的药物发现至关重要。最近，图形研究界一直在试图复制自然语言处理预处理的成功，并获得了一些成功。但是，我们发现在许多情况下，自我监督预审计对分子数据的益处可以忽略不计。我们对GNN预处理的关键组成部分进行了彻底的消融研究，包括预处理目标，数据拆分方法，输入特征，预处理数据集量表和GNN体系结构，以决定下游任务的准确性。我们的第一个重要发现是，在许多情况下，自我监督的图表预处理没有统计学上的显着优势。其次，尽管可以通过额外的监督预处理可以观察到改进，但通过更丰富或更平衡的数据拆分，改进可能会减少。第三，实验性超参数对下游任务的准确性具有更大的影响，而不是训练训练的任务。我们假设对分子进行预训练的复杂性不足，从而导致下游任务的可转移知识较低。

虽然最近在许多科学领域都变得无处不在，但对其评估的关注较少。对于分子生成模型，最先进的是孤立或与其输入有关的输出。但是，它们的生物学和功能特性（例如配体 - 靶标相互作用）尚未得到解决。在这项研究中，提出了一种新型的生物学启发的基准，用于评估分子生成模型。具体而言，设计了三个不同的参考数据集，并引入了与药物发现过程直接相关的一组指标。特别是我们提出了一个娱乐指标，将药物目标亲和力预测和分子对接应用作为评估生成产量的互补技术。虽然所有三个指标均在测试的生成模型中均表现出一致的结果，但对药物目标亲和力结合和分子对接分数进行了更详细的比较，表明单峰预测器可能会导致关于目标结合在分子水平和多模式方法的错误结论，而多模式的方法是错误的结论。因此优选。该框架的关键优点是，它通过明确关注配体 - 靶标相互作用，将先前的物理化学域知识纳入基准测试过程，从而创建了一种高效的工具，不仅用于评估分子生成型输出，而且还用于丰富富含分子生成的输出。一般而言，药物发现过程。

分子特性预测是与关键现实影响的深度学习的增长最快的应用之一。包括3D分子结构作为学习模型的输入可以提高它们对许多分子任务的性能。但是，此信息是不可行的，可以以几个现实世界应用程序所需的规模计算。我们建议预先训练模型，以推理仅给予其仅为2D分子图的分子的几何形状。使用来自自我监督学习的方法，我们最大化3D汇总向量和图形神经网络（GNN）的表示之间的相互信息，使得它们包含潜在的3D信息。在具有未知几何形状的分子上进行微调期间，GNN仍然产生隐式3D信息，并可以使用它来改善下游任务。我们表明3D预训练为广泛的性质提供了显着的改进，例如八个量子力学性能的22％的平均MAE。此外，可以在不同分子空间中的数据集之间有效地传送所学习的表示。

Many applications of machine learning require a model to make accurate predictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naïve strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.However, pre-training on graph datasets remains a hard challenge. Several key studies (

自我监督学习（SSL）是一种通过利用数据中固有的监督来学习数据表示的方法。这种学习方法是药物领域的焦点，由于耗时且昂贵的实验，缺乏带注释的数据。使用巨大未标记数据的SSL显示出在分子属性预测方面表现出色的性能，但存在一些问题。 （1）现有的SSL模型是大规模的；在计算资源不足的情况下实现SSL有限制。 （2）在大多数情况下，它们不利用3D结构信息进行分子表示学习。药物的活性与药物分子的结构密切相关。但是，大多数当前模型不使用3D信息或部分使用它。 （3）以前对分子进行对比学习的模型使用置换原子和键的增强。因此，具有不同特征的分子可以在相同的阳性样品中。我们提出了一个新颖的对比学习框架，用于分子属性预测的小规模3D图对比度学习（3DGCL），以解决上述问题。 3DGCL通过不改变药物语义的预训练过程来反映分子的结构来学习分子表示。仅使用1,128个样本用于预训练数据和100万个模型参数，我们在四个回归基准数据集中实现了最先进或可比性的性能。广泛的实验表明，基于化学知识的3D结构信息对于用于财产预测的分子表示学习至关重要。

人工智能（AI）在过去十年中一直在改变药物发现的实践。各种AI技术已在广泛的应用中使用，例如虚拟筛选和药物设计。在本调查中，我们首先概述了药物发现，并讨论了相关的应用，可以减少到两个主要任务，即分子性质预测和分子产生。然后，我们讨论常见的数据资源，分子表示和基准平台。此外，为了总结AI在药物发现中的进展情况，我们介绍了在调查的论文中包括模型架构和学习范式的相关AI技术。我们预计本调查将作为有兴趣在人工智能和药物发现界面工作的研究人员的指南。我们还提供了GitHub存储库（HTTPS:///github.com/dengjianyuan/survey_survey_au_drug_discovery），其中包含文件和代码，如适用，作为定期更新的学习资源。

Despite significant progress of generative models in the natural sciences, their controllability remains challenging. One fundamentally missing aspect of molecular or protein generative models is an inductive bias that can reflect continuous properties of interest. To that end, we propose the Regression Transformer (RT), a novel method that abstracts regression as a conditional sequence modeling problem. This introduces a new paradigm of multitask language models which seamlessly bridge sequence regression and conditional sequence generation. We thoroughly demonstrate that, despite using a nominal-scale training objective, the RT matches or surpasses the performance of conventional regression models in property prediction tasks of small molecules, proteins and chemical reactions. Critically, priming the same model with continuous properties yields a highly competitive conditional generative model that outperforms specialized approaches in a substructure-constrained, property-driven molecule generation benchmark. Our dichotomous approach is facilitated by a novel, alternating training scheme that enables the model to decorate seed sequences by desired properties, e.g., to optimize reaction yield. In sum, the RT is the first report of a multitask model that concurrently excels at predictive and generative tasks in biochemistry. This finds particular application in property-driven, local exploration of the chemical or protein space and could pave the road toward foundation models in material design. The code to reproduce all experiments of the paper is available at: https://github.com/IBM/regression-transformer

诸如GPT-3之类的大型审慎模型通过利用自学学习的学习来学习明显的表现，从而对现代自然语言处理产生了巨大影响，这些表现可以轻易地对各种下游任务进行挑剔。我们通过使用微笑语言构建化学基础模型Chemberta-2来研究将这种进步转移到分子机器学习中的可能性。虽然标记的分子预测任务数据通常很少，但微笑字符串的库很容易获得。在这项工作中，我们通过优化预处理过程来建立Chemberta。我们比较了通过不同的超参数和预处理数据集尺寸的多任务和自我监督预训练的预测，来自PubChem最多77m化合物。据我们所知，77m集合构成了迄今为止用于分子预处理的最大数据集之一。我们发现，通过这些预处理的改进，我们与Moleculenet基准套件上现有的最先进的体系结构具有竞争力。我们分析了预读的改进的程度，转化为下游任务的改进。

它是科学技术的基础，能够预测化学反应及其性质。为实现此类技能，重要的是要培养良好的化学反应表示，或者可以自动从数据中学习此类表示的良好深度学习架构。目前没有普遍和广泛采用的方法，可强健地代表化学反应。大多数现有方法患有一个或多个缺点，例如：（1）缺乏普遍性; （2）缺乏稳健性; （3）缺乏可解释性;或（4）需要过度手动预处理。在这里，我们利用基于图的分子结构表示，以开发和测试一个超图注意神经网络方法，以一次解决反应表示和性能 - 预测问题，减轻了上述缺点。我们使用三个独立数据集化学反应评估三个实验中的这种超照片表示。在所有实验中，基于超图的方法与其他表示和它们相应的化学反应模型相匹配或优于相应的模型，同时产生可解释的多级表示。

Molecular representation learning is crucial for the problem of molecular property prediction, where graph neural networks (GNNs) serve as an effective solution due to their structure modeling capabilities. Since labeled data is often scarce and expensive to obtain, it is a great challenge for GNNs to generalize in the extensive molecular space. Recently, the training paradigm of "pre-train, fine-tune" has been leveraged to improve the generalization capabilities of GNNs. It uses self-supervised information to pre-train the GNN, and then performs fine-tuning to optimize the downstream task with just a few labels. However, pre-training does not always yield statistically significant improvement, especially for self-supervised learning with random structural masking. In fact, the molecular structure is characterized by motif subgraphs, which are frequently occurring and influence molecular properties. To leverage the task-related motifs, we propose a novel paradigm of "pre-train, prompt, fine-tune" for molecular representation learning, named molecule continuous prompt tuning (MolCPT). MolCPT defines a motif prompting function that uses the pre-trained model to project the standalone input into an expressive prompt. The prompt effectively augments the molecular graph with meaningful motifs in the continuous representation space; this provides more structural patterns to aid the downstream classifier in identifying molecular properties. Extensive experiments on several benchmark datasets show that MolCPT efficiently generalizes pre-trained GNNs for molecular property prediction, with or without a few fine-tuning steps.

Although substantial efforts have been made using graph neural networks (GNNs) for AI-driven drug discovery (AIDD), effective molecular representation learning remains an open challenge, especially in the case of insufficient labeled molecules. Recent studies suggest that big GNN models pre-trained by self-supervised learning on unlabeled datasets enable better transfer performance in downstream molecular property prediction tasks. However, they often require large-scale datasets and considerable computational resources, which is time-consuming, computationally expensive, and environmentally unfriendly. To alleviate these limitations, we propose a novel pre-training model for molecular representation learning, Bi-branch Masked Graph Transformer Autoencoder (BatmanNet). BatmanNet features two tailored and complementary graph autoencoders to reconstruct the missing nodes and edges from a masked molecular graph. To our surprise, BatmanNet discovered that the highly masked proportion (60%) of the atoms and bonds achieved the best performance. We further propose an asymmetric graph-based encoder-decoder architecture for either nodes and edges, where a transformer-based encoder only takes the visible subset of nodes or edges, and a lightweight decoder reconstructs the original molecule from the latent representation and mask tokens. With this simple yet effective asymmetrical design, our BatmanNet can learn efficiently even from a much smaller-scale unlabeled molecular dataset to capture the underlying structural and semantic information, overcoming a major limitation of current deep neural networks for molecular representation learning. For instance, using only 250K unlabelled molecules as pre-training data, our BatmanNet with 2.575M parameters achieves a 0.5% improvement on the average AUC compared with the current state-of-the-art method with 100M parameters pre-trained on 11M molecules.

分子表示学习有助于多个下游任务，例如分子性质预测和药物设计。为了适当地代表分子，图形对比学习是一个有前途的范式，因为它利用自我监督信号并没有人类注释要求。但是，先前的作品未能将基本域名知识纳入图表语义，因此忽略了具有共同属性的原子之间的相关性，但不通过键连接连接。为了解决这些问题，我们构建化学元素知识图（KG），总结元素之间的微观关联，并提出了一种用于分子代表学习的新颖知识增强的对比学习（KCL）框架。 KCL框架由三个模块组成。第一个模块，知识引导的图形增强，基于化学元素kg增强原始分子图。第二模块，知识意识的图形表示，利用用于原始分子图的公共曲线图编码器和通过神经网络（KMPNN）的知识感知消息来提取分子表示来编码增强分子图中的复杂信息。最终模块是一种对比目标，在那里我们在分子图的这两个视图之间最大化协议。广泛的实验表明，KCL获得了八个分子数据集上的最先进基线的优异性能。可视化实验适当地解释了在增强分子图中从原子和属性中了解的KCL。我们的代码和数据可用于补充材料。

基于深度学习的分子建模的最新进步令人兴奋地加速硅药发现。可获得血清的生成模型，构建原子原子和键合或逐片键的分子。然而，许多药物发现项目需要固定的支架以存在于所生成的分子中，并纳入该约束仅探讨了该约束。在这里，我们提出了一种基于图形的模型，其自然地支持支架作为生成过程的初始种子，这是可能的，因为它不调节在发电历史上。我们的实验表明，Moler与最先进的方法进行了相当的方法，在无约会的分子优化任务上，并且在基于脚手架的任务上优于它们，而不是比现有方法从培训和样本更快的数量级。此外，我们展示了许多看似小设计选择对整体性能的影响。

We present $\textbf{MolT5}$ $-$ a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings. $\textbf{MolT5}$ allows for new, useful, and challenging analogs of traditional vision-language tasks, such as molecule captioning and text-based de novo molecule generation (altogether: translation between molecules and language), which we explore for the first time. Since $\textbf{MolT5}$ pretrains models on single-modal data, it helps overcome the chemistry domain shortcoming of data scarcity. Furthermore, we consider several metrics, including a new cross-modal embedding-based metric, to evaluate the tasks of molecule captioning and text-based molecule generation. Our results show that $\textbf{MolT5}$-based models are able to generate outputs, both molecules and captions, which in many cases are high quality.

逆合合成是一种将分子转化为潜在反应物的过程，因此鉴定了合成途径。我们提出了一个新颖的生成框架，称为$ \ mathsf {g^2retro} $，用于一步回曲预测。 $ \ mathsf {g^2retro} $模仿合成反应的反向逻辑，也就是说，首先预测反应中心以将靶分子转换为名为合成的片段，然后将合成剂转化为反应剂，然后按照先前的基于半电压的方法转换为反应剂。在预测反应中心时，$ \ mathsf {g^2retro} $定义了一组全面的反应中心类型，并通过考虑多个反应中心候选者来实现预测反应的多样性。在完成合成子时，$ \ mathsf {g^2retro} $部署了一系列子结构附件，以将合成物转换为反应物，该反应物利用了要完成的合成结构的最新结构的整体视图，以及所有所涉及的合成物和所有合成的结构产品结构。在这里，我们证明$ \ mathsf {g^2retro} $能够更好地对基准数据集中最可能的反应物进行优先级，而不是最先进的方法，并且发现了不包括在该方法中基准数据集。