Recent years have seen rapid progress at the intersection between causality and machine learning. Motivated by scientific applications involving high-dimensional data, in particular in biomedicine, we propose a deep neural architecture for learning causal relationships between variables from a combination of empirical data and prior causal knowledge. We combine convolutional and graph neural networks within a causal risk framework to provide a flexible and scalable approach. Empirical results include linear and nonlinear simulations (where the underlying causal structures are known and can be directly compared against), as well as a real biological example where the models are applied to high-dimensional molecular data and their output compared against entirely unseen validation experiments. These results demonstrate the feasibility of using deep learning approaches to learn causal networks in large-scale problems spanning thousands of variables.

Pre-publication draft of a book to be published byMorgan & Claypool publishers. Unedited version released with permission. All relevant copyrights held by the author and publisher extend to this pre-publication draft.

Inferring causal structure poses a combinatorial search problem that typically involves evaluating structures with a score or independence test. The resulting search is costly, and designing suitable scores or tests that capture prior knowledge is difficult. In this work, we propose to amortize causal structure learning. Rather than searching over structures, we train a variational inference model to directly predict the causal structure from observational or interventional data. This allows our inference model to acquire domain-specific inductive biases for causal discovery solely from data generated by a simulator, bypassing both the hand-engineering of suitable score functions and the search over graphs. The architecture of our inference model emulates permutation invariances that are crucial for statistical efficiency in structure learning, which facilitates generalization to significantly larger problem instances than seen during training. On synthetic data and semisynthetic gene expression data, our models exhibit robust generalization capabilities when subject to substantial distribution shifts and significantly outperform existing algorithms, especially in the challenging genomics domain. Our code and models are publicly available at: https://github.com/larslorch/avici.

因果关系是理解世界的科学努力的基本组成部分。不幸的是，在心理学和社会科学中，因果关系仍然是禁忌。由于越来越多的建议采用因果方法进行研究的重要性，我们重新制定了心理学研究方法的典型方法，以使不可避免的因果理论与其余的研究渠道协调。我们提出了一个新的过程，该过程始于从因果发现和机器学习的融合中纳入技术的发展，验证和透明的理论形式规范。然后，我们提出将完全指定的理论模型的复杂性降低到与给定目标假设相关的基本子模型中的方法。从这里，我们确定利息量是否可以从数据中估算出来，如果是的，则建议使用半参数机器学习方法来估计因果关系。总体目标是介绍新的研究管道，该管道可以（a）促进与测试因果理论的愿望兼容的科学询问（b）鼓励我们的理论透明代表作为明确的数学对象，（c）将我们的统计模型绑定到我们的统计模型中该理论的特定属性，因此减少了理论到模型间隙通常引起的规范不足问题，以及（d）产生因果关系和可重复性的结果和估计。通过具有现实世界数据的教学示例来证明该过程，我们以摘要和讨论来结论。

本文提出了一种新的因果发现方法，即结构不可知的建模（SAM）。SAM利用条件独立性和分布不对称性，旨在从观察数据中找到潜在的因果结构。该方法基于不同玩家之间的游戏，该游戏将每个变量分布有条件地作为神经网估算，而对手则旨在区分生成的数据与原始数据。结合分布估计，稀疏性和无环限制的学习标准用于通过随机梯度下降来实施图形结构和参数的优化。SAM在合成和真实数据上进行了实验验证。

因果推断的一个共同主题是学习观察到的变量（也称为因果发现）之间的因果关系。考虑到大量候选因果图和搜索空间的组合性质，这通常是一项艰巨的任务。也许出于这个原因，到目前为止，大多数研究都集中在相对较小的因果图上，并具有多达数百个节点。但是，诸如生物学之类的领域的最新进展使生成实验数据集，并进行了数千种干预措施，然后进行了数千个变量的丰富分析，从而增加了机会和迫切需要大量因果图模型。在这里，我们介绍了因子定向无环图（F-DAG）的概念，是将搜索空间限制为非线性低级别因果相互作用模型的一种方法。将这种新颖的结构假设与最近的进步相结合，弥合因果发现与连续优化之间的差距，我们在数千个变量上实现了因果发现。此外，作为统计噪声对此估计程序的影响的模型，我们根据随机图研究了F-DAG骨架的边缘扰动模型，并量化了此类扰动对F-DAG等级的影响。该理论分析表明，一组候选F-DAG比整个DAG空间小得多，因此在很难评估基础骨架的高维度中更统计学上的稳定性。我们提出了因子图（DCD-FG）的可区分因果发现，这是对高维介入数据的F-DAG约束因果发现的可扩展实现。 DCD-FG使用高斯非线性低级结构方程模型，并且在模拟中的最新方法以及最新的大型单细胞RNA测序数据集中，与最新方法相比显示出显着改善遗传干预措施。

因果表示学习是识别基本因果变量及其从高维观察（例如图像）中的关系的任务。最近的工作表明，可以从观测的时间序列中重建因果变量，假设它们之间没有瞬时因果关系。但是，在实际应用中，我们的测量或帧速率可能比许多因果效应要慢。这有效地产生了“瞬时”效果，并使以前的可识别性结果无效。为了解决这个问题，我们提出了ICITRI，这是一种因果表示学习方法，当具有已知干预目标的完美干预措施时，可以在时间序列中处理瞬时效应。 Icitris从时间观察中识别因果因素，同时使用可区分的因果发现方法来学习其因果图。在三个视频数据集的实验中，Icitris准确地识别了因果因素及其因果图。

因果推断对于跨业务参与，医疗和政策制定等领域的数据驱动决策至关重要。然而，关于因果发现的研究已经与推理方法分开发展，从而阻止了两个领域方法的直接组合。在这项工作中，我们开发了深层端到端因果推理（DECI），这是一种基于流动的非线性添加噪声模型，该模型具有观察数据，并且可以执行因果发现和推理，包括有条件的平均治疗效果（CATE） ）估计。我们提供了理论上的保证，即DECI可以根据标准因果发现假设恢复地面真实因果图。受应用影响的激励，我们将该模型扩展到具有缺失值的异质，混合型数据，从而允许连续和离散的治疗决策。我们的结果表明，与因果发现的相关基线相比，DECI的竞争性能和（c）在合成数据集和因果机器学习基准测试基准的一千多个实验中，跨数据类型和缺失水平进行了估计。

Causal learning has attracted much attention in recent years because causality reveals the essential relationship between things and indicates how the world progresses. However, there are many problems and bottlenecks in traditional causal learning methods, such as high-dimensional unstructured variables, combinatorial optimization problems, unknown intervention, unobserved confounders, selection bias and estimation bias. Deep causal learning, that is, causal learning based on deep neural networks, brings new insights for addressing these problems. While many deep learning-based causal discovery and causal inference methods have been proposed, there is a lack of reviews exploring the internal mechanism of deep learning to improve causal learning. In this article, we comprehensively review how deep learning can contribute to causal learning by addressing conventional challenges from three aspects: representation, discovery, and inference. We point out that deep causal learning is important for the theoretical extension and application expansion of causal science and is also an indispensable part of general artificial intelligence. We conclude the article with a summary of open issues and potential directions for future work.

从观察数据中推断出因果关系很少直接，但是在高维度中，问题尤其困难。对于这些应用，因果发现算法通常需要参数限制或极端稀疏限制。我们放松这些假设，并专注于一个重要但更专业的问题，即在已知的变量子中恢复因果秩序，这些变量已知会从某些（可能很大的）混杂的协变量（即$ \ textit {Confounder Blanset} $）中降下。这在许多环境中很有用，例如，在研究具有背景信息的遗传数据的动态生物分子子系统时。在一个称为$ \ textit {混杂的毯子原理} $的结构假设下，我们认为这对于在高维度中的可拖动因果发现至关重要，我们的方法可容纳低或高稀疏性的图形，同时保持多项式时间复杂性。我们提出了一种结构学习算法，相对于所谓的$ \ textit {Lazy Oracle} $，该算法是合理且完整的。我们设计了线性和非线性系统有限样本误差控制的推理过程，并在一系列模拟和现实世界数据集上演示了我们的方法。随附的$ \ texttt {r} $ package，$ \ texttt {cbl} $可从$ \ texttt {cran} $获得。

本文提出了在适当的监督信息下进行分解的生成因果代表（亲爱的）学习方法。与实施潜在变量独立性的现有分解方法不同，我们考虑了一种基本利益因素可以因果关系相关的一般情况。我们表明，即使在监督下，先前具有独立先验的方法也无法解散因果关系。在这一发现的激励下，我们提出了一种称为DEAR的新的解开学习方法，该方法可以使因果可控的产生和因果代表学习。这种新公式的关键要素是使用结构性因果模型（SCM）作为双向生成模型的先验分布。然后，使用合适的GAN算法与发电机和编码器共同训练了先验，并与有关地面真相因子及其基本因果结构的监督信息合并。我们提供了有关该方法的可识别性和渐近收敛性的理论理由。我们对合成和真实数据集进行了广泛的实验，以证明DEAR在因果可控生成中的有效性，以及在样本效率和分布鲁棒性方面，学到的表示表示对下游任务的好处。

Latent variable models such as the Variational Auto-Encoder (VAE) have become a go-to tool for analyzing biological data, especially in the field of single-cell genomics. One remaining challenge is the interpretability of latent variables as biological processes that define a cell's identity. Outside of biological applications, this problem is commonly referred to as learning disentangled representations. Although several disentanglement-promoting variants of the VAE were introduced, and applied to single-cell genomics data, this task has been shown to be infeasible from independent and identically distributed measurements, without additional structure. Instead, recent methods propose to leverage non-stationary data, as well as the sparse mechanism shift assumption in order to learn disentangled representations with a causal semantic. Here, we extend the application of these methodological advances to the analysis of single-cell genomics data with genetic or chemical perturbations. More precisely, we propose a deep generative model of single-cell gene expression data for which each perturbation is treated as a stochastic intervention targeting an unknown, but sparse, subset of latent variables. We benchmark these methods on simulated single-cell data to evaluate their performance at latent units recovery, causal target identification and out-of-domain generalization. Finally, we apply those approaches to two real-world large-scale gene perturbation data sets and find that models that exploit the sparse mechanism shift hypothesis surpass contemporary methods on a transfer learning task. We implement our new model and benchmarks using the scvi-tools library, and release it as open-source software at \url{https://github.com/Genentech/sVAE}.

Machine learning on graphs is an important and ubiquitous task with applications ranging from drug design to friendship recommendation in social networks. The primary challenge in this domain is finding a way to represent, or encode, graph structure so that it can be easily exploited by machine learning models. Traditionally, machine learning approaches relied on user-defined heuristics to extract features encoding structural information about a graph (e.g., degree statistics or kernel functions). However, recent years have seen a surge in approaches that automatically learn to encode graph structure into low-dimensional embeddings, using techniques based on deep learning and nonlinear dimensionality reduction. Here we provide a conceptual review of key advancements in this area of representation learning on graphs, including matrix factorization-based methods, random-walk based algorithms, and graph neural networks. We review methods to embed individual nodes as well as approaches to embed entire (sub)graphs. In doing so, we develop a unified framework to describe these recent approaches, and we highlight a number of important applications and directions for future work.

Graph machine learning has been extensively studied in both academia and industry. Although booming with a vast number of emerging methods and techniques, most of the literature is built on the in-distribution hypothesis, i.e., testing and training graph data are identically distributed. However, this in-distribution hypothesis can hardly be satisfied in many real-world graph scenarios where the model performance substantially degrades when there exist distribution shifts between testing and training graph data. To solve this critical problem, out-of-distribution (OOD) generalization on graphs, which goes beyond the in-distribution hypothesis, has made great progress and attracted ever-increasing attention from the research community. In this paper, we comprehensively survey OOD generalization on graphs and present a detailed review of recent advances in this area. First, we provide a formal problem definition of OOD generalization on graphs. Second, we categorize existing methods into three classes from conceptually different perspectives, i.e., data, model, and learning strategy, based on their positions in the graph machine learning pipeline, followed by detailed discussions for each category. We also review the theories related to OOD generalization on graphs and introduce the commonly used graph datasets for thorough evaluations. Finally, we share our insights on future research directions. This paper is the first systematic and comprehensive review of OOD generalization on graphs, to the best of our knowledge.

考虑基于AI和ML的决策对这些新兴技术的安全和可接受的使用的决策的社会和道德后果至关重要。公平，特别是保证ML决定不会导致对个人或少数群体的歧视。使用因果关系，可以更好地实现和衡量可靠的公平/歧视，从而更好地实现了敏感属性（例如性别，种族，宗教等）之间的因果关系，仅仅是仅仅是关联，例如性别，种族，宗教等（例如，雇用工作，贷款授予等） ）。然而，对因果关系解决公平性的最大障碍是因果模型的不可用（通常表示为因果图）。文献中现有的因果关系方法并不能解决此问题，并假设可获得因果模型。在本文中，我们没有做出这样的假设，并且我们回顾了从可观察数据中发现因果关系的主要算法。这项研究的重点是因果发现及其对公平性的影响。特别是，我们展示了不同的因果发现方法如何导致不同的因果模型，最重要的是，即使因果模型之间的轻微差异如何对公平/歧视结论产生重大影响。通过使用合成和标准公平基准数据集的经验分析来巩固这些结果。这项研究的主要目标是强调因果关系使用因果关系适当解决公平性的因果发现步骤的重要性。

In this review, we discuss approaches for learning causal structure from data, also called causal discovery. In particular, we focus on approaches for learning directed acyclic graphs (DAGs) and various generalizations which allow for some variables to be unobserved in the available data. We devote special attention to two fundamental combinatorial aspects of causal structure learning. First, we discuss the structure of the search space over causal graphs. Second, we discuss the structure of equivalence classes over causal graphs, i.e., sets of graphs which represent what can be learned from observational data alone, and how these equivalence classes can be refined by adding interventional data.

从视觉观察中了解动态系统的潜在因果因素被认为是对复杂环境中推理的推理的关键步骤。在本文中，我们提出了Citris，这是一种变异自动编码器框架，从图像的时间序列中学习因果表示，其中潜在的因果因素可能已被干预。与最近的文献相反，Citris利用了时间性和观察干预目标，以鉴定标量和多维因果因素，例如3D旋转角度。此外，通过引入归一化流，可以轻松扩展柑橘，以利用和删除已验证的自动编码器获得的删除表示形式。在标量因果因素上扩展了先前的结果，我们在更一般的环境中证明了可识别性，其中仅因果因素的某些成分受干预措施影响。在对3D渲染图像序列的实验中，柑橘类似于恢复基本因果变量的先前方法。此外，使用预验证的自动编码器，Citris甚至可以概括为因果因素的实例化，从而在SIM到现实的概括中开放了未来的研究领域，以进行因果关系学习。

建议图表神经网络（GNNS）在不考虑训练和测试图之间的不可知分布的情况下，诱导GNN的泛化能力退化在分布外（OOD）设置。这种退化的根本原因是大多数GNN是基于I.I.D假设开发的。在这种设置中，GNN倾向于利用在培训中存在的微妙统计相关性用于预测，即使它是杂散的相关性。然而，这种杂散的相关性可能在测试环境中改变，导致GNN的失败。因此，消除了杂散相关的影响对于稳定的GNN来说是至关重要的。为此，我们提出了一个普遍的因果代表框架，称为稳定凝球。主要思想是首先从图数据中提取高级表示，并诉诸因因果推理的显着能力，以帮助模型摆脱虚假相关性。特别是，我们利用图形池化层以提取基于子图的表示作为高级表示。此外，我们提出了一种因果变量区别，以纠正偏置训练分布。因此，GNN将更多地集中在稳定的相关性上。对合成和现实世界ood图数据集的广泛实验良好地验证了所提出的框架的有效性，灵活性和可解释性。

了解因果关系有助于构建干预措施，以实现特定的目标并在干预下实现预测。随着学习因果关系的越来越重要，因果发现任务已经从使用传统方法推断出潜在的因果结构从观察数据到深度学习涉及的模式识别领域。大量数据的快速积累促进了具有出色可扩展性的因果搜索方法的出现。因果发现方法的现有摘要主要集中在基于约束，分数和FCM的传统方法上，缺乏针对基于深度学习的方法的完美分类和阐述，还缺乏一些考虑和探索因果关系的角度来探索因果发现方法范式。因此，我们根据变量范式将可能的因果发现任务分为三种类型，并分别给出三个任务的定义，定义和实例化每个任务的相关数据集以及同时构建的最终因果模型，然后审查不同任务的主要因果发现方法。最后，我们从不同角度提出了一些路线图，以解决因果发现领域的当前研究差距，并指出未来的研究方向。

药物的因果模型已用于分析机器学习系统的安全性方面。但是，识别代理是非平凡的 - 通常只是由建模者假设而没有太多理由来实现因果模型 - 建模失败可能会导致安全分析中的错误。本文提出了对代理商的第一个正式因果定义 - 大约是代理人是制度，如果他们的行为以不同的方式影响世界，则可以改善其政策。由此，我们得出了第一个用于从经验数据中发现代理的因果发现算法，并提供了用于在因果模型和游戏理论影响图之间转换的算法。我们通过解决不正确的因果模型引起的一些混乱来证明我们的方法。