神经网络越来越依赖于复杂安全系统（例如自动驾驶汽车）的组成部分。对在更大的验证周期中嵌入神经网络验证的工具和方法的需求很高。但是，由于关注的广泛验证属性，很难进行神经网络验证，通常每个验证属性仅适用于专用求解器中的验证。在本文中，我们展示了最初设计用于验证，验证和仿真金融基础架构的功能编程语言的Imandra如何为神经网络验证提供整体基础架构。我们开发了一个新颖的图书馆Checkinn，该图书馆在Imandra的神经网络上形式化，并涵盖了神经网络验证的不同重要方面。

We present AI 2 , the first sound and scalable analyzer for deep neural networks. Based on overapproximation, AI 2 can automatically prove safety properties (e.g., robustness) of realistic neural networks (e.g., convolutional neural networks).The key insight behind AI 2 is to phrase reasoning about safety and robustness of neural networks in terms of classic abstract interpretation, enabling us to leverage decades of advances in that area. Concretely, we introduce abstract transformers that capture the behavior of fully connected and convolutional neural network layers with rectified linear unit activations (ReLU), as well as max pooling layers. This allows us to handle real-world neural networks, which are often built out of those types of layers.We present a complete implementation of AI 2 together with an extensive evaluation on 20 neural networks. Our results demonstrate that: (i) AI 2 is precise enough to prove useful specifications (e.g., robustness), (ii) AI 2 can be used to certify the effectiveness of state-of-the-art defenses for neural networks, (iii) AI 2 is significantly faster than existing analyzers based on symbolic analysis, which often take hours to verify simple fully connected networks, and (iv) AI 2 can handle deep convolutional networks, which are beyond the reach of existing methods.

由于它们在计算机视觉，图像处理和其他人领域的优异性能，卷积神经网络具有极大的普及。不幸的是，现在众所周知，卷积网络通常产生错误的结果 - 例如，这些网络的输入的小扰动可能导致严重的分类错误。近年来提出了许多验证方法，以证明没有此类错误，但这些通常用于完全连接的网络，并且在应用于卷积网络时遭受加剧的可扩展性问题。为了解决这一差距，我们在这里介绍了CNN-ABS框架，特别是旨在验证卷积网络。 CNN-ABS的核心是一种抽象细化技术，它通过拆除卷积连接，以便在这种方式创造原始问题的过度逼近来简化验证问题;如果产生的问题变得过于抽象，它会恢复这些连接。 CNN-ABS旨在使用现有的验证引擎作为后端，我们的评估表明它可以显着提高最先进的DNN验证引擎的性能，平均降低运行时间15.7％。

我们介绍了一种新颖的方法来对计算机程序进行自动终止分析：我们使用神经网络来表示排名功能。排名函数映射程序状态为从下面界限并随着程序运行而减小的值；排名函数的存在证明了程序终止。我们从程序的采样执行轨迹训练神经网络，以使网络的输出沿轨迹降低；然后，我们使用符号推理正式验证其对所有可能执行的概括。通过肯定的答案，我们获得了该计划的正式终止证书，我们称之为神经排名函数。我们证明，由于神经网络代表非线性功能的能力，我们的方法成功地超过了最先进工具的程序。这包括在其循环条件和包括非线性表达式的程序中使用析取的程序。

神经网络已广泛应用于垃圾邮件和网络钓鱼检测，入侵预防和恶意软件检测等安全应用程序。但是，这种黑盒方法通常在应用中具有不确定性和不良的解释性。此外，神经网络本身通常容易受到对抗攻击的影响。由于这些原因，人们对可信赖和严格的方法有很高的需求来验证神经网络模型的鲁棒性。对抗性的鲁棒性在处理恶意操纵输入时涉及神经网络的可靠性，是安全和机器学习中最热门的主题之一。在这项工作中，我们在神经网络的对抗性鲁棒性验证中调查了现有文献，并在机器学习，安全和软件工程领域收集了39项多元化研究工作。我们系统地分析了它们的方法，包括如何制定鲁棒性，使用哪种验证技术以及每种技术的优势和局限性。我们从正式验证的角度提供分类学，以全面理解该主题。我们根据财产规范，减少问题和推理策略对现有技术进行分类。我们还展示了使用样本模型在现有研究中应用的代表性技术。最后，我们讨论了未来研究的开放问题。

归纳逻辑编程（ILP）是一种机器学习的形式。ILP的目标是诱导推广培训示例的假设（一组逻辑规则）。随着ILP转30，我们提供了对该领域的新介绍。我们介绍了必要的逻辑符号和主要学习环境;描述ILP系统的构建块;比较几个维度的几个系统;描述四个系统（Aleph，Tilde，Aspal和Metagol）;突出关键应用领域;最后，总结了未来研究的当前限制和方向。

Deep neural networks have achieved impressive experimental results in image classification, but can surprisingly be unstable with respect to adversarial perturbations, that is, minimal changes to the input image that cause the network to misclassify it. With potential applications including perception modules and end-to-end controllers for self-driving cars, this raises concerns about their safety. We develop a novel automated verification framework for feed-forward multi-layer neural networks based on Satisfiability Modulo Theory (SMT). We focus on safety of image classification decisions with respect to image manipulations, such as scratches or changes to camera angle or lighting conditions that would result in the same class being assigned by a human, and define safety for an individual decision in terms of invariance of the classification within a small neighbourhood of the original image. We enable exhaustive search of the region by employing discretisation, and propagate the analysis layer by layer. Our method works directly with the network code and, in contrast to existing methods, can guarantee that adversarial examples, if they exist, are found for the given region and family of manipulations. If found, adversarial examples can be shown to human testers and/or used to fine-tune the network. We implement the techniques using Z3 and evaluate them on state-of-the-art networks, including regularised and deep learning networks. We also compare against existing techniques to search for adversarial examples and estimate network robustness.

Automatic differentiation (AD) is a technique for computing the derivative of a function represented by a program. This technique is considered as the de-facto standard for computing the differentiation in many machine learning and optimisation software tools. Despite the practicality of this technique, the performance of the differentiated programs, especially for functional languages and in the presence of vectors, is suboptimal. We present an AD system for a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source forward-mode AD and global optimisations such as loop transformations. In combination, gradient computation with forward-mode AD can be as efficient as reverse mode, and the Jacobian matrices required for numerical algorithms such as Gauss-Newton and Levenberg-Marquardt can be efficiently computed.

General mathematical reasoning is computationally undecidable, but humans routinely solve new problems. Moreover, discoveries developed over centuries are taught to subsequent generations quickly. What structure enables this, and how might that inform automated mathematical reasoning? We posit that central to both puzzles is the structure of procedural abstractions underlying mathematics. We explore this idea in a case study on 5 sections of beginning algebra on the Khan Academy platform. To define a computational foundation, we introduce Peano, a theorem-proving environment where the set of valid actions at any point is finite. We use Peano to formalize introductory algebra problems and axioms, obtaining well-defined search problems. We observe existing reinforcement learning methods for symbolic reasoning to be insufficient to solve harder problems. Adding the ability to induce reusable abstractions ("tactics") from its own solutions allows an agent to make steady progress, solving all problems. Furthermore, these abstractions induce an order to the problems, seen at random during training. The recovered order has significant agreement with the expert-designed Khan Academy curriculum, and second-generation agents trained on the recovered curriculum learn significantly faster. These results illustrate the synergistic role of abstractions and curricula in the cultural transmission of mathematics.

在本文中，我们介绍了两级晶格神经网络（FAST BATLLLNN）的刀具快速箱分析，作为两级格子（TLL）神经网络（NNS）的盒状输出约束的快速验证器。特别地，快速Batllnn可以验证给定TLL NN的输出是否始终在指定的超矩形内呈现，只要其输入约束到指定的凸多特级（不一定是超矩形）。 FAST BATLLNN使用TLL架构的唯一语义和盒状输出约束的解耦性质，从而显着提高具有通用多粒输出约束的TLL的已知多项式验证算法的验证性能。在本文中，我们评估了快速Batllnn的性能和可扩展性，无论是自身的权利，也与应用于TLL NNS的最先进的NN Verifers相比。快速的Batllnn比较最快的NN Verifiers非常有利地比较，完成我们的合成TLL测试台超过400倍，而不是最近的竞争对手。

深度神经网络（DNN）越来越多地用于安全至关重要的系统中，迫切需要保证其正确性。因此，验证社区设计了多种技术和工具来验证DNN。当DNN验证者发现触发错误的输入时，很容易确认；但是，当他们报告不存在错误时，就无法确保验证工具本身没有缺陷。由于在DNN验证工具中已经观察到了多个错误，因此这将DNN验证的适用性提出了质疑。在这项工作中，我们提出了一种具有证明生产能力的基于简单的DNN验证符的新型机制：产生易于检查的不可满足性的见证人，这证明了没有错误的情况。我们的证明生产是基于众所周知的Farkas引理的有效适应，并结合了处理分段线性函数和数值精确误差的机制。作为概念的证明，我们在Marabou DNN验证者之上实施了我们的技术。我们对避免空中碰撞的安全至关重要系统的评估表明，在几乎所有情况下，证明生产都成功了，只需要最小的开销。

Alphazero，Leela Chess Zero和Stockfish Nnue革新了计算机国际象棋。本书对此类引擎的技术内部工作进行了完整的介绍。该书分为四个主要章节 - 不包括第1章（简介）和第6章（结论）：第2章引入神经网络，涵盖了所有用于构建深层网络的基本构建块，例如Alphazero使用的网络。内容包括感知器，后传播和梯度下降，分类，回归，多层感知器，矢量化技术，卷积网络，挤压网络，挤压和激发网络，完全连接的网络，批处理归一化和横向归一化和跨性线性单位，残留层，剩余层，过度效果和底漆。第3章介绍了用于国际象棋发动机以及Alphazero使用的经典搜索技术。内容包括minimax，alpha-beta搜索和蒙特卡洛树搜索。第4章展示了现代国际象棋发动机的设计。除了开创性的Alphago，Alphago Zero和Alphazero我们涵盖Leela Chess Zero，Fat Fritz，Fat Fritz 2以及有效更新的神经网络（NNUE）以及MAIA。第5章是关于实施微型α。 Shexapawn是国际象棋的简约版本，被用作为此的示例。 Minimax搜索可以解决六ap峰，并产生了监督学习的培训位置。然后，作为比较，实施了类似Alphazero的训练回路，其中通过自我游戏进行训练与强化学习结合在一起。最后，比较了类似α的培训和监督培训。

我们在HOL4互动定理证明书的顶部实施了自动战术证据Tacticeoe。Tactice从人类证据中学习，数学技术适用于每个证明情况。然后在蒙特卡罗树搜索算法中使用这种知识来探索有前途的策略级证明路径。在一个CPU上，时间限制为60秒，Tactictoe在Hol4的标准图书馆中证明了7164定理的66.4％，而自动调度的电子箴言解决了34.5％。通过结合Tactice和电子证明者的结果，成功率上升至69.0％。

Deep neural networks have emerged as a widely used and effective means for tackling complex, real-world problems. However, a major obstacle in applying them to safety-critical systems is the great difficulty in providing formal guarantees about their behavior. We present a novel, scalable, and efficient technique for verifying properties of deep neural networks (or providing counter-examples). The technique is based on the simplex method, extended to handle the non-convex Rectified Linear Unit (ReLU ) activation function, which is a crucial ingredient in many modern neural networks. The verification procedure tackles neural networks as a whole, without making any simplifying assumptions. We evaluated our technique on a prototype deep neural network implementation of the next-generation airborne collision avoidance system for unmanned aircraft (ACAS Xu). Results show that our technique can successfully prove properties of networks that are an order of magnitude larger than the largest networks verified using existing methods.

即使机器学习算法已经在数据科学中发挥了重要作用，但许多当前方法对输入数据提出了不现实的假设。由于不兼容的数据格式，或数据集中的异质，分层或完全缺少的数据片段，因此很难应用此类方法。作为解决方案，我们提出了一个用于样本表示，模型定义和培训的多功能，统一的框架，称为“ Hmill”。我们深入审查框架构建和扩展的机器学习的多个范围范式。从理论上讲，为HMILL的关键组件的设计合理，我们将通用近似定理的扩展显示到框架中实现的模型所实现的所有功能的集合。本文还包含有关我们实施中技术和绩效改进的详细讨论，该讨论将在MIT许可下发布供下载。该框架的主要资产是其灵活性，它可以通过相同的工具对不同的现实世界数据源进行建模。除了单独观察到每个对象的一组属性的标准设置外，我们解释了如何在框架中实现表示整个对象系统的图表中的消息推断。为了支持我们的主张，我们使用框架解决了网络安全域的三个不同问题。第一种用例涉及来自原始网络观察结果的IoT设备识别。在第二个问题中，我们研究了如何使用以有向图表示的操作系统的快照可以对恶意二进制文件进行分类。最后提供的示例是通过网络中实体之间建模域黑名单扩展的任务。在所有三个问题中，基于建议的框架的解决方案可实现与专业方法相当的性能。

We present an approach for the verification of feed-forward neural networks in which all nodes have a piece-wise linear activation function. Such networks are often used in deep learning and have been shown to be hard to verify for modern satisfiability modulo theory (SMT) and integer linear programming (ILP) solvers.The starting point of our approach is the addition of a global linear approximation of the overall network behavior to the verification problem that helps with SMT-like reasoning over the network behavior. We present a specialized verification algorithm that employs this approximation in a search process in which it infers additional node phases for the non-linear nodes in the network from partial node phase assignments, similar to unit propagation in classical SAT solving. We also show how to infer additional conflict clauses and safe node fixtures from the results of the analysis steps performed during the search. The resulting approach is evaluated on collision avoidance and handwritten digit recognition case studies.

We consider the algorithmic problem of finding the optimal weights and biases for a two-layer fully connected neural network to fit a given set of data points. This problem is known as empirical risk minimization in the machine learning community. We show that the problem is $\exists\mathbb{R}$-complete. This complexity class can be defined as the set of algorithmic problems that are polynomial-time equivalent to finding real roots of a polynomial with integer coefficients. Furthermore, we show that arbitrary algebraic numbers are required as weights to be able to train some instances to optimality, even if all data points are rational. Our results hold even if the following restrictions are all added simultaneously. $\bullet$ There are exactly two output neurons. $\bullet$ There are exactly two input neurons. $\bullet$ The data has only 13 different labels. $\bullet$ The number of hidden neurons is a constant fraction of the number of data points. $\bullet$ The target training error is zero. $\bullet$ The ReLU activation function is used. This shows that even very simple networks are difficult to train. The result explains why typical methods for $\mathsf{NP}$-complete problems, like mixed-integer programming or SAT-solving, cannot train neural networks to global optimality, unless $\mathsf{NP}=\exists\mathbb{R}$. We strengthen a recent result by Abrahamsen, Kleist and Miltzow [NeurIPS 2021].

在过去十年中，已经开发出新的深度学习（DL）算法，工作负载和硬件来解决各种问题。尽管工作量和硬件生态系统的进步，DL系统的编程方法是停滞不前的。 DL工作负载从DL库中的高度优化，特定于平台和不灵活的内核，或者在新颖的操作员的情况下，通过具有强大性能的DL框架基元建立参考实现。这项工作介绍了Tensor加工基元（TPP），一个编程抽象，用于高效的DL工作负载的高效，便携式实现。 TPPS定义了一组紧凑而多才多艺的2D张镜操作员（或虚拟张量ISA），随后可以用作构建块，以在高维张量上构建复杂的运算符。 TPP规范是平台 - 不可行的，因此通过TPPS表示的代码是便携式的，而TPP实现是高度优化的，并且特定于平台。我们展示了我们使用独立内核和端到端DL＆HPC工作负载完全通过TPPS表达的方法的效力和生存性，这在多个平台上优于最先进的实现。

这本数字本书包含在物理模拟的背景下与深度学习相关的一切实际和全面的一切。尽可能多，所有主题都带有Jupyter笔记本的形式的动手代码示例，以便快速入门。除了标准的受监督学习的数据中，我们将看看物理丢失约束，更紧密耦合的学习算法，具有可微分的模拟，以及加强学习和不确定性建模。我们生活在令人兴奋的时期：这些方法具有从根本上改变计算机模拟可以实现的巨大潜力。

这是一门专门针对STEM学生开发的介绍性机器学习课程。我们的目标是为有兴趣的读者提供基础知识，以在自己的项目中使用机器学习，并将自己熟悉术语作为进一步阅读相关文献的基础。在这些讲义中，我们讨论受监督，无监督和强化学习。注释从没有神经网络的机器学习方法的说明开始，例如原理分析，T-SNE，聚类以及线性回归和线性分类器。我们继续介绍基本和先进的神经网络结构，例如密集的进料和常规神经网络，经常性的神经网络，受限的玻尔兹曼机器，（变性）自动编码器，生成的对抗性网络。讨论了潜在空间表示的解释性问题，并使用梦和对抗性攻击的例子。最后一部分致力于加强学习，我们在其中介绍了价值功能和政策学习的基本概念。