QNNVerifier是第一个用于验证神经网络实现的开源工具，以考虑其操作数的有限字长（即量化）。通过采用最先进的软件模型检查（SMC）技术来实现对量化的新颖支持。它将神经网络的实现基于可满足模数理论（SMT）来将神经网络的实现到一阶逻辑的可解除片段。通过给定硬件确定的精度，通过直接实现来表示固定和浮点操作的影响。此外，Qnnverifier允许指定定制安全性能，并使用不同的验证策略（增量和K-Incuction）和SMT求解器来验证所产生的模型。最后，QNNVerifier是第一个通过间隔分析和非线性激活功能的离散化来组合不变推论的工具，以加快级别验证神经网络的级数。 qnnverifier的视频呈现可在https://youtu.be/7jmgol41zty中获得

While deep neural networks (DNNs) have demonstrated impressive performance in solving many challenging tasks, they are limited to resource-constrained devices owing to their demand for computation power and storage space. Quantization is one of the most promising techniques to address this issue by quantizing the weights and/or activation tensors of a DNN into lower bit-width fixed-point numbers. While quantization has been empirically shown to introduce minor accuracy loss, it lacks formal guarantees on that, especially when the resulting quantized neural networks (QNNs) are deployed in safety-critical applications. A majority of existing verification methods focus exclusively on individual neural networks, either DNNs or QNNs. While promising attempts have been made to verify the quantization error bound between DNNs and their quantized counterparts, they are not complete and more importantly do not support fully quantified neural networks, namely, only weights are quantized. To fill this gap, in this work, we propose a quantization error bound verification method (QEBVerif), where both weights and activation tensors are quantized. QEBVerif consists of two analyses: a differential reachability analysis (DRA) and a mixed-integer linear programming (MILP) based verification method. DRA performs difference analysis between the DNN and its quantized counterpart layer-by-layer to efficiently compute a tight quantization error interval. If it fails to prove the error bound, then we encode the verification problem into an equivalent MILP problem which can be solved by off-the-shelf solvers. Thus, QEBVerif is sound, complete, and arguably efficient. We implement QEBVerif in a tool and conduct extensive experiments, showing its effectiveness and efficiency.

本文提出了一种验证网络物理安全 - 关键系统中发现的非线性人工神经网络（ANN）行为的方法。我们将Sigmoid函数的专用间隔约束传播器实施到SMT求解器ISAT中，并将这种方法与组成方法进行比较，该方法通过ISAT中可用的基本算术特征和近似方法来编码Sigmoid函数。我们的实验结果表明，专用和组成方法明显优于近似方法。在我们所有的基准中，专门的方法与组成方法相比表现出相等或更好的性能。

We study the problem of training and certifying adversarially robust quantized neural networks (QNNs). Quantization is a technique for making neural networks more efficient by running them using low-bit integer arithmetic and is therefore commonly adopted in industry. Recent work has shown that floating-point neural networks that have been verified to be robust can become vulnerable to adversarial attacks after quantization, and certification of the quantized representation is necessary to guarantee robustness. In this work, we present quantization-aware interval bound propagation (QA-IBP), a novel method for training robust QNNs. Inspired by advances in robust learning of non-quantized networks, our training algorithm computes the gradient of an abstract representation of the actual network. Unlike existing approaches, our method can handle the discrete semantics of QNNs. Based on QA-IBP, we also develop a complete verification procedure for verifying the adversarial robustness of QNNs, which is guaranteed to terminate and produce a correct answer. Compared to existing approaches, the key advantage of our verification procedure is that it runs entirely on GPU or other accelerator devices. We demonstrate experimentally that our approach significantly outperforms existing methods and establish the new state-of-the-art for training and certifying the robustness of QNNs.

神经网络已广泛应用于垃圾邮件和网络钓鱼检测，入侵预防和恶意软件检测等安全应用程序。但是，这种黑盒方法通常在应用中具有不确定性和不良的解释性。此外，神经网络本身通常容易受到对抗攻击的影响。由于这些原因，人们对可信赖和严格的方法有很高的需求来验证神经网络模型的鲁棒性。对抗性的鲁棒性在处理恶意操纵输入时涉及神经网络的可靠性，是安全和机器学习中最热门的主题之一。在这项工作中，我们在神经网络的对抗性鲁棒性验证中调查了现有文献，并在机器学习，安全和软件工程领域收集了39项多元化研究工作。我们系统地分析了它们的方法，包括如何制定鲁棒性，使用哪种验证技术以及每种技术的优势和局限性。我们从正式验证的角度提供分类学，以全面理解该主题。我们根据财产规范，减少问题和推理策略对现有技术进行分类。我们还展示了使用样本模型在现有研究中应用的代表性技术。最后，我们讨论了未来研究的开放问题。

神经网络是基于学习的软件系统的重要组成部分。但是，它们的高计算，内存和功率要求使在低资源域中使用它们具有挑战性。因此，在部署前通常对神经网络进行量化。现有的量化技术倾向于降低网络准确性。我们提出了反示例引导的神经网络量化改进（CEG4N）。该技术结合了基于搜索的量化和等效性验证：前者最小化了计算要求，而后者保证网络的输出在量化后不会改变。我们根据包括大型和小型网络在内的各种基准测试对CEG4N〜进行评估。我们的技术成功地量化了我们评估中的网络，同时生产的模型比最先进的技术高达72％。

We present an approach for the verification of feed-forward neural networks in which all nodes have a piece-wise linear activation function. Such networks are often used in deep learning and have been shown to be hard to verify for modern satisfiability modulo theory (SMT) and integer linear programming (ILP) solvers.The starting point of our approach is the addition of a global linear approximation of the overall network behavior to the verification problem that helps with SMT-like reasoning over the network behavior. We present a specialized verification algorithm that employs this approximation in a search process in which it infers additional node phases for the non-linear nodes in the network from partial node phase assignments, similar to unit propagation in classical SAT solving. We also show how to infer additional conflict clauses and safe node fixtures from the results of the analysis steps performed during the search. The resulting approach is evaluated on collision avoidance and handwritten digit recognition case studies.

我们提出了Caisar，这是一个正在积极开发的开源平台，以表征AI Systems的鲁棒性和安全性。Caisar提供了一个统一的入口点，用于使用WhyML（Why3验证平台的成熟和表达语言）来定义验证问题。此外，Caisar协调并创建了最先进的机器学习验证工具，该工具单独无法有效地处理所有问题，但总体可以涵盖越来越多的属性。我们的目的是一方面，通过减少选择针对给定验证问题量身定制的方法的负担来协助V \＆v流程，另一方面描述一个平台。CAISAR很快将在https://git.frama-c.com/pub/caisar提供。

我们介绍了一种新颖的方法来对计算机程序进行自动终止分析：我们使用神经网络来表示排名功能。排名函数映射程序状态为从下面界限并随着程序运行而减小的值；排名函数的存在证明了程序终止。我们从程序的采样执行轨迹训练神经网络，以使网络的输出沿轨迹降低；然后，我们使用符号推理正式验证其对所有可能执行的概括。通过肯定的答案，我们获得了该计划的正式终止证书，我们称之为神经排名函数。我们证明，由于神经网络代表非线性功能的能力，我们的方法成功地超过了最先进工具的程序。这包括在其循环条件和包括非线性表达式的程序中使用析取的程序。

This report summarizes the 3rd International Verification of Neural Networks Competition (VNN-COMP 2022), held as a part of the 5th Workshop on Formal Methods for ML-Enabled Autonomous Systems (FoMLAS), which was collocated with the 34th International Conference on Computer-Aided Verification (CAV). VNN-COMP is held annually to facilitate the fair and objective comparison of state-of-the-art neural network verification tools, encourage the standardization of tool interfaces, and bring together the neural network verification community. To this end, standardized formats for networks (ONNX) and specification (VNN-LIB) were defined, tools were evaluated on equal-cost hardware (using an automatic evaluation pipeline based on AWS instances), and tool parameters were chosen by the participants before the final test sets were made public. In the 2022 iteration, 11 teams participated on a diverse set of 12 scored benchmarks. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this iteration of this competition.

Fairness of machine learning (ML) software has become a major concern in the recent past. Although recent research on testing and improving fairness have demonstrated impact on real-world software, providing fairness guarantee in practice is still lacking. Certification of ML models is challenging because of the complex decision-making process of the models. In this paper, we proposed Fairify, an SMT-based approach to verify individual fairness property in neural network (NN) models. Individual fairness ensures that any two similar individuals get similar treatment irrespective of their protected attributes e.g., race, sex, age. Verifying this fairness property is hard because of the global checking and non-linear computation nodes in NN. We proposed sound approach to make individual fairness verification tractable for the developers. The key idea is that many neurons in the NN always remain inactive when a smaller part of the input domain is considered. So, Fairify leverages whitebox access to the models in production and then apply formal analysis based pruning. Our approach adopts input partitioning and then prunes the NN for each partition to provide fairness certification or counterexample. We leveraged interval arithmetic and activation heuristic of the neurons to perform the pruning as necessary. We evaluated Fairify on 25 real-world neural networks collected from four different sources, and demonstrated the effectiveness, scalability and performance over baseline and closely related work. Fairify is also configurable based on the domain and size of the NN. Our novel formulation of the problem can answer targeted verification queries with relaxations and counterexamples, which have practical implications.

本文提出了一种新的可达性分析工具，用于计算给定输入不确定性下的前馈神经网络的输出集的间隔过度近似。所提出的方法适应神经网络的现有混合单调性方法，用于可动力分析的动态系统，并将其应用于给定神经网络内的所有可能的部分网络。这确保了所获得的结果的交叉点是可以使用混合单调性获得的每层输出的最紧密的间隔过度近似。与文献中的其他工具相比，专注于小类分段 - 仿射或单调激活功能，我们方法的主要优势是其普遍性，它可以处理具有任何嘴唇智能连续激活功能的神经网络。此外，所提出的框架的简单性允许用户通过简单地提供函数，衍生和全局极值以及衍生物的相应参数来非常容易地添加未实现的激活功能。我们的算法经过测试，并将其与1000个随机生成的神经网络上的五个基于间隔的工具进行了比较，用于四个激活功能（Relu，Tanh，Elu，Silu）。我们表明我们的工具总是优于间隔绑定的传播方法，并且我们获得比Reluval，神经化，Verinet和Crown（适用于案件的时）更严格的输出界限。

深度神经网络的鲁棒性对于现代AI支持系统至关重要，应正式验证。在广泛的应用中采用了类似乙状结肠的神经网络。由于它们的非线性，通常会过度评估乙状结肠样激活功能，以进行有效的验证，这不可避免地引入了不精确度。已大量的努力致力于找到所谓的更紧密的近似值，以获得更精确的验证结果。但是，现有的紧密定义是启发式的，缺乏理论基础。我们对现有神经元的紧密表征进行了彻底的经验分析，并揭示它们仅在特定的神经网络上是优越的。然后，我们将网络紧密度的概念介绍为统一的紧密度定义，并表明计算网络紧密度是一个复杂的非convex优化问题。我们通过两个有效的，最紧密的近似值从不同的角度绕过复杂性。结果表明，我们在艺术状态下的方法实现了有希望的表现：（i）达到高达251.28％的改善，以提高认证的较低鲁棒性界限； （ii）在卷积网络上表现出更为精确的验证结果。

随着深度学习在关键任务系统中的越来越多的应用，越来越需要对神经网络的行为进行正式保证。确实，最近提出了许多用于验证神经网络的方法，但是这些方法通常以有限的可伸缩性或不足的精度而挣扎。许多最先进的验证方案中的关键组成部分是在网络中可以为特定输入域获得的神经元获得的值计算下限和上限 - 并且这些界限更紧密，验证的可能性越大，验证的可能性就越大。成功。计算这些边界的许多常见算法是符号结合传播方法的变化。其中，利用一种称为后替代的过程的方法特别成功。在本文中，我们提出了一种使背部替代产生更严格的界限的方法。为了实现这一目标，我们制定并最大程度地减少背部固定过程中发生的不精确错误。我们的技术是一般的，从某种意义上说，它可以将其集成到许多现有的符号结合的传播技术中，并且只有较小的修改。我们将方法作为概念验证工具实施，并且与执行背部替代的最先进的验证者相比，取得了有利的结果。

深度神经网络（DNN）的巨大进步导致了各种任务的最先进的性能。然而，最近的研究表明，DNNS容易受到对抗的攻击，这在将这些模型部署到自动驾驶等安全关键型应用时，这使得非常关注。已经提出了不同的防御方法，包括：a）经验防御，通常可以在不提供稳健性认证的情况下再次再次攻击; b）可认真的稳健方法，由稳健性验证组成，提供了在某些条件下的任何攻击和相应的强大培训方法中的稳健准确性的下限。在本文中，我们系统化了可认真的稳健方法和相关的实用和理论意义和调查结果。我们还提供了在不同数据集上现有的稳健验证和培训方法的第一个全面基准。特别是，我们1）为稳健性验证和培训方法提供分类，以及总结代表性算法的方法，2）揭示这些方法中的特征，优势，局限性和基本联系，3）讨论当前的研究进展情况TNN和4的可信稳健方法的理论障碍，主要挑战和未来方向提供了一个开放的统一平台，以评估超过20种代表可认真的稳健方法，用于各种DNN。

最近，图形神经网络（GNN）已应用于群集上的调整工作，比手工制作的启发式方法更好地表现了。尽管表现令人印象深刻，但仍然担心这些基于GNN的工作调度程序是否满足用户对其他重要属性的期望，例如防止策略，共享激励和稳定性。在这项工作中，我们考虑对基于GNN的工作调度程序的正式验证。我们解决了几个特定领域的挑战，例如网络，这些挑战比验证图像和NLP分类器时遇到的更深层和规格更丰富。我们开发了拉斯维加斯，这是基于精心设计的算法，将这些调度程序的单步和多步属性验证的第一个通用框架，它们结合了抽象，改进，求解器和证明传输。我们的实验结果表明，与以前的方法相比，维加斯在验证基于GNN的调度程序的重要特性时会达到显着加速。

The rising popularity of intelligent mobile devices and the daunting computational cost of deep learning-based models call for efficient and accurate on-device inference schemes. We propose a quantization scheme that allows inference to be carried out using integer-only arithmetic, which can be implemented more efficiently than floating point inference on commonly available integer-only hardware. We also co-design a training procedure to preserve end-to-end model accuracy post quantization. As a result, the proposed quantization scheme improves the tradeoff between accuracy and on-device latency. The improvements are significant even on MobileNets, a model family known for run-time efficiency, and are demonstrated in ImageNet classification and COCO detection on popular CPUs.

神经网络越来越依赖于复杂安全系统（例如自动驾驶汽车）的组成部分。对在更大的验证周期中嵌入神经网络验证的工具和方法的需求很高。但是，由于关注的广泛验证属性，很难进行神经网络验证，通常每个验证属性仅适用于专用求解器中的验证。在本文中，我们展示了最初设计用于验证，验证和仿真金融基础架构的功能编程语言的Imandra如何为神经网络验证提供整体基础架构。我们开发了一个新颖的图书馆Checkinn，该图书馆在Imandra的神经网络上形式化，并涵盖了神经网络验证的不同重要方面。

Recurrent neural networks (RNN) are the backbone of many text and speech applications. These architectures are typically made up of several computationally complex components such as; non-linear activation functions, normalization, bi-directional dependence and attention. In order to maintain good accuracy, these components are frequently run using full-precision floating-point computation, making them slow, inefficient and difficult to deploy on edge devices. In addition, the complex nature of these operations makes them challenging to quantize using standard quantization methods without a significant performance drop. We present a quantization-aware training method for obtaining a highly accurate integer-only recurrent neural network (iRNN). Our approach supports layer normalization, attention, and an adaptive piecewise linear (PWL) approximation of activation functions, to serve a wide range of state-of-the-art RNNs. The proposed method enables RNN-based language models to run on edge devices with $2\times$ improvement in runtime, and $4\times$ reduction in model size while maintaining similar accuracy as its full-precision counterpart.

Deep neural networks have achieved impressive experimental results in image classification, but can surprisingly be unstable with respect to adversarial perturbations, that is, minimal changes to the input image that cause the network to misclassify it. With potential applications including perception modules and end-to-end controllers for self-driving cars, this raises concerns about their safety. We develop a novel automated verification framework for feed-forward multi-layer neural networks based on Satisfiability Modulo Theory (SMT). We focus on safety of image classification decisions with respect to image manipulations, such as scratches or changes to camera angle or lighting conditions that would result in the same class being assigned by a human, and define safety for an individual decision in terms of invariance of the classification within a small neighbourhood of the original image. We enable exhaustive search of the region by employing discretisation, and propagate the analysis layer by layer. Our method works directly with the network code and, in contrast to existing methods, can guarantee that adversarial examples, if they exist, are found for the given region and family of manipulations. If found, adversarial examples can be shown to human testers and/or used to fine-tune the network. We implement the techniques using Z3 and evaluate them on state-of-the-art networks, including regularised and deep learning networks. We also compare against existing techniques to search for adversarial examples and estimate network robustness.