随着社会越来越多的机器学习（ML）和深度学习（DL）进行各种智能解决方案，它将在普通开发实践和资源下规范ML / DL型号的ML / DL型号的常见措施变得越来越迫切。人们可以在共同的地面上基准和比较模型质量和性能。 MLCommons最近出现了来自工业和学术界的推动力，以协调这种努力。尽管其广泛采用作为标准化基准，但MLCommons推断只包括有限数量的ML / DL模型（实际上总共七种型号）。这显着限制了MLCommons推论的基准测试结果的一般性，因为研究界有更多的新颖ML / DL模型，解决了不同输入和输出方式的广泛问题。为了解决这种限制，我们提出了MLHARNESS，用于MLCommons推断的可扩展基准线束系统，具有三种独特的特点：（1）它根据MLCommons推理定义的标准基准过程，包括模型，数据集，DL框架和软件和硬件系统; （2）它为模型开发人员提供了一种简单且声明的方法，使其模型和数据集贡献到MLCommons推断; （3）它包括具有不同输入/输出模态的各种模型的支持，以便我们可以在不同的数据集，框架和硬件系统上缩放这些模型。该线束系统是在MLModelscope系统的顶部开发的，并将开放到社区。我们的实验结果表明，对于MLCommons推理基准测试，这种线束系统的卓越灵活性和可扩展性。

基于深度学习的（DL）申请越来越受欢迎，并以前所未有的步伐推动。虽然正在进行许多研究工作以增强深度神经网络（DNN） - DL应用的核心 - 云和边缘系统中这些应用的实际部署挑战，它们对应用程序的可用性的影响并未充分调查。特别是，部署不同虚拟化平台的影响由云和边缘提供的DL应用程序的可用性（在端到端（E2E）推理时间）仍然是一个打开的问题。重要的是，资源弹性（通过放大），CPU固定和处理器类型（CPU VS GPU）配置已经显示在虚拟化开销上有影响力。因此，本研究的目标是研究这些潜在决定的部署选项对E2E性能的影响，从而实现了DL应用程序的可用性。为此，我们在改变处理器配置时，我们测量四种流行的执行平台（即，裸机，虚拟机（VM），容器和容器中的裸机，虚拟机（VM），容器和容器）的影响（扩展，CPU固定）和处理器类型。本研究揭示了一系列有趣的，有时是反向直观的发现，可以用作云解决方案架构师的最佳实践，以便在各种系统中有效地部署DL应用程序。值得注意的发现是，解决方案架构师必须了解DL应用特征，特别是它们的预处理和后处理要求，能够最佳选择和配置执行平台，确定使用GPU，并决定有效扩展范围。

随着机器学习系统的计算要求以及机器学习框架的规模和复杂性的增加，基本框架创新变得具有挑战性。尽管计算需求驱动了最近的编译器，网络和硬件的进步，但通过机器学习工具对这些进步的利用却以较慢的速度发生。这部分是由于与现有框架原型制作新的计算范式有关的困难。大型框架将机器学习研究人员和从业人员作为最终用户的优先级优先，并且很少关注能够向前推动框架的系统研究人员 - 我们认为两者都是同等重要的利益相关者。我们介绍了手电筒，这是一个开源库，旨在通过优先考虑开放式，模块化，可定制的内部设备以及最新的，可用于研究的模型和培训设置，以刺激机器学习工具和系统的创新。手电筒使系统研究人员能够快速原型并尝试机器学习计算中的新思想，并且开销低，与其他流行的机器学习框架竞争并经常超过其他流行的机器学习框架。我们将手电筒视为一种工具，可以使可以使广泛使用的图书馆受益，并使机器学习和系统研究人员更加紧密地结合在一起。手电筒可从https://github.com/flashlight/flashlight获得。

受益于扩大云基础设施，今天深度神经网络（DNN）在云中培训时具有越来越高的性能。研究人员花了几个月的努力，竞争额外的模型精度百分比。但是，当这些模型实际上在实践中部署在边缘设备上时，通常情况可能会突然下降超过10％而无明显原因。关键挑战是，在边缘设备上对ML推理执行并不多的可见性，并且在边缘部署过程中对潜在问题的认识很少。我们呈现ml-exray，一个端到端的框架，它提供了ML执行的层级细节的可见性，并帮助开发人员分析和调试云到边缘部署问题。更常见的是，子最佳边缘性能的原因不仅可以在模型本身中介绍，而是在整个数据流和部署过程中的每一个操作。评估显示ML-EXRARE可以有效地捕获部署问题，例如使用ML-EXRARE的预处理错误，量化问题，次优内核等，用户需要写入不到15行代码以完全检查边缘部署管道。消除这些问题，ML-EXRARE可以通过最多30％的模型性能，Pinpoint忽略层，指导用户通过两个数量级来优化内核执行延迟。代码和API将被释放为开源多语言仪表库和Python部署验证库。

Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.

There is an increasing need to bring machine learning to a wide diversity of hardware devices. Current frameworks rely on vendor-specific operator libraries and optimize for a narrow range of server-class GPUs. Deploying workloads to new platforms -such as mobile phones, embedded devices, and accelerators (e.g., FPGAs, ASICs) -requires significant manual effort. We propose TVM, a compiler that exposes graph-level and operator-level optimizations to provide performance portability to deep learning workloads across diverse hardware back-ends. TVM solves optimization challenges specific to deep learning, such as high-level operator fusion, mapping to arbitrary hardware primitives, and memory latency hiding. It also automates optimization of low-level programs to hardware characteristics by employing a novel, learning-based cost modeling method for rapid exploration of code optimizations. Experimental results show that TVM delivers performance across hardware back-ends that are competitive with state-ofthe-art, hand-tuned libraries for low-power CPU, mobile GPU, and server-class GPUs. We also demonstrate TVM's ability to target new accelerator back-ends, such as the FPGA-based generic deep learning accelerator.The system is open sourced and in production use inside several major companies.

负责将数据从存储转移到GPU的同时，在培训机器学习模型的同时，数据加载器可能会大大提高培训工作的绩效。最近的进步不仅通过大大减少训练时间，而且还提供了新功能，例如从远程存储（如S3）加载数据，这表明了希望。在本文中，我们是第一个将数据加载器区分为深度学习（DL）工作流程中的单独组件并概述其结构和功能的组件。最后，我们提供了可用的不同数据库，其功能，可用性和性能方面的权衡以及从中获得的见解的全面比较。

New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This technology provides more flexibility for users to support both deep learning training and inference workloads, but efficiently utilizing it can still be challenging. The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts. To achieve this vision, the paper presents MIGPerf, an open-source tool that streamlines the benchmark study for MIG. Using MIGPerf, the authors conduct a series of experiments, including deep learning training and inference characterization on MIG, GPU sharing characterization, and framework compatibility with MIG. The results of these experiments provide new insights and guidance for users to effectively employ MIG, and lay the foundation for further research on the orchestration of hybrid training and inference workloads on MIGs. The code and results are released on https://github.com/MLSysOps/MIGProfiler. This work is still in progress and more results will be published soon.

Video, as a key driver in the global explosion of digital information, can create tremendous benefits for human society. Governments and enterprises are deploying innumerable cameras for a variety of applications, e.g., law enforcement, emergency management, traffic control, and security surveillance, all facilitated by video analytics (VA). This trend is spurred by the rapid advancement of deep learning (DL), which enables more precise models for object classification, detection, and tracking. Meanwhile, with the proliferation of Internet-connected devices, massive amounts of data are generated daily, overwhelming the cloud. Edge computing, an emerging paradigm that moves workloads and services from the network core to the network edge, has been widely recognized as a promising solution. The resulting new intersection, edge video analytics (EVA), begins to attract widespread attention. Nevertheless, only a few loosely-related surveys exist on this topic. A dedicated venue for collecting and summarizing the latest advances of EVA is highly desired by the community. Besides, the basic concepts of EVA (e.g., definition, architectures, etc.) are ambiguous and neglected by these surveys due to the rapid development of this domain. A thorough clarification is needed to facilitate a consensus on these concepts. To fill in these gaps, we conduct a comprehensive survey of the recent efforts on EVA. In this paper, we first review the fundamentals of edge computing, followed by an overview of VA. The EVA system and its enabling techniques are discussed next. In addition, we introduce prevalent frameworks and datasets to aid future researchers in the development of EVA systems. Finally, we discuss existing challenges and foresee future research directions. We believe this survey will help readers comprehend the relationship between VA and edge computing, and spark new ideas on EVA.

本文介绍了有关如何架构，设计和优化深神经网络（DNN）的最新概述，以提高性能并保留准确性。该论文涵盖了一组跨越整个机器学习处理管道的优化。我们介绍两种类型的优化。第一个改变了DNN模型，需要重新训练，而第二个则不训练。我们专注于GPU优化，但我们认为提供的技术可以与其他AI推理平台一起使用。为了展示DNN模型优化，我们在流行的Edge AI推理平台（Nvidia Jetson Agx Xavier）上改善了光流的最先进的深层网络体系结构之一，RAFT ARXIV：2003.12039。

TensorFlow is a machine learning system that operates at large scale and in heterogeneous environments. Tensor-Flow uses dataflow graphs to represent computation, shared state, and the operations that mutate that state. It maps the nodes of a dataflow graph across many machines in a cluster, and within a machine across multiple computational devices, including multicore CPUs, generalpurpose GPUs, and custom-designed ASICs known as Tensor Processing Units (TPUs). This architecture gives flexibility to the application developer: whereas in previous "parameter server" designs the management of shared state is built into the system, TensorFlow enables developers to experiment with novel optimizations and training algorithms. TensorFlow supports a variety of applications, with a focus on training and inference on deep neural networks. Several Google services use TensorFlow in production, we have released it as an open-source project, and it has become widely used for machine learning research. In this paper, we describe the TensorFlow dataflow model and demonstrate the compelling performance that Tensor-Flow achieves for several real-world applications.

机器学习传感器代表了嵌入式机器学习应用程序未来的范式转移。当前的嵌入式机器学习（ML）实例化遭受了复杂的整合，缺乏模块化以及数据流动的隐私和安全问题。本文提出了一个以数据为中心的范式，用于将传感器智能嵌入边缘设备上，以应对这些挑战。我们对“传感器2.0”的愿景需要将传感器输入数据和ML处理从硬件级别隔离到更广泛的系统，并提供一个薄的界面，以模拟传统传感器的功能。这种分离导致模块化且易于使用的ML传感器设备。我们讨论了将ML处理构建到嵌入式系统上控制微处理器的软件堆栈中的标准方法所带来的挑战，以及ML传感器的模块化如何减轻这些问题。 ML传感器提高了隐私和准确性，同时使系统构建者更容易将ML集成到其产品中，以简单的组件。我们提供了预期的ML传感器和说明性数据表的例子，以表现出来，并希望这将建立对话使我们朝着传感器2.0迈进。

联合学习（FL）作为边缘设备的有希望的技术，以协作学习共享预测模型，同时保持其训练数据，从而解耦了从需要存储云中的数据的机器学习的能力。然而，在规模和系统异质性方面，FL难以现实地实现。虽然有许多用于模拟FL算法的研究框架，但它们不支持在异构边缘设备上进行可扩展的流程。在本文中，我们呈现花 - 一种全面的FL框架，通过提供新的设施来执行大规模的FL实验并考虑丰富的异构流程来区分现有平台。我们的实验表明花卉可以仅使用一对高端GPU在客户尺寸下进行FL实验。然后，研究人员可以将实验无缝地迁移到真实设备中以检查设计空间的其他部分。我们认为花卉为社区提供了一个批判性的新工具，用于研究和发展。

人工智能（AI）对计算的巨大需求正在推动对AI的硬件和软件系统的无与伦比的投资。这导致了专用硬件设备数量的爆炸，现在由主要的云供应商提供。通过通过基于张量的界面隐藏低级复杂性，张量计算运行时间（TCR）（例如Pytorch）允许数据科学家有效利用新硬件提供的令人兴奋的功能。在本文中，我们探讨了数据库管理系统如何在AI空间中乘坐创新浪潮。我们设计，构建和评估张量查询处理器（TQP）：TQP将SQL查询转换为张量程序，并在TCR上执行它们。 TQP能够通过在张量例程中实现与关系运算符的新颖算法来运行完整的TPC-H基准。同时，TQP可以支持各种硬件，而仅需要通常的开发工作。实验表明，与专用CPU和仅GPU的系统相比，TQP可以将查询执行时间提高到10美元$ \ times $。最后，TQP可以加速查询ML预测和SQL端到端，并在CPU基线上输送高达9 $ \ times $速度。

Artificial Intelligence (AI) is having a tremendous impact across most areas of science. Applications of AI in healthcare have the potential to improve our ability to detect, diagnose, prognose, and intervene on human disease. For AI models to be used clinically, they need to be made safe, reproducible and robust, and the underlying software framework must be aware of the particularities (e.g. geometry, physiology, physics) of medical data being processed. This work introduces MONAI, a freely available, community-supported, and consortium-led PyTorch-based framework for deep learning in healthcare. MONAI extends PyTorch to support medical data, with a particular focus on imaging, and provide purpose-specific AI model architectures, transformations and utilities that streamline the development and deployment of medical AI models. MONAI follows best practices for software-development, providing an easy-to-use, robust, well-documented, and well-tested software framework. MONAI preserves the simple, additive, and compositional approach of its underlying PyTorch libraries. MONAI is being used by and receiving contributions from research, clinical and industrial teams from around the world, who are pursuing applications spanning nearly every aspect of healthcare.

Recent advances in deep learning (dl) have led to the release of several dl software libraries such as pytorch, Caffe, and TensorFlow, in order to assist machine learning (ml) practitioners in developing and deploying state-of-the-art deep neural networks (DNN), but they are not able to properly cope with limitations in the dl libraries such as testing or data processing. In this paper, we present a qualitative and quantitative analysis of the most frequent dl libraries combination, the distribution of dl library dependencies across the ml workflow, and formulate a set of recommendations to (i) hardware builders for more optimized accelerators and (ii) library builder for more refined future releases. Our study is based on 1,484 open-source dl projects with 46,110 contributors selected based on their reputation. First, we found an increasing trend in the usage of deep learning libraries. Second, we highlight several usage patterns of deep learning libraries. In addition, we identify dependencies between dl libraries and the most frequent combination where we discover that pytorch and Scikit-learn and, Keras and TensorFlow are the most frequent combination in 18% and 14% of the projects. The developer uses two or three dl libraries in the same projects and tends to use different multiple dl libraries in both the same function and the same files. The developer shows patterns in using various deep-learning libraries and prefers simple functions with fewer arguments and straightforward goals. Finally, we present the implications of our findings for researchers, library maintainers, and hardware vendors.

近年来，在移动设备上部署深度学习（DL）一直是一个显着的趋势。为了支持对开发DL的快速推断，DL库作为算法和硬件扮演着至关重要的角色。不幸的是，先前的工作从未深入现代DL Libs的生态系统，并为其性能提供定量结果。在本文中，我们首先建立了一个全面的基准，其中包括6个代表性DL LIB和15种多元化的DL模型。然后，我们在10个移动设备上进行了广泛的实验，这有助于揭示当前移动DL LIBS生态系统的完整景观。例如，我们发现表现最佳的DL LIB在不同的模型和硬件中严重碎片，这些DL Libs之间的差距可能相当巨大。实际上，DL LIB的影响会淹没算法或硬件的优化，例如模型量化和基于GPU/DSP的异质计算。最后，在观察结果上，我们总结了对DL Lib生态系统中不同角色的实际意义。

深度学习（DL）模型越来越多地为应用程序提供多种应用。不幸的是，这种普遍性也使它们成为提取攻击的有吸引力的目标，这些目标可以窃取目标DL模型的体系结构，参数和超参数。现有的提取攻击研究观察到不同DL模型和数据集的攻击成功水平不同，但其易感性背后的根本原因通常仍不清楚。确定此类根本原因弱点将有助于促进安全的DL系统，尽管这需要在各种情况下研究提取攻击，以确定跨攻击成功和DL特征的共同点。理解，实施和评估甚至单一攻击所需的绝大部分技术努力和时间都使探索现有的大量独特提取攻击方案是不可行的，当前框架通常设计用于仅针对特定攻击类型，数据集和数据集，以及硬件平台。在本文中，我们介绍捏：一个有效且自动化的提取攻击框架，能够在异质硬件平台上部署和评估多个DL模型和攻击。我们通过经验评估大量先前未开发的提取攻击情景以及次级攻击阶段来证明捏合的有效性。我们的主要发现表明，1）多个特征影响开采攻击成功跨越DL模型体系结构，数据集复杂性，硬件，攻击类型和2）部分成功的提取攻击显着增强了进一步的对抗攻击分期的成功。

如今，由于最近在人工智能（AI）和机器学习（ML）中的近期突破，因此，智能系统和服务越来越受欢迎。然而，机器学习不仅满足软件工程，不仅具有有希望的潜力，而且还具有一些固有的挑战。尽管最近的一些研究努力，但我们仍然没有明确了解开发基于ML的申请和当前行业实践的挑战。此外，目前尚不清楚软件工程研究人员应将其努力集中起来，以更好地支持ML应用程序开发人员。在本文中，我们报告了一个旨在了解ML应用程序开发的挑战和最佳实践的调查。我们合成从80名从业者（以不同的技能，经验和应用领域）获得的结果为17个调查结果;概述ML应用程序开发的挑战和最佳实践。参与基于ML的软件系统发展的从业者可以利用总结最佳实践来提高其系统的质量。我们希望报告的挑战将通知研究界有关需要调查的主题，以改善工程过程和基于ML的申请的质量。

深度学习（DL）框架现在被广泛使用，简化了复杂模型的创建以及它们对各种应用的集成甚至到非DL专家。但是，就像任何其他程序一样，他们容易发生错误。本文与命名静默错误的错误分类：它们会导致错误的行为，但它们不会导致系统崩溃或挂起，也不会向用户显示错误消息。这种错误在DL应用程序和框架中更危险，因为系统的“黑匣子”和系统的随机性质（最终用户无法理解模型如何做出决定）。本文介绍了Keras和Tensorflow Silent错误的第一个实证研究，以及它们对用户节目的影响。从Tensorflow Github存储库中提取与KERA相关的封闭问题。在我们收集的1,168个问题中，77个影响了影响用户程序的可重复静音错误。我们根据“用户程序的影响”和“发生问题”的“发生问题的组件”归类错误。然后，我们根据用户程序的影响，我们为每个问题派生威胁级别。为了评估所确定的类别和影响规模的相关性，我们使用103个DL开发人员进行了在线调查。参与者普遍同意DL库中静音错误的重大影响，并承认了我们的研究结果（即，沉默错误的类别和拟议的影响量表）。最后，利用我们的分析，我们提供了一套指导方针，以促进对DL框架中的这些错误的保护。