深度学习培训是一个昂贵的过程，可广泛使用GPU，但并非所有模型训练都饱和现代强大的GPU。 Multi-Instance GPU（MIG）是NVIDIA引入的一项新技术，可以分区GPU，以更好地适合不需要所有内存和计算完整GPU的资源的工作负载。在本文中，我们研究了在深度学习工作负载下的三种尺寸工作负载下的MIG启用A100 GPU的性能，这些尺寸重点是使用Resnet模型进行图像识别培训。当在GPU允许的各种MIG实例上孤立运行时，我们还研究了这些工作负载的行为，此外还可以在同一GPU共同列入同类的同质实例上并行运行它们。我们的结果表明，当工作负载太小而无法孤立地利用整个GPU时，使用MIG可以显着改善GPU的利用率。通过并行训练多个小型型号，尽管每单位时间的时间增加了，但每单位时间的GPU可以执行更多的工作，导致$ \ sim $ \ sim $ 3倍吞吐量。相比之下，对于已经很好地利用了整个GPU的中型和大型工作量，MIG仅提供边际性能的改进。然而，我们观察到，使用单独的MIG分区并行的训练模型不会表现出强调具有MIG在现代GPU上具有功能的价值的干扰。

我们旨在通过引入全面的分布式深度学习（DDL）探索器来解决此问题，该研究人员可以确定DDL在公共云上运行时遭受的各种执行“失速”。我们已经通过扩展先前的工作来估算两种类型的通信失速 - 互连和网络摊位来实现剖面。我们使用Profiler培训流行的DNN模型来表征各种AWS GPU实例，并列出了用户做出明智决定的优势和缺点。我们观察到，较昂贵的GPU实例可能不是所有DNN型号的性能最多，并且AWS可能会在次优的硬件互连资源分配次优。具体而言，与单个实例的培训相比，机内互连可以引入高达90％的DNN培训时间和网络连接的实例的通信开销，而与网络连接的实例可能会遭受高达5倍的速度。此外，我们对DNN宏观特征的影响进行建模，例如层的数量和通信摊位上的梯度数量。最后，我们为用户提出了一个基于衡量的建议模型，以降低DDL的公共云货币成本。

负责将数据从存储转移到GPU的同时，在培训机器学习模型的同时，数据加载器可能会大大提高培训工作的绩效。最近的进步不仅通过大大减少训练时间，而且还提供了新功能，例如从远程存储（如S3）加载数据，这表明了希望。在本文中，我们是第一个将数据加载器区分为深度学习（DL）工作流程中的单独组件并概述其结构和功能的组件。最后，我们提供了可用的不同数据库，其功能，可用性和性能方面的权衡以及从中获得的见解的全面比较。

A growing number of Machine Learning Frameworks recently made Deep Learning accessible to a wider audience of engineers, scientists, and practitioners, by allowing straightforward use of complex neural network architectures and algorithms. However, since deep learning is rapidly evolving, not only through theoretical advancements but also with respect to hardware and software engineering, ML frameworks often lose backward compatibility and introduce technical debt that can lead to bottlenecks and sub-optimal resource utilization. Moreover, the focus is in most cases not on deep learning engineering, but rather on new models and theoretical advancements. In this work, however, we focus on engineering, more specifically on the data loading pipeline in the PyTorch Framework. We designed a series of benchmarks that outline performance issues of certain steps in the data loading process. Our findings show that for classification tasks that involve loading many files, like images, the training wall-time can be significantly improved. With our new, modified ConcurrentDataloader we can reach improvements in GPU utilization and significantly reduce batch loading time, up to 12X. This allows for the use of the cloud-based, S3-like object storage for datasets, and have comparable training time as if datasets are stored on local drives.

Video, as a key driver in the global explosion of digital information, can create tremendous benefits for human society. Governments and enterprises are deploying innumerable cameras for a variety of applications, e.g., law enforcement, emergency management, traffic control, and security surveillance, all facilitated by video analytics (VA). This trend is spurred by the rapid advancement of deep learning (DL), which enables more precise models for object classification, detection, and tracking. Meanwhile, with the proliferation of Internet-connected devices, massive amounts of data are generated daily, overwhelming the cloud. Edge computing, an emerging paradigm that moves workloads and services from the network core to the network edge, has been widely recognized as a promising solution. The resulting new intersection, edge video analytics (EVA), begins to attract widespread attention. Nevertheless, only a few loosely-related surveys exist on this topic. A dedicated venue for collecting and summarizing the latest advances of EVA is highly desired by the community. Besides, the basic concepts of EVA (e.g., definition, architectures, etc.) are ambiguous and neglected by these surveys due to the rapid development of this domain. A thorough clarification is needed to facilitate a consensus on these concepts. To fill in these gaps, we conduct a comprehensive survey of the recent efforts on EVA. In this paper, we first review the fundamentals of edge computing, followed by an overview of VA. The EVA system and its enabling techniques are discussed next. In addition, we introduce prevalent frameworks and datasets to aid future researchers in the development of EVA systems. Finally, we discuss existing challenges and foresee future research directions. We believe this survey will help readers comprehend the relationship between VA and edge computing, and spark new ideas on EVA.

培训深神经网络（DNNS）在企业和云数据中心都广受欢迎。现有的DNN培训调度程序将GPU视为主要资源，并分配其他资源，例如CPU和内存与作业要求的GPU数量成正比。不幸的是，这些调度程序不考虑作业对CPU，内存和存储资源分配的敏感性的影响。在这项工作中，我们提出了Synergy，这是一种对共享GPU群集的资源敏感调度程序。通过乐观的分析，协同作用侵犯了DNN对不同资源的敏感性；某些工作可能会从GPU育儿分配中受益更多，而某些工作可能不会受到GPU育儿分配的影响。 Synergy使用新的近乎最佳的在线算法在共享的多租户集群上安排的一组作业进行了多余的工作量感知作业。我们的实验表明，与传统的GPU育儿计划相比，工作量感知的CPU和内存分配可以提高平均JCT高达3.4倍。

With the rise of AI in recent years and the increase in complexity of the models, the growing demand in computational resources is starting to pose a significant challenge. The need for higher compute power is being met with increasingly more potent accelerators and the use of large compute clusters. However, the gain in prediction accuracy from large models trained on distributed and accelerated systems comes at the price of a substantial increase in energy demand, and researchers have started questioning the environmental friendliness of such AI methods at scale. Consequently, energy efficiency plays an important role for AI model developers and infrastructure operators alike. The energy consumption of AI workloads depends on the model implementation and the utilized hardware. Therefore, accurate measurements of the power draw of AI workflows on different types of compute nodes is key to algorithmic improvements and the design of future compute clusters and hardware. To this end, we present measurements of the energy consumption of two typical applications of deep learning models on different types of compute nodes. Our results indicate that 1. deriving energy consumption directly from runtime is not accurate, but the consumption of the compute node needs to be considered regarding its composition; 2. neglecting accelerator hardware on mixed nodes results in overproportional inefficiency regarding energy consumption; 3. energy consumption of model training and inference should be considered separately - while training on GPUs outperforms all other node types regarding both runtime and energy consumption, inference on CPU nodes can be comparably efficient. One advantage of our approach is that the information on energy consumption is available to all users of the supercomputer, enabling an easy transfer to other workloads alongside a raise in user-awareness of energy consumption.

现代深度学习应用程序需要越来越多地计算培训最先进的模型。为了解决这一需求，大型企业和机构使用专用的高性能计算集群，其建筑和维护既昂贵又远远超出大多数组织的预算。结果，一些研究方向成为几个大型工业甚至更少的学术作用者的独家领域。为了减轻这种差异，较小的团体可以汇集他们的计算资源并运行有利于所有参与者的协作实验。这种范式称为网格或志愿者计算，在众多科学领域看到了成功的应用。然而，由于高延迟，不对称带宽以及志愿者计算独特的几个挑战，使用这种用于机器学习的方法是困难的。在这项工作中，我们仔细分析了这些约束，并提出了一种专门用于协作培训的新型算法框架。我们展示了我们在现实条件下的SWAV和Albert预先预价的方法的有效性，并在成本的一小部分中实现了与传统设置相当的性能。最后，我们提供了一份成功的协作语言模型预先追溯的详细报告，有40名参与者。

Accurate reporting of energy and carbon usage is essential for understanding the potential climate impacts of machine learning research. We introduce a framework that makes this easier by providing a simple interface for tracking realtime energy consumption and carbon emissions, as well as generating standardized online appendices. Utilizing this framework, we create a leaderboard for energy efficient reinforcement learning algorithms to incentivize responsible research in this area as an example for other areas of machine learning. Finally, based on case studies using our framework, we propose strategies for mitigation of carbon emissions and reduction of energy consumption. By making accounting easier, we hope to further the sustainable development of machine learning experiments and spur more research into energy efficient algorithms.

New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This technology provides more flexibility for users to support both deep learning training and inference workloads, but efficiently utilizing it can still be challenging. The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts. To achieve this vision, the paper presents MIGPerf, an open-source tool that streamlines the benchmark study for MIG. Using MIGPerf, the authors conduct a series of experiments, including deep learning training and inference characterization on MIG, GPU sharing characterization, and framework compatibility with MIG. The results of these experiments provide new insights and guidance for users to effectively employ MIG, and lay the foundation for further research on the orchestration of hybrid training and inference workloads on MIGs. The code and results are released on https://github.com/MLSysOps/MIGProfiler. This work is still in progress and more results will be published soon.

Persistent Memory (PMEM), also known as Non-Volatile Memory (NVM), can deliver higher density and lower cost per bit when compared with DRAM. Its main drawback is that it is typically slower than DRAM. On the other hand, DRAM has scalability problems due to its cost and energy consumption. Soon, PMEM will likely coexist with DRAM in computer systems but the biggest challenge is to know which data to allocate on each type of memory. This paper describes a methodology for identifying and characterizing application objects that have the most influence on the application's performance using Intel Optane DC Persistent Memory. In the first part of our work, we built a tool that automates the profiling and analysis of application objects. In the second part, we build a machine learning model to predict the most critical object within large-scale graph-based applications. Our results show that using isolated features does not bring the same benefit compared to using a carefully chosen set of features. By performing data placement using our predictive model, we can reduce the execution time degradation by 12\% (average) and 30\% (max) when compared to the baseline's approach based on LLC misses indicator.

由于新兴的深度神经网络（DNN）模型的规模继续增大，使用大型GPU集群培训DNN是实现可接受培训时间的基本要求。在本文中，我们考虑了集群大小的未来增加的情况将导致全局批量大小用于培训模型以达到基本限制：超出某个点，更大的全球批量尺寸会导致样品效率降低，总体上升准确性的时间。因此，为了实现培训性能的进一步改进，我们必须考虑“强大的缩放”策略，该策略保持全局批量大小常量，并将较小的批次分配给每个GPU。不幸的是，这使得能够有效地使用群集资源。我们呈现DeepPool，通过两个关键思想解决这种效率挑战的系统。首先，突发并行性将大量GPU分配给突发中的前景作业，以利用整个层的并行性的不均匀性。其次，GPU多路复用优先考虑前台培训工作的吞吐量，而背景培训作业包装以回收未充分利用的GPU资源，从而提高集群范围利用率。这两个想法在一起使DeepPool能够在群集刻度大的单一任务中通过标准数据并行度进行2.2 - 2.4倍的完整性。

深度学习领域目睹了对极端计算和内存密集型神经网络的显着转变。这些较新的较大模型使研究人员能够推进各种领域的最先进的工具。这种现象刺激了在更多的硬件加速器上产生了针对神经网络的分布式训练的算法。在本文中，我们讨论并比较了当前的最先进的框架，以实现大规模的分布式深度学习。首先，我们调查分布式学习中的当前实践，并确定所使用的不同类型的并行性。然后，我们提出了对大型图像和语言培训任务的性能进行了经验结果。此外，我们解决了他们的统计效率和内存消耗行为。根据我们的结果，我们讨论了阻碍性能的每个框架的算法和实现部分。

随着越来越多的机器和深度学习应用在高能量物理中，方便地访问专用基础设施代表了快速高效的研发需求。这项工作探讨了不同类型的云服务，以使用TensorFlow数据并行策略在并行环境中训练生成的对冲网络（GaN）。更具体地，我们并将培训过程并行化多个GPU和Google Tensor处理单元（TPU），我们比较两个算法：TensorFlow内置逻辑和自定义循环，优化，以便更高控制分配给每个GPU工作者的元素或TPU核心。将所生成的数据的质量与Monte Carlo仿真进行比较。获得训练过程的线性加速，同时在物理结果方面保留大部分性能。此外，我们根据多个GPU节点，以规模，在多个GPU节点上进行基准测试，在不同的公共云提供商上部署培训过程，寻求整体效率和成本效益。数据科学，云部署选项和相关经济学的组合允许异构地突发，探索基于云的服务的全部潜力。

Deep learning based recommendation models (DLRM) are widely used in several business critical applications. Training such recommendation models efficiently is challenging primarily because they consist of billions of embedding-based parameters which are often stored remotely leading to significant overheads from embedding access. By profiling existing DLRM training, we observe that only 8.5% of the iteration time is spent in forward/backward pass while the remaining time is spent on embedding and model synchronization. Our key insight in this paper is that access to embeddings have a specific structure and pattern which can be used to accelerate training. We observe that embedding accesses are heavily skewed, with almost 1% of embeddings represent more than 92% of total accesses. Further, we observe that during training we can lookahead at future batches to determine exactly which embeddings will be needed at what iteration in the future. Based on these insight, we propose Bagpipe, a system for training deep recommendation models that uses caching and prefetching to overlap remote embedding accesses with the computation. We designed an Oracle Cacher, a new system component which uses our lookahead algorithm to generate optimal cache update decisions and provide strong consistency guarantees. Our experiments using three datasets and two models shows that our approach provides a speed up of up to 6.2x compared to state of the art baselines, while providing the same convergence and reproducibility guarantees as synchronous training.

变形金刚是一种深入学习语言模型，用于数据中心中的自然语言处理（NLP）服务。在变压器模型中，生成的预训练的变压器（GPT）在文本生成或自然语言生成（NLG）中取得了显着的性能，它需要在摘要阶段处理大型输入上下文，然后是产生一个生成阶段的一次单词。常规平台（例如GPU）专门用于在摘要阶段平行处理大型输入，但是由于其顺序特征，它们的性能在生成阶段显着降低。因此，需要一个有效的硬件平台来解决由文本生成的顺序特征引起的高潜伏期。在本文中，我们提出了DFX，这是一种多FPGA加速器，该设备在摘要和发电阶段中执行GPT-2模型端到端，并具有低延迟和高吞吐量。 DFX使用模型并行性和优化的数据流，这是模型和硬件感知的设备之间快速同时执行执行。其计算核心根据自定义说明运行，并提供GPT-2操作端到端。我们在四个Xilinx Alveo U280 FPGAS上实现了建议的硬件体系结构，并利用了高带宽内存（HBM）的所有频道，以及用于高硬件效率的最大计算资源数量。 DFX在现代GPT-2模型上实现了四个NVIDIA V100 GPU的5.58倍加速度和3.99倍的能效。 DFX的成本效益比GPU设备更具成本效益，这表明它是云数据中心中文本生成工作负载的有前途解决方案。

操作系统包括许多启发式算法，旨在提高整体存储性能和吞吐量。由于此类启发式is不能适用于所有条件和工作负载，因此系统设计人员诉诸用户对用户的众多可调参数揭示 - 基本上负担用户不断优化自己的存储系统和应用程序。存储系统通常负责I / O重型应用中的大多数延迟，因此即使是小的总延迟改善也可能很重要。机器学习（ml）技术承诺学习模式，从它们概括，并实现适应更改工作负载的最佳解决方案。我们提出ML解决方案成为OSS中的一流组件，并更换了动态优化存储系统的手动启发式。在本文中，我们描述了我们所提出的ML架构，称为KML。我们开发了一个原型KML体系结构，并将其应用于两个问题：最佳readAhead和NFS读取大小值。我们的实验表明，KML消耗了很少的操作系统资源，延迟可忽略不计，但可以学习可以分别为两种用例的2.3倍或15倍提高I / O吞吐量的模式 - 即使是复杂的，也不是为了复杂 - 在不同的存储设备上同时运行混合工作负载。

已经提出了高效和自适应计算机视觉系统以使计算机视觉任务，例如图像分类和对象检测，针对嵌入或移动设备进行了优化。这些解决方案最近的起源，专注于通过设计具有近似旋钮的自适应系统来优化模型（深神经网络，DNN）或系统。尽管最近的几项努力，但我们表明现有解决方案遭受了两个主要缺点。首先，系统不考虑模型的能量消耗，同时在制定要运行的模型的决定时。其次，由于其他共同居民工作负载，评估不考虑设备上的争用的实际情况。在这项工作中，我们提出了一种高效和自适应的视频对象检测系统，这是联合优化的精度，能量效率和延迟。底层Virtuoso是一个多分支执行内核，它能够在精度 - 能量 - 延迟轴上的不同运行点处运行，以及轻量级运行时调度程序，以选择最佳的执行分支以满足用户要求。要与Virtuoso相当比较，我们基准于15件最先进的或广泛使用的协议，包括更快的R-CNN（FRCNN），YOLO V3，SSD，培训台，SELSA，MEGA，REPP，FastAdapt和我们的内部FRCNN +，YOLO +，SSD +和高效+（我们的变体具有增强的手机效率）的自适应变体。通过这种全面的基准，Virtuoso对所有上述协议显示出优势，在NVIDIA Jetson Mobile GPU上的每一项效率水平上引领精度边界。具体而言，Virtuoso的准确性为63.9％，比一些流行的物体检测模型高于10％，51.1％，yolo为49.5％。

TensorFlow is a machine learning system that operates at large scale and in heterogeneous environments. Tensor-Flow uses dataflow graphs to represent computation, shared state, and the operations that mutate that state. It maps the nodes of a dataflow graph across many machines in a cluster, and within a machine across multiple computational devices, including multicore CPUs, generalpurpose GPUs, and custom-designed ASICs known as Tensor Processing Units (TPUs). This architecture gives flexibility to the application developer: whereas in previous "parameter server" designs the management of shared state is built into the system, TensorFlow enables developers to experiment with novel optimizations and training algorithms. TensorFlow supports a variety of applications, with a focus on training and inference on deep neural networks. Several Google services use TensorFlow in production, we have released it as an open-source project, and it has become widely used for machine learning research. In this paper, we describe the TensorFlow dataflow model and demonstrate the compelling performance that Tensor-Flow achieves for several real-world applications.

本文探讨了超线性增长趋势的环境影响，从整体角度来看，跨越数据，算法和系统硬件。我们通过在行业规模机器学习用例中检查模型开发周期来表征AI计算的碳足迹，同时考虑系统硬件的生命周期。进一步迈出一步，我们捕获AI计算的操作和制造碳足迹，并为硬件 - 软件设计和尺度优化的结束分析以及如何帮助降低AI的整体碳足迹。根据行业经验和经验教训，我们分享关键挑战，并在AI的许多方面上绘制了重要的发展方向。我们希望本文提出的关键信息和见解能够激发社区以环保的方式推进AI领域。