Differentiable Architecture Search (DARTS) has attracted considerable attention as a gradient-based Neural Architecture Search (NAS) method. Since the introduction of DARTS, there has been little work done on adapting the action space based on state-of-art architecture design principles for CNNs. In this work, we aim to address this gap by incrementally augmenting the DARTS search space with micro-design changes inspired by ConvNeXt and studying the trade-off between accuracy, evaluation layer count, and computational cost. To this end, we introduce the Pseudo-Inverted Bottleneck conv block intending to reduce the computational footprint of the inverted bottleneck block proposed in ConvNeXt. Our proposed architecture is much less sensitive to evaluation layer count and outperforms a DARTS network with similar size significantly, at layer counts as small as 2. Furthermore, with less layers, not only does it achieve higher accuracy with lower GMACs and parameter count, GradCAM comparisons show that our network is able to better detect distinctive features of target objects compared to DARTS.

In this paper, we present a modified Xception architecture, the NEXcepTion network. Our network has significantly better performance than the original Xception, achieving top-1 accuracy of 81.5% on the ImageNet validation dataset (an improvement of 2.5%) as well as a 28% higher throughput. Another variant of our model, NEXcepTion-TP, reaches 81.8% top-1 accuracy, similar to ConvNeXt (82.1%), while having a 27% higher throughput. Our model is the result of applying improved training procedures and new design decisions combined with an application of Neural Architecture Search (NAS) on a smaller dataset. These findings call for revisiting older architectures and reassessing their potential when combined with the latest enhancements.

可区分的架构搜索（飞镖）大大促进了NAS技术的发展，因为其搜索效率很高，但遭受了性能崩溃的影响。在本文中，我们努力从两个方面减轻飞镖的性能崩溃问题。首先，我们研究了飞镖中超级网的表达能力，然后仅使用训练batchnorm来得出新的飞镖范式设置。其次，从理论上讲，随机特征稀释了跳过连接在超网优化中的辅助连接作用，并使搜索算法专注于更公平的操作选择，从而解决了性能崩溃问题。我们具有随机功能的实例化飞镖和PC-Darts，分别为每个命名的RF-Darts和RF-PCDART构建一个改进的版本。实验结果表明，RF-darts在CIFAR-10上获得\ TextBf {94.36 \％}测试精度（这是NAS Bench-201的最接近最佳结果），并实现了最新的最新最先进的TOP-1从CIFAR-10传输时，ImageNet上\ TextBf {24.0 \％}的测试错误。此外，RF-DARTS在三个数据集（CIFAR-10，CIFAR-100和SVHN）和四个搜索空间（S1-S4）上进行稳健性能。此外，RF-PCDARTS在Imagenet上取得了更好的结果，即\ textbf {23.9 \％} top-1和\ textbf {7.1 \％} top-5 top-5测试错误，超越了代表性的方法，例如单路径，训练免费， ，直接在Imagenet上搜索部分通道范例。

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.

视觉识别的“咆哮20S”开始引入视觉变压器（VITS），这将被取代的Cummnets作为最先进的图像分类模型。另一方面，vanilla vit，当应用于一般计算机视觉任务等对象检测和语义分割时面临困难。它是重新引入多个ConvNet Priors的等级变压器（例如，Swin变压器），使变压器实际上可作为通用视觉骨干网，并在各种视觉任务上展示了显着性能。然而，这种混合方法的有效性仍然在很大程度上归功于变压器的内在优越性，而不是卷积的固有感应偏差。在这项工作中，我们重新审视设计空间并测试纯粹的Convnet可以实现的限制。我们逐渐“现代化”标准Reset朝着视觉变压器的设计设计，并发现几个有助于沿途绩效差异的关键组件。此探索的结果是一个纯粹的ConvNet型号被称为ConvNext。完全由标准的Convnet模块构建，ConvNexts在准确性和可扩展性方面与变压器竞争，实现了87.8％的ImageNet Top-1精度和表现优于COCO检测和ADE20K分割的Swin变压器，同时保持了标准Convnet的简单性和效率。

We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. ENAS constructs a large computational graph, where each subgraph represents a neural network architecture, hence forcing all architectures to share their parameters. A controller is trained with policy gradient to search for a subgraph that maximizes the expected reward on a validation set. Meanwhile a model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss. Sharing parameters among child models allows ENAS to deliver strong empirical performances, whilst using much fewer GPU-hours than existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. On Penn Treebank, ENAS discovers a novel architecture that achieves a test perplexity of 56.3, on par with the existing state-of-the-art among all methods without post-training processing. On CIFAR-10, ENAS finds a novel architecture that achieves 2.89% test error, which is on par with the 2.65% test error of NASNet (Zoph et al., 2018).

是否可以在深网络中重组非线性激活函数以创建硬件有效的模型？为了解决这个问题，我们提出了一个称为重组激活网络（RANS）的新范式，该范式操纵模型中的非线性数量以提高其硬件意识和效率。首先，我们提出了RAN-STHICER（RAN-E） - 一个新的硬件感知搜索空间和半自动搜索算法 - 用硬件感知的块替换效率低下的块。接下来，我们提出了一种称为RAN-IMPLICIC（RAN-I）的无训练模型缩放方法，从理论上讲，我们在非线性单元的数量方面证明了网络拓扑与其表现性之间的联系。我们证明，我们的网络在不同尺度和几种类型的硬件上实现最新的成像网结果。例如，与有效网络-lite-B0相比，RAN-E在ARM Micro-NPU上每秒（FPS）提高了1.5倍，同时提高了类似的精度。另一方面，ran-i以相似或更好的精度表现出#macs的#macs降低2倍。我们还表明，在基于ARM的数据中心CPU上，RAN-I的FPS比Convnext高40％。最后，与基于Convnext的模型相比，基于RAN-I的对象检测网络在数据中心CPU上获得了类似或更高的映射，并且在数据中心CPU上的fps高达33％。

We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller discovers neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on a validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss. Sharing parameters among child models allows ENAS to deliver strong empirical performances, while using much fewer GPUhours than existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. On the Penn Treebank dataset, ENAS discovers a novel architecture that achieves a test perplexity of 55.8, establishing a new state-of-the-art among all methods without post-training processing. On the CIFAR-10 dataset, ENAS finds a novel architecture that achieves 2.89% test error, which is on par with the 2.65% test error of NAS-Net (Zoph et al., 2018).

深层神经网络（DNN）是通过依次执行线性和非线性过程产生的。使用线性和非线性程序的组合对于生成足够深的特征空间至关重要。大多数非线性运算符是激活函数或合并函数的推导。数学形态是数学的一个分支，为各种图像处理问题提供了非线性操作员。我们调查了将这些操作集成到本文端到端深度学习框架中的实用性。 DNN旨在获得特定工作的现实代表。形态运算符给出拓扑描述符，以传达有关图像中描述的物体形状的显着信息。我们提出了一种基于元学习的方法，将形态算子纳入DNN。博学的结构展示了我们的新型形态操作如何显着提高各种任务（包括图片分类和边缘检测）的DNN性能。

Designing accurate and efficient ConvNets for mobile devices is challenging because the design space is combinatorially large. Due to this, previous neural architecture search (NAS) methods are computationally expensive. ConvNet architecture optimality depends on factors such as input resolution and target devices. However, existing approaches are too resource demanding for case-by-case redesigns. Also, previous work focuses primarily on reducing FLOPs, but FLOP count does not always reflect actual latency. To address these, we propose a differentiable neural architecture search (DNAS) framework that uses gradient-based methods to optimize Con-vNet architectures, avoiding enumerating and training individual architectures separately as in previous methods. FBNets (Facebook-Berkeley-Nets), a family of models discovered by DNAS surpass state-of-the-art models both designed manually and generated automatically. FBNet-B achieves 74.1% top-1 accuracy on ImageNet with 295M FLOPs and 23.1 ms latency on a Samsung S8 phone, 2.4x smaller and 1.5x faster than MobileNetV2-1.3[17] with similar accuracy. Despite higher accuracy and lower latency than MnasNet[20], we estimate FBNet-B's search cost is 420x smaller than MnasNet's, at only 216 GPUhours. Searched for different resolutions and channel sizes, FBNets achieve 1.5% to 6.4% higher accuracy than Mo-bileNetV2. The smallest FBNet achieves 50.2% accuracy and 2.9 ms latency (345 frames per second) on a Samsung S8. Over a Samsung-optimized FBNet, the iPhone-Xoptimized model achieves a 1.4x speedup on an iPhone X. FBNet models are open-sourced at https://github. com/facebookresearch/mobile-vision. * Work done while interning at Facebook.… Figure 1. Differentiable neural architecture search (DNAS) for ConvNet design. DNAS explores a layer-wise space that each layer of a ConvNet can choose a different block. The search space is represented by a stochastic super net. The search process trains the stochastic super net using SGD to optimize the architecture distribution. Optimal architectures are sampled from the trained distribution. The latency of each operator is measured on target devices and used to compute the loss for the super net.

Automl的一个重要目标是自动化在探索域内的新任务上的神经网络设计。通过这一目标激励，我们研究了使用户能够发现来自其特定域的数据的正确神经操作的问题。我们介绍一个名为XD-Operation的搜索空间，这些操作模仿标准多通道卷曲的归纳偏差，同时更具表现力：我们证明它包括多个应用程序区域的许多命名操作。从Reset等任何标准骨干开始，我们展示了如何通过XD操作将其转换为搜索空间以及如何使用简单的权重共享方案遍历空间。在各种任务组合 - 求解PDES，距离蛋白质折叠和音乐建模的距离预测 - 我们的方法一致地产生比基线网络更低的误差的模型，并且通常更低的误差比专业设计的域特定方法更低。

最近，变压器和多层感知器（MLP）体系结构在各种视觉任务上取得了令人印象深刻的结果。但是，如何有效地结合这些操作员形成高性能混合视觉体系结构仍然是一个挑战。在这项工作中，我们通过提出一种新型的统一体系结构搜索方法来研究卷积，变压器和MLP的可学习组合。我们的方法包含两个关键设计，以实现高性能网络的搜索。首先，我们以统一的形式对截然不同的可搜索运算符进行建模，从而使操作员能够用相同的配置参数进行表征。这样，总体搜索空间规模大大减少，总搜索成本变得负担得起。其次，我们提出上下文感知的倒数采样模块（DSM），以减轻不同类型的操作员之间的差距。我们提出的DSM能够更好地适应不同类型的操作员的功能，这对于识别高性能混合体系结构很重要。最后，我们将可配置的运算符和DSM集成到统一的搜索空间中，并使用基于增强学习的搜索算法进行搜索，以充分探索操作员的最佳组合。为此，我们搜索一个基线网络并扩大规模，以获得一个名为UNINET的模型系列，该模型的准确性和效率比以前的Convnets和Transformers更好。特别是，我们的UNET-B5在ImageNet上获得了84.9％的TOP-1精度，比效应网络-B7和Botnet-T7分别少了44％和55％。通过在Imagenet-21K上进行预处理，我们的UNET-B6获得了87.4％，表现优于SWIN-L，拖鞋少51％，参数减少了41％。代码可在https://github.com/sense-x/uninet上找到。

There is growing interest in automating neural network architecture design. Existing architecture search methods can be computationally expensive, requiring thousands of different architectures to be trained from scratch. Recent work has explored weight sharing across models to amortize the cost of training. Although previous methods reduced the cost of architecture search by orders of magnitude, they remain complex, requiring hypernetworks or reinforcement learning controllers. We aim to understand weight sharing for one-shot architecture search. With careful experimental analysis, we show that it is possible to efficiently identify promising architectures from a complex search space without either hypernetworks or RL.

我们提出了三种新型的修剪技术，以提高推理意识到的可区分神经结构搜索（DNAS）的成本和结果。首先，我们介绍了DNA的随机双路构建块，它可以通过内存和计算复杂性在内部隐藏尺寸上进行搜索。其次，我们在搜索过程中提出了一种在超级网的随机层中修剪块的算法。第三，我们描述了一种在搜索过程中修剪不必要的随机层的新技术。由搜索产生的优化模型称为Prunet，并在Imagenet Top-1图像分类精度的推理潜伏期中为NVIDIA V100建立了新的最先进的Pareto边界。将Prunet作为骨架还优于COCO对象检测任务的GPUNET和EFIDENENET，相对于平均平均精度（MAP）。

Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation. One crucial aspect for this progress are novel neural architectures. Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and errorprone process. Because of this, there is growing interest in automated neural architecture search methods. We provide an overview of existing work in this field of research and categorize them according to three dimensions: search space, search strategy, and performance estimation strategy.

作为梯度引导的搜索方法，可区分的神经体系结构搜索（飞镖）大大降低了计算成本，并加快了搜索的速度。在飞镖中，将体系结构参数引入候选操作，但是某些配备权重的操作的参数可能在初始阶段训练不好，这会导致候选操作之间的不公平竞争。无重量的操作大量出现，导致性能崩溃现象。此外，在训练超网中将占用许多内存，这会导致内存利用率较低。在本文中，提出了基于通道注意的部分通道连接，以进行可区分的神经体系结构搜索（ADARTS）。一些具有较高权重的通道是通过注意机制选择的，并将其他通道直接与处理的通道接触到操作空间。选择一些具有较高注意力权重的通道可以更好地将重要的功能信息传输到搜索空间中，并大大提高搜索效率和内存利用率。也可以避免由随机选择引起的网络结构的不稳定性。实验结果表明，ADART在CIFAR-10和CIFAR-100上分别达到了2.46％和17.06％的分类错误率。 Adarts可以有效地解决一个问题，即搜索过程中出现过多的跳过连接并获得具有更好性能的网络结构。

Recently, Neural Architecture Search (NAS) has successfully identified neural network architectures that exceed human designed ones on large-scale image classification. In this paper, we study NAS for semantic image segmentation. Existing works often focus on searching the repeatable cell structure, while hand-designing the outer network structure that controls the spatial resolution changes. This choice simplifies the search space, but becomes increasingly problematic for dense image prediction which exhibits a lot more network level architectural variations. Therefore, we propose to search the network level structure in addition to the cell level structure, which forms a hierarchical architecture search space. We present a network level search space that includes many popular designs, and develop a formulation that allows efficient gradient-based architecture search (3 P100 GPU days on Cityscapes images). We demonstrate the effectiveness of the proposed method on the challenging Cityscapes, PASCAL VOC 2012, and ADE20K datasets. Auto-DeepLab, our architecture searched specifically for semantic image segmentation, attains state-of-the-art performance without any ImageNet pretraining. 1 * Work done while an intern at Google.

与其他基于架构的NAS方法不同，广泛的神经结构搜索（BNA）提出了一个广泛的，它由卷积和增强块组成，被称为广泛的卷积神经网络（BCNN）作为搜索空间，以惊人的效率改进。 BCNN重用卷积块中的单元格的拓扑，使得BNA可以使用很少的小区以获得有效的搜索。此外，提出了多尺度特征融合和知识嵌入，以提高BCNN具有浅层拓扑的性能。然而，BNA遭受了一些缺点：1）特征融合和增强的代表性多样性不足，2）人类专家对知识嵌入设计的耗时。在本文中，我们提出了堆叠的BNA，其搜索空间是名为堆叠BCNN的开发的广泛可扩展架构，性能比BNA更好。一方面，堆叠的BCNN将Mini-BCNN视为保存综合表示的基本块，并提供强大的特征提取能力。另一方面，我们提出了知识嵌入搜索（KES）来学习适当的知识嵌入。实验结果表明，1）堆叠的BNA获得比BNA，2）KES有助于降低具有令人满意的性能的学习架构参数，3）堆叠BNA可提供0.02 GPU天的最新效率。

Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks-PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS [41], a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators are not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks - Trainability, Expressivity, Generalization - all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of "TEG" characteristics, observing that while a good local minimum is easier to find on NAS-Bench-201 given its simple topology, balancing "TEG" characteristics is much harder on the DARTS search space due to its complex landscape geometry. Our code is available at https://github.com/VITA-Group/TEGNAS.