我们将受约束的线性数据特征映射模型提出作为使用卷积神经网络(CNN)的图像分类的可解释数学模型。从这个角度来看,我们建立了线性系统的传统迭代方案与Reset-and Mgnet型模型的基本块体系结构之间的详细连接。使用这些连接,我们介绍了一些修改的Reset模型,与原始模型相比具有更少的参数,但可以产生更准确的结果,从而展示该受约束的学习数据特征映射假设的有效性。基于此假设,我们进一步提出了一般的数据特征迭代方案来展示MGNet的合理性。我们还对MGNet提供系统的数值研究,以显示其在图像分类问题中的成功和优势,并展示其与已建立的网络相比的优点。
translated by 谷歌翻译
本文着重于在二维空间中建立深层卷积神经网络(CNN)的$ l^2 $近似属性。该分析基于具有较大空间大小和多通道的卷积内核的分解定理。鉴于分解结果,relu激活函数的性质和通道的特定结构,通过显示其与一层隐藏层的Relu神经网络(NNS)的联系,获得了具有经典结构的深层relu CNN的通用近似定理。此外,基于这些网络之间的连接,可以为具有重新NET,PER-ACT RESNET和MGNET体系结构的一个版本的神经网络获得近似属性。
translated by 谷歌翻译
Current state-of-the-art deep neural networks for image classification are made up of 10 - 100 million learnable weights and are therefore inherently prone to overfitting. The complexity of the weight count can be seen as a function of the number of channels, the spatial extent of the input and the number of layers of the network. Due to the use of convolutional layers the scaling of weight complexity is usually linear with regards to the resolution dimensions, but remains quadratic with respect to the number of channels. Active research in recent years in terms of using multigrid inspired ideas in deep neural networks have shown that on one hand a significant number of weights can be saved by appropriate weight sharing and on the other that a hierarchical structure in the channel dimension can improve the weight complexity to linear. In this work, we combine these multigrid ideas to introduce a joint framework of multigrid inspired architectures, that exploit multigrid structures in all relevant dimensions to achieve linear weight complexity scaling and drastically reduced weight counts. Our experiments show that this structured reduction in weight count is able to reduce overfitting and thus shows improved performance over state-of-the-art ResNet architectures on typical image classification benchmarks at lower network complexity.
translated by 谷歌翻译
我们提出了一种多移民通道(MGIC)方法,该方法可以解决参数数量相对于标准卷积神经网络(CNN)中的通道数的二次增长。因此,我们的方法解决了CNN中的冗余,这也被轻量级CNN的成功所揭示。轻巧的CNN可以达到与参数较少的标准CNN的可比精度。但是,权重的数量仍然随CNN的宽度四倍地缩放。我们的MGIC体系结构用MGIC对应物代替了每个CNN块,该块利用了小组大小的嵌套分组卷积的层次结构来解决此问题。因此,我们提出的架构相对于网络的宽度线性扩展,同时保留了通道的完整耦合,如标准CNN中。我们对图像分类,分割和点云分类进行的广泛实验表明,将此策略应用于Resnet和MobilenetV3等不同体系结构,可以减少参数的数量,同时获得相似或更好的准确性。
translated by 谷歌翻译
神经运营商最近成为设计神经网络形式的功能空间之间的解决方案映射的流行工具。不同地,从经典的科学机器学习方法,以固定分辨率为输入参数的单个实例学习参数,神经运算符近似PDE系列的解决方案图。尽管他们取得了成功,但是神经运营商的用途迄今为止仅限于相对浅的神经网络,并限制了学习隐藏的管理法律。在这项工作中,我们提出了一种新颖的非局部神经运营商,我们将其称为非本体内核网络(NKN),即独立的分辨率,其特征在于深度神经网络,并且能够处理各种任务,例如学习管理方程和分类图片。我们的NKN源于神经网络的解释,作为离散的非局部扩散反应方程,在无限层的极限中,相当于抛物线非局部方程,其稳定性通过非本种载体微积分分析。与整体形式的神经运算符相似允许NKN捕获特征空间中的远程依赖性,而节点到节点交互的持续处理使NKNS分辨率独立于NKNS分辨率。与神经杂物中的相似性,在非本体意义上重新解释,并且层之间的稳定网络动态允许NKN的最佳参数从浅到深网络中的概括。这一事实使得能够使用浅层初始化技术。我们的测试表明,NKNS在学习管理方程和图像分类任务中占据基线方法,并概括到不同的分辨率和深度。
translated by 谷歌翻译
Solving variational image segmentation problems with hidden physics is often expensive and requires different algorithms and manually tunes model parameter. The deep learning methods based on the U-Net structure have obtained outstanding performances in many different medical image segmentation tasks, but designing such networks requires a lot of parameters and training data, not always available for practical problems. In this paper, inspired by traditional multi-phase convexity Mumford-Shah variational model and full approximation scheme (FAS) solving the nonlinear systems, we propose a novel variational-model-informed network (denoted as FAS-Unet) that exploits the model and algorithm priors to extract the multi-scale features. The proposed model-informed network integrates image data and mathematical models, and implements them through learning a few convolution kernels. Based on the variational theory and FAS algorithm, we first design a feature extraction sub-network (FAS-Solution module) to solve the model-driven nonlinear systems, where a skip-connection is employed to fuse the multi-scale features. Secondly, we further design a convolution block to fuse the extracted features from the previous stage, resulting in the final segmentation possibility. Experimental results on three different medical image segmentation tasks show that the proposed FAS-Unet is very competitive with other state-of-the-art methods in qualitative, quantitative and model complexity evaluations. Moreover, it may also be possible to train specialized network architectures that automatically satisfy some of the mathematical and physical laws in other image problems for better accuracy, faster training and improved generalization.The code is available at \url{https://github.com/zhuhui100/FASUNet}.
translated by 谷歌翻译
Recent years have witnessed a growth in mathematics for deep learning--which seeks a deeper understanding of the concepts of deep learning with mathematics, and explores how to make it more robust--and deep learning for mathematics, where deep learning algorithms are used to solve problems in mathematics. The latter has popularised the field of scientific machine learning where deep learning is applied to problems in scientific computing. Specifically, more and more neural network architectures have been developed to solve specific classes of partial differential equations (PDEs). Such methods exploit properties that are inherent to PDEs and thus solve the PDEs better than classical feed-forward neural networks, recurrent neural networks, and convolutional neural networks. This has had a great impact in the area of mathematical modeling where parametric PDEs are widely used to model most natural and physical processes arising in science and engineering, In this work, we review such methods and extend them for parametric studies as well as for solving the related inverse problems. We equally proceed to show their relevance in some industrial applications.
translated by 谷歌翻译
神经网络的经典发展主要集中在有限维欧基德空间或有限组之间的学习映射。我们提出了神经网络的概括,以学习映射无限尺寸函数空间之间的运算符。我们通过一类线性积分运算符和非线性激活函数的组成制定运营商的近似,使得组合的操作员可以近似复杂的非线性运算符。我们证明了我们建筑的普遍近似定理。此外,我们介绍了四类运算符参数化:基于图形的运算符,低秩运算符,基于多极图形的运算符和傅里叶运算符,并描述了每个用于用每个计算的高效算法。所提出的神经运营商是决议不变的:它们在底层函数空间的不同离散化之间共享相同的网络参数,并且可以用于零击超分辨率。在数值上,与现有的基于机器学习的方法,达西流程和Navier-Stokes方程相比,所提出的模型显示出卓越的性能,而与传统的PDE求解器相比,与现有的基于机器学习的方法有关的基于机器学习的方法。
translated by 谷歌翻译
在现代诊所中,医学成像至关重要,可以指导疾病的诊断和治疗。医学图像重建是医学成像的最基本和重要组成部分之一,其主要目的是以最低的成本和对患者的风险获取高质量的医学图像来临床使用。医学图像重建中的数学模型或更普遍的计算机视觉中的图像恢复一直在发挥重要作用。较早的数学模型主要是由人类知识或对要重建图像的假设设计的,我们将这些模型称为手工制作的模型。后来,手工制作的以及数据驱动的建模开始出现,这主要基于人类的设计,而从观察到的数据中学到了部分模型。最近,随着更多的数据和计算资源可用,基于深度学习的模型(或深度模型)将数据驱动的建模推向了极端,该模型主要基于以最小的人类设计为基础的学习。手工制作和数据驱动的建模都有自己的优势和缺点。医学成像的主要研究趋势之一是将手工制作的建模与深层建模相结合,以便我们可以从两种方法中享受好处。本文的主要部分是从展开的动态观点对一些有关深层建模的最新作品进行概念回顾。该观点通过优化算法和数值微分方程的灵感来刺激神经网络体系结构的新设计。鉴于深层建模的普及,该领域仍然存在巨大的挑战,以及我们将在本文结尾处讨论的机会。
translated by 谷歌翻译
These notes were compiled as lecture notes for a course developed and taught at the University of the Southern California. They should be accessible to a typical engineering graduate student with a strong background in Applied Mathematics. The main objective of these notes is to introduce a student who is familiar with concepts in linear algebra and partial differential equations to select topics in deep learning. These lecture notes exploit the strong connections between deep learning algorithms and the more conventional techniques of computational physics to achieve two goals. First, they use concepts from computational physics to develop an understanding of deep learning algorithms. Not surprisingly, many concepts in deep learning can be connected to similar concepts in computational physics, and one can utilize this connection to better understand these algorithms. Second, several novel deep learning algorithms can be used to solve challenging problems in computational physics. Thus, they offer someone who is interested in modeling a physical phenomena with a complementary set of tools.
translated by 谷歌翻译
卷积神经网络(CNN)的量化是缓解CNN部署的计算负担,尤其是在低资源边缘设备上的常见方法。但是,对于神经网络所涉及的计算类型,固定点算术并不是自然的。在这项工作中,我们探索了使用基于PDE的观点和分析来改善量化CNN的方法。首先,我们利用总变化方法(电视)方法将边缘意识平滑应用于整个网络的特征图。这旨在减少值分布的异常值并促进零件恒定图,这更适合量化。其次,我们考虑用于图像分类的常见CNN的对称和稳定变体,以及用于图源分类的图形卷积网络(GCN)。我们通过几个实验证明,正向稳定性的性质保留了在不同量化速率下网络的作用。结果,稳定的量化网络的行为与非量化的网络相似,即使它们依赖于较少的参数。我们还发现,有时,稳定性甚至有助于提高准确性。对于敏感,资源受限,低功率或实时应用(例如自动驾驶),这些属性特别感兴趣。
translated by 谷歌翻译
Deep neural networks provide unprecedented performance gains in many real world problems in signal and image processing. Despite these gains, future development and practical deployment of deep networks is hindered by their blackbox nature, i.e., lack of interpretability, and by the need for very large training sets. An emerging technique called algorithm unrolling or unfolding offers promise in eliminating these issues by providing a concrete and systematic connection between iterative algorithms that are used widely in signal processing and deep neural networks. Unrolling methods were first proposed to develop fast neural network approximations for sparse coding. More recently, this direction has attracted enormous attention and is rapidly growing both in theoretic investigations and practical applications. The growing popularity of unrolled deep networks is due in part to their potential in developing efficient, high-performance and yet interpretable network architectures from reasonable size training sets. In this article, we review algorithm unrolling for signal and image processing. We extensively cover popular techniques for algorithm unrolling in various domains of signal and image processing including imaging, vision and recognition, and speech processing. By reviewing previous works, we reveal the connections between iterative algorithms and neural networks and present recent theoretical results. Finally, we provide a discussion on current limitations of unrolling and suggest possible future research directions.
translated by 谷歌翻译
深度神经网络一直是分类任务成功的推动力,例如对象和音频识别。许多最近提出的架构似乎已经取得了令人印象深刻的结果和概括,其中大多数似乎是断开连接的。在这项工作中,我们在统一框架下对深层分类器进行了研究。特别是,我们以输入的不同程度多项式的形式表达最新的结构(例如残留和非本地网络)。我们的框架提供了有关每个模型的电感偏差的见解,并可以在其多项式性质上进行自然扩展。根据标准图像和音频分类基准评估所提出模型的功效。提出的模型的表达性既是在增加模型性能和模型压缩方面都突出的。最后,在存在有限的数据和长尾数据分布的情况下,此分类法所允许的扩展显示。我们希望这种分类法可以在现有特定领域的架构之间提供联系。源代码可在\ url {https://github.com/grigorisg9gr/polynomials-for-aigmenting-nns}中获得。
translated by 谷歌翻译
深层剩余网络(RESNET)在各种现实世界应用中显示出最先进的性能。最近,重新聚集了重新分解模型并将其解释为连续的普通微分方程或神经模型的解决方案。在这项研究中,我们提出了一个具有层变化参数的神经通用的普通微分方程(神经 - 理)模型,以进一步扩展神经模块以近似离散的重新NET。具体而言,我们使用非参数B-Spline函数来参数化神经形成,以便可以轻松平衡模型复杂性和计算效率之间的权衡。证明重新结构和神经码模型是所提出的神经形模型的特殊情况。基于两个基准数据集,MNIST和CIFAR-10,我们表明,与标准神经模板相比,与层变化的神经形成更加灵活和通用。此外,神经学享有计算和记忆益处,同时在预测准确性方面具有相当的性能。
translated by 谷歌翻译
标准化技术已成为现代卷积神经网络(Convnets)中的基本组件。特别是,许多最近的作品表明,促进重量的正交性有助于培训深层模型并提高鲁棒性。对于Courmnets,大多数现有方法基于惩罚或归一化矩阵判断或施加卷积核的重量矩阵。这些方法经常摧毁或忽视核的良性卷积结构;因此,对于深扫描器来说,它们通常是昂贵或不切实际的。相比之下,我们介绍了一种简单富有高效的“卷积归一化”(ConvNORM)方法,可以充分利用傅立叶域中的卷积结构,并用作简单的即插即用模块,以方便地结合到任何围栏中。我们的方法是通过最近关于卷积稀疏编码的预处理方法的工作启发,可以有效地促进每个层的频道方向等距。此外,我们表明我们的判断可以降低重量矩阵的层状频谱标准,从而改善网络的嘴唇,导致培训更容易培训和改善深扫描器的鲁棒性。在噪声损坏和生成的对抗网络(GAN)下应用于分类,我们表明CONVNOMOL提高了常见扫描仪(如RENET和GAN性能)的稳健性。我们通过Cifar和Imagenet的数值实验验证了我们的研究结果。
translated by 谷歌翻译
离散的不变学习旨在在无限维函数空间中学习,其能力将功能的异质离散表示作为学习模型的输入和/或输出。本文提出了一个基于整体自动编码器(IAE-NET)的新型深度学习框架,用于离散不变学习。 IAE-NET的基本构建块由编码器和解码器组成,作为与数据驱动的内核的积分转换,以及编码器和解码器之间的完全连接的神经网络。这个基本的构建块并行地在宽的多通道结构中应用,该结构反复组成,形成了一个具有跳过连接作为IAE-NET的深度连接的神经网络。 IAE-NET接受了随机数据扩展的培训,该数据具有随机数据,以生成具有异质结构的培训数据,以促进离散化不变性学习的性能。提出的IAE-NET在预测数据科学中进行了各种应用,解决了科学计算中的前进和反向问题,以及信号/图像处理。与文献中的替代方案相比,IAE-NET在现有应用中实现了最先进的性能,并创建了广泛的新应用程序。
translated by 谷歌翻译
使用卷积神经网络(CNN)已经显着改善了几种图像处理任务,例如图像分类和对象检测。与Reset和Abseralnet一样,许多架构在创建时至少在一个数据集中实现了出色的结果。培训的一个关键因素涉及网络的正规化,这可以防止结构过度装备。这项工作分析了在过去几年中开发的几种正规化方法,显示了不同CNN模型的显着改进。该作品分为三个主要区域:第一个称为“数据增强”,其中所有技术都侧重于执行输入数据的更改。第二个,命名为“内部更改”,旨在描述修改神经网络或内核生成的特征映射的过程。最后一个称为“标签”,涉及转换给定输入的标签。这项工作提出了与关于正则化的其他可用调查相比的两个主要差异:(i)第一个涉及在稿件中收集的论文并非超过五年,并第二个区别是关于可重复性,即所有作品此处推荐在公共存储库中可用的代码,或者它们已直接在某些框架中实现,例如Tensorflow或Torch。
translated by 谷歌翻译
对称性一直是探索广泛复杂系统的基本工具。在机器学习中,在模型和数据中都探索了对称性。在本文中,我们试图将模型家族架构引起的对称性与该家族的内部数据表示的对称性联系起来。我们通过计算一组基本的对称组来做到这一点,我们称它们称为模型的\ emph {Intertwiner组}。这些中的每一个都来自模型的特定非线性层,不同的非线性导致不同的对称组。这些组以模型的权重更改模型的权重,使模型所代表的基础函数保持恒定,但模型内部数据的内部表示可能会改变。我们通过一系列实验将Intertwiner组连接到模型的数据内部表示,这些实验在具有相同体系结构的模型之间探测隐藏状态之间的相似性。我们的工作表明,网络的对称性在该网络的数据表示中传播到对称性中,从而使我们更好地了解架构如何影响学习和预测过程。最后,我们推测,对于Relu网络,交织组可能会为在隐藏层而不是任意线性组合的激活基础上集中模型可解释性探索的共同实践提供理由。
translated by 谷歌翻译
Time Series Classification (TSC) is an important and challenging problem in data mining. With the increase of time series data availability, hundreds of TSC algorithms have been proposed. Among these methods, only a few have considered Deep Neural Networks (DNNs) to perform this task. This is surprising as deep learning has seen very successful applications in the last years. DNNs have indeed revolutionized the field of computer vision especially with the advent of novel deeper architectures such as Residual and Convolutional Neural Networks. Apart from images, sequential data such as text and audio can also be processed with DNNs to reach state-of-the-art performance for document classification and speech recognition. In this article, we study the current state-ofthe-art performance of deep learning algorithms for TSC by presenting an empirical study of the most recent DNN architectures for TSC. We give an overview of the most successful deep learning applications in various time series domains under a unified taxonomy of DNNs for TSC. We also provide an open source deep learning framework to the TSC community where we implemented each of the compared approaches and evaluated them on a univariate TSC benchmark (the UCR/UEA archive) and 12 multivariate time series datasets. By training 8,730 deep learning models on 97 time series datasets, we propose the most exhaustive study of DNNs for TSC to date.
translated by 谷歌翻译
在本文中,我们提出了解决稳定性和卷积神经网络(CNN)的稳定性和视野的问题的神经网络。作为提高网络深度或宽度以提高性能的替代方案,我们提出了与全球加权拉普拉斯,分数拉普拉斯和逆分数拉普拉斯算子有关的基于积分的空间非识别算子,其在物理科学中的几个问题中出现。这种网络的前向传播由部分积分微分方程(PIDE)启发。我们在自动驾驶中测试基准图像分类数据集和语义分段任务的提出神经架构的有效性。此外,我们调查了这些密集的运营商的额外计算成本以及提出神经网络的前向传播的稳定性。
translated by 谷歌翻译