衍射深神经网络（D2NNS）定义了一个由空间工程的被动表面组成的全光计算框架，该框架通过调节传播光的幅度和/或相位来共同处理光学输入信息。衍射光学网络通过薄衍射量以光的速度来完成其计算任务，而无需任何外部计算能力，同时利用了光学的巨大并行性。证明了衍射网络以实现对象的全光分类并执行通用线性变换。在这里，我们首次证明了使用衍射网络的“延时”图像分类方案，通过使用输入对象的横向运动和/或衍射网络，可以显着提高其在复杂输入对象上的分类准确性和概括性性能。 ，相对于彼此。在不同的上下文中，通常将对象和/或相机的相对运动用于图像超分辨率应用程序；受其成功的启发，我们设计了一个延时衍射网络，以受益于由受控或随机横向移动创建的互补信息内容。我们从数值探索了延时衍射网络的设计空间和性能限制，从CIFAR-10数据集的对象进行光学分类中揭示了62.03％的盲测精度。这构成了迄今使用CIFAR-10数据集上的单个衍射网络达到的最高推理精度。延时衍射网络将对使用全光处理器的输入信号的时空分析广泛有用。

随机且未知的散射介质背后的对象的分类为计算成像和机器视野字段的具有挑战性的任务。最新的基于深度学习的方法证明了使用图像传感器收集的扩散器延伸模式对对象进行分类。这些方法需要使用在数字计算机上运行的深神经网络进行相对大规模的计算。在这里，我们提出了一个全光处理器，使用单个像素检测到的宽带照明通过未知的随机相扩散器直接对未知对象进行分类。使用深度学习进行了优化的一组传播衍射层，形成了一个物理网络，该物理网络全面地绘制了随机扩散器后面输入对象的空间信息，以进入通过单个像素在输出平面上检测到的输出光的功率谱，衍射网络。我们在数值上使用宽带辐射通过随机新扩散器对未知手写数字进行分类，在训练阶段从未使用过，并实现了88.53％的盲目测试准确性。这种通过随机扩散器的单像素全光对象分类系统基于被动衍射层，该层可以通过简单地缩放与波长范围的衍射范围来缩放衍射特征，从而在电磁光谱的任何部分中运行，并且可以在电磁光谱的任何部分工作。这些结果在例如生物医学成像，安全性，机器人技术和自动驾驶中具有各种潜在的应用。

置换矩阵构成了一个重要的计算构建块，这些构建块在各个领域中经常使用，例如通信，信息安全和数据处理。具有相对较大数量的基于功率，快速和紧凑型平台的输入输出互连的置换运算符的光学实现是非常可取的。在这里，我们提出了通过深度学习设计的衍射光学网络，以全面执行置换操作，可以使用被动的传播层在输入和视场之间扩展到数十万个互连，这些互连是在波长规模上单独构造的。 。我们的发现表明，衍射光网络在近似给定置换操作中的容量与系统中衍射层和可训练的传输元件的数量成正比。这种更深的衍射网络设计可以在系统的物理对齐和输出衍射效率方面构成实际挑战。我们通过设计不对对准的衍射设计来解决这些挑战，这些设计可以全面执行任意选择的置换操作，并首次在实验中证明了在频谱的THZ部分运行的衍射排列网络。衍射排列网络可能会在例如安全性，图像加密和数据处理以及电信中找到各种应用程序；尤其是在无线通信中的载波频率接近THZ波段的情况下，提出的衍射置换网络可以潜在地充当无线网络中的通道路由和互连面板。

Diffractive optical networks provide rich opportunities for visual computing tasks since the spatial information of a scene can be directly accessed by a diffractive processor without requiring any digital pre-processing steps. Here we present data class-specific transformations all-optically performed between the input and output fields-of-view (FOVs) of a diffractive network. The visual information of the objects is encoded into the amplitude (A), phase (P), or intensity (I) of the optical field at the input, which is all-optically processed by a data class-specific diffractive network. At the output, an image sensor-array directly measures the transformed patterns, all-optically encrypted using the transformation matrices pre-assigned to different data classes, i.e., a separate matrix for each data class. The original input images can be recovered by applying the correct decryption key (the inverse transformation) corresponding to the matching data class, while applying any other key will lead to loss of information. The class-specificity of these all-optical diffractive transformations creates opportunities where different keys can be distributed to different users; each user can only decode the acquired images of only one data class, serving multiple users in an all-optically encrypted manner. We numerically demonstrated all-optical class-specific transformations covering A-->A, I-->I, and P-->I transformations using various image datasets. We also experimentally validated the feasibility of this framework by fabricating a class-specific I-->I transformation diffractive network using two-photon polymerization and successfully tested it at 1550 nm wavelength. Data class-specific all-optical transformations provide a fast and energy-efficient method for image and data encryption, enhancing data security and privacy.

A unidirectional imager would only permit image formation along one direction, from an input field-of-view (FOV) A to an output FOV B, and in the reverse path, the image formation would be blocked. Here, we report the first demonstration of unidirectional imagers, presenting polarization-insensitive and broadband unidirectional imaging based on successive diffractive layers that are linear and isotropic. These diffractive layers are optimized using deep learning and consist of hundreds of thousands of diffractive phase features, which collectively modulate the incoming fields and project an intensity image of the input onto an output FOV, while blocking the image formation in the reverse direction. After their deep learning-based training, the resulting diffractive layers are fabricated to form a unidirectional imager. As a reciprocal device, the diffractive unidirectional imager has asymmetric mode processing capabilities in the forward and backward directions, where the optical modes from B to A are selectively guided/scattered to miss the output FOV, whereas for the forward direction such modal losses are minimized, yielding an ideal imaging system between the input and output FOVs. Although trained using monochromatic illumination, the diffractive unidirectional imager maintains its functionality over a large spectral band and works under broadband illumination. We experimentally validated this unidirectional imager using terahertz radiation, very well matching our numerical results. Using the same deep learning-based design strategy, we also created a wavelength-selective unidirectional imager, where two unidirectional imaging operations, in reverse directions, are multiplexed through different illumination wavelengths. Diffractive unidirectional imaging using structured materials will have numerous applications in e.g., security, defense, telecommunications and privacy protection.

Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72 times the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially-repeating virtual spectral filter array with 2x2=4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.

波前调节器的限制空间散宽产品（SBP）阻碍了大型视野（FOV）上图像的高分辨率合成/投影。我们报告了一种深度学习的衍射显示设计，该设计基于一对训练的电子编码器和衍射光学解码器，用于合成/项目超级分辨图像，使用低分辨率波形调节器。由训练有素的卷积神经网络（CNN）组成的数字编码器迅速预处理了感兴趣的高分辨率图像，因此它们的空间信息被编码为低分辨率（LR）调制模式，该模式通过低SBP Wavefront调制器投影。衍射解码器使用薄的传播层处理该LR编码的信息，这些层是使用深度学习构成的，以在其输出FOV处进行全面合成和项目超级分辨图像。我们的结果表明，这种衍射图像显示可以达到〜4的超分辨率因子，表明SBP增加了约16倍。我们还使用3D打印的衍射解码器在THZ光谱上进行实验验证了这种衍射超分辨率显示器的成功。该衍射图像解码器可以缩放以在可见的波长下运行，并激发紧凑，低功率和计算效率的大型FOV和高分辨率显示器的设计。

Photonic neural networks are brain-inspired information processing technology using photons instead of electrons to perform artificial intelligence (AI) tasks. However, existing architectures are designed for a single task but fail to multiplex different tasks in parallel within a single monolithic system due to the task competition that deteriorates the model performance. This paper proposes a novel optical multi-task learning system by designing multi-wavelength diffractive deep neural networks (D2NNs) with the joint optimization method. By encoding multi-task inputs into multi-wavelength channels, the system can increase the computing throughput and significantly alle-viate the competition to perform multiple tasks in parallel with high accuracy. We design the two-task and four-task D2NNs with two and four spectral channels, respectively, for classifying different inputs from MNIST, FMNIST, KMNIST, and EMNIST databases. The numerical evaluations demonstrate that, under the same network size, mul-ti-wavelength D2NNs achieve significantly higher classification accuracies for multi-task learning than single-wavelength D2NNs. Furthermore, by increasing the network size, the multi-wavelength D2NNs for simultaneously performing multiple tasks achieve comparable classification accuracies with respect to the individual training of multiple single-wavelength D2NNs to perform tasks separately. Our work paves the way for developing the wave-length-division multiplexing technology to achieve high-throughput neuromorphic photonic computing and more general AI systems to perform multiple tasks in parallel.

由于其潜在的优势，如可扩展性，低延迟和功率效率，光学计算在过去几十年中已经看到了快速进步。潜在的全光处理器的核心单元将是NAND门，其可以级联以执行任意逻辑操作。在这里，我们使用衍射神经网络呈现可级可级联的全光NAND门的设计和分析。我们使用两个空间分离的孔的相对光功率在衍射NAND门的输入和输出平面上进行了编码的逻辑值。基于该架构，我们使用光的衍射来了数值优化了由4个无源层组成的衍射神经网络的设计，并通过光的衍射来实现这些衍射NAND操作，通过连续地馈送输出来级联这些衍射NAND门来执行复杂的逻辑功能一个衍射NAND门进入另一个。我们通过使用相同的衍射设计来显示我们的衍射NAND门的级联性，以及全光学执行和或操作以及半加法器。由空间工程化无源衍射层组成的可级可抵级光学NAND门可以用作各种光学计算平台的核心组件。

计算光学成像（COI）系统利用其设置中的光学编码元素（CE）在单个或多个快照中编码高维场景，并使用计算算法对其进行解码。 COI系统的性能很大程度上取决于其主要组件的设计：CE模式和用于执行给定任务的计算方法。常规方法依赖于随机模式或分析设计来设置CE的分布。但是，深神经网络（DNNS）的可用数据和算法功能已在CE数据驱动的设计中开辟了新的地平线，该设计共同考虑了光学编码器和计算解码器。具体而言，通过通过完全可区分的图像形成模型对COI测量进行建模，该模型考虑了基于物理的光及其与CES的相互作用，可以在端到端优化定义CE和计算解码器的参数和计算解码器（e2e）方式。此外，通过在同一框架中仅优化CE，可以从纯光学器件中执行推理任务。这项工作调查了CE数据驱动设计的最新进展，并提供了有关如何参数化不同光学元素以将其包括在E2E框架中的指南。由于E2E框架可以通过更改损耗功能和DNN来处理不同的推理应用程序，因此我们提出低级任务，例如光谱成像重建或高级任务，例如使用基于任务的光学光学体系结构来增强隐私的姿势估计，以维护姿势估算。最后，我们说明了使用全镜DNN以光速执行的分类和3D对象识别应用程序。

与常规深层神经网络（DNN）相比，衍射光学神经网络（DONNS）在功率效率，并行性和计算速度方面具有显着优势，因此引起了很多关注，这些神经网络（DNN）在数字平台上实现时具有内在的限制。但是，反相反的算法训练的物理模型参数上具有离散值的现实世界光学设备是一个非平凡的任务，因为现有的光学设备具有非统一的离散级别和非单调属性。这项工作提出了一个新颖的设备对系统硬件软件代码框架，该框架可以对Donns W.R.T的有效物理意识培训进行跨层的任意实验测量的光学设备。具体而言，使用Gumbel-SoftMax来启用从现实世界设备参数的可区分映射到Donns的正向函数，在Donn中，Donn中的物理参数可以通过简单地最小化ML任务的损耗函数来训练。结果表明，我们提出的框架比传统的基于基于量化的方法具有显着优势，尤其是使用低精确的光学设备。最后，在低精度设置中，通过物理实验光学系统对所提出的算法进行了充分的验证。

光学成像通常用于行业和学术界的科学和技术应用。在图像传感中，通过数字化图像的计算分析来执行一个测量，例如对象的位置。新兴的图像感应范例通过设计光学组件来执行不进行成像而是编码，从而打破了数据收集和分析之间的描述。通过将图像光学地编码为适合有效分析后的压缩，低维的潜在空间，这些图像传感器可以以更少的像素和更少的光子来工作，从而可以允许更高的直通量，较低的延迟操作。光学神经网络（ONNS）提供了一个平台，用于处理模拟，光学域中的数据。然而，基于ONN的传感器仅限于线性处理，但是非线性是深度的先决条件，而多层NNS在许多任务上的表现都大大优于浅色。在这里，我们使用商业图像增强器作为平行光电子，光学到光学非线性激活函数，实现用于图像传感的多层预处理器。我们证明，非线性ONN前处理器可以达到高达800：1的压缩率，同时仍然可以在几个代表性的计算机视觉任务中高精度，包括机器视觉基准测试，流程度图像分类以及对对象中对象的识别，场景。在所有情况下，我们都会发现ONN的非线性和深度使其能够胜过纯线性ONN编码器。尽管我们的实验专门用于ONN传感器的光线图像，但替代ONN平台应促进一系列ONN传感器。这些ONN传感器可能通过在空间，时间和/或光谱尺寸中预处处理的光学信息来超越常规传感器，并可能具有相干和量子质量，所有这些都在光学域中。

Ever since the first microscope by Zacharias Janssen in the late 16th century, scientists have been inventing new types of microscopes for various tasks. Inventing a novel architecture demands years, if not decades, worth of scientific experience and creativity. In this work, we introduce Differentiable Microscopy ($\partial\mu$), a deep learning-based design paradigm, to aid scientists design new interpretable microscope architectures. Differentiable microscopy first models a common physics-based optical system however with trainable optical elements at key locations on the optical path. Using pre-acquired data, we then train the model end-to-end for a task of interest. The learnt design proposal can then be simplified by interpreting the learnt optical elements. As a first demonstration, based on the optical 4-$f$ system, we present an all-optical quantitative phase microscope (QPM) design that requires no computational post-reconstruction. A follow-up literature survey suggested that the learnt architecture is similar to the generalized phase contrast method developed two decades ago. Our extensive experiments on multiple datasets that include biological samples show that our learnt all-optical QPM designs consistently outperform existing methods. We experimentally verify the functionality of the optical 4-$f$ system based QPM design using a spatial light modulator. Furthermore, we also demonstrate that similar results can be achieved by an uninterpretable learning based method, namely diffractive deep neural networks (D2NN). The proposed differentiable microscopy framework supplements the creative process of designing new optical systems and would perhaps lead to unconventional but better optical designs.

在2015年和2019年之间，地平线的成员2020年资助的创新培训网络名为“Amva4newphysics”，研究了高能量物理问题的先进多变量分析方法和统计学习工具的定制和应用，并开发了完全新的。其中许多方法已成功地用于提高Cern大型Hadron撞机的地图集和CMS实验所执行的数据分析的敏感性;其他几个人，仍然在测试阶段，承诺进一步提高基本物理参数测量的精确度以及新现象的搜索范围。在本文中，在研究和开发的那些中，最相关的新工具以及对其性能的评估。

具有最小延迟的人工神经网络的决策对于诸如导航，跟踪和实时机器动作系统之类的许多应用来说是至关重要的。这要求机器学习硬件以高吞吐量处理多维数据。不幸的是，处理卷积操作是数据分类任务的主要计算工具，遵循有挑战性的运行时间复杂性缩放法。然而，在傅立叶光学显示器 - 光处理器中同心地实现卷积定理，使得不迭代的O（1）运行时复杂度以超过1,000×1,000大矩阵的数据输入。在此方法之后，这里我们展示了具有傅里叶卷积神经网络（FCNN）加速器的数据流多核图像批处理。我们将大规模矩阵的图像批量处理显示为傅立叶域中的数字光处理模块执行的被动的2000万点产品乘法。另外，我们通过利用多种时空衍射令并进一步并行化该光学FCNN系统，从而实现了最先进的FCNN加速器的98倍的产量改进。综合讨论与系统能力边缘工作相关的实际挑战突出了傅立叶域和决议缩放法律的串扰问题。通过利用展示技术中的大规模平行性加速卷积带来了基于VAN Neuman的机器学习加速度。

由于深度学习在许多人工智能应用中显示了革命性的性能，其升级的计算需求需要用于巨大并行性的硬件加速器和改进的吞吐量。光学神经网络（ONN）是下一代神经关键组成的有希望的候选者，由于其高并行，低延迟和低能量消耗。在这里，我们设计了一个硬件高效的光子子空间神经网络（PSNN）架构，其针对具有比具有可比任务性能的前一个ONN架构的光学元件使用，区域成本和能量消耗。此外，提供了一种硬件感知培训框架，以最小化所需的设备编程精度，减少芯片区域，并提高噪声鲁棒性。我们在实验上展示了我们的PSNN在蝴蝶式可编程硅光子集成电路上，并在实用的图像识别任务中显示其实用性。

过去的十年见证了深度学习在各种计算成像，传感和显微镜任务中的变革性应用。由于采用了有监督的学习方案，因此大多数方法取决于大规模，多样化和标记的培训数据。此类培训图像数据集的获取和准备通常很费力且昂贵，也导致对新样本类型的估计和概括有限。在这里，我们报告了一种称为Gedankennet的自制学习模型，该模型消除了对标签或实验培训数据的需求，并证明了其对全息图重建任务的有效性和卓越的概括。如果没有关于要成像的样本类型的先验知识，则使用物理矛盾的丢失和人为的随机图像进行了培训，这些模型是合成生成的，没有任何实验或与现实世界样本的相似之处。在其自制训练之后，Gedankennet成功概括为各种看不见的生物样品的实验全息图，并使用实验获得的测试全息图重建了不同类型对象的相位和振幅图像。 Gedankennet的自我监督学习实现了与Maxwell的方程相一致的复杂图像重建，无需访问实验数据或知识的真实样本或其空间特征的知识，就意味着其输出推理和对象解决方案准确地表示波传播，这实现了复杂的图像重建。在自由空间中。对图像重建任务的自我监督学习为全息，显微镜和计算成像领域的各种反问题打开了新的机会。

随着深度神经网络（DNN）的发展以解决日益复杂的问题，它们正受到现有数字处理器的延迟和功耗的限制。为了提高速度和能源效率，已经提出了专门的模拟光学和电子硬件，但是可扩展性有限（输入矢量长度$ k $的数百个元素）。在这里，我们提出了一个可扩展的，单层模拟光学处理器，该光学处理器使用自由空间光学器件可重新配置输入向量和集成的光电，用于静态，可更新的加权和非线性 - 具有$ k \ \ 1,000 $和大约1,000美元和超过。我们通过实验测试MNIST手写数字数据集的分类精度，在没有数据预处理或在硬件上进行数据重新处理的情况下达到94.7％（地面真相96.3％）。我们还确定吞吐量（$ \ sim $ 0.9 examac/s）的基本上限，由最大光带宽设置，然后大大增加误差。我们在兼容CMOS兼容系统中宽光谱和空间带宽的组合可以实现下一代DNN的高效计算。

Machine learning methods have revolutionized the discovery process of new molecules and materials. However, the intensive training process of neural networks for molecules with ever-increasing complexity has resulted in exponential growth in computation cost, leading to long simulation time and high energy consumption. Photonic chip technology offers an alternative platform for implementing neural networks with faster data processing and lower energy usage compared to digital computers. Photonics technology is naturally capable of implementing complex-valued neural networks at no additional hardware cost. Here, we demonstrate the capability of photonic neural networks for predicting the quantum mechanical properties of molecules. To the best of our knowledge, this work is the first to harness photonic technology for machine learning applications in computational chemistry and molecular sciences, such as drug discovery and materials design. We further show that multiple properties can be learned simultaneously in a photonic chip via a multi-task regression learning algorithm, which is also the first of its kind as well, as most previous works focus on implementing a network in the classification task.

The ever-growing deep learning technologies are making revolutionary changes for modern life. However, conventional computing architectures are designed to process sequential and digital programs, being extremely burdened with performing massive parallel and adaptive deep learning applications. Photonic integrated circuits provide an efficient approach to mitigate bandwidth limitations and power-wall brought by its electronic counterparts, showing great potential in ultrafast and energy-free high-performance computing. Here, we propose an optical computing architecture enabled by on-chip diffraction to implement convolutional acceleration, termed optical convolution unit (OCU). We demonstrate that any real-valued convolution kernels can be exploited by OCU with a prominent computational throughput boosting via the concept of structral re-parameterization. With OCU as the fundamental unit, we build an optical convolutional neural network (oCNN) to implement two popular deep learning tasks: classification and regression. For classification, Fashion-MNIST and CIFAR-4 datasets are tested with accuracy of 91.63% and 86.25%, respectively. For regression, we build an optical denoising convolutional neural network (oDnCNN) to handle Gaussian noise in gray scale images with noise level {\sigma} = 10, 15, 20, resulting clean images with average PSNR of 31.70dB, 29.39dB and 27.72dB, respectively. The proposed OCU presents remarkable performance of low energy consumption and high information density due to its fully passive nature and compact footprint, providing a highly parallel while lightweight solution for future computing architecture to handle high dimensional tensors in deep learning.