计算机生成的全息图（CGH）算法在匹配模拟与物理全息显示的结果中通常不足。我们的工作通过学习全息图显示中的全息光传输来解决这一不匹配。使用摄像头和全息显示器，我们捕获了依靠理想模拟生成数据集的优化全息图的图像重建。受理想模拟的启发，我们学习了一个复杂的价值卷积内核，该内核可以传播给定的全息图，以在我们的数据集中捕获的照片。我们的方法可以显着提高全息图显示中的模拟精度和图像质量，同时为身体知情的学习方法铺平道路。

A large portion of today's world population suffer from vision impairments and wear prescription eyeglasses. However, eyeglasses causes additional bulk and discomfort when used with augmented and virtual reality headsets, thereby negatively impacting the viewer's visual experience. In this work, we remedy the usage of prescription eyeglasses in Virtual Reality (VR) headsets by shifting the optical complexity completely into software and propose a prescription-aware rendering approach for providing sharper and immersive VR imagery. To this end, we develop a differentiable display and visual perception model encapsulating display-specific parameters, color and visual acuity of human visual system and the user-specific refractive errors. Using this differentiable visual perception model, we optimize the rendered imagery in the display using stochastic gradient-descent solvers. This way, we provide prescription glasses-free sharper images for a person with vision impairments. We evaluate our approach on various displays, including desktops and VR headsets, and show significant quality and contrast improvements for users with vision impairments.

光学系统的可区分模拟可以与基于深度学习的重建网络结合使用，以通过端到端（E2E）优化光学编码器和深度解码器来实现高性能计算成像。这使成像应用程序（例如3D定位显微镜，深度估计和无透镜摄影）通过优化局部光学编码器。更具挑战性的计算成像应用，例如将3D卷压入单个2D图像的3D快照显微镜，需要高度非本地光学编码器。我们表明，现有的深网解码器具有局部性偏差，可防止这种高度非本地光学编码器的优化。我们使用全球内核傅里叶卷积神经网络（Fouriernets）基于浅神经网络体系结构的解码器来解决此问题。我们表明，在高度非本地分散镜头光学编码器捕获的照片中，傅立叶网络超过了现有的基于网络的解码器。此外，我们表明傅里叶可以对3D快照显微镜的高度非本地光学编码器进行E2E优化。通过将傅立叶网和大规模多GPU可区分的光学模拟相结合，我们能够优化非本地光学编码器170 $ \ times $ \ times $ tos 7372 $ \ times $ \ times $ \ times $比以前的最新状态，并证明了ROI的潜力-type特定的光学编码使用可编程显微镜。

Spatially varying spectral modulation can be implemented using a liquid crystal spatial light modulator (SLM) since it provides an array of liquid crystal cells, each of which can be purposed to act as a programmable spectral filter array. However, such an optical setup suffers from strong optical aberrations due to the unintended phase modulation, precluding spectral modulation at high spatial resolutions. In this work, we propose a novel computational approach for the practical implementation of phase SLMs for implementing spatially varying spectral filters. We provide a careful and systematic analysis of the aberrations arising out of phase SLMs for the purposes of spatially varying spectral modulation. The analysis naturally leads us to a set of "good patterns" that minimize the optical aberrations. We then train a deep network that overcomes any residual aberrations, thereby achieving ideal spectral modulation at high spatial resolution. We show a number of unique operating points with our prototype including dynamic spectral filtering, material classification, and single- and multi-image hyperspectral imaging.

Control of light through a microscope objective with a high numerical aperture is a common requirement in applications such as optogenetics, adaptive optics, or laser processing. Light propagation, including polarization effects, can be described under these conditions using the Debye-Wolf diffraction integral. Here, we take advantage of differentiable optimization and machine learning for efficiently optimizing the Debye-Wolf integral for such applications. For light shaping we show that this optimization approach is suitable for engineering arbitrary three-dimensional point spread functions in a two-photon microscope. For differentiable model-based adaptive optics (DAO), the developed method can find aberration corrections with intrinsic image features, for example neurons labeled with genetically encoded calcium indicators, without requiring guide stars. Using computational modeling we further discuss the range of spatial frequencies and magnitudes of aberrations which can be corrected with this approach.

Lensless cameras are a class of imaging devices that shrink the physical dimensions to the very close vicinity of the image sensor by replacing conventional compound lenses with integrated flat optics and computational algorithms. Here we report a diffractive lensless camera with spatially-coded Voronoi-Fresnel phase to achieve superior image quality. We propose a design principle of maximizing the acquired information in optics to facilitate the computational reconstruction. By introducing an easy-to-optimize Fourier domain metric, Modulation Transfer Function volume (MTFv), which is related to the Strehl ratio, we devise an optimization framework to guide the optimization of the diffractive optical element. The resulting Voronoi-Fresnel phase features an irregular array of quasi-Centroidal Voronoi cells containing a base first-order Fresnel phase function. We demonstrate and verify the imaging performance for photography applications with a prototype Voronoi-Fresnel lensless camera on a 1.6-megapixel image sensor in various illumination conditions. Results show that the proposed design outperforms existing lensless cameras, and could benefit the development of compact imaging systems that work in extreme physical conditions.

由少量镜头组成的全景环形镜头（PAL）在全景周围具有巨大潜力，该镜头围绕着移动和可穿戴设备的传感任务，因为其尺寸很小，并且视野很大（FOV）。然而，由于缺乏畸变校正的镜头，小体积PAL的图像质量仅限于光学极限。在本文中，我们提出了一个环形计算成像（ACI）框架，以打破轻质PAL设计的光学限制。为了促进基于学习的图像恢复，我们引入了基于波浪的模拟管道，用于全景成像，并通过多个数据分布来应对合成间隙。提出的管道可以轻松地适应具有设计参数的任何PAL，并且适用于宽松的设计。此外，我们考虑了全景成像和物理知识学习的物理先验，我们设计了物理知情的图像恢复网络（PI2RNET）。在数据集级别，我们创建了Divpano数据集，其广泛的实验表明，我们提出的网络在空间变化的降级下在全景图像恢复中设置了新的最新技术。此外，对只有3个球形镜头的简单PAL上提议的ACI的评估揭示了高质量全景成像与紧凑设计之间的微妙平衡。据我们所知，我们是第一个探索PAL中计算成像（CI）的人。代码和数据集将在https://github.com/zju-jiangqi/aci-pi2rnet上公开提供。

计算光学成像（COI）系统利用其设置中的光学编码元素（CE）在单个或多个快照中编码高维场景，并使用计算算法对其进行解码。 COI系统的性能很大程度上取决于其主要组件的设计：CE模式和用于执行给定任务的计算方法。常规方法依赖于随机模式或分析设计来设置CE的分布。但是，深神经网络（DNNS）的可用数据和算法功能已在CE数据驱动的设计中开辟了新的地平线，该设计共同考虑了光学编码器和计算解码器。具体而言，通过通过完全可区分的图像形成模型对COI测量进行建模，该模型考虑了基于物理的光及其与CES的相互作用，可以在端到端优化定义CE和计算解码器的参数和计算解码器（e2e）方式。此外，通过在同一框架中仅优化CE，可以从纯光学器件中执行推理任务。这项工作调查了CE数据驱动设计的最新进展，并提供了有关如何参数化不同光学元素以将其包括在E2E框架中的指南。由于E2E框架可以通过更改损耗功能和DNN来处理不同的推理应用程序，因此我们提出低级任务，例如光谱成像重建或高级任务，例如使用基于任务的光学光学体系结构来增强隐私的姿势估计，以维护姿势估算。最后，我们说明了使用全镜DNN以光速执行的分类和3D对象识别应用程序。

波前调节器的限制空间散宽产品（SBP）阻碍了大型视野（FOV）上图像的高分辨率合成/投影。我们报告了一种深度学习的衍射显示设计，该设计基于一对训练的电子编码器和衍射光学解码器，用于合成/项目超级分辨图像，使用低分辨率波形调节器。由训练有素的卷积神经网络（CNN）组成的数字编码器迅速预处理了感兴趣的高分辨率图像，因此它们的空间信息被编码为低分辨率（LR）调制模式，该模式通过低SBP Wavefront调制器投影。衍射解码器使用薄的传播层处理该LR编码的信息，这些层是使用深度学习构成的，以在其输出FOV处进行全面合成和项目超级分辨图像。我们的结果表明，这种衍射图像显示可以达到〜4的超分辨率因子，表明SBP增加了约16倍。我们还使用3D打印的衍射解码器在THZ光谱上进行实验验证了这种衍射超分辨率显示器的成功。该衍射图像解码器可以缩放以在可见的波长下运行，并激发紧凑，低功率和计算效率的大型FOV和高分辨率显示器的设计。

计算机生成的全息图（CGHS）用于全息三维（3D）显示器和全息投影。使用阶段的CGHS的重建图像的质量降低，因为重建图像的幅度难以控制。迭代优化方法，例如Gerchberg-Saxton（GS）算法是提高图像质量的一个选项。它们以迭代方式优化CGHS以获得更高的图像质量。然而，这种迭代计算是耗时的，并且图像质量的改善通常是停滞的。最近，已经提出了基于深度学习的全息图计算。深神经网络直接从输入图像数据推断出CGHS。然而，它仅限于重建与全息图相同的图像。在这项研究中，我们使用深度学习来优化使用缩放衍射计算和随机相位的方法生成的阶段CGHS。通过将随机相移方法与缩放的衍射计算组合，可以处理大于全息图的缩放重建图像。与GS算法相比，所提出的方法优化高质量和速度。

光学成像通常用于行业和学术界的科学和技术应用。在图像传感中，通过数字化图像的计算分析来执行一个测量，例如对象的位置。新兴的图像感应范例通过设计光学组件来执行不进行成像而是编码，从而打破了数据收集和分析之间的描述。通过将图像光学地编码为适合有效分析后的压缩，低维的潜在空间，这些图像传感器可以以更少的像素和更少的光子来工作，从而可以允许更高的直通量，较低的延迟操作。光学神经网络（ONNS）提供了一个平台，用于处理模拟，光学域中的数据。然而，基于ONN的传感器仅限于线性处理，但是非线性是深度的先决条件，而多层NNS在许多任务上的表现都大大优于浅色。在这里，我们使用商业图像增强器作为平行光电子，光学到光学非线性激活函数，实现用于图像传感的多层预处理器。我们证明，非线性ONN前处理器可以达到高达800：1的压缩率，同时仍然可以在几个代表性的计算机视觉任务中高精度，包括机器视觉基准测试，流程度图像分类以及对对象中对象的识别，场景。在所有情况下，我们都会发现ONN的非线性和深度使其能够胜过纯线性ONN编码器。尽管我们的实验专门用于ONN传感器的光线图像，但替代ONN平台应促进一系列ONN传感器。这些ONN传感器可能通过在空间，时间和/或光谱尺寸中预处处理的光学信息来超越常规传感器，并可能具有相干和量子质量，所有这些都在光学域中。

Ever since the first microscope by Zacharias Janssen in the late 16th century, scientists have been inventing new types of microscopes for various tasks. Inventing a novel architecture demands years, if not decades, worth of scientific experience and creativity. In this work, we introduce Differentiable Microscopy ($\partial\mu$), a deep learning-based design paradigm, to aid scientists design new interpretable microscope architectures. Differentiable microscopy first models a common physics-based optical system however with trainable optical elements at key locations on the optical path. Using pre-acquired data, we then train the model end-to-end for a task of interest. The learnt design proposal can then be simplified by interpreting the learnt optical elements. As a first demonstration, based on the optical 4-$f$ system, we present an all-optical quantitative phase microscope (QPM) design that requires no computational post-reconstruction. A follow-up literature survey suggested that the learnt architecture is similar to the generalized phase contrast method developed two decades ago. Our extensive experiments on multiple datasets that include biological samples show that our learnt all-optical QPM designs consistently outperform existing methods. We experimentally verify the functionality of the optical 4-$f$ system based QPM design using a spatial light modulator. Furthermore, we also demonstrate that similar results can be achieved by an uninterpretable learning based method, namely diffractive deep neural networks (D2NN). The proposed differentiable microscopy framework supplements the creative process of designing new optical systems and would perhaps lead to unconventional but better optical designs.

我们提出了一种依赖工程点扩散功能（PSF）的紧凑型快照单眼估计技术。微观超分辨率成像中使用的传统方法，例如双螺旋PSF（DHPSF），不适合比稀疏的一组点光源更复杂的场景。我们使用cram \'er-rao下限（CRLB）显示，将DHPSF的两个叶分开，从而捕获两个单独的图像导致深度精度的急剧增加。用于生成DHPSF的相掩码的独特属性是，将相掩码分为两个半部分，导致两个裂片的空间分离。我们利用该属性建立一个基于紧凑的极化光学设置，在该设置中，我们将两个正交线性极化器放在DHPSF相位掩码的每一半上，然后使用极化敏感的摄像机捕获所得图像。模拟和实验室原型的结果表明，与包括DHPSF和Tetrapod PSF在内的最新设计相比，我们的技术达到了高达50美元的深度误差，而空间分辨率几乎没有损失。

本文涉及从由此产生的刻薄的单个图像重建折射物体形状的高度挑战性问题。由于日常生活中透明折射物体的难以达到透明折射物体，其形状的重建需要多种实际应用。最近从焦散（SFC）方法的形状作为用于合成苛性图像的光传播仿真的问题，这可以通过可微分的渲染器来解决。然而，通过折射表面的光传输的固有复杂性当前限制了相对于重建速度和鲁棒性的实用性。为了解决这些问题，我们从焦散（N-SFC）引入神经形状，这是一种基于学习的扩展，将两个组件包含在重建管道中：一个去噪模块，该模块减轻了光传输模拟的计算成本和优化基于学习梯度下降的过程，它可以使用较少的迭代来更好地收敛。广泛的实验证明了我们的神经扩展在3D玻璃印刷中质量控制的情况下的有效性，在那里我们在计算速度和最终表面误差方面显着优于当前最先进的。

傅立叶Ptychographic显微镜（FPM）是一种成像过程，它通过计算平均值克服了传统的传统显微镜空间带宽产品（SBP）的限制。它利用使用低数值孔径（NA）物镜捕获的多个图像，并通过频域缝线实现高分辨率相成像。现有的FPM重建方法可以广泛地分为两种方法：基于迭代优化的方法，这些方法基于正向成像模型的物理学以及通常采用馈送深度学习框架的数据驱动方法。我们提出了一个混合模型驱动的残留网络，该网络将远期成像系统的知识与深度数据驱动的网络相结合。我们提出的架构LWGNET将传统的电线流优化算法展开为一种新型的神经网络设计，该设计通过复杂的卷积块增强了梯度图像。与其他传统的展开技术不同，LWGNET在PAR上执行时使用的阶段较少，甚至比现有的传统和深度学习技术更好，尤其是对于低成本和低动态范围CMOS传感器。低位深度和低成本传感器的性能提高有可能显着降低FPM成像设置的成本。最后，我们在收集到的实际数据上显示出始终提高的性能。

我们报告了张力层造影差异相位对比度显微镜（T2DPC），这是一种用于同时测量相和各向异性的无定量标签层析成像方法。T2DPC扩展了差异相位对比显微镜（一种定量相成像技术），以突出光的矢量性质。该方法求解了从配备有LED矩阵，圆极偏振器和偏振敏感摄像机的标准显微镜获得的强度测量的各向异性样品的介电常数张量。我们证明了各种验证样品的折射率，双折射和方向的准确体积重建，并证明生物标本的重建极化结构是病理学的预测。

本书章节介绍了如何利用散射光场中的光谱相关性来进行高精度的飞行时间感测。本章应作为温和的介绍，旨在用于计算成像科学家和新手合成波长成像主题的学生。技术细节（例如检测器或光源规格）将在很大程度上省略。取而代之的是，不同方法之间的相似性将被强调“绘制更大的图景”。

具有多核光纤（MCF）无透镜微观镜片的定制光的产生广泛用于生物医学。然而，用于这种应用的计算机生成的全息图（CGHS）通常由迭代算法产生，这需要高计算工作，限制在体内光源刺激和光纤细胞操纵中的高级应用。纤维芯的随机和离散分布对CGHS引起了强烈的空间偏大，因此，非常需要一种能够快速生成MCF的量身定制的CGHS的方法。我们展示了一种新型阶段编码器深神经网络（Coreenet），它可以在近视频速率下为MCF产生精确定制的CGHS。模拟表明，与传统的CGH技术相比，CoreNet可以将计算时间加速两个大小，并增加产生的光场的保真度。首次，实时生成的定制CGHS在飞行中加载到仅相位的SLM，用于通过MCF微内窥镜在实验中产生动态光场。这铺设了实时细胞旋转的途径和几种需要在生物医学中实时高保真光传递的几种进一步的应用。

Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72 times the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially-repeating virtual spectral filter array with 2x2=4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.

We present a novel single-shot interferometric ToF camera targeted for precise 3D measurements of dynamic objects. The camera concept is based on Synthetic Wavelength Interferometry, a technique that allows retrieval of depth maps of objects with optically rough surfaces at submillimeter depth precision. In contrast to conventional ToF cameras, our device uses only off-the-shelf CCD/CMOS detectors and works at their native chip resolution (as of today, theoretically up to 20 Mp and beyond). Moreover, we can obtain a full 3D model of the object in single-shot, meaning that no temporal sequence of exposures or temporal illumination modulation (such as amplitude or frequency modulation) is necessary, which makes our camera robust against object motion. In this paper, we introduce the novel camera concept and show first measurements that demonstrate the capabilities of our system. We present 3D measurements of small (cm-sized) objects with > 2 Mp point cloud resolution (the resolution of our used detector) and up to sub-mm depth precision. We also report a "single-shot 3D video" acquisition and a first single-shot "Non-Line-of-Sight" measurement. Our technique has great potential for high-precision applications with dynamic object movement, e.g., in AR/VR, industrial inspection, medical imaging, and imaging through scattering media like fog or human tissue.