神经压缩算法通常基于需要专门编码器和解码器体系结构的自动编码器，以实现不同的数据模式。在本文中，我们提出了Coin ++，这是一种神经压缩框架，无缝处理广泛的数据模式。我们的方法基于将数据转换为隐式神经表示，即映射坐标（例如像素位置）为特征（例如RGB值）的神经函数。然后，我们不用直接存储隐式神经表示的权重，而是存储应用于元学习的基础网络作为数据的压缩代码的调制。我们进一步量化和熵代码这些调制，从而导致大量压缩增益，同时与基线相比，将编码时间缩短了两个数量级。我们通过压缩从图像和音频到医学和气候数据的各种数据方式来证明我们方法的有效性。

最近隐含的神经表示（INRS）作为各种数据类型的新颖且有效的表现。到目前为止，事先工作主要集中在优化其重建性能。这项工作从新颖的角度来调查INRS，即作为图像压缩的工具。为此，我们提出了基于INR的第一综合压缩管线，包括量化，量化感知再培训和熵编码。使用INRS进行编码，即对数据示例的过度装备，通常是较慢的秩序。为缓解此缺点，我们基于MAML利用META学习初始化，以便在较少的渐变更新中达到编码，这也通常提高INR的速率失真性能。我们发现，我们对INR的源压缩方法非常优于类似的事先工作，具有专门针对图像专门设计的常见压缩算法，并将基于速率 - 失真自动分析器的差距缩小到最先进的学习方法。此外，我们提供了对我们希望促进这种新颖方法对图像压缩的未来研究的重要性的广泛消融研究。

It is common practice in deep learning to represent a measurement of the world on a discrete grid, e.g. a 2D grid of pixels. However, the underlying signal represented by these measurements is often continuous, e.g. the scene depicted in an image. A powerful continuous alternative is then to represent these measurements using an implicit neural representation, a neural function trained to output the appropriate measurement value for any input spatial location. In this paper, we take this idea to its next level: what would it take to perform deep learning on these functions instead, treating them as data? In this context we refer to the data as functa, and propose a framework for deep learning on functa. This view presents a number of challenges around efficient conversion from data to functa, compact representation of functa, and effectively solving downstream tasks on functa. We outline a recipe to overcome these challenges and apply it to a wide range of data modalities including images, 3D shapes, neural radiance fields (NeRF) and data on manifolds. We demonstrate that this approach has various compelling properties across data modalities, in particular on the canonical tasks of generative modeling, data imputation, novel view synthesis and classification. Code: https://github.com/deepmind/functa

深度学习中的最新工作重新想象了数据的表示形式，因为函数从坐标空间映射到基础连续信号。当神经网络近似此类功能时，这引入了更常见的多维阵列表示的引人注目的替代方案。关于这种隐式神经表示（INR）的最新工作表明，仔细体系结构搜索 - INR可以超越建立的压缩方法，例如JPEG（例如Dupont等，2021）。在本文中，我们提出了至关重要的步骤，以使这种想法可扩展：首先，我们采用最先进的网络稀疏技术来大大改善压缩。其次，引入第一种方法，允许在常用的元学习算法的内环中使用稀疏性，从而极大地改善了压缩和学习INR的计算成本。这种形式主义的普遍性使我们能够对各种数据模式提出结果，例如图像，歧管，签名距离功能，3D形状和场景，其中一些建立了新的最新结果。

我们提出了一种压缩具有隐式神经表示的全分辨率视频序列的方法。每个帧表示为映射坐标位置到像素值的神经网络。我们使用单独的隐式网络来调制坐标输入，从而实现帧之间的有效运动补偿。与一个小的残余网络一起，这允许我们有效地相对于前一帧压缩p帧。通过使用学习的整数量化存储网络权重，我们进一步降低了比特率。我们呼叫隐式像素流（IPF）的方法，提供了几种超简化的既定神经视频编解码器：它不需要接收器可以访问预先磨普的神经网络，不使用昂贵的内插基翘曲操作，而不是需要单独的培训数据集。我们展示了神经隐式压缩对图像和视频数据的可行性。

Implicit Neural Representations (INR) have recently shown to be powerful tool for high-quality video compression. However, existing works are limiting as they do not explicitly exploit the temporal redundancy in videos, leading to a long encoding time. Additionally, these methods have fixed architectures which do not scale to longer videos or higher resolutions. To address these issues, we propose NIRVANA, which treats videos as groups of frames and fits separate networks to each group performing patch-wise prediction. This design shares computation within each group, in the spatial and temporal dimensions, resulting in reduced encoding time of the video. The video representation is modeled autoregressively, with networks fit on a current group initialized using weights from the previous group's model. To further enhance efficiency, we perform quantization of the network parameters during training, requiring no post-hoc pruning or quantization. When compared with previous works on the benchmark UVG dataset, NIRVANA improves encoding quality from 37.36 to 37.70 (in terms of PSNR) and the encoding speed by 12X, while maintaining the same compression rate. In contrast to prior video INR works which struggle with larger resolution and longer videos, we show that our algorithm is highly flexible and scales naturally due to its patch-wise and autoregressive designs. Moreover, our method achieves variable bitrate compression by adapting to videos with varying inter-frame motion. NIRVANA achieves 6X decoding speed and scales well with more GPUs, making it practical for various deployment scenarios.

我们引入基于实例自适应学习的视频压缩算法。在要传输的每个视频序列上，我们介绍了预训练的压缩模型。最佳参数与潜在代码一起发送到接收器。通过熵编码在合适的混合模型下的参数更新，我们确保可以有效地编码网络参数。该实例自适应压缩算法对于基础模型的选择是不可知的，并且具有改进任何神经视频编解码器的可能性。在UVG，HEVC和XIPH数据集上，我们的CODEC通过21％至26％的BD速率节省，提高了低延迟尺度空间流量模型的性能，以及最先进的B帧模型17至20％的BD速率储蓄。我们还证明了实例 - 自适应FineTuning改善了域移位的鲁棒性。最后，我们的方法降低了压缩模型的容量要求。我们表明它即使在将网络大小减少72％之后也能实现最先进的性能。

Recent neural compression methods have been based on the popular hyperprior framework. It relies on Scalar Quantization and offers a very strong compression performance. This contrasts from recent advances in image generation and representation learning, where Vector Quantization is more commonly employed. In this work, we attempt to bring these lines of research closer by revisiting vector quantization for image compression. We build upon the VQ-VAE framework and introduce several modifications. First, we replace the vanilla vector quantizer by a product quantizer. This intermediate solution between vector and scalar quantization allows for a much wider set of rate-distortion points: It implicitly defines high-quality quantizers that would otherwise require intractably large codebooks. Second, inspired by the success of Masked Image Modeling (MIM) in the context of self-supervised learning and generative image models, we propose a novel conditional entropy model which improves entropy coding by modelling the co-dependencies of the quantized latent codes. The resulting PQ-MIM model is surprisingly effective: its compression performance on par with recent hyperprior methods. It also outperforms HiFiC in terms of FID and KID metrics when optimized with perceptual losses (e.g. adversarial). Finally, since PQ-MIM is compatible with image generation frameworks, we show qualitatively that it can operate under a hybrid mode between compression and generation, with no further training or finetuning. As a result, we explore the extreme compression regime where an image is compressed into 200 bytes, i.e., less than a tweet.

We describe an end-to-end trainable model for image compression based on variational autoencoders. The model incorporates a hyperprior to effectively capture spatial dependencies in the latent representation. This hyperprior relates to side information, a concept universal to virtually all modern image codecs, but largely unexplored in image compression using artificial neural networks (ANNs). Unlike existing autoencoder compression methods, our model trains a complex prior jointly with the underlying autoencoder. We demonstrate that this model leads to state-of-the-art image compression when measuring visual quality using the popular MS-SSIM index, and yields rate-distortion performance surpassing published ANN-based methods when evaluated using a more traditional metric based on squared error (PSNR). Furthermore, we provide a qualitative comparison of models trained for different distortion metrics.

By optimizing the rate-distortion-realism trade-off, generative compression approaches produce detailed, realistic images, even at low bit rates, instead of the blurry reconstructions produced by rate-distortion optimized models. However, previous methods do not explicitly control how much detail is synthesized, which results in a common criticism of these methods: users might be worried that a misleading reconstruction far from the input image is generated. In this work, we alleviate these concerns by training a decoder that can bridge the two regimes and navigate the distortion-realism trade-off. From a single compressed representation, the receiver can decide to either reconstruct a low mean squared error reconstruction that is close to the input, a realistic reconstruction with high perceptual quality, or anything in between. With our method, we set a new state-of-the-art in distortion-realism, pushing the frontier of achievable distortion-realism pairs, i.e., our method achieves better distortions at high realism and better realism at low distortion than ever before.

Recent models for learned image compression are based on autoencoders, learning approximately invertible mappings from pixels to a quantized latent representation. These are combined with an entropy model, a prior on the latent representation that can be used with standard arithmetic coding algorithms to yield a compressed bitstream. Recently, hierarchical entropy models have been introduced as a way to exploit more structure in the latents than simple fully factorized priors, improving compression performance while maintaining end-to-end optimization. Inspired by the success of autoregressive priors in probabilistic generative models, we examine autoregressive, hierarchical, as well as combined priors as alternatives, weighing their costs and benefits in the context of image compression. While it is well known that autoregressive models come with a significant computational penalty, we find that in terms of compression performance, autoregressive and hierarchical priors are complementary and, together, exploit the probabilistic structure in the latents better than all previous learned models. The combined model yields state-of-the-art rate-distortion performance, providing a 15.8% average reduction in file size over the previous state-of-the-art method based on deep learning, which corresponds to a 59.8% size reduction over JPEG, more than 35% reduction compared to WebP and JPEG2000, and bitstreams 8.4% smaller than BPG, the current state-of-the-art image codec. To the best of our knowledge, our model is the first learning-based method to outperform BPG on both PSNR and MS-SSIM distortion metrics.32nd Conference on Neural Information Processing Systems (NIPS 2018),

这项工作解决了基于深度神经网络的端到端学习图像压缩（LIC）的两个主要问题：可变速率学习，其中需要单独的网络以不同的质量生成压缩图像，以及可微分近似之间的列车测试不匹配量化和真正的硬量化。我们介绍了LIC的在线元学习（OML）设置，将Meta学习和在线学习中的思想结合在条件变分自动编码器（CVAE）框架中。通过将条件变量视为元参数并将生成的条件特征视为元前沿，可以通过元参数控制所需的重建以适应变量质量的压缩。在线学习框架用于更新元参数，以便为当前图像自适应地调整条件重建。通过OML机制，可以通过SGD有效更新元参数。条件重建基于解码器网络中的量化潜在表示，因此有助于弥合训练估计与真正量化的潜在分布之间的间隙。实验表明，我们的OML方法可以灵活地应用于不同的最先进的LIC方法，以实现具有很少的计算和传输开销的额外性能改进。

量化在隐式/坐标神经网络中的作用仍未完全理解。我们注意到，在训练过程中使用规范的固定量化方案在训练过程中的网络重量分布发生变化，在训练过程中会导致低速表现不佳。在这项工作中，我们表明神经体重的不均匀量化会导致显着改善。具体而言，我们证明了群集量化可以改善重建。最后，通过表征量化和网络容量之间的权衡，我们证明使用二进制神经网络重建信号是可能的（而记忆效率低下）。我们在2D图像重建和3D辐射场上实验证明了我们的发现；并表明简单的量化方法和体系结构搜索可以使NERF的压缩至小于16KB，而性能损失最小（比原始NERF小323倍）。

隐式神经表示是通过学习作为神经网络参数化的连续功能来代表一般信号的有前途的新大道，将信号的域映射到其Codomain;例如，从图像的空间坐标映射到其像素值。能够在高尺寸信号中传送细细节，其域，隐式神经表示确保了与传统离散表示的许多优点。然而，目前的方法难以为大量信号或数据集缩放，因为学习神经表示 - 这是自身沉重的参数 - 对于每个信号，每个信号都需要大量的存储器和计算。为了解决这个问题，我们建议在稀疏性约束下结合网络压缩来利用元学习方法，使得它呈现出良好的初始化稀疏参数化，以便在随后的训练中快速发展以表示一组未见信号。我们经验证明，Meta学习的稀疏神经表示比使用相同数量的优化步骤训练时，比较稀疏的稀疏神经表示比具有相同数量的参数的致密荟萃学习模型。

由于深层网络的计算复杂性和功率约束的移动硬件的计算复杂性，因此在移动设备上实现神经视频编解码器的潜力是一项巨大的技术挑战。我们通过利用高通公司的技术和创新来证明可行性，从而弥合了从基于神经网络的编解码器模拟在壁式工作站运行的差距，再到由Snapdragon技术供电的移动设备上的实时操作。我们显示有史以来第一个在商用手机上运行的框架间神经视频解码器，实时解码高清视频，同时保持低比特率和高视觉质量。

Image compression is a fundamental research field and many well-known compression standards have been developed for many decades. Recently, learned compression methods exhibit a fast development trend with promising results. However, there is still a performance gap between learned compression algorithms and reigning compression standards, especially in terms of widely used PSNR metric. In this paper, we explore the remaining redundancy of recent learned compression algorithms. We have found accurate entropy models for rate estimation largely affect the optimization of network parameters and thus affect the rate-distortion performance. Therefore, in this paper, we propose to use discretized Gaussian Mixture Likelihoods to parameterize the distributions of latent codes, which can achieve a more accurate and flexible entropy model. Besides, we take advantage of recent attention modules and incorporate them into network architecture to enhance the performance. Experimental results demonstrate our proposed method achieves a state-of-the-art performance compared to existing learned compression methods on both Kodak and high-resolution datasets. To our knowledge our approach is the first work to achieve comparable performance with latest compression standard Versatile Video Coding (VVC) regarding PSNR. More importantly, our approach generates more visually pleasant results when optimized by MS-SSIM. The project page is at https://github.com/ZhengxueCheng/ Learned-Image-Compression-with-GMM-and-Attention.

上下文自适应熵模型的应用显着提高了速率 - 渗透率（R-D）的性能，在该表现中，超级培训和自回归模型被共同利用来有效捕获潜在表示的空间冗余。但是，潜在表示仍然包含一些空间相关性。此外，这些基于上下文自适应熵模型的方法在解码过程中无法通过并行计算设备，例如FPGA或GPU。为了减轻这些局限性，我们提出了一个学识渊博的多分辨率图像压缩框架，该框架利用了最近开发的八度卷积，以将潜在表示形式分配到高分辨率（HR）和低分辨率（LR）部分，类似于小波变换，这进一步改善了R-D性能。为了加快解码的速度，我们的方案不使用上下文自适应熵模型。取而代之的是，我们利用一个额外的超层，包括超级编码器和超级解码器，以进一步删除潜在表示的空间冗余。此外，将跨分辨率参数估计（CRPE）引入提出的框架中，以增强信息流并进一步改善速率延伸性能。提出了对总损耗函数提出的其他信息损失，以调整LR部分对最终位流的贡献。实验结果表明，与最先进的学术图像压缩方法相比，我们的方法分别将解码时间减少了约73.35％和93.44％，R-D性能仍然优于H.266/VVC（4：4：：4：： 2：0）以及对PSNR和MS-SSIM指标的一些基于学习的方法。

我们研究如何代表具有隐式神经表示（INRS）的视频。经典INRS方法通常利用MLP将输入坐标映射到输出像素。尽管最近的一些作品试图直接使用CNN重建整个图像。但是，我们认为，以上像素和图像策略都不利于视频数据。取而代之的是，我们提出了一个贴片解决方案PS-NERV，该解决方案将视频表示为贴片的函数和相应的补丁坐标。它自然继承了图像方法的优势，并以快速解码速度实现出色的重建性能。整个方法包括常规模块，例如位置嵌入，MLP和CNN，同时还引入了ADAIN以增强中间特征。这些简单而基本的更改可以帮助网络轻松拟合高频细节。广泛的实验证明了其在几个与视频有关的任务中的有效性，例如视频压缩和视频介绍。

标量和矢量场的神经近似（例如签名距离函数和辐射场）已成为准确的高质量表示。最先进的结果是通过从可训练的特征网格中进行查找的调节来获得的，这些近似是按照学习任务的一部分，并允许较小，更有效的神经网络。不幸的是，与独立的神经网络模型相比，这些特征网格通常以明显增加的记忆消耗成本。我们提出了一种词典方法，用于压缩此类特征网格，将其内存消耗降低至100倍，并允许多分辨率表示，这对于核心外流很有用。我们将词典优化作为矢量定量的自动码头问题提出，使我们能够在没有直接监督以及具有动态拓扑和结构的空间中学习端到端离散的神经表示。我们的源代码将在https://github.com/nv-tlabs/vqad上找到。

我们展示了如何使用变压器来大大简化神经视频压缩。以前的方法一直依赖越来越多的建筑偏见和先进的方法，包括运动预测和翘曲操作，从而产生复杂的模型。取而代之的是，我们独立地将输入帧映射到表示形式，并使用变压器对其依赖性进行建模，让它预测给定过去的未来表示的分布。最终的视频压缩变压器优于标准视频压缩数据集上的先前方法。合成数据的实验表明，我们的模型学会了处理复杂的运动模式，例如纯粹从数据中模糊和褪色。我们的方法易于实施，我们发布代码以促进未来的研究。