人类文明对地球系统具有越来越强大的影响,地球观察是评估和减轻负面影响的宝贵工具。为此,观察地球表面上精确定义的变化是必不可少的,我们提出了一种实现这一目标的有效方法。值得注意的是,我们的变更检测(CD)/分割方法提出了一种新颖的方式,以通过将不同的地球观察程序通过不同的扩散概率模型来纳入数百万个现成的,未标记的,未标记的,遥感的图像到训练过程中。我们首先通过使用预训练的denoding扩散概率模型,利用这些现成,未经贴贴和未标记的遥感图像的信息,然后采用来自扩散模型解码器的多尺度特征表示来训练轻量级CD分类器检测精确的更改。在四个公开可用的CD数据集上执行的实验表明,所提出的方法比F1,IOU和总体准确性中的最新方法取得了更好的结果。代码和预培训模型可在以下网址找到:https://github.com/wgcban/ddpm-cd
translated by 谷歌翻译
Semantic segmentation from aerial views is a vital task for autonomous drones as they require precise and accurate segmentation to traverse safely and efficiently. Segmenting images from aerial views is especially challenging as they include diverse view-points, extreme scale variation and high scene complexity. To address this problem, we propose an end-to-end multi-class semantic segmentation diffusion model. We introduce recursive denoising which allows predicted error to propagate through the denoising process. In addition, we combine this with a hierarchical multi-scale approach, complementary to the diffusion process. Our method achieves state-of-the-art results on UAVid and on the Vaihingen building segmentation benchmark.
translated by 谷歌翻译
Human civilization has an increasingly powerful influence on the earth system. Affected by climate change and land-use change, natural disasters such as flooding have been increasing in recent years. Earth observations are an invaluable source for assessing and mitigating negative impacts. Detecting changes from Earth observation data is one way to monitor the possible impact. Effective and reliable Change Detection (CD) methods can help in identifying the risk of disaster events at an early stage. In this work, we propose a novel unsupervised CD method on time series Synthetic Aperture Radar~(SAR) data. Our proposed method is a probabilistic model trained with unsupervised learning techniques, reconstruction, and contrastive learning. The change map is generated with the help of the distribution difference between pre-incident and post-incident data. Our proposed CD model is evaluated on flood detection data. We verified the efficacy of our model on 8 different flood sites, including three recent flood events from Copernicus Emergency Management Services and six from the Sen1Floods11 dataset. Our proposed model achieved an average of 64.53\% Intersection Over Union(IoU) value and 75.43\% F1 score. Our achieved IoU score is approximately 6-27\% and F1 score is approximately 7-22\% better than the compared unsupervised and supervised existing CD methods. The results and extensive discussion presented in the study show the effectiveness of the proposed unsupervised CD method.
translated by 谷歌翻译
本文介绍了一种基于变压器的暹罗网络架构(由Cradiformer缩写),用于从一对共同登记的遥感图像改变检测(CD)。与最近的CD框架不同,该CD框架基于完全卷积的网络(CoundNets),该方法将具有多层感知(MLP)解码器的分层结构化变压器编码器统一,以暹罗网络架构中的多层感知器,以有效地呈现所需的多尺度远程详细信息用于准确的CD。两个CD数据集上的实验表明,所提出的端到端培训变换器架构比以前的同行实现更好的CD性能。我们的代码可在https://github.com/wgcban/changeFormer获得。
translated by 谷歌翻译
扩散模型是一类深入生成模型,在具有密集理论建立的各种任务上显示出令人印象深刻的结果。尽管与其他最先进的模型相比,扩散模型的样本合成质量和多样性令人印象深刻,但它们仍然遭受了昂贵的抽样程序和次优可能的估计。最近的研究表明,对提高扩散模型的性能的热情非常热情。在本文中,我们对扩散模型的现有变体进行了首次全面综述。具体而言,我们提供了扩散模型的第一个分类法,并将它们分类为三种类型,即采样加速增强,可能性最大化的增强和数据将来增强。我们还详细介绍了其他五个生成模型(即变异自动编码器,生成对抗网络,正常流量,自动回归模型和基于能量的模型),并阐明扩散模型与这些生成模型之间的连接。然后,我们对扩散模型的应用进行彻底研究,包括计算机视觉,自然语言处理,波形信号处理,多模式建模,分子图生成,时间序列建模和对抗性纯化。此外,我们提出了与这种生成模型的发展有关的新观点。
translated by 谷歌翻译
建筑变更检测是许多重要应用,特别是在军事和危机管理领域。最近用于变化检测的方法已转向深度学习,这取决于其培训数据的质量。因此,大型注释卫星图像数据集的组装对于全球建筑更改监视是必不可少的。现有数据集几乎完全提供近Nadir观看角度。这限制了可以检测到的更改范围。通过提供更大的观察范围,光学卫星的滚动成像模式提出了克服这种限制的机会。因此,本文介绍了S2Looking,一个建筑变革检测数据集,其中包含以各种偏离Nadir角度捕获的大规模侧视卫星图像。 DataSet由5000个批次图像对组成的农村地区,并在全球范围内超过65,920个辅助的变化实例。数据集可用于培训基于深度学习的变更检测算法。它通过提供(1)更大的观察角来扩展现有数据集; (2)大照明差异; (3)额外的农村形象复杂性。为了便于{该数据集的使用,已经建立了基准任务,并且初步测试表明,深度学习算法发现数据集明显比最接近的近Nadir DataSet,Levir-CD +更具挑战性。因此,S2Looking可能会促进现有的建筑变革检测算法的重要进步。 DataSet可在https://github.com/s2looking/使用。
translated by 谷歌翻译
Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models.
translated by 谷歌翻译
与生成的对抗网(GAN)相比,降级扩散概率模型(DDPM)在各种图像生成任务中取得了显着成功。关于语义图像综合的最新工作主要遵循\ emph {de exto}基于gan的方法,这可能导致生成图像的质量或多样性不令人满意。在本文中,我们提出了一个基于DDPM的新型框架,用于语义图像合成。与先前的条件扩散模型不同,将语义布局和嘈杂的图像作为输入为U-NET结构,该结构可能无法完全利用输入语义掩码中的信息,我们的框架处理语义布局和嘈杂的图像不同。它将噪声图像馈送到U-NET结构的编码器时,而语义布局通过多层空间自适应归一化操作符将语义布局馈送到解码器。为了进一步提高语义图像合成中的发电质量和语义解释性,我们介绍了无分类器的指导采样策略,该策略承认采样过程的无条件模型的得分。在三个基准数据集上进行的广泛实验证明了我们提出的方法的有效性,从而在忠诚度(FID)和多样性〜(LPIPS)方面实现了最先进的性能。
translated by 谷歌翻译
监督的深度学习模型取决于大量标记的数据。不幸的是,收集和注释包含所需更改的零花态样本是耗时和劳动密集型的。从预训练模型中转移学习可有效减轻遥感(RS)变化检测(CD)中标签不足。我们探索在预训练期间使用语义信息的使用。不同于传统的监督预训练,该预训练从图像到标签,我们将语义监督纳入了自我监督的学习(SSL)框架中。通常,多个感兴趣的对象(例如,建筑物)以未经切割的RS图像分布在各个位置。我们没有通过全局池操纵图像级表示,而是在每个像素嵌入式上引入点级监督以学习空间敏感的特征,从而使下游密集的CD受益。为了实现这一目标,我们通过使用语义掩码在视图之间的重叠区域上通过类平衡的采样获得了多个点。我们学会了一个嵌入式空间,将背景和前景点分开,并将视图之间的空间对齐点齐聚在一起。我们的直觉是导致的语义歧视性表示与无关的变化不变(照明和无关紧要的土地覆盖)可能有助于改变识别。我们在RS社区中免费提供大规模的图像面罩,用于预训练。在三个CD数据集上进行的大量实验验证了我们方法的有效性。我们的表现明显优于Imagenet预训练,内域监督和几种SSL方法。经验结果表明我们的预训练提高了CD模型的概括和数据效率。值得注意的是,我们使用20%的培训数据获得了比基线(随机初始化)使用100%数据获得竞争结果。我们的代码可用。
translated by 谷歌翻译
Change detection (CD) aims to find the difference between two images at different times and outputs a change map to represent whether the region has changed or not. To achieve a better result in generating the change map, many State-of-The-Art (SoTA) methods design a deep learning model that has a powerful discriminative ability. However, these methods still get lower performance because they ignore spatial information and scaling changes between objects, giving rise to blurry or wrong boundaries. In addition to these, they also neglect the interactive information of two different images. To alleviate these problems, we propose our network, the Scale and Relation-Aware Siamese Network (SARAS-Net) to deal with this issue. In this paper, three modules are proposed that include relation-aware, scale-aware, and cross-transformer to tackle the problem of scene change detection more effectively. To verify our model, we tested three public datasets, including LEVIR-CD, WHU-CD, and DSFIN, and obtained SoTA accuracy. Our code is available at https://github.com/f64051041/SARAS-Net.
translated by 谷歌翻译
病理学家对患病组织的视觉微观研究一直是一个多世纪以来癌症诊断和预后的基石。最近,深度学习方法在组织图像的分析和分类方面取得了重大进步。但是,关于此类模型在生成组织病理学图像的实用性方面的工作有限。这些合成图像在病理学中有多种应用,包括教育,熟练程度测试,隐私和数据共享的公用事业。最近,引入了扩散概率模型以生成高质量的图像。在这里,我们首次研究了此类模型的潜在用途以及优先的形态加权和颜色归一化,以合成脑癌的高质量组织病理学图像。我们的详细结果表明,与生成对抗网络相比,扩散概率模型能够合成各种组织病理学图像,并且具有较高的性能。
translated by 谷歌翻译
在不利天气条件下的图像恢复对各种计算机视觉应用引起了重大兴趣。最近的成功方法取决于深度神经网络架构设计(例如,具有视觉变压器)的当前进展。由最新的条件生成模型取得的最新进展的动机,我们提出了一种基于贴片的图像恢复算法,基于脱氧扩散概率模型。我们的基于贴片的扩散建模方法可以通过使用指导的DeNoising过程进行尺寸 - 不足的图像恢复,并在推理过程中对重叠贴片进行平滑的噪声估计。我们在基准数据集上经验评估了我们的模型,以进行图像,混合的降低和飞行以及去除雨滴的去除。我们展示了我们在特定天气和多天气图像恢复上实现最先进的表演的方法,并在质量上表现出对现实世界测试图像的强烈概括。
translated by 谷歌翻译
Diffusion-based generative models have achieved remarkable success in image generation. Their guidance formulation allows an external model to plug-and-play control the generation process for various tasks without fine-tuning the diffusion model. However, the direct use of publicly available off-the-shelf models for guidance fails due to their poor performance on noisy inputs. For that, the existing practice is to fine-tune the guidance models with labeled data corrupted with noises. In this paper, we argue that this practice has limitations in two aspects: (1) performing on inputs with extremely various noises is too hard for a single model; (2) collecting labeled datasets hinders scaling up for various tasks. To tackle the limitations, we propose a novel strategy that leverages multiple experts where each expert is specialized in a particular noise range and guides the reverse process at its corresponding timesteps. However, as it is infeasible to manage multiple networks and utilize labeled data, we present a practical guidance framework termed Practical Plug-And-Play (PPAP), which leverages parameter-efficient fine-tuning and data-free knowledge transfer. We exhaustively conduct ImageNet class conditional generation experiments to show that our method can successfully guide diffusion with small trainable parameters and no labeled data. Finally, we show that image classifiers, depth estimators, and semantic segmentation models can guide publicly available GLIDE through our framework in a plug-and-play manner.
translated by 谷歌翻译
There has been a recent explosion of impressive generative models that can produce high quality images (or videos) conditioned on text descriptions. However, all such approaches rely on conditional sentences that contain unambiguous descriptions of scenes and main actors in them. Therefore employing such models for more complex task of story visualization, where naturally references and co-references exist, and one requires to reason about when to maintain consistency of actors and backgrounds across frames/scenes, and when not to, based on story progression, remains a challenge. In this work, we address the aforementioned challenges and propose a novel autoregressive diffusion-based framework with a visual memory module that implicitly captures the actor and background context across the generated frames. Sentence-conditioned soft attention over the memories enables effective reference resolution and learns to maintain scene and actor consistency when needed. To validate the effectiveness of our approach, we extend the MUGEN dataset and introduce additional characters, backgrounds and referencing in multi-sentence storylines. Our experiments for story generation on the MUGEN, the PororoSV and the FlintstonesSV dataset show that our method not only outperforms prior state-of-the-art in generating frames with high visual quality, which are consistent with the story, but also models appropriate correspondences between the characters and the background.
translated by 谷歌翻译
使用遥感图像进行建筑检测和变更检测可以帮助城市和救援计划。此外,它们可用于自然灾害后的建筑损害评估。当前,大多数用于建筑物检测的现有模型仅使用一个图像(预拆架图像)来检测建筑物。这是基于这样的想法:由于存在被破坏的建筑物,后沙仪图像降低了模型的性能。在本文中,我们提出了一种称为暹罗形式的暹罗模型,该模型使用前和垃圾后图像作为输入。我们的模型有两个编码器,并具有分层变压器体系结构。两个编码器中每个阶段的输出都以特征融合的方式给予特征融合,以从disasaster图像生成查询,并且(键,值)是从disasaster图像中生成的。为此,在特征融合中也考虑了时间特征。在特征融合中使用颞变压器的另一个优点是,与CNN相比,它们可以更好地维持由变压器编码器产生的大型接受场。最后,在每个阶段,将颞变压器的输出输入简单的MLP解码器。在XBD和WHU数据集上评估了暹罗形式模型,用于构建检测以及Levir-CD和CDD数据集,以进行更改检测,并可以胜过最新的。
translated by 谷歌翻译
Denoising diffusion (score-based) generative models have recently achieved significant accomplishments in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data into noise and a backward denoising process for sampling data from noise. Unfortunately, the generation process of current denoising diffusion models is notoriously slow due to the lengthy iterative noise estimations, which rely on cumbersome neural networks. It prevents the diffusion models from being widely deployed, especially on edge devices. Previous works accelerate the generation process of diffusion model (DM) via finding shorter yet effective sampling trajectories. However, they overlook the cost of noise estimation with a heavy network in every iteration. In this work, we accelerate generation from the perspective of compressing the noise estimation network. Due to the difficulty of retraining DMs, we exclude mainstream training-aware compression paradigms and introduce post-training quantization (PTQ) into DM acceleration. However, the output distributions of noise estimation networks change with time-step, making previous PTQ methods fail in DMs since they are designed for single-time step scenarios. To devise a DM-specific PTQ method, we explore PTQ on DM in three aspects: quantized operations, calibration dataset, and calibration metric. We summarize and use several observations derived from all-inclusive investigations to formulate our method, which especially targets the unique multi-time-step structure of DMs. Experimentally, our method can directly quantize full-precision DMs into 8-bit models while maintaining or even improving their performance in a training-free manner. Importantly, our method can serve as a plug-and-play module on other fast-sampling methods, e.g., DDIM.
translated by 谷歌翻译
作为生成部件作为自回归模型的向量量化变形式自动化器(VQ-VAE)的集成在图像生成上产生了高质量的结果。但是,自回归模型将严格遵循采样阶段的逐步扫描顺序。这导致现有的VQ系列模型几乎不会逃避缺乏全球信息的陷阱。连续域中的去噪扩散概率模型(DDPM)显示了捕获全局背景的能力,同时产生高质量图像。在离散状态空间中,一些作品已经证明了执行文本生成和低分辨率图像生成的可能性。我们认为,在VQ-VAE的富含内容的离散视觉码本的帮助下,离散扩散模型还可以利用全局上下文产生高保真图像,这补偿了沿像素空间的经典自回归模型的缺陷。同时,离散VAE与扩散模型的集成解决了传统的自回归模型的缺点是超大的,以及在生成图像时需要在采样过程中的过度时间的扩散模型。结果发现所生成的图像的质量严重依赖于离散的视觉码本。广泛的实验表明,所提出的矢量量化离散扩散模型(VQ-DDM)能够实现与低复杂性的顶层方法的相当性能。它还展示了在没有额外培训的图像修复任务方面与自回归模型量化的其他矢量突出的优势。
translated by 谷歌翻译
DeNoising扩散模型代表了计算机视觉中最新的主题,在生成建模领域表现出了显着的结果。扩散模型是一个基于两个阶段的深层生成模型,一个正向扩散阶段和反向扩散阶段。在正向扩散阶段,通过添加高斯噪声,输入数据在几个步骤中逐渐受到干扰。在反向阶段,模型的任务是通过学习逐步逆转扩散过程来恢复原始输入数据。尽管已知的计算负担,即由于采样过程中涉及的步骤数量,扩散模型对生成样品的质量和多样性得到了广泛赞赏。在这项调查中,我们对视觉中应用的denoising扩散模型的文章进行了全面综述,包括该领域的理论和实际贡献。首先,我们识别并介绍了三个通用扩散建模框架,这些框架基于扩散概率模型,噪声调节得分网络和随机微分方程。我们进一步讨论了扩散模型与其他深层生成模型之间的关系,包括变异自动编码器,生成对抗网络,基于能量的模型,自回归模型和正常流量。然后,我们介绍了计算机视觉中应用的扩散模型的多角度分类。最后,我们说明了扩散模型的当前局限性,并设想了一些有趣的未来研究方向。
translated by 谷歌翻译
遥感图像的更改检测(CD)是通过分析两个次时图像之间的差异来检测变化区域。它广泛用于土地资源规划,自然危害监测和其他领域。在我们的研究中,我们提出了一个新型的暹罗神经网络,用于变化检测任务,即双UNET。与以前的单独编码BITEMAL图像相反,我们设计了一个编码器差分注意模块,以关注像素的空间差异关系。为了改善网络的概括,它计算了咬合图像之间的任何像素之间的注意力权重,并使用它们来引起更具区别的特征。为了改善特征融合并避免梯度消失,在解码阶段提出了多尺度加权方差图融合策略。实验表明,所提出的方法始终优于流行的季节性变化检测数据集最先进的方法。
translated by 谷歌翻译
In recent years, generative models have undergone significant advancement due to the success of diffusion models. The success of these models is often attributed to their use of guidance techniques, such as classifier and classifier-free methods, which provides effective mechanisms to trade-off between fidelity and diversity. However, these methods are not capable of guiding a generated image to be aware of its geometric configuration, e.g., depth, which hinders the application of diffusion models to areas that require a certain level of depth awareness. To address this limitation, we propose a novel guidance approach for diffusion models that uses estimated depth information derived from the rich intermediate representations of diffusion models. To do this, we first present a label-efficient depth estimation framework using the internal representations of diffusion models. At the sampling phase, we utilize two guidance techniques to self-condition the generated image using the estimated depth map, the first of which uses pseudo-labeling, and the subsequent one uses a depth-domain diffusion prior. Experiments and extensive ablation studies demonstrate the effectiveness of our method in guiding the diffusion models toward geometrically plausible image generation. Project page is available at https://ku-cvlab.github.io/DAG/.
translated by 谷歌翻译