诸如DALL-E 2之类的生成模型可以代表放射学中人工智能研究的图像生成，增强和操纵的有希望的未来工具，前提是这些模型具有足够的医疗领域知识。在这里，我们证明DALL-E 2在零拍的文本到图像生成方面，学习了具有有希望的功能的X射线图像的相关表示，将图像的延续超出其原始边界或删除元素，尽管病理产生或CT，MRI和超声图像仍然受到限制。因此，即使事先需要对这些模型进行进一步的微调和适应，也需要使用生成模型来增强和生成放射学数据似乎是可行的。

胸部计算机断层扫描（CT）成像为肺部传染病（如结核病（TB））的诊断和管理增添了宝贵的见解。但是，由于成本和资源的限制，只有X射线图像可用于初步诊断或在治疗过程中进行后续比较成像。由于其投影性，X射线图像可能更难解释临床医生。缺乏公开配对的X射线和CT图像数据集使训练3D重建模型的挑战。此外，胸部X射线放射学可能依赖具有不同图像质量的不同设备方式，并且潜在的种群疾病谱可能会在输入中产生多样性。我们提出了形状诱导，也就是说，在没有CT监督的情况下从X射线中学习3D CT的形状，作为一种新型技术，可以在训练重建模型的训练过程中结合现实的X射线分布。我们的实验表明，这一过程既提高了产生的CT的感知质量，也可以提高肺传染病的下游分类的准确性。

Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models in particular have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen and Stable Diffusion. However, their use in medicine, where image data typically comprises three-dimensional volumes, has not been systematically evaluated. Synthetic images may play a crucial role in privacy preserving artificial intelligence and can also be used to augment small datasets. Here we show that diffusion probabilistic models can synthesize high quality medical imaging data, which we show for Magnetic Resonance Images (MRI) and Computed Tomography (CT) images. We provide quantitative measurements of their performance through a reader study with two medical experts who rated the quality of the synthesized images in three categories: Realistic image appearance, anatomical correctness and consistency between slices. Furthermore, we demonstrate that synthetic images can be used in a self-supervised pre-training and improve the performance of breast segmentation models when data is scarce (dice score 0.91 vs. 0.95 without vs. with synthetic data).

The release of ChatGPT, a language model capable of generating text that appears human-like and authentic, has gained significant attention beyond the research community. We expect that the convincing performance of ChatGPT incentivizes users to apply it to a variety of downstream tasks, including prompting the model to simplify their own medical reports. To investigate this phenomenon, we conducted an exploratory case study. In a questionnaire, we asked 15 radiologists to assess the quality of radiology reports simplified by ChatGPT. Most radiologists agreed that the simplified reports were factually correct, complete, and not potentially harmful to the patient. Nevertheless, instances of incorrect statements, missed key medical findings, and potentially harmful passages were reported. While further studies are needed, the initial insights of this study indicate a great potential in using large language models like ChatGPT to improve patient-centered care in radiology and other medical domains.

大型文本对图像模型在AI的演变中取得了显着的飞跃，从而使图像从给定的文本提示中实现了高质量和多样化的图像合成。但是，这些模型缺乏在给定的参考集中模仿受试者的外观，并在不同情况下合成它们的新颖性。在这项工作中，我们提出了一种新的方法，用于“个性化”文本图像扩散模型（将它们专门针对用户的需求）。仅作为一个主题的几张图像给出，我们将验证的文本对图像模型（图像，尽管我们的方法不限于特定模型），以便它学会了将唯一标识符与该特定主题结合。一旦将受试者嵌入模型的输出域中，就可以使用唯一标识符来合成主题的完全新颖的光真逼真的图像在不同场景中的上下文化。通过利用具有新的自动构基特异性的先前保存损失的语义先验嵌入到模型中，我们的技术可以在参考图像中未出现的不同场景，姿势，视图和照明条件中合成主题。我们将技术应用于几个以前无用的任务，包括主题重新定义，文本指导的视图合成，外观修改和艺术渲染（所有这些都保留了主题的关键特征）。项目页面：https：//dreambooth.github.io/

We introduce M-VADER: a diffusion model (DM) for image generation where the output can be specified using arbitrary combinations of images and text. We show how M-VADER enables the generation of images specified using combinations of image and text, and combinations of multiple images. Previously, a number of successful DM image generation algorithms have been introduced that make it possible to specify the output image using a text prompt. Inspired by the success of those models, and led by the notion that language was already developed to describe the elements of visual contexts that humans find most important, we introduce an embedding model closely related to a vision-language model. Specifically, we introduce the embedding model S-MAGMA: a 13 billion parameter multimodal decoder combining components from an autoregressive vision-language model MAGMA and biases finetuned for semantic search.

The text-to-image model Stable Diffusion has recently become very popular. Only weeks after its open source release, millions are experimenting with image generation. This is due to its ease of use, since all it takes is a brief description of the desired image to "prompt" the generative model. Rarely do the images generated for a new prompt immediately meet the user's expectations. Usually, an iterative refinement of the prompt ("prompt engineering") is necessary for satisfying images. As a new perspective, we recast image prompt engineering as interactive image retrieval - on an "infinite index". Thereby, a prompt corresponds to a query and prompt engineering to query refinement. Selected image-prompt pairs allow direct relevance feedback, as the model can modify an image for the refined prompt. This is a form of one-sided interactive retrieval, where the initiative is on the user side, whereas the server side remains stateless. In light of an extensive literature review, we develop these parallels in detail and apply the findings to a case study of a creative search task on such a model. We note that the uncertainty in searching an infinite index is virtually never-ending. We also discuss future research opportunities related to retrieval models specialized for generative models and interactive generative image retrieval. The application of IR technology, such as query reformulation and relevance feedback, will contribute to improved workflows when using generative models, while the notion of an infinite index raises new challenges in IR research.

数据的多样性对于对深度学习模型的成功培训至关重要。通过经常发生的生成对抗网络杠杆，我们提出了在胸部CT扫描的小型数据集上培训时产生大规模3D合成CT-SCAN卷（$ \ GEQ224 \ Times224 $）的CT-SGAG。CT-SGAN为医学成像机器学习面临的两个主要挑战提供有吸引力的解决方案：少数给定的I.I.D。培训数据，以及对患者数据共享的限制，防止迅速获得更大和更多样化的数据集。我们使用包括FR \'电位距离和成立分数的各种度量来评估生成的图像的保真度和定量。我们进一步表明，通过在大量合成数据上预先训练分类器，CT-SGAN可以显着提高肺结核检测精度。

数字艺术合成在多媒体社区中受到越来越多的关注，因为有效地与公众参与了艺术。当前的数字艺术合成方法通常使用单模式输入作为指导，从而限制了模型的表现力和生成结果的多样性。为了解决这个问题，我们提出了多模式引导的艺术品扩散（MGAD）模型，该模型是一种基于扩散的数字艺术品生成方法，它利用多模式提示作为控制无分类器扩散模型的指导。此外，对比度语言图像预处理（剪辑）模型用于统一文本和图像模式。关于生成的数字艺术绘画质量和数量的广泛实验结果证实了扩散模型和多模式指导的组合有效性。代码可从https://github.com/haha-lisa/mgad-multimodal-guided-artwork-diffusion获得。

Recent large-scale image generation models such as Stable Diffusion have exhibited an impressive ability to generate fairly realistic images starting from a very simple text prompt. Could such models render real images obsolete for training image prediction models? In this paper, we answer part of this provocative question by questioning the need for real images when training models for ImageNet classification. More precisely, provided only with the class names that have been used to build the dataset, we explore the ability of Stable Diffusion to generate synthetic clones of ImageNet and measure how useful they are for training classification models from scratch. We show that with minimal and class-agnostic prompt engineering those ImageNet clones we denote as ImageNet-SD are able to close a large part of the gap between models produced by synthetic images and models trained with real images for the several standard classification benchmarks that we consider in this study. More importantly, we show that models trained on synthetic images exhibit strong generalization properties and perform on par with models trained on real data.

每年医生对患者的基于形象的诊断需求越来越大，是最近的人工智能方法可以解决的问题。在这种情况下，我们在医学图像的自动报告领域进行了调查，重点是使用深神经网络的方法，了解：（1）数据集，（2）架构设计，（3）解释性和（4）评估指标。我们的调查确定了有趣的发展，也是留下挑战。其中，目前对生成的报告的评估尤为薄弱，因为它主要依赖于传统的自然语言处理（NLP）指标，这不准确地捕获医疗正确性。

The success of Deep Learning applications critically depends on the quality and scale of the underlying training data. Generative adversarial networks (GANs) can generate arbitrary large datasets, but diversity and fidelity are limited, which has recently been addressed by denoising diffusion probabilistic models (DDPMs) whose superiority has been demonstrated on natural images. In this study, we propose Medfusion, a conditional latent DDPM for medical images. We compare our DDPM-based model against GAN-based models, which constitute the current state-of-the-art in the medical domain. Medfusion was trained and compared with (i) StyleGan-3 on n=101,442 images from the AIROGS challenge dataset to generate fundoscopies with and without glaucoma, (ii) ProGAN on n=191,027 from the CheXpert dataset to generate radiographs with and without cardiomegaly and (iii) wGAN on n=19,557 images from the CRCMS dataset to generate histopathological images with and without microsatellite stability. In the AIROGS, CRMCS, and CheXpert datasets, Medfusion achieved lower (=better) FID than the GANs (11.63 versus 20.43, 30.03 versus 49.26, and 17.28 versus 84.31). Also, fidelity (precision) and diversity (recall) were higher (=better) for Medfusion in all three datasets. Our study shows that DDPM are a superior alternative to GANs for image synthesis in the medical domain.

现在，人工智能（AI）可以自动解释医学图像以供临床使用。但是，AI在介入图像中的潜在用途（相对于参与分类或诊断的图像），例如在手术期间的指导，在很大程度上尚未开发。这是因为目前，使用现场分析对现场手术收集的数据进行了事后分析，这是因为手术AI系统具有基本和实际限制，包括道德考虑，费用，可扩展性，数据完整性以及缺乏地面真相。在这里，我们证明从人类模型中创建逼真的模拟图像是可行的替代方法，并与大规模的原位数据收集进行了补充。我们表明，对现实合成数据的训练AI图像分析模型，结合当代域的概括或适应技术，导致在实际数据上的模型与在精确匹配的真实数据训练集中训练的模型相当地执行的模型。由于从基于人类的模型尺度的合成生成培训数据，因此我们发现我们称为X射线图像分析的模型传输范式（我们称为Syntheex）甚至可以超越实际数据训练的模型，因为训练的有效性较大的数据集。我们证明了合成在三个临床任务上的潜力：髋关节图像分析，手术机器人工具检测和COVID-19肺病变分割。 Synthex提供了一个机会，可以极大地加速基于X射线药物的智能系统的概念，设计和评估。此外，模拟图像环境还提供了测试新颖仪器，设计互补手术方法的机会，并设想了改善结果，节省时间或减轻人为错误的新技术，从实时人类数据收集的道德和实际考虑方面摆脱了人为错误。

文本指导的图像生成模型，例如DALL-E 2和稳定的扩散，最近受到了学术界和公众的关注。这些模型提供了文本描述，能够生成描绘各种概念和样式的高质量图像。但是，此类模型接受了大量公共数据的培训，并从其培训数据中隐含地学习关系，这些数据并不明显。我们证明，可以通过简单地用视觉上类似的非拉丁字符替换文本描述中的单个字符来触发并注入生成的图像中的常见多模型模型，这些偏见可以被触发并注入生成的图像。这些所谓的同符文更换使恶意用户或服务提供商能够诱导偏见到生成的图像中，甚至使整个一代流程变得无用。我们实际上说明了对DALL-E 2和稳定扩散的这种攻击，例如文本引导的图像生成模型，并进一步表明夹子的行为也相似。我们的结果进一步表明，经过多语言数据训练的文本编码器提供了一种减轻同符替代效果的方法。

综合正电子发射断层扫描/磁共振成像（PET/MRI）扫描仪通过PET和形态信息促进了同时获得代谢信息，并使用MRI进行了高软组织对比度。尽管PET/MRI促进了捕获高精度融合图像，但其主要缺点可以归因于进行衰减校正时遇到的困难，这对于定量PET评估是必不可少的。合并后的宠物/MRI扫描需要从MRI中产生衰减 - 校正图，这是由于伽马射线衰减信息与MRI之间没有直接关系。尽管可以轻松地为头部和骨盆区域执行基于MRI的骨组织分割，但通过胸部CT生成来实现准确的骨骼分割仍然是一项艰巨的任务。这可以归因于胸部发生的呼吸和心脏运动，以及其解剖学上复杂的结构和相对较薄的骨皮质。本文提出了一种方法，可以通过使用独立于模态的邻域描述符（思维）添加结构性约束，从而最大程度地减少解剖结构变化，而无需人类注释，从而将结构性变化（MID）添加到可以转换不配对图像的生成对抗网络（GAN）中。在这项研究中获得的结果揭示了拟议的U-Gat-It +思维方法，以优于所有其他竞争方法。这项研究的发现暗示了可能在没有人类注释的情况下从胸部MRI中合成临床上可接受的CT图像的可能性，从而最大程度地减少了解剖结构的变化。

X射线成像是最受欢迎的医学成像技术。虽然X射线射线造影相当成本效益，但组织结构沿X射线路径叠加。另一方面，计算断层扫描（CT）重建内部结构，但CT增加辐射剂量，复杂且昂贵。在这里，我们提出了“X射线分析缩放”，以在深度学习框架中以少量的射线照相投影来分化以数字的靶器官/组织提取靶器官/组织。作为示例性实施例，我们提出了一般的X射线分解网络，专用的X射线绝地形网络和X射线成像系统以实现这些功能。我们的实验表明，在这种情况下，可以实现X射线立体术中孤立的器官，如这种情况下，表明将常规放射线读数转化为孤立器官的立体检查的可行性，这可能允许更高的敏感性和特异性，甚至目标的断层可视化。随着进一步的改进，X射线分解缩放有望成为辐射剂量和系统成本的CT级诊断的新X射线成像模型，与射线照相或造影术成像相当。

The availability of large-scale chest X-ray datasets is a requirement for developing well-performing deep learning-based algorithms in thoracic abnormality detection and classification. However, biometric identifiers in chest radiographs hinder the public sharing of such data for research purposes due to the risk of patient re-identification. To counteract this issue, synthetic data generation offers a solution for anonymizing medical images. This work employs a latent diffusion model to synthesize an anonymous chest X-ray dataset of high-quality class-conditional images. We propose a privacy-enhancing sampling strategy to ensure the non-transference of biometric information during the image generation process. The quality of the generated images and the feasibility of serving as exclusive training data are evaluated on a thoracic abnormality classification task. Compared to a real classifier, we achieve competitive results with a performance gap of only 3.5% in the area under the receiver operating characteristic curve.

Generating photos satisfying multiple constraints find broad utility in the content creation industry. A key hurdle to accomplishing this task is the need for paired data consisting of all modalities (i.e., constraints) and their corresponding output. Moreover, existing methods need retraining using paired data across all modalities to introduce a new condition. This paper proposes a solution to this problem based on denoising diffusion probabilistic models (DDPMs). Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Since each sampling step in the DDPM follows a Gaussian distribution, we show that there exists a closed-form solution for generating an image given various constraints. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task through our proposed sampling strategy. We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints. We perform experiments on various standard multimodal tasks to demonstrate the effectiveness of our approach. More details can be found in https://nithin-gk.github.io/projectpages/Multidiff/index.html

文本对图像模型提供了前所未有的自由，可以通过自然语言指导创作。然而，尚不清楚如何行使这种自由以生成特定独特概念，修改其外观或以新角色和新颖场景构成它们的图像。换句话说，我们问：我们如何使用语言指导的模型将猫变成绘画，或者想象基于我们喜欢的玩具的新产品？在这里，我们提出了一种简单的方法，可以允许这种创造性自由。我们仅使用3-5个用户提供的概念（例如对象或样式）的图像，我们学会通过在冷冻文本到图像模型的嵌入空间中通过新的“单词”表示它。这些“单词”可以组成自然语言句子，以直观的方式指导个性化的创作。值得注意的是，我们发现有证据表明单词嵌入足以捕获独特而多样的概念。我们将我们的方法比较了各种基线，并证明它可以更忠实地描绘出一系列应用程序和任务的概念。我们的代码，数据和新单词将在以下网址提供：https：//textual-inversion.github.io

Large, text-conditioned generative diffusion models have recently gained a lot of attention for their impressive performance in generating high-fidelity images from text alone. However, achieving high-quality results is almost unfeasible in a one-shot fashion. On the contrary, text-guided image generation involves the user making many slight changes to inputs in order to iteratively carve out the envisioned image. However, slight changes to the input prompt often lead to entirely different images being generated, and thus the control of the artist is limited in its granularity. To provide flexibility, we present the Stable Artist, an image editing approach enabling fine-grained control of the image generation process. The main component is semantic guidance (SEGA) which steers the diffusion process along variable numbers of semantic directions. This allows for subtle edits to images, changes in composition and style, as well as optimization of the overall artistic conception. Furthermore, SEGA enables probing of latent spaces to gain insights into the representation of concepts learned by the model, even complex ones such as 'carbon emission'. We demonstrate the Stable Artist on several tasks, showcasing high-quality image editing and composition.