Weakly Supervised Airway Orifice Segmentation in Video Bronchoscopy

Video Bronchoscopy (VB) is commonly applied in conjunction with lung diseases. It is a fundamental procedure for diagnosis of lung cancer enabling biopsy of deep airway tissue. In addition to that, VB is routinely conducted for monitoring Chronic Obstructive Pulmonary Disease (COPD) patients and clarification of acute respiratory problems at Intensive Care Units (ICU). The navigation within the bronchial tree is challenging and physically demanding for physicians due to homogenous textures and perceptually similar appearance of bronchial orifices. This is particularly the case in the absence of prior CT scans and Electromagnetic Tracking (EMT) systems at ICUs. Airway orifice segmentation which is the main objective of this paper enables image-based guidance, e.g. by providing graphical overlays on top of the VB images. In conjunction with EMT or image-based tracking[8] these overlays can be accompanied by airway labels w.r.t. generic or interpatient lung models which however is not addressed by the approach presented in this paper. The variety of tissue appearance, illumination, image artifacts, secrete and patient anatomy poses a challenge for airway segmentation which we aim to address by deep learning-based approaches. However, this is currently hampered due to a lack of readily available ground truth labels motivating the incorporation of traditional (non-learning-based) methods as weak supervision. In particular, we incorporate an airway phantom dataset collection accompanied by ground truth depth images to generate airway orifice labels for training a deep learning-based segmentation model. Our proposed methods are also developed with a focus on their complexity and runtime, keeping them real-time capable even on low-end devices for intervention guidance.

The qualitative results in Fig.2 demonstrate that the model trained on the phantom dataset was able to extract a semantic knowledge about airways from the noisy ground truth generated based on depth images using our proposed data-driven pipeline. In some cases, it even outperforms this noisy ground truth, particularly when airways were not detected properly due to their low depth profiles (see Fig.1(a)). This happens if an orifice belongs to an airway, that has a high angle to the camera perspective, and therefore only its wall remains visible. The data augmentation method [1] was able to close the domain gap from the phantom to the in-vivo datasets, when comparing the model’s performance to the models being directly trained on the in-vivo data. Even cases without any airways present were correctly predicted, even though such situations were not directly covered by the ground truth during training (see Fig.1(b)). Having those results in mind, our training can be considered as successful for the real world use case.

Our quantitative results shown in Tab.1 have only limited significance for the Phantom dataset due to lack of real ground truth, which is compensated by our noisy generated one. However, the ground truth by the in-vivo dataset was created by human experts and is, beside the very high observer variability, reliable. The model trained on the Phantom dataset shows a great domain robustness, considering the resulting DSC. The domain gap seems larger coming from the in-vivo domain on the first sight. But this is a perfect example for our AMCD ${\overline{d}}_c$ and its motivation; even if the DSC on the phantom data is twice as low as with the model trained on the phantom dataset, the AMCD is in the very same range, going hand in hand with the correct prediction visible in the qualitative results. This is due to the observer variability of the ground truth segmentation, resulting in different diameters of the segmentations depending on the observer.

Considering the quantitative but especially the qualitative results, the training on the Phantom data with the noisy ground truth as weak supervision enables the learning of a semantic knowledge of airways. We had also shown that this knowledge can be transferred to in-vivo data while reaching comparable performance as the models directly trained on these datasets.

This paper presents a novel approach for weakly-supervised CNN-based airway orifice segmentation in video bronchoscopy which is trained on phantom and evaluated on in-vivo patient data. To our best knowledge, this is the first paper presenting a deep learning-based approach omitting the use of depth images for inference.

In this work, we presented a very real-time capable pipeline that extracts airway segmentations from a given bronchoscopy depth image using efficient data-driven classical methods. However, this method has some disadvantages: On the one hand it requires a depth image, which is not native given by the endoscope due to hardware limitation and therefore has to be generated via a complex non-linear domain translation like a Generative Adversarial Networks (GAN). On the other hand, a data-driven approach is not equal to a semantic understanding. Considering this and due to the lack of robustness to some edge cases, we consider this pipeline alone as not suitable for real world applications. However, paired with the RGB image of the bronchoscopy, these generated segmentation maps can be used as weak supervision during training of a shallow CNN for airway orifice segmentation. We showed that this model being trained on phantom data gains tremendous semantic knowledge of airway structures overcoming noisy ground truth on edge cases and is even applicable directly to in-vivo VB due thanks to a substantial data augmentation. With all of this, our proposed method allows the generation of segmentation masks directly on RGB images without the need of hand annotated datasets. We argue that this direct prediction from RGB images is superior to the segmentation approaches on the depth images, because it comes without the risks of a domain translation from RGB to depth via GANs, which is mainly based on unsupervised manner of the GAN training, likely to cause the generation of wrong anatomies[6] like additional or absent airway branches in the synthesized depth images.