Airway measurement by refinement of synthetic images improves mortality prediction in idiopathic pulmonary fibrosis

Deep learning frameworks have been applied to the measurement of airways in the lung in a bid to improve measurement accuracy. However, these machine learning methods are extremely data hungry and can be biased towards the training data sample [9]. Synthetic data by way of generative models has been employed to improve the training of deep learning models. This helps overcome the data limitations that are ubiquitous to medical imaging studies [23].

A state of the art method in measuring airway lumen radius and wall thickness on CT imaging, simGAN [15, 20], takes labelled simplistic representations of airway patches (synthetic images) and aims to transforms them in to the emulations of real airways by generative adversarial training (GAN) [5]. These are then used for supervised training of a convolutional neural regressor (CNR) which learns to measure airway radius and wall thickness and ultimately to run inference on real CT images.

The driving loss for realism in simGAN is cross-entropy loss computed on the classifications of the discriminator. For successful synthetic refinement by image transformation, the synthetic and refined images must have good correspondence in their shared label. To this end, a per-pixel $\parallel l{\parallel }_1$ regularisation loss is applied between input and output of the refiner.

GAN training is inherently unstable with mode collapse complicating and lengthening training times. As an alternative strategy, in this paper we propose the first use of perceptual losses to generate labelled synthetic airway images. Perceptual loss functions have been applied to image style transfer and super-resolution tasks[10]. We explore the clinical benefits of learning from perceptual loss generated synthetic data in mortality prediction.

Details of airway parameters and synthesis pipeline have been previously described [15]. Airway characteristics are sampled from a set of distribution parameters informed by [22]. We deviate from these parameters in two ways. First, we use an airway lumen radius (LR) interval of [0.3, 6] to permit measurement of smaller airways. Second, we use an airway wall thickness [$0.1\cdot LR+0.2$,$0.3\cdot LR+0.8$] mm to reflect the lack of airway wall thickening in IPF. We add four further parameters: (i) parameters for the airway centre determined by a normal distribution $X\sim N(0,1)$ mm to account for airway skeletons that are not perfectly positioned within the centre of the airway lumen. (ii) $p=0.4$ that an adjacent airway of similar diameter is randomly added. This is performed to accommodate airway patches close to airway bifurcations and to train the CNR to correctly identify the airway in the centre of the patch. (iii) We model our airways as ellipsoids, we achieve this by an ellipsoidness characteristic, sampled from a uniform distribution, $X\sim U(0.9,1)$ which determines the ratio in major and minor radii of the ellipse. (iv) Uniformly random rotation applied to the airway in the horizontal axis. We include our synthetic airway generator and configuration parameters in the open-source code repository.

We implement perceptual losses for computing high level perceptual differences between synthetic and real images as described by [10]. These losses are computed by comparing the activations in particular layers, $j$ of a pretrained convolutional neural network (CNN), $\varphi$ between a pair of images. Different activation layers of a trained CNN learn to represent different image features on the same sampled patch. In minimising for perceptual losses we are looking to reduce the differences in the activation of these layers between the refiner output and some objective image. For each calculation of perceptual losses on a synthetic input image, $x$ we have a refiner prediction, $\hat{y}$. As a modification of the original style transfer implementation [10], a randomly chosen real image is selected as the style target, $y_s$. Perceptual losses are then calculated and summed for different layers ${\varphi }_j$.

We utilise feature reconstruction loss. This is defined as the mean euclidean distance between activations of the input and output images of the refiner, where $C$, $H$, and $W$ are the number of channels, height and width of layer $j$ respectively. We use a VGG-16 [21] network pretrained on the ImageNet dataset [1] in our calculations of style and feature losses.

We also employ style reconstruction loss, which considers those features that tend to be activated together between the refiner output and the given style target image, a random real airway, where $G_j^{\varphi }$ is the gram matrix for a given layer $j$ of $\varphi$ as described in [4].

We examined CT images from 113 IPF patients diagnosed at the University Hospitals Leuven, Belgium. CTs were evaluated by an experienced chest radiologist (author JJ) for quality i.e. absence of breathing artefacts and infection. The quality of the automated segmentation was also visually inspected to ensure contiguous airway segmentations without oversegmentation blowouts. Airway segmentations were also required to reach the sixth airway generation in the upper and lower lobes to be selected for analysis. Pulmonary function tests were considered if they occurred within 90 days of the CT scan: Forced Vital Capacity (FVC, n=111)); diffusing capacity of the lung for carbon monoxide (DLco, n=103).

The trachea and first generation bronchi were excluded from analysis. We define an airway segment as the length of airway that runs between airway branching points or an airway endpoint. All airway segments were pruned by 1 mm at either end to avoid bifurcating patches. $80\times 80$ pixel size orthogonal airway patches were linearly interpolated with a pixel size of $0.5\times 0.5$ mm from the CT at 0.5 mm intervals along each segment. This resulted in a final set of 546,790 real CT-derived airway patches. A synthetic dataset of 375,000 patches was generated to train our refiner.

27% of patients were female. 74% of patients had smoked previously. The median patient age was 71, with 57% of patients having died. All patients had received antifibrotic drug treatment.

Measures of intertapering, intratapering [13] and absolute airway volume were derived from the airway measurements for each airway segment. Segmental intertapering represents the relative difference in diameter of an airway segment when compared to its parent segment. Segmental intertapering is calculated as the difference in mean diameter, $\overline{d}$ of an airway segment and its parent segment, $\overline{d_p}$, divided by the mean diameter of the parent segment. Segmental intratapering is the gradient of change in diameter of the airway segment relative to the diameter of the origin of the segment²²2Segments are considered to be oriented from the centre of the lung to the periphery. Accordingly, measurement of the airway origin beings at the end closest to the trachea. Segmental intratapering is computed by dividing the gradient, $m$ by the zero-intercept, $c$ of a line $y=mx+c$ fitted to the diameter measurements of an airway segment. Segmental volume is computed by summing area measurements along an airway segment, and multiplying this value by the measurement interval, i.e. an integration of area along the segment’s length.

Univariable and multivariable Cox proportional hazards models were used to examine patient survival. Multivariable models included patient age (years), gender, smoking status (never vs ever) and either FVC or DLco (as measures of disease severity) as covariates. The goodness of fit of the model was denoted by the concordance index [7]. A p-value of $<$0.05 was considered statistically significant.

We use the same refiner architecture as in [20, 15], the refiner is a purely convolutional network with four repeating 3x3, 64 feature ResNet blocks [8]. The measurement CNR, described in [15], is a convolutional network that feeds into two fully connected layers to learn the airway ellipse parameters. Instead of the custom CNR loss described in [15], we implemented a mean square error (MSE) loss for regressing to the airway ellipse parameters.

Synthetic images were generated to $0.5\times 0.5$ mm pixel size making $80\times 80$ pixel patches, corresponding to the real patch generation noted in section 2.3. All images were standardised and augmented on the fly, adding random Gaussian noise [25,25] Hounsfield units, random levels of Gaussian blurring with standard deviation scalled in the interval [0.5, 0.875] and random flipping ($p=0.2$). We apply random scaling on real images only, in the interval [0.75,1.25] to increase diversity in airway size. Finally, a centre crop was applied to make a $32\times 32$ pixel input patch.

Both simGAN and ATN models were trained for 10000 steps, where the simGAN refiner had 50 training iterations and the discriminator 1 iteration for every 1 step. The simGAN discriminator was implemented as described in the original method, with a memory buffer and local patch discrimination [20].