Compound Figure Separation of Biomedical Images: Mining Large Datasets for Self-supervised Learning

In biomedical articles, about 40-60$\%$ of figures are multi-panel (kalpathy2015evaluating). Several methods have been proposed in the document analysis community that envolve, extracting figures and their semantic information. For example, huang2005associating presented their recognition results of textual and graphical information in literary figures. davila2020chart presented a survey of approaches of several data mining pipelines for future research.

Supervised learning refers the usage of a set of input variables to predict the value of a labeled output variable. It requires labeled data (like an answer key that the model can use to evaluate its performance). Conversely, self-supervised learning (celebi2016unsupervised) refers to inferring underlying patterns from an unlabeled dataset without any reference to labeled outcomes or predictions.

Recently, a new family of self-supervised representation learning, called contrastive learning, shows its superior performance in various vision tasks  (wu2018unsupervised; noroozi2016unsupervised; zhuang2019local; hjelm2018learning). Learning from large-scale unlabeled data, contrastive learning can learn discriminative features for downstream tasks. SimCLR (chen2020simple) maximizes the similarity between images in the same category and repels the representations of different category images. wu2018unsupervised uses an offline dictionary to store all data representation and randomly selects training data to maximize negative pairs. MoCo (he2020momentum) introduces a momentum design to maintain a negative sample pool instead of an offline dictionary. Such works demand a large batch size in order to include sufficient negative samples. To eliminate the needs of negative samples, BYOL  (grill2020bootstrap) was proposed to train a model with an asynchronous momentum encoder. Recently, SimSiam (chen2020exploring) was proposed to further eliminate the momentum encoder in BYOL, allowing for less GPU memory consumption.

YOLOv5, the latest version in the YOLO family (bochkovskiy2020yolov4), is employed as the backbone network for sub-figure detection. The rationale for choosing YOLOv5 is that the sub-figures in compound figures are typically located in horizontal or vertical orders. Herein, the grid-based design with anchor boxes is well adaptable to our application. A new Side loss is introduced to the detection network that further optimizes the performance of compound figure separation.

Our goal is to only utilize individual images, which are non-compound images with weak classification labels in training a compound image separation method. In previous studies, the same task typically requires stronger bounding box annotations of subplots using real compound figures. In compound figure separation tasks, a unique advantage is that the sub-figures are not overlapped. Moreover, their spatial distributions are more ordered as compared with natural images in object detection. Therefore, we propose to directly simulate compound figures from individual images as the training data for the downstream sub-figure detection.

tsutsui2017data proposed a compound figure synthesis approach (Fig. 3). The method first randomly samples a number of rows and generates random heights for each row. Then a random number of single figures fills the empty template. However, the single figures are naively resized to fit the template, with large distortion (Fig. 3).

Inspired by prior arts (tsutsui2017data), we propose a simple augmentation strategy that is specific to compound figure separation data, called SimCFS-AUG, to perform compound figure simulation. The inputs of the simulator are single images with specified classes. Two groups are generated when simulating compound figures; these groups are row-restricted and column-restricted. The length of each row or column is randomly generated within a certain range. Then, images from our database are randomly selected and concatenated together to fit in the preset space. As opposed to previous studies, the original ratio of individual images is kept within our SimCFS-AUG simulator so as to mimic more realistic common compound images without distortion in individual images.

For object detection on natural images, there is no specific preference between over detection and under detection as objects can be randomly located and even overlapped. In medical compound images, however,objects are typically closely attached to each other without overlapping. In this case, over detection would introduce undesired pixels from the nearby plots (Fig.  4), which are not ideal for downstream deep learning tasks. Unfortunately, over detection is often encouraged by the current Intersection Over Union (IoU) loss in object detection (Fig.  4), as compared with under detection.

In the SimCFS-DET network, we introduce a simple side loss, which will penalize over detection. We define a predicted bounding box as $B^p$ and a ground truth box as $B^g$, with coordinates: $B^p=(x_1^p,y_1^p,x_2^p,y_2^p)$,$B^g=(x_1^g,y_1^g,x_2^g,y_2^g)$. The over detection penalty of vertices for each box is computed as:

Then, the Side loss is defined as:

The side loss is combined with canonical loss functions in YOLOv5, including bounding box loss ($L_{box}$), object probability loss ($L_{obj}$), and classification loss ($L_{cls}$).
$L_{total}={\lambda }_1{L}_{box}+{\lambda }_2{L}_{obj}+{\lambda }_3{L}_{cls}+{\lambda }_4{L}_{side}$ ,where ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, ${\lambda }_4$ are constant weights to balance the four loss functions. Following YOLOv5’s implementation ¹¹1https://github.com/ultralytics/yolov5, the parameters were set as ${\lambda }_1$ = $box\times \left(3/nl\right)$, ${\lambda }_2$ = $obj\times (imgsize/640{)}^2\times (3/nl)$, ${\lambda }_3$ = $(cls\times num\_cls/80)\times \left(3/nl\right)$, where $num\_cls$ was the number of classes, $nl$ was the number of layers, and $imgsize$ was the image size.The ${\lambda }_4$ of the Side loss was empirically set to ${\lambda }_1/30$ across all experiments as the Side loss and Box loss are all based on the coordinates.

We collected two in-house datasets for evaluating the performance of different compound figure separation strategies. One compound figure dataset (called Glomeruli-2000) consisted of 917 training and 917 testing real figure plots from the American Journal of Kidney Diseases (AJKD), with keywords “glomerular OR glomeruli OR glomerulus”. Each figure was annotated manually with four classes, including glomeruli from (1) light microscopy, (2) fluorescence microscopy, (3) electron microscopy, and (4) charts/plots.

To obtain individual images to simulate compound figures, we downloaded 5,663 single individual images from online resources. Briefly, we obtained 1,037 images from Twitter, and obtained 4,626 images from Google search, with five classes, including individual images from (1) glomeruli with light microscopy, (2) glomeruli with fluorescence microscopy, (3) glomeruli with electron microscopy, (4) charts/plots, and (5) others. The individual images were combined using the SimCFS-AUG simulator in order to generate 7,000 pseudo training images. 2,000 of the pseudo images (with multiple sub-figures) were simulated using intra-class augmentation. In addition, 2,947 individual images were further employed as training data. The implementation of SimCFS-DET was based on YOLOv5 with PyTorch implementations. Google Colab was used to perform all experiments in this study.

In the experiment setting, the parameters are empirically chosen. We set the learning rate to 0.01, weight decay to 0.0005 and momentum to 0.937. The input image size was set to 640, $box$ to 0.5, $obj$ to 1, $cls$ to 0.5, and the number of layers to 3. For our in-house datasets, we trained 50 epochs using a batch size of 64. For the imageCLEF2016 dataset (GSB2016), we trained 50 epochs using a smaller batch size of 8.

Mean average precision was the primary metric used to evaluate detection performance. For a given threshold IOU, average precision was obtained by calculating the area under the 101-point interpolated precision-recall curve. Then, mean average precision ($AP$) is the mean of the average precision for IOU thresholds from 0.5 to 0.95 with a step size of 0.05. $A{P}_{50}$ is the average precision with an IOU threshold at 0.5. $A{P}_{75}$ is the average precision with an IOU threshold at 0.75. $A{P}_S$ is the mean average precision for small objects (area less than ${32}^2$). $A{P}_M$ is the mean average precision for medium objects (area between ${32}^2$ and ${96}^2$). Since no objects contained an area greater than ${96}^2$, the large mean average precision ($A{P}_L$) was not utilized.

In this ablation study, we evaluate the image separation performance via 917 real compound images with manual box annotations as testing data in 1 and Fig. 5. For training, we assessed the performance of using 917 real compound training images (“Real Training Images”), as well as the performance when only using simulated training images (“Simulated Training Images”).

From the result, the proposed Side loss consistently improves the detection performance by a decent margin. The proposed compound image simulation method (with intra-class self-augmentation) achieves superior performance as compared to the benchmarks.

We also compare CFS-DET with the state-of-the-art approaches including tsutsui2017data and zou2020unified using the ImageCLEF2016 dataset (GSB2016). ImageCLEF2016 is the commonly accepted benchmark for compound figure separation, including total 8,397 annotated multi-panel figures (6,783 figures for training and 1,614 figures for testing). Table 2 shows the results of the ImageCLEF2016 dataset. The proposed CFS-DET approach consistently outperforms other methods by considering evaluation metrics. Additionally, we applied five-fold cross validation to our model training using weighted boxes fusion as proposed by (solovyev2021weighted). To merge the bounding boxes results from the five predictions, the proposed method used the confidence scores of all of the proposed bounding boxes in order to construct the average boxes. Eventually, when combining SimCFS with the weighted boxes fusion (SimCFS-DET ensemble), the performance was further improved.

We demonstrate the application of our SimCFS framework and how it helps to provide massive biomedical image data and benefits further data analysis with self-supervised representation learning.

In this study, self-supervised contrastive learning was employed as an example downstream task for our SimCFS compound image separation approach. We demonstrate how our approach helps to provide massive biomedical image data and benefits further data analysis with self-supervised representation learning. To evaluate the performance of introducing separated images, a semi-supervised method was evaluated beyond the supervised benchmark to present the performance of using the same set of unannotated images as the contrastive learning approach.(Table 3) Specifically, the stain and imaging modality classification task is employed to evaluate the performance of different approaches.