Segmentation of Weakly Visible Environmental Microorganism Images Using Pair-wise Deep Learning Features ¹¹1

Different techniques are implemented to enhance good segmentation performances of microorganisms. These techniques can be categorized into classical and machine learning based techniques refer_01. Table 1, gives a summary of the categories and subcategories of microorganisms image segmentation methods.

Classical methods are the traditional techniques which have found broad applications. For instance, in refer_13 outstanding results are achieved by applying Otsu thresholding in the segmentation of floc and filament. In order to enhance shape feature extraction, an active contour method is used in refer_18 for segmentation of Rotavirus-A. A seed watershed algorithm is applied in refer_20 for segmentation of Bacillus subtilis bacteria in clustered biofilm. Generally, classical methods are associated with challenges such as, they can not work direct on colour images, they need pre-processing like denoising and colour conversion and they cannot perform well on images which have uneven background colours. To overcome above challenges machine learning based methods have been adopted for segmentation.

Machine learning based methods can be categorised into unsupervised and supervised refer_01, as shown in table 1. Unsupervised machine learning (ML) techniques build their mathematical models from a set of data that contain only input without target output labels (segmentation can be referred to as pixel level classification, in that context the target labels are the individual pixel values/ranges in the ground truth mask images. Where, for the case of of unsupervised ML they are not required. An example of unsupervised ML algorithms is the $k$-means clustering). These techniques unsupervisely discover the data pattern and cluster them into segments refer_22. For instance, in order to automate the detection of pulmonary tuberculosis (TB) which is caused by Mycobacterium tuberculosis, $k$-means and self organizing map (SOM) clustering were proposed in the segmentation of the basilli from Ziehl-Neelsen sputum smears refer_23 and refer_24. While in refer_26a, a modified fuzzy divergence clustering method which is based on Cauchy membership function is leveraged in the segmentation of Plasmodium vivax from C channel CMYk color model of images containing the parasites in blood smears. Although unsupervised methods are simple to apply, their ability to learn the pattern of data is inadequate in transparent images, which is the case for the weakly visible EM.

In recent years the use of supervised methods has shown promising results in segmentation tasks. Supervised machine learning algorithms build mathematical models from a set of labeled data. Example of supervised techniques are convolution neural networks (CNN), support vector machine (SVM) and naive Bayes model. Due to the ability of CNN to capture pattern of data in challenging datasets, they have been used in many works. For instance, refer_27 increases the receptive field by applying $7\times 7$ filter size on fully convolutional network (FCN), this results in an outstanding segmentation performance of 99.7% accuracy on feline calicivirus images. In refer_28, refer_29, in order to tackle the challenge of imbalance between the foreground and background, a dice coefficient is applied as a loss function in U-net for segmentation of the rift valley virus and Leishmania parasites. To exploit fully the benefits of CNN, a large amount of training dataset is needed. One of the challenges we have in the weakly visible EM is the scarcity of datasets, However the innovation of strong models such as SegNet refer_31 and U-net refer_30 which are capable of working in small number of datasets, gives us a suitable option for our dataset. Moreover, SegNet shows more superiority for having few parameters and hence faster to train, because it passes pooling indeces to the upsampling layers and does not use the heavy deconvolution layers. U-net has been applied in many works for segmentation of EM. Nevertheless, to the best of our knowledge no any work has been done on segmentation of EMs using SegNet, except for one work which uses SegNet directly without any network changes from the original one on sementation of yeast cells refer_32. Thus, in this paper we attempt to leverage SegNet for segmentation of weakly visible microorganisms.

Feature extraction is an important stage in the image processing pipeline. In most cases features are used in image classification and object matching works such as refer_33, refer_34 and refer_35. Mainly there are two categories of feature extraction methods, hand crafted and feature learning refer_36, as indicated in table 2.

Hand crafted features are manual features which are extracted based on prior knowledge. For example, color (ie. RGB, HSV, LAB, HUE color modes), texture which is defined by the spatial distribution of pixels in the neighbourhood of an image (ie. energy, entropy, homogeneity, correlation, and contrast refer_71) , geometric features (area, perimeter and length), global shape (ie. Krawtchouk moment) and local shape features (ie. SURF and SIFT). Local features are the collection of basic and frequent features that can be used to estimate a class’s shape knowledge as they learns from finite samples of training data. Besides, two classes which are fairly similar cannot be distinguished by local features alone. Utilizing global features convey greater discriminative information of a class domain by making use of more specific and uncommon features refer_70. Hand crafted features, particularly local features (SIFT and SURF) are very useful in detection of interest points. Interest points are distinctive spots/regions that help to distinguish between different objects (images). refer_39. Corner, blob, and ridge descriptors are examples of interest points. They play an important role in image classification and matching tasks. For example, in refer_40, refer_41 image matching of EMs is achieved used SIFT features, where these features are derived from corner interest points of 10 channels of different color modes. In refer_42 edge and Fourier descriptors are applied for classification of EMs using SVM classifier. Interest points (descriptors) are useful in classification and image matching due to the fact that they are invariant to changes of illumination, rotation, and translation. Besides, local discriminant information content is abundant in the local image structure surrounding the interest point refer_43. Thus, we leverage the corner descriptors’ locations in enhancing the segmentation of weakly visible EM. However, corner descriptors (hand crafted features) are not sufficient to present diverse appearance of weakly visible EM. Therefore, we complement them by using deep learning features (feature learning).

Feature learning (features) are high dimension features generated by the composition of local features such as SIFT. Bag of visual words (BoVW) refer_37, sparse coding (which analyse a large number of images to learn set of bases where each expresses a characteristics pattern of a patch refer_38), and deep learning features are examples of feature learning refer_72. In most cases deep learning features are genereted from training the deep (convolutional) neural networks such as VGG-16, ResNet, and AlexNet. Deep learning networks represent high level features composed from low level ones. They have superior descriptive power than hand crafted features methods refer_44, because they replicate the feature extraction capability of visual cortex in human brain refer_45. VGG-16 is among the most superior and used models in segmentation and classification tasks because of its high ability in learning features. For example, in refer_46, VGG-16 achieves an outstanding performance on classification of viral pneumonia and bacteria from x-ray images. In refer_02, a VGG-16 pre-trained is used as a base model for segmentation in the Deeplab-VGG, this is achieved by replacing the fully connected layers with average pooling, three convolutions and interpolation layer, then use it for initial segmentation of EMs. Leveraging the capability of VGG-16, in this study we employ it in extracting the deep learning features at every location of the detected corner descriptor. Because of its robustness and simplicity the Bag of visual words (BoVW) is among the most used feature learning technique. However, because of the orderless representation of local features in it, it does not achieve maximum performance. To remedy that and improve the performance of BoVW, some studies have considered spatial arrangement of features to discover higher order in BoVW for object matching and classification refer_50, refer_51. Among the methods of arranging spatial features is by pairing of close visual words refer_52. For instance, in refer_53 and refer_54 pairing is done on visual words (where Prior to pairing, feature descriptions are mapped to the visual words, and then pairing is carried out on the visual words). Yet, the underlying distribution of pairs of neighboring local feature descriptors appears to be ignored by the pairing of visual words. To address that, refer_48 and refer_49 suggested that the pairing of spatial close local descriptors (such as SIFT) can be done before the building of BoVW. This seem to achieve maximum improvement on classification of challenging dataset. Motivated by the concept of pairing features and to the best of the authors’ knowledge, there is no any work which has been done on pairing of deep learning features for segmentation task, thus in this study we pair deep learning features generated from corner interest points’ locations and concatenate them to the base model for segmentation of weakly visible EMs.

SegNet is one of the powerful models in computer vision for semantic segmentation refer_31. It consists of the encoder and decoder, as shown in figure 3.

The encoder of the SegNet consists of 13 convolutional layers similar to VGG-16, without the last fully connected layers. Thus, the encoder network is largely reduced in parameters compared to VGG-16 and can easily be trained. Each of the 13 encoder layers constitutes of a series of convolutional layer with 64 filter banks (contrary to the original SegNet which use $7\times 7$ filter size, we apply $5\times 5$ to reduce the receptive field suitable for small size of EMs in images). Followed by batch normalization. Then a ReLu activation function $f\left(x\right)=max(0,x)$ which eliminates negative values follows. To achieve translation invariance over small spatial shift of input images, max-pooling with window size of $2\times 2$ and stride 2 (non overlapping window) follows, which results into output being sub-sampled by the factor of 2 after each step. The application of 13 max-pooling down sampling layers in the encoder achieves more robust pixel level classification but there is a loss in spatial resolution of feature maps (boundary details). To overcome this, the boundary information in the encoder feature maps are captured and stored before next sub-sampling in each stage by storing the max-pooling indices which are more efficient for restoring boundary information and require less memory. The decoder network (which has similar convolution layers in up sampling manner) upsamples the input feature maps using the memorized max-pooling indices from corresponding encoder feature maps. Each upsampling is followed by convolution and batch normalization layer to produce dense features that are similar in size to the corresponding inputs at the encoder. Finally, the softmax is used as the classification layer. We utilize SegNet as the base model for binary segmentation of weakly visible EMs. For all experiments we use binary cross entropy as a loss function and SGD optimizer with learning rate of 0.01 and momentum of 0.9. Although ReLu has shown some drawbacks such as decreasing the performance in the gradient descent operations because all gradient values would be zero when the activation values are zero refer_66, we still opt to use it instead of LeakyReLu which provides effective learning even when the values of activation are zero. This is because during our preliminary experiments on activation functions, the average results for ReLu were slightly highter than LeakyReLu by the margin of 0.19% accuracy.

Due to the challenges on weakly visible EMs dataset, the base model misses fine information from images during training, which gives poor segmentation results when using SegNet alone. Therefore, we use external pairwise features to enhance the performance of the base model by combining the advantage of interest points’ locations (hand crafted features) and deep learning features. Specific techniques are describes below;

To pair feature maps which have been extracted from the interest points’ coordinates, we use the Delaunay triangulation theorem.

At this stage, we join the features formed on the interest points’ coordinates ($F1,F2$…) and pairwise features ($Fm$…). The average amount of pairwise features for each image is between 36 and 43. 10 features originate from interest points. Thus, we form the joint feature maps by appending these features vertically. This joining style has shown best results from the tests done during experiments. The average joint feature maps sizes range from $46\times 1000$ to $53\times 1000$ for different images. Therefore, each joint feature map corresponds to one original image. Because the dominant features are pairwise features, we name the features as joint pairwise features (Pairwise features). After formation of joint features, they are stored parallel to their original images and ground truth images.

Both the original images and their corresponding joint pairwise features point to similar ground truth (GT) images. During training, the original images and corresponding ground truth images are fed to the input (first block) of the base model (SegNet). The joint pairwise features are resized at different sizes to fit the spatial dimensions of the encoder blocks of the the SegNet. These dimensions are $384\times 512$, $192\times 256$, $96\times 128$, $48\times 64$ and $24\times 32$ for first, second, third, fourth and fifth blocks respectively. Then we concatenate the joint pairwise features at different blocks of the encoder in the SegNet, as shown in figure 8 of the general proposed network.

We apply different options for concatenating the joint feature maps to the SegNet model. Example of the options are, concatenation at block 1 only, block 2 only, block 3 only, block 1 and 2 only, block 3 and 5 only, block 1, 2 and 5 only.

In order to evaluate quantitatively and compare the segmentation results of different approaches, we use accuracy (Acc), Dice, intersection of union (IoU, volumetric overlap error (VOE), Sensitivity (Sens), Precision (Prec) and Specificity (Spec). Accuracy: measures the percentage of pixels in an image which are correctly classified. Accuracy and specificity sometimes mislead results on segmentation when the object of interest is small compared to background (which is the case for our dataset). Because these measures are biased mainly on how well negative pixels (background) is predicted. Thus, we use more than one metric for correct analysis of the results. Dice coefficient: also known as F1 score, is widely used for evaluation of segmentation performance. The definition of Dice is given in table 3. Intersection over union: Also known as Jaccard coefficient, measures the percentage overlap between the target mask and the prediction output. Volumetric overlap error: Is the complement of Jaccard coefficient. Table 3, summaries definition of these metrics.

From table 3, $W_{prd}$ represents the predicted foreground by the model. $W_{gt}$ represents the foreground in the ground truth image. During segmentation of the EM, images are partitioned into two class pixels representing the foreground (the EMs) and the background. True positive (TP): is an outcome when the model correctly predicts the positive class. True negative (TN): is when the model predicts the negative class correctly. False negative (FN): is the outcome when the model predicts negative while it is actually positive. True negative (TN): is when the model predicts negative and it is actually negative. All the evaluation metrics are defined based on these terms TN, TP, FN, FP as shown in table 3. For analysis purposes, the greater the values of accuracy, Dice, IoU, sensitivity, precision and specificity indicate better segmentation results and the smaller the value of VOE indicates better results and vice versa.

Because the PDLF-Net originates from SegNet, therefore in this section we compare the segmentation performance of the PDLF-Net and SegNet on a small dataset (Each class having 5:5:10 dataset for training, validation and testing respectively). In our initial experiments we examined the performance of the PDLF-Net on different options of concatenating the joint pairwise features to different blocks of the encoder, such as concatenation at one block only of the PDLF-Net encoder as shown in figure 8, two blocks simultaneously, three blocks simultaneously, four blocks simultaneously and five block simultaneously. Referring to figure 8, these concatenation options can be described as concatenation at block 1 only, block 2 only, block 3 only, block 4 only, block 5, block 1 and 2 only, block 1 and 3 only following this order up to block 1, 2, 3, 4, and 5 only. We found that the performances are better when the concatenation is only at one block either block 1, block 2, block 3, block 4 and block 5 only. The increase in the number of concatenation blocks simultaneously leads to over-segmentation. Thus, we focus our research on concatenation at one block for all other experiments which we present in this paper. We compare and examine the performance of the PDLF-Net on small dataset of weakly visible classes by treating each class alone. Table 4 shows the performance of the PDLF-Net with concatenation at different blocks and the original SegNet.

From table 4, the application of pairwise features show improvement of the segmentation results. Block 5 and Block 2 results of the PDLF-Net are presented because they show consistent improvement in all classes compared to other block options. This is because, the deep layers at block 5 (bottom neck layers) in the deep network (PDLF-Net) are responsible for learning specific features of the foreground, therefore adding the joint pairwise features at block 5 emphases more the network to focus on learning the foreground (EM) thus improves the performance. The application of pairwise features on different blocks improves the segmentation performance by 6.21% acc, 2.9% IoU, 2.8% Dice, 14.83% sens and 15.57% spec on weak data class 1 (DC1). 6.06% acc, 1.95% IoU, 2.08% Dice, 4.98% sens, 7.23% prec and 5.15% spec on weak data class 3 (DC3). 6.30% acc, 2.59% IoU, 2.61% Dice, 17.26% sens, 0.25% prec and 15.94% spec on DC4. 5.00% acc, 8.00% IoU, 8.48% Dice, 19.32% sens, 17.00% prec and 20.6% spec on DC5. 12.90% acc, 4.32% IoU, 4.30% Dice, 7.45% sens and 8.76% spec on DC6. 2.73% IoU and 2.70% Dice on DC8. (The comparison above is obtained by taking the original SegNet result for a particular dataset class as a reference and compare it with maximum value of the PDLF-Net result of any block in that particular data class). The average performance results of the original SegNet and PDLF-Net at block 2 and 5 on all classes are given in figure 9 (This is obtained by averaging the results of all classes on a particular method separately and drawing the performance chart for each method).

From the general figure 9, even though the number of dataset for training is very small the PDLF-Net shows improvement in IoU, Dice, VoE, Sens, Prec, Spec and Acc by about 1.66%, 1.50%, 1.55%, 5.34%, 1.65%, 5.37% and 0.28% respectively. Generally the PDLNet shows improvement, however, the individual errors (VOE) are still high as shown in table 4. This is due to the small dataset which cause the networks not to generalize well during training. In order to reduce such errors and increase segmentation performance more, we apply augmentation on all weakly visible dataset and their corresponding GT images.

In order to enhance the performance of PDLF-Net, we augment all the weakly visible EMs and their GT images. Then joint pairwise features are extracted from each image and concatenated to different blocks. Each block is trained and tested independently for each dataset class. Table 5 shows the results of the most performing network concatenation configurations.

From table 5, there is an overall improvement of segmentation performance contributed by all blocks. Comparing with the original SegNet, the PDLF-Net shows improvement by an increase of 2.1% acc, 4.11% IoU, 2.90% dice, 2.96% sens, 3.15% prec and 2.96% spec on weakly visible data class 1 (DC1). 1.51% acc, 4.28% IoU, 3.33% dice, 3.08% sens, 0.29% prec and 3.20% spec on DC2. 2.34% acc, 0.94% IoU, 0.8% dice, 4.80% sens, 1.49% prec and 3.76% spec on DC3. 4.45% acc, 6.71% IoU, 5.26% dice, 3.32% sens, 3.05% prec and 2.86% spec on DC4. 3.18% IoU, 2.24% dice, 0.44% sens, 1.5% prec and 0.71% spec on DC5. 0.53% IoU, 0.37% dice, 0.30% sens, 2.00% prec and 1.56% spec on DC6. 3.19% acc, 3.46% IoU, 2.67% dice, 4.89% sens, 4.19% prec and 5.27% spec on DC7. 5.15% acc, 5.59% IoU, 4.46% dice, 5.18% sens, 7.92% prec and 5.97% spec on DC8. The average performances on all dataset for original SegNet and PDLF-Net are given in figure 10.

Figure 10 shows that the average improvement of about 1.09% acc, 2.20% IoU, 1.75% dice, 2.00% sens, 2.17% prec and 2.15% spec is observed on segmentation using PDLF-Net compared to the original SegNet. The overall average maximum results achieved by the PDLF-Net are 89.33%, 63.26%, 77.35%, 36.74%, 88.10%, 91.79% and 87.48% by acc, IoU, Dice, VOE, sens, prec and spec respectively. Moreover, the visual comparison of segmented images on original SegNet and PDLF-Net at blocks 2, 3, 4 and 5 are given in figure 11.

Comparing the observation performance from figure 11, the PDLF-Net shows better segmentation results. For instance, in data class DC3 and DC4 ($2^{nd}$ and $3^{rd}$ rows from the top) SegNet in (c) has not been able to show the foreground while there is a good segmented output of the same image by PDLF-Net in (e) and (f). In DC8 (last row), SegNet over-segments the image while good visual results are observed by PDLF-Net when concatenation of pairwise feature is at block 2, 3 and 5. Generally, the visual results show great improvement of segmentation results when using PDLF-Net.

To evaluate more the effectiveness of the PDLF-Net, we examine it on the test dataset. The test dataset contains 480 images, which are twice in number to the training and validation sets. The average segmentation performance of the PDLF-Net on test dataset for all classes is shown in figure 12. The graph shows a comparison of both the test set and validation set for each block performance.

From figure 12, each pair of bars from left to right, are of similar configuration (blocks) applied on validation and test sets respectively. The performance of the PDLF-Net is almost similar in both validation and test sets although the number of test dataset is twice. This shows the great effectiveness of the PDLF-Net on unseen dataset (test set). The highest average performances of PDLF-Net on test dataset are 89.24% accuracy, 63.20% IoU, 77.27% Dice, 35.15% VOE, 89.72% sensitivity, 91.44% precision and 89.30% specificity .

It should be noted that, although the proposed method has shown potential on EMs, it focuses only on segmentation of one (single) or two microorganisms on the image and not biofilms. Example of the segmentation of two EMs on the same image can be seen in Fig. 11 in classes DC3 and DC4, while other classes contain only one EM on every image. In the future work, we will extend our scope to testing our novel method on microorganim dataset with more than two microorganisms, clusters, and biofilms.