Weakly Supervised Faster-RCNN+FPN to classify animals in camera trap images

To reduce the amount of annotation work, we identify three main tracks: extracting knowledge from similarly labeled datasets; using crowd-sourced data labeling software and collaborating with many annotators, or applying weak supervision.

Open dataset (7) contains terabytes of camera traps images in some areas of the world which can be unlabeled, partially labeled, or contain multiple annotation errors. A popular way to extract useful knowledge from other datasets is referred to as transfer learning, under this umbrella, pre-training weights is a common method (Tan et al., 2018).

Another natural method is the usage of crowd-sourced labeling data software (Swanson et al., 2015) where many annotators can work together to divide the huge workload out between them. However, only a few ecologists worldwide know to distinguish and annotate some wild animal taxa on a given area of the world.

Another way to tackle annotation effort is to use weak supervision. At the difference of “fully supervised” (or just “supervised”) machine learning models are applied to large datasets with reliable and complete labels. There are 3 kinds of weakly supervised machine learning (Zhou, 2017): inaccurate, i.e., noise affecting labels; incomplete (or “semi-supervision” machine learning) i.e., consisting to train an algorithm on both unlabeled and label data samples; inexact meaning labels are too coarse-grained in comparison to the task. In Object Detection, the case is said inexact when human experts only label at image-level without qualifying which region of the image is decisive. This is known as Weakly Supervised Object Detection (WSOD).

Weakly Supervised Object Detection is a method able to provide localization of objects with only class annotation. The first approaches to WSOD formulate this task as a workflow of 3 consecutive steps: region extraction to extract candidate regions from the image; region representation to compute features representations of regions; region final decision such a regressor allows to refine the bounding boxes or a classifier to decide if a bounding box contains an object.

1. Region extraction to extract candidate regions from the image. The most common region proposals are today the Edge Boxes (Zitnick and Dollár, 2014) and the Selective Search methods. (Uijlings et al., 2013)

2. Region representation to compute features representations of regions such SIFT (Lampert et al., 2008), and HOG (Pandey and Lazebnik, 2011), and CNN-based (Wang et al., 2014).

3. Region final decision to refine the object localization (Pandey and Lazebnik, 2011) or enable/disable the objectness. More precisely, a regressor allows refining the bounding boxes coordinates or a classifier to decide if a bounding box contains an object (Lampert et al., 2008).

Over the last years, the landscape of weak object detection pushed forward the adoption of efficient end-to-end learning frameworks. Some methods extract the saliency map (i.e., object localization probability map) from a CNN classifier trained with an image-level annotation (Wei et al., 2017b) (Zhou et al., 2016) (Wei et al., 2017a). More advanced methods with the same label requirement consist in combining a CNN classifier and, with a saliency map extracted from it, training either another object detection CNN (Wan et al., 2019) or a semantic segmentation CNN (Jiang et al., 2013), (Yu et al., 2019). However, they share in common that the majority of works focus on VOC2007 or VOC2012 datasets where objects are rather centered and occupy a large portion of the image.

Another weak supervision method consists in using a standard strongly supervised model but a labeling rule-based function (Ratner et al., 2017) to automatically build labels with unknown accuracy. For example, JFT-300M dataset was built automatically gathering 375 million images from the web associated with one of the 19 thousand categories depending on web signals. It can be noisy (estimated to  20% error) in terms of label confusion and incorrect labels which are not cleaned by humans. This weak supervision machine learning method is popular when it comes to entity extraction where a label function can be found with some relevant hypothesis. If the system is unbiased, we may expect the quantity of automatically labeled data samples may compensate for their low quality.

This label function is domain-specific which is why we propose one to localize animals on camera traps and evaluate its performance. In the ‘experimental results’ section, we compare the complete workflow compared to some relevant baselines. In our raw images, no such heuristic exists to classify and localize our animals in images from camera traps. That is why we propose a new heuristic to localize the animal.

Weak supervision is very useful in video recognizing (P.Ochs and T.Brox, 2011) (Xiao and Lee, 2016) (Singh and Lee, 2019) containing tens of frames per second. They often rely on motion cues on images and propagate the objectness information over the neighboring pixels (spatial) and neighboring frames (temporal). Generally, the Weakly Supervised Object Detection in videos assumes that relevant objects are similar from frame to frame which is not a relevant assumption in camera traps data where images are taken with a lower time resolution. In a sequence of frames, the animal can be present in only one frame or present on multiple frames but with different positions and different locations.

Match and retrieve regions from the video to an external image dataset have been also used to automatically annotate bounding boxes and their associated class (Singh et al., 2016). This would be difficult to implement in practice in our case because an external training base should contain all animals’ taxa. For instance, only ecologists and a supervised trained neural network can infer efficiently animal taxa.

Many object detectors including R-FCN, Faster-RCNN are ‘meta-architectures’ and features extractors based on CNN architecture such as VGG or ResNet50. The meta-architecture defines the different modules and how they work together. The feature extractor is a deep learning architecture only made of convolution layers to extract spatial features.

To do object detection, researchers have developed fully integrated meta-architectures like Faster-RCNN (Ren et al., 2015) which both yield object localization and their classes. The meta-architecture defines the different modules and how they work together. To find an acceptable trade-off between accuracy and computational cost, researchers have proposed SSD or YOLO which are especially popular to recognize objects in videos, that is to say, millions of low-resolution images or usable for real-time applications. Our need is to recognize a few thousand highly detailed images containing often challenging small objects. That is why we give priority to accuracy over inference speed in this work, Faster-RCNN is a known robust meta-architecture containing three specialized trainable modules trained end-to-end. The CNN extractor is a standard deep learning architecture without any classification layer which extracts spatial features shared with the Region Proposal Network (RPN). The RPN is trained to localize Region of Interests (ROIs) such as the position of the animal in the picture and it sends them to the “classifier and regressor module”. This last module classifies each ROI and is trained to classify the localized animal and also to refine the ROIs coordinates.

In theory, standard CNN is adapted to detect different levels of abstraction in the input image. In practice, it fails to preserve tiny signals in the input image of the greatest importance on the final prediction. FPN (Lin et al., 2016) presented in 1 is trained to extract and make decisions on different levels of abstraction and resolution (the features pyramid). Faster-RCNN with FPN shows a significant improvement in several applications compared to basic Faster-RCNN models.

In summary, this paper is built on previous research on weak supervision methods and object detection to recognize small objects. The goal is to save the annotation effort on camera traps and to benefit from the state-of-the-art supervised object detection method to spot tiny objects (animals). While most recent initiatives are focusing on how to apply WSOD to localize objects with an end-to-end CNN classifier with some reasonable size objects (VOC2007, VOC2012) or video with multiple similar frames, we rather attempt to build a labeling function applied to small moving objects on short sequences of images

Before applying deep learning, the ecologists manually check all images and they input a mapping file that maps the name of the photograph with a taxon number or ‘0’ if there is no animal in the photo. Now, using our proposed system, the ecologist labels a few thousand images, then he/she runs the training phase before running the deep learning inference model on the remaining images.

Because no deep learning model is perfect, to save ecologists’ time and reduce errors, all predictions of the systems are sorted in ascending order to their Softmax posterior probability. Therefore, the first images have more chance to contain challenging images to recognize, like taxa absent from the training dataset. After that, the ecologist goes through the predictions and associated images and corrects the mapping file when needed.

The main criticism against deep learning is the long labeling time needed for the algorithms to be accurate. The advantage of the proposed classification Deep Learning method is that only one piece of information is needed by an animal in a picture - its taxon - which makes it faster and easier to label training images.

Our weakly supervised Faster-RCNN+FPN method is winning on both sides: it can accurately localize and classify a small signal in a cluttered background thanks to the FPN module while requiring classification labeling effort (taxa) and not the usual object detection effort (localization annotations and taxa). The overall proposed method is shown in figure 2 and described in the following paragraphs.

The localization algorithm. The presence of burst mode on modern camera traps makes it possible to capture the same animal in a short sequence of frames in a few seconds when the camera is triggered. Therefore, the animal motion in those frames is possible to distinguish from the background. A motion-based localization allows the ecologists to handle the workload induced by automatically computing bounding boxes of the animals and it enables to feed the object detection neural network.

Our localization algorithm proceeds following those 6 steps and they are illustrated in figure 3:

1. Input a short sequence of images $I_1,I_2,...,I_n$.

2. It computes a background $B$ computed with median filtering of all the $n$ images of the burst.

3. Motion map $M_1,M_2,...,M_n$ are euclidean distances between each pixel into $I_1,I_2,...,I_n$ and each associated pixel in $B$.

4. A binary threshold $t$ is applied on all motion maps $M$ with t=12%. Those new maps named $T$ may contain salt and pepper noise.

5. A morphological opening operation is applied on those previous binary maps $T$. First an erosion operation with kernel 3x3 allows us to erase noisy connected components. Then, a dilation operation with kernel 151x151 ensures all animal parts are connected. The denoised motion maps are named $D$.

6. The bounding boxes of the largest connected components (if any) are computed from $D$ and returned.

The accuracy of the localization algorithm is 77.6% on the Papua New Guinea dataset test dataset, with 14.7% of false-positive error (motion from raindrop, flying bugs, or wind in the vegetation) and 7.7% of false-negative error (The animal motion is missed).

The FP box correction algorithm. All the FP errors are automatically corrected based on a simple if-then-else code structure. It compares the presence of a detected box and the presence of an animal in the class label to avoid FP errors. Thus, Faster-RCNN+FPN is trained on 92.3% of correct labels (77.6%+14.7%).

The time performance of the workflow depends mostly on the hardware, the resolution of images, and the implementation. Our 3,776x2,124 resolution images are predicted with a throughput of 1.78 images/sec on an NVIDIA Tesla V100 GPU inside an NVIDIA DGX1 computer. Regarding previous studies on mammalian species in the African Savannah (Swanson et al., 2015), it took over 28,000 registered citizen scientists between 2 and 3 months to classify 3.2 million images. Our code would take 21 days with one single modern GPU with equivalent resolution. An actual production code would be faster than the current inference throughput due to software optimizations.

Those images are collected from a monitoring campaign that took place in Papua New Guinea. 8 motion-triggered wildlife camera traps have been used for 15 months with the same settings. They have been operated tied to a tree from 43 to 101 days and took between 474 and 1,272 photographs. When triggered, they shot a burst of three photographs spaced by 1 second, increasing the chances to have at least one good image of the animal. As a majority of mammals are nocturnal, camera traps are also equipped with a flash mode. To decrease potential errors made by our localization algorithm, tuning the settings of cameras is an axis of improvement including the number of shots per sequence and the delay between shots in one sequence.

25 taxa have been captured (dorcopsis luctuosa, bandicoot, scrubfowl, emerald dove…) into 5400 images with 8 cameras. Each camera is fixed to a tree and captures similar background when it is triggered, even if the background may slowly evolve due to leaves falling, weather, flowers blooming … We want our system to be trained on some camera traps and may be able to generalize on new ones. 6 cameras contain 4,320 labeled pictures, 95% of sequences serve as training and the remaining ones are the validation dataset. The testing dataset contains 1,080 pictures taken by the 2 remaining cameras. All pictures are colored with a resolution of 3,776x2,124 pixels. Each image is associated with one label, those labels are edited in a file which maps the identifier number of the picture and the identifier number of the class. If a picture contains an animal, the class identifier is set to the animal taxa starting at ‘1’. Otherwise, the class identifier is set to ‘0’.

This open-access dataset comes from the Missouri campaign (8) (Zhang et al., 2016). It contains approximately 24,673 camera trap images. There are about 1277 fully labeled images with bounding boxes and classes (squirrel, European hare, red deer, red fox, …). Among them, a statistic we made on 100 randomly drawn images shows 10% false-positive errors (bounding box and animal taxon whereas we do not see animal into), no false-negative errors, and a majority of badly framed bounding box annotations. The tiny portion of the labeled dataset and the fact that annotations contain errors make the task particularly challenging. We checked and corrected 1000 images to create the test dataset.

Wildlife camera trap automatic recognition present some challenging situation for deep learning methods. First, the animals are small in a cluttered background making it difficult to recognize them. We measure on the Papua New Guinea dataset that the surface of animals can occupy from the entire image to $<0.2\%$ of pixels of the photo depending on its actual size and its proximity to the camera, with a median of 4%. Additionally, both datasets contain unbalanced classes.

In addition, we summarize the difficulty to recognize an animal in five challenging situations: when the animal is hidden, when the animal is badly framed, when the animal appear blurred (e.g., motion blur when it is running or jumping), hidden behind foliage, unusually small ($<0.2\%$ of pixels of the image), when the animal is unknown to the training dataset. They are illustrated in supplementary material section Challenging image types.

The accuracy comparison of different workflows is shown in table 1. Those workflows, their settings, and their characteristics are discussed in the following text.

CNN Classifier. It is a trained classification deep learning model ResNet50 (He et al., 2015). It takes as an input the overall photograph and classifies among 26 classes, 25 animal taxa, or the empty class. We tested also VGG16, InceptionV3, EfficientNet-B4 with similar settings: 50 epochs and batch size of 32. ResNet50 is more accurate than VGG16 and performs the same as InceptionV3. The neural network is trained with SGD and the learning rate starts at ${10}^{-3}$ and is divided by 10 after the ${20}^{th}$ and the ${40}^{th}$ epochs. Despite we spend time assessing many neural network architecture and optimizer settings, we have not obtained satisfactory results. It takes only 1 hour to be trained, thus this workflow and RP+Classifier are the fastest to converge.

RP+Classifier. (Region proposals+Classifier) This method uses also a ResNet50 classifier with the same training set as the CNN Classifier workflow. The difference is that it is trained on the region of interests yielded by the localization algorithm presented in section 3 and the overall image such as Faster-RCNN and classifier models. An empty class is added because the localization algorithm provides frequent (14.7%) false-positive error motions (again, wind, flying bugs, …). We explain the performance gained compared to the standard classifier by the fact that the localization algorithm crops most of the cluttered background, thus the classifier part is trained and predicts with a better focus on the moving object.

Auto-Keras. (version 1.0.12) It is an AutoML strategy named Auto-Keras Image Classifier (Jin et al., 2019). It is a Bayesian optimization algorithm searching the best neural network architecture using validation metrics and calibrating the weights of the candidate on the training dataset. We observe a small improvement compared to the ResNet50 classifier after tuning 100 models trained a maximum of 20 epochs but the training computing cost is about multiplied by 100 (4 days on 1 NVIDIA Tesla GPU). It shows that the famous ResNet50 neural architecture is relevant in our datasets and spending days searching for a specific neural architecture may only improve the results a little bit.

Weakly Supervised Faster-RCNN. We use ResNet50 inside Faster-RCNN and Faster-RCNN+FPN meta-architecture. The neural network converges after 200 epochs during 4 hours on Tesla GPU with a batch size of 32. The model is trained with SGD and the learning rate starts at ${10}^{-3}$, then divided by 10 after the ${100}^{th}$, ${170}^{th}$ and ${190}^{th}$ epochs. In the case of the Papua New Guinea dataset, we compare our workflow with and without manually correcting the bounding box annotations. And in the case of the Missouri dataset, we compare with our workflow and the original bounding boxes downloaded with the Missouri dataset.

The downloaded dataset annotation of the Missouri dataset ((3) in the figure 1) (Zhang et al., 2016) contains inaccurate bounding box annotations. It suffers from about 10% images which contain wrong animal labels or boxes without a visible animal in them. Our proposed workflow can cancel a bounding box when the class of the image is “empty” but it cannot handle when a wrong animal class is given and thus the biggest move (e.g., the wind on the grass) is used as a bounding box associated to a false taxon. We run our Localization based on the motion on this same dataset and provide better annotations ((2) in the figure 1).

RP+Classifier works better than classifier thanks to its ability to focus on the region of interest but is still inferior to Faster-RCNN+FPN. RP+Classifier loses contextual information: the relative size of the region of interest is lost because it is resized to the fixed size 256x256 to feed the classifier. And more, the surrounding is also lost it may be a major issue when the localization poorly frames the animal, thus some animal parts are cropped and not given to the neural network.

Finally, we observe on the Papua New Guinea dataset that fully supervised deep learning performs +4.6% on the test compared to the weakly supervised counterpart (again, 7.7% of the training dataset is affected by false-negative error boxes).

Table 2 shows that our method displays an overall accuracy of 81.7%. In practice, when an error is made, other images in the sequence often contain at least once the right animal identification.

We do our best to break down all the test datasets into those types of challenging images but choosing for each image is sometimes subject to interpretation. However, we show that the majority of errors are caused by those challenging images and 7% of errors are unavoidable due to the discovery of new species. We also show that 1/3 of images are challenging.

We observe in the previous section that on two datasets Faster-RCNN+FPN was superior to other approaches. To better understand this accuracy, we use the nMNIST testbed (Pawlowski et al., 2019). It is an easily reproducible testbed that consists to classify if the image contains or not the digit ‘3’ in a randomly generated cloud of digits. The experiment is repeated with four different O2I (object-to-image): {19.1%, 4.8%, 1.2%, 0.3%} and the corresponding number of digits in each image generated is {3, 6, 26, 101}. For all values of O2I ratio, 11276, 1972, 4040 of training, validation, and testing images, out of which 50% are negative and 50% are positive images.

We report results of the workflow previously proposed (Pawlowski et al., 2019) and meta-architectures. We observe roughly similar results when changing the architecture (such replace ResNet50 with EfficientNet-B4) so we decided do not to show them to keep the figure readable.

We observe that when O2I=19.1%, all workflows perform around 95% but [Pawlowski et al 2020] which is around 92%. When O2I=0.3%, Faster-RCNN+FPN outperforms all other workflows with a 76.83% accuracy, and [Pawlowski et al 2020] shows very unstable results with different run times. More surprisingly, when decreasing O2I, Faster-RCNN without FPN accuracy drops faster than all the other workflows. However, on our biodiversity datasets, Faster-RCNN performs better than classification neural networks.

Those experiments confirm that Faster-RCNN+FPN is the more stable to recognize needle-in-haystacks and performs also very well when the object appears bigger. In camera trap applications, due to the accuracy of Faster-RCNN+FPN, we conclude it deserves to use an additional localization algorithm to compute bounding box annotations to feed it.