Weakly and Semi-Supervised Detection, Segmentation and Tracking of Table Grapes with Limited and Noisy Data¹¹1This work extends the one titled ”Pseudo-label Generation for Agricultural Robotics Applications” presented at the 3rd International Workshop on Agriculture-Vision, CVPR 2022, New Orleans.

Computer Vision techniques applied to Agriculture have been a central subject of recent research. Often these systems are focused on yield estimation, but pest and sickness detection are also relevant applications of detection and segmentation in Agriculture. In this Section we outline mainly the methods that rely on vision sensors, such as simple RGB cameras, since they are readily available and relatively cheap. In this context, a number of studies are relevant to this work. In Bellocchio et al. (2019) Bellocchio et al. present an olive counting solution that is explicitly trained with weak labels and consistency losses. This work is close to ours for the focus on working on data with minimal labelling, however it is based on simple direct fruit counting, which can lead to huge errors in cases where self occlusion is typical. While many early works Skrabanek and Majerík (2016); Nuske et al. (2014), and some recent ones Pérez-Zavala et al. (2018), use hand crafted features, most of the recent approaches use representation learning Bargoti and Underwood (2017); Wan and Goudos (2020); Bellocchio et al. (2019). When moving into this kind of technique, data availability is a key issue. In this respect, Korala et al. Koirala et al. (2019) present an overview of Deep Learning methods applied to fruit detection, pointing out in particular the critical role of data availability and recommending the use of public data and benchmarks to compare results.

Often, agricultural tasks are cast as activities that could be performed by an autonomous robot. In this line of research there are numerous contributions, both in single and multi robot scenarios. Halstead et al. Halstead et al. (2018) present one such systems where they detect red peppers with a camera mounted on an AGV. The authors also estimate ripeness by considering ripeness stages as different classes. Among the robotic solutions for yield estimation, there are a few that use a multi robot setup, in particular mixing different types of robots, such as Unmanned Ground Vehicles (UGVs) and Unmanned Aerial Vehicles (UAVs) as in Rahnemoonfar and Sheppard (2017). A more recent example of this kind of system is given in Pretto et al. (2021) by Pretto et al. where the multi robot approach is explored in depth together with the use of multi spectral cameras to detect and monitor both crop and weeds. These approaches, while effective, require a good deal of time from experienced technicians and are not applicable on cultures such as table grapes, where the foliage covers the fruit from above. However, in Ballesteros et al. (2020), Ballesteros et al. use UGVs equipped with multi spectral equipment to compute Vegetation Indexes and then train a yield estimator leveraging the correlation between plant vigour and yield quantity. This approach, using different sensors and indirect measures, is valuable and could be considered as complementary to the one we propose, where yield can be estimated by directly tracking and counting the fruit instances.

As we have shown in the related works, supervised solutions to detection and tracking have been proposed many times and show interesting performances, but at the same time the generality of these solutions is naturally impaired by the limited data each solution is trained on. Even if for the majority of the most common fruits a supervised solution with a relative amount of labelled data exists (e.g. apples Bargoti and Underwood (2017), grapes Santos et al. (2020), tomatoes Liu et al. (2020a)), every time we want to train a new detector for the same fruit in a different field, there is considerable covariate shift due to different sensor and environmental characteristics, illumination conditions and fruit intra-species variability. Put in simpler terms, given the limited variety of labelled data for this kind of application, the overfitting of the algorithm to the specific data collected is inevitable. We propose then a different approach to the problem: collect data for the target application and find easy and robust ways to bridge the covariate shift from any existing data and the target one. We work on this problem using the case of table grapes, for which there is some labelled data available (e.g. the wine grape dataset from Santos et al. Santos et al. (2020)), but which is not enough to work on a different variety (e.g. Pizzutello instead of Cabernet), cultivated with slightly different techniques (tendone structure instead of standard trellis structure). We propose algorithmic techniques to produce pseudo labelled data in order to bridge the gap of covariate shifts that occur whenever a new specific crop becomes the target of a computer vision system for precision agriculture. We explicitly tackle the problem of doing so with limited hardware and software resources, to address the needs of small and medium business. For this reason all the pseudo labelling strategies presented are based on simple videos collected with a cellphone camera. With this in mind, the specific pseudo labelling strategies we propose are of two kinds:

• Automatic bounding boxes generation for objects contained in consecutive video frames, based on a starting estimate and 3D structure geometrical considerations. We show that, leveraging a simple initial labelling - which could be manual or automatic - and the information that we can get from feature matching and structure from motion, we are able to generate new labelled data that greatly increases the performance of the detector.

• Pseudo mask generation for instance segmentation: we show how, starting from a simple bounding box - which could be the one automatically generated in the previous step - it is possible to use a segmentation network together with a refining strategy to generate new mask labels.

An overview of the system is depicted in Figure (1). The main inspiring principle of this work is the economy of data labelling and data reuse, for this reason the only two sources of data are the source dataset (data available from a similar task) and a video collected on the target field. The source dataset is used to train the initial detection and segmentation models, namely the Source Detector Network (SDet) and the Source Segmentation Network (SSeg). SDet is not perfectly tuned on the target environment, still it can be used on selected frames of the video input that we call keyframes. To keep this solution simple, we consider equally spaced keyframes starting from the first one, but other strategies could be devised. A set of initial bounding boxes is extracted from this keyframe, using a high confidence threshold, to limit the false positives. Then, the whole video is passed in a Geometric Consistency block (GC block) that extracts features from each frame and associates them. We tested two different options for this block, as will be shown in the following Sections. Using this geometric information, together with the initial bounding boxes extracted from the keyframes, it is possible to interpolate the bounding boxes positions for the remaining frames with high accuracy. These new bounding boxes are our pseudo labels for training the detector on the target environment, which we call Target Detector (TDet).

The Detection Pseudo-Labels Generation (DPLG) sub-system could be used independently by the Segmentation Pseudo-Labels Generation (SPLG). To prove the effectiveness of the approach, we compare the performance of TDet on the bunches tracking problem, i.e. counting the number of grape bunches by counting the instances tracked along a video. This problem is relevant, since it can be used for yield estimation purposes. We test two different tracking algorithms, that are described in Section 2.6.3 and evaluated in Section 3.2.

The goal of the second part of the system is to generate pseudo masks for training an instance segmentation network. This sub-system can be seen both as an independent pseudo label generator, or as part of a bigger system such as the one we describe here. As mentioned before, the SSeg is trained only on source data and is not able to produce good segmentation masks on the Target Data. However, it is possible to give the network some information cues that can greatly improve the mask estimates. The first one is the bounding box region in which the instance should be segmented. This cue comes easily from the previous step of pseudo bounding box generation, but it could be produced otherwise. This generates the initial pseudo masks. It is possible to use these pseudo masks for training the Target Segmentation Network (TSeg) but this would lead to poor performances due to confirmation bias. We need therefore to inject external information from other cues that we have. In our system, this is the role of the pseudo masks Refining block. In Section 2.7.1 three different solutions for refinement will be described. Thanks to these refined pseudo masks it is finally possible to train the TSeg Network. Section 3.3 reports the results of the refining strategies and compares the performance of TSeg with SSeg.

Finally, the experiments of the whole system, trained only on videos and tested for instance segmentation performance are given in Section 3.4.

The experimental field is located in southern Lazio (Italy). The vineyard is composed of two plots approximately 114 m x 51 m (0,58 ha) and 122 m x 48 m (0,58 ha). Vineyards are structured as a traditional trellis system called Tendone with a wide distance between each plant, 3 x 3 $m^2$. Plantations are all older than 3 years and so in full production and health, thus representing a typical working condition for the validation of agronomic activities such as fruit harvesting or vine pruning. All structures are traditionally covered with plastic and net to protect grapes from rain and hail. The average extension of each plot is around 1 hectare and dimensions (length and width) are on average between 25 m and 50 m according to plot extension and geometry. The selected vineyard in Aprilia has currently four different table grape varieties which are described in the following: White Pizzutello, Black Pizzutello, Red Globe and Black Magic. Figure (3) shows some examples of the these grape varieties while Figure (2) shows images of the experimental field as well as the approximate extension of each grape variety in the vineyard.

Of the four varieties that were present in the vineyard, Black Magic was of very low quality and thus untended by the field owner. White Pizzutello is identical in shape to the Black one, and the latter has the same color of the former when not ripe. Together, white and black Pizzutello are a peculiar variety of the Lazio Region and present the highest variability in shape and color with respect to standard rounded berry variants. For these reasons, while we collected images of all of the varieties, we finally concentrated our data labelling effort only on Black Pizzutello.

The proposed system deals with the covariate shift from a generic source dataset to a target dataset that is representative of the images that could be collected on the field. We assembled our target dataset with two different kinds of data. The first are videos recorded using a mid range cellphone camera (MotoG8 Plus), which simulates a data collection operation that could be performed by a farmer with ease. We collected videos moving along the vineyard (i.e. tangential to the rows), without any requirement on distance from the fruits or height from the ground. In this work we use HD (128$\times$720) videos at 10Hz with a total of 1469 frames. Examples of these frames are shown in Figure 4. A short segment of 10 seconds has been labelled for test use in the case of the tracking algorithm evaluation, while the rest has been used without labelling thanks to the semi-supervised nature of the system. We briefly call this target video dataset TVid.

The second kind of data is composed of static images of Black Pizzutello. This data simulates the images a farmer, or a robot, could collect to perform some agricultural action on specific grape bunches (e.g. quality estimation, disease detection, automatic harvesting). The dataset consists of 134 images of 3000x4000 resolution, collected with the same cellphone camera used for the videos, however the optics and chip used for video and still images are different, as often is the case with cellphones. This is intentional, since it adds a very common source of covariate shift related to the device and capturing mode (motion vs still images). All the images in this case have been labelled for detection (bounding boxes), while a small subset has also been labelled for instance segmentation (70 images), using the Innotescus labelling application Innotescus LLC (2022). All these labels are used for validation and testing of the algorithms described in this Section. We call this still images dataset TImg.

Together these datasets (TVid and TImg) constitute our Target dataset (TD).

As mentioned before, we work under the hypothesis that a small amount of labelled data of the same fruit exists, but that it has considerable covariate shift with respect to the TD distribution. In this work, our Source Data is the one presented by Santos et al.(Santos et al., 2020). For the details about these data the reader can check the cited work, here we give a short summary to underline the differences between SD and TD:

• the grape varieties (wine vs table, berry shape and color)

• the illumination conditions (full sun vs shadows)

• the camera device (Reflex vs cellphone camera)

• scale of the images (standard scale vs variable scale)

To quantify the covariate shift gap, in Section 3 the performance drop for detectors and instance segmentation networks trained on SD and tested on TD are given.

In this Section we describe the metrics used to evaluate and compare the detectors, the trackers and the instance segmentation algorithms. To evaluate the detectors and instance segmentation algorithms, the standard metrics of Precision, Recall and Intersection over Union (IoU) have been used. In addition, for instance segmentation, the Average Precision, as defined in the MS COCO challenges Lin et al. (2014) has been used. Usually AP is computed for each class and then averaged to obtain the mean average precision (mAP). In this work, since there is only one class (grape), the AP coincides with the mAP. In the MS COCO metrics, the AP is calculated by computing the precision at every recall level from 0 to 1 with a step size of 0.01. The mAP is then computed by averaging the AP over all the object categories and ten IoU thresholds from 0.5 to 0.95 with a step size of 0.05.

To evaluate the trackers we follow the common practice of Multiple Object Tracking (MOT) as defined by Wu and Nevatia Wu and Nevatia (2006) and the CLEAR MOT metrics Bernardin and Stiefelhagen (2008). MOT is a difficult task to evaluate, since the performance metrics should capture both the precision in detecting individual instances and the accuracy in tracking each instance across multiple frames, without losing track, or switching between instances. Given a number of objects $o_j,j\in \left[\mathrm{0..}m\right]$, the tracker produces a number of hypotheses $h_i,i\in \left[\mathrm{0..}n\right]$. The performance of association of hypothesis and objects can be measured frame by frame by using the classic True Positive, True Negative, False Positive and False Negative figures, together with their direct descendants Precision and Recall. However, recently a number of compound indexes have been proposed to better capture the general tracker’s performance. The first one is the Multiple Object Tracking Accuracy (MOTA) defined as follows:

where $FN$ and $FP$ are False Negatives and False Positives, $I{D}_{sw}$ represents the number of instances whose ID has been erroneously switched, GT is the real number of instances in the video. This index accounts for three sources of error, namely the false positive ratio, the false negative ratio and the mismatch ratio. Together, they give an idea of the general tracking accuracy. To evaluate the precision a second index was proposed:

where $c_t$ denotes the total number of matches in frame $t$, and $d_{t,i}$ in general represent the distance of the hypothesis and the object, but in our case can be computed as the overlap of the ground truth and the hypothesis bounding boxes. This second index gives only a measure of the precision in detecting the instances, without giving any information on the tracking and association capability.

As explained in Section 2.1, the general pseudo label generation system is based on pre-trained detection and segmentation networks (SDet and SSeg), and is meant produce the pseudo labelled data to train new networks that are able to perform better on TD (TDet and TSeg).

The main parameters that influence the choice of the architectures are speed and accuracy. It is well known (Liu et al. (2020b)) that SotA detection networks can be divided into two main categories: two stage and single stage. The first kind separates detection into two phases, the first is called region proposal and gives object bounding boxes candidates, while the second filters and refines these candidates to produce the bounding boxes and classifies the objects. The second kind instead extracts both region proposal and class prediction in one pass. The main advantage of the single stage detectors is speed, which is much higher than the two stage one, but at the cost of a general reduced accuracy. The main examples of single stage detectors are the YOLO variants, in particular the recent YOLOv5 Redmon and Farhadi (2018). One of the best known and best performing two stage architectures is Mask R-CNN He et al. (2017), which is also a segmentation network, more accurate than any YOLO variants, but slower and difficult to tweak for real-time use.

In this work we use the single stage YOLOv5s architecture for the experiments on tracking, since real-time detection is needed for this kind of application. In addition, some of the variants have a small number of parameters, which makes them viable for embedded applications, such as robotic harvesting. The pseudo bounding box generation could be performed offline, thus allowing use of the better performing Mask R-CNN, but we decided to use the YOLO detector to keep this sub system self-contained. In addition, using an architecture with lower detection performance stresses and tests the robustness of the generation process. For segmentation and pseudo mask generation a segmentation network is needed, so the choice falls on Mask R-CNN. The details of the pretraining and fine tuning of the detection and segmentation networks are given in Sections 3.1.1 and Section 3.3.1, respectively.

In this Section we detail the elements of the DPLG system depicted in Figure 5, i.e. the pseudo bounding box generation system, together with the tracking algorithm used for yield estimation as a possible application.

While detection is enough for counting tasks, for quantitative yield estimations or for tasks that require a physical interaction with the volumes of the grapes (e.g. harvesting), segmentation, and in particular instance segmentation, is required. Instance segmentation requires labels that are ideally pixel perfect masks, however Bellocchio et al. Bellocchio et al. (2019) showed how, even with minimal labelling signal (e.g. presence or absence of an object in a image) the task network is able to learn representations that are close to masks of the object of interest. For this reason we again adopt a pseudo-labelling approach to this problem, starting with a pretrained network on the WGISD source dataset and then using simple external cues to work as our external information signal that helps in refining the label. The overview of this sub-system is depicted in Figure 8.

Our SSeg network is Mask R-CNN trained on WGISD, as usual. Mask R-CNN in its basic form extracts region proposals and uses them to predict bounding boxes and instance segmentation masks. However, it is possible to use the segmentation subnetwork of Mask R-CNN as the pseudo mask initial generator. In particular, Mask R-CNN is wired differently at inference time than at training time, since the bounding boxes predicted by the detection head are directly fed to the mask head. The network will use this bounding box as a cue, or as an attention mechanism, which helps the segmentation sub network to output a useful pseudo mask. This is depicted in Figure 9. This strategy will mitigate the problem of confirmation bias, since the box comes from an external information source. In our system the bounding boxes could come from the output of the DPLG sub-system. At the same time, in Section 3 we show the performances of the pseudo mask generation starting from ground truth bounding boxes so as to better isolate the performance contribution of the mask generation process. In this way, the number of pseudo-masks will coincide with the actual number of grape clusters in the image, and the measured error will only be due to the mask generation process. The qualitative difference of segmentation mask between the standard wiring of the Mask R-CNN network, and the one with an external attention mechanism is shown in Figure 10.

In this section we describe the results of the detection experiments. Table 1 shows our preliminary experiments to compare different version of the detector. All models in this initial comparison are trained and tested on WGISD. Results show that the models with a large number of parameters offer a minimal performance increase on basic detection, compared to the lightweight versions S and N. This can be explained by a general homogeneity of the distribution of the grape images in WGISD, which does not require huge number of parameters to learn a good estimator. This is expected, since in agriculture we do not work with huge amounts of data. For this reason we decided to base all the trackers on the S and N variants to reduce overfitting.

In this section we describe the results of the tracking experiments. Each tracker is built on a YOLOv5 detector version. As explained in Section 2.6.3, we compare two tracking schemes combined with two pseudo label generation strategies. Pseudo labels depend, among other intrinsic and extrinsic variables, on the frame rate combined with the skip value, that was described in Section 2.6.2. As mentioned in that section, with the bounding box interpolation system we use the best performing skip value is 2, as showed in Figure 14. This is expected, as if too many frames are interpolated, the motion becomes too large to be compensated. In addition, from the same results it is clear that the SfM approach has higher MOTA than DeepSORT at most skip values, due to its use of the geometrical representation of the scene. However, it is not meant for real time computation. In Table 4 the MOT metrics for the best models have been summarized.

Among the MOT metrics described in Section 2.4, the MOTA and MOTP give a general idea of the tracking performances. However, to use trackers as yield estimators, one of the figures of interest is the number of IDs that the tracker finds, which could be considered an estimate of the number of bunches found. However, during the tracking process, some of the bunches IDs are switched. This can happen, for example, when two bunches are occluding each other, and the IDs are inverted after the occlusion situation disappears, or when there are errors in feature association in the SfM block (Figure 15). Whatever the reason, this situation is captured by the $I{D}_{sw}$ metric, together with the MOTA score. More details on these challenges of MOT can be found in Bernardin and Stiefelhagen (2008). For all these reasons we focus our attention on MOTA for the tracking accuracy, and on the number of IDs for the yield estimation.

From Table 4 it is evident that the pseudo label generation is highly beneficial, with an increase of more than 10% compared to the source dataset WGISD. Looking closely at the performance metrics, it can be seen that the capacity to track the same IDs for the entire trajectory is stronger in DeepSORT. This is probably due to Kalman filtering, since taking into account the bounding box movement dynamics implicitly avoids errors such as the one shown in Figure 15.

In this Section we show the results of the experiments concerning the performance of the SPLG sub-system. The pseudo-mask generator can be seen as an independent system, or in conjunction with the DPLG sub-system. In the following the experiments of the SPLG sub-system are described, while the results of the whole system are described in Section 3.4.

In this section we describe the results obtained by using the detector described in section 3.1 to generate the bounding boxes required by the SPLG sub-system described in the previous section 3.3. First the best YOLOv5 detector, namely that obtained with the use of the pseudo-labels generation method, was used to predict the bounding boxes that are used to generate the pseudo-masks by Mask R-CNN. This was done both for TImg training set and for TVid. The test data for this experiment is the TImg test set, so the training and test distributions, although being target data, are different. Table 7 again shows the comparison of TSeg with and without the Refining Block. In the case of refined TImg the improvement is still substantial with an improvement of more than 40% over the baseline. Moreover, TVid is able to give an even greater improvement thanks to the greater number of images in the training set. Despite the fact that the video frames present many differences with respect to the target dataset, the TSeg still manages to increase the performance by 42% with respect to the baseline. Finally, the increase in mAP is even greater when considering TImg and TVid as training data. Also in those experiments, the Refinement Block gives an improvement over the non refined counterpart. From the values of $mA{P}_{0.50}$ and $mA{P}_{0.75}$ we deduce that the increase is mainly due to an improvement at IoU higher than $0.75$.

Thomas A. Ciarfuglia: Conceptualization, Methodology, Validation, Investigation, Data Curation, Writing - Original Draft, Writing - Review & Editing, Visualization, Supervision, Project administration; Ionut M. Motoi: Methodology, Software, Validation, Formal analysis, Investigation, Data Curation, Writing - Original Draft; Leonardo Saraceni: Methodology, Software, Validation, Formal analysis, Investigation, Data Curation, Writing - Original Draft; Mulham Fawakherji: Methodology; Alberto Sanfeliu: Supervision; Daniele Nardi: Supervision, Project administration, Resources, Funding acquisition, Writing - Review & Editing;