BioSLAM: A Bio-inspired Lifelong Memory System for General Place Recognition

Place recognition targets identifying the exact areas under different perspectives and environmental conditions [37]. There are mainly addressed with two types of approaches, namely visual-based and LiDAR-based place recognition.

For visual place recognition, the visual inputs are usually affected by illuminations and viewpoints. The traditional geometry descriptors (e.g., scale-invariant feature transform (SIFT) [39] and oriented FAST and rotated BRIEF (ORB) [43]) are widely used in visual place recognition because of their invariant properties to scale, orientation and illumination changes. Based on these handcrafted features, FAB-MAP [15] build a Bag-of-visual-words (BoW) architecture to achieve large-scale visual re-localization. iBoW-LCD [21] uses an incremental BoW scheme based on binary descriptors to retrieve matched images more efficiently. An et al. introduces FILD++ [1], an incremental loop closure detection approach via constructing a hierarchical small‐world graph. With the booming of deep learning, new convolutional neural network (CNNs) features, such VGG [47], ResNet [23], Transformer [53], provide significent improvements in feature/semantic extraction. NetVLAD [2] combined the CNN features and an differentiable VLAD [3] layer to enable deep learning for visual place recognition; and based on [2], recent deep learning approaches [26, 19, 25] further improve the recognition accuracy by combining with different networks.

For LiDAR-based place recognition, LiDAR inputs will not be affected by environmental conditions, such as illumination, weather, and season differences. In non-learning based 3D localization, M2DP [24] and Scan-context [28] utilize the histogram of LiDAR projection to achieve long-term 3D re-localization. With the developments of 3D deep feature extraction, recent learning-based approaches have also gotten increasing attention. PointNetVLAD [50] combines the point-based feature extraction and VLAD layer for 3D place recognition. LPDNet [35] extend [50] by including local geometric features. PCAN [63], from another perspective, uses the attention-enhanced VLAD layer to improve feature association for accurate localization. OverlapNet [12] provides a differentiable projection layer to estimate the similarity of local 3D sub-maps. In our previous works, FusionVLAD [58] provides a fusion based approach to improve the features adaptation between different perspectives; and SphereVLAD [55] provides a viewpoint-invariant place descriptor by combining spherical harmonics [56] and sequential matching [40].

Despite the success of existing place recognition methods, the non-learning-based approaches are sensitive to parameter tuning under different scenarios; and learning-based techniques are trained in a supervised learning manner, restricting their generalization ability within the offline training datasets. However, in real-world localization tasks, the data stream is infinite with the combination of different areas under varying environmental conditions; meanwhile, robotic systems can’t stop and wait for the network model to update for newly encountered scenarios. In this work, we target lifelong learning, where the place observations can be viewed only once in the sequential order [16]. In lifelong localization, one important evaluation metric is to analyze the re-localization ability after long-term navigation (i.e., catastrophic forgetting), while most exciting place recognition methods mainly focus on short-term localization or fixed pattern localization [60]. To the best of our knowledge, this is the first work to handle real-world long-term/large-scale lifelong localization.

Lifelong learning, also known as continual learning, aims at providing incrementally updated knowledge in ever-changing environments. Though this area has been studied for a long time, most approaches are still restricted to simulation or toy datasets [16], and can not be applied in real robotic applications. As mentioned in [33], the fundamental challenge for lifelong learning is not necessarily finding solutions that work in the real world but rather finding stable algorithms that can learn in the real world and overcome the catastrophic forgetting problem. Recent works can be roughly divided into four families: dynamic architectures, regularization-based, rehearsal, and generative replay approach.

Dynamic architecture-based methods either 1) add additional parameters to the models, such as LwF [34], which use shared early feature extraction layers and fixed task layers; or 2) use model adaptation to avoid catastrophic forgetting, such as PackNet [38], which defines the mask layer to protect weights when learning new tasks. Regularization-based methods in the context of lifelong learning can add constraints to avoid overfitting to new tasks and keep inference ability for the previous mission, such as Elastic Weight Consolidation (EWC) [29] and Synaptic Intelligence (SI) [61]. However, the above methods must deal with specific network structures and can quickly converge to undesired local optima for complex tasks. Rehearsal-based methods, on the other hand, use memory replays to enhance the knowledge from the previous tasks or processes such as iCaRL [42], GEM [36], which use a small subset of the previous dataset to balance the knowledge distribution for different tasks. Instead of maintaining the knowledge based on past data samples, generative replay [52] combines the actual raw data and generated artificial data for model updating. In [13], the authors use a dual teacher-student generative replay method for incremental learning, where the teacher network is frozen to guide new networks, and the networks will switch the role when the student network surpasses the teacher.

Hence, the ideal approach would be tackling the real-world localization problem in an embodied platform: an autonomous agent that can efficiently and incrementally update its localization ability with limited computation resources. Our BioSLAM method combines rehearsal, general replay mechanisms, and a specific dynamic and static memory to tackle long-term complex environments.

We define a sequence of place observations under domain $D$ (i.e., visual, LiDAR, etc.) as $O^D=\{O_1,...,O_M\}$, and a query of observations under the same domain as $Q^D=\{Q_1,...,Q_N\}$. The task of traditional place recognition is to learn a feature extraction function $F$ with parameter $\theta$ to help each frame in $Q^D$ find the matched (positive) place from the reference sets $O^D$. Let $d(\cdot ,\cdot )$ denotes the difference matrix (i.e. Euclidean distance). The objective is to make the feature differences of positive places (or matched) smaller than negative places (or unmatched) by feature extraction function $F_{{\theta }^{\ast }}$.

where $Q_k^D$ is the current $k$-th query, $O_{\approx }^D$ is the positive reference within a predefined neighbor range (e.g. 3m) near $Q_k^D$, and $O_{\ne }^D$ is the negative reference away from $Q_k^D$ more than a predefined threshold (e.g. 10m).

In lifelong localization, both reference set $O_t^D$ and query set $Q_t^D$ are obtained incrementally, and the environmental domains can also be varying under different environmental conditions (illuminations, weathers, etc.) or sensor modalities. As depicted in Fig. 2, the lifelong localization problem is to incrementally learn and update the feature extraction function $F_{\theta }$, that can quickly adapt its feature extraction ability in the newest domain $\{O^{D_T},Q^{D_T}\}$, and also in parallel maintain the feature distinguish ability for previous domains $\{O^{D_t},Q^{D_t}\}{|}_{t=1,...,T-1}$. Since we are considering continual learning [16], raw data of different domains are fed sequentially for one-time usage and cannot be stored for offline training. Thus, when optimizing feature extraction function $F_{{\theta }_T}$ at the current domain $\{O^{D_T},Q^{D_T}\}$, we cannot access the previous raw data from $\{O^{D_t},Q^{D_t}\}{|}_{t=1,...,T-1}$. Recall Eq. 1, for time step $T$, lifelong localization can be formulated as,

For the lifelong purpose of long-term localization, we developed a General Place Descriptor (GPD) based on our previous works in visual [57] and LiDAR-based [56] localization, which can be referred to as the feature extraction function $F_{\theta }$ as in Eq. 2. We use the shared spherical convolution network to achieve LiDAR and visual place localization simultaneously. The spherical harmonic-based convolution can help the learned descriptor have the viewpoint-invariant propriety for the same place recognition. The major difference between the current GPD and our previous works is that GPD does not contain any domain-transfer module, which has been used to reduce the feature differences for the same areas under different domains [57]. This modification is because we want to evaluate the adaptation ability for the same network. On the other hand, there are no task-specific network layers as used in the dynamic architecture-based lifelong modules as stated in section II-B. We want to avoid uncertain parameters and only focus on how memory mechanisms can help incremental learning for real-world applications.

Inspired by the memory system in human-being and other mammal animals [7], we provide a dual-memory (i.e., dynamic memory zone (DMZ) and static memory zone (SMZ)) enhanced lifelong learning mechanism to deal with catastrophic forgetting in continual localization. As studied in [48], to create long-term memories in our brain, we have so-called sleep circle during our sleep:

• 1) the brain can encoding our daily observation into the hippocampus zone with decay along the time;

• 2) then, a consolidation mechanism is triggered between the hippocampus and the neocortex to store essential memory traces and forget the rest traces;

• 3) finally, humans can retrieve the relative memory traces based on the consolidated ones in the neocortex.

In BioSLAM, we re-build the ‘sleep circle’ for the lifelong localization task. As we can see in Fig. 2, the memory system of BioSLAM also includes the place feature ‘encoding’ procedure for new observations, the memory ‘consolidation’ controlled by a behavior cost module to filter out necessary traces for more extended storage, and the ‘retrieved’ memory to re-enhance the long-term place recognition ability. Based on the above architectural, BioSLAM system construct two major systems, the General Place Learning (GPL) system and the Bio-inspired Lifelong Memory (BiLM) system, which will be investigated in section IV and section V respectively.

GPL applies the encoder module $E$ to convert raw sensor observations into the ‘memory codes’ with VGG [47]-based networks, which are also the basic materials in the BiLM system. In parallel, GPL constructs the decode module mirrored to the encode module, which can reconstruct the stored memory into the synthetic observations. Since in lifelong localization, both viewpoint difference and environmental appearance changes will affect the final localization performance in real-world applications. Based on orientation-equivalent property of spherical harmonics, we utilize the spherical convolution [57, 56] in the encoder module to provide viewpoint-invariant descriptor to reduce the viewpoint differences in long-term re-localization. As shown in Fig. 3, the extracted place feature descriptor is not involved with the BiLM memory system, which indicates that our lifelong localization is mainly designed for the long-term domain differences, which also helps simplify quantitative and qualitative analysis.

The GPL system encodes both panorama camera and 3D local point cloud with the same encoding network structure $E$. For the visual inputs, we convert the raw image to $[H\times W]$ spherical perspectives, which are fed into the encoder module for memory storage and place descriptor extraction. For the LiDAR inputs, instead of a single scan, we generate dense local 3D maps using the similar voxel mapping mechanism in our previous work [58] and map the points onto the spherical projections, which have the same omnidirectional view as a panorama camera. The default size for visual and LiDAR views is $H=W=64$, with $3$ channels for visual and LiDAR inputs (repeated three depth channels ranging from $0$ to $1$). We can obtain the encoded ‘memory’ $z_k$ from observation $q_k$ by,

To extract the orientation-invariant place descriptor from $z_k$, we utilize the spherical convolution based on the spherical harmonics [18]. In theory, Spherical convolution can avoid space-varying distortions in Euclidean space by convolving spherical signals in the harmonic domain. Let $f$ is the signal on spherical harmonic, which satisfy the orientation-equivariant [14] property with the signal $E$,

where $H_R(R\in SO(3\left)\right)$ is the rotation operator for spherical signals. $f{\star }_{SO\left(3\right)}E$ denotes the spherical convolution between $f$ and $E$. Practically, the spherical convolution is computed in three steps. We first expand $f$ and $E\left(q_k\right)$ to their spherical harmonic basis, then compute the point-wise product of harmonic coefficients, and finally invert the spherical harmonics.

Let $V$ be the unsupervised VLAD layer [2]. Then the feature extraction function $F_{\theta }=V\circ \left[f{\star }_{SO\left(3\right)}E\right]$ is the orientation-invariant function. Where $\theta$ are learnable parameters of the feature extraction function. Given the data sample $q_k$, the place descriptor (or learned feature) $F_k$ can be denoted as

For details about the viewpoint-invariant analysis, please refer to our previous works [57, 56].

The above procedure is relevant to the biological ‘encoding’ procedure within the ‘sleep cycle’ as we mentioned section III, and the extracted ‘memory codes’ $z_k$ will be used for later ‘memory consolidation’ in next section V and ‘retrieval’ for generative replay in next section IV-B

As depicted in Fig. 3, in our generative memory replay for the lifelong localization task, the data stream under different environmental conditions is fed into the system sequentially, and we generate synthetic samples from stored place memories through a deep generative memory replay framework. In particular, the retrieved ‘memories’ will include the abstracted place latent codes under different domains as shown in Fig. 4, which enforce the generative memory play to extract a portion of history samples under all different domains to maintain the localization performance. The next section relates the memory extraction mechanism to the BiLM memory management.

To ensure the generalization ability of synthetic samples, we provide a deep generative adversarial network (GANs) to mimic the distribution differences between the raw data and synthetic samples and parallel with a $L1$ reconstruction loss between encoder and decoder modules. The GANs-based generative model defines a zero-sum minimax game with the memory decoder $G$ and the discriminator $D$ as stated in [22], the objective function is thereby defined by,

where $P_z$ is the retrieved memory buffer from the BiLM system, and $P_{data}$ is the new observed data samples. $L_{gan}$ denotes the generator and discriminator losses [22]. The detailed generative play strategy for lifelong learning can be found in [46, 51].

Given the combination of both new streaming data and retrieved synthetic samples $\{P_{data},P_z\}$, we can obtain observation pair $\{Q,O\}$ with tuple sets $(q_k,\{o_k^{pos}\},\{o_k^{neg}\left\}\right)$, where for each query sample $q_k$ we have a set of potential positives (close-by samples ) $\left\{o_k^{pos}\right\}$ and the set of negatives (far away samples) $\left\{o_k^{neg}\right\}$. The localization loss metric is defined by:

the above equation is the triplet loss version of Eq. 1, where $L_{loc}\left(q_k\right)$ is the localization loss metric for single query $q_k$, ($\alpha >0$) is a margin to control the feature difference threshold, and $L_{loc}$ is the localization loss for the joint $\{P_{data},P_z\}$ sets.

To keep the consistency of memory encoding-decoding, we further use a reconstruction loss of the memory $z^{\prime }$ and the generated memory $E\left(G\right(z^{\prime }\left)\right)$ with,

And the joint loss metric for the generative memory replay enhanced place recognition can be written as,

The major difference between our work and the traditional generative play [46] is that BioSLAM can manage the retrieved memory based on their long-term behavior instead of treating all data on the same manifold distribution. The next section will deeply investigate the lifelong memory system.

When the memory system encounters a new place ‘memory traces’ $z$, we define the hybrid cost to control the learning behavior: an external reward $R_{ex}$ which indicates localization ability, and the internal reward $R_{in}$ which can present the intrinsic familiarity on observations.

The memory of human beings is highly connected with long-term memory (the neocortex) and short-term memory (the hippocampus) mechanisms within our brains. BioSLAM also constructs such paired dual-memory mechanisms,

• Static Memory $M_S$ is similar to the long-term memory of human beings and belongs to rehearsal-based mechanisms with large storage for lifelong learning. Static memory stores the selected memory traces $\left\{m_k\right\}$ to the static memory zone by memory consolidation.

• Dynamic Memory $M_D$ is similar to the short-term memory of human beings, which is a quick access memory with a portion of pre-stored historical memory traces. Dynamic memory is automatically refreshed from the static memory and connected with the memory decoder module, which belongs to generative replay mechanisms with small memory buffer for lifelong learning.

Based on the dual-memory structure, we construct two important operations for static memory: memory consolidation and forgetting, and two important operations for dynamic memory: memory refreshing and memory replay.

Fig. 5 shows our data collection platform, which includes an omnidirectional camera, a Velodyne VLP-16 LiDAR device, an inertial measurement unit (Xsense MTI $30$, ${0.5}^{\circ }$ error in roll/pitch, $1^{\circ }$ error in yaw, $550m$W), and an embedded GPU device (Nvidia Xavier, $8$G memory). To collect time-synced LiDAR projection and omnidirectional images, we first generated dense 3D maps through well-known LiDAR odometry [62]. Then project the point cloud within a certain distance (default is $30m$) to the spherical projections, which have the same perspective as the omnidirectional images. We will revisit the same area under large-scale and long-term assumptions in lifelong localization. To provide the relative ground truth position between different visits: to outdoor environments, we rely on the GNSS system and Generalize-ICP [44] to estimate the relative transformation; For indoor environments, we mainly rely on Generalize-ICP. Please note that we cannot guarantee the meter-level global absolute localization, but we can provide accurate relative localization, which is enough for the lifelong localization task. Based on the collected datasets, we have hosted a General Place Recognition Competition for long-term place recognition. For more details on the data collection platform and the datasets, please refer to our dataset paper(https://github.com/MetaSLAM/ALITA) and competition site (http://gprcompetition.com/).

We intend to analyse the lifelong performance under large-scale and long-term two perspectives. To this end, our localization datasets include two tracks:

• City dataset: shown in Fig. 6, which is targeting at large-scale lifelong performance. We collected $50$ trajectories within the city of Pittsburgh. Since we mainly care about large-scale localization, we only collected the LiDAR inputs within a short-term drive. The total trajectory distance for this dataset is $110$km.

• Campus dataset: shown in Fig. 7, which is targeted at long-term lifelong performance. We picked up $10$ trajectories within Carnegie Mellon University. Each trajectory is revisited by $8$ times under different day- and night- time to satisfy the long-term requirements.

For both datasets, we feed the sequential data stream into the BioSLAM training procedure as depicted in Fig. 2. Please note that each data will be only fed into the system once, and BioSLAM will not save the copy of that data sample. For Campus dataset, the LiDAR inputs, day- and night- visual inputs are fed into the system one by one. For City dataset, we will only feed the continuous LiDAR inputs to the system.

To evaluate the localization performance on the large-scale City dataset and long-term Campus dataset in incremental learning, we divide the trajectory into individual trajectory segments and feed them into different place recognition systems in a sequential manner. We evaluate the online localization performance mainly through Weighted Recall (WR) of top-6 retrievals over incremental training, which is defined by $\text{WR}={\sum }_{k=1}^6{\omega }_k{r}_k,{\omega }_1=0.5,{\omega }_k=0.1fork\ne 1$, where $r_k(1\le k\le 6)$ denotes the recall@k. Recall@k is the proportion of matched references found in the top-k retrieval. Specifically, the query image is deemed correctly localized (matched) if at least one of the top $k$ retrieved reference images is within the predefined neighbor range (e.g. 3m) from the ground truth position.

Since our place recognition task involve different sensor modalities, non-learning/learning methods, and lifelong/non-lifelong methods, it is impossible to cover all the relevant state-of-the-arts. We focus on the performance comparison from a 2D perspective and ignore Point-like [50] 3D methods. As a comparison, we select the following well-known non-learning methods (Bag-of-wards (BOW) [20], CoHOG [59]), learning-based methods (NetVLAD [2], RegionVLAD [27]) and lifelong-based methods (Generative Replay (GR) [45], Synaptic Intelligence (SI) [61]). Among the above methods, GR and SI are the most related and important baselines to BioSLAM. Although BioSLAM and GR both use memory replay, BioSLAM has more efficient and effective memory replay mechanisms. Because 1) BioSLAM replays samples according to reward (importance), while GR replays randomly and evenly. 2) BioSLAM has static memory to refresh the dynamic memory buffer to keep diverse and important memory traces, as well as easier to adapt to new trajectory observations.

We evaluate the performance of the BioSLAM in a large scale lifelong learning scenario with the City dataset. For the localization task under city-scale environments, robots may encounter multiple types of 3D geometric structures within the urban environments, such as open-street, bridges, parks, big buildings, and residential areas. We divide the $50$ trajectories with $120$km distance within the city into $3$ different areas based on their geometric properties: area 1 for commercial buildings, area 2 for parks, and area 3 for residential districts. For the large-scale City dataset, observations from different areas can be treated as different domains $D_t$ in Eq. 2.

In the training procedure, we incrementally feed the place recognition methods with trajectory observations from 3 different areas. Fig. (a)a shows the weighted recall curve of trajectory observations within area 1, area 2, and area 3, respectively. Fig. (b)b shows the average weighted recall curve of all trajectories during training. As can be seen, BioSLAM outperforms other methods during training and is at least $14\%$ better than other baselines in terms of final average recall. More importantly, BioSLAM could keep the knowledge about previous trajectory observations when trained with new trajectory observations. For example, at epoch 240, when the training observations switched from area 2 to area 3, the performance drop on previous trajectories for BioSLAM is much smaller than in other methods, as shown in Fig. (b)b. Because BioSLAM retrains important previous knowledge by replaying related memory traces. Note that, BioSLAM replays important and highly rewarded memory traces, while GR only replays randomly. Thus BioSLAM has a much higher convergence rate and final performance than other baselines.

After training, we evaluate the generalization ability of the final trained model on the fixed test set. Fig. 9 shows the comparison between BioSLAM and other baselines in terms of top-$k$ recall on the test set of City dataset. As can be seen, although BioSLAM learns incrementally, it still performs better than baselines on classic (non-lifelong learning) offline test set evaluation, while some non-lifelong learning methods are designed or trained for offline evaluation.

Training place recognition models on independent domains are inefficient because no information will be shared. We thus demonstrate the merit of BioSLAM in more reasonable settings where the model benefits from solving place recognition from multiple domains (Lidar, day-time vision, night-time vision). A place recognition model operating in multiple domains has several advantages. First, the knowledge of one domain can help better and faster understand other domains, because the domains are not completely independent in place recognition tasks. Second, generalization over multiple domains may result in more universal knowledge that applies to unseen domains. Such phenomenon is also observed in infants learning [4, 9].

We evaluate the performance of BioSLAM on long-term and cross-domain lifelong learning scenario with Campus dataset. In the training procedure, we incrementally feed the place recognition methods with trajectory observations from different domains (ordered with Lidar, day-time vision, night-time vision), and evaluate the performance on all domains. Observations from Lidar, day-time vision, and night-time visual signals can be treated as different domains $D_t$ in Eq. 2.

Fig. (a)a shows the performance comparison between BioSLAM and baselines on different domains of the Campus dataset. In $0\sim 600$ epochs, we train the place recognition model on the Lidar domain and the performance of all methods on all domains increases within 600 epochs. This verifies that the knowledge of one domain can help better and faster understand other domains. In $600\sim 1200$ epochs, we train the model in the day-time visual domain. For all methods, the performance in the day-time visual domain increases but the Lidar performance decrease around the switching point of epoch=600. This is reasonable because the background of Lidar and day-time visual images are totally different. In the Lidar domain, the performance drop of BioSLAM is much smaller than in other methods. This shows that BioSLAM could learn observations in a new domain without forgetting observations from past domains. In $1200\sim 1800$ epochs, we train the model in the night-time visual domain. In addition to the performance increases in the night-time visual domain, the performance of BioSLAM also increases in the Lidar domain. With efficient replay mechanisms of BioSLAM, the knowledge of one domain can help better understand other domains. The average weighted recall of all domains is shown in Fig. (b)b. As can be seen, the performance of all methods is similar in the beginning, but as new observations from new domains are added, BioSLAM converges faster and better than baselines and outperforms other methods by at least 10% in terms of final average recall.

After training, we evaluate the cross-domain generalization ability of the final trained model on the fixed test set. Fig. 11 shows the comparison between BioSLAM and other baselines in terms of top-$k$ recall on the test set of the Campus dataset. As can be seen, although BioSLAM learns incrementally, it still performs better than baselines on classic (non-lifelong learning) offline test set evaluation, while some non-lifelong learning methods are designed or trained for offline evaluation.

Table II shows the comparison between different methods both on the fixed test set of City and Campus datasets. In City dataset, BioSLAM outperforms a state-of-the-art lifelong learning method GR by 7.6 %, and a non-learning method CoHOG by 5%. In Campus dataset, BioSLAM outperforms a lifelong learning method GR by 19.8 %, and a non-learning method CoHOG by 7.2%. Note that, this paper focuses on incremental and lifelong learning scenarios, so the most important evaluation metric is the recall curve over incremental learning (as shown in Fig. 10 and 8). For a recall on the fixed test set, some non-lifelong learning methods (i.e. CoHOG) may perform very well, but these methods can not learn incrementally, so the performance of the non-lifelong learning methods is limited. Thus lifelong learning methods have a higher potential in wider and changing real environments.

As mentioned in section V, BioSLAM has several novel mechanisms that differ from previous lifelong learning methods: (1) external reward $R_{ex}$ to indicate localization performance; (2) internal reward $R_{in}$ to indicate the robustness of feature representation; (3) Static memory consolidation to abstract concise memory traces, and clustering (Eq. 14) is the key of memory consolidation; (4) Dynamic memory refreshing to effectively replay important memory, in which, the time-decay mechanism (Eq. 18) for importance weight is critical. We further evaluate the effectiveness of the above mechanisms of BioSLAM by comparing BioSLAM with the following variants: (1) w/o $R_{ex}$: without applying external reward, then Eq. 13 becomes $R_k=R_{in}\left(q_k\right)$; (2) w/o $R_{in}$: without applying internal reward, then Eq. 13 becomes $R_k=R_{ex}\left(q_k\right)$; (3) w/o consolidation-clustering: without using clustering in static memory consolidation, then Algorithm 1 becomes directly storing all memory traces in static memory; (4) w/o time-decay: without using the time decay factor in importance sampling, which is equivalent to set $\gamma =1$ in dynamic memory refresh Eq. 18. Note that these variants follow the control variates method. They cover all important mechanisms of BioSLAM without overlapping functionalities.

The results for the ablation study on City dataset are shown in Fig. (a)a, and the results on Campus dataset are shown in Fig. (b)b. As can be seen, BioSLAM outperforms its variants on both City and Campus datasets. The removal of any component leads to a significant performance drop. In particular, the performance grain of BioSLAM with respect to “w/o time-decay” validates the necessity of decay weights (rewards) of importance sampling during dynamic memory refresh. The larger performance drop is caused by removing internal rewards, which means the indicator (internal reward) for the robustness of feature representation is critical to retrieving memories. The performance of “ w/o cluster-consolidation” is close to BioSLAM, but BioSLAM is much memory efficient by clustering and downsampling.

In this section, we visualize and evaluate the BioSLAM learned features (place descriptor, Eq. 5) $F\left(q_k\right)$ with similarity matrix and Principle Component Analysis (PCA). Similarity Matrix $M_{\text{sim}}$ is defined by the cosine similarity between reference $O^D$ and query $Q^D$ features, with $M_{\text{sim}}(i,j)=\cos \left(F\right(O_i^D),F(Q_j^D\left)\right)$. A high-contrast similarity matrix indicates that the learned feature $F$ has a strong expression and discrimination ability.

The similarity matrix of BioSLAM over training on City dataset is shown in Fig. (a)a. The left column represents the sampled trajectories from area 1, area 2, and area 3. The right three columns represent the similarity matrices of the corresponding trajectories (from left to right) after incrementally training on area 1, area 2, and area 3, respectively. After training on area 1, the similarity matrices of all areas increase contrast. Then training on area 2 and area 3, the similarity matrix of area 1 is almost non-decayed. That means BioSLAM still has strong expression ability on past trajectories when learning from different areas.

The similarity matrix of BioSLAM over training on Campus dataset is shown in Fig. (b)b. The left column represents the sampled observations from Lidar, day-time visual, and night-time visual inputs. The right three columns represent the similarity matrices of the corresponding observations (from left to right) after incrementally training on Lidar, day-time visual, and night-time visual domains, respectively. After training on the Lidar domain, the similarity matrix of the Lidar observations increases contrast. Then incrementally training the model on day-time and night-time visual domain, the similarity matrix of the corresponding domain become more contrastive, and the similarity matrix of Lidar almost does not decay. That means BioSLAM could remember the past domains when learning from totally different domains.

We used PCA to reduce the dimension of BioSLAM learned features to 2d. The PCA visualization of learned features of observations from different areas on the City dataset is shown in Fig. (a)a. The subfigures from left to right represent PCA visualization results at the initial step, and after incrementally training on area 1, area 2, and area 3. As can be seen, with BioSLAM training, observations within the same area are almost clustered together. The clusters of the different areas become easier to discriminate over incremental learning.

The PCA visualization of learned features from different domains and trajectories on the Campus dataset is shown in Fig. (b)b . (For clearer visualization, we only visualize three trajectory segments in each domain). The sub-figures from left to right represents PCA visualization results at the initial step, and after incrementally training on Lidar, day-time visual, and night-time visual domains. As can be seen, BioSLAM not only differentiates different domains (3 clusters from left to right) but also differentiates different trajectories within each domain. As shown in the right sub-figure in Fig. (b)b, the same trajectories of different domains are relatively close. As an example, for trajectory 1, the PCA results of the Lidar domain and night-time visual domains are close to each other and located in the lower part of the PCA visualization results. That means, BioSLAM could incrementally learn trajectories from different domains, and it has the potential to find the cross-domain relationship of place observations from different domains.

As described in section V-B, static memory $M_s$ stores selected memory traces by memory consolidation. Because clustering and downsampling are based on feature and spatial property (Algorithm 1), the static memory stores concise and diverse observations in terms of feature and spatial property. Then dynamic memory $M_d$ samples memory traces from static memory with importance sampling, and the sampling weight is proportional to its reward value.

The memory traces in the dynamic memory zone has a direct impact on lifelong learning performance because the replayed samples from dynamic memory are used for training at every iteration. In this section, we visualize the dynamic memory zone to see the proportion of memory traces from different domains or trajectory segments. Note that, if memory traces of a domain have higher rewards than other domains, then the dynamic memory zone holds more memory traces (samples) from the domain. Thus, to better understand the proportion between different domains, we also visualize the (normalized) reward ratio of each domain or trajectory segment. For trajectory $i$, $\text{reward\_ratio}\left(i\right)=\frac{{\overline{R}}_{\left(i\right)}}{{\sum }_j{\overline{R}}_{\left(j\right)}}$, where ${\overline{R}}_{\left(i\right)}$ is the average reward for observations from trajectory $i$.

For the City dataset, the proportion of observations from different trajectory segments in the dynamic memory zone is shown in Fig. (a)a. As the new trajectory segments incrementally feed into BioSLAM (every 60 epochs), the trajectory diversity in the dynamic memory zone increases. Fig. (b)b shows the reward ratio of different trajectories of training. As can be seen, the proportion of different trajectories in the dynamic buffer is consistent with the reward ratio of corresponding trajectories. Because given a trajectory segment, a higher reward means worse performance, BioSLAM uses higher sampling weights to retrieve more memory traces from the high-rewarded trajectory to achieve better performance. As an example, from epoch 360 to 420, the reward ratio of trajectory 1 increases in Fig. (b)b, and dynamic memory samples more memory traces of trajectory 1.

For the Campus dataset, the proportion of observations from different domains in the dynamic memory zone is shown in Fig. (a)a. Fig. (b)b shows the reward ratio of different domains of training. As can be seen, the proportion of different domains in the dynamic buffer is consistent with the reward ratio of corresponding domains. As an example, in the final step, the reward ratio of the night-visual domain is lower than other domains in Fig. (b)b, which means BioSLAM already achieves better performance in the night-visual domain. Then the dynamic memory tends to retain only a small amount of night-visual memory traces, leaving valuable memory zone for other high-rewarded domains (i.e. LiDAR).

Thus, we have the following relationship between dynamic memory zone and rewards: the performance on a domain (or trajectory) $i$ is lower $\to$ higher reward on the domain (or trajectory) $i\to$ more memory traces of domain (or trajectory) $i$ in dynamic memory $\to$ training on more replayed samples from the domain (or trajectory) $i\to$ the performance on the domain (or trajectory) $i$ may increases. The above relationship between dynamic memory and rewards can serve as feedback compensation.¹¹1Note that, the performance on the domain (or trajectory) $i$ may be saturated and not increase. But the relationship still works as feedback compensation.

To illustrate the incremental learning property, we evaluate the BioSLAM with a confidence map during training. The confidence score of a place is defined by the cosine similarity between the corresponding reference $O_k$ and query $Q_k$ features, with $\text{confidence}\left(k\right)=\cos \left(F\right(O_k),F(Q_k\left)\right)$. A higher $\text{confidence}\left(k\right)$ denotes the robust feature representation of place $Q_k$, because the feature representations of query and reference in the same place are similar. The confidence map is measured by calculating the confidence score for all observations and visualizing the confidence scores on real trajectories.

The confidence map of BioSLAM over training on City dataset is shown in Fig. (a)a. The left column represents the sampled trajectories from area 1, area 2, and area 3. The right three columns represent the confidence map of the corresponding trajectory (from left to right) after incrementally training on area 1, area 2, and area 3, respectively. After training on area 1, the confidence map of all areas becomes better. Then training on area 2 and area 3, the confidence map of area 1 is almost non-decayed. That means training BioSLAM on a trajectory also helps to improve the performance of others.

The confidence map of BioSLAM over training on Campus dataset is shown in Fig. (b)b. The left column represents the sampled observations from Lidar, day-time visual, and night-time visual inputs of the same trajectory. The right three columns represent the confidence map of the corresponding domains after incrementally training on Lidar, day-time visual, and night-time visual inputs of the trajectory. After training on the Lidar domain, the confidence map of the Lidar domain increases. Then incrementally training the model on the day-time and night-time visual domain, the confidence map of the corresponding domain increases, and other domains almost do not change. That means BioSLAM can remember past domains when learning from other domains.

In this section, we introduce memory and time usage for lifelong learning. For both methods, we evaluate an Ubuntu 18.04 system, using an Nvidia RTX 2080 Ti (12 GB) graphics processing unit (GPU), Intel Core i9-7900x processors, and $64$ gigabyte (GB) memory. Table III shows the total memory usage on City datasets. The GPU memory usages of BioSLAM are acceptable under the current embedded system structure.

Fig. 18 shows the time usage of the BioSLAM lifelong learning procedure on the City dataset when incrementally feeding new trajectory segments. The distance for each segment is about 2km and is composed of about 200 observation frames. The data inference procedure takes $<1$s for each trajectory segment, which is efficient in real-world inference. The average time for memory consolidation and forgetting is around $40$s for each trajectory, which is fast enough to analyze the newly captured memory traces, and update the memory system. And the memory replay takes $2$s to generate replayed samples for place recognition training. Finally, place recognition optimization takes $40$s to optimize a $2$km new trajectory in one epoch. For multi-epoch training, BioSLAM runs inference, replay, and optimization multi-times, but only runs memory consolidation once. In general, training a trajectory segment about 50 times can get convergence results. So the total learning time for a 2km trajectory segment is $40s+(1+2+40)\ast 50s=2190s\approx 36min$. Given that the distance between neighbor keyframes is $10$m, in this case, BioSLAM can learn $100$m new areas in around 1.8 minutes.

We can note the most important property, the time usage in the above memory operations will not be affected by scale differences in either spatial or temporal. This is mainly benefited by our memory forgetting mechanism, which can maintain the searching space of $M_S,M_d$ and keep up-to-date memory traces to balance the localization. The above properties indicate that BioSLAM can be applied to low-cost robotic systems for long-term place recognition tasks on NVIDIA embedded systems.