Dynamic Memory-based Curiosity: A Bootstrap Approach for Exploration

In RL, the exploration issue is a long standing challenge. Previous attempts suggest that: if there is no additional reward, exploration can be regarded as a hunt for information theoretically, which also can be viewed as the curiosity Berlyne (1950); Schmidhuber (1991); Kidd and Hayden (2015); de Abril and Kanai (2018); Jaegle et al. (2019); Friston et al. (2016); Peterson and Verstynen (2021). One intuitive formulation of curiosity is the count-based methods, where the less visited state has more state novelty for exploration. But it can not scale to large-scale or continuous state spaces Kearns and Singh (2002); Charikar (2002). Inspired by count-based methods, RND calculates the state novelty by distilling a random fixed network (target network) into another prediction network (predictor network). The predictor network is trained to minimize the prediction error for each state and take the prediction error as the intrinsic reward. Apart from count-based methods, prediction-based methods also show competitive or better performance by modeling the environment dynamics Pathak et al. (2017, 2019); Kim et al. (2020); Burda et al. (2018). With the assumption that more visited state-action pairs will result in more accurate prediction, the intrinsic reward can be applied as the variance of predictions of ensembles or the distance between prediction states and true states, such as the Disagreement method Pathak et al. (2019) and ICM Pathak et al. (2017) method. There have been few attempts to design a curiosity that contains memory and effectively uses information consolidated in memory, which however is the main goal of this paper.

Our work is also related to the uncertainty estimation, as uncertainty is crucial which allows an agent to discern when to exploit and when to explore its environment in RL Szepesvári (2009). Previous intrinsic rewards can also be interpreted from the perspective of uncertainty estimation, which can evaluate curiosity by estimating the deep learning model’s uncertainty (confidence). Take Disagreement as an example, instead of comparing the prediction to the ground-truth, they suggest to evaluate the uncertainty of multiple prediction models using the deep ensemble Dietterich (2000), despite incurring additional computation. RND also claims that the distillation error can be viewed as a quantification of the uncertainty. Unlike RND, in our work, we evaluate the uncertainty of given states though measuring the information gap between dual learners which rely on dynamic memory instead of a random fixed network.

In our framework, if the information in current memory cannot handle the encountered state, then the curiosity is aroused. We model the memory as a learnable neural network, but there is a dilemma that we do not have a “benchmark” encoded network in the parameter space to encode the encountered state accurately, since not enough supervised signal provided here. Thus it is difficult and not sound to directly define the curiosity by comparing a random encoded state with the output latent state from a dynamic memory network. We instead introduce dual online learners for state encode representations. These dual learners have the same network architecture as the memory network but with each own different parameters. In their network parameter space, the dual networks are supervised by the memory encoding ability. And then our curiosity can be defined by the gap of the encodings of the same state output by dual learners’ networks. The intuition of our dual learners is: if the state information has been squeezed out by the memory, then the memory can completely know and resolve the state, and the dual learners which can be seen as the two imitators to the memory are easier to get the similar encodings to the current state. In other words, if one state is little known by the memory, then the dual learners may produce quite different encodings to it, which represents the larger information value $E$ and thus stimulates the agent to explore this state. Here, for the uniform description, we refer the encoded states and encodings to the latent states, which reflect the cognition of states by the memory and learner networks.

In our implementation, such kind of latent gap $E$ will spark the curiosity as the intrinsic reward for agents exploration. After the RL agents learning, such information will be consolidated to the memory for memory better growing. In terms of the consolidation method $g$, it is externalized as updating the memory parameters via the exponential moving average (EMA) Haynes et al. (2012); Klinker (2011); Grebenkov and Serror (2014) of the dual learners’ parameters.

From such analysis, we see the dual learners first learn based on the memory network for measure the information value for exploration, and then the memory network consolidate information gain based on dual learners in the EMA way. The memory is actually seeking for the appropriate position in the parameter space dynamically, in order that its network can better characterize the memory and cognition ability of the seen states in environments. In a word, our dynamic memory is updated in a bootstrap Grill et al. (2020) way. Figure 1 and algorithm 1 present the whole framework and pseudo-code of DyMeCu.

• Learning of Dual Learners:

Dual online learner models $f_{{\theta }_1}$ and $f_{{\theta }_2}$ are defined by a set of weights ${\theta }_1$ and ${\theta }_2$ with the same architecture as the memory network $M_{\omega }$. The memory provides the regression targets for the learning of dual learners $f_{{\theta }_1}$ and $f_{{\theta }_2}$. Given a current state $s_t$, the learners transform it into the latent states $z_t^{{\theta }_1}\triangleq f_{{\theta }_1}\left(s_t\right)$ and $z_t^{{\theta }_2}\triangleq f_{{\theta }_2}\left(s_t\right)$ respectively, and the memory network outputs $z_t^{\omega }\triangleq M_{\omega }\left(s_t\right)$. The mean squared error (MSE) between them is:

Based on $L_{{\theta }_1}$ and $L_{{\theta }_2}$, the dual learners are updated as :

where $\mathrm{optim}$ and $\eta$ represent the optimizer and learning rate.

• Intrinsic Reward based on Curiosity:

The curiosity relies on the information value of current state. In our method, such information value can be measured by the information gap between dual learners. This information gap can also be considered following the $\delta$-Progress  Achiam and Sastry (2017); Graves et al. (2017) to form the curiosity. We obtain the intrinsic reward to agents based on the curiosity from the information value:

From another point of view, the dual-learner mechanism can be regarded as the variant of ensemble Mai et al. (2022) for uncertainty estimation. Compared with previous attempts which requires heavily ensembling (such as the Disagreement), our lightweight solution can previous better performance while retaining computation efficiency.

Overall, we can get the optimization goal for the agent:

where $\gamma$ is the discount factor and $\varphi$ represents parameters of policy $\pi$; $\zeta$ and $\beta$ are the coefficients of the intrinsic reward and extrinsic reward respectively.

• Consolidating Information into Memory:

The memory model is updated in an EMA way for sake of its stability to the old state information and the plasticity to the current new state information. In other words, the memory is dynamically growing taking the contextual environment information into account. Specifically, given a decay rate $\alpha \in [0,1]$ and after each training step, the memory $M_{\omega }$ can be updated as:

• Intuitions on DyMeCu’s behavior:

The dynamic memory-based curiosity is closer to human curiosity mechanism. It is the cognitive difference compared to the memory that stimulates our curiosity to explore the world, and then we will consolidate the cognition information to the memory dynamically. In addition, from the knowledge distillation view, such memory can also be regarded as the teacher model in Mean Teacher-based approach Tarvainen and Valpola (2017). The memory is essentially a self-ensemble of the intermediate models of learners. The paradigm we proposed is one type of replay mechanism that is thought to play an important role in memory formation, retrieval, and consolidation Hayes et al. (2021). Moreover, we consider our way to form the memory can also be used in the continual learning to address the issue of catastrophic forgetting Arani et al. (2021).

We evaluate our method in both pre-training and traditional RL situations utilizing two widely used benchmarks: DeepMind Control Suite (DMC) Tunyasuvunakool et al. (2020) and Atari Suite Bellemare et al. (2013). We follow the RND Burda et al. (2019) experimental settings for Atari Suite and settings of URLB Laskin et al. (2021) which is the pre-training benchmark for DeepMind Control Suite. We apply PPO algorithm Schulman et al. (2017) to train the agent. The hyper-parameter $\alpha$ was set as 0.99 in all experiments, and the implementation details and hyper-parameters can be found in the appendix.

Many well-performing approaches like URLB Laskin et al. (2021) use the pre-training and fine-tuning paradigm to improve sample efficiency for RL, especially in the experiment benchmark like DMC containing various domains and complex tasks. We evaluate DyMeCu on all three domains of DMC, namely Walker, Quadruped, and Jaco Arm (from easiest to hardest), and each of them has four tasks. During the pre-training phase, the agents are trained for 2 million steps with only intrinsic rewards produced by the curiosity. During the fine-tuning phase, the agents are trained for 100k steps with only extrinsic rewards.

Table 1 reports the final scores and standard deviations of DyMeCu and other competitive methods. We compare DyMeCu with both intrinsic reward-based methods (ICM, Disagreement, RND, APT Liu and Abbeel (2021b)) and other pre-training strategies (ProtoRL Yarats et al. (2021), SMM Lee et al. (2019), DIAYN Eysenbach et al. (2018), APS Liu and Abbeel (2021a)). DyMeCu improves average performance by 18.3%, 26.6%, and 21.0% on these three domains respectively. From the quantitative results, we can see our DyMeCu achieve the new state-of-the-art across all 12 tasks, demonstrating DyMeCu’s ability to improve the model performance and robustness through pre-training paradigm. Figure 2 plots 6 learning curves (fine-tuning phase) of DyMeCu and three competitive curiosity-based methods. All learning curves can be found in the appendix. DyMeCu shows a superior convergence speed than other methods. Meanwhile, the convergence result of DyMeCu also surpasses others significantly. DyMeCu’s speed increase may sbe mainly due to the contextual state information being consolidated into memory dynamically, rather than a random fixed setting like the RND. Based on the dynamic memory, the exploration of agents can be much more efficient.

For the Atari suite, we first record the performance of agents with both intrinsic and extrinsic rewards. The experiments conducts 100M running steps - equivalent to 400M frames and the intrinsic and extrinsic rewards coefficients are set to $\zeta =1$ and $\beta =2$ respectively for all methods, following the setup of the previous curiosity-based methods. Table 2 lists the aggregate metrics and scores of three methods trained with both intrinsic and extrinsic rewards on the Atari 26 games. Human and random scores are adopted from Hessel et al. (2018). As done in previous works Liu and Abbeel (2021b); Yarats et al. (2020); Schwarzer et al. (2020), we normalize the episode reward as human-normalized scores (HNS) by expert human scores to account for different score scales in each game. #SOTA denotes the number of games that the current method exceeds other methods and mean HNS is calculated as the average of $(\text{agent score}-\text{random score})/(\text{human score}-\text{random \% score})$ of all games. From Table 2, DyMeCu displays the superiority over Disagreement and ICM with its highest mean HNS and #SOTA.

Figure 3 displays the learning curves using both intrinsic and extrinsic rewards. We compare DyMeCu with three widely-used baselines, including Disagreement, ICM and RND, on 6 random chosen Atari games. DyMeCu shows evident advantages in most games on the performance and learning speed. For example, on Jamesbond, the convergence plot reward of DyMeCu is more than three times that of other methods. Moreover, we also compare the performance of agents trained with only intrinsic rewards. As shown in Figure 4, of the 6 environments, DyMeCu outperforms Disagreement baseline in all environments, outperforms ICM and RND baselines both in 4 environments. Overall, the results in Atari Suite show that DyMeCu outperforms other curiosity-based methods, demonstrating DyMeCu’s ability to generate more accurate intrinsic rewards and provide more useful information for better exploration.

Further analysis including ablation studies on DyMeCu are presented to give an intuition of its behavior and performance. We run the experiments across 3 random seeds and all following experiments conducts 50M running steps - equivalent to 200M frames.

• Dual learners:

Here we explore to design the curiosity under the naive setting, that is, using one encoding network to learn to encode the latent space, and thus the curiosity-based intrinsic reward can be defined as the gap with the memory network:

The memory is updated with $\omega \leftarrow \alpha \omega +(1-\alpha )\theta$. As shown in Figure 5, one-learner mechanism does not show significant advantages over other methods, whereas our dual-learner mechanism performs much better with more accurate curiosity and corresponding intrinsic rewards.

• Update of memory network:

The memory network in DyMeCu is updated with dual learners, we additionally evaluate the performance of DyMeCu when the memory is updated using only one of the learner’s parameters. The results in Table 3 indicate that both learners can consolidate state information into the memory well. Combined with Figure 5, it is useful and necessary to assign and train dual learners, and then we can update the memory with dual or one-learner, while dual-learner update mechanism shows a little superior performance.

• Structure of learners:

The bootstrap idea has been explored and used in some previous researches. The most similar one to ours is BYOL Grill et al. (2020), which uses the bootstrap method for self-supervised learning in computer vision. Furthermore, Grill et al. add another predictor module to the online network, and compare the output of predictor to the target network, and it is the key to generating well-performed representations Chen and He (2021). Similarly, in this ablation study, we also design the controlled trials, in which additional two convolution layers are added to each of dual learners. In Table 3, we can find that such learnable additional module does not lead to significant improvement. Under our analysis, unlike previous work using the bootstrap method, we aim to generate the intrinsic reward by calculating the information value (i.e., information gap between dual learners) as accurate as possible, instead of better representations for downstream tasks.

• Robustness to hyper-parameter $\alpha$:

There is a concern of the updating speed of memory network in the EMA way. It is about how much and how fast to accept and consolidate the new environment information. Therefore, to further analyze the updating effect of the hyper-parameter $\alpha$, we evaluate DyMeCu with different values of $\alpha$ in a rational interval, and we assess the agents’ performance in three different Atari games: Alien, Kangaroo, and Krull. For more direct and visual comparison, we normalize the episode reward of DyMeCu as baseline-normalized scores (BNS) which is calculated as the average of $(\text{DyMeCu score}-\text{random score})/(\text{baseline score}-\text{random \% score})$ where the baseline score is the average score of baselines. As illustrated in Figure 6, all values of the hyper-parameter $\alpha$ between 0.99 and 0.9999 yield satisfied performance, generally greater than twice that of the baseline average. DyMeCu shows acceptable robustness to the updating hyper-parameter.