Monocular Camera-based Complex Obstacle Avoidance via Efficient Deep Reinforcement Learning

Collision avoidance is a challenging research problem for robot navigation, which has been studied for decades and it can be applied in many important applications in real-world scenarios, e.g., unmanned driving [1] and freight logistics [38]. In recent years, deep reinforcement learning (DRL) based methods [42, 27, 6, 13] have been studied to address the collision avoidance problem in the robot system. Compared with deep learning-based methods [36, 8, 5, 57] the DRL-based methods can collect a large amount of data based on simulation without relying on manually-labeled data. Specifically, in virtual environments, the agent can be simulated to navigate, continuously interact with the environment in a ‘trial-error’ manner, and obtain the environmental feedback for learning. Therefore, in practice, their collision avoidance policies can be trained in simulation and then migrated to real robots.

As the major limitation of learning policy using simulation data, the large gap between simulation and real-world makes it difficult to directly migrate the learned policy from virtual agents to real robots. In order to reduce this gap, one feasible solution is to apply lasers to sense the surrounding environment and formulate the observation as the obstacle distance information of discretized orientations in the form of one-dimensional vector [27]. Such laser-based observation possesses the advantage of accuracy and simplicity, while it not only ignores the discrepancy of texture and color between the simulation environment and the real environment, but also contains less redundant information than image data, which benefits the training of DRL frameworks.

However, laser sensors notoriously have several shortcomings, i.e., one-dimensional laser observations cannot fully exploit the scene semantic information, and they are difficult to perceive the complex obstacles (as shown in Fig. 1, the irregular obstacles, the complex ground scenes, and special materials). Besides, its high price makes it unsuitable to deploy in large-scale industrial applications.

In this work, we aim to set up a collision avoidance framework for a real robot based on a monocular RGB camera using efficient DRL. Compared with laser sensors, RGB cameras are not only low-cost but also contain necessary semantic information for understanding the environment. However, the image data captured by RGB cameras contains some redundant information [4] (e.g., intensity, texture, etc.), which causes a large sim-to-real gap that makes DRL difficult to converge and reduces the robustness of robots in unseen scenes. Thus, to utilize the advantage of RGB cameras while reducing redundant information, we propose a novel pseudo-laser measurement generated from the single RGB image captured by a deployed monocular camera for the task of collision avoidance in the complex scene. Unlike [50] directly mapping the depth map to Pseudo-LiDAR to improve the accuracy of 3D bounding box prediction, our pseudo-laser measurement fuses the distance and semantic information of the complex objects and the scene, which thus can more effectively handle the complex obstacles than the laser-based measurement. On the other hand, this pseudo-laser measurement is represented by one-dimensional data, which can provide more concise input for DRL to narrow the sim-to-real gap and promote the effective convergence of DRL and the generalization of the model.

In particular, our approach for generating pseudo-laser measurement is composed of two stages: semantic masking and deep slicing. We first estimate the scene depth map based on the RGB image captured by a monocular camera. It is reasonable to assume that the ground or floor depth that occupies a larger area in the depth map has no substantive significance for obstacle avoidance. Previous researchers [6] usually used the method of projecting a depth map as a point cloud and then performing horizontal threshold-based cutting to remove the ground information. However, this method may fail in some complex ground scenes, e.g., water on the ground, clothes on the floor, and slopes (as shown in Fig. 1). Therefore, we exploit the semantic information of the scene to infer the traversable region, which locates these ‘special obstacles’ (e.g., clothes, water, glass, etc.) on the ground. We design the semantic mask to remove the distance information of the traversable region from the estimated depth map. In addition, we also extract the edge contours of obstacles and combine them with the later depth slicing module to improve the perception of complex obstacle shapes. In this way, the agent can fully exploit pixel-wise semantic information in the field of view (FOV), which is beneficial to make DRL efficient.

After culling out the traversable region from the depth map, we propose a depth slicing module to generate the pseudo-laser measurement. A naive way to accomplish pseudo-laser measurement is to slice the depth map along a horizontal axis. However, this approach ignores the shape of irregular obstacles. For instance, as shown in Fig. 1, if only slicing the feet of the table, the robot may collide with the upper part of it. Thus, we propose a dynamic local minimum pooling operation, which slices the depth information of each column from the semantic-depth map, performs minimum pooling for each column and composes them into a one-dimensional vector, i.e. the pseudo-laser measurement, in which each element of the vector represents the distance away from the nearest obstacle along the vertical axis.

With the pseudo-laser measurement, we adopt a DRL framework to sample the collision avoidance policy of the robot to efficiently avoid obstacles. In particular, to overcome the limitation of $90{}^{\circ }$-FOV and enhance the performance of the policy, we introduce the LSTM [18] submodule to leverage the historical state of the agent to model its temporal actions. Then, we propose the Feature Extraction Guidance (FEG) module to improve the expressiveness of the pseudo-laser measurement. Generally, the state of the agent in DRL includes observation, goal, and velocity. There are two methods of state fusion as input to the DRL network: data-level fusion [6] and decision-level fusion [27]. However, they are not conducive to process the input information from different modalities, so they result in unreliable DRL policies. Thus, we design the FEG module to fully exploit the interrelationship of the input information across the channel domain and the spatial domain. Specifically, FEG weights the pseudo-laser measurement according to the destination, distance, and velocity data, then the weighted result (feature mask) is producted with the observation data. It can pay more attention to the pseudo-laser measurement that guides the current obstacle avoidance task thus the subsequent DRL framework is more purposeful. Due to the preprocessing of the input, the initial DRL stage is not so blind, which accelerates the network convergence.

Finally, to optimize the training process of DRL, we design several simulation environments with different levels of difficulty. The algorithm starts training from simple scenes, then we migrate the trained model to more complex scenes. In order to transfer our model from the simulation environment to the real world, we augment the observations collected from the simulation scenes by adding specially-designed noises similar to the real world to the training data, thereby narrowing the sim-to-real gap. We deploy 7 models in the Gazebo simulation environment, and the results show that our noise model has the highest success rate in complex environments. In addition, this model shows smooth motion and a high success rate in real scene experiments. Therefore, the data augmentation we adopted benefits the sim-to-real migration of our model.

The main contributions of our work are as follows:

• We present a vision-based collision avoidance method based on deep reinforcement learning for robot navigation, which only relies on a single RGB monocular camera as the sensor.

• To utilize the semantic information and share the advantage of laser sensors, we propose using RGB images to encode one-dimensional pseudo-laser measurement for collision avoidance, which greatly improves collision avoidance ability of robots in complex unknown scenes.

• We design a feature extraction guidance module to improve the expressiveness of pseudo-laser measurement in the DRL process, thereby enhancing the ability of robots to understand and handle the obstacle avoidance task.

• To accommodate our vision-based policy, we introduce a new data augmentation to enhance the robustness of the model and reduce the sim-to-real gap. Experiments show that the robot moves smoothly through the adaptive data augmentation.

In this paper, we propose the pseudo-laser measurement, which is generated by distance and semantic information from an RGB image, to improve the environmental-perception ability of robots for the efficient DRL policy. In addition, we design an FEG module and a data augmentation method to make the pseudo-laser measurement play a more reasonable role in the DRL module. Based on this, our method can effectively perceive irregular obstacles and complex ground information in the scene, and reasonably model the current observation, to perform efficient DRL and obstacle avoidance in complex scenes.

As shown in Fig. 2, we obtain the corresponding depth map and semantic mask from the input RGB image. Then, we cull out the traversable region from the depth map through the semantic mask, since the depth of the traversable region under real conditions affects collision avoidance. After that, we propose a dynamic local minimum pooling operation that slices the semantic-depth map to generate the pseudo-laser measurement. Based on this, we input the pseudo-laser to the DRL network and sample the actions to guide the agent to avoid obstacles. In order to enable robots to capture the information that needs more reasonable attention in the input pseudo-laser, we design a FEG module. In the following, we first describe the method we proposed to generate pseudo-laser measurement, then introduce the DRL framework and submodules. Finally, we depict the data augmentation method of our model.

In this section, we first test the DRL framework and evaluate its performance. We also show the visualization results of the FEG module. Then, we conduct experiments on the entire monocular camera-based collision avoidance method. Finally, we apply our model to a robot in real-world scenarios.

Implementation details. Our algorithm is implemented in Pytorch. We train the DRL policy on a computer equipped with an i7-7700 CPU and NVIDIA GTX 1080Ti GPU for approximately 40 hours. We build training and testing scenarios based on the Stage mobile robot simulator (as shown in Fig. 4) and train 4 models with hyperparameters (see Table I) while adopting a multi-scene multi-stage alternating training method. During training, the convergence of our reward function is shown in Fig. 5. Specifically, we train our model in three training scenes (see Fig. 4) one after another. Note that we use a multi-agent collision avoidance task where each agent is assigned a shared collision avoidance policy, which can increase the robustness and training speed of our model.

We use the following metrics to evaluate the performance of different navigation algorithms:

• Success Rate. The number of times a robot reaches the target waypoint without collision and overtime.

• Average Time. The average time it takes for a robot to reach the goal safely.

Network structure experiment. To demonstrate that the Control Decision Stage of the $90{}^{\circ }$-FOV is reliable, we evaluate 4 different network structures (CNN models with $180{}^{\circ }$ and $90{}^{\circ }$ FOV, LSTM based model and LSTM+Attention model with $90{}^{\circ }$-FOV) in DRL models relying on laser sensors and ORCA [47] as baseline in the test scenarios shown in Fig. 4. Note that the CNN model with $180{}^{\circ }$-FOV is [27], while the CNN model with $90{}^{\circ }$-FOV is the same model yet trained with sensor observation of $90{}^{\circ }$-FOV. The test scenario includes the collision avoidance tasks against static/dynamic obstacles: crossing, through walls, through fixed obstacles, and multi-robot obstacle avoidance, which are challenging scenes for evaluating the collision avoidance ability of our model against different types of obstacles. We randomly assign an initial position and a goal position to each robot and test it 100 times in different scenarios. The results are shown in Table II. We also visualize the example trajectories of each model, as shown in Fig. 6. Compared with peer models, our model can generate a smooth trajectory that successfully passes through two waypoints and reaches the goal without getting stuck or collisions.

We observe that our method is superior to the collision avoidance method ORCA [47] and comparable to the model with $180{}^{\circ }$-FOV, while our model has a better success rate. For the $90{}^{\circ }$-FOV model, the success rate drops significantly. With the LSTM module introduced into the model, the performance of the model in a limited FOV is improved. Based on this, we propose the FEG module that CNN has reasonable attention to the current observation. Intuitively, it can make the policy reach or even exceed the model with $180{}^{\circ }$-FOV. However, due to the complexity of the network structure, the robot may occasionally decelerate significantly in dangerous situations, resulting in an increase of average time.

Convergence speed experiment. We find that after introducing the FEG module, the convergence speed of the algorithm is greatly accelerated, and the policy can achieve a better effect in a shorter time. Therefore, we conduct a comparative experiment of convergence speed. Specifically, we compare the three models (model without mask [27], model with the manually set mask [6], model with the network learning mask), and train a fixed number of episodes for each model. The value of episodes represents the training time. We conduct a test navigation success rate for different training time and different model policies. As shown in Fig. 8, the results show that our model has the fastest convergence rate, with a success rate of 75% at 10000 episodes, and a success rate of more than 90% at 15000 episodes. The model without mask only reaches a 90% success rate at 25000 episodes, showing a steady growth trend. Although the manually set mask has a faster convergence rate than the model without the mask, the effect is not ideal compared with the model where we obtain the mask through learning. In the same training times, our FEG module always maintains the highest success rate. Intrinsically, by obtaining masks that have hierarchical attention to the current observation, the network can more efficiently extract reasonable information for obstacle avoidance tasks and accelerate policy convergence.

Feature extraction guidance module analysis. To study the internal effect of the FEG module, we show the visualization results of the learned mask. Fig. 7(a) shows the current state of the robot, green represents the laser range, the red arrow is the current relative direction to the goal, the yellow arrow is the current velocity direction, and the red dot is the goal position. Fig. 7(b) to 7(d) are the visualization of laser input, feature mask and fusion result. Note that the visualization result is divided into three layers from top to bottom, corresponding to three consecutive times of laser measurement. We observe that the weight of the FEG module mask gradually increases from top to bottom. The network has different attention to different moments and pays more attention to the data at the current moment. Mask is obtained by fusion of laser input, goal, and velocity. Therefore the laser measurement in the two directions of the current position to goal direction and speed direction are given higher weight. The position of the obstacle (small value) in the laser input has a higher weight, thus the network pays more attention to the obstacles at closer distances, while the safety distance pays less attention.

Dense crowd experiment. Our model is learned in a multi-agent scenario. Agents act as dynamic obstacles to each other, and all agents share the same policy. To show that our model is not only applicable to multi-agent scenarios with the same policy, but also to dense and complex crowd scenarios, we conduct dense crowd experiments. We use the pedestrian motion simulator [34] proposed by Billy et al. to generate pedestrian trajectories in RVIZ of ROS, use simple cubes instead of pedestrian models for testing in Gazebo, and transmit pedestrian coordinates from RVIZ to Gazebo in real-time, thus the cubes can simulate the movement of pedestrians. We deploy the robot model to test in three different scenarios (as shown in Fig. 9), randomly generate the initial position and the end position, compare the success rate and time of the five models, and each model is tested 100 times in each scenario. As shown in Table III, we can observe that it is similar to Table II. The success rate of our $90{}^{\circ }$-FOV model is similar to the $180{}^{\circ }$-FOV CNN model, while the $90{}^{\circ }$-FOV CNN model performs poorly. Because pedestrians are too dense and have irregular movements, the success rate is generally low. Due to the deepening of the network, our policy has taken a relatively longer time.

The entire monocular camera-based method. To demonstrate the effectiveness of our pseudo-laser measurement, we perform a visual simulation experiment. We build a 3D simulation environment based on Gazebo, as shown in Fig.11. Scene 7 consists only of challenging irregular objects, while Scene 8-10 are complex corridor scenes with pedestrians. We build a simple two-wheel differential robot in Gazebo, deploying laser sensors and RGB-D cameras. The height of the robot is higher than the height of the table in the scene. In order to make the depth estimation and semantic segmentation models suitable for the Gazebo simulation environment, we collect scene-specific image data and pretrain the model for 6 hours so that it can be applied to virtual scenes. In order to improve the efficiency of image processing, we reduce the depth map by 5 times. The model runs on a computer with NVIDIA GTX 1080Ti GPU and an i7-7700 CPU, which can reach 10 frames per second.

We compare 7 models, lasers on the bottom of the robot (E${}_{Btm.Las.}$), laser on top of the robot (E${}_{Top.Las.}$), one-dimensional slice of depth map (E${}_{Dep.1-D.}$), dynamic local minimum pooling of depth map (E${}_{Dep.Pol.}$), one-dimensional slice of depth map combined with semantic mask (E${}_{Dep.1-D.Sem.}$), dynamic local minimum pooling of depth maps combined with semantic mask (E${}_{Dep.Pol.Sem.}$), and dynamic local minimum pooling of depth map and semantic mask with noise (E${}_{Dep.Sem.Noi.}$). We randomly generate a starting position and a goal position for a robot in each scene and test it 100 times. The corridor scene is set with 4 waypoints, and the robot moves circularly in the scene and tests for 20 times. The results are shown in Table IV.

We observe that, in Scene 7 consisting only of irregular objects, almost all the comparison models fail to accomplish the task. Only our slicing method takes the shape of the obstacle into account and achieves the best performance. Since the model E${}_{Btm.Las.}$ considers the shape of the bottom of obstacles only, it reacts poorly towards obstacles such as desks. Similarly, the model E${}_{Top.Las.}$ can only scan obstacle information at a certain height, which is not able to handle lower objects and cone objects. The model E${}_{Dep.1-D.}$ does not perform well for all objects since it falsely integrates the depth of the traversable region. The model E${}_{Dep.Pol.}$ shows some successful collision avoidance behaviors for some objects. However, the traversable region information is not considered, therefore the navigation success rate is still poor. Although the model E${}_{Dep.1-D.Sem.}$ considers the depth of the traversable region, it still only captures one-dimensional data, which is not effective. In contrast, the model E${}_{Dep.Pol.Sem.}$ considers both the depth of traversable region and the overall shape of the obstacle, which can achieve a better success rate of obstacle avoidance. We also test in more complex scenes, i.e., Scene 8-10, and the results show that our method has better robustness and generalization, and even better than laser sensors. Based on this, we also introduce a model trained via our proposed data augmentation method E${}_{Dep.Sem.Noi.}$. During training, the data augmentation setting details are as below. For identifying junction boundaries in laser measurements, we set the threshold for adjacent values as 0.5, and the range of neighborhoods is set as 8. For non-boundaries, we add the Gaussian white noise to each value of the laser measurement for data augmentation. The scale of the noise is 0.07 times the value. Table IV shows that the addition of noise in data augmentation for training the model has the highest success rate. Without such data augmentation, the agent will behave hesitantly and it may rotate at a large angle due to the difference between the pseudo-laser estimated from the depth map and the actual laser sensing data in the virtual environment.

Single irregular object experiment. To further illustrate the effectiveness of our policy on obstacle avoidance for irregular objects, we tested 5 different irregular obstacles (cafe table, table, fire hydrant, construction cone, and cabinet) in the Gazebo environment (as shown in Fig. 10). For the robot to randomly generate an initial position in front of each obstacle and a target position after the obstacle, each obstacle is tested 50 times. The results are shown in Table V. We can observe that all strategies have a high success rate for fire hydrant objects because it is similar to a cylindrical shape and is more regular than other objects. However, for the conical object and the table, the simple laser obstacle avoidance policy and the simple depth slicing policy are not ideal. And our policy has a higher success rate of obstacle avoidance for all irregular objects, showing good generalization.

Complex ground scenes experiment. In order to further explain the role of our fusion of semantic information, as shown in Fig. 12, we visualize the process of acquiring pseudo-laser measurement in three complex scenarios. Taking Fig. 12(A) as an example, we obtain a depth map from an RGB image and a binary classification semantic mask with only the traversable region and background. After fusing into a semantic-depth map, we use the depth slicing method we proposed to obtain the final pseudo-laser measurement. The traditional method obtains laser measurement from the depth map by horizontal slice mapping of the point cloud. Since semantic information is not considered, the water surface is also cut off from the ground. However, in reality, the water surface is not the safe passable region of the robot. Therefore we use semantic information to view the water as an unpassable region. The depth information of the water surface is also taken into account during the slicing process, thus providing the robot with the safest pseudo-laser measurement. In Fig. 12(B), there is a highly misaligned slope. For a simple horizontal slice, it is difficult to set a threshold, resulting in ground depth affecting navigation. Through semantic information, we can obtain the traversable region of robots, so as to generate the most efficient and reasonable pseudo-laser measurement. There is a piece of clothing on the ground in Fig. 12(C). In fact, the clothing cannot be crushed and should be regarded as an obstacle. We can obtain the pseudo-laser measurement considering the depth of the clothing. Therefore, the method of obtaining the semantic mask of the traversable region through the semantic information and then performing depth slicing can enable the robot to effectively deal with scenes with the complex ground.

Visualization experiment of complex obstacles in real-world scenes. We deploy our method in a real-world scene, and visualize the generation process of pseudo-laser measurement when faced with complex objects, as shown in Fig. 13. Compared with laser device observation data, our approach with semantic information can effectively segment the traversable region, extract the edge contours of irregular objects, and identify ‘special obstacles’ in complex ground scenes (as shown in the red boxes in Fig. 13). Thereby our framework can perform efficient DRL to obtain reasonable actions through the generated pseudo-laser measurement. See the supplementary video for more details.

Monocular-visual limitation experiment. Visual obstacle avoidance will be affected by the current camera perspective. Although our policy will avoid obstacles in advance when facing obstacles, in order to study the limitations of our method, we manually initialize the location of obstacles to test the navigation success rate. We find that navigation failed when obstacles occupied most of the FOV, thus different obstacles have different navigation safety distances. We use walls of different sizes as test obstacles, and each wall tests multiple different distances.

• Distance - The quotient of the distance between the robot and the obstacle and the maximum distance taken by the virtual camera (6.0m).

The result is shown in Fig. 14. The navigation success rate is lower when the initial position is closer to the obstacle. Obviously, the greater the space occupied by obstacles, the greater the distance for policy failure. When the obstacle occupies 65% of the maximum angle of view ($90{}^{\circ }$-FOV), our policy starts to perform poorly, and the success rate decreases. When it occupies 80%, our policy will fail, showing a hover or even collision.

In this paper, we propose a framework that uses a low-cost monocular RGB camera to accomplish obstacle avoidance for mobile robots. Against the complex obstacles in challenging environments, we design the pseudo-laser, which fuses distance and semantic information and exhibits good collision avoidance performance in unseen complex scenes. The FEG module we proposed can reasonably weight the observed data, thus the mobile robot can capture the surrounding environment well and achieve efficient collision avoidance. In addition, we use data augmentation to improve the performance of the policy in the real-world and introduce an LSTM module to solve the problem with the limited FOV. However, our method has limitations. Due to the poor performance of the depth estimation method for the mirror, our method is invalid in some indoor scenes with mirrors [56], as shown in Fig. 18. In future work, we will consider more complex concepts, such as robot positioning via pure vision [17], visual servoing [22] and the application of hierarchical reinforcement learning [25] in obstacle avoidance tasks.

This work was supported in part by National Key Research and Development Program of China (2021ZD0112400), the National Natural Science Foundation of China under Grant 61972067/U21A20491/U1908214, and the Innovation Technology Funding of Dalian (2020JJ26GX036).