UAS Navigation in the Real World Using Visual Observation

Visual navigation research in the virtual environment has shown promising results to traverse to the target with the shortest trajectory [11], [2], [4]. Some recent work focus on introducing memory to the navigation task, such as graph attention memory (GAM) based system [6] and visual graph memory (VGM) structure to embed its navigation history information [5]. However, the majority of work in this field is dedicated to use simulation environment and lack validation in the real world .

Image matching as a fundamental concept in computer vision and pattern recognition is the general name of keypoint detection, description and matching. The goal is to recognize the same item in images taken from any angles, with any lighting and scale. Image matching are generally categorized as handcrafted and deep learning. Handcrafted image matching requires expert acknowledge to model local keypoint texture regardless of rotation or scale variation. Over past decades, growing amount of methods have been proposed [8]. SIFT and SURF [7, 1] are pieces of the art among them. Nowadays, many applications are still using them. Apart from handcrafted methods, deep descriptors are the main stream in recent years. Those descriptors keep the same vector structure but learnt by a deep learning framework. Paul-Edouard et al. proposed SuperGlue [9], an end-to-end framework to find matched features between two or more images.

We formulate the navigation task as a Markov decision process (MDP) and implement the RL approach. The RL algorithm we adopt is the DCQN model proposed by [3]. The goal of the agent is to exploit a policy that rewards reaching the destination starting from any location. The navigation process contains a sequence of states, actions and rewards $(s_0,a_0,r_0,s_1,...,a_{N-1},r_{N-1},s_N)$. The state in this task is the UAS’s observation which is a RGB image. We make it a policy that the UAS makes decision on where to go while encountering the landmark. The actions are moving forward, backward, left and right. There are three situations for reward design: 0.1 upon reaching the goal, -0.001 while colliding and -0.0001 for time penalty to encourage finding the shortest trajectory. The agent continuously interacts in the environment to explore the optimal navigation policy by trial and error and maximize the accumulated reward.

The UASNAV environment is designed for training the agent in the policy learning phase. This environment spans a 400 meters by 300 meters area of residential block. We select 100 landmarks distributed in a 10 by 10 pattern in this area. The horizontal spacing is 40 meters and the vertical spacing is 30 meters between each landmark. These landmarks act as ground control points guiding the UAS to the goal. We collect satellite images from Google Maps at these locations to represent the landmark observation. This image is used as the input to the RL algorithm. The output of the DRL module is the optimal action the agent should take at this location (see Fig.2). The raw image resolution is $1280\times 720$, we use the $640\times 480$ resized image in the image matching phase.

In the UASNAV environment, we have the satellite image as a descriptor corresponded to each landmark. While navigating in the real environment, the UAS takes real-time observation using the embedded downward-looking camera in a certain frequency. In order to make decision on actions at these ground control point based on the learned policy, we need to enable the UAS to recognize the landmark during the navigation process. Since the UAS observation and satellite image at the same landmark position would contain similar texture. We therefore utilized image matching approach to match the environment landmark descriptor (satellite image) and UAS observation (real-time taken image).

During the navigation process, the UAS continuously matches the current observation with four landmark (front, back, left, right) descriptors. We choose the one with the most matching points as the matched landmark. To ensure the UAS exactly arrived at the matched landmark position, we use affine matrix to calculate the center distance between the current observation and landmark satellite image. Once it has the shortest center distance, we consider the UAS has arrived at that landmark position and should take action correspondingly.

We adopt SuperGlue [9], which is a state of the art algorithm for image matching. It has robust performance with regard to illumination variation or seasonal changes (see Fig.3). Our method takes this advantage, therefore, shows the ability to resist environment variation and is applicable in different seasons and most weather conditions.

We implement the DCQN [3] model for policy learning and observe that the agent successfully learns the navigation task. The training progress in Fig.4 shows the fast convergence of our algorithm. The reward in this figure indicates the cumulative reward computed at each episode as the agent reaches the destination. As shown in Fig.4, the agent learns the navigation policy within 200 episodes. We also run evaluation for 100 episodes in UASNAV environment and demonstrate that the agent is able to find the target in 6.53 steps in average.

For the real world test, we use UAS to fly in the residential area which is the same as the UASNAV environment. The UAS is first set to a known starting point and then uses only a down-looking camera to get observation during the whole process. Once it matches with the landmark, the drone will choose an action to either fly forward, backward, left or right based on the output policy of RL algorithm. We adopt the pretrained SuperGlue algorithm for the purpose of image matching in our approach without additional training. The UAS trajectory and image matching results at the landmark location are shown in Fig.5.