Motion Robust High-Speed Light-weighted Object Detection with Event Camera

Currently, representative approaches to applying event cameras to high-level tasks have mainly been done by processing events in batches, converting them into regular, tensor-like representations, making them compatible with deep learning techniques based on image data, and feeding them into traditional deep convolutional network-based target detectors. There has been much work demonstrating that the statistics of these temporal representation tensors overlap with those of natural images, thus also enabling transfer learning using networks pre-trained on image data [11, 22, 34]. These methods are divided into two main categories: representation methods based on discretizing Event Measurement Field, collectively called Event Spike Tensor, and representation methods emulating spike neural networks.

The Event Measurement Field is defined in the work of Gehrig et al. [11] by representing each event as a Dirac pulse in a continuous spatio-temporal manifold. The Event Measurement Field is generated by assigning a measurement to each event, and the representation tensor is then obtained by using kernel convolutions to aggregate the events. The representation tensor has at least two spatial dimensions and one temporal dimension, and the temporal dimension is considered the channel dimension when feeding the tensor into the detector. The polarity can be either encoded into the values or separated in the channel dimension. The manually generated Event Spike Tensor includes Event Frame[27], Event Count Image[22, 34], Event Volume[35]. Such methods require specifying hyperparameters including the measurement function, the convolution kernel function, and timestamps for executing kernel convolutions. There are also some works [11, 18] that explore the possibility of learning the measurement function or the convolution kernel function directly from the raw event stream, but the timestamps for performing kernel convolutions are still a hyperparameter that needs to be specified.

A common approach to the representation of emulating spike neural networks is to maintain a tensor with two spatial dimensions and one channel dimension, and always ensure that the spatial locations of newly occurring events in the tensor have larger values. Surface of Active Events[1, 34] gives a 2D snapshot of the latest timestamp of the events in the field of view. In practice, a global numerical transformation is applied to it so that its values satisfy a certain value domain. Surface of Active Events requires the global numerical transformation function as the hyperparameter. Leaky Surface[3] is a process capable of accumulating events, where sparse events are integrated into a leaking surface. Whenever an event is triggered at a spatial location, the corresponding spatial location of the tensor increments the value by a fixed amount, and the values in other locations decrease. By an attenuation function, the amount of decrement is determined by the difference between the current timestamp and the timestamp of the last received event at a specific location. The hyperparameters to be specified by Leaky Surface are the incremental value and the attenuation function.

There are existing works on applying event cameras to target detection tasks based on deep convolutional neural networks, mainly in traffic scenarios.

Conventional approaches using feedforward target detector include that designed by Chen et al [6], using Event frame as event representation tensor and YOLO [28] model as feedforward object detector, and that designed by Hu et al [14], using Event Volume as the event representation tensor and YOLOv3[29] model as the feedforward object detector. Some works focus on exploiting the spatial sparse nature of the Event Representation Tensor to reduce the computational effort of the model. Cannici et al. [3] designed an approach using Leaky Surface as the Event Representation Tensor and a fully convolutional YOLO [28] model with sparsely updateable feature map as a feedforward object detector. Messikommer et al. [23] designed a method using a sparse Event Count Image as the event representation tensor and an asynchronous sparse convolution-based implementation of the VGG[33] model as a feedforward object detector.

Recurrent-Convolutional neural network object detector is currently an advanced approach in the field of event camera event stream object detection. Among them, the RED algorithm designed by Perot et al [25] uses Event Volume as the event representation tensor and replaces the deep layer of SE-ResNet [13] backbone with the ConvLSTM [32] module. The ASTMnet algorithm designed by Li et al. [18] uses an asynchronous attention embedding learned directly from the original event stream and the VGG[33] backbone with a lightweight memory module called Rec-Conv. These memory mechanisms are not designed according to the characteristics of the event stream data, so their effectiveness depends on a large number of parameters and datasets with high labeling frequency for training. Therefore, they are expensive to train and slow to run.

Methods to fuse the event stream representation tensor and traditional APS camera images all use Event Frame as the event representation tensor and the YOLOv3[29] as feedforward target detector. Among them, Li et al. [17] post-fused the detection results using Dempster-Shafer theory [16], while Jiang et al. [15] performed fusion on the confidence map and output the detection results. The fusion method requires traditional APS cameras to be involved in data acquisition, implying more energy consumption and data storage costs. Besides, not all event camera datasets provide the corresponding traditional APS frames at the same time.

Messikommer et al.[23] have also explored maintaining asynchronous queues of events at each spatial location to sample the event stream asynchronously while using a VGG[33] model implemented based on asynchronous sparse convolution as a feedforward object detector. This approach can theoretically overcome the problem of hyperparameter setting in the event stream representation tensor, but it is not mature enough to be used for designing high-accuracy detectors.

Event cameras bring a new computer vision paradigm by representing visual information in a different way [19, 26, 9]. Instead of encoding dynamic visual scenes using a series of static images acquired at a fixed frame rate like traditional APS cameras, event cameras generate data in the form of sparse and asynchronous event streams. According to the event generation mechanism, an event $\{e_i=(x_i,y_i,p_i,t_i\left)\right\}$ will be triggered at moment $t$ when the log scale illumination change at the pixel $(x,y)$ exceeds a predefined threshold $\eta$ compared to the previous moment, i.e.,

where $\Delta t$ is a value of the order of microseconds. The polarity $p\in \{0,1\}$ indicates whether the illumination is increasing or decreasing. The asynchronous and sparse nature of the event stream data is explored in detail in the work of Messikommer et al.[23]; in short, when the global illumination is constant, the event camera responds mainly to the edges in the scene. This means that events are not present at the vast majority of spatial locations and moments.

A paradigm for the event camera object detection problem was defined by Perot et al. It is briefly modified here to make it more precise and to facilitate the discussion later on.

• Event stream data: for a camera with a field of view height of $H$ and width of $W$, its event stream data is defined as:

  $$E=\{e_i=(x_i,y_i,p_i,t_i){\}}_{i\in N}$$

  (2)

  where $x_i\in \{0,1,...,W-1\}$, $y_i\in \{0,1,...,H-1\}$, $p_i\in \{0,1\}$, $t_i\in [0,T_{max})$, in which $T_{max}$ is the maximum duration of the event stream record.

• Annotation:

  $$B^{\ast }=\{b_j^{\ast }=(x_j,y_j,w_j,h_j,l_j,t_j\left)\right\}$$

  (3)

  where $l_j\in \{0,...,L\}$ is the class, $t_j$ is the timestamp of the appearance of the annotation. The labeling is performed at a specific time interval, so $t_j$ has only a fixed set of taken values.

• Detection result:

  $$D^{\left(k\right)}\left(E\right)\approx \left\{D\right(\left\{e_i{\}}_{t_i\in [0,t^{\left(n\right)})}\right)\}$$

  (4)

  where $D(.)$ is the detector, $k$ means it is the $n$th detection on the event stream, $n\in \{1,2,...\}$. $t^{\left(n\right)}$ denotes the timestamp of the $n$th detection occurrence with $t^{\left(n\right)}>t^{(n-1)},\forall n$. The detection bounding boxes with timestamp $t^{\left(n\right)}$ can be matched with the annotation with $t_j$ near $t^{\left(n\right)}$, thus applying the evaluation protocol defined under synchronized frames. Ideally, the detector will detect at each $t_j$.

When the global illumination of the environment is constant, an object generates events depending on its relative motion to the event camera. The edges of objects moving fast relative to the event camera tend to be more likely to trigger events, while objects moving slow or even stationary relative to the event camera do not easily, or even not, trigger events.

The most accurate measure of motion velocity is the optical flow calculated from camera imaging information. There have been many efforts to estimate optical flow from event stream data. We chose the method proposed by Nagata et al. [24] because of its computational simplicity and efficiency. They estimated the TV-L1 optical flow [5] by matching adjacent timestamps using the Surface of Active Events [1, 34] representation tensor built from the event stream data, which allows the estimation of the TV-L1 optical flow without recovering luminance or additional sensor information. At the timestamp $t_j$ with a labeling bounding box $b_j^{\ast }$, the dense optical flow can be represented as a horizontal optical flow $u_{t_j}(x,y)$ and a vertical optical flow $v_{t_j}(x,y)$, i.e., both a horizontal and a vertical component exist at each spatial location, to describe the short-time motion velocity.

Event Spike Tensor is a data representation method for event streams based on discretizing the Event Measurement Field. According to the definition proposed in the work of Gehrig et al [11], we define the Event Measurement Field build from $E^{\left(n\right)}=\{e_i{\}}_{t_i\in [0,t^{\left(n\right)})}$ as $S^{\left(n\right)}(x,y,p,t)={\sum }_{e_i\in E^{\left(n\right)}}f(x,y,p,t)\delta (x-x_i,y-y_i,p-p_i,t-t_i)$, where $x\in \{0,1,...,W-1\},y\in \{0,1,...,H-1\},p\in \{0,1\}$, $f(.)$ is the measurement used to extract event features, usually defined manually, but can also be learned from the event stream [11, 18].

The temporal dimension of the Event Measurement Field is indefinite and continuous, to transform it into the Event Spike Tensor with a fixed and discrete temporal dimension, it is necessary to use kernel convolutions to aggregate events at specific timestamps. The general formula of Event Spike Tensor is $S^{\left(n\right)}(x,y,p,b)={\sum }_{e_i\in E^{\left(n\right)}}f(x_i,y_i,p_i,t_i)k(x-x_i,y-y_i,p-p_i,{\tau }_b^{\left(n\right)}-t_i)$.

$T^{\left(n\right)}=\{{\tau }_b^{\left(n\right)}{\}}_{b\in \{0,1,...,B-1\}}$ is a set of timestamps for performing kernel convolutions, and $B$ is the length of the temporal dimension of the Event Spike Tensor. In the existing work on Event Spike Tensor, $T^{\left(n\right)}$ are a set of globally synchronized timestamps independent of $x$, $y$, $p$. Timestamps in $T^{\left(n\right)}$ are usually set at a fixed time interval $\Delta \tau$, i.e., $T^{\left(n\right)}=\{{\tau }_b^{\left(n\right)}=t^{\left(n\right)}-(B-b)\Delta \tau {\}}_{b\in \{0,1,...,B-1\}}$.

$k(.)$ is the convolution kernel for aggregating events, usually manually defined but can also be learned from the event stream [11, 18]. Define $k_{upper}$ as the upper bound of $t$ that does not satisfy $k(x,y,p,t)\to 0$ and $k_{lower}$ as the lower bound of $t$ that does not satisfy $k(x,y,p,t)\to 0$, the currently available work on Event Spike Tensor has $k_{upper}$ and $k_{lower}$ as global synchronized values independent of $x$, $y$, and $p$.

With a dimension transformation $S_{c,y,x}=S^{\left(n\right)}(x,y,c-p\lfloor \frac{c}{p}\rfloor ,\lfloor \frac{c}{p}\rfloor )$, the Event Spike Tensor can finally be in a form suitable for inputting to the detector.

Event Volume[35] is a typical Event Spike Tensor with $f(x,y,p,t)=1$ and $k(x,y,p,t)=\delta (x,y,p)max(0,1-|\frac{t}{\Delta \tau }|)$, then its $k_{upper}=\Delta \tau$. Event Volume has the problem of the global synchronized time window and the global synchronized temporal resolution. The Event Volume can only consider events satisfying $t_i\in [{\tau }_0^{\left(n\right)}-k_{upper},t^{\left(n\right)})$ at $t^{\left(n\right)}$, and we call this interval the global synchronized time window of the Event Volume. The global synchronized temporal resolution means that regardless of the frequency of events triggered at each spatial and polarity location at a moment, kernel convolutions will be performed at a globally consistent time interval $\Delta \tau$.

Since the temporal dimension will be used as the channel dimension when feeding the tensor into the object detector, the value of $B$ should not be too large, otherwise, it may lead to the occurrence of overfitting. With constant $B$, the global synchronized time window length is proportional to the interval size, but a larger interval symbolizes a lower temporal resolution for sampling the event stream, implying that the time window length is inversely proportional to the temporal resolution, which means that having both a long global synchronized time window and a large global synchronized temporal resolution is not possible.

Fig.2 visualizes the problems that can result from the global synchronized time window and the global synchronized temporal resolution. It can be seen that for spatial and polarity locations recently trigger events less frequently, a short global synchronized time window cannot aggregate enough information, while a too long global synchronized time window leads to a decrease in global synchronized temporal resolution for spatial and polarity locations recently trigger events more frequently.

Event Count Image[22, 34] is also an Event Spike Tensor. unlike Event Volume, the convolution kernel size of Event Count Image can be dynamically adjusted according to the frequency of events triggered within the recent event stream. The $B$ of Event Count Image is usually taken as 1. The hyperparameter controlling the size of the convolution kernel is an integer $N$. In our implementation, we take $f(x,y,p,t)=0.05$ and $k(x,y,p,t)=\{\begin{array}{cc}u(x,y,p,t),& t\le t^{\left(n\right)}-t_{i^{\left(n\right)}-N}\\ 0,& Others\end{array}$, where $i^{\left(n\right)}=\arg {max}_i{t}_i,t_i\in [0,t^{\left(n\right)})$, then its $k_{upper}=t^{\left(n\right)}-t_{i^{\left(n\right)}-N}$, which means that Event Count Image will consider the latest $N$ events at $t^{\left(n\right)}$. Its global synchronized time window is $t_i\in [t_{i^{\left(n\right)}-N},t^{\left(n\right)})$.

The global synchronized time window in Event Count Image can be dynamically adjusted according to the frequency of events triggered within the recent event stream since it considers the nearest fixed-length event stream. This approach is valid if there isn’t much difference between the frequency of events triggered at each spatial and polarity location. However, in the object detection problem, we often have to detect multiple objects within the field of view at the same time. When there are objects with different motion speeds relative to the event camera in the field of view, there will be different event generation frequencies at different spatial locations simultaneously. In this case, as long as the global synchronized time window exists, it is impossible to set a reasonable time window for each spatial location.

Fig.3 further illustrates this problem. It can be seen that for spatial and polarity locations recently trigger events less frequently, too short a global synchronized time window cannot aggregate enough information, while for spatial and polarity locations recently trigger events more frequently, too long a global synchronized time window makes the convolution kernel cover excessive events at one time, resulting in a loss of information.

The representation method of emulating spike neural networks always ensures that the spatial locations of newly occurring events in the tensor have larger values than other positions. Surface of Active Events[1, 34] belongs to this type of method, which gives a 2D snapshot of the time surface with the latest timestamp of the event in the field of view, i.e., $S^{\left(n\right)}(x,y,p)=\underset{i,x_i=x,y_i=y,p_i=p}{max}{t}_i$. In practice, an exponential transforming function is applied to it to restrict its value range, i.e. $S^{\left(n\right)}(x,y,p)=\exp \left[\lambda \right(\underset{i,x_i=x,y_i=y,p_i=p}{max}{t}_i-t^{\left(n\right)}\left)\right]$, where $\lambda$ is a hyperparameter. This gives a value range of $(0,1]$, where the spatial locations of newly triggered events have larger values in the value range, while the values of the spatial locations without new events for a long time will continue to decay, and the $\lambda$ is the hyperparameter to controls the decay rate. By dimensional transformation $S_{c,y,x}=S^{\left(n\right)}(x,y,c)$, Surface of Active Events can eventually be in a form suitable for inputting to the detector.

Fig.4 shows the mapping between $\Delta t=t-t^{\left(n\right)}$ and the value in $S^{\left(n\right)}$ when $\lambda$ takes different values. It can be seen that under $\lambda =1\times {10}^{-5}$, when $\Delta t$ takes $-5\times {10}^4\mu s$ and $1\times {10}^4\mu s$, the value in $S$ takes a larger difference, which indicates that Surface of Active Events gives the newly triggered events a larger the temporal resolution. However, when the value of $\Delta t$ is less than $-1\times {10}^6\mu s$, the values in $S$ are close to zero, which indicates that the information of the events triggered further than $1s$ from the current time $t^{\left(n\right)}$ will hardly be retained. Therefore, there is also a global time window for Surface of Active Events. Define $k_{lower}$ as the lower bound of $t$ that does not satisfy $e^{\lambda t}\to 0$, then the global time window of Surface of Active Events is $[t^{\left(n\right)}+k_{lower},t^{\left(n\right)})$.

The length of the global time window can be increased by decreasing the value of $\lambda$, as shown in Figure.4, when $\lambda =1\times {10}^{-6}$, there is still a large difference in the value of $S$ after $\Delta t$ takes a value smaller than $-1\times {10}^6\mu s$. However, relatively, when $\Delta t$ takes values of $-5\times {10}^4\mu s$ and $1\times {10}^4\mu s$, the difference in the values in $S$ decreases. If the value of $\lambda$ is continue to reduce, the newly triggered events will be more difficult to distinguish from each other by the value in $S$.

Column 1 in Fig.5 shows the TV-L1 optical flow plot of the scene, Columns 2 to 6 show the visualization of Event Volume, Event Count Image and Surface of Active Events with different hyperparameters. It can be found that for objects with slow motion relative to event camera, a long global synchronized time window is required to obtain sufficient information, but for objects with fast motion relative to event camera, an excessively long global synchronized time window instead leads to the occurrence of motion blur. When there are objects with different motion speeds relative to camera in the field of view at the same time, it is hard to find a hyperparameter that takes into account all objects.

The feedforward target detector in the state-of-the-art approaches are designed mainly based on the YOLO series model [6, 14, 3]. Also, we designed the Agile Event Detector (AED) mainly based on the YOLOX model [10], which has the characteristics of light weight and high speed. The AED model receives event representation tensors as input, and outputs the detection results. The general structure of the AED model is shown in Fig.6.

Compared to YOLOX, AED does not use the CSPDarknet[2] as the backbone network. AED’s backbone network is adapted from the Darknet21 used in YOLOv3[29]. Compared to the CSPDarknet, the Darknet has a faster inference speed. The input layer of the backbone network accepts the event representation tensor as input. The event representation tensor $S_{c,y,x}$ of shape $(C\times H\times W)$ is resampled to $(C\times H^{\ast }\times W^{\ast })$ using nearest neighbor interpolation to match the network input size. We still retain the Focus layer in CSPDarknet as the input layer of the backbone network instead of the traditional $7\times 7$ convolution used in ResNet[12] and Darknet. The Focus layer downsamples the input tensor spatially while encoding the spatial information into the channels, which reduces the spatially redundant local information to some extent. Then the features are extracted using the $3\times 3$ convolution. After downsampling the feature map, the $3\times 3$ convolution actually possesses a larger field of perception than the traditional $7\times 7$ convolution. In this way, the number of parameters in the input layer is reduced, while the operation speed as well as the field of perception is increased.

In the traditional Darknet, the feature map output from the input layer often has a small number of channels. With the deepening of the network, the reduction of the feature map size, and the enhancement of the semantics, the number of the feature map channels will increase exponentially. In AED, however, the feature map output by the input layer has 64 channels, which is a large value for the model with the same number of parameters. This is because the event representation tensor depends on the channel dimension to encode time and polarity information, and its semantics is more complex than the channels of conventional RGB images. Therefore, more neurons need to be used in the early stage to fully fit the information in the event representation tensor. In order to maintain the lightness of the detector, we keep the number of output channels of Residual Layer 3 and 4 to 256. Reducing the number of channels in the last few layers of the backbone network can significantly reduce the number of parameters, improve the inference speed, while having a limited impact on the accuracy. We use the SiLU function used in the CSPDarknet of YOLOX as the activation function in the backbone network.

The Feature Pyramid Network (FPN) and the detection head are similar to the settings in YOLOX. The FPN mainly refers to the Path Aggregation Feature Pyramid Network (PAFPN) used in YOLOv4[2]. The PAFPN in AED is also lighter due to the lower number of channels in the output feature map from the backbone network. The detection head network structure refers to the decoupled detection head used in YOLOX, which branches the feature maps through two convolutional layers, classification and regression, respectively. The decoupled detection head leads to better accuracy, while the increased computational effort can be compensated by the smaller number of feature map channels.

To improve the generalization capability of the lightweight detector, we also designed a simple but effective data augmentation method for event stream representation tensor data:

1. During training, each event representation tensor $S$ has $p_1$ probability to be flipped horizontally, i.e., $S_{c,y,x}^{\ast }=S_{c,y,W-x-1}$.

2. During training, each event representation tensor $S$ of shape $(C\times H\times W)$ has $p_2$ probability to be resampled to $(C\times \lfloor \alpha H\rfloor \times \lfloor \alpha W\rfloor )$ using nearest neighbor interpolation, where $\alpha \in [1,\infty )$, and then randomly cropped back to $(C\times H\times W)$, i.e. $S_{c,y,x}^{\ast }=S_{c,y^{\ast }:y^{\ast }+H,x^{\ast }:x^{\ast }+W}$, where $y^{\ast }\sim U(0,\lfloor \alpha H\rfloor -H),x^{\ast }\sim U(0,\lfloor \alpha W\rfloor -W)$, and $U(a,b)$ is the uniform distribution on $[a,b],a,b\in R$.

Temporal Active Focus (TAF) is an event stream representation tensor that we propose, which makes full use of the asynchronous generation characteristics of the event stream data. By sampling the Event Measurement Field[11], which is continuous and of indefinite length in the temporal dimension, at different spatial and polarity locations asynchronously in time, the generated event stream representation tensor is discrete and of fixed length in the temporal dimension, but different spatial and polarity locations all contain information of different time ranges. Moreover, the incremental update ability of the TAF tensor makes it much less computationally intensive to construct in continuous event stream detection scenarios than the state-of-the-art methods. In this subsection, we will describe the principle and the implementation of TAF respectively.

The input layer of conventional detectors is often a convolutional layer, which not only extracts information in the channel but also models spatial dependencies to extract spatial information. It works well when using the normal RGB image as the input since the semantics in channels are often weak. However, the TAF tensor has very rich information in the temporal dimension, and it is important to note that the TAF tensor also belongs to a kind of Event Spike Tensor, so before being fed into the detector, its temporal dimension will be transformed to the channel dimension first by the dimensional transformation. Therefore, the semantics of the TAF tensor in channels is much stronger than the normal RGB image. Bearing this in mind, we designed a module called Bifurcated Folding Module (BFM) to fully extract information in the temporal dimension of the TAF tensor before being fed into the detector, which is point-wisely applied at each spatial location.

The design of the BFM follows two main concepts: first, gradually aggregating values at adjacent times; second, assigning greater importance to information from events that are triggered more recently. Following the first concept, we design the Folding Layer, whose structure is shown in Fig.10(b). the Folding Layer gradually reduces the number of input channels using the $1\times 1$ depthwise convolution, whose number of output channels is smaller than the number of input channels. Since the temporal dimension is transformed to the channel dimension by dimensional transformation, the $1\times 1$ depthwise convolution is actually a local connection of adjacent times. We also use Weight Normalization[30] to make the Folding Layer converge better during training while playing a similar role to Batch Normalization during inference. Following the second idea, we are inspired by the Cross-Stage-Partial-connections (CSP)[2] mechanism. After each temporal aggregation, we make slicing in channel dimensions to get the channels encoding temporally latest information, and finally connect all sliced channels together. This will make the output channels contain more information about recent events, and also act as a residual connection to a certain extent, making the module easier to converge during training. We finally fuse the output channels with a Multi-Layer Perceptron (MLP), the structure of which is shown in Fig.10(c).

The overall structure of the Bifurcated Folding Module is shown in Fig.10(a). For a TAF tensor with temporal dimension length $B$, we set ${\log }_{2}B$ Folding Layers. The first Folding Layer will use $B/2$ convolution groups to reduce the number of channels from $2B$ to $2^{{\log }_{2}B-1}d$, where $d$ is a hyperparameter. Then the $i,i\in \{2,...,{\log }_{2}B\}$th Folding Layer will use $B/2^i$ convolution groups to reduce the number of channels from $2^{{\log }_{2}B-i+1}d$ to $2^{{\log }_{2}B-i}d$. The channels $c\in \{2^{{\log }_{2}B-i}d-d,2^{{\log }_{2}B-i}d-d+1,...,2^{{\log }_{2}B-i}d-1\}$ of the output of $i$th Folding Layer will be taken and connected altogether in the channel dimension, serving as the input of the Multi-Layer Perceptron (MLP).

We compare our method with the state-of-the-art methods. We first compare our overall method with other existing works in terms of accuracy, model inference speed, and model parameters. Then, for the event representation methods, we conduct a separate comparison experiment, using AED as the detector, comparing the accuracy and the speed of TAF with other methods.

In the ablation experiments, we focus on the effectiveness of the different components of our method and the factors that affect the performance of TAF.