GaitFi: Robust Device-Free Human Identification via WiFi and Vision Multimodal Learning

The WiFi-based gait recognition method uses RF signals from WiFi-enabled devices to determine human identity. The transmitter emits WiFi signals, which are reflected by different body parts of the walking subject and then recorded by CSI data at the receiver [yang2022benchmark], which has empowered many applications including occupancy detection [zou2017freedetector], crowd counting [zou2018device, zou2017freecount], human activity recognition [zou2018deepsense, zou2017multiple, zou2019wifi, yang2018carefi, wang2021multimodal], person identification [zou2018identification, wang2022caution], vital sign detection [hu2022resfi], pose estimation [yang2022metafi] and gesture recognition [zou2018robust, yang2019learning, zou2018gesture]. To use WiFi sensing in the real world, current research aims at efficient communication [yang2022efficientfi], model security [yang2022robustsense] and data-efficient training [yang2022autofi].

Recently, research on human identification using WiFi-enabled devices has begun to emerge because of ubiquitous WiFi-enabled IoT devices. This paper focuses on the research of WiFi in the field of gait recognition. A WiFi-based gait feature extraction system named WiFiU [wang2016gait] is proposed by Wang et al. to classify humans with different identities. Zhang et al. [zhang2016wifi] propose WiFi-ID, a WiFi-based gait recognition method that can be used in small offices or smart homes. Zeng et al. [zeng2016wiwho] utilize gait to recognize human identity by measuring the time domain information of WiFi signals. Lv et al. [lv2017wii] propose Wii, which improved gait recognition accuracy by performing autocorrelation on the torso reflection to remove imperfection in spectrograms. Cao et al. [cao2021lightweight] propose a lightweight deep learning algorithm named LW-WiID, which can achieve a relatively high recognition accuracy by extracting the spatial information of subcarriers. CAUTION [wang2022caution] proposes to employ few-shot learning for data-efficient human identification. From the above work, research on WiFi-based single-modal gait recognition is getting more appealing for IoT-enabled human identification.

Vision-based solution plays an essential role in gait recognition methods. Johansson et al. [johansson1973visual] use moving light displays and reflectors on different joints of the human body and observe that gait patterns are unique, so that gait can be a biomarker feature that is recognized by vision. With the development of computer vision, vision-based gait recognition methods are gradually gaining widespread attention. Nowadays, vision-based gait recognition methods can be divided into two main categories, template-based and video sequence-based methods.

For the template-based gait recognition method, the gait silhouette contour sequence needs to be obtained using background subtraction [wang2003silhouette]. Then, the resulting gait profile is aligned by cropping and then pixel-level operations are performed to generate a gait template, such as Gait Energy Image (GEI) [han2005individual]. The obtained gait templates can be used to obtain feature representations by machine learning methods [xing2016complete]. After obtaining the gait representation, the similarity between the representation pairs can be measured by metric learning methods [takemura2017input]. Recently, an increasing number of deep learning methods have been applied to template-based gait recognition tasks [wu2016comprehensive, he2018multi].

The video sequence-based gait recognition directly uses the silhouette sequence generated by background subtraction as the input to the deep learning neural network. This method can collect more temporal information, so specialized neural network structures need to be designed to extract such temporal information. Liao et al. [liao2017pose] use a LSTM-based approach to extract temporal information from gait sequences. Chao et al. [chao2019gaitset] propose GaitSet, a network that can blur time information of gait sequence. Lin et al. [lin2021gait] propose to aggregate local temporal and local spatial information for gait recognition.

Multimodal machine learning aims to build models that can process and correlate information from multiple modalities [ngiam2011multimodal]. The motivation for multimodal machine learning comes from the fact that every single modality has its own drawbacks that make them perform sub-optimally. In addition, humans perceive the world in a multimodal way, such as vision, sounds and text, encouraging the existence of multimodal learning. Therefore, when a research question or dataset contains multiple modalities, it is characterized as a multimodal task. Multimodal machine learning involves many research directions including representation, translation, alignment, fusion and co-learning. Fusion is responsible for combining the information of multiple modalities to perform target prediction (i.e., classification or regression). It is one of the earliest research directions of multimodal machine learning and is currently the most widely used one. According to the level of fusion, multimodal fusion can be divided into input-level [li2017pixel], feature-level [ross2005feature, haghighat2016discriminant] and decision-level fusion [chatzis1999multimodal]. For our system GaitFi, the multimodal machine learning method is feature-level fusion. Co-learning is another popular research topic in the multimodal machine learning domain, which can model resource-poor modalities by leveraging knowledge from other resource-rich modalities [rahate2022multimodal]. It achieves this capability by using transfer learning and domain adaptation methods [zou2019consensus]. There is also a type of work in co-learning called co-training [ning2021review], which is responsible for studying how to expand a small number of annotations in multimodal data to obtain more annotation information.

Different from the existing WiFi-based gait recognition methods that simply formulate the problem as a standard classification problem, we formulate it as a gait retrieval task that is more practical in reality. The gait retrieval task is similar to the visual pedestrian ReID task [lin2019improving] and visual gait recognition [chao2019gaitset]. Given gallery samples and probe samples (i.e. test samples), the objective is to find those samples in the probe that have the same identity as the gallery samples. Therefore, the process of gait recognition is to match the test gait sample with existing gallery gait data, which allows users to enlarge the categories easily in practice. Our system uses two modalities, WiFi and vision, to get richer gait information from different levels. As shown in Fig. 2, the two modalities can reflect the gaits of different people and indicate the occupancy condition. Then we introduce the two modalities of gait data.

Having data from the two modalities, the multimodal learning module accounts for representing and recognizing human gaits. Given a set of $N$ subjects with $M$ samples per subject, we denote the dataset as $D^T$

where $x_w^{ij}$ denotes the $j$-th sample of the $i$-th subject in the WiFi modality, $x_v^{ij}$ denotes the corresponding sample of vision modality and $y^{ij}$ denotes the ground truth of its human ID. The objective of our GaitFi is to map a test sample to its subject ID, denoted as

where $\Phi (\cdot )$ is our gait recognition model.

System setup. To evaluate the performance of the GaitFi system using the gait recognition method, we use two commercial TP-LINK N750 routers as WiFi transmitters and receivers respectively to acquire CSI data, and an Intel RealSense camera to acquire vision data for human gaits. The routers and the camera are shown in Fig. 4(a). The testbed is set up in an indoor environment, as shown in Fig. 4(b), where the camera and the router as the WiFi receiver are on one side of the photo, while the router as the WiFi transmitter is set on the other side. The receiver and the RealSense camera are connected to the same mini-PC for synchronization and data annotations. The WiFi routers are set to run at 5GHz with 40MHz bandwidth, whose firmware is upgraded as described in Sec. III-A1 to collect 114 subcarriers of CSI data for each TX-RX pair. The receiver is equipped with 3 antennas while the transmitter is equipped with 1 antenna. The distance of the TX-RX pair is $2.1m$. Fig. 4(c) is a top view of the testbed layout, where the sensing devices and the facilities are illustrated.

Data collection. To test the performance of the GaitFi system, we collect a dataset for performance evaluation on the above platform. We invite 12 volunteers with heights between $1.55m$ and $1.85m$ as subjects, where their genders and heights are shown in Tab. II. In comparative experiments, this dataset can effectively demonstrate the advantage and correctness of using GaitFi, compared to other gait recognition methods. The area of the dashed box in Fig. 4(c) shows the area where the subjects walk, while the walking direction of each subject is perpendicular to the line of sight (LoS) of the two routers. Walking from one side to the other side is recorded as a sample. 30 samples (i.e., 15 back and forth) are collected for each subject. The WiFi sensor and camera sensor simultaneously record $2s$ of gait information for each walk. In this manner, we can obtain gait samples of 12 different groups (i.e., subjects), and each group contains 30 WiFi CSI frames and corresponding gait videos. During training, 20 samples of each subject serve as the training set, i.e., the gallery set, while the remaining 10 samples are utilized as the probe set. The gait videos obtained by the camera are drawn at an interval of $0.035s$ to form the original visual gait frame sequence, and the pixel size of each frame is $640\times 480$. For the CSI data obtained by the WiFi sensor, the sampling rate of the receiver is 800 $packets/s$, and the sensing time is $2s$, so a WiFi CSI data frame has 1600 $packets$. Because the transmitter router has 1 antenna, and the receiver router has 3 antennas, with Eq. 3, the size of each WiFi CSI data frame is $3\times 114\times 1600$.

Data pre-processing and implementation details. Before inputting the data into the end-to-end training model shown in Fig. 1, the data collected in Sec. IV-A needs to be preprocessed first. For each raw WiFi CSI data frame, we first remove all NaN (Not a Number) values that are caused by the loss of packet in the CSI data. Then the CSI data is normalized and sampled into a size of $3\times 114\times 500$ which is the input of the WiFi sensor module. As far as the vision data frame is concerned, the frame images extracted by the camera contain too much redundant information, which is not conducive to the extraction of gait features. As shown in Fig. 6, we use a Gaussian mixture model (GMM) for background subtraction to get a binarized gait silhouette [wang2003silhouette] first. Then we cut and align the silhouettes, which is the standard pipeline for vision-based gait recognition methods [takemura2018multi]. In this process, we can discard the silhouettes that do not contain any person. If the video sequence length is less than 32 frames, we repeat the last frame of the sequence to make up for 32 frames. The model structure has been illustrated in Tab. I. The learning scheme of the GaitFi is implemented by PyTorch, and the model is trained on one NVIDIA GTX 1660Ti. The Adam optimizer is leveraged for better convergence. The batch size is set to 32 with a learning rate of ${10}^{-3}$ and a total of 30 epochs.

To evaluate the performance of the GaitFi system for the human identification task utilizing the gait recognition method, we process our dataset by utilizing methods from other research on gait recognition based on WiFi modality or vision modality. In the case of WiFi modality, we compare our method with novel WiFi-based human sensing methods including BeAware [jia2020beaware], CSAR [wang2018channel], DeepSense [zou2018deepsense], and the vision-based gait recognition method including CNN-LB [wu2016comprehensive], PTSN [liao2017pose] and GaitSet [chao2019gaitset]. In Tab. III, our two-modality method achieves the state-of-the-art performance of 94.2% accuracy. In comparison, the recognition accuracy of the BeAware is 73.3%, since it only uses WiFi and a simple CNN module. When learning the WiFi modality using CSAR [wang2018channel] that consists of 4 LSTM modules, its recognition accuracy is 81.7%. DeepSense [zou2018deepsense] innovatively combines CNN and LSTM to process WiFi CSI data, and its recognition accuracy can reach 85.0%. For the vision modality, we first evaluate CNN-LB [wu2016comprehensive] which contains CNN feature extractors with a MLP (Multi-layer perceptron) classifier, and the recognition accuracy is 68.3%. The PTSN [liao2017pose] proposed by Liao et al. is a very representative sequence-based gait recognition method utilizing LSTM for video gait recognition, where the recognition accuracy is 88.3%. Then we compare our method with the state-of-the-art vision-based solution, GaitSet [chao2019gaitset], which has outstanding recognition accuracy on the public gait dataset CASIA-B. The GaitSet achieves an accuracy of 92.5% when it is applied to the single modality of vision in our dataset. Since the illumination condition is not ideal in the lab, the vision-based method may be affected and its performance is therefore degrading.

We also investigate our backbone network using different combinations of modalities and network structures. When we use the optimized lightweight residual convolution network WiFi-LRCN shown in Tab. I for WiFi modality, the accuracy can reach 90.8%. For vision modality, if we only use LRCN to extract the features of each frame and perform element-wise addition to get the gait features, the recognition accuracy is only 69.2%. The reason for this is that the element-wise addition at the frame level ignores the correlation between consecutive frames in a sequence, which is important for gait. To extract sequence-level features, we use LSTM to act on the output features from LRCN, achieving 90.8% accuracy. Although the performance of the lightweight backbone network is not as good as a complete vision-based solution GaitSet, it can save computing resources and be more efficient in identity inference. By fusing WiFi and vision two modalities, our GaitFi system achieves an accuracy rate of 94.2%, which demonstrates the advantages of multimodal sensing. GaitFi can learn gait features of vision and WiFi modalities at the same time, improve the recognition accuracy, and enable the system to achieve better effectiveness than a single modality.

To study the robustness of the GaitFi system, we select 6 subjects to conduct experiments in the scene with poor illumination conditions, where 40 samples of gaits are collected from each subject, 25 of which are used as the training set and the gallery set, and the other 15 are used as the probe set. The results are shown in Tab. IV. Methods based on vision modalities perform poorly. The CNN-LB achieves 57.8%, and the PTSN only attains 62.2%. Even the state-of-the-art vision solution, the GaitSet, only achieves 76.7%. In contrast, the WiFi modality has better robustness against poor illumination. The BeAware, the CSAR and the DeepSense achieve 71.1%, 74.4%, and 80.0%, respectively. By utilizing WiFi and vision modalities, our GaitFi system achieves the best accuracy of 85.6%. The results illustrate that the CSI data extracted from WiFi is a good complementarity to vision modality, which can enhance the robustness of our system against poor illumination.