Deep-Learning-Based Device Fingerprinting for Increased LoRa-IoT Security: Sensitivity to Network Deployment Changes ^††thanks: An IEEE-formatted version of this article is published in IEEE Network. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Fig. 1 depicts our testbed, consisting of $25$ Pycom LoRa-enabled devices and Ettus USRP B$210$ receivers. USRP receivers used for signal sampling were configured at a center frequency of $915$MHz and a sampling rate of $1$MS/s. Each device was connected to a dedicated antenna and configured for Raw-LoRa mode transmissions with: $125$KHz bandwidth, a spreading factor (SF) of $7$, a preamble of $8$, a TX power of $20$dBm, and a coding rate of $4/5$. We used MicroPython to program and configure the Pycom transmitter boards, and GNURadio software to configure the USRP B$210$ receivers to capture and preprocess LoRa transmissions and to store sampled data into files. PyCom devices were configured to transmit their data using the LoRa protocol. LoRa uses a tunable parameter, termed spreading factor (SF), to adjust the data rates and energy usage so at to compensate for signal strength degradation or power capacity limitation. SF ranges from 7 to 12, with higher values yielding longer ranges and lower data rates.

Our RF datasets, collected using the testbed described in Section II-A, provide both time-domain IQ samples and their corresponding FFT representations for multiple different scenarios, specifically designed to enable thorough assessment of fingerprinting approaches. In all scenarios, we consider a total bandwidth of $1$MHz that covers in-band as well as an adjacent out-of-band spectrum of LoRa transmissions. The reason for choosing this bandwidth is explained in Section IV when we present our proposed technique. Recorded IQ and FTT data samples are stored into binary files in compliance with the Signal Metadata Format (SigMF), where a metadata file written in plaintext JSON (JavaScript object notation) is created for each binary file to include recording information such as sampling rate, recording time/day, carrier frequency, and other parameters. In this work, we use the following experimental scenarios (The complete RF datasets—with a more comprehensive scenario set, detailed description and downloading links—can found at [elmaghbub2021comprehensive]).

• Different Days (Indoor and Outdoor) Scenario, with data collected over multiple consecutive days using the same receiver. For each day, each device generated ten $20$s-long transmissions, separated one minute apart from each other. This multi-day scenario allows us to study the robustness of deep learning models when trained on data collected on one day but tested on data captured on a different day.

• Different Locations (Indoor and Outdoor) Scenario, to study of how well the deep learning models perform when trained and tested on data captured in different locations. For this, we considered three different locations/environments—room, office, and outdoor, and captured LoRa transmissions at each of these locations, all taken on the same day using the same receiver. One $20$s-long transmission is collected from each device at each location, with $1$-minute duration separating the collections from the consecutive devices.

• Different Configurations (Indoor) Scenario, to study the sensitivity of fingerprinting to LoRa configuration changes. Ideally, the learning models should still perform well even when tested using an SF configuration that is different from that used for training. To assess this, we collected data samples under: 
  Configuration 1: SF = 7; bit rate = 5470 bps 
  Configuration 2: SF = 8; bit rate = 3125 bps  
  Configuration 3: SF = 11; bit rate = 537 bps 
  Configuration 4: SF = 12; bit rate = 293 bps. 
  All four configurations have a transmit power of 20dBm, a bandwidth of 125KHz, and a coding rate of 4/5. For each configuration, one $20$s-long transmission was collected from each device in an indoor setup. All experiments were taken on the same day using the same receiver.

• Different Receivers (Indoor) Scenario, to assess the sensitivity of the learning models to changes in the receiver hardware. This is because receivers, like transmitters, also have hardware impairments that can impact the robustness of the learning models. For this, we collected data from the 25 Pycom transmitting devices with two identical USRP B$210$ receivers. For each receiver, one $20$s-long transmission was captured from each transmitter in an indoor setup.

In all these experiments, in order to allow for the study of the impact of one parameter at time (e.g., time, location, configuration, and receiver), we chose a transmitter-receiver distance that can be fixed for all scenarios. Because $5m$ was the maximum feasible distance in the indoor case, we set that distance to $5m$ for all cases. Unless mentioned otherwise, SF is set to $7$, which is the default value of LoRa protocol.

Typical direct conversion transmitters use quadrature mixer configuration to upconvert, at the carrier frequency $w_c$, the in-phase $\left(I\right)$ and quadrature $\left(Q\right)$ components of the baseband signal using two independent mixers, one used for the $I$ and one for the $Q$ components. First, using the two mixers, the $I$ and $Q$ components are each multiplied (separately and in parallel) with one of the two oscillating signals, one coming from the LO port and the other also coming from the LO port but shifted by 90°. The outputs of the two mixers are summed together to form the upconverted signal modulated at the carrier frequency $w_c$. For an ideal LO, the periodic oscillating signal generated by the LO can be represented as a pure sinusoidal waveform $\cos \left(w_{c}t\right)$, which preserves the original spectrum shape in the upconversion operation. For real scenarios, however, the time domain instability of the oscillating signals generated by real LOs causes random phase fluctuations in the signal, which results in a spectral expansion around the spectrum of the carrier signal. More formally, the oscillating signal generated by real LOs can be expressed as $\cos (w_{c}t+\theta (t\left)\right)$, where $\theta \left(t\right)$ represents the phase noise, which results in a random rotation of the signal constellation observed at the receiver and thus incurs undesired out-of-band (OOB) emission.

To assess the effectiveness of the proposed technique, we implemented a Convolutional Neural Network (CNN) model (see Section V-A for detail) and evaluated the classification accuracy, obtained under each of the proposed (in-band and OOB) and conventional (in-band only) techniques, using our RF datasets described in Section II-B. Fig. 3 shows the average testing accuracy achieved under both techniques for the Indoor and Outdoor setups while considering both domain-time IQ and frequency-domain FFT representations as an input to the learning models. First, note that when the FFT is used as the input to the learning models, the proposed (in-band and OOB) approach achieves a much higher accuracy compared to the conventional approach. More specifically, in both indoor and outdoor scenarios, the proposed approach achieves an accuracy of about $80\%$ whereas that achieved under the conventional one is only about $30\%$. This increase in the accuracy is due to the fact that the proposed technique extracts its features from both the signal’s in-band spectrum and the adjacent spectrum, thereby capturing out-of-band distortion information caused by the hardware impairments as explained in Section III. Second, observe that the proposed approach does slightly better than the conventional approach only in the indoor setting when IQ is used as the input to the learning models. More detail on the evaluation setup and learning model architecture is provided in Section V.

System complexity and training time are other essential metrics when proposing authentication techniques as they impact the speed of the process and challenge the scalability. Our proposed technique does not incur additional computational costs at the transmitters, and instead, it only requires a slightly higher sampling rate at the receiver side, which is feasible without a considerable overhead due to the advances in the sampling capabilities of radio technologies. Furthermore, compared to the in-band only approach, our technique does not result in an increase in the training time, nor does involve more model parameters.

Deep CNNs have been the most commonly used architectures in deep-learning-based RF fingerprinting with the state of the art performances. And hence, CNNs have been perceived as the appropriate candidate for showing the advantage of our technique. For our evaluation, we employ a CNN model similar to the one used in [liu2017deep], where the data input, represented as 2-dimensional (one for real, one for imaginary) tensor, is fed to the first convolution layer of the CNN. This first layer has sixteen $1$x$4$ filters, each learning I and Q time variations, all generating $16$ distinct feature maps (with the exception of the last convolution layer whose filters are of size $2$x$4$ generating features for both I and Q at the same time). After each (except the last) convolution layer there is a Batch Normalization layer, a Leaky Rectified Linear Unit activation, and a Maximum Pooling layer of [$1$ $2$] stride and $1$x$2$ size. The last convolution layer is followed by a $1$x$256$ Average Pooling layer whose output is fed an input to $25$-neuron Fully Connected layer, which is then followed by another activation layer and dropout layer (dropout rate = $0.5$). The output of the Fully Connected layer is fed at last to the classifier layer. The probabilities of each input frame is produced by a Softmax classifier employed at the last layer of the CNN.

Stochastic Gradient Descent is used to update the CNN weights. The momentum optimizer of the gradient descent has an initial learning rate, rate drop factor, rate drop period, and a L2-regularization value of $0.07$, $0.1$, $19$, and $0.0001$, respectively. The categorical cross-entropy is used as the loss function and the back-propagation method is used to minimize the prediction error. We used MATLAB’s Deep Learning Toolbox to implement the CNN model, and run the classifier using our collected datasets (with $80\%$ for training, $10\%$ for validation, and $10\%$ for testing) on a computing cluster containing 6 Nvidia dgx2 nodes with 16 V100 GPUs. For CNN’s input data, we used a sliding window of size $8192$ samples with stride size that is equal to the window size.

We study the use of two different representations of the data input to the learning model: raw time-domain IQ data and its frequency-domain FFT representation.

Fig. 4 shows the fingerprinting accuracy behavior observed when the deep learning models are trained and tested under same/different channel conditions while considering both IQ and FFT representations as input to the models. Fig. 4(a) reveals the accuracy when training and testing are done using the same channel conditions; i.e., the training and testing data is collected during the same time at the same location, Loc 1, which is an Indoor (room) environment. Figs. 4(b) and 4(c) show the accuracy achieved when the channel conditions used during training are different from those used during testing, due to temporal (Fig. 4(b)) or spacial (Fig. 4(c)) variations. The results shown in Fig. 4(b) are based on the Different Days Scenario (Indoor) datasets, and those shown in Fig. 4(c) are based on the Different Locations Scenario datasets, all described in Section II-B. Similar results were obtained for the Outdoor scenarios, but omitted here due to space limitation.

We begin by looking at the results depicted in Fig. 4(a), when the channel conditions used for training and testing are the same. First, observe that the achieved accuracy is consistent throughout each of the 4 studied days. Second, observe that the FFT data representation achieves an accuracy of about $10\%$ higher than that achieved under the IQ representation.

Now when the training and testing are done under different channel conditions, whether due to time change (Fig. 4(b)) or due to location change (Fig. 4(c)), the accuracy drops substantially for both types of input representation, IQ and FFT. Our findings are consistent with previous studies, indicating that the learning models tend to latch into channel-related as opposed to device-related features, making them unable to maintain their high accuracy under channel condition changes.

Another interesting observation we make is that when training and testing are performed under different channels, FFT performs worse than IQ, regardless of whether the channel condition variation occurs due to a temporal or a spatial change. In both types of change, the accuracy drops from about $72\%$ to about $46\%$ under IQ but from about $82\%$ to about $6\%$ under FFT. Our conclusion is that the learning models seem to be more resilient to channel condition changes when using time-domain IQ representation as its input as opposed to using the frequency-domain FFT representation. This suggests that the impact of channel impairment variations is more profound in the frequency domain than in the time domain.

Fig. 5 shows the accuracy when the training and testing are performed while keeping the same wireless channel conditions (same time and same location) but varying the LoRa protocol configuration and/or the receiver hardware. Specifically, it presents the fingerprinting accuracy behavior when training and testing are done under: same LoRa configuration and same receiver hardware (Fig. 5(a)), different LoRa configuration but same receiver hardware (Fig. 5(b)), and same LoRa configuration but different receiver hardware (Fig. 5(c)). For this experiment, we consider the four different LoRa configurations described in Section II-B and two identical USRP B$210$ receivers: Rec 1 and Rec 2. Recall that, as discussed in Section II-B, changing the spreading factor (SF) of LoRa modulation changes the data rate, reception sensitivity, and power consumption, which in turn results in a change of the shape of the spectrum. Fig. 6 shows spectrum shapes of the four LoRa configurations considered in our datasets, which are described in Section II-B. Note the difference in the spectrum shape observed under each of the four LoRa configurations.

From Fig. 5(a), we first observe that FFT input representation consistently outperforms IQ representation when same configuration is used during training and testing. Also, we observe that for the IQ case, the higher the spreading factor, the higher the degradation in the accuracy. Now surprisingly, as can clearly be seen from Fig. 5(b), the learning models perform very poorly, under both IQ and FFT input representations, when tested and trained using different SF configurations, even when data capturing is done during the same time and at the same location. The figure shows that the testing accuracy decreases significantly to about $4\%$ (what random guesses achieve) when configuration 1 is used for training while configuration 2, 3 or 4 is used for testing, and this is regardless of whether FFT or IQ is used for data input.

Fig. 5(c), showing the accuracy behavior when using a receiver during testing that is different from that used during training, allows us to assess the impact of the receiver’s impairments on the performance. This is done by using two identical USRP B$210$ receivers (Rec 1 and Rec 2) that capture the transmissions of the $25$ devices at the same location (indoor/room), using the same LoRa configuration, and during the same time. This set of results is obtained based on the RF datasets presented in the Different Receivers Scenario, explained in Section II-B. Like in the case of changes occurring in the LoRa configuration, Fig. 5(c) shows that the accuracy drops significantly when training and testing are done on different receivers, with a lesser severity in the case of IQ representation; this is true for both: train on Rec1 / test on Rec2 and train on Rec2 / test on Rec1.

We conclude that unlike when changes occur in the channel conditions, the learning models completely lose their ability to classify devices when changes occur in the LoRa configuration and/or the receiver hardware, implying that changes in the system setting yield drastic changes in the data distribution, severely crippling fingerprinting performances.