Classical-to-quantum convolutional neural network transfer learning

Machine learning (ML) with a parameterized quantum circuit (PQC) is a promising approach for improving existing methods beyond classical capabilities Romero_2017 ; PhysRevA.98.032309 ; Benedetti_2019 ; cong_quantum_2019 ; cerezo2020variational ; mangini_quantum_2021 ; PRXQuantum.2.040337 . This is a classical-quantum hybrid algorithm in which the cost function and its corresponding gradient are computed using quantum circuits PhysRevLett.118.150503 ; PhysRevA.99.032331 and the model parameters are updated classically. Such hybrid ML models are particularly advantageous when cost function minimization is difficult to perform classically peruzzo_variational_2014 ; McClean_2016 ; cong_quantum_2019 . These models optimize the quantum gate parameters under the given experimental setup, and hence can be robust to systematic errors. Furthermore, they are less prone to decoherence because iterative computation can be exploited to reduce the quantum circuit depth. Thus, the hybrid algorithm has the potential to achieve quantum advantage in solving various problems in the noisy intermediate-scale quantum (NISQ) era Preskill2018quantumcomputingin ; bharti2021noisy .

A critical challenge in the utilization of PQC for solving real-world problems is the barren plateau phenomenon in the optimization landscape, which makes training the quantum model that samples from the Haar measure difficult as the number of qubits increases McClean2018 . One way to avoid the barren plateau is to adopt a hierarchical structure grant_hierarchical_2018 ; pesah2020absence , in which the number of qubits decreases exponentially with quantum circuit depth, such as in quantum convolutional neural networks (QCNNs) cong_quantum_2019 . The hierarchical structure is interesting from a theoretical perspective because of its close connection to tensor networks grant_hierarchical_2018 ; HUANG202189 . Moreover, the shallow depth of a QCNN, which grows logarithmically with the number of input qubits, makes it well suited for NISQ computing. In addition, an information-theoretic analysis shows that the QCNN architecture can help reduce the generalization error PRXQuantum.2.040321 , which is one of the central goals of machine learning. All these factors motivate the application of QCNN for machine learning. QCNNs have been shown to be useful for solving both quantum cong_quantum_2019 ; maccormack_branching_2020 and classical hur2021quantum problems despite their restricted structure with a shallow-depth quantum circuit. In Ref. hur2021quantum , for binary classification with the MNIST 726791 and Fashion-MNIST xiao2017fashionmnist datasets, QCNN yielded higher classification accuracy than the classical convolutional neural network (CNN) when only 51 or fewer parameters were used to construct these models. The best-known classical CNN-based classifiers for the same datasets typically employ millions of parameters. However, the size of the quantum circuits that can be implemented with current quantum devices is too small to incorporate such a large number of parameters. Therefore, two important issues remain. The first is to verify whether a QCNN can continue to outperform its classical counterpart as the number of trainable model parameters increases. The second is to utilize small QCNNs that can be realized in the near future to the full extent, so that a quantum advantage can be achieved in solving practical problems. The latter is the main focus of this work.

An ML problem for which the quantum advantage in the few-parameter regime can be exploited is transfer learning (TL) Bozinovski2020ReminderOT ; 5288526 ; 10.1007/978-3-030-01424-7_27 ; Goodfellow-et-al-2016 . TL aims to utilize what has been learned in one setting to improve generalization in another setting that is independent of the former. TL can be applied to classical-quantum hybrid networks such that the parameters learned for a classical model are transferred to training a quantum model or vice versa Mari2020transferlearning . In the classical-to-quantum (C2Q) TL scheme, the number of qubits increases with the number of output nodes (or features) of the pre-trained classical neural network. This indicates that the transferred part of a classical neural network should have a small number of output nodes to find applications in the NISQ era. For example, using a pre-trained feedforward neural network with a large number of nodes throughout the layers would not be well suited for near-term hybrid TL. By contrast, building a TL model with a classical and quantum CNN is viable because the number of features in the CNN progressively decreases via subsampling (i.e., pooling), and the QCNN has already exhibited an advantage with a small number of input qubits.

Motivated by the aforementioned observations, we present TL from a classical to quantum convolutional neural network (C2Q-CNN). By conducting numerical simulations with PennyLane bergholm2020pennylane , we investigate the performance of the C2Q-CNN for MNIST data classification, where the classical CNN is pre-trained with Fashion-MNIST data. The simulation benchmarks the classification accuracy under various quantum convolution and pooling operations. We compare C2Q-CNN with various classical-to-classical CNN (C2C-CNN) TL schemes and show that C2Q-CNN achieves a higher classification accuracy than C2C-CNN under similar training conditions.

The remainder of this paper is organized as follows. Section 2 reviews QCNNs and TL. This section also introduces the generalization of the pooling operation of the QCNN. Section 3 explains the general framework for C2Q-CNN TL. The simulation results are presented in section 4. MNIST data classification was performed with a CNN pre-trained for Fashion-MNIST data, and the performance of the C2Q-CNN models was compared with that of various C2C-CNN models. The conclusions and outlook are presented in Section 5.

An extension of TL to quantum ML was proposed, and its general concept was formulated in Ref. Mari2020transferlearning . Although the performances of the quantum models were not compared with those of their classical counterparts, three different scenarios of quantum TL, namely C2Q, quantum-to-classical, and quantum-to-quantum, were shown to be feasible. Among these three possible scenarios, we focus on C2Q TL as mentioned in Section 1, because we aim to utilize QCNNs in the few-parameter regime to the full extent. Sufficient reduction of the data dimensionality (i.e., the number of attributes or features) by classical learning would ensure that the size of a quantum circuit subject to training is sufficiently small for implementation with NISQ devices. The dimensionality reduction technique is also necessary to simplify expensive quantum state preparation routines to represent classical data in a quantum state PhysRevA.64.014303 ; PhysRevA.73.012307 ; Mosca:01 ; PhysRevA.83.032302 ; Mottonen:2005:TQS:2011670.2011675 ; araujo_divide-and-conquer_2021 ; 9259210 ; PhysRevA.102.032420 ; araujo2021configurable .

C2Q TL has been utilized for image data classification Mari2020transferlearning ; mogalapalli_classicalquantum_2022 and spoken command recognition qi2021classical . These works serve as proof of the principle of the general idea and present interesting examples to motivate further investigations and benchmarks. The parameterized quantum circuits therein are vulnerable to the barren plateau problem, because they follow the basic structure of a fully connected feedforward neural network with the same number of input and output qubits. Moreover, these studies used classical neural networks to significantly reduce the number of data features to only four or eight. This means that most of the feature extraction is performed classically; hence, the necessity of the quantum part is unclear. These studies encode the reduced data onto a quantum circuit using simple single-qubit rotations, also known as qubit encoding grant_hierarchical_2018 ; PhysRevA.102.032420 , which makes the number of model parameters grow polynomially with the number of data features. In contrast, the number of model parameters in our ML algorithm scales logarithmically with the number of input qubits. Furthermore, all of these works use only one type of ansatz based on repetitive applications of single-qubit rotations and controlled-NOT gates. Finally, the performance of C2Q TL was not compared with that of the C2C version in any of these studies. Because the pre-trained classical neural network performs a significant dimensionality reduction (and hence feature extraction), the absence of a direct comparison with C2C TL raises the question of whether the quantum model achieves any advantage over its classical counterparts.

In this study, we developed C2Q TL with a QCNN and compared C2Q TL results with C2C TL. Transferring pre-learned information to QCNN is important because QCNN can avoid the barren plateau effect. Moreover, because previous studies have shown the advantages of QCNNs over their classical counterpart in the few-parameter regime, fine-tuning an ML model with a QCNN is expected to improve the classification performance compared to fine-tuning it with a CNN. An example of TL using C2Q-CNN, which was used in our benchmark studies, is illustrated in Fig. 4.

The general model is flexible with the choice of data encoding, which loads classical data features to a quantum state $|\psi {⟩}_d$, and ansatz, the quantum circuit model subject to training. We performed extensive benchmarking over the various ansatzes presented in Section 2.1 to classify MNIST data using a classical model pre-trained with Fashion-MNIST data. Finally, we compared the classification accuracies of C2Q and various C2C models. The C2Q models performed noticeably better than all C2C models tested in this study under similar training conditions. More details on the simulation and results are presented in the following section.

To demonstrate the advantage of using a QCNN in TL, we performed classical simulations of binary classification using PennyLane bergholm2020pennylane . The benchmark was performed using two standard image datasets, MNIST and Fashion-MNIST, which were accessed through Keras chollet2015keras . Examples of the datasets are shown on the left in Fig. 4. Note that both datasets have 28$\times$28 features and 10 classes. Among the 10 classes of MNIST data, we performed three independent binary classification tasks aimed at distinguishing between 0 and 1, between 2 and 3, and between 8 and 9. To represent classical data as a quantum state in a QCNN, the classical data must be encoded into a quantum state. The number of data features that can be encoded in $N$ qubits ranges from $N$ to $2^N$ depending on the choice of the encoding method grant_hierarchical_2018 ; PhysRevA.102.032420 ; Havlicek2019 ; araujo_divide-and-conquer_2021 ; araujo2021configurable . Among many options, we used amplitude encoding to represent as many features as possible (see  A). All C2Q-CNN simulations were performed with eight input qubits, to which the amplitude encoding loads 256 features. Quantum feature maps that encode only eight features, such as qubit encoding, are not considered because they require extreme dimensionality reduction on the classical end, which may dominate the classification result as described in the previous section. Furthermore, amplitude encoding was shown to work well with QCNNs for classical data classification hur2021quantum .

The reference classical CNN model (Fig. 4) was trained using 60,000 Fashion-MNIST data for multinomial classification. For the fine-tuning process, we used 10,000 MNIST data to train the QCNN part of the hybrid TL model. In the meantime, the classical CNN part is frozen and uses the set of parameters found during the Fashion-MNIST data classification, as required in TL. The trainable parameters in the QCNN were optimized by minimizing the cross-entropy cost function with the Adam optimizer kingma2014adam using PennyLane bergholm2020pennylane . The number of MNIST test data was approximately 2,000.

C2Q transfer models can be split into three sets based on different pooling variations. The first set uses ZX pooling with convolution circuits (a)-(j), as shown in Fig. 2. The second set includes the general pooling circuit and convolution circuits (a)-(j), as shown in Fig. 2. Finally, we constructed transfer models without parameterized quantum gates in pooling layers. We refer to this pooling strategy that merely traces out one of the qubits as trivial pooling. Trivial pooling is tested with convolution circuits (j) and (k), as shown in Fig. 2.

To compare C2Q-CNN TL classification results with its classical counterparts, 1D and 2D CNN C2C TL models were constructed with a similar number of trainable parameters as C2Q-CNN models. The 1D CNN model was composed of a 1D convolution layer and 1D max pooling layer with 64 trainable parameters. Similarly, the 2D CNN model was composed of a 2D convolution layer and 2D max pooling layer with 76 trainable parameters. The CNNs subjected to fine-tuning for the MNIST data use the cross-entropy cost function with the Adam optimizer, as in the C2Q-CNN case. These classical CNN architectures are built using Keras chollet2015keras . Detailed descriptions of C2C models are provided in B. The training process used mini-batch gradient descent with a batch size of 50 and a learning rate of 0.01. We also fixed the number of training iterations at 200 for the C2Q TL and C2C TL models. The other training conditions were kept the same in the C2Q and C2C transfer models to make the comparison as fair as possible.

The TL classification results are shown in Fig. 5. Each bar represents the C2Q classification accuracy averaged over ten randomly initialized parameters. Different bars along the x-axis represent different convolutions, labeled according to Fig. 2. The unfilled, filled, and hatched bars represent the results of ZX pooling, general pooling, and trivial pooling, respectively. The blue dashed line and green solid line represent the results of the C2C TL using 1D and 2D CNN architectures, respectively. A full list of simulation results for each of the ten random initialization instances is provided in C.

The 0 vs. 1 classification results are shown in Fig. 5 (a). For ZX and general pooling, most of the average test accuracies were greater than 95% and 98%, respectively. The test accuracy with ZX pooling with convolution 2, 4, 7, 8, 9, and 10 ansatz and general pooling with all convolution ansatz is greater than that of 1D and 2D C2C TL. The accuracy with trivial pooling with convolution 10 ansatz was also higher than that of C2C TL. In addition, the accuracy of trivial pooling with convolution ansatz 10 is higher than that of trivial pooling with convolution ansatz 11. This is because convolution ansatz 10 has more trainable parameters than convolution ansatz 11. The test accuracy of general pooling is greater than that of ZX pooling when the convolution ansatz is the same. This can be inferred from the fact that general pooling can be trained to learn ZX pooling.

The 2 vs. 3 classification results are shown in Fig. 5 (b). Most of the ZX pooling average accuracies were between 70% and 90%, and most of the general pooling average accuracies were between 85% and 90%. The test accuracies in (b) are lower than those of (a) because 2 vs. 3 image classifications are more difficult than 0 vs. 1 image classifications. All C2Q TL classification accuracies are higher than the C2C classification accuracies except for ZX pooling with convolution 1 and 3, which use a much smaller number of parameters than the purely classical TL. As shown in (a), the test accuracy of general pooling is greater than that of ZX pooling when the convolution is the same. In addition, the accuracy of trivial pooling with convolution ansatz 10 is higher than that with ansatz 11.

The 8 vs. 9 classification results are shown in Fig. 5 (c). Most of the ZX pooling average accuracies were between 70% and 90%, and most of the general pooling average accuracies were between 85% and 90%. The test accuracies in (c) are lower than those in (a) because 8 vs. 9 image classification is more difficult than 0 vs. 1 image classification, but the accuracies in (c) are similar to those in (b). All C2Q TL classification accuracies are higher than the C2C TL classification accuracies except for ZX pooling with convolution 1, 2, and 3, which use a much smaller number of parameters than the purely classical TL. As shown in (a) and (b), trivial pooling with convolution ansatz 10 has higher accuracy than trivial pooling with ansatz 11. The accuracy of general pooling is mostly greater than that of ZX pooling when the convolution is the same, but there are exceptional cases such as convolution ansatz 7. These exceptional cases show that increasing the number of trainable parameters does not always improve the test accuracy.

The results in Fig. 5(a), (b), and (c) show a tendency for higher convolution ansatz numbers to yield higher accuracy. It can be interpreted as QCNN’s tendency to perform better when the convolution circuits have a larger number of trainable parameters. However, increasing the number of trainable parameters does not always guarantee to improve the test accuracy because it is affected by various conditions, such as statistical error and quantum gate arrangement. For example, ZX pooling with convolution ansatz 5, 6, 7 in (b) has the same number of trainable parameters, but their average accuracies are different. We did not observe overfitting in our benchmark, but care must be taken to avoid it when the number of model parameters grows beyond what is used in this work.

In summary, general pooling mostly produces higher classification test accuracy than ZX pooling with the same convolution circuit. The accuracy of all ZX pooling, general pooling, and trivial pooling circuits tends to be higher when the convolution circuits have a larger number of gate parameters. Although C2Q models have fewer trainable parameters than C2C models, most C2Q models outperform C2C models.

In this study, we propose a classical-to-quantum CNN (C2Q-CNN), a transfer learning (TL) model that uses some layers of a pre-trained CNN as a starting point for a quantum CNN (QCNN). The QCNN constitutes an extremely compact machine learning (ML) model because the number of trainable parameters grows logarithmically with the number of initial qubits cong_quantum_2019 and is promising because of the absence of barren plateaus pesah2020absence and generalization capabilities PRXQuantum.2.040321 . Supervised learning with a QCNN has also demonstrated classification performance superior to that of its classical counterparts under similar training conditions for a number of canonical datasets hur2021quantum . C2Q-CNN TL provides an approach to utilize the advantages of QCNN in the few-parameter regime to the full extent. Moreover, the proposed method is suitable for implementation in quantum hardware expected to be developed in the near future because it is robust to systematic errors and can be implemented with a shallow-depth quantum circuit. Therefore, C2Q-CNN TL is a strong candidate for practical applications of NISQ computing in ML with a quantum advantage.

To verify the quantum advantage of C2Q-CNN, we compared two classical-to-classical (C2C) TL models with C2Q models. These C2C and C2Q TL models have the same pre-trained CNN model, and the C2C TL models are designed to have slightly larger numbers of model parameters than C2Q TL models. The pre-training was carried out with the Fashion-MNIST dataset for multinomial classification, and the TL models were used for three independent binary classification tasks with the MNIST data. Classification results obtained via simulations using PennyLane and Keras showed that C2Q models produced higher classification accuracy in most cases, even though the number of trainable parameters in the C2Q model was smaller than that in the C2C model.

The potential future research directions are as follows. First, the reason behind the quantum advantage demonstrated by C2Q-CNN remains unclear. Although rigorous analysis is lacking, we speculate that this advantage is related to the ability of a quantum measurement to discriminate non-orthogonal states, for which a classical analog does not exist. Moreover, verifying whether the quantum advantage would continue to hold as the number of trainable parameters increases and for other datasets would be interesting. To increase the number of model parameters for a fixed number of features and input qubits, one may consider generalizing the QCNN model to utilize multiple channels, as in many classical CNN models. Note that, in the TL tested in our experiment, the final dense layer was replaced with a model subjected to fine-tuning, while the entire convolutional part was frozen. Testing the various depths of frozen layers would be an interesting topic for future research. For example, freezing a smaller number of layers to use the features of an earlier layer of the convolutional stage can be beneficial when the new dataset is small and significantly different from the reference. Finally, the focus of this study was on inductive TL, for which both the reference and new datasets were labeled. Exploring the possibility of applying quantum techniques to other TL scenarios, such as self-taught, unsupervised, and transductive TL 5288526 , remains an open challenge.

This research was supported by the Yonsei University Research Fund of 2022 (2022-22-0124), the National Research Foundation of Korea (Grant Nos. 2019R1I1A1A01050161, 2021M3H3A1038085, 2019M3E4A1079666, 2022M3E4A1074591, and 2021M3E4A1038308), and the KIST Institutional Program (2E31531-22-076).