Unrestricted Black-box Adversarial Attack Using GAN with Limited Queries

Many deep-learning applications have been deployed in security-important areas such as face recognition [balaban2015deep], self-driving car [grigorescu2020survey], and malware detection [Droid]. However, the recent deep neural network (DNN) models have been known to be vulnerable to adversarial examples [carlini2017towards, goodfellow2014explaining, papernot2016limitations]. A lot of studies have presented adversarial attack methods in various domains such as image, text, and audio applications [alzantot2018generating, carlini2018audio, ebrahimi2017hotflip]. The adversarial examples can be also used to improve the robustness of automated authentication systems such as CAPTCHAs [shi2021adversarial, na2020captchas].

One key aspect in categorizing the threat model of adversarial attacks is the accessibility to components of DNN models, known as white-box or black-box (Figure 2). In the white-box threat model, the attacker can access the whole information of a DNN model, including its weights and hyper-parameters. Many adversarial attacks assume the white-box threat model. The fast gradient sign method (FGSM) is proposed to generate an adversarial example by calculating the gradient only once [goodfellow2014explaining]. The projected gradient descent (PGD) is then proposed to efficiently generate a strong adversarial example with several gradient calculation steps [madry2017towards]. The CW attack is commonly used to find an adversarial example with a small perturbation by calculating a gradient vector typically more than thousands of times [carlini2017towards].

In the black-box threat model, the attacker can access only the output of a DNN model. The black-box threat model can be divided into two variants, the score-based threat model and the decision-based threat model. The score-based threat model assumes that an attacker is able to access the output of the softmax layer of a DNN model. Natural Evolution Strategy (NES) attack generates adversarial examples by estimating the gradient based on the top-$k$ prediction scores of a DNN model [ilyas2018black]. On the other hand, the decision-based threat model assumes that an attacker is able to get the final decision of a DNN model, i.e., a predicted label alone. Boundary Attack (BA) uses random walks along the boundary of a DNN model to generate an adversarial example that looks similar to the source image [brendel2017decision]. HopSkipJump-Attack (HSJA) is then proposed to produce an adversarial example efficiently by combining binary search and gradient estimation [chen2020hopskipjumpattack]. As decision-based attacks (BA and HSJA) can access only the top-1 label, they start with an image already classified as the target class and maintain the classification result of the adversarial example during the whole attack procedure [brendel2017decision, chen2020hopskipjumpattack].

In this paper, we consider the decision-based threat model known as the most difficult black-box setting where an attacker can access only the top-1 label. This threat model is suitable for a real-world adversarial attack scenario because recent real-world applications such as the Clarifai service may provide only the top-1 label. Specifically, we present a variant of the HSJA [chen2020hopskipjumpattack], namely Latent-HSJA, by adding an encoding step to the original HSJA procedure. The most significant difference between the original HSJA and our attack is that our attack leverages a latent space, not an input image space.

Generative Adversarial Network (GAN) has been proposed to generate plausible new data examples [goodfellow2014generative]. Especially, GANs with deep convolutional layers have shown remarkable achievements and are able to produce realistic examples in an image-specific domain [radford2015unsupervised]. Previous studies with GAN have shown that it is possible to generate high-resolution images up to 1024 $\times$ 1024 resolution in various domains such as the human face, vehicles, and animals [karras2017progressive, karras2019style]. Recently, StyleGAN architecture shows an outstanding quality of the synthesized image by combining progressive training and the idea of style transfer [karras2019style, karras2020analyzing].

With the advent of GAN, some previous studies have shown that adversarial examples can exist in the distribution of GAN [poursaeed2019fine, athalye2018obfuscated]. It implies that an attacker can generate various adversarial examples by manipulating latent vectors of GAN. From this intuition, we propose a novel method that efficiently utilizes GAN for generating unrestricted adversarial examples that look perceptually natural. With extensive experiments, we have found that the StyleGAN2 architecture is suitable for our Latent-HSJA to efficiently generate realistic unrestricted adversarial examples in a black-box setting [karras2020analyzing].

Most previous studies consider attack methods that generate adversarial examples constrained to the specific $p$-norm bound [goodfellow2014explaining, carlini2017towards, papernot2016limitations]. Specifically, $l_0$, $l_1$, $l_2$, and $l_{\infty }$ norms are commonly used  [brendel2019accurate]. Therefore, previous defense methods also focus on a norm-constrained adversarial perturbation. Especially, recently proposed defense methods [cohen2019certified, wong2018provable] provide certified robustness against an adversarial perturbation with a specific size of $p$-norm bound. On the other hand, recent studies have proposed various unrestricted adversarial attack methods that are not norm-constrained [poursaeed2019fine, kakizaki2019adversarial, bhattad2019unrestricted, brown2018unrestricted] (Figure 3). Moreover, some studies have shown that these unrestricted attacks can bypass even the certified defense methods [song2018constructing, ghiasi2020breaking].

Nonetheless, the current unrestricted adversarial attacks have not yet successfully expanded to the black-box threat model. A related study has proposed an unrestricted black-box attack using an image-to-image translation network. However, it requires a large number of queries (more than hundreds of thousands of queries) in a black-box setting, which is not desirable for fooling real-world applications [kakizaki2019adversarial]. In this paper, we present a targeted attack method that generates unrestricted adversarial examples with a limited query budget (less than 20,000 queries). To the best of our knowledge, we are the first to propose a targeted black-box attack method that generates unrestricted adversarial examples with limited queries.

We propose a method to conduct an unrestricted black-box attack, namely, Latent-HSJA, consisting of two steps (Figure 4). First, we find latent vectors of source and target images ($w_{src}$ and $w_{trg}$). We use the latent vector of the target image $w_{trg}$ as an initial latent vector for an adversarial example $w_{adv}$. The corresponding adversarial example $G\left(w_{adv}\right)$ should be adversarial (i.e., classified as the target class) at the start of the attack. Second, we conduct a decision-based update procedure in the latent space. Our method always maintains the predicted label of the adversarial example to be adversarial during the whole attack procedure ($F\left(G\right(w_{adv}\left)\right)=y_{trg}$). We illustrate our attack algorithm in Figure 4.

We utilize the HSJA method [chen2020hopskipjumpattack] for the second step of the proposed method (the decision-based update). Our decision-based attack minimizes $D(w_{src},w_{adv})$ while preserving that the model output $F\left(G\right(w_{adv}\left)\right)$ is always classified as the target class $y_{trg}$. After we run the attack algorithm, we get an adversarial latent vector $w_{adv}$ such that $G\left(w_{adv}\right)\approx x_{src}$ in human perception. Our method tends to change the semantic information of an adversarial example $G\left(w_{adv}\right)$ because our method chooses to update the latent vector $w_{adv}$ in the latent space rather than directly update the image $x_{adv}$ in the image space.

As a result, we can generate a perceptually natural adversarial example even in an early stage of the attack because our Latent-HSJA updates coarse-grained semantic features of the image. On the other hand, previous decision-based attack methods based on $p$-norm metrics suffer from a limitation that the generated adversarial example $x_{adv}$ is not perceptually plausible in an early stage of the attack (less than 5,000 queries).

Our attack requires an accurate encoding method that maps an image $x$ into a latent vector $w$ such that $F\left(x\right)=F\left(G\right(Encoder\left(x\right)\left)\right)$. Ideally, a perfect encoding method can satisfy this requirement. We first prepare $w_{trg}$ by using the encoder so that $G\left(w_{trg}\right)$ is classified as an adversarial class $y_{trg}$. Secondly, we also prepare $w_{src}$ by using a given source image $x_{src}$. With this pair ($w_{src}$ and $w_{trg}$), we could finally get the adversarial example $G\left(w_{adv}\right)$ by conducting the Latent-HSJA (Algorithm 1).

For developing an encoding method that finds a latent vector according to an image for the StyleGAN-based generators, two approaches are commonly used. The first is an optimization-based approach that updates a latent vector using gradient descent steps [abdal2019image2stylegan, abdal2020image2stylegan++]. The second approach trains an additional encoder network that embeds an image to a latent vector [DonahueKD16, richardson2021encoding, tov2021designing]. Recent studies have also shown that hybrid approaches combining the two methods could return better-encoded results [zhu2020domain, bau2019seeing]. We have found that the optimization-based encoding method is unsuitable for our attack because it could cause severe overfitting. Previous studies have also shown that such overfitting is not desirable for semantic image manipulation [zhu2020domain, tov2021designing]. When the initial latent vector is highly overfitted, our attack could fail. We have observed that the recent pSp encoder provides successful encoding results for our attack method [richardson2021encoding].

Gender classification is a common binary classification task for classifying a face image. For the gender classification model, the targeted attack setting is the same as the untargeted attack setting. We demonstrate that adversarial examples made from our Latent-HSJA are sufficiently similar to the source images $x_{src}$ (Figure 5). Especially, our method is more efficient in the initial steps than the norm-based adversarial attack (Table 2). Our adversarial examples show better results in the evaluation metrics of SIM and LPIPS compared to the norm-based adversarial attack below 5,000 queries.

For the evaluation of our method against the facial identity recognition task, we have experimented with the targeted attack. We demonstrate that our method is query-efficient for the targeted attack. Our targeted attack is based on Algorithm 1. As illustrated in Figure 6, our method mainly changes the coarse-grained semantic features in the initial steps (Table 2). This property is desirable for the black-box attack since our adversarial example will quickly be semantically far away from the target image. Moreover, the previous decision-based attacks often generate disrupted images that contain perceptible artificial noises with a limited query budget (under 5,000 queries). In contrast, our method maintains the adversarial examples to be always perceptually feasible.