Laplacian Pyramid-Like Autoencoder

Recently, most research about the autoencoder concentrates on its connection to other applications rather than developing it independently. Generative autoencoder [31, 10] occupies a significant portion as a base model for application. In [17], variational autoencoder (VAE) solves a problem of insufficient data by producing synthetic data from a given sample data. [27] develops a reference-based SR approach using VAE, which can take any image as a reference. [36] suggests a novel generative model with hierarchical structure based on VAE. By exploiting unusual convolutions such as depthwise separable convolution and regular convolution, authors get high performance without modifying the loss function of VAE. Other existing autoencoders, such as sparse autoencoder and denoising autoencoder, are used to solve specific problems. For example, [15] composes stacked sparse denoising autoencoder for detecting electricity theft and [13] exploits a sparse autoencoder for landslide susceptibility prediction. Because the autoencoder extracts the feature efficiently and reconstructs data with a simple structure, it is a popular tool for developing a framework combined with machine learning. Our model, LPAE, is a model possessing the properties of LP. So LPAE can provide new approaches to diverse problems where LP properties are helpful.

After the substantial progress of the convolutional neural network (CNN), a lot of research focuses on the acceleration of the networks with keeping high performance. There are several approaches to accelerate networks. [6, 3] propose methods that modify an architecture or overall structure. [3] suggests an accelerator using the point that can compute the convolution layer similar to the fully connected layer. Moreover, to accelerate model training, some researches present a new training framework. [41] makes an assistant model along with the main model to remove trivial instances during training, then gets the results with the new training algorithm. [25] suggests a new training plan for fine-tuning. It accelerates the fine-tuning by adaptively freezing the nearly converged portion of the network. Since many natural language processing and classification models exploit fine-tuning the pre-trained network, this method is tempting. Furthermore, the study to achieve harmony between the hardware and the software is actively performed [30, 16]. Considering rapidly advanced hardware, both propose a hardware-based approach to speed up a CNN and reduce the energy required for computation.

In the super-resolution area, many of the studies try to build a deep convolution-based model using up-sampling. For example, [19] introduces the way of the deep network for super-resolution by a residual learning and high initial learning rate, while [22] suggests enhancing the power of the deep network by removing the batch normalization and setting a training pipeline. [12] makes a model to repeat the up-scaling and down-scaling iteratively and to reflect the feedback of error. In addition to the construction of deep models, some researchers are focusing on other points. By learning the feature correlations of hidden layers by second-order attention, [8] improves the expressional power of CNN. [34] trains the content-adaptive resampling (CAR) model with the main super-resolution (SR) network to create a low-resolution image. Since unsupervised training is conducted on CAR, CAR makes a down-scaled image to keep important information for super-resolution impartially. The primary concern of [7, 20] is the speed of the SR. While both of them construct a new network structure and framework, neural architecture search is a noticeable characteristic of [7]. The LPAE that we propose in this paper is an assistant model to reconstruct images trained with the main SR model, similar to the approach in [34]. Also, LPAE learns the correlation between the components for super-resolution and can improve the reconstructing power of the main SR model similar to [8]. However, LPAE is an autoencoder with a straightforward structure and is easy to connect with diverse architectures without too much modification.

The Laplacian pyramid (LP) is introduced in [5] as a technique for compact image encoding. This technique has its root in subtracting a low-pass filtered image from the original image. Such subtraction reduces redundant information by decreasing the correlation between neighboring pixels in an image, in other words, data compression. Moreover, this compression process can be repeated on low-pass filtered images having different scales. Then the repetitions form a pyramid-like structure and accelerate the reduction. Since this process is similar to the Laplacian operator sampling on diverse scales, the pyramid is named the Laplacian pyramid.

The following is the overall process of establishing LP. For an input image $I_0$, a filtered image $I_1$ is obtained with the decreased resolution by some low-pass filter. A filtered image $I_2$ is similarly obtained from $I_1$, and proceeding successively results in a sequence $\{I_k{\}}_{k=0}^K$. For each fixed $I_k$ with $k\ge 1$, the image ${\tilde{I}}_k$ with the same size as $I_{k-1}$ is defined by expanding $I_k$ by interpolation. Subtracting ${\tilde{I}}_k$ from $I_{k-1}$ produces $d_k$ with the same size as $I_{k-1}$ but with compressed image information. The sequence $\{d_k{\}}_{k=1}^K$ of such differences corresponds to the pyramid of LP.

According to the above procedure, LP decomposes an image $I$ into a low-pass filtered image $I_c$ and a difference $I_d$ after one stage. The decomposition result of LP is redundant compared to the wavelet transform often used as the analysis tool because the $I_d$ part of LP has the same size as the original image. But some advantages are originated from the structure of LP. For example, the implementation that requires only low-pass filtering and subtraction is straightforward. Another strength of LP is that there exists no scrambled frequency [9] derived from high-pass filtering. And the organizational style of LP enables the perfect reconstruction of the original image. Such advantages encourage many usages of LP as a tool in traditional image processing.

In addition, there have been lots of efforts in machine learning to treat LP, both independently and as a concept in a model. For the super-resolution problem, [2] and [43] get competitive results by introducing the style of LP in the attention module and by considering a generator of the conditional generative adversarial network, respectively. On the other hand, [23, 21] use LP itself to make input images. After generating the image pyramid from the original image, [23] put components into two modules for high-quality transfer, a drafting module that transforms artistic style from low-resolution image and a revision module that refines the transformed image with a high-resolution image. On top of these, there are applications for diverse problems such as object detection [42], image compression [37]. These researches show a possibility of a connection between LP and deep learning, presenting satisfactory results. As will be seen later in this paper, our LPAE presents convincing results, which add to the above possibility.

LPAE has two parts, encoding (analysis) and decoding (reconstruction) parts. Fig. 1 shows the overall structure of LPAE. In the analysis part, LPAE decomposes the original image $I$ into images $I_c$ and $I_d$. $I_c$ is the down-sampled approximation image that is low-pass filtered, and $I_d$ is the detail image representing the difference between the original image and the prediction from approximation. Both outputs pass through $4$ convolution layers. When we downsample images, we use the convolution layer with stride $2$ instead of a pooling layer. In the reconstruction part, the output image $I^{\prime }$ is reconstructed by an element-wise sum between the detail image $I_d$ and the prediction image $\varphi \left(I_c\right)$, where $\varphi$ is a deconvolution process. The process $\varphi$ consists of up-sampling with $4\times 4$ transposed convolution layer and filtering with $3$ of convolution layers. In our LPAE, except for the output layers, we use convolution layers with filter-size $3\times 3$ and 16 output channels for simplicity and efficiency, but the network can be made deeper or more redundant.

Our loss function consists of three components that make the autoencoder similar to LP. Since we want LPAE to reconstruct the original image as much as possible, we define our first loss function to be the reconstruction loss,

For the approximation image representing the low-frequency channel of the original image well, we prepare an approximation image $I_{\downarrow }$ by using the bicubic interpolation and apply the mean square error (MSE). we then define our second loss function as the energy (or approximation) loss,

For the approximation image $I_{\downarrow }$, many other methods, including the wavelet transform by CDF 9/7 filters, can also be used, but we find no significant difference in the loss for using different approximation methods. Hence in this paper, we fix the bicubic interpolation to get $I_{\downarrow }$. The bicubic interpolation makes a natural connection with the super-resolution problem.

To constrain the detail image to be sparse, we set our last loss function to be the sparsity loss,

This sparsity loss makes the detail image carry a high-frequency channel of the original image and provide textures to the connected network.

The overall loss of LPAE is defined as the weighted sum of the three losses:

where $\alpha =\gamma =1,\beta =0.8$ to give less weight on the approximation loss than other losses.

Fig. 2 shows the result of LPAE and WAE applied to two different images from ImageNet. For the approximation image in Fig. 2(b), there is no doubt that the result of LPAE is more vivid, original-like image. Besides, Fig. 2(c), the result of detail images, demonstrates that this precise approximation influences the detail image to be sparse. Although sparse, the detail contains many information for the original texture since it has the same spatial size as the original image, unlike WAE. Eventually, LPAE gets a more sharp reconstruction, as seen from the images in Fig. 2(d). The differences prove the validity of our loss function.

In [6], the authors show that WAE can accelerate classification networks while keeping the accuracy almost the same by using two outputs of the encoding part. We show that LPAE can accelerate classification networks at about the similar level, even with a slight improvement in accuracy. We think that using both the approximation and the detail, similar to [6], has contributed to getting comparable accuracy, even if the input in VGG is not the original image. In this sense, we speculate that there is an improvement in accuracy for LPAE because LPAE gets a better approximation and a high-resolution detail than WAE.

To be comparable with the experiments in [6], we set the same structure except for the autoencoder part, replacing WAE with LPAE. Below we describe the use of the autoencoder only for VGG, as the process for ResNet is similar. Recall that the encoder part of LPAE decomposes an original image $I$ into an approximation image $I_c$ and a detail image $I_d$. The features $f_c$ for $I_c$ are extracted from the feature extraction part of VGG. The features $f_d$ for $I_d$ are extracted from another lighter feature extraction part consisting of convolution layers with only a quarter of the output channels for $f_c$. Using fully connected layers, we obtain the classification score $s_c$ from $f_c$, and $s_d$ from the concatenation of $f_c$ and $f_d$. The final score $s$ is the average of the two scores $s_c$ and $s_d$.

From the acceleration perspective, according to [6], the total complexity of all the convolutional layers can be represented by

Here, $d$ is the number of the convolution layers. For the $l$-th layer, $n_l$ is the number of the output channels, $s_l$ is the size of the kernels, and $m_l\times m_l$ is the spatial size of the output features.

Since LPAE has $\frac{1}{2}$ size of feature maps for the approximation, the complexity $N$ in (1) is $\frac{1}{4}$ of the original network. However, unlike WAE, there is no change in the size of feature maps for the detail. Still following the setup of WAE for the detail, the number of the output channels for the LPAE’s detail becomes $\frac{1}{4}$ compared to the approximation case, so the complexity becomes $\frac{1}{16}$ of the original network. As a result, LPAE has $\frac{5}{16}$ total complexity compared to the original. Thus the acceleration rate is about $1/\frac{5}{16}=3.2$. This number is comparable to $3.76$, the acceleration rate of WAE.

As observed earlier, LPAE’s approximation image in Fig. 2 tends to be similar to some other low-pass filtered images, such as the approximation image of the wavelet transform or the bicubic interpolation. This approximation image helps the detail to be more sparse. Besides, LPAE can make the hierarchical structure because the approximation carries low-frequency information sufficiently. Based on these observations, we try to expand the application domain of LPAE to super-resolution.

There are lots of algorithms in the super-resolution problem. Some of them are hard to be connected to LPAE directly in a natural way. So we try to choose the models having room for substitution, and WaveletSRNet (WaveSR) in [14] is one such model. The basic concept of the network is the use of the wavelet packet transform (WPT) as a reconstructor. For example, to train WaveSR for magnification of $4$, an input image is decomposed into two levels using WPT. Then the network gets the approximation image of $\frac{1}{4}$ size and extracts features from the approximation image through the embedding part. The features form new details of the number needed in the reconstruction process (15 for magnification of 4) and one approximation image. At last, through WPT, the network creates the high-resolution image of magnification of 4. In this procedure, the authors set the loss function of WaveSR to make the new details and approximation similar to the decomposition results of WPT.

In this algorithm, we insert LPAE as a substitute for WPT. The divided encoding and decoding parts can take the role of the existing decomposition and reconstruction, respectively. Moreover, the hierarchical structure of LPAE does not restrict its use only for a magnification of 2 but allows it for an arbitrary magnification of $2^k$. Although LPAE produces larger-sized details than WPT, the redundancy of information is helpful to reconstruct. And the fact that it has a smaller number of detail images makes the network efficient. In addition, we expect the substitute to improve the high-resolution result because it has the flexibility that the reconstruction of LPAE gets better by a fine-tune on various datasets in contrast with WPT.

We modify the loss function given in [14] to fit LPAE’s situation when we train WaveSR substituted with LPAE (named LPSR). The reconstruction loss is defined by using the L1 distance between the reconstruction $I^{\prime }$ and the original high-resolution image $I$,

The use of L1 loss for the whole procedure can reduce the smoothing effect and enhance the quality of the image [44]. Based on this observation, we define the pyramid loss by

where $\left\{{\lambda }_i\right\}$ are weights for the detail part. For instance, we take ${\lambda }_1=0.8,{\lambda }_2=1.2$ in case of magnification of 4.

Finally, the loss function of LPSR is the weighted sum of two losses:

If the new details $I_{d_i}^{\prime }$ and approximation $I_c^{\prime }$ produced by LPSR are close to the outputs $\{I_{d_i},I_c\}$ of LPAE sufficiently, the reconstruction using the new ones will have high quality on high-resolution stage. From this point of view, we focus on the closeness between the outputs of LPAE and LPSR. Hence in all of our experiments, weights $\gamma =1$ and $\delta =10$ are used.

In classification, as mentioned in our paper, we use two backbones, VGG16, ResNet50, and use two datasets, ImageNet2012 (ImageNet), Intel Image Classification datasets (Natural Scene). ImageNet is a large dataset having train images of 1.2 million approximately and validation images of 50k. The images are classified as labels of 1000 and are with varying image sizes. In comparison to ImageNet, Natural Scene is a tiny dataset of 6 categories. There are 14k in train, 10k in validation and test. Moreover, the spatial size of each image is fixed as $150\times 150$.

We mostly follow the training strategies in [6] to train the autoencoders for comparison with WAE. In particular, for both WAE and LPAE, the Xavier algorithm is used for initialization of parameters, and the SGD algorithm with momentum 0.9 and weight decay of 0.0005 is used. However, some options are chosen differently because of the dataset size. Although we keep a batch of 4 as in [6] for Natural Scene, we choose 256 for ImageNet to shorten training time. Also, our training epochs are fixed as 100 for Natural Scene and 20 for ImageNet. Considering the difference between the two autoencoders, we choose the initial learning rate differently. For WAE, it is set as 0.000001 with the decay factor of 0.1 after every 10 epochs as in [6], but for LPAE as 0.01 with the same decaying strategy.

To train the connected classification network, for Natural Scene images, we randomly crop to $128\times 128$, and for ImageNet images, we resize to $256\times 256$ and randomly crop to $224\times 224$. The only data augmentation we select is the random horizontal flip. We choose the batch size to be 256 regardless of datasets, and choose the SGD algorithm with the same options. But the decay strategy and the number of epochs are different along with the dataset. For ImageNet, training is performed with 20 epochs, and then the learning rate is multiplied by 0.1 after 10 epochs. For Natural Scene, the learning rate is multiplied by 0.5 after every 10 epochs during 100 training epochs.

In the super-resolution, we train on the DIV2K dataset and test on the Set5 dataset. Additionally, we use the CelebA dataset for comparison with the original task of WaveSR. DIV2K has 800 diverse images of the large size of 2k resolution, but CelebA has 162,770 images of center-arranged (to some degree) face with the size of $178\times 218$. Because of this substantial dissimilarity, we perform different data augmentations on two datasets.

The autoencoders’ training options are chosen similarly to the classification. For DIV2K, we randomly crop the high-resolution image to $192\times 192$ with random horizontal/vertical flips. We train LPAE using the Adam algorithm with a batch of $4$ and adjust the initial learning rate of $0.001$ to be divided by $2$ after every $50$ epochs during $400$ epochs. To train LPSR, we set a batch of $8$ and select the Adam algorithm with the initial learning rate of $0.001$ decaying to half after every 50 epochs for $300$ training epochs. For the other dataset, CelebA, the training image is resized to $144\times 144$ then randomly cropped to $128\times 128$ with a random horizontal flip only. LPSR is trained during $40$ epochs using the Adam algorithm with a batch of $256$, the initial learning rate of $0.01$ multiplied by $0.1$ after every $10$ epochs. Options for other networks, WaveSR and WSR, are chosen to be identical to LPSR. All of our codes are available at https://github.com/sangjun7/LPAE.

LPAE is motivated by WAE. However, there is a considerable difference between the two autoencoders in their network structure and loss formulation. As we can see in Fig. 2, LPAE gets a high-resolution detail image compared to the detail image of WAE. The large-sized detail image has more abundant information for textures and high-frequency data of the original image. Furthermore, LPAE makes a more specific approximation image than WAE, as seen in Fig. 2. This fact comes from the constraint on the approximation image to resemble the bicubic interpolated image. Also, the change of the MSE loss between the input image and the reconstructed image to the L1 loss with a large learning rate returns high-quality reconstruction [44]. Table 1 shows a difference between the two autoencoders. The PSNR values in the table are calculated between the original image and the reconstruction image. Although PSNR alone cannot determine the quality of reconstruction, many researchers indeed consider a large PSNR value as an essential indication for better reconstruction. For Bicubic, we put back to the original image using bicubic interpolation after reducing the spatial size to half. For ImageNet, we get the large PSNR value of 47.89 dB, far superior to the others, i.e., 28.57 dB for WAE and 28.44 dB for Bicubic. In the DIV2K dataset, LPAE makes the closer reconstruction image to the original than the case of ImageNet. The PSNR value of LPAE is 54.73 dB, and that of WAE is 19.90 dB, which shows a significant difference between PSNR values. LPAE gets one compressed image and another sparse image, and based on the above results, we see that LPAE restores the original image much better. Thus LPAE can accomplish the role of accelerator comparable to WAE and be extended to the super-resolution problem.

As mentioned in Section 4, we expect that the link between LPAE and a classification network reduces the computational cost and accelerates algorithms with a slight drop in accuracy. To show this, we train the basic classification networks (VGG, ResNet) and those connected with WAE and LPAE, using two datasets (ImageNet, Natural Scene). We use our codes for WAE. Although not able to reproduce exactly, we identify a tendency of accelerating presented in [6]. Table 2 shows a comparison of performances between networks. For all cases, classification networks connected with LPAE (LPVGG, LPResNet) get better precision than those connected with WAE (WVGG, WResNet). For Natural Scene, resulting in 89.97$\%$, LPVGG even has the same accuracy as the original VGG. We speculate that more information of the original image is kept after the LPAE’s encoding helps classify. Table 2 also reports the number of trainable parameters for each case for reference. The training times of LPVGG/LPResNet are reduced to a similar level as WVGG/WResNet. For ImageNet, WVGG saves about 2.63 hr than the basic, and LPVGG saves about 2.99 hr that is about 10$\%$ of the whole training time of the basic. For Natural Scene, LPVGG reduces about 31$\%$ of training time than the basic.

Table 3 shows each network’s FLOPs, complexity (Compl.), and test times for different batch sizes. Our complexity here is calculated similarly to $N$ in (1) for convolution layers and fully connected layers. For instance, with VGG on ImageNet, we get an acceleration rate of $\frac{15.47}{5.48}\approx 2.82$ for LPAE, and $\frac{15.47}{4.46}\approx 3.47$ for WAE. These numbers are similar to the rough computation of acceleration rate in Section 4, i.e., $3.2$ for LPAE and $3.76$ for WAE. When we check the results of FLOPs about VGG, we get 31.02 billion (B) for the basic, 8.95 B for LPVGG, and 11.01 B for WVGG. Thus the FLOPs of LPVGG and WVGG are about 0.35 and 0.29 of that of VGG. Hence, the computational cost of LPVGG, similar to WVGG, is vastly reduced. We think that this point leads to the result accelerating LPVGG sufficiently in the test. LPVGG gets 12.67 ms in the test with 1 batch, which is decreased by 1.78 ms. If we raise batch size to 20 or 50 (cf. [39]), the decrease rate becomes larger because LPVGG has 80.77 ms instead of 127.07 ms for size 20, and 171.98 ms instead of 274.55 ms for size 50. Although the connection with LPAE accelerates VGG less than the connection with WAE, there is a meaningful difference in test time between LPVGG and VGG. For other cases with 1 batch, results are unexpected, representing that basic VGG takes the shortest time for the test. We think the unexpected situations are due to the small size of the input. For increased batch, the test time of networks connected with LPAE again becomes smaller than that of original networks.

We evaluate super-resolution networks (WaveSR, WSR, LPSR) on two datasets (CelebA, Set5) using metrics PSNR and SSIM. For the result of WaveSR, we used our codes and could not reproduce results reported in [14] despite the same options. The values in Table 4 are obtained by applying our codes with the same environment to the three WaveletSRNet-based networks, namely, WaveSR, WSR, and LPSR. For CelebA, LPSR has top values among the three networks for all scales. In particular, for $\times 2$ scale, there is the most significant gap when we change WPT to LPAE because PSNR is 36.04 dB for LPSR and 30.87 dB for WaveSR. This shows the power of LPAE reconstructing the image and having a learning method that fits the model’s parameter on data distribution. If we focus on the comparison between WSR and LPSR, the PSNR value of WSR takes a sharp drop with increased scale. WSR for $\times 2$ scale gets 31.93 dB, which is even higher than the PSNR of WaveSR, but for $\times 4$ scale, WSR reaches 17.96 dB. This result describes the limit of WAE, which means that WAE does not consider multi-scale analysis and reconstruction of the image (c.f. Section 1). The same tendency appears once more in the results for the Set5 dataset. LPSR obtains the biggest PSNR, SSIM values among the three networks. Its PSNR values are 37.62 dB for $\times 2$ scale, 32.25 dB for $\times 4$ scale, and 27.07 dB for $\times 8$ scale. For WSR, since Set5 is dissimilar to CelebA, which is the center-aligned data of constant size, reconstruction by WAE falls down to 25.70 dB in $\times 2$ scale. For $\times 4$ or $\times 8$ scale, it is hard to do reconstruction using WSR. Table. 5 shows the comparison of our LPSR on Set5 with SOTA networks. Our PSNR values rank about the middle on average, which is good, considering the fact that LPSR is obtained by a simple change using LPAE from WaveSR. Fig. 2(o) shows the reconstruction of WaveSR, WSR, LPSR for two images of Set5. WaveSR works well for$\times 2$ scale, but as the scale is getting bigger, the reconstruction images of WaveSR are blurred and have a checkers pattern. However, LPSR gets high-quality reconstruction images for all scales.