Leveraging Symmetrical Convolutional Transformer Networks for
Speech to Singing Voice Style Transfer

The proposed SymNet architecture is shown in Fig 2. The input speech spectrogram along with the melody features are passed to two convolutional layers to model the local spectro-temporal patterns. The output of the convolutional layer is passed through a recurrent layer for modeling long-term dependencies. We consider bi-directional long-short term memory (BLSTM) or a transformer layers in this stage. The BLSTM model is a recurrent neural architecture implements using LSTM cells while the transformer models use self-attention layers to incorporate temporal dependency. The next layer of processing involves a separate attention network which performs the non-linear time alignment between the speech and singing voice spectrogram based on the target melody features. The rest of the network consisting of a BLSTM/transformer layer and the two convolutional layers allows the architecture to form a symmetrical model before and after the attention layer. We highlight the importance of the symmetrical architecture and the combination of three types of layers by comparing other possible architectures in our experiments. The detailed SymNet configuration is summarized in Table 1. Each transformer layer has $4$ attention heads along with $2$ encoder and $2$ decoder layers. The dimension of the feed forward layer is set to $1024$.

We utilize a data augmentation method that generates singing voice data from natural speech data. Our prior work [our_work] used a data generation approach for automatic lyric recognition by leveraging the WORLD vocoder [world12]. Specifically, the approach consisted of decomposing the natural speech and singing voice (opera vocals) into three constituent components - spectral envelope, fundamental frequency (F0) and aperiodicity parameter. The F0 contour from the singing voice is then used along with the spectral envelope and the aperiodic parameter from the natural speech in the synthesizer. The synthesized output is the singing voice version of the natural speech. The western opera vocal dataset consists of both male and female opera singers and during the synthesis it was made sure that the speech and the opera vocals are gender and F0 matched [our_work]. We use the synthesized output and the input speech data as the paired data for augmentation. While this approach already generates a singing voice version of the natural speech, we find that the reconstruction quality is not sufficient for commercial applications. More details about the data augmentation can be found in Basak et. al. [our_work],

The boundary equilibrium generative adversarial networks (BEGAN)  [berthelot2017began] is a energy based GAN, which matches the distribution of loss using an auto-encoder architecture, unlike the original GAN which matches the distributions between the real and generated samples directly. Moreover, the BEGAN relaxes the equilibrium of the auto-encoder loss using a hyper-parameter $\gamma \in [0,1]$. In order to implement the BEGAN loss ($L_G$) [BEGAN], the STS model also has an additional discriminator network.

We experimented with two different training settings, (i) without the discriminator (and therefore without the BEGAN loss), and (ii) with the BEGAN discriminator. For the first one, our learning objective is based on mean square error ($L_{MSE}$) between predicted ($\hat{Y}$) and the true log-magnitude spectrograms of singing voice ($Y\in R^{F\times T}$). The combined loss then is, $L=L_{MSE}+\zeta L_G$. Here, $\zeta$ controls the trade-off between the MSE and the GAN loss.

We explore a loss function annealing, where the regularization term $\zeta$ in the joint loss function is annealed as a function of the iteration to focus on the reconstruction MSE in the latter part of the training. In our experiments (Sec 3.6), we explored various annealing schedules, such as a linear decay and a step decay.

We use the NUS data [NUS] as the paired training dataset containing $115$ mins of singing data and the corresponding $54$ mins of speech data. The $48$ recordings of $20$ unique English songs are sung/read by $12$ subjects, and each song-speech paired audio can be time-aligned with their phone annotations. The phone annotations are in accordance with the CMU dictionary ($39$ phonemes). Out of the $20$ unique songs in the dataset, one song (with two recordings) is used as the test set for NUS (similar to the dataset division followed in [sp2si]). We also use NHSS [sharmanhss] for training/testing the models. It consist of $100$ songs with their respective speech recorded by $10$ singers, resulting in a total of $7$ hours of audio data. For data augmentation, the LibriSpeech [panayotov2015librispeech] is converted to synthetic singing voice and about $10$ hours of augmented data is added for model training (Sec. 2.2). We use the log spectral distance (LSD) as the performance metric in our objective evaluation. Further, a subjective test is performed to affirm the quality improvements seen in the objective metric.

Given an input time-domain speech signal and a target F0 contour, the pre-processing consists of the following steps:
Silence frame removal :  All the silence frames from the speech signal are removed. This is achieved via a short-time energy threshold set at $40$dB below the maximum energy frame. Any set of three or more consecutive silent frames (longer than approximately $50$ms) are removed.
Time stretching :  The input spectrogram is linearly time-warped to the length of the F0 contour.
Singing melody contour :  The melody contour is extracted using the CREPE algorithm  [crepe] applied on the singing voice. We also convert the continuous-valued F0 to one of the $128$ MIDI levels. For training with paired data, we extract the melody contour from the singing counterpart of the input speech.
Random re-sampling (RR) :  We have re-sampled the input speech in random fashion to change the rhythm and to disentangle the content and rhythm [BEGAN].

The magnitude specrogram is recovered by applying $g\left(\hat{Y}\right)=e^{\hat{Y}-1}$ element-wise (the target spectrograms are $\hat{Y}=log(1+\hat{y})$, where $y$ is the magnitude spectrogram of the singing voice). The reconstructed magnitude spectrogram $g\left(\hat{Y}\right)$ is then converted to log-mel spectrogram with $80$ frequency bins. Then, the MelGAN vocoder [kumar2019melgan] is used to generate audio waveforms from the spectrogram.

The time domain signals are re-sampled from $44$kHz to $16$kHz. We compute short-time Fourier transform (STFT) with $1024$ frequency samples using $64$ ms window size and $16$ms hop size. For the first setting (the one with just the MSE loss), we use Adam optimizer [kingma2014adam] with initial learning rate of $0.003$ and an exponential decay factor of $0.9$. All the networks were trained for $30$ epochs, (with $1000$ mini-batches in each epoch). For the second setting, the one with BEGAN loss, we use Adam optimizer with initial learning rate of $0.001$ and exponential decay factor of $0.99$. All the neural network implementations are performed using PyTorch [paszke2019pytorch] and Librosa [mcfee2015librosa].

The first set of results, reported in Table 2, show the results for the ablation study with different model architecture choices. All the models reported here use the first setting of the loss function (only MSE). The results reported here are for models with various architectures - three BLSTM layers (with $256$ and $512$ cells), convolutional and BLSTM layers, BLSTM layers with attention, convolutional layers followed by BLSTM layers, convolutional and transformer layers and the proposed SymNet architectures. The key observations from there experiments are,

• [leftmargin=*]

• The mixture of convolutional and transformer/LSTM layers is superior to the use of convolutional layers alone [BEGAN]. The use of $512$D is slightly better than $256$D.

• The use of attention in the model allows the non-linear time alignment between the speech and singing voice counterparts with only a minor increase in the number of parameters.

• The transformer improves over the BLSTM counterparts.

• Combining all three types of layers (Conv-Trans-A) demonstrates the effectiveness of the three types of layers, namely, local modeling with convolution, long-term modeling with the transformer, and dynamic alignment with attention.

• SymNet improves the LSD over the models with the same set of layers and parameters, but are asymmetrical. To the best of our knowledge, this is the first attempt to highlight the importance of the symmetric architecture for audio style transfer tasks.

In Table 3, we consider three schedules for $\zeta$ which controls relative weighting of the BEGAN loss and the MSE loss - (i) $\zeta$ is fixed at $0.3$, (ii) $\zeta$ is initialized to $0.3$ and gradually reduced by $0.001$ for every epoch, (iii) $\zeta$ is initialized to $0.3$ and changed to 0 after $15$ epochs. Table 3 shows the results for the scheduling choices along with the system trained without BEGAN loss ($\zeta =0$). Even though BEGAN loss improves the LSD for fixed $\zeta$, annealing $\zeta$ further improves the performance.

The results comparing different components of the proposed model are given in Table 4. The encoder-decoder model reported in [BEGAN] serves as the baseline system. The results reported here show that the incorporation of the SymNet architecture with BEGAN loss improves over the baseline system. Further, the incorporation of the data augmentation method improves the reconstruction quality. The effect of additional training data is explored by training the $SymNe{t}_{512}$ model with additional data from NHSS dataset [sharmanhss]. Further the combination of the data-augmentation approach along with the addition of the NHSS dataset improves the reconstruction quality with an absolute improvement of $0.97$ dB over the baseline system on the NUS dataset and $1.46$ dB on the NHSS dataset.

We conducted a mean opinion score (MOS) listening experiment¹¹1The study is hosted at https://symnettesting.herokuapp.com/ with $20$ audio clips, each with three versions - hidden reference (ground truth), style transferred from speech using the proposed SymNet approach and using the baseline system [BEGAN]. The three audio files for each snippet were presented in a random order. The participants belonged to the age group of $21-39$ with normal hearing. The subjects were asked to listen to the audio stimuli using high quality headphones and rate the files according to naturalness, reconstruction quality, intelligibility of the lyrics, and the preservation of speaker identity. The participants gave the rating on a $5$ point scale, with $1$ being extremely poor quality while $5$ represents the natural singing voice. In total, $30$ participants took part in the subjective listening test with $20$ audio snippets. The summary of the subjective listening tests results is given in Table. 5. The high rating for the hidden reference makes the results credible. The subjective results confirm the improved quality of the proposed SymNet model over the baseline system. The SymNet model improves the baseline system by an absolute MOS value of $0.37$.