Hybrid Fusion Based Interpretable Multimodal Emotion Recognition with Insufficient Labelled Data

The traditional feature-based SER systems extract audio features such as cepstrum coefficient, voice tone, prosody, and pitch and use them for SER (el2011survey). For instance, Rong et al. (rong2009acoustic) worked on extracting the most important audio features from the speech samples, whereas Lee et al. (lee2005toward) used the extracted audio features to identify negative and positive emotions in speech samples. The feature-based SER systems depend on the polarity of the emotional features. The features of high-key classes (happiness and anger) are similar in properties among themselves, and they are very different from the low-key classes (sad and despair) (trigeorgis2016adieu). In the context of machine learning-based SER, Support Vector Machine (SVM) based classifiers and Hidden Markov model (HMM) based statistical techniques have also been explored (jain2018cubic; lorenzo2015emotion). However, they require manual crafting of acoustic features, and HMM-based models cannot always reliably estimate the parameters of global speech features (el2011survey). Hence, it is challenging to develop an end-to-end SER system using them. The deep learning-based approaches using spectrogram features and attention mechanisms have shown state-of-the-art SER results (dai2019learning). In this context, Xu et al. (xu2020improve) generated multiple attention maps, fused them, and used them for SER. They observed an increased performance as compared to non-fusion-based approaches. In another work, Mao et al. (mao2014learning) used a Convolutional Neural Network (CNN) for spectrogram processing. On the other hand, Majumder et al. (majumder2019dialoguernn) performed Recurrent Neural Network (RNN) based SER. They determined speech embeddings and used them for speaker identification. In another work using attention maps, Seyedmahdad et al. (mirsamadi2017automatic) implemented local attention to learn the emotion features automatically.

Deep learning-based approaches have shown state-of-the-art TER results. With the evolution of deep learning, it has become possible to convert text into vectors and use Deep Neural Networks (DNNs) to process them. Emotion recognition in conversation could be useful to mine opinions from conversational data on platforms such as YouTube, Facebook, Reddit, Twitter, and others (poria2019emotion). More examples of Deep Learning-based emotion analysis include personality detection from text using document modeling (majumder2017deep) and text-based emotion recognition using YouTube comments (hajar2016using). In another work, Huang et al. (huang2019emotionx) used a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model for emotion recognition in text dialogs. Shrivastava et al. (shrivastava2019effective) applied sequence-based CNNs, whereas Bambaataa et al. (batbaatar2019semantic) utilized semantic and emotional information to train a deep TER model. Deep models focus on learning parameters such as features, sequence-related information, and contextual information from the text input.

One of the most informative ways for machine perception of emotions is through facial expressions in images and videos. Identifying human emotions from facial expressions is a relatively more saturated research area than emotion recognition in other modalities. Various techniques such as face localization, face registration, micro-expression analysis, tracking the landmark points, shape feature analysis, eye gaze prediction, face segmentation, and detection have been developed for facial emotion recognition (li2018deep; corneanu2016survey). Image Emotion Recognition (IER) research is also an active domain. For instance, Kim et al. (kim2018building1) built a deep feed-forward neural network to combine different levels of emotion features obtained by using the semantic information of the image. In another work, Rao et al. (rao2019learning) prepared hierarchical notations for emotion recognition in the visual domain. Traditional, feature-based IER analysis with low-level (shape, color, and edge) and mid-level (composition and optical balance) image features  (hanjalic2006extracting; joshi2011aesthetics; machajdik2010affective) used the semantic content of the images for emotion analysis. However, all the low and mid-level features are difficult to accommodate by handcrafted feature extraction techniques used for machine traditional and machine learning-based methods. On the other hand, deep learning-based IER approaches are capable of extracting high-level visual features, but they struggle to extract low and mid-level features. Moreover, they require well-labeled large-scale datasets for training (rao2019learning).

In the context of recognizing emotions from speech utterances and corresponding text transcripts, Chuang et al. (chuang2004multi) worked on analyzing the overlapping emotional information in speech and text. A simple fusion of spoken and textual information has also been used for emotion recognition in some more works. For instance, Makuuchi et al. (makiuchi2021multimodal) performed separate acoustic and textual analyses and determined the emotional context based on their collective result. In another work, Yoon et al. (yoon2018multimodal) extracted the audio and text information using dual RNNs and then combined it to perform emotion recognition. On the other hand, some research attempts used textual information to improve the SER performance. For example, Tripathi et al. (tripathi2018multi) performed emotion recognition on the Interactive EMOtional dyadic motion CAPture (IEMOCAP) dataset (busso2008iemocap) using data from speech and text modalities and complementing it with hand movements and facial impressions data. In another work, Siriwardhana et al. (siriwardhana2020jointly) fine-tuned transformers-based models to improve the performance of multimodal speech emotion recognition.

Several attempts have been made to recognize the emotional content portrayed in visual and textual modalities. In this direction, Kahou et al. (kahou2016emonets) developed a framework, EmoNets for emotion recognition in video and text. Fortin et al. (page2019multimodal) implemented a multi-task architecture-based emotion recognition approach to perform predictions with one or two missing modalities by using a classifier for each combination of image, text, and tags. In another work, Xu et al. (xu2018co) modeled the interplay of visual and textual content for sentiment recognition using a co-memory based-network.

Multimodal emotion analysis from audio-visual data has also started getting researchers’ attention lately (hossain2019emotion). For instance, Aytar et al. (aytar2016soundnet) proposed SoundNet that extracts the emotional information by self-supervised learning of sound representations. In another work, Guanghui et al. (guanghui2021multi) implemented the feature correlation analysis algorithm for multimodal emotion recognition. They extracted speech and visual features using two-dimensional and three-dimensional CNNs, fused the features, and used SVM for emotion classification on the fused features.

There have been several attempts regarding emotion recognition in more than two modalities simultaneously. For example, Poria (poria2016fusing) used information fusion techniques to combine the context from audio, vision, and textual modalities for the sentiment analysis. They found the fusion of the textual and spoken description of the emotional information as an aid to the emotion analysis for visual modality. In another work, Tzirakis et al. (tzirakis2017end) extracted the features from speech, image, and text modalities, analyzed the correlation among them, and trained an end-to-end emotion recognition system using them combinedly.

In these methods, the attribution values denoting the relevance of inputs concerning outputs are determined. A popular attribution value is ‘Shapley Values’ (SHAPley1953value) . The attribution techniques are frequently used for local interpretability, which explains the impact of one instance instead of the overall model. The Shapley values have been used by Lundberg et al. (lundberg2017unified) who implemented an interpretability framework, SHAP (Shapley Additive exPlanations), which determines each feature’s contribution by analyzing its Shapley values (malik2021towards). The computation of Shaply values is very expensive because $2^n$ models are required to be trained for a model with $n$ features (lundberg2017unified; castro2009polynomial). Different approximations have been used to speed us Shapley values’ computation. For instance, Shapley values sampling (castro2009polynomial) and KernelSHAP (lundberg2017unified). The attribution techniques are further classified into perturbation and back-propagation-based approaches, which are explained further.

These techniques make a small change in the input and observe its impact. The insights thus obtained are used to interpret the model’s working (fong2019understanding). The most frequently used perturbation technique is Local Interpretable Model-agnostic Explanations (LIME) (ribeiro2016should) that perturbs the given instance and synthesizes new data. The new data is weighted as per the closeness of the new instance to the original instance. The output is computed by training the original model on the perturbed data. The trained model’s weights denote the approximate values of each feature’s contribution. The LIME can be used with any machine learning model, though it is computationally expensive as it involves the generation of new data.

The backpropagation-based interpretability techniques calculate the attributions by backpropagating through the network multiple times. A popular backpropagation-based technique is ‘Saliency Map’ (simonyan2013deep) that has the label output’s absolute gradient of each input feature as an attribution. Another popular technique is Gradient-weighted Class Activation Map (Grad-CAM), which assigns a score to each feature and computes the activation map using this score (selvaraju2017grad). The Grad-CAM goes until the last convolutional layer instead of backpropagating back to the image. It generates a map that highlights the important features of the input image. If the input image is slightly changed, it generates an entirely different map (kindermans2019reliability).

As suggested by the above survey, LIME, SHAP, and Grad-CAM are the most frequently used interpretability techniques for machine learning models. The LIME is involved with a very high computational cost, whereas Grad-CAM is incapable of withstanding small changes in the input image. The SHAP technique does not suffer from the aforementioned limitations. Furthermore, the DNN interpretability has been applied to the visual modality but it has not been fully explored for the speech and text modalities and multimodal analysis. It inspired us to develop an interpretability technique for multimodal emotion recognition to explain the importance of each modality and identify the important features of each modality that lead to the prediction of a particular class.

The original B-T4SA dataset contained 4.7M data samples labeled as ‘positive,’ ‘negative,’ and ‘neutral.’ While constructing the IIT-R MMEmoRec dataset with discrete emotion labels, i.e., ‘angry,’ ‘happy,’ ‘hate,’ and ‘sad,’ it was essential to retain only the samples having high confidence in the associated emotion label. After passing the image, speech, and text components of the inputs to respective emotion recognition models as discussed in Section 3.2, we computed a value for each data sample in each class representing at what percentage compared to the class maximum that sample is in its particular class. It gave us the confidence of each data sample in its particular class. To determine the appropriate threshold, we plotted possible threshold values vs. the ratio of the class present (the number of each class sample and the total number of samples) as shown in Fig. 2.

The higher the threshold, the higher the confidence and the better the quality of data. However, a higher threshold value also leads to two issues – i) reduction in the size of the dataset and ii) disruption in the distribution of emotion classes compared to its original distribution. As seen in Fig. 2 the distribution of each class at a threshold approaching $1$ is very different as compared to when all samples are taken at the $0$ thresholds.

An appropriate threshold value needs to be chosen, leading to a good trade-off between high confidence and appropriate size & distribution of the dataset. Till the threshold value of $0.33$, the distribution is almost the same as the original, but this confidence is too low to be acceptable. The next possible value is above $0.5$ but below $0.6$. Between these two values, the distribution of various classes is almost the same, and the confidence is also above $0.5$, which is acceptable. Hence an average value of $0.55$ is chosen as the threshold confidence value.

The MMEmoRec dataset has been evaluated by having 8 people evaluate the data samples. We had two human readers (one male and one female) who spoke out and recorded the text components of the data samples. The evaluators listened to the machine synthesized speech against the human speech recorded by the human readers and scored the contextual similarity between them on a scale of 0 to 100. The human evaluators also evaluated whether the data samples’ speech, image, and text components agree with the annotated emotion sample individually and combinedly. The samples have been picked randomly, and the average of the evaluators’ scores has been reported in Table 3 where $S_{ss-hs}$ denotes the percentage of evaluators reporting the synthetic speech (ss) to be similar to human speech (hs). $S_{ss}$ & $S_{hs}$ denotes the percentage of speech components of synthetic and human speech portraying the annotated emotion. Likewise, $S_i$ and $S_t$ denote the agreement of annotated emotion class by image and text components. $S_{ss-i-t}$ and $S_{hs-i-t}$ show the samples showing agreement of the annotated emotion class by all three modalities on considering synthetic and human speech, respectively.

We had two readers read the text of the data samples and called their output human synthesized speech. 60.72% evaluators found the synthetic speech to be contextually similar to the human synthesized speech. 74.49% synthetic speech samples and 78.91% human synthesized speech samples were found to be portraying the annotated emotion labels. As per the further observations, 69.26% images and 78.81% text components of the data samples correspond to the annotated emotion labels. Moreover, the evaluators also reported that 72.99% of the samples considering machine synthesized speech along with the corresponding text & image were in line with the determined emotion label, whereas this is comparable to the value of 76.74% on considering human synthesized speech along with the corresponding text & image.

The images of dimension $(128,128,3)$ are fed into $P_i$ and $S_i$ respectively, with $P_i$ consisting of VGG16 (simonyan2014very) and a $512$-dimensional dense layer and $S_i$ containing 3 convolution layers of $64$, $128$ and $256$ filters of size $(3,3)$ and a dense layer of $512$ dimensions. The spectrogram of size $(128,128,1)$ from speech input is passed from a $3$ filter convolution layer of size $(1,1)$, to make it compatible with VGG16. Further, it is passed from $P_s$ and $S_s$, consisting of the same architecture as $P_i$ and $S_i$ respectively.

The text input is similarly passed from $T_i$ containing a BERT (devlin2018bert) and $T_s$ consisting of an embedding and LSTM layer with $64$ units. Both $T_i$ & $T_s$ are followed by $512$-dimensional dense layers. In the intermediate fusion, all pairs of the pre-trained and simpler networks from different modalities are created by passing them from the $WeightedAdd$ layer that we have defined. It gives us six such combinations passed from $2$ dense layers with $1024$ neurons, giving the classification based on each pair. The Eq. 1 shows all the possible pairs formed from the combination of pre-trained and simpler networks such that both the networks do not belong to the same modality.

In this phase, the information from various modalities’ all possible pairs is combined in a hybrid manner. The intermediate classification outputs obtained from above Eq. 1 are passed from another $WeightedAdd$ layer, which combines these outputs dynamically, giving us the final output $O$ as depicted in Eq. 2. The output $O$ is passed from a dense layer with dimensions equal to the number of emotion classes, i.e., four.

For a model with $k$ features $\{f_1,f_2,\dots f_k\}$, K exPlanable (KP) value of feature $f_i$, $k{p}_{f_i}$ denotes its importance. Fig. 5 depicts an example calculation.

Consider four nodes, Node $1$ with no feature i.e NULL, Node $2$ with a single feature $f_i$, Node $3$ with all the remaining features left in Node $1$, i.e. , $\{f_1,f_2,\dots ,f_{(i-1)},f_{(i+1)},\dots ,f_k\}$, and Node $4$ with all the features $\{f_1,f_2,\dots ,f_{(i-1)},f_i,f_{(i+1)},\dots ,f_k\}$. The ‘Marginal Contribution’ of an edge connecting Node $i$ and Node $j$ is defined as the difference between the prediction probabilities on using their features. For a given predicted label $c$, the marginal contribution of the feature $f_i$ for the edge from Node $1$ to Node $2$ is calculated using Eq. 3. Here, $pro{b}_{\left\{f_i\right\}}$ is the probability of label $c$ calculated by having only feature $f_1$ in the input and perturbing all other features to zero.

To calculate the overall importance of $f_i$, we need to calculate the weighted average of all ‘marginal contribution’ of $f_i$ given by Eq. 4.

This Section computes the KAAP values and uses them to determine the importance of each modality and its features. The information of image, text, and speech modalities are in the same data format, i.e., continuous format. A single pixel can not define an object that can lead to a particular emotion for an image, but a group of pixels will. For speech, the spectrogram at a single instance of time & frequency alone can not define anything, but a time interval will. Likewise, for text, a single letter may not define an emotion, but a word is capable of doing so. KAAP values have been defined based on the motivation from the aforementioned fact. They are computed using the KP values for a group of features.

First, the input of size $l$ is divided into $k$ parts, where $k$ is a hyperparameter decided through ablation study in Section 4.3. These $k$ parts correspond to the $k$ features of the input. Then, for a feature group $f_i$, $K{P}_{f_i}\left(k\right)$ values are computed for the given value of $k$ using Eq 6. It represents how a group of features $f_i$ will perform compared to all remaining groups. However, these groups can vary in size, i.e., $k$ can have various values that lead to different groups and thus to different KP values from groups of different sizes, thus affecting the original features’ importance. To deal with this issue, the weighted average of all the KP values is taken for $k\in \{2,3\dots ,k\}$ where weights are equal to the number of features in that group of features, given by the Eq. 8. It should be noted that $k$ = 1 is ignored here, as the whole input as one feature will not make any sense.

For input image and speech spectrogram, both of width $128$ and height $128$, their KP values for a given $k$ are calculated by dividing the input into $k$ parts along both the axes. As a matrix defines both image and speech spectrogram, this gives us with $k\ast k$ feature group, the equation for calculating the KAAP values for the above two inputs is given by Eq. 9. It gives us a matrix showing the importance of each pixel for a given image and speech input. This matrix directly represents the importance of the image. At the same time, for speech input, the values are averaged along the frequency axis to reduce the KAAP value matrix to the time axis, hence giving importance to speech at a given time.

For input text, the division is done such that each text word is considered a feature, as the emotion can only be defined by a word, not a single letter, as discussed above. Then the text is divided into $k$ parts, and as a linear array can represent text, the KAAP values are calculated using Eq. 8. Also, the value of $k$ used for image, speech, and text modalities have been determined as $7$, $7$, and $5$ respectively in Section 4.3.2. Furthermore, the modalities’ importance defined by symbols $\upsilon$, $\delta$, and $\tau$ for visual, spoken, and textual features, respectively, are computed assuming that image, speech, and text are three distinct features and calculating each modality’s KAAP value for $k$ = $3$. While finding the importance of the features of a particular modality, all the other modalities are perturbed to zero. The KAAP technique is depicted in Algorithms 3 which uses Algorithm 2 that calculates the KAAP values for each data instance and 1 for probability prediction.

To begin with, the emotion recognition has been performed for a single modality at a time, i.e., separate IER, SER, and TER using pre-trained VGG models (simonyan2014very) for Image & speech and BERT (devlin2018bert) for text. The performance has been evaluated in terms of Accuracy, CohenKappa metric (CK), F1 score, Precision, and Recall and summarized in Table 4. The CK metric measures if the distribution of the predicted class is in line with the ground truth or not.

Next, we moved on to the combination of two modalities. The chosen two modalities are fed into respective pre-trained models and then passed from a dense layer of $512$ neurons. Then the information from these modalities is added using the $WeightedAdd$ layer defined in 3.2.1, this output is next passed from three dense layers of size $1024$, $1024$ and $4$ neurons, which then classifies the emotion. Image + text comes out to be the best combination, beating the remaining two combinations in both Accuracy and CK values.

At last, the information from all three modalities is combined and fed into their respective pre-trained models and is then passed from a dense layer of size $512$, which is then passed from a $WeightedAdd$ layer; the output of this layer is passed from $3$ dense layers as in the combination of two modalities. Combining all three modalities has performed better than the remaining models in all the evaluation metrics. As observed during the experiments above, combining the information from the complementary modalities has led to better emotion recognition performance. Hence, the baselines and proposed model have been formulated, including all three modalities and various information fusion mechanisms in Section 4.4.

An in-depth ablation study has been conducted here to decide the value of $k$ used in Section 3.3.2. The dice coefficient (deng2018learning) is used to determine the best $k$ values. It measures the similarity of two data samples; the value of $1$ denotes that the two compared data samples are completely similar, whereas a value of $0$ denotes their complete dis-similarity. For each modality, KAAP values are calculated at $k$ $\in$ $\{2,3\dots ,10\}$. The dice coefficient is calculated for two adjacent $k$ values. For example, at $k=3$, the KAAP values at $k=2$ and $k=3$ are used to calculate the dice coefficient. The procedure mentioned above has been performed for all three modalities, and the results are visualized in Fig. 6.