TFusion: Transformer based N-to-One Multimodal Fusion Block ^††thanks: ${}^{\ast }$ corresponding author

For multiple modalities, let $k\in K\subseteq \{1,2,\dots ,S\}$ index a specific modality, within the available modality set of $K$, where $S$ is the number of all possible modalities. Given an input $f_k\in R^{B\times C\times R_f}$, B and C denote the batch size and the number of channels, respectively. $R_f$ represents the shape of feature representation extracted from the $k$-th modality of a sample data, which can be 1D (L), 2D (H$\times$W), 3D (D$\times$H$\times$W) or higher-dimensional. In addition, $I=\{f_k|k\in K\}$ denotes the input set of feature representations from all the available modalities. Our goal is to learn a fusion function $F$ that can project $I$ into a shared feature representation $f_s$, denoted as $F\left(I\right)\to f_s$. To achieve the goal, we design an N-to-One fusion block, TFusion. The architecture is shown in Fig. 2, which consists of two modules: correlation extraction (CE) module and modal attention (MA) module.

Given the feature representation $f_k\in R^{B\times C\times R_f}$, we first flatten the $R_f$ dimensions of $f_k$ into one dimension and get a $B\times C\times R$ feature representation, where $R=L$ (1D), $R=H\times W$ (2D), $R=D\times H\times W$ (3D), etc. It can be viewed as $B\times R$ $C$-dimensional tokens $t_k$. Then, we obtain the concatenation of all the tokens $z_0\in R^{B\times T\times C}$, where $T=R\times |K|$, and $|K|$ denotes the number of available modalities.

Given $z_0$, the stack of eight transformer layers are introduced to learn the latent multimodal correlations. Each layer has a standard architecture that includes a multi-head attention (MHA) [18] block and a fully connected feed-forward network (FFN) [18]. Layer normalization (LN) [18] is applied before every block. The outputs of the $x$-th ($x\in [1,2,\dots ,8]$) layer can be describe as:

Therefore, we get $z_l\in R^{B\times T\times C}$, which is the last transformer layer output. By reverting $z_l$ to the size of $|K|\times B\times C\times R_f$, we obtain the output $I^{\prime }=\{f_k^{\prime }|k\in K\}$ of CE as:

where $r(\cdot )$ and $split(\cdot )$ are the reshape and split operations, and $I^{\prime }$ is the set of transformed feature representations $f_k^{\prime }\in R^{B\times C\times R_f}$ which contains multimodal correlations and has the same size as the original input $f_k$.

Given the transformed feature representations set $I^{\prime }$, the weight map $m_k$ is generated with the modal attention mechanism. Feature representations extracted from different modalities are expected to have different weights for fusion at the voxel level. Therefore, we introduce a modal-wise and voxel-level softmax function to generate the weight maps from $I^{\prime }$, as shown in Fig. 3.

We denote the $i$-th voxel of $f_k^{\prime }$ and $m_k$ as $v_k^i$ and $m_k^i$, respectively. $e$ is the natural logarithm. The value of weight map $m_k$ can be defined as:

By element-wise multiplying input feature map $f_k$ with the corresponding weight map $m_k$ and summing all the modalities, we can obtain a fused feature map $f_s$ as:

Since the sum of $m_1^i,\dots m_{|K|}^i$ is 1, the value range of fused feature representation $f_s$ remains stable to improve the robustness for variable input modalities. Moreover, the relative sizes of $v_1^i,\dots v_{|K|}^i$ (contain the latent multi-modal correlations learned from the CE module) are retained in the corresponding weights. In particular, when only one modality is available, all the values of the weight map are 1, which means $f_s=f_k$ ($k\in K$, $|K|=1$). In this case, the input feature representation remains unchanged. It enables the backbone network (the upstream processing model and the downstream decision model) to enhance its capability to encode and decode information from different modalities rather than relying on a particular one. It is crucial for variable multimodal data analysis.

It is worth noting that TFusion is a flexible data-dependent fusion strategy by which complementary information from variable input feature representations can be fused automatically. In addition, TFusion does not require the introduction of any redundant information (e.g., padding zero), which is different from existing convolutional fusion strategies. Experiments described in Sect.III are carried out on human activity recognition and brain tumor segmentation to evaluate the TFusion.

SHL2019. The SHL (Sussex-Huawei Locomotion) Challenge 2019 [20] dataset provides data from seven sensors of a smartphone to recognize eight modes of locomotion and transportation (activities), including still, walking, run, bike, car, bus, train, and subway. The sensor data are collected from smartphones of a person with four locations, including the bag, trousers front pocket, breast pocket and hand. Each location is called “Bag”, “Hips”, “Torso”, and “Hand”, respectively. The seven sensors are an accelerometer, gravity, gyroscope, linear accelerometer, magnetometer, orientation and pressure sensor. The orientation consists of four channels, expressed as a spatial rotation in a quaternion (i.e., w, x, y, and z-axes). The pressure has a single channel. Each of the remaining sensor data consists of three channels corresponding to the three-dimensional axes (i.e., x, y, and z-axes).

The SHL2019 dataset is split into three subsets: train, validation, and test, which contain 588,215, 48,708, and 55,811 available samples, respectively. Each sample consists of 500 sensor values that are acquired for five seconds with a sampling rate of 100 Hz. Data acquired from the locations except the “Hand” are given in the train subset, while the validation subset provides the data of all four locations. In the test subset, only unlabeled “Hand” location data are available.

BraTS2020. The BraTS2020 [21] dataset provide four modality scans: T1ce, T1, T2, FLAIR for brain tumor segmentation. It contains 369 subjects. There are four labels in the segmentation mask namely NCR (label 1: the necrotic tumor core), ED (label 2: the peritumoral edematous/invaded tissue), NET (label 3: the non-enhancing tumor core), and ET (label 4: the enhancing tumor). To better represent the clinical application tasks, different structures have been grouped into three mutually inclusive tumor regions: ET: the enhancing tumor, TC (Union of labels 1, 3 and 4): the tumor core region, and WT (Union of all labels): the whole tumor region [21]. All the modalities from the same subject are co-registered, resampled to isotropic $1m{m}^3$ and skull-stripped, with the size of $155\times 240\times 240$. We select 70% data as training data, while 10% and 20% as validation and test data respectively. To prevent overfitting, two data augmentation techniques (randomly flip the axes and rotate with a random angle in $[-{10}^{\circ },{10}^{\circ }]$) are applied during training. We apply z-score normalization [22] to the volumes individually and randomly crop $128\times 128\times 128$ patches as inputs to the networks.

EmbraceNet. In the experiments on activity recognition, we compare TFusion with EmbraceNet [23], which employs a selection strategy (shown in Fig. 1(b)) by generating feature masks ($r_1,r_2,\dots ,r_7$) with the rule of giving equal chances to all available modalities during each value selection. For a fair comparison, as shown in Fig. 4, we adopt the same processing (P) and decision (D) model as used in [23]. We obtain the performance of our fusion strategy by replacing EmbraceNet with TFusion.

Following [23] setting, five decisions are obtained per sample, and each one is the result for a one-second time duration. The batch size is set to 8. A cross-entropy loss and the Adam optimization method [24] with ${\beta }_1=0.9$, ${\beta }_2=0.999$ are employed. The learning rate is initially set to $1\times {10}^{-4}$ and reduced by a factor of 2 at every $1\times {10}^5$ steps. A total of $5\times {10}^5$ training steps are executed.

GFF. In the experiments on brain tumor segmentation, we compare TFusion with a gated feature fusion block (GFF) [17], which belongs to the convolution strategy (shown in Fig. 1(c)). As shown in Fig. 5, a feature disentanglement architecture is employed. Multimodal medical images are decomposed into the modality-invariant content and the modality-specific appearance code by encoders $E^c$ and $E^a$, respectively. The content codes (e.g., $c_2$ and $c_3$, shown in Fig. 5) of missing modalities are simulated with zero values. Then, all content codes are fused into a shared representation $c_s$ by GFF. Given $c_s$ and an appearance code $a_i$, the corresponding image is reconstructed by the decoder $D_i^r$. Given $c_s$ , the tumor segmentation results are generated by the decoder $D^s$. For a fair comparison, we adopt the same encoders ($E_i^c$ and $E_i^a$) and decoders ($D^s$ and $D_i^r$) as used in [17]. We obtain the performance of our fusion strategy by replacing GFF with TFfusion and removing the zero-padding operation.

The training max_epoch is set to 200. Following [17] setting, the batch size is set to 1. Adam [24] is utilized with a learning rate of $1\times {10}^{-4}$ and progressively multiplies it by (1 - epoch / max_epoch${)}^{0.9}$. Losses of $L_{KL}$, $L_{rec}$ and $L_{seg}$ are employed as [17]. During training, to simulate real missing modalities scenarios, each training patient’s data is fixed to one of 15 possible missing cases. For a comprehensive evaluation, we test the performance of all 15 cases for each test patient.

Our implementations are on an NVIDIA RTX 3090 (24G) with PyTorch 1.8.1.

Activity recognition. We compare TFusion with the EmbraceNet [23] on SHL2019. In addition, as shown in TABLE II, we also compare the results of other fusion methods, which use the same processing (P) model and decision (D) model as shown in Fig 4. (1) In the early fusion method, the data of seven sensors are concatenated along their $C$ dimension. The prediction results are obtained by inputting the concatenation into a network of P and D in series. (2) For the intermediate fusion approach, the EmbraceNet is replaced with the concatenation of feature representations along their $R_f$ dimension. (3) In the late fusion method, an independent network of P and D in series is trained for each sensor, and then the decision is made from the averaged softmax outputs. (4) In the confidence fusion model, the EmbraceNet is replaced with the confidence calculation and fusion layers in [25]. The results of different fusion methods on the validation data are presented in TABLE II. Our proposed TFusion outperforms the EmbraceNet in all four smartphone locations and improves the overall accuracy from 65.22% to 67.47%.

Brain tumor segmentation. The quantitative segmentation results are shown in Table I. Compared with GFF, the network integrated with TFusion achieves better average performance over the 15 possible combinations in all three tumor segmentation tasks. In particular, TFusion outperforms GFF for all the possible combinations in the TC segmentation task. Overall, TFusion achieves better Dice scores in most situations (13,15,13 situations for WT, TC and ET segmentation, respectively). In addition, we conduct the statistical significance analysis. The number of situations with significant improvement are 6, 10 and 8 for WT, TC and ET, respectively. It is provided by a Wilcoxon test ($p$-values $<0.05$). Besides, we find no significant drop in performance caused by TFusion. The visualization of segmentation results is shown in Fig. 6.

Ablation experiments.The correlation extraction (CE) module and the modal attention (MA) module are two key components in TFusion. To verify the contribution of these modules, we evaluate the TFusion without CE and MA, respectively. TFusion without CE denotes that feature representations are directly fed into the MA module (Fig. 2). TFusion without MA means that we directly add the transformed feature representations ($I^{\prime }$) up to get the fusion result. For activity recognition, as shown in TABLE II, we can find that TFuison without CE performs worse than other methods. Compared with EmbraceNet, the improvement of TFusion without MA is inconspicuous. For brain tumor segmentation, as shown in TABLE. III, we present the averaged performance over the 15 possible combinations of the input modalities on BraTS2020. It shows that both the CE and MA module lead to performance improvement across all the tumor regions. Therefore, ablation experiments on two different tasks show that both CE and MA play an important role in TFusion.