Scenario-Adaptive and Self-Supervised Model for Multi-Scenario Personalized Recommendation

In the user side, every training sample contains user profiles, scenario context features and two groups of user-scenario cross behavior features which are drawn from two scenarios separately. Specifically, user profiles contain age, gender, etc. Scenario context features mainly include scenario ID. User-scenario cross behavior features including user behavior sequences, category preferences and user statistic characteristics in the corresponding scenario.

For user behavior sequence features, the list of feature field embeddings will be concatenated to form the item embedding for each item in the behavior sequence. The final user behavior sequence embeddings will be generated by sequence pooling strategies. We utilize self-attention mechanism(Devlin et al., 2018) as our sequence pooling operation, other strategies(Shen et al., 2021; Lv et al., 2019) can be investigated for better performance, which we remain for future research.

In the item side, item profiles contain item ID, item category ID, account ID, etc. item-scenario cross features mainly include item statistic characteristics in the corresponding scenario.

To highlight the significance of scenario context features, we introduce a separate auxiliary network to model scenario characteristic, as formulated below:

where $x_a$ is the embedding of scenario context features, $f(\ast )$ is a multi-layer perceptron.

We propose a novel network to extract representation vectors in corresponding scenario both for users and items. The structures between user side and item side are nearly the same, as shown in Figure 3. Thus, we only describe the user side network in the following of the paper.

To make full use of whole-scenario information and regulate the fine-grained information transfer from whole scenarios to specific scenario, we introduce a global shared network to learn the information of all scenarios, and propose a Multi-Layer Scenario Adaptive Transfer Module(ML-SAT) as the scenario specific network for each individual scenario. As shown in Figure 4(a), the global shared network is a multi-layer perceptrons shared by all the scenarios. Training samples coming from the whole scenarios will be fed into global shared network to train a whole-scenario model. ML-SAT, which is parameter-specific in separate scenario, is trained with training samples coming from corresponding scenario.

Motivated by GRU(Cho et al., 2014), in each network layer, we design a scenario-adaptive gate unit with explicit gate mechanisms to regulate the fine-grained information transfer from whole scenarios to specific scenario. As shown in Figure 4(b), the scenario-adaptive gate unit can be formulated as follows:

where $g_l$ denote the $l$th layer output of global shared network. $s_{l-1}$ denote the $(l-1)$th layer output of scenario specific network. $W_z^l$ and $W_{bz}$ are the projection matrix weights and bias matrix weights of update gate $z_l$ in $l$th layer. $W_r^l$ and $W_{br}$ are the projection matrix weights and bias matrix weights of adaptive gate $r_l$ in $l$th layer. $a$ is the output of the scenario auxiliary network, which is introduced as a strong scenario indicator bias.

In Equation (3), adaptive gate $r_l$ decides the degree of useful information the global shared network can transfer. $h_l$ reflects the new adaptive state transfer information, considering the correlation between $g_l$ and $s_{l-1}$, as depicted in Equation (4). With the update gate $z_l$ as fusion weights, the new output $s_l$ can be updated by the fusion of $s_{l-1}$ and adaptive transfer output $h_l$. The network layer with scenario-adaptive gate unit can be stacked to multiple layers for more progressive layered extraction(Tang et al., 2020).

After the multi-layer network with scenario-adaptive gate unit, we can get the scenario main output of the corresponding scenario. Considering the significant importance of scenario context features in distinguishing scenario characteristics, we treat the output of scenario auxiliary network as scenario bias and fuse it with scenario main output to generate the final scenario-specific representation vector, as shown in Figure 4(c).

where $e_s$ is the final output, $s_T$ is the scenario main output, with $T$ as the final layer number. $a$ is the output of the scenario auxiliary network. $\sigma$ is the sigmoid function.

For each training sample with two groups of scenario-specific features in corresponding scenario, we can get two output vectors for the same user or item separately, denoted as $e_s^i$ and $e_s^j$. The objective of pre-training task is to extract agreements and model corrections between different scenarios. Thus, we adopt the same self-supervised contrastive loss of (Chen et al., 2020a) as our optimizing loss. Specifically, for a minibatch training samples with batch size $N$, we can get $2N$ vectors after ML-SAT. we treat $e_s^i$ and $e_s^j$ as a positive pair. The other $2(N-1)$ scenario vectors are regarded as negative vectors. The loss function for a positive pair $(e_s^i,e_s^j)$ is defined as

where $sim(e_s^i,e_s^j)=\frac{(e_s^i{)}^T{e}_s^j}{\parallel e_s^i\parallel \parallel e_s^j\parallel }$. $\tau$ is the temperature parameter. The final loss is the sum of all losses in the minibatch, as denoted in equation (10).

The training samples in fine-tune stage are labeled data drawn from target scenario. A user can only access one scenario at a time, so the features in one training sample only contain features in a single scenario. User side features contain user profiles, scenario context feature (scenario ID), user’s behavior sequences, statistic features and prefer features of user in target scenario. Item side features contain item profiles, scenario context feature (scenario ID), statistic features and prefer features of item in target scenario. Then, as shown in Figure 5, user side features and item side features are separately fed into embedding layer and upper ML-SAT modules to generate final scenario-specific vectors in corresponding scenario both for users and items. It should be emphasized that the embedding layer and networks weights of ML-SAT in fine-tune stage are restored from pre-training stage.

The final optimization loss in fine-tune stage is the combination of Scenario Specific Loss and Global Auxiliary Loss.

Scenario Specific Loss Function: Similar with other matching tasks(Huang et al., 2020; Nigam et al., 2019; Zhang et al., 2020), we adapt pairwise loss to optimize our fine-tune matching task. For a scenario $s$, the $k$th training sample in fine-tune stage is a triplet ($u_s^k$, $p_s^k$, $n_s^k$). $u_s^k$ denotes user representation vector, $p_s^k$ and $n_s^k$ denote corresponding positive and negative item vector, respectively. Negative items are randomly sampled from whole candidate set with negative sampling strategy(Mikolov et al., 2013). The scenario-specific loss function is defined as

where $\sigma$ is sigmoid function and $sim(\ast )$ is cosine function.

Global Auxiliary Loss Function: In fine-tune stage, the global shared network will also output a representation vector $g_T$ for each sample, as shown in Figure 5. The global shared network is trained with samples from all scenarios. So the output $g_T$ can be treated as the user or item representations in a global perspective. Modeling the similarity of users and items in the global scope is beneficial to the convergence of training and performance improvement. So we introduce a global auxiliary loss defined as:

where $u_g^k$, $p_g^k$, $n_g^k$ denote the representation vectors of a triplet (user, positive item, negative item) respectively, which are all generated though global shared network. Finally, the loss function in fine-tune stage is formulated as below with a hyper-parameter $\beta$:

We conduct experiments on our industrial dataset and two public datasets. Table 1 summarizes the basic information of these datasets.

• [topsep=2pt, leftmargin=8pt]

• Ali-MSSV. Our industrial Muiti-Scenario Short Video (MSSV) dataset in Taobao. Data from 2022-03-24 to 2022-04-04 is utilized for training and 2022-04-05 for testing. We evaluate models on two dense scenarios (denoted as #A1 and #A2) with abundant user behaviors and two spare scenarios (#A3 and #A4) with sparse data.

• Ali-CCP(Ma et al., 2018b). A public dataset released by Taobao with prepared training and testing set. We split the dataset into 3 scenarios according to scenario id, denoted as #B1 to #B3 for simplicity.

• Ali-Mama(Gai et al., 2017). A public dataset released by Alimama, an online advertising platform in China. Data from 2017-05-06 to 2017-05-011 is utilized for training and 2022-05-11 for testing. We arrange the dataset into 5 scenarios according to the city level, denoted as #C1 to #C5 for simplicity.

We release two types of SASS models to compare with other methods.

• [topsep=2pt, leftmargin=8pt]

• SASS-Base: It is the model trained with labeled data without pre-training. We will evaluate the performance of SASS-Base on all of the three datasets.

• SASS: It is our proposed model with two stages. The Ali-CCP dataset is unsuitable for pre-training task due to the lack of indispensable features, so we only evaluate the performance of SASS on our Ali-MSSV dataset and Ali-Mama dataset.

The compared Single-Scenario matching models (trained with single-scenario data) are listed as follows:

• [topsep=2pt, leftmargin=8pt]

• YoutubeDNN: YoutubeDNN(Davidson et al., 2010) adopts average pooling to extract user’s interest with a sampled softmax loss to optimize similarities between users and items.

• DSSM: DSSM(Huang et al., 2013) builds a relevance score model to extract user and item representations with double-tower architecture.

• BST: BST(Chen et al., 2019) leverages transformer to build the user behavior sequence. In this paper we use the inner product of user and item representations instead of MLP.

• MIND: MIND(Li et al., 2019) clusters users’ multiple interests by capsule network to improve the effect of multi-interest promotion.

The model ment above will also be trained with all-scenario data for Mix-Scenario versions.

The multi-scenario matching models are listed as follows. To the best of our knowledge, most of the existing multi-scenario models are mainly proposed for ranking problems, Therefore, for original multi-scenario ranking modes, some necessary but slight modifications are made for unified evaluation on matching tasks, denoted by a postfix -M for each model.

• [topsep=2pt, leftmargin=8pt]

• SAR-Net-M: Modified version of SAR-Net(Shen et al., 2021) which propose a multi-scenario architecture for scenario information migration with scenario-specific users’ behaviors and attention mechanism;

• STAR-M: Modified version of STAR(Sheng et al., 2021) which constructs star topology for multi scenarios and operate an element-wise operation to control information transferred from central network to specific networks;

• HMoE-M: Modified version of HMoE(Li et al., 2020) which constructs relationship between multiple scenarios in label implicit space through stacked model;

• ZEUS-M: Modified version of ZEUS(Gu et al., 2021) which learns unlabeled data through a self-supervised Next-Query Prediction task;

• ICAN: ICAN(Xie et al., 2020) is one of the SOTA models for multi-domain matching, which is most related works of our task. It highlights the interactions between feature fields in different domains for cold-start matching.

For all methods, the truncation length of user behavior is 50. AdamGrad is used as the optimizer with learning rate of 0.001 for all methods and the batch size is 512. Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) are adopted as the performance metrics. We set HR@20 and NDCG@20 in top-20 matching results as final metrics.

Scenario-Adaptive Gate Unit controls the information transfer from whole scenarios to specific scenario, which is essential for modeling scenario commonalities and distinctions. In this subsection, we investigate different transfer gate mechanisms and compare their performance on the SASS-Base model (SASS without pre-training stage).

The baseline model is SASS${}^{\ast }$, which is a SASS-Base version without transfer gate. SASS${}^{\ast }$ can be treated as a variant framework of MOE(Shazeer et al., 2017). In particular, we compare the following settings: (1) Sigmoid Gate: concatenating scenario specific layer’s output $s_l$ with global shared layer’s output $g_l$ and feeding the results into MLP with sigmoid to control information transfer. SASS${}^{\ast }$+Sigmoid Gate can be treated as a variant framework of MMoE(Ma et al., 2018a) or Cross-Stitch(Misra et al., 2016), which is also the fundamental transfer and fusion structure in SAR-Net(Shen et al., 2021) and HMoE(Li et al., 2020). (2) Production Gate: taking the element-wise production of scenario specific layer’s output $s_l$ and global shared layer’s output $g_l$ to generate the final scenario output, which can be partly treated as an equivalent of information mapping of STAR(Sheng et al., 2021). (3) Simnet Gate: an updated version for the fusion of information with the concatenations of element-wise production, element-wise subtraction and element-wise addition as the input. Then the input is feed into MLP with sigmoid. It can be summarized from Table 3 that (1) SASS${}^{\ast }$ performs worst and SASS${}^{\ast }$ with other gate mechanisms are beneficial to scenario-specific modeling both in #A1 and #A3. (2) The proposed SASS-Base model achieves the best performance. It shows that the fine-grained and decoupled gate mechanism in ML-SAT can get better control on information transfer from whole scenarios to individual scenario.

Motivated by PLE(Tang et al., 2020), the Scenario-Adaptive Transfer Layer can be stacked multiple layers for better performances. Thus, we set SASS-Base model with different layers for comparison. The results can be shown in Figure 6. The performances become better when the number of layers are stacked from 2 to 3, while getting worse with the further increase of layer number. We suspect that, with the layer number increasing, the distinctions of scenario-specific representation are decreasing if much more information is transferred from whole scenarios to specific scenario.

In this part, we investigate the performances of different self-supervised strategies in pre-training stage. We set SASS-Base (model without pre-training) as baseline model. We compare SASS-Base with SASS (with pre-training task) and another self-supervised Next Video Prediction task, which is introduced in ZEUS(Gu et al., 2021). The results shown in Table 4 exhibit the superior performance of SASS, especially in spare scenarios (such as #A3).

To certify the performance of generating distinguishing item representations in different scenarios, we implement a variant of SASS with only one single item representation vector shared in multiple scenarios. Results in Table 5 indicate that the performances gets better when generating different item representations in multiple scenarios. The reason is that when we consider the different representations of items in various scenarios, the characteristics of individual scenario can be more directly captured.

Scenario Bias Fusion is proposed to highlight the importance of scenario context information, while the global auxiliary loss task is introduced to strengthen the learning of global shared network. Both of the two strategies are expected to improve the whole performances of SASS. Experimental results in Table 6 demonstrate our hypothesises.