Federated Online Clustering of Bandits

Stochastic multi-armed bandit (MAB) [auer2002finite] is a well-known sequential decision-making problem, where a learner sequentially selects actions so as to maximize the cumulative rewards (or minimize the cumulative regret). One fruitful application area of MAB is the online recommendation systems (RecSys) [chu2011contextual, abbasi2011improved, gentile2014online, li2019improved, zhang2020conversational, li2021unifying], where MAB algorithms provide a principled way to handle the challenge of exploration-exploitation trade-off [lattimore2020bandit].

To advance the bandit algorithm for large-scale applications, contextual linear bandits add the simple yet effective linear structure assumptions on actions and reward functions [chu2011contextual, li2010contextual, abbasi2011improved]. One limitation, however, is that such a model mainly works in a content-dependent manner, ignoring the often used tool of collaborative filtering. To address this issue, the clustering of bandits (CLUB) are proposed [gentile2014online, li2016collaborative, li2018online, li2019improved]. The CLUB algorithms adaptively cluster similar users and utilize the collaborative information given by the cluster structure, which dramatically improves the recommendation quality.

While most existing bandit algorithms are designed under a centralized setting, in response to the increasing application scale and public concerns about privacy, there is a growing demand to keep user data decentralized and push the learning of bandit models to the client or the local server side . This paradigm is now known as federated learning [kairouz2021advances]. Owing to its overall applicability, there has been a surge of interest in studying federated MAB [dubey2020differentially, zhu2021federated, shi2021federated], which promises cooperative bandit learning with larger amounts of data (across multiple local servers) while keeping the data decentralized. This motivates us to study the CLUB problem to its federated counterpart, i.e., the federated clustering of bandits (FCLUB).

In FCLUB, each local server can conduct its own local clustering of bandit algorithms. To enable the collaborative effects of users across different servers, the local server could also collaborate with other local servers under the coordination of a global server, whose communication needs to satisfy specific privacy and communication requirements. The goal of this work is to design an federated online clustering of bandit framework, so as to minimize the $T$-round regret under the privacy protection requirements and communication cost considerations.

The key challenge of FCLUB is designing collaborative bandit learning procedures and cluster detection strategies to identify the overall cluster structures across different local servers, where each local server only holds part of the users with unknown interests. Such a problem is more challenging due to the following privacy and communication cost requirements, which are two first-order requirements for any federated applications [kairouz2021advances].

Privacy protection: To reduce the privacy leakage of each user, we expect local servers to only share user clusters’ data instead of individuals’ raw data. In addition, we still need a mechanism to protect the uploaded (cluster) information against possible adversaries outside the local server, for which we adopt the solution concept of differential privacy (DP). However, the off-the-shelf DP notion is defined on individual users, hence unsuitable for FCLUB. It is challenging and unclear what is a suitable notion of privacy over the clustering of users and how to devise algorithms to guarantee the corresponding privacy requirements.

Communication: Communication is critical for collaborative learning, but may also be expensive or time-consuming. For FCLUB, it is desired to minimize the total regret while keeping the communication costs (in terms of communication rounds between the global server and local servers) as low as possible. Another requirement is to design an asynchronous communication protocol incorporating the randomly arriving users and possibly lagging servers, preventing commonly used synchronous protocols [dubey2020differentially].

In this section, we formulate the setting of “Federated Clustering of Bandits” (FCLUB). We use $\left[n\right]$ to represent set $\{1,...,n\}$. We use boldface lowercase letters and boldface capitalized letters for column vectors and matrices, respectively. For the norms, $\parallel x\parallel$ denotes the ${\ell }_2$ norm of vector $x$. For any symmetric positive semi-definite (PSD) matrix $M$ (i.e., $x^{\top }Mx\ge 0,\forall x$), ${\parallel x\parallel }_M=\sqrt{x^{\top }Mx}$ denotes the matrix norm of $x$ regarding matrix $M$.

At the global level, there are $n$ users, denoted by the set $U=\{u_1,...,u_n\}$. Each user $i\in U$ has an unknown preference vector ${\theta }_i\in R^d$ and for simplicity we assume $\parallel {\theta }_i\parallel \le 1$. Since users may have the same/similar preference vector, we assume there exists $m$ (unknown) different preference vectors, i.e., $|\{{\theta }_1,...,{\theta }_n\}|=m$. Users with the same preference vector form an underlying cluster and we denote these $m$ (unrevealed) clusters by $C=\{C_1,...,C_m\}$. Different from CLUB, users in FCLUB are distributed in $L$ local servers denoted by $\{1,...,L\}$. At the local level, the local server $\ell$ contains $n^{\ell }$ users $U^{\ell }=\{u_1^{\ell },...,u_{n^{\ell }}^{\ell }\}$ (with ${\bigcup }_{\ell \in \left[L\right]}{U}^{\ell }=U$) and similarly, these $n^{\ell }$ users form local cluster $C^{\ell }=\{C_1^{\ell },...,C_{m_{\ell }}^{\ell }\}$ where $m_{\ell }\le m$.

The learning agent interacts with the bandit game as the follows. At each time $t$, a user $i_t\in \left[n\right]$ randomly arrives with probability $1/n$. Then $K$ items are generated to form a item set $D_t$, where the feature of each item $x\in D_t$ is drawn independently from a fixed but unknown distribution $\rho$ over $\{x\in R^d:\parallel x\parallel \le 1\}$. The learning agent identifies the local server ${\ell }_t$ that $i_t$ belongs to and the current user cluster $j_t$ (detected by our algorithm) which $i_t$ lies in. The local server then recommends an item $x_t\in D_t$ to the user based on the aggregated information from cluster $j_t$. After $i_t$ receives the recommendation, the learning agent receives a random reward $y_t\in [0,1]$. Let $H_t=\{i_1,x_1,y_1,...,i_{t-1},x_{t-1},y_{t-1},i_t\}$ be the historical information before time $t$. We assume the expectation of reward $y_t$ is linear in the feature vector $x\in D_t$ and the unknown preference vector ${\theta }_{i_t}$, i.e., $E_t\left[y_t|x\right]={\theta }_{i_t}^{\top }x$, and $\{y_t-{\theta }_{i_t}^{\top }x{\}}_{t=1,2,...}$ have sub-Gaussian tails ${\sigma }_0^2$.

Now we give some assumptions on preference vectors and item feature vectors. Note that all the assumptions follow the previous works gentile2014online, gentile2017context, li2018online, li2019improved.

Learning Efficiency. The goal of the learning agent is to accumulate as much reward as possible. Let the optimal item for user $i_t$ at time $t$ be $x_{i_t}^{\ast }={argmax}_{x\in D_t}{\theta }_{i_t}^{\top }x$. The learning performance is measured by the regret, defined as

where $r_t$ is the regret at time $t$ and the expectation is taken over the randomness of the algorithm and the environment regarding the users $i_1,...,i_T$ and the item sets $D_1,...,D_T$.

In addition to the regret, privacy protection and the communication cost are two important criterion in federated learning. In this work, we aim to ensure that the user data are protected under privacy constraints and the communication complexity is low.

Privacy Requirements. To protect the user data, we introduce two privacy requirements. First, we desire the local server only uploads the user clusters’ sufficient statistics (or clustered data) for the learning procedure, instead of individual users’ raw data. Second, to protect the clustered data against third-party adversaries outside the local server, we adopt the notion of DP, which encodes the intuition that any observable output changes very little (in probability) when any input datum changes. Since existing DP notions are defined on user-level data shariff2018differentially, dubey2020differentially, we introduce a new differential privacy requirement to protect the cluster-level data.

The contextual MAB problem involves two sets of variables against the adversaries outside the local server: the decision sets $D_t$ and the observed rewards $y_t$. Since the users only receive and store observations regarding the chosen action $x_t$ and the observed reward $y_t$, it suffices to protect $(x_t,y_t{)}_{t\in \left[T\right]}$ to achieve DP requirements. Let ${\tau }_{\ell ,j}$ be time slots when user in cluster $j$ appears at local server $\ell$, we denote two sequences $S_{\ell ,j}=(x_t,y_t{)}_{t\in {\tau }_{\ell ,j}}$ and $S_{\ell ,j}^{\prime }=(x_t^{\prime },y_t^{\prime }{)}_{t\in {\tau }_{\ell ,j}}$ as $t$-neighboring if $(x_{\tau },y_{\tau })=(x_{\tau }^{\prime },y_{\tau }^{\prime })$ for $\tau \ne t\in {\tau }_{\ell ,j}$.

Note that our CDP notion formalizes the intuition that the action chosen by any local server $\ell$ (at the cluster level $j$) must be sufficiently indistinguishable (in probability) to any single $(x,y)$ pair from any other local cluster $({\ell }^{\prime },j^{\prime })$. Such a notion does not require each cluster is private to its own observations, i.e., each cluster of users can be trusted with its own data, which is different from the local DP [zheng2020locally] or Joint DP [shariff2018differentially] that assume even itself cannot be trusted.

Communication Complexity. To evaluate the communication complexity, we count one upload operation (or one download operation) between any local server and the global server as one communication round. Our communication complexity is the total number of communication rounds $C\left(T\right)$ over the time horizon $T$.

In this section, we introduce our phase-based federated clustering of bandit algorithms with CDP (FCLUB-CDP).

Identify the Underlying Cluster Structure. To correctly identify the cluster structure, we design a phase-based clustering detection algorithm in Algorithm 2. The high level idea is to first conduct local-level clustering of bandits on each local server and merge the local clusters on the global server.

At the local level, each server $\ell$ maintains a profile of $(V_{t,i},b_{t,i},T_{t,i})$ for its own local users $i\in U^{\ell }$, where $V_{t,i}$ is Gramian matrix, $b_{t,i}$ is the moment vector of regressand by the regressors, and $T_{t,i}$ is the number of times that $i$ has appeared up to time $t$. At the beginning of $s$ phase (or $t=2^s+1$), based on profiles $(V_{t,i},b_{t,i},T_{t,i}{)}_{i\in U^{\ell }}$, each server $\ell$ maintains an undirected graph structure $G_s^{\ell }$, where nodes represent all local users $U^{\ell }$ and a pair of users are connected by an edge if they are similar. We initialize the graph by a complete graph $G_0^{\ell }$ and gradually delete edges at every beginning of each phase $s$. Specifically, in line 3, we delete edge between any user $i_1$ and $i_2$ if the distance between their estimated preference vector are larger than the following threshold.

where ${\hat{\theta }}_{t,i}=(\lambda I+V_{t,i}{)}^{-1}{b}_{t,i}$ and $F\left(x\right)=\sqrt{\frac{1+\ln (1+x)}{(1+x)}}$. After the deletion, users in connected components $j\in C\left(G_s^{\ell }\right)$ are grouped into local cluster $j$. In line 5, server $\ell$ uploads the clustered information $I_{s,\ell }=(C_{s,j}^{\ell },{\tilde{V}}_{s,j}^{\ell },{\tilde{b}}_{s,j}^{\ell },{\tilde{T}}_{s,j}^{\ell }{)}_{j\in C\left(G_s^{\ell }\right)}$ to the global server, which contains clustered information for each cluster $j\in C\left(G_s^{\ell }\right)$. Note that $I_{s,\ell }$ are added with random perturbation to protect the users’ data, which will be introduced shortly after.

In line 7, when the global server receives the privatized clustered information from all servers, it performs a merge operation to merge clusters from different servers whose estimated clustered preference vectors are close into a global clusters according to the following inequality.

where ${\hat{\theta }}_{s,j}^{\ell }=({\tilde{V}}_{s,j}^{\ell }{)}^{-1}{\tilde{b}}_{s,j}^{\ell }$ and $F\left(x\right)=\sqrt{\frac{1+\ln (1+x)}{(1+x)}}$. $P_s$ denotes the set of $m_s$ global clusters. Note that the local clusters in the same global cluster indexed by $k\in \left[m_s\right]$ will communicate and share protected clustered information with each other in an asynchronous manner. At the beginning of phase $s$, if the new global cluster structure $P_s$ is different from $P_{s-1}$ at phase $s-1$, we will renew the shared global information $(S_{t,k}^g$, $u_{t,k}^g$, $T_{t,k}^g)$ for $k\in \left[m_s\right]$. For local servers $(\ell ,j)$ in the same global cluster $P_{s,k}$, the local synchronized information $(S_{t,j}^{\ell },u_{t,j}^{\ell },T_{t,j}^{\ell })$, the upload buffers $(\Delta S_{t,j}^{\ell },\Delta u_{t,j}^{\ell },\Delta T_{t,j}^{\ell })$ and download buffers $(\Delta S_{t,j}^{-\ell },\Delta u_{t,j}^{-\ell },\Delta T_{t,j}^{-\ell })$ are renewed. We also generate a new tree-based privatizer PVT$(\ell ,j)$ for each cluster $j$ at server $\ell$, which will be introduced later on.

Asynchronous Communication Protocol. In this work, we design a novel asynchronous communication protocol to incorporate the randomly arriving users. To reduce the communication cost, our high-level idea is to use the delayed communication, where the feedback are temporarily stored in buffers and only if the stored information exceeds a threshold, the upload/download events are triggered. Such a threshold will ensure that the local information will not diverge too far from the global information, which in turn will not diverge too far from the scenario when information are fully synchronized. Also note that our communication is conducted in the asynchronous manner at the local cluster level. In other words, all local clusters indexed by $(\ell ,j)\in P_{s,k}$ will establish connection with each other within the global cluster $k$. For each local cluster $(\ell ,j)$, it stores a local copy of the sufficient statistics $(S_{t,j}^{\ell },u_{t,j}^{\ell },T_{t,j}^{\ell })$ and a upload buffer $(\Delta S_{t,j}^{\ell },\Delta u_{t,j}^{\ell },\Delta T_{t,j}^{\ell })$. For the global server, it prepares for each local cluster $(\ell ,j)$ a download buffer $(\Delta S_{t,j}^{-\ell },\Delta u_{t,j}^{-\ell },\Delta T_{t,j}^{-\ell })$, which are used to send other local servers’ information to the local cluster. It also maintains the global statistics $(S_{t,k}^g,u_{t,k}^g,T_{t,k}^g)$ to save the data uploaded from local clusters in global cluster $k$.

Our proposed communication framework consists of two components: the upload protocol (Algorithm 3) and the download protocol (Algorithm 4). For the upload protocol, at each time step $t$, user $i_t$ visits the server $l_t$ and receives recommended item $x_t$. After the user $i_t$ interacts with the environment and observes feedback $(x_t,y_t)$, the local server updates the upload buffers in line 1 and checks the following condition to decide whether to upload the upload buffer:

where $H_{t,j_t}^{l_t}$ and ${\overline{H}}_{t,j_t}^{l_t}$ are tentative and current perturbation for privacy protection, respectively. If the condition is satisfied, the local server sends $(\Delta S_{t,j_t}^{l_t},u_{t,j_t}^{l_t},T_{t,j_t}^{l_t})$ to the global cluster $k_t$. The global server then merges the uploaded information into the global information in line 4 and also sends it to download buffers for other local clusters $(\ell ,j)\ne (l_t,j_t)\in P_{s,k_t}$ in line 6. For local cluster $(l_t,j_t)$ itself, the local server updates the local statistics and initializes the upload buffer using the newly generated perturbation in lines 8 to 10. For the download protocol, at each time step $t$, the global server will check the deviation between global statistics and the local statistics via following condition:

independently for local clusters $(\ell ,j)$ in global cluster $P_{s,k_t}$. If any cluster $(\ell ,j)$ satisfies such condition, the global server sends the information from other clusters to $(\ell ,j)$, which is used to update $(\ell ,j)$’s local statistics. Finally, the global server cleans the download buffer.

Tree-Based Privacy Protocol. To ensure the uploaded information are privatized, we adopt the tree-based privatizer to generate random perturbations $H_{t,j}^{\ell }$ and $h_{t,j}^{\ell }$ whenever an upload event happens. Note that the privatier subroutine is at the local cluster level and a new privatizer is created if the cluster structure changes at the start of any phase $s$.

Let $x_1,...,x_T$ be a (matrix-valued) sequence of length $T$, and $s_i={\sum }_{t=1}^i{x}_i$ be the partial sum of the first $i$ elements that will be realised privately. Generally speaking, the tree-based mechanism dwork2006calibrating maintains a binary tree $T$ of depth $1+\lceil \log T\rceil$, where the leaf nodes contain the elements $x_i$ and the parent node maintains the sum of its children. For each node with value $n_i$, the tree-base mechanism protects privacy by adding noise $h_i$ to each node and release $n_i+h_i$ if queried. The key advantage is that such a tree only accesses $\nu =O\left(\log T\right)$ nodes to compute and release the partial sum $s_i$, which means the perturbation is at most $O\left(\nu \right)$ instead of $O\left(T\right)$.

Following this general idea, we implement the tree-based privatizer $(\ell ,j)$ that satisfies the requirements of CDP. Recall that we only need to protect the information uploaded to the global server, it suffices to maintain a tree $T_j^{\ell }$ of depth $\nu =O(1+\lceil \log t_c\rceil )$ for the upload event, where $t_c$ is the total number of uploads. To make the partial sums private, we insert a random noise matrix to each node in $T_j^{\ell }$, similar to that of shariff2018differentially and dubey2020differentially,. Specifically, we sample a random matrix $\overline{N}\in R^{(d+1)\times (d+1)}$ where each entry ${\overline{N}}_{p,q}$ is drawn from i.i.d. Gaussian distribution $N(0,{\sigma }_{\text{noise}})$ and symmetrize it to get $N=({\overline{N}}^{\top }+\overline{N})/\sqrt{2}$. It follows that in order to ensure the whole tree is $(\epsilon ,\delta )$-DP, each node should preserve $\left(\epsilon /\sqrt{8\nu \log (2/\delta }\right),\delta /2)$-DP. In other words, it suffices to set the variance ${\sigma }_{\text{noise}}=64\nu \log (2/\delta {)}^2/{ϵ}^2$ for each tree node. Note that at each upload round $t$, the total noise added to the partial sum is the summation of at most $\nu$ random matrices with size $(d+1)\times (d+1)$, where the top-left $(d\times d)$-submatrix forms $H_{t,j}^{\ell }$ and the first $d$ elements from the right-most $(d+1)\times 1$ vector forms $h_{t,j}^{\ell }$. By concentration of random matrices tao2011topics, we have with probability at least $(1-\frac{\alpha }{mL})$, the operator norm of $H_{t,j}^{\ell }$ is

for any $\ell \in \left[L\right],j\in \left[m\right],t\in \left[T\right]$.

Recommendation Procedure. At each time step $t$, the recommended item $x_t$ for user $i_t$ is selected as follows. When the current cluster is correct (which is guaranteed after $O\left(\log T\right)$ rounds and to be proved later), the estimated ${\hat{\theta }}_t=(S_{t,j_t}^{l_t}{)}^{-1}{u}_{t,j_t}^{l_t}$ is computed using the local information $S_{t,j_t}^{l_t}$ and $u_{t,j_t}^{l_t}$. Since by Lemma 1, ${\Vert {\theta }_{i_t}-{\hat{\theta }}_t\Vert }_2\le {\beta }_{t,j_t}^{l_t}$, the confidence radius is ${\beta }_{t,j_t}^{l_t}{\parallel x\parallel }_{(S_{t,j_t}^{l_t}{)}^{-1}}$, which characterizes the exploration bonus for item $x\in D_t$. Then the local server will recommend the item $x_t\in D_t$ that maximizes the $x^{\top }{\widehat{theta}}_t$ plus the above exploration bonus. Finally, the user will receive feedback $y_t$ and the system updates corresponding statistics for better decision in future rounds.