DRL Enabled Coverage and Capacity Optimization in STAR-RIS Assisted Networks

As can be seen from related works, the CCO problem of STAR-RIS assisted wireless networks is still at its early stage. In this research direction, there are two main challenges:

• Characterizing Coverage in STAR-RIS assisted Networks: To efficiently characterize the performance metric of the coverage, it’s better to extend to the two-dimensional (2D) coverage instead of 1D range distance. Additionally, simultaneously characterizing both coverage and capacity is challenging when evaluating the performance of STAR-RIS assisted networks.

• Designing MORL Algorithms to Solve COO Problem: Conventional Pareto-based MO optimization (MOO) solutions mainly aim to find a Pareto front (PF) of objectives within one time step, which ignore the dynamic requirements of temporal correlations in long-term wireless communications. PPO is a policy gradients method where policy updates use a surrogate loss function to avoid catastrophic drops in performance. In addition, the new MOO methods [20, 21] have the capability to dynamically update the weights of objectives. Therefore, how to obtain the Pareto optimal (PO) solution based on PPO algorithm and these two new MOO methods is challenging.

To solve these challenges and fully reap the advantages of STAR-RISs, in this paper, we propose a new RL approach based on the PPO algorithm, named multi-objective PPO (MO-PPO) algorithm, to provide the maximum coverage and capacity for STAR-RIS assisted networks. The optimal results obtained by the MO-PPO algorithm are different according to the different update strategies. The main contributions of this paper can be summarized as follows:

• We propose a new model for a narrow-band downlink STAR-RIS assisted network consisting of two single-antenna BSs, where the serving range is defined as a square region. To quantitatively analyze the coverage and capacity, the serving range is discretized into numerous square grids and the center point of each grid acts as the evaluating sample point. Based on this framework, this work formulates the CCO problem of STAR-RIS assisted networks by jointly optimizing the transmit power, the reflection phase shift matrix, and the transmission phase shift matrix.

• We investigate an action value-based update strategy (AVUS) for the MO-PPO algorithm to solve the CCO problem. The core point of this strategy is to integrate the action values of both coverage and capacity by random sampling preferences, and further invoke a coefficient to update the policy by homotopy optimization. This update strategy with high performance is able to provide the optimal coverage and capacity, while it has to spend a long time to achieve convergence. Therefore, the AVUS has strict requirements on the computation resource, which is suitable for networks with strong computation capability.

• We adopt a loss function-based update strategy (LFUS) for the MO-PPO algorithm to reduce the complexity brought by the AVUS. The improved point is to assign dynamic weights for both loss functions of coverage and capacity, and to update the whole MO-PPO policy with an integrated loss function of coverage and capacity. The dynamic weights are re-calculated by a min-norm solver at every update. Compared to the AVUS, this strategy has slightly worse performance, but it still has acceptable performance gain when compared to the existing CCO solutions.

• We illustrate that both AVUS and LFUS based MO-PPO algorithms are capable of striking a balance between the conflicting goals in terms of coverage and capacity. Then, AVUS and LFUS based algorithms are able to provide the Pareto optimality compared to conventional fixed weights MOO algorithms. With the same bandwidth, millimeter wave (mmWave) is able to provide better capacity while sub-6 GHz provides better coverage. Next, the coverage and capacity have a positive correlation with the number of STAR-RISs. Finally, when the number of elements in STAR-RISs is fixed, the coverage and capacity have a negative correlation with the physical size of STAR-RISs.

The rest of this paper is organized as follows. Section II presents the system model for the considered STAR-RIS assisted networks, and the coverage and capacity optimization problems are formulated. Section III provides the preliminaries, including the principles of the PPO algorithm and the PO solution. In Section IV, we investigate the two updated strategies-based MO-PPO algorithms, i.e., AVUS and LFUS, which are updated for different parts of the algorithm. Section V presents numerical results to verify the effectiveness of the proposed MO-PPO algorithms, by considering the different number of sample grids, the different number of elements in STAR-RISs, the different number of STAR-RISs, and the different physical sizes of STAR-RISs module. Finally, Section VI concludes this paper.

Notations: Scalars, vectors, and matrices are denoted by lower-case, bold-face lower-case, and bold-face upper-case letters, respectively. The conjugate transpose of vector $a$ is denoted by $a^H$. The diag($a$) denotes a diagonal matrix with the elements of vector $a$ on the main diagonal. The $|a|$ denotes the norm of vector $a$. The Mod($a,b$) denotes the modulus operation between values $a$ and $b$. The $\lfloor a\rfloor$ denotes the truncated argument of value $a$. The $\ast$ denotes the dot multiplication operation. The $E\left[A\right]$ is the expectation operator of matrix $A$. The log${}_2$($A$) represents a logarithmic function with a constant base of 2 for matrix $A$. The $tr\left(A\right)$ denotes the trace of matrix $A$.

Assuming a three-dimensional (3D) Cartesian coordinate system, where the origin is set at the top-left corner. The locations of two BSs and $n_s$-th STAR-RISs are denoted by $B_1=(R_s,0,h_b)$, $B_2=(R_s,R_s,h_b)$, and $A_{n_s}=(x_{n_s},y_{n_s},h_{n_s})$, respectively. Note that $h_{n_s}$ is the height of STAR-RISs module, and the thickness of STAR-RISs are ignored. The height $h_{n_s}$ and width ${\omega }_{n_s}$ are depicted in Fig. 1. The indicators $I_{h_{n_s}}$ and $I_{{\omega }_{n_s}}$ are invoked to characterize $h_{n_s}$ and ${\omega }_{n_s}$, which can be expressed as follows:

To characterize the coverage and capacity, the region is discretized into numerous square grids with the length of the side $R_g$, while the center point of each grid acts as the sample point . Accordingly, the total number of grids is $N=\lceil R_s/R_g{\rceil }^2$, where the set of sample points can be denoted as $s=\{s_1,s_2,...,s_N\}$. In practical networks, in order to characterize the importance of each grid at each time step $t$, two time-related weights, $w_{cov,i}\left(t\right)$ and $w_{cap,i}\left(t\right)$, are assigned for coverage and capacity of each sample points $s_i$ ($i\in [1,N]$), respectively. Moreover, the weights have been unified, i.e., ${\sum }_{i=1}^N{w}_{cov,s_i}\left(t\right)=1$ and ${\sum }_{i=1}^N{w}_{cap,s_i}\left(t\right)=1$. In this system model, we study long-term communication with a time period $T$. For each sample point at any time step, the weighted assignments $w_{cov,s_i}\left(t\right)$ and $w_{cap,s_i}\left(t\right)$ are influenced by the previous network performance and resource allocation strategy. Therefore, the considered problem can be regarded as a Markov Decision Process (MDP).

In this section, the fading channels from BSs to STAR-RISs, from STAR-RISs to sample points, and from BSs to sample points are introduced, as well as their spatial channel correlations. Denote ${\Phi }_{\delta ,n_s}$ as the coefficients of $n_s$-th STAR-RISs with mode $\delta$, where $\delta \in \{Re,Tr\}$ represents the reflection and transmission modes. Due to the high path loss, this work assume that signals are only reflected and transmitted by the STAR-RISs once. We consider the non-ideal STAR-RISs with same constant amplitude and continuous phase shifters in each mode, where the phase shifters can be expressed as [23]: ${\varphi }_{\delta ,n_s,k}\in [0,2\pi ),\forall k\in \{1,2,\cdots ,K\}$. The coefficients of $n_s$-th STAR-RISs are denoted as ${\Phi }_{\delta ,n_s}=diag(\sqrt{{\beta }_{\delta ,n_s}}{e}^{j{\varphi }_{\delta ,n_s,1}},...,\sqrt{{\beta }_{\delta ,n_s}}{e}^{j{\varphi }_{\delta ,n_s,K}}),\forall k\in \{1,2,\cdots ,K\}$, where $\sqrt{{\beta }_{\delta ,n_s}}\in (0,1],{\beta }_{Re,n_s}+{\beta }_{Tr,n_s}=1$ [24]. As shown in Fig. 1, a spherical coordinate system is defined with azimuth angel $\psi$ and elevation angel $\theta$ based on the 3D space. Denote the area of each element as $M=M_H{M}_V$, where $M_H$ and $M_V$ are the horizontal width and vertical height, respectively. Thus, the total area of $K$ elements can be expressed as $M_a=KM$. For the $k$-th element, its location can be expressed as [25]:

Assume that these channels are independently distributed and corresponding channel state information are perfect. Denote $h_{a,n_s}$, $h_{\delta ,n_s,s_i}$, and $h_{a,s_i}$ as the channel from $a$-th BS to $n_s$-th STAR-RISs, from $n_s$-th STAR-RISs to $s_i$-th sample point with mode $\delta$, and from $a$-th BS to $s_i$-th sample point, respectively. Here, the channels $h_{a,n_s}$, $h_{\delta ,n_s,s_i}$, and $h_{a,s_i}$ can be modeled as Rician fading model, which are expressed as:

Since the size of STAR-RISs module affects the direct link, the received signal $y_{a,n_s,s_i}\in C$ from the $a$-th BS to the $s_i$-th sample point via $n_s$-th STAR-RISs is determined by $I_{n_s}$. Thus, the recieved signal $y_{a,n_s,s_i}$ can be written as [26]:

PPO algorithm is based on trust region policy optimization [27] and utilizes the typical actor-critic architecture. The actor network is to determine the action according to the current state and the critic network, where the critic network is to evaluate how well the actor network performs the action. The configuration of the PPO algorithm is shown in Fig. 2. Note that, the design of the architecture is modularized to separate the cohesion between neural networks, PPO algorithm, and environments. The action-value function in the PPO algorithm is replaced by an advantage function ${\hat{A}}_t$ at every $\overline{T}$ time steps, which is expressed as:

In multi-objective optimization problems, each objective function may have an individual optimal solution, while these solutions usually have significant differences. Therefore, multi-objective optimization with such conflicting objective functions provides a set of optimal solutions, namely, PO solutions [29]. As shown in Fig. 3, considering two conflict objectives, both of which aim to be maximized. The point $C_1$ represents a solution that $F_2$ is near-maximum, but $F_1$ is low, while point $C_4$ indicates a solution $F_1$ is near-maximum, but $F_2$ is small. However, it is difficult to distinguish whether solution $C_1$ is better than $C_4$, or vice versa. In fact, there exist many such solutions belonging to the PO set, which forms a primary PF. Additionally, $C_5$, $C_6$, and $C_7$ are the feasible solutions. $C_5$ belongs to the second PF, while $C_6$ and $C_7$ are the part of the third PF [29].

In this work, the locations of STAR-RISs are randomly pre-selected. Note that, the locations of STAR-RISs are not overlapped. Moreover, STAR-RISs are placed along the y-axis direction to ensure that transmit signal from any BS is reflected and transmitted using the same planar of each STAR-RISs.

In MO-PPO algorithm, the MDP can be represented by a tuple $⟨\overline{S},\overline{A},p,\overline{R}⟩$ with state space $\overline{S}$, action space $\overline{A}$, reward space $\overline{R}$. The $p$ is the transition probability matrix indicating the probability of changing the current state to the next state. Define a controller as an agent, which controls both two BSs, to develop the policy from the BSs to sample points via STAR-RISs, i.e., the adjustment policies of phase shifts and transmit power. At each time step $t$, the controller observes the state $S_t$ from state space $\overline{S}$, and carries out an action $A_t$ from action space $\overline{A}$. The received reward $R\subseteq \overline{R}$ is calculated by the current state and action, and determine the transition probability to the next state $S_{t+1}$. Additionally, since the locations of STAR-RISs are pre-determined, the distance between any BS and $n_s$-th STAR-RISs is fixed. The coverage and capacity are determined by the distance between STAR-RISs and $s_i$-th point and the corresponding phase shift of the STAR-RISs, according to the (13) and (14). Thus, the state $S_t$ can be defined symbolically as follows: