Formation control with connectivity assurance for missile swarm: a natural co-evolutionary strategy approach

This paper focus on the formation control in two-dimensional space, i.e., the $OXY$ planar space of the inertial coordinate frame. As show in \prettyreffig:engagement1, the swarm is consist of multiple missiles, each missile can be treated as point mass, In the simplified dynamics model we do not consider aerodynamic factors and the effect of the missile’s own inertia tensor. Consider a total of $N$ missiles, for missile $i$ denoted by $M_i$, $V_{mi},{\alpha }_{mi}$ $x_{mi}$ and $y_{mi}$ are used to represent the velocity and heading angle, and its coordinates in the global coordinate frame, respectively. Let $X_i=[x_{mi},y_{mi},{\alpha }_{mi}{]}^T$ represents the state vector of the missile, which are not necessarily required hereinafter, and $X_r=[x_r,y_r,{\alpha }_r{]}^T$ represents the state of the reference target. For brevity, let ${\hat{X}}_i=[X_i^T,V_{mi}{]}^T$ be the extended state vector with speed inserted. As shown in the partial enlargement of the actuation decomposition, the missiles have two independent input $a_{vi},a_{li}$ which are the acceleration commands both along and perpendicular to the direction of the velocity, respectively. The second-order system dynamic of the $i^{th}$ missile can be expressed as

where $u_i=[a_{vi},a_{li}{]}^T$ is the input for $M_i$, and

where $I_n,0_{m,n}$ denote the identity and zero matrix, and $j$ represents the neighboring missile. Note that system \prettyrefeqn:systemDynamic is similar to the system discussed in aicardiClosedLoopSteering1995 which is often called unicycle-like vehicle. In this paper, the missile is a nonholonomic system with controllable speed and capable of maneuvering with limited actuation and dynamic constrains. We assume that under certain communication relationship, the relative state $z_{ji}$ can be sensed by missile $i$. In addition we make the following assumptions about the system:

The formation pattern is defined by $\lambda =\{{\lambda }_{pi}=[{\lambda }_{xi},{\lambda }_{yi}{]}^T:i=1,2,...,N\}$, which is the set of Cartesian coordinates of the origin of each node that are expressed in the coordinate system, such that the center of the formation is the origin which should satisfies

Let $N_i$ denotes the set of neighboring missiles of the $i^{th}$ missile. In the displacement-based framework, each missile has to align its own coordinate system with the global coordinate system and to be able to sense the relative positions and orientations of its neighbors, whereas its position in the global coordinate system is not necessarily required. Suppose that the reference target should be kept in the formation center, then the tracking error of the $i^{th}$ missile is

where $P_i=[{\lambda }_{xi},{\lambda }_{yi},0{]}^T$ and ${\zeta }_i=1$ if the state of the target is available to the $i^{th}$ missile, else ${\zeta }_i=0$, $e_i\in R^3$. It is worth noting that the first term represents the formation maintenance error, while the second term represents the target tracking error when it is able to obtain the target information. However, the error vector is defined in the global coordinate system, in order to standardize the effect of individual missile’s heading angles on the error vector to the relative coordinate system that are aligned with the missile’s orientation, we define the rotated error vector as $e_{ri}=R_3(-{\alpha }_i)e_i,$ where $R_3(-\alpha )\in R^{3\times 3}$ is the three dimensional rotation matrix defined as

Let’s $e_{ri}={\left[\begin{array}{ccc}{e}_{ix}& e_{iy}& e_{i\theta }\end{array}\right]}^T$. Considering system \prettyrefeqn:systemDynamic, by differentiating $e_{ri}$ with respect to time and considering the discretization of system control loop, we obtain the following error dynamic

where

$L_i$ is the number of missiles belonging to the set of neighbors $N_i$ and $\tau$ is the simulation step size. Furthermore, let $u_i=[a_{vi},a_l{]}^T$and $u_r=[v_r,w_r{]}^T$ denote the vector of control command, and $\tau$ be the control loop constant which is selected by the controller.

Generally, the objective metric at time $t$ can be defined by function of the error vector, which is

where $K_C=diag[k_1,k_2,k_3]$ is the symmetric positive defined matrix with positive weights. By choosing $K_C$ properly, the swarm balances between keeping the formation shape or tracking the reference target as a priority. The optimal design of the formation controller $u_i$ can be formulated as a nonlinear optimization problem

subjected to

in which $T$ denotes the total flight time, and $u_{min},u_{max},V_{min},V_{max}$ represents the system constraints.

In this paper we discuss the parametric definition of the formation geometry, which is rarely mentioned in the other works. We define two formation patterns, the regular polygon and the straight-line formation which are shown in \prettyreffig:formationPatternsDefinition, and denote them by ${\lambda }_{({\alpha }_p,l_f)}^P$ and ${\lambda }_{({\alpha }_p,l_f)}^L$. $l_f$ and ${\alpha }_p$ are the parameters that control the size and rotation of the formation, specifically we have

and $R_2\left(\beta \right)$ is a two-dimensional rotation matrix similar to \prettyrefeqn:rotationmatrixClockwise, which is

From the above definition, it can be intuitively observed that the straight-line formation is symmetric with respect to the origin of the coordinate system that defines it, so that the sum of its $x$ and $y$ coordinates is 0, which satisfies \prettyrefeqn:formationDefPrerequisite. Similarly when the number of nodes of a regular polygon is even, we can also get this conclusion. Still when the number of nodes is odd, we can prove that it still satisfies this prerequisite, and the following is a simple proof.

Suppose that the regular polygon formation defined in \prettyrefeqn:formationDef consists of $N\in \{2n+1:n\in Z\}$ nodes, the Cartesian coordinates of each node in the formation are denoted by $P_i=[{\lambda }_{xi},{\lambda }_{yi}{]}^T$, so that the sum of all coordinates is

Suppose the angle between the vector from the coordinate origin to an arbitrary node and the positive direction of the $Y$-axis is $\rho$. We can obtain the following sum of coordinates by multiplying with the rotation matrix to rotate node one to the $Y$-axis

At this configuration, since the formation is symmetric about the $Y$-axis, the sum of its x-coordinates can be obtained as $0$, so we have

thus ${\xi }_x={\xi }_y=0$, \prettyrefeqn:formationDefPrerequisite holds. It is shown that many other formation patterns can be considered as variants of the above two formation patterns, such as Row, Column, Square patterns limFormationControlLeader2009. Moreover, as discussed in the following \prettyrefsection:Implementation of Natural Co-evolutionary Strategy based formation control via neural networks, this formation paradigm can also generate some asymmetric patterns such as Wedge, Crescent patterns lanAdaptiveNeuralNetworkBasedShapeControl2018, etc., by deleting nodes appropriately.

With limited sensing capability, the MAS system can be modeled as multi-agent partial observable Markov decision process(POMDP), which is an extension on the basis of the fundamental Markov decision process(MDP). Such process can be expressed as

with $S$ and $S^{\prime }$ representing the system state before and after transition, and $A=[u_1,...,u_n]$ is the actions of all agents in the system. $P(\cdot )$ is the transition probability function. It is said to be deterministic optimization problem if $P(\cdot )=1$ for all time, otherwise it is stochastic optimization problem.

Considering of the simplified system model neglecting uncertainties, the formation control problem discussed in general can be solved using optimization algorithms under multi-agent POMDP. Assuming that the fitness function measuring the performance of agent $i$ is $f_i({\theta }_i,{\theta }_{N_i})$, in which ${\theta }_i$ is the controller(policy) parameter representing agent $i$, and ${\theta }_{N_i}=\{{\theta }_j:j\in N_i\}$ is set of neighboring parameters of agent $i$. The objectives of optimal control is then to find the optimal control strategy ${\theta }_i^{\ast },i\in (1,...,N)$ , such that

where $F(\cdot )$ denote the overall performance of all agents in the MAS. Such solution is often referred to as the Nash equilibrium strategy, meaning that can no further improvement can be made to individual solutions without deteriorating the performance. It has been found to be difficult to find a Nash equilibrium strategy while satisfying the system constraints \prettyrefeqn:systemConstraints, and also limited by the nonlinear continuous system and the cost function with coupled parameters sefriouiNashGeneticAlgorithms2000; seung-mokleeCooperativeCoevolutionaryAlgorithmBased2015. For the formation control problem which requires sufficient cooperation among neighboring agents, $F(\cdot )$ can be regarded as a simple summation of all individual $f_i(\cdot )$, and the formation control problem can be viewed as a constrained dynamic optimization problem with the objective of minimizing the total cost of formation error. Thus the Nash equilibrium strategy can be obtained using a co-evolutionary algorithm that is designed for evolving simultaneous to reach the overall optimum fitness. In our previous work chenCooperativeGuidanceMultiple2022, we improved on the natural evolutionary strategy (NES) and proposed an natural co-evolutionary strategy (NCES) algorithm that seeks global optimality for constrained multi-objective optimization problem in multi-agent system. In brief, NCES is a bio-inspired population-based algorithm that is capable of optimizing high-dimensional parameters, such as the weights of neural networks, towards the direction of higher fitness. The NCES algorithm usually proceeds as follows: first, the parameters ${\theta }_i$ for $i\in (1,...,N)$ are initialized, and the optimization objective is determined for a specific control problem. The fitness function $f(\cdot )$ which is embedded within the system model is thus obtained. Then, in the second step, iterative optimization is performed, and new perturbations ${ϵ}_i$ are sampled $m$ times at the beginning of each iteration step which obey a probability distribution $p(\cdot )$ to obtain the perturbed population ${\theta }_i^{\prime }={\theta }_i+{ϵ}_i$ corresponding to each agent, and their fitness values are then evaluated in the system in a distributed way, so that each population obtains the following corresponding gradient information

and updates its parameters in a gradient ascent manner

Finally, loop the second step until convergence or the Nash equilibrium strategy is found. For more detailed procedure of the algorithm, please refer to the original research.

Motivated by previous works which implement NNs in MAS, we use a multi-layer perceptron (MLP) NNs with a single hidden layer of 16 nodes as the distributed controller. The schematic of the neural networks controller is depicted in \prettyreffig:NeuralNetworkSchematic.

The weighting matrix for each layer are represented by $W_{1i}\text{and}{W}_{2i}$, with $\varphi (\cdot )$ and $\psi (\cdot )$ as their activation function, respectively. Specifically, $\varphi (\cdot )$ is the Sigmoid function and $\psi (\cdot )$ is selected as the hyperbolic tangent function $\left(Tanh\right)$ in order to impose restrictions on the output to satisfy the system constraints. Another advantage of the neural network controller is shown that saturated control can be achieved by a reasonable choice of activation function without restricting the solution space. With $z=[{\alpha }_{mi},V_{mi},e_{ri}{]}^T$ denoting the input, the output of the neural network controller is

Note that the dashed network input in the schematic represents the transform signal (TS), which is included in the input only when needed, as described in \prettyrefsection:Simulation and result analysis. The same network configuration will be used in the following experiments for all agents. Applying the previously mentioned NCES algorithm to train this controller, it is firstly determine that the parameters of the controller are ${\theta }_i=[W_{1i},W_{2i}]$. The fitness function is the objective function \prettyrefeqn:objectiveFunction, i.e., $f({\theta }_i,{\theta }_{N_i})=J_i$ which is non-linearly coupled with the sates of neighboring agents and can be only evaluated through the interaction feedback within the system. In order to apply NCES to the training of the missile formation controller in this paper, the plain NCES algorithm is not sufficient to guarantee the global stability and the convergence speed of the algorithm. To further improve the performance of the algorithm and to perform specific optimization for the formation problem, we propose a series of supplementary techniques to effectively enhance the convergence speed and accuracy of the algorithm.

In chenCooperativeGuidanceMultiple2022 we point out the importance of learning rate on the accuracy of the algorithm and proposed an algorithm for learning rate adaptation, however, adjusting the learning rate often consumes a lot of time, in order to ensure the speed of the algorithm here a novel population size adaptation algorithm is considered, which can be used to adjust the evolutionary process adaptatively. Based on the previous works nishidaPSACMAESCMAESPopulation2018; nomuraPrincipledLearningRate2022, we can learn that the trend of the gradient is related to the complexity of the objective function, usually for the more complex optimization region such as multi-modal or noised function the estimated gradient is gentle, while in the flatter region such as the spherical space the gradient is relatively steep. To estimate the accuracy of the estimated natural gradient under the movement of the parameter distribution, the evolution path ${\rho }_{\theta }$ is introduced to detect the resistance in the evolutionary process. The population size is adapted based on the length of the evolutionary path following an empirical common sense that larger population size will lead to higher accuracy of the estimated gradient. Considering of the parameters of the $i^{th}$ agent as the vector consisting of the weighting matrices $W_{1i},W_{2i}$, that is ${\theta }_i\in R^s$, and $s$ is the total number of weights. The evolution path in iteration $t$ is calculated by accumulated the square of the Mahalanobis distance of the parameter movement of all agents as

where $\Sigma$ is the covariance of the probability distribution from which the new populations are sampled, since the evolution path should not depend on the parameterization of the probability distribution. The population size ${\eta }_p\left(t\right)$ is then adjusted according to the evolutionary path as follows:

where $\beta$ is a constant factor that determines the growth rate of the population size and ${\eta }_p^{min},{\eta }_p^{max}$ are the minimum and maximum population size which are sent to the clip function $\text{clip}(\cdot )$in case of undesirable adapted values. Note that since the total optimization complexity tends to increase as evolution progresses, we adopt a non-decreasing strategy to adjust the population size to ensure stability. The initial population size is set to be ${\eta }_p\left(0\right)=10+5\ln \left(s\right)$ referring to the default setup in glasmachersExponentialNaturalEvolution2010, which should be a good candidate, and the boundaries are determined as

Note that $s$ is the number of parameters of one agent, the above configuration is found to be appropriate according to experimental results. Through the empirical observation, we came to an intuitive conclusion that the increment of population size does not necessarily lead to the improvement of evolutionary quality, sometimes even leads to the difficulty of convergence or falls into the local optimum, which is presumably because a large population size will greatly average the contribution of individuals and thus reduce the exploratory nature of individual, so it is wise to adjust the size appropriately rather than just thinking that the larger the better.

The inter-agent connectivity in the filed of multi-agent system have been primarily modeled and characterized by means of graph theory desaiControllingFormationsMultiple1998; ohSurveyMultiagentFormation2015, which in this paper will be utilized to identify the observable neighbors of the missiles in the formation. In this subsection we review some of the basics of graph theory. Suppose there are $n$ agents in the MAS (agents can be represented by nodes), and a graphs $G$ is defined as $(V,ϵ)$, where $V=(v_1,v_2,...,v_n)$ represents the set of nodes, and $ϵ\subseteq V\times V$ represents the set of edges composed of directed connections of different nodes. The neighbor set of node $i$ is defined as $N_i=\{j\in V:(i,j)\in ϵ\}$. $j\in V$ is said to be connected to $i\in V$ in a directed way if $(i,j)\in ϵ$ and $(j,i)\notin ϵ$, otherwise $j$ is said to be connected to $i$ in an undirected way if both $(i,j)$ and $(j.i)$ belongs to $ϵ$ . A directed path of $G$ is a series of adjacent edges of the form$(v_1,v_2),(v_2,v_3),...,(v_i,v_j),(v_j,v_k)$, and a graph is said to have a spanning tree if there exist a directed path from one node to any other nodes, and a graph is said to be connected in directed (undirected) manners if there is a directed (undirected) path between any pair of distinct nodes. We use the adjacency matrix $A=\left[a_{ij}\right]\in R^{|V|\times |V|}$ to represent the above node connectivity, in which

The associated Laplacian matrix is $L=D-A$. $D=diag\left(a_i\right)$ is the diagonal matrix of vertex degrees where $a_i$ is the degree of vertex $i$, and $a_i={\sum }_{j=1}^n{a}_{ij}$. The Laplacian matrix $L$ is symmetric semi-positive definite matrix.

One of the foremost goal of designing communication method is to ensure highly reliable and low latency communication for all nodes in the swarm. The communication quality of missile swarm is easily affected by various conditions especially when performing in hostile environment. In another word, swarm of missiles are constantly subject to threats from enemy’s air defense system during flight, and some missiles may get intercepted and lost communication with the rest of the swam. Therefore, dynamic communication approaches needs to ensure that when some nodes fail, the swarm can still maintain effective communication to complete the mission. Based on the previous work in network topology and swarm communication, we developed a novel adaptive cluster-based network, which adaptively reconfigure the communication topology to achieve robust and fault-tolerant communication. Under the guidance of the well known Molloy-Reed criterion barabasiNETWORKSCIENCENETWORK, which is

$k$ denotes the average degree of an arbitrary node which can be considered somehow as the node degrees $a_i$, it can be inferred that each node in the communication network should be at least connected to two other nodes. What is also known is that for displacement-based formation, the formation is persistent only if there exists at least a spinning-tree in the communication topology. When establishing a communication connection, the formation error tends to grow as the communication distance between nodes increases, so we favour the connection method which minimizes the length of the communication chains dasVisionbasedFormationControl2002. With the minimal constraint of satisfying the above conditions, we propose to set one node(usually node $1$) of the network as the cluster head and the other nodes choose another communication node according to the inter-node distances that are derived from the formation definition. For achieving this, we define the inter-agent distance matrix as

where $d_{ij}$ is the Euclidean distance between nodes that can be calculated from the definition of the formation in \prettyrefsection:Preliminaries and Problem Formulation, and $D_g$ is symmetric positive definite matrix. Given that node $h$ is selected as the cluster head, the element of the adjacency matrix is determined as

In the proposed adaptive topology, the cluster head undertakes the task of broadcasting its own state to other nodes in the network, or follower nodes need to follow the cluster head, in either case the cluster head is not constrained by nodes other than the reference target. It is also important to note that only the cluster head has access to the reference target information, which greatly reduces the communication or detection burden of follower nodes.

Using the above communication configuration method for a network with 5 nodes as an example, the obtained communication topology is shown in \prettyreffig:adaptiveTopology_fiveNodes and its adjacent matrix is

Node 1 is the cluster head by default. When node 1 fails, node 2 successively inherits the head position as shown in \prettyreffig:adaptiveTopology_failedNodes, and the adjacent matrix becomes

Using the proposed adaptive communication method, the network connectedness can be ensured no matter how many nodes fail. The proof is that graph $G$ is uniformly connected since for any $t\ge t_0$, there exists a node $h\in V$ such that $h$ is the root of a spanning tree linStateAgreementContinuous2007.

The NCES algorithm explores the optimal policy that maximizes fitness by continuously interacting with the environment; consequently, the NCES algorithm and the reinforcement learning algorithm are prone to fail to achieve global convergence of the control policy in many cases due to excessive cumulative time or inherent defects in the design of the objective function. In much of the previous literature, control policies were allowed to freely explore regions of the environment under system constraints. In recent years, the idea of constrained policy has gradually emerged achiamConstrainedPolicyOptimization2017, which states that control policies should be executed under constraints that can be imposed either by human-developed rules or by the feedback state of the system. Constrained policy is also known as safety exploration rayBenchmarkingSafeExploration has been increasingly applied to simulation and training of realistic robotic controllers, by which not only can personal safety be ensured during the training process, but also global convergence can be accelerated to some extent. Based on the above facts, we propose a model-based constrained policy strategy. The non-linear error dynamic of the second-order system has been obtained in \prettyrefeqn:errorDynamic, which is a function of the system input. To apply the method, we assume that the control input of the communication object can be obtained by the agent, so that at time $t$, the predicted formation error at next step can be calculated as follows:

where $\tau$ is the time step, and the error deviation $△e_{ri}\left(t\right)\in R^3$ is

then the aggregate matrix is defined as $△E\left(t\right)=[...△e_{ri}(t)...]\in R^{3\times n}$, for $i\in \{1,2,...,n\}$ . A termination indicator $ST$ is assigned to the system so that if it is a nonzero value, the system terminates and restart the training algorithm, and it is defined as

and ${\delta }_s$ is the threshold that measures the maximum error increment that the algorithm can tolerate. It satisfies ${\delta }_s<(2l+1)\tau V_{max}$, where $l$ is the number of neighbors for the agent with the most neighbors in the MAS. With a reasonable choice of ${\delta }_s$, the algorithm is expected to exclude control strategies that deviate too much from the desired trajectory at an early stage. Experimental evidence shows that without such a constrained policy, the convergence speed and global optimality of the algorithm will be degraded. Applying the above adaptation techniques, the detailed pseudo-code of the proposed algorithm that is used to train the missile formation controllers of interest is shown in \prettyrefalg:fcalgorithm. Since the network topology adaptation and constrained policy strategies are embedded in the process of fitness evaluation, they are not exhibited in the pseudo-code.

First of all the basic formation control tasks are implement to examine the validity of the algorithm under basic situations, the missile swarm of five missiles entails tracking the reference target moving along diagonal or spiral trajectory while keeping the formation geometric. The reference target, or virtual leader is set to be at the center of the formation in order to achieve error-free formation control. The objective is to achieve zero tracking error as well as zero formation maintenance error for all time, that is ${\int }_t|e_{ri}\left(t\right)|dt\to 0_3,$ for $i\in \{1,...,N\}$. The hyper-parameters of the algorithm is listed in \prettyreftab:hyperParameters and it is noted that this parameter setting applies to all of the following experiments. Additionally, the system constraints are shown in \prettyreftab:systemConstraints, the missiles as well as the reference target are subjected to saturated control input and limited states. For the linear trajectory, the target is driven by constant control input, while for the spiral trajectory, the control input of the target is $v_r=0.65-0.01t\text{(km/s)}$ and ${\omega }_r=0.1+0.01t\text{(rad/s)}$.

For convenience, we use $|e_{ri}|$ to represent the resultant error of each agent. The trajectories of the two situations are shown in \prettyreffig:BasiclinearTraj and \prettyreffig:BasicspiralTraj, and the corresponding analytical results are presented in \prettyreffig:BasicAnalyticLinear and \prettyreffig:BasicAnalyticSpiral. It can be observed that the resultant error is maintained in a small interval (within 0.1) in both cases and that a good synchronization of the velocity and heading angles among the missiles is achieved, despite the fact that velocity information is not provided.

Moving into formation is however different from cases where the formation is in the ideal geometry in the initial state. The missiles are separated from the reference target with their positions initialized randomly in an area of 4 km wide and 3 km long. The missiles need to first move to the designated formation shape and then track the reference target in a consistent motion direction, and in this case, the target moves along the y-axis with a constant speed of 0.5km. \prettyreffig:mIfTraj shows the motion trajectory of the swarm formation, and \prettyreffig:resErrorMvForm shows the resultant error during the flight. It is evident from the results that the formation is able to adjust to the desired formation shape rapidly (mostly within 10 seconds) and achieve high accuracy in tracking and maintaining the formation in the case of random initial distribution.

Further, we discuss the case in which the missile swarm are supposed to switch among formation patterns in order to avoid obstacles and go through narrow spaces. To achieve this transformation, an additional node $TS$ is appended to the input layer of the policy network as depicted in \prettyreffig:NeuralNetworkSchematic. It’s assumed that whenever the obstacle is detected by the leader missile, or more literally the cluster head, the missile then send a signal to all connected missiles to perform formation switch ${\lambda }^R\to {\lambda }^L$. Although formation geometries are predefined by the controller, the shapes can be controlled by the parameter $l_f$ and $\beta$, which is changeable during the flight.

In this scenario, two walls are placed in the trajectory of the formation flight to create narrow spaces, and the missiles need to negotiate the obstacles by changing the shape or size of the formation. We implement a event-based formation switching strategy, in which nodes in the formation are able to detect obstacles within a certain range $d_c$ and send a formation switching signal to all other nodes if an obstacle is detected. Similarly, after crossing the obstacle and reaching a safe distance, a formation recovery signal will be sent to restore the original formation.

To cross the obstacle, we consider both changing the formation pattern and formation size, and the time-varying definition of the formation in both cases are as follows:

The result trajectories are shown in \prettyreffig:switchTypeTraj and \prettyreffig:switchSizeTraj, it can be observed that when obstacles are detected, the swarm can swiftly adjust by changing the formation pattern or size, and then recover the formation quickly after going through.

To verify the effectiveness of the proposed algorithm under node failure. In this experimental scenario, a swarm with six nodes are designed to pursue the reference target in a regular polygon formation, and the reference target moves in a sinusoidal fashion. At $t=20s$ during the pursuit, the cluster head node 1 and the cluster member node 4 suffer attacks and will disconnect from the other nodes in the swarm, and the remaining nodes of the swarm need to maintain their original formation and complete the formation task.

From the results \prettyreffig:nodeFailTraj and \prettyreffig:resErrNodeFail, it can be observed that the swarm selects node 2 as the cluster head to reorganize the communication topology after the node failure and successfully maintains the original formation shape after a short fluctuation. Finally, the robustness of the proposed formation control algorithm against node failure is validated. To investigate the effect of policy constraint on control performance, we compared the results of the NCES-based formation control method that impose policy constraints with the one without policy constraints. In some cases such as node failure and switching formation, there is a certain chance that the algorithm will not converge, moreover, in all cases the convergence rate is generally improved by more than 20 percents with policy constraint than without it, therefore, policy constraint is essential for the training of NCES-based neural network controller.