Prospect Theory-inspired Automated P2P Energy Trading with Q-learning-based Dynamic Pricing

Peer-to-peer (P2P) energy trading is a recently proposed decentralized modality for energy sharing aiming at solving limitations of centralized techniques. This modality has been gaining significant traction recently [27, 26]. Specifically, P2P energy trading allows prosumers to trade energy among each other at a negotiated price with or without the involvement of the grid [26]. It generates better monetary incentives for prosumers compared to existing mechanisms while also reducing their grid dependency [27]. Additionally, increased local energy generation/consumption resulting from P2P trading leads to the minimization of overall system energy loss while providing an effective way to achieve demand side management[32]. Benefits extend also to the grid operator, by providing savings in investments that would have been otherwise required to develop/maintain transmission infrastructure in a centralized power distribution architecture [16, 26].

P2P energy trading has received attention from the research community in recent years. The works in [29, 28] present game theoretic approaches in a P2P setting, while a greedy rule-based P2P mechanism to assign energy among prosumers is proposed in [3] that includes mid-market pricing. Similarly, the physical aspects of P2P energy trading, such as power loss minimization and voltage regulation, have been explored in [15, 17]. These works, however, largely overlook the user behavior in designing their solutions. As established in [16, 27, 25], accommodating the user behavioral modeling in P2P energy trading ensures sustained participation from prosumers while incentivizing their contribution. In fact, the papers [29, 28] consider prosumers to be actively involved and fully compliant with the system as rational decision-makers. First concern with this assumption is that the continuous online presence of participating prosumers with the system might not always be possible in real-world application. Secondly, research on user behavioral models and decision making [11, 2] have found users to have bounded rationality. Therefore, requiring constant active participation overwhelms the users and incentivizes non-rational decisions [7]. In the worst case, it might even result in users opting to terminate their participation altogether [2, 25]. In that light, the works in [25, 1] incorporates bounded rationality and user preferences into P2P energy trading. However, it requires continuous human participation and assumes a simplistic linear model for user perception. Conversely, the authors of [27] limit their focus on coalition formation in game theoretic setting and do not explicitly consider user behavioral modeling.

As a result, a prosumer-centric P2P energy trading model, that effectively incorporates the prosumers’ decision-making behavior and their perceived loss/gain value from trading, is still lacking in the existing literature. Such a trading modality is expected to require minimal active participation from users while also ensuring their sustained involvement through the adoption of user behavioral modeling. To this end, the framework of Prospect Theory (PT) [13] can be used to model the non-rational user behavior in the face of uncertain decision-making. It is often regarded as fairly accurate mathematical representation of human behavior [13, 9, 8].

Recently, there has been few efforts in integrating PT in energy related applications as well to capture the irrationality of users [20, 8, 30, 31]. In relation to P2P energy trading, the authors in [31] have proposed a PT-based distributed energy trading model to optimize trading decisions for prosumers in a competitive market. Although these papers model the user behavior in some ways, they require active participation from users and also assume that such behavior (e.g., the parameters of PT) is homogeneous for all the users. Social science studies, such as the one conducted in Italy [6] to investigate the social acceptance of nuclear energy using an online survey, show that users exhibit significant heterogeneity in their preferences for the sources of energy. Neuroscience studies have also stressed the heterogeneity of humans in reference to PT parameters [10]. Not capturing such heterogeneity provides little benefits in terms of user behavioral modeling.

In this paper, we design a PT-based optimization framework for prosumer-centric P2P energy trading as shown in Fig. 1. The framework aims at matching energy production and consumption (step $1$ in Fig. 1) to maximize the perceived utility of individual buyers while taking into account the intrinsic heterogeneity of human perception. Given that the optimization problem is non-linear and non-convex, we further devise a Differential Evolution-based [23] metaheuristic algorithm called $DEbATE$ to solve the problem (energy allocation, step $2$). In order to ensure minimal active participation of prosumers, we employ a Reinforcement Learning (RL) framework, called $PQR$, in tandem with $DEbATE$ to automate the pricing mechanism for sellers (pricing mechanism, step $3$). In doing so, $PQR$ learns the selling price for each sellers using a PT-based risk-sensitive Q-learning algorithm [21]. The output of the algorithms is then returned to the prosumers for executing the physical energy transactions (step $4$). Using real datasets for energy production and consumption, paired with recent survey data for PT perception modeling, results show that $DEbATE$ performs $25\%$ higher in buyer’s perception and $7\%$ higher in seller’s reward compared to state-of-the-art approach.

The major contributions of the paper are the following:

• We develop a PT-inspired optimization framework for P2P energy trading;

• We design a metaheuristic algorithm $DEbATE$ to solve the non-linear energy allocation problem;

• We design dynamic pricing mechanism with $PQR$ algorithm using risk-sensitive Q-learning approach;

• Experiments using real data show the superiority of proposed approach compared to the state-of-the-art;

$DEbATE$ is executed at each trading period (e.g., 12 hours) to solve the non-linear optimization problem in Eq. (2). It uses differential evolution to determine an optimal amount of energy to be traded between prosumers that maximizes the perceived utility of buyers. DEbATE initially updates the list of buyers ($B$) and sellers ($S$) based on the expected production and consumption for current trading period. These can be predicted accurately with recent approaches [14, 5]. The differential evolution-based optimization begins on line $2$ where an initial population $X$ is generated with population size of $NP$. An element $x_k\in X$, with $k=1,2,\dots ,NP$ is a candidate solution vector of variables $x_{ij}$ representing the amount of energy to be traded between each seller $i$ and buyer $j$ . These variables correspond to the decision variables of our optimization problem.

The $while-$loop (line $3-19$) is the differential evolution loop that aims at finding solution to the non-linear optimization problem with Eq. (2) as the fitness function. The loop is executed for $G_{max}$ iterations. At each iteration, for each candidate solution $x_k\in X$, the algorithm creates a mutated solution ${\overline{x}}_k$. Initially, ${\overline{x}}_k=x_k$. The mutated solution is subsequently updated through mutation and crossover with $3$ random candidates $x_a,x_b,x_c\in X$ (line $5$). A value $R\in [1,|S|\times |B|]$ is selected at random. $R$ will be used in the following $for-$loop to ensure a minimum mutation. The for loop in line $7$ iterates over the components (dimensions in evolutionary terms) of ${\overline{x}}_k$. During each iteration, a value $u\in [0,1]$ is sampled at random as mutation probability (line $8$). Subsequently, a mutation occurs for the component $ij$ of ${\overline{x}}_k$ with crossover probability $CR$ (line $9$). The mutation occurs irrespective of the probability if $(i\times j)=R$ (to ensure at least one minimum mutation). A mutation is executed by combining the corresponding component of $x_a$, $x_b$, and $x_c$ with the differential weight parameter $F\in [0,2]$ as in line $10$. The mutated component ${\overline{x}}_{ij}^{\left(k\right)}$ is clipped to ensure that it falls within $[0,1]$ as minimum and maximum threshold to satisfy constraint Eq. (2e) in line $11$ of the algorithm.

After the mutated solution is finalized, it is checked, and adjusted if needed, to meet the constraints in Eqs. (2a)-(2c) of the optimization problem. Specifically, line $13$ ensures that no exchange occurs (i.e., ${\overline{x}}_{ij}^{\left(k\right)}=0$) between users having a loss higher than $l_{max}$. Lines $14-15$ ensure that the production of a seller and the demand of each buyer are not exceeded, respectively. Finally, in line $16$, the fitness function $f(.)$ of the mutated solution ${\overline{x}}_k$ is compared against the original candidate solution $x_k$. If $f\left({\overline{x}}_k\right)>f\left(x_k\right)$, then ${\overline{x}}_k$ replaces $x_k$ in the set of candidate solutions $X$. At the end of the while loop, $DEbATE$ selects the best solution $x^{\ast }$ in $X$ (line $20)$ and executes the transactions accordingly (line $21$). In the following theorem 1, we show that the $DEbATE$ has polynomial complexity and hence, computationally efficient.

After determining the solution to the energy allocation problem in $DEbATE$, the selling price for sellers is then updated through the $PQR$ algorithm. In order to learn the optimal selling price dynamically over time, we model the sellers as independent learning agents. Note that, to preserve the privacy and avoid the conflict between prosumers, these agents do not have access to information about other sellers or buyers. The state space in the Q-learning formulation consists of the prices between the grid buying (${\rho }_{gb}$) and selling (${\rho }_{gs}$), discretized by a step size, $\delta$, i.e., ${\rho }_i\in \{{\rho }_{gb},{\rho }_{gb}+\delta ,{\rho }_{gb}+2\delta ,...,{\rho }_{gb}+(\frac{{\rho }_{gs}-{\rho }_{gb}}{\delta }-1)\delta ,{\rho }_{gs}\}.$

The action space consists of a price increasing action, price decreasing action, and no change action, i.e. $a\in \{+\delta ,-\delta ,0\}$, where $\delta$ is the amount by which price is increased or decreased. Seller $i$ reward function is the total revenue generated at the current trading period i.e. $R_i=({\rho }_i+a){\sum }_{j\in B}{x}_{ij}{w}_j$. For updating Q-values, we modify the approach proposed in [21] by considering the following Q-learning update rule that includes the PT-based perceived utility of sellers.

where, $y_i=R_i+\gamma {max}_{a}Q(s_{new},a)-Q(s,a)$ is the Temporal Difference (TD) error of $i^{th}$ seller for current iteration, and $v\left(y_i\right)$ is transformation of TD error to capture each seller’s personalized perceived utility on loss and gain. $\alpha$ refers to the learning rate for updating Q-values in Eq. (3). The action is selected based on an $ϵ$-greedy exploration-exploitation strategy [24]. Specifically, $ϵ$ refers to the probability of exploration and it is initially set to $1$. It is then decreased over time using an $ϵ-$decay value, as the system learns the optimal policy. Based on the selected action, the new selling price, reward, and Q-value are updated as per Eqs. (3) and (4). Updated selling price is then sent to the respective seller $i$ for next trading period.

The system runs both $DEbATE$ and $PQR$ sequentially at every trading period. Input of $DEbATE$ is updated based on the prices computed by $PQR$. $PQR$ then takes as input the reward from executing energy transactions by $DEbATE$.

In this section, we evaluate the performance of DEbATE and PQR, hereafter jointly referred as $DEbATE-PQR$, against a recent state-of-the-art approach referred to as Rule [3]. $Rule$ allocates energy using a greedy heuristic that assigns cheapest sellers to buyers based on their registration order in the system, while final price of each transaction follows mid-market pricing, i.e., mid value of seller’s and buyer’s asking price. We consider a system with $40$ prosumers, split evenly as buyers and sellers. This is considered a representative number of prosumers in a microgrid or set of houses supplied by a single distribution transformer. We use a realistic dataset for buyers’ energy consumption obtained from [18]. Similarly, we consider sellers equipped with $4kW$ rooftop solar located in Lexington, Kentucky, USA. The energy generated is estimated using NREL’s PVWatts Calculator [22] given the solar irradiance in Lexington and size of solar panels. Losses are assigned uniformly at random from set $\{1\%,2\%,3\%,4\%\}$ and maximum loss threshold $L_{max}=2.5\%$.

We assume that prosumers complete a survey before joining the system to estimate their individual prospect theory parameters, similar to [19, 4, 10], and use realistic prospect theory parameters determined by them. Specifically, we sample the risk-averting parameter for gains $\left({\zeta }_{+}\right)\in [0.60,0.88]$, the risk-seeking parameter for losses $\left({\zeta }_{-}\right)\in [0.52,1.0]$, the loss-aversion parameters for gain and loss $\left(k_{+}\right),\left(k_{-}\right)\in [2.10,2.61]$ for each individual prosumers. The grid energy buying price is set to ${\rho }_{gb}=\$0.06$ and the selling price to ${\rho }_{gs}=\$0.12$. The reference price for each sellers is initially randomly sampled from range $[0.09,0.12]$. It is then updated using $PQR$ at each iteration. The reference price for each buyer is selected in the range $[0.06,0.10]$ and considered static for the duration of experiments, which is $365$ days. The parameters for $PQR$ algorithm are set as follows: learning rate $\alpha ={10}^{-4}$, step size for discretizing state space $\delta =\$0.001$, and $ϵ-$decay $=0.965$.

We consider several experimental scenarios and performance metrics, as discussed in the following.

Experimental Scenario 1: We first run experiments to study the convergence of DEbATE. We considered different system size by scaling the number of sellers and buyers. Fig. 2 shows the normalized objective value as a function of the number of iterations using a population size $NP=20$. The plot averaged over 10 runs shows that $10,000$ iterations are sufficient for the algorithm to converge in the considered settings. As a result, in the following scenarios we set $G_{max}=10,000$ and the population size $NP=20$.

Experimental Scenario 2: In the second experimental scenario we study the performance of the considered approaches over time. Two performance metrics are considered, namely the buyers’ objective value and the sellers’ cumulative reward. These are represented in Figs. 5 and 5, respectively, with a moving average of $10$ days. In this experiments we consider $15$ buyers and $15$ sellers. The benefits of $DEbATE-PQR$ over $Rule$ are more prominent from April through October, when the energy demand and production is higher. The greedy nature of $Rule$ penalizes the quality of the resulting matching, significantly reducing the buyers’ perceived value. Note that, the buyers’ objective values are negative because they are paying higher prices than their reference purchase price. Therefore, transactions are seen as loss from a prospect theory perspective. Nevertheless, our approach optimizes the energy assignment to maximize the buyers perceived value. Additionally, our approach is able to generate higher rewards than $Rule$ by dynamically learning the prices for sellers through the $PQR$ algorithm. The the sellers’ reward decreases after mid-september for both the approaches due to the reduced energy production during winter.

We further study the performance over time by considering the evolution of average and individual sellers’ prices. We consider a smaller system of $5$ sellers and $5$ buyers for ease of representation of the results. Fig. 5 shows the individual prices. $DEbATE-PQR$ is able to learn and adjust the price over time to improve the buyers’ perceived value considering their competitiveness. The competitiveness is a function of a buyer’s reference price, their production, and their location in the system (e.g., loss w.r.t. sellers). As a result, our approach is able to improve the perception of both buyers and sellers while ensuring the competitiveness of the market.

Experimental Scenario 3: In this scenario we test the scalability with respect to the system size. Specifically, we increase the system proportionately from $5$ sellers and $5$ buyers to $20$ sellers and $20$ buyers. Figs. 7-7 show the buyers’ total perceived value and the sellers’ reward, respectively, over a year. By considering the loss-averse and risk-seeking PT-value functions, $DEbATE-PQR$ achieves an increasing advantage as the system size increases compared to $Rule$, for both sellers and buyers. As a numerical example, $DEbATE-PQR$ achieves as much as $26\%$ increase in buyers’ perceived value while ensuring $7\%$ profit improvement for sellers.