Micro-grid source-load storage energy minimization method based on improved competitive depth Q - network algorithm and digital twinning

Abstract

Aiming at the frequency instability caused by insufficient energy in microgrids and the low willingness of grid source and load storage to participate in optimization, a microgrid source and load storage energy minimization method based on an improved competitive deep Q network algorithm and digital twin is proposed. We have constructed a basic framework structure for the coordinated operation of source grid load and energy storage, and analyzed the modules on the power supply side, grid side, load side, and energy storage side. Under the improved competitive deep Q network algorithm, modifications were made to the energy storage of microgrid loads. Based on the processing results, the objective function for optimizing microgrid source load energy storage is constructed using digital twin technology, and the optimization of the objective function is achieved to solve the optimization objective function for microgrid source load energy storage and complete the optimization of microgrid source load energy storage. The experimental results show that this method can control the distortion rate within 5.12%, with frequency fluctuations around 50.0 Hz, and relatively good MSE, MAE, and R2 values. This method can effectively control frequency fluctuations and has a good effect on optimizing energy storage for microgrid power sources and loads.

Introduction

Microgrids, as an important way to achieve efficient, flexible, and sustainable energy systems, have attracted widespread attention worldwide in recent years. By integrating distributed power sources ( [1], energy storage systems, conversion devices, loads, and monitoring and protection systems  [2], microgrids can not only effectively utilize clean energy, but also solve grid connection problems  [3] and operate independently [4]. Improve the acceptance of renewable energy [5] reduce fossil energy consumption and greenhouse gas emissions. However, the energy management of microgrids faces many challenges, such as coordinated scheduling of energy, load, and energy storage  [6], as well as handling uncertainties in complex environments. Reference [7] proposed an AC/DC hybrid microgrid power quality optimization scheme that reduces harmonics through power filters, but this method is greatly affected by environmental factors. Reference [8] designed an energy management system based on standard communication protocols. Although it can adapt to environmental changes, communication reliability remains a challenge. To address the uncertainty in microgrids, researchers have proposed various strategies. Reference [9] constructs a load deviation index for intermittent power fluctuations in microgrids and proposes a hybrid energy storage strategy with double-layer fuzzy optimization. This strategy stabilizes the energy storage state and reduces load deviation, which is superior to single-layer fuzzy control. Reference [10] uses robust planning methods for photovoltaic microgrid energy storage to handle uncertainty, but there is still room for improvement in the precise setting of control rules and the accuracy of model predictions. Reference [11] discusses the effectiveness of real-time scheduling based on the chance constraint of system security under model load. Reference [12] is based on high penetration resource dynamic interaction regulation technology to address the impact of data latency and communication failures. Reference [13] proposed dynamic interactive adjustment of flexible resources, but its effectiveness in handling uncertainty still needs to be verified.Reference [14] states that the adaptive protection coordination algorithm can dynamically adjust the setting of relay protection, but changes in the system topology have a significant impact on it.

Among the numerous energy management methods in microgrids, deep reinforcement learning has shown broad application prospects with its powerful learning and decision-making abilities  [15,16,17]. Deep Q-network (DQN) combines deep learning and reinforcement learning to effectively solve high-dimensional decision problems  [18,19,20]. However, the exploration and utilization of traditional DQN in complex microgrids are unbalanced, resulting in low learning efficiency [21, 22]. The improved competitive DQN decomposes the value function into state value and dominant position [23, 24], improving learning efficiency and dynamic adaptability. In addition, digital twin technology has opened up new avenues for microgrid energy management. Digital twin refers to the faithful mapping of physical objects in virtual space by integrating multiple physical quantities and multi-scale simulation models, sensor update data, and historical data, and can simulate their behavior and performance in real environments  [25]. Digital twin technology creates virtual images of microgrids, synchronizes physical states and energy flows in real-time, and assists in developing efficient energy management strategies  [26]. In summary, the proposed microgrid source load energy storage minimization method based on improved competitive deep Q-network algorithm and digital twin aims to integrate the advantages of existing research, overcome its shortcomings, and provide a new efficient, flexible, and sustainable solution for energy management in microgrids. This research will not only contribute to the development of microgrid technology, but also promote the widespread application of renewable energy, providing strong support for addressing the global energy crisis.

Optimization of source-load storage energy in microgrid

Construction of the basic frame structure for coordinated operation of load and storage of source network

The proportion of renewable energy generation has increased [27], and a source grid load storage friendly interaction system has emerged [28]. It integrates source, grid, load, and storage resources, optimizes and adjusts equipment, ensures supply-demand balance, avoids losses, and has various control methods. Figure 1 shows its basic framework.

Fig. 1

Basic frame structure diagram of coordinated operation of load and storage in source network

The framework in Fig. 1 is divided into two layers: the upper layer coordinates decision-making, optimizes calculations using real-time data, and sends the results to lower level devices; Real time regulation at the lower level to achieve power dispatch.

The four parts of the source network load storage communicate bidirectionally with the coordination system, uploading status information and executing instructions to achieve global control. Electricity comes from renewable and traditional energy sources  [29]; The network includes flexible transmission and distribution equipment; The load is divided into two categories: rigid and flexible, with adjustable flexibility; Common energy storage methods include lead-acid and lithium batteries  [30].

Mains side

(1) Thermal power generation

Thermal power remains the mainstay, supporting flexible regulation of the power grid  [31].$Q_{{IT}}^{max}$ .$Q_{{IT}}^{min}$  is the maximum and minimum power of thermal power, $Q_{{IT}}^{m}$ is the output power of unit  during the period, and the maximum value is:

$$Q_{{IT}}^{+}=\hbox{min} \left( {Q_{{IT}}^{{\hbox{max} }} - Q_{{IT}}^{\operatorname{m} }} \right) \times F \times \Delta t$$

(1)

$$Q_{{IT}}^{ - }=\hbox{min} \left( {Q_{{IT}}^{\operatorname{m} } - Q_{{IT}}^{{\hbox{min} }}} \right) \times E \times \Delta t$$

(2)

 $Q_{{IT}}^{+}$. $Q_{{IT}}^{-}$is the maximum upward and downward adjustment of I thermal power during T period,  F and E are the maximum climbing and landslide rates, and $\triangle t$is the time unit. The output of coal-fired power plants is limited by the increase in climbing rate and the decrease in landslide rate.

Thermal power generation not only undertakes the main task of power supply, but also has an important function of supporting flexible regulation of the power grid. During operation, the output power can be adjusted according to demand to achieve the maximum and minimum power output, ensuring the stability of the power grid and the reliability of power supply.

Therefore, the maximum adjustment value is:

$${Q_{IT}}=Q_{{IT}}^{{\hbox{max} }} - Q_{{IT}}^{{\hbox{min} }}$$

(3)

If the equipment allows, a wide adjustment range leads to high accuracy and ensures system stability.

(2) Photovoltaic power generation

The power generation is obtained by multiplying the photovoltaic panel area S by the light intensity $L_{{x}}$ :

$$Q_{{IT}}^{q}={Q_{IT}}+\left( {Q_{{IT}}^{+}+Q_{{IT}}^{ - }} \right) \times S \times {L_x} \times \lambda$$

(4)

 $Q_{{IT}}^{q}$ is the power of the I photovoltaic unit during the T period, and $\lambda$ is the power generation efficiency.

The prediction of photovoltaic power generation, with the help of a model that integrates Bi LSTM (Bidirectional Long Short Term Memory Network) and attention mechanism, can deeply analyze the changes in light intensity in time series data. By capturing historical photovoltaic power, weather information, and time characteristics, and emphasizing the importance of key time points with attention mechanism  [32], the maximum flexible and adjustable photovoltaic capacity can be scientifically planned to optimize power resource allocation and ensure stable operation of the power grid  [33]. The predicted photovoltaic power generation can be expressed as:

$$Q_{{IT}}^{{q - }}=Q_{{IT}}^{q} \times Q_{{IT}}^{{qv}}\left( {W\sum\limits_{{t=1}}^{T} {{\alpha _t}h_{t}^{{}}} +b} \right)$$

(5)

  $Q_{{IT}}^{qv}$ is the output I of  photovoltaic T  period, and $Q_{{IT}}^{q-}$ is the dimming output under $Q_{{IT}}^{q}$  control. W and b are the weights and biases of the linear layer, and ${\alpha _t}$ is the attention weight, representing the importance of the information at time t  to the final prediction. $h_{t}^{{}}$ is the hidden state of Bi LSTM at time t.

(3) Wind power generation

Wind power is converted into wind energy, and the power varies with wind speed  [34]. V  is the wind speed,$V_{a}^{{}}$  is cut in, $V_{b}^{{}}$  is rated, $V_{c}^{{}}$ is cut out, and an output model is constructed:

$$Q_{T}^{w}\left( v \right)=\left\{ {\begin{array}{*{20}{c}} {0,v< {v_a},v<{v_b}} \\ {Q_{{IT}}^{q},\frac{{v-{v_a}}}{{{v_b}-{v_a}}},{v_a}< v<{v_c}} \\ {Q_{{IT}}^{q},{v_c}< v< {v_b}} \end{array}} \right.$$

(6)

 $Q_{T}^{{w }}(v)$is the active power of wind power in phase T,$Q_{IT}^{{v}}$ and is the rated power generation of the wind turbine.

Although wind and photovoltaic power generation are limited by natural conditions, with the support of advanced control technology and smart grids, they can flexibly respond to the instability of output power through fuzzy decision systems, achieving maximum utilization. Through real-time data analysis and fuzzy rule inference, the system can intelligently adjust the output of power plants, ensuring that the power system maintains supply-demand balance and stable operation during fluctuations in renewable energy  [35], reflecting the potential similarity and practical application value of the two in improving the flexibility of the power system.which is:

$$Q_{T}^{{wt - }}=Q_{T}^{{wt}} \times Q_{T}^{w}\left( v \right)$$

(7)

$Q_{T}^{{wt }}$ is the output of the I wind turbine during the T period, and $Q_{T}^{{wt - }}$ is the lowest decrease to $Q_{T}^{{wt }}$ .

Power grid side

Connecting lines to bridge the electrical system, scheduling flexible operating states, limiting power and network structure. The transmission capacity is expressed in power, as follows:

$$Q_{{ITJ}}^{{ABC}} \in \left[ { - Q_{{\hbox{max} }}^{{ABC}},Q_{{\hbox{max} }}^{{ABC}}} \right]$$

(8)

$Q_{{ITJ}}^{{ABC}}$ is the output power from  I to  J during the T period, and $Q_{{max}}^{{ABC}}$  is the maximum carrying power or capacity of the line.

Although the structure and lines of the power grid are not flexible resources, they affect scheduling  [36]. Unreasonable transmission increases scheduling difficulty and cost. Reasonably design the power grid, enhance the capacity of interconnection lines, and promote the utilization of power system resources.

Load side

There are two types of loads: rigid and flexible. Rigid is difficult to move, while flexible is easy. Flexibility is like a residential building, with large adjustment space and benefits for peak shaving and green energy consumption. Rigidity is like industry, and continuity is difficult to transfer.

Both rigid and flexible loads can be guided to transfer, but the adjustable proportion of flexibility is difficult to measure. Big data analysis helps transfer, but errors still disrupt scheduling modeling. The maximum adjustable amount during period T can be expressed as:

$$Q_{T}^{{K - }}=\gamma \times \left( {Q_{T}^{{GX}}+Q_{T}^{{RX}}} \right)$$

(9)

In the formula, $Q_{T}^{{K - }}$ means that it can only be adjusted downward by $Q_{T}^{{K - }}=\gamma \times \left( {Q_{T}^{{GX}}+Q_{T}^{{RX}}} \right)$  ; $\gamma$ represents $Q_{{T}}^{GX}$ the $Q_{{T}}^{RX}$load transfer coefficient; and respectively represent the amount of rigid load and flexible load in  T period.

Energy storage side

Energy storage is beneficial for flattening the curve, and the cost depends on the system. Flexible resource complementarity and rational deployment of energy storage. Flexible adjustment of charging and discharging;

$$Q_{T}^{D} \in \left[ {Q_{T}^{{D,\hbox{min} }},Q_{T}^{{D,\hbox{max} }}} \right],Q_{T}^{D} \leqslant S_{T}^{{\hbox{max} }} - S_{T}^{\operatorname{m} }$$

(10)

$$Q_{T}^{R} \in \left[ {Q_{T}^{{R,\hbox{min} }},Q_{T}^{{R,\hbox{max} }}} \right],Q_{T}^{R} \leqslant S_{T}^{\operatorname{m} } - S_{T}^{{\hbox{min} }}$$

(11)

 $Q_{{T}}^{D}$ and $Q_{{T}}^{R}$ are the discharge power of T , $Q_{{T}}^{D,max}$ is the upper limit of $Q_{{T}}^{D,min}$ and the lower limit of $Q_{{T}}^{R,max}$ is the upper limit of $Q_{{T}}^{R,min}$ . $S_{{T}}^{max}$ and $S_{{T}}^{min}$ are the upper and lower charge limits, and $S_{{T}}^{m}$ is the current charge.

Modification of microgrid load storage energy under improved competitive depth Q-network algorithm

Competitive deep Q-network optimization of microgrids, adjusting load energy storage through reinforcement learning to improve energy efficiency and system stability.

Improving the competitive deep Q-network by integrating CNN and Q-learning, intelligent decision-making for microgrid source load energy storage, with CNN assisting in optimization.CNN takes initial image data as input, extracts features through multi-layer convolution and pooling operation, and finally outputs the obtained feature items. These characteristics can reflect the state information in the microgrid system, such as energy supply and load demand, and serve as the basis for decision-making. Q-learning is based on MDP and includes states, actions, and rewards. In microgrid applications, the state includes supply and demand, energy storage; Adjust the supply and use of energy storage. The reward is based on the performance indicators of microgrid, such as energy efficiency and economic cost. Improve Q-Net with reinforcement learning modeling and optimization  [37], and use deep learning to solve state strategies. Retrieve data from the environment, input it into the Q network, periodically replicate the main Q to the target Q, update the main network with the backpropagation loss function, and iterate until convergence.

Improve the Q-network by fitting the Q-value with CNN  [38], optimize with gradient descent, select the optimal action with the same target Q-value as the fitting function, and obtain the results:

$$Y=\hbox{max} Q \times \left( {Q_{T}^{D}+Q_{T}^{R}} \right)$$

(12)

In this process, $Q$ is not updated, which will lead to a higher estimated value of $Q$ . If the estimated value is too high, all potential decisions will be inconsistent, and the decision will be selected to the suboptimal solution. As for the reinforcement learning task of continuous location in state space, the limited learning samples can’t make the neural network fit the Q-value function that adapts to all state action pairs, so the fitted Q-value function curve will float around the actual Q-value curve, because the target Q-value function needs to be selected to achieve the maximum action, and the Q-value estimated by the network will be higher than the actual Q-value.

Microgrid load correction requires preprocessing of historical data to eliminate interference and ensure accuracy. Distingter obtaining mutation load data, correction is anuishing between normal and mutation data, the load data undergoes drastic changes and its continuity is disrupted, resulting in the load data being:

$$\left\{ \begin{gathered} \frac{{F\left( {x,y} \right) \times Y}}{{F\left( {x,y - 1} \right)}}> {H_J},y \ne 1 \hfill \\ \frac{{F\left( {x,y - 1} \right)}}{{F\left( {x,y} \right) \times Y}}> {H_J},y=1 \hfill \\ \end{gathered} \right.$$

(13)

F(x,y) is the data of x at y, $H_J$  is the threshold set to determine the rate of change. After obtaining mutation load data, correction is an important step to ensure the accuracy and reliability of the data, thereby supporting more precise decision-making. Linking this correction process with previous discussions on fuzzy decision-making, using the corrected data as part of the input for the fuzzy decision-making system to improve the system’s responsiveness to dynamic changes in the power system. Firstly, obtain sudden load data from sensors or data recording systems, perform preliminary cleaning on the data, and remove obvious errors or invalid data points. Secondly, based on continuity, set the distance interval between the previous and next time points for each data point as the corrected value for that point. Finally, the corrected mutation load data is used as an input variable for the fuzzy decision system, and the membership degree of each set is calculated based on the corrected data values. By following the above steps, improve the responsiveness of the power system to dynamic changes and the accuracy of decision-making. The formula is:

$${X_Z}=\frac{{f\left( {x,y} \right)+F^{\prime}\left( {x,y} \right)}}{2}$$

(14)

$F^{\prime}( x,y)$ is to correct the load. The load storage energy correction processing of microgrid uses historical data and other microgrid operation parameters to predict the load data of microgrid at a certain time in the future under the premise of training and learning, so it is necessary to obtain effective data through preprocessing, which is a normative condition that must be met in the load storage energy correction processing of microgrid and lays a foundation for constructing the objective function.

In the process of optimizing the energy storage strategy for microgrid loads, the key parameters of the Q network were adjusted by integrating CNN and DQN. The learning rate is set to a small initial value to balance the learning speed and stability of the algorithm. Meanwhile, the discount coefficient is selected as a value close to but less than 1 to ensure that the algorithm considers long-term energy efficiency and system stability while pursuing immediate supply-demand balance. In addition, attention was paid to parameters such as batch size, iteration times, and optimizer selection, which were validated through multiple experiments to find the most suitable parameter combination for the current task, thereby improving the performance of the entire system.

Construction of energy optimization objective function of micro-grid source, load and storage under digital twinning

Digital twins are crucial in optimizing load energy storage in microgrids. By incorporating optimized data into wind power models, load forecasting can aid in decision-making. Figure 2 shows the wind power output model based on twin technology.****

Fig. 2

Structure diagram of wind power output model based on digital twinning technology

Figure 2 shows the real-time reflection of the real system state by the digital twin model, which is calibrated using real-time data from the microgrid.

Step 1: Dynamic assimilation. During the simulation process, the status and parameters of the digital twin model are updated in real-time to reflect the actual changes in the physical object. Simultaneously conducting operational simulations, utilizing the constructed digital twin model for simulation experiments, to simulate the behavior of physical objects under different conditions.

Step 2: Model modification. Based on the results of running simulations and dynamic assimilation, make necessary modifications and optimizations to the digital twin model.

Step 3: Iterative optimization. Repeat the process of running simulations, dynamic assimilation, and model modification to form a closed-loop feedback mechanism.

Step 4: Perceive and predict load. Optimize the framework adjustment strategy and optimize the load energy storage scheme.

Step 5: Synchronize data feedback to improve the accuracy and reliability of predictions.

Construct a digital twin optimal microgrid load energy storage objective function  [39], with a technical framework as shown in Fig. 3, to improve energy efficiency and meet load demand.

Fig. 3

Technical framework for energy optimization of microgrid source, load and storage under digital twinning

From Fig. 3, it can be seen that the LSTM model (especially the Bi LSTM combined with attention mechanism), as a prediction tool, improves the accuracy of predicting photovoltaic power generation and load demand by integrating and fusing the prediction results of different models. These predicted results are then used to construct and optimize the objective function, and through techniques such as multi-agent coordinated optimization models, remote virtual interaction, and real-time synchronous control, efficient energy management and load satisfaction of microgrids are achieved.

Digital twin connected source negative energy storage optimization and intelligent synchronization  [40], simulating and controlling microgrid operation. The key is to establish coordination strategies and obtain excellent cases through letter writing.

Sensor data may have delays, assimilation, or occurrence errors. Construct twin dataset vectors, correct load forecasting errors, and calculate as follows:

$${O_{YW}}=\left( {{\eta _1},{\eta _2}, \cdots ,{\eta _n}} \right) \times {X_Z}$$

(15)

${O_{YW}}$ is the load forecasting error, and ${\eta_{n}}$ is the weighted system of the -th twin data. Calculate the load value with the error using the above formula, which is as follows:

$${\chi _S}=\frac{{{M_V}}}{{Z{}_{l} - {Z_r}}} \times {Z_e} \times {O_{YW}}$$

(16)

${x_{s}}$ is the hourly power generation load, ${M_{V}}$  is the output compensation of the unit, ${Z_{l}}$  is the daily real value, ${Z_{e}}$

According to the detection of power disturbance in the photovoltaic microgrid, an objective function for optimizing the distribution of power load energy can be constructed, which can be divided into two layers: internal and external. The external model aims to minimize the total cost of microgrids during the planning period, including equipment investment and economic operating costs. The economic cost involves light storage, transportation, maintenance, and network interaction; Equipment investment refers to the investment in photovoltaic power generation. The formula for the objective function of the external model is:

$${M_{md}}={A_u} \times \zeta \times {g_w} \times {\chi _S}$$

(17)

${A_{u}}$ is the cost of Lianyou equipment, $\zeta$ is the planning year, and ${g_{w}}$ is the inflation rate.

External model constraints include energy storage and photovoltaic assembly machines.

The internal model takes into account power disturbances and energy storage losses, with the aim of minimizing annual operating costs and constructing formulas:

$${M_{nd}}{\text{=}}{B_{sd}} \times {M_{md}} \times \varpi \times {v_g}$$

(18)

In the formula, ${B_{sd}}$  represents the weight coefficient corresponding to the energy storage loss cost; $\varpi$  represents the interaction cost between external public power grid and optical storage microgrid; ${v_g}$ represents the actual operation and cost.

Cross side power balance and photovoltaic output are internal model target constraints.

Objective function solution

Using the goal letter of improving competitive Q network and twin technology optimization, solve the problem of optimizing the allocation of micro network sources and negative storage.

Retain group interaction, self inertia, high precision, strong optimization, and fast convergence. The solving process is as follows:

Step 1: Set algorithm parameters: file set, initial value, weight, iteration times, total number.

Step 2: Optimal particle swarm optimization achieves the best field of view, center, and group velocity position.

Step 3: Judge the dominant relationship in the initialized population.

Step 4: Liberating the non-inferiority in the archive set in the initialized population;

Step 5: Select the optimal particles, update the new generation population by the following formula, and use the comparison result to generate the new generation population.

$$V\left( {{y_{j,o+1}}} \right){\text{=}}\left\{ \begin{gathered} \begin{array}{*{20}{c}} {{L_{j,o}}+{V_{j,o+1}}}&{\frac{{{W_{v,o}}}}{{{m_m}}}>\varepsilon } \end{array} \hfill \\ \begin{array}{*{20}{c}} {{E_{rey}}\left( {{y_{j,o+1}}} \right)}&{else} \end{array} \hfill \\ \end{gathered} \right.$$

(19)

In the formula, $V\left( {{y_{j,o+1}}} \right)$  stands for clustering and foraging behavior, Clustering and foraging behavior work together in optimization algorithms to guide particles towards the optimal solution. Clustering behavior helps particles form high-density regions in the search space, accelerating convergence speed; Foraging behavior helps particles discover new and potentially better solutions by exploring and utilizing mechanisms.${L_{j,o}}$ stands for the o fish; ${V_{j,o+1}}$ represents the velocity of particle j  in the o+1 iteration; ${m_{m}}$  represents the visual field of the o fish; $\varpi$  stands for inertia weight factor; ${{E_{rey}}\left( {{y_{j,o+1}}} \right)}$  stands for foraging and random behavior; ${W_{v,o}}$ stands for rear-end collision and foraging behavior.

In the process of solving the objective function, clustering and foraging behavior are reflected through particle swarm optimization (PSO) algorithm. Although not directly implemented as a clustering algorithm, particles naturally cluster in the solution space due to the attraction of local and global optimal solutions, forming potential favorable search areas. The foraging behavior is simulated by updating the formula based on the velocity and position of particles, allowing them to move towards a more optimal solution. Clustering and foraging behavior complement each other, jointly driving the algorithm to converge towards the global optimal solution.

Step 6: judge the dominant relationship of the new generation population and whether it is within the restricted range, so as to generate a new generation of non-inferior solutions. Putting a new generation of non-inferior solutions in the archive set means judging whether the archive set needs to be deleted. When deletion is not needed, step 7 is executed; When deletion is needed, step 6 is executed to maintain the archive set.

If the maximum iteration algorithm is reached, increase the number of iterations to step 4. Evaluate and verify the solution, and make improvements based on the actual situation. Based on the transformation of competitive Q network and twin, it has become a micro network source with excellent negative energy storage.

In summary, the optimization process of microgrid source load energy storage is shown in Fig. 4.

Fig. 4

Optimization process of microgrid source load energy storage

Experimental analysis

Verify and improve the twin energy subtraction of Q-Net and Microgrid, and simulate it using TensorFlow and SciPy libraries in Python. Select a typical microgrid model with multi-source load energy storage. Measure the power, distortion rate, and frequency variation of the connecting line, and optimize the energy storage of the power load. See Table 1 for energy storage.

Table 1 Parameter selection of source-load energy storage system in microgrid

The experimental process is as follows:

Step 1: Install the initial microgrid model and set parameters for energy, load, and energy storage.

Step 2: Modify the competitive Q website and the twin optimization process.

Step 3, After the initial state parameter is tested, the optimal time for the Q-network is changed.According to the predicted action, the cooperative scheduling strategy of energy and energy storage devices is calculated and the model state is updated; According to the model state and the actual energy supply and demand situation, the experimental indexes are calculated and stored; According to the current state and feedback signal, the parameters of the improved competition depth Q-network model are updated.

Step 4: Record the experimental indexes in each training process, and judge whether the experiment is finished according to the set stopping criteria.

Step 5: Analyze and compare the experimental results obtained by different optimization methods, and evaluate their performance and effect.

Collect the number of microgrids, compare and improve the twin method and literature method of competing Q-networks, analyze the data, and evaluate the performance differences.

During the experiment, the parameter settings of the proposed algorithm are shown in Table 2.

Table 2 Experimental parameter settings

Using a 10 kV local microgrid as a test, set it up as shown in Fig. 5.

Fig. 5

Experimental object microgrid architecture diagram

Contact line power conversion shadow microgrid source negative storage. Increasing leads to reducing load and promoting energy storage and charging; Reduce or cut off, relying on internal supply. Summer and winter optimization, as shown in Fig. 6.

Fig. 6

(a) Summer  (b) Winter   Variation results of tie line power in summer and winter after optimization by different methods

From Fig. 6, it can be seen that after optimization, the maximum power of the summer line is 4 kW, and the maximum power of the winter line is 6 kW. After optimization, the fluctuation value significantly increased and the standard deviation significantly decreased, especially for this method, which fully verified its superiority. This is because the improved competitive deep Q network algorithm combines deep learning and reinforcement learning to intelligently make optimal energy storage and load management decisions based on the real-time status of microgrids. This intelligent decision-making capability enables microgrids to maintain high operational efficiency and stability under different seasons and load conditions.

Optimize the original formula and adjust it according to the distortion rate constraint. The calculation formula is:

$${f_s}=\frac{{{D_S} \times {S_S}}}{{{Z_S} \times {W_S}}} \times 100\%$$

(20)

The formula includes ${D_S}$ frequency, ${S_S}$ variable capacity, ${Z_S}$ negative capacity, and ${W_S}$ source storage capacity. Calculate harmonic under standard conditions to improve harmonic control efficiency. Extremely low, high quality, excellent sex. Table 3 shows the false accuracy rate under the comparison method.

Table 3 Experimental comparison results of distortion rate under different methods

From Table 3, it can be seen that the distortion rates of all four methods increase with the increase of experimental times. When the number of experiments reaches 600, the distortion rate of the proposed method is within 5.12%, which is relatively low. This is because the proposed method constructs a basic framework structure for the coordinated operation of source grid load and energy storage, and analyzes in detail the modules of the power supply side, grid side, load side, and energy storage side. Digital twin technology can ensure the accuracy and reliability of the model. Not only did it consider the optimization of energy storage, but it also took into account the coordinated operation of the power supply side, grid side, and load side. This comprehensive optimization strategy helps to achieve overall performance improvement of microgrids. Therefore, the proposed method can ensure a lower distortion rate.

Using this method to optimize microgrid source negative storage and specific frequency variation. As shown in Fig. 7.

Fig. 7

Frequency change results before and after adopting the proposed method

From the analysis of Fig. 7, it can be seen that when the research object is subjected to power interference, its frequency will fluctuate. The larger the fluctuation amplitude, the worse the frequency stability, which will have adverse effects on the normal operation of the power system. After using this method for optimization, the frequency fluctuation of the research object is effectively controlled, and the rated frequency remains stable at around 50.0 Hz. This is because the real-time feedback mechanism provided by digital twin technology enables algorithms to adjust their strategies in a timely manner to cope with frequency fluctuations caused by power interference. By continuously optimizing the objective function, the algorithm can gradually find the most suitable control strategy for the current operating conditions, thereby maintaining a stable frequency of around 50.0 Hz.

The optimization effects of different methods were validated using MSE, MAE, and R2 as evaluation metrics. Optimize the average difference by calculating MAE. The smaller the MAE, the stronger the recognition ability. R2 (R-squared, coefficient of determination) has a value range of 0 to 1. The closer it is to 1, the better the explanatory power of the data. The results are shown in Table 4.

Table 4 Index results of different methods

According to Table 4 analysis, the MSE of the proposed method is 0.11, MAE is 0.09, and r2 is 0.97. This indicates that the proposed method constructs the objective function for optimizing microgrid source load energy storage based on the processing results under digital twin technology. This objective function can quantitatively evaluate the system performance under different strategies, which enables the algorithm to undergo multiple iterations and optimizations in a virtual environment without the need for trial and error in actual systems, greatly reducing experimental costs and risks. At the same time, the real-time feedback mechanism provided by digital twin technology enables algorithms to adjust strategies in a timely manner, thereby improving overall performance and enhancing the accuracy and stability of prediction and optimization.

Conclusion and prospect

Conclusion

An innovative microgrid energy optimization strategy has been proposed, which integrates an improved CDQN algorithm and digital twin technology. This strategy not only verifies its excellent performance, such as low dropout rate, strong power output, and effective utilization of solid-state power negative storage, but also successfully controls frequency fluctuations and stabilizes at 50 Hz, significantly improving the stability and reliability of the power supply. Looking ahead, the actual deployment of this strategy needs to focus on aspects such as network security and economic cost analysis, and continue to conduct research on adaptive learning and multi-objective optimization. At the same time, promoting the standardization process, establishing an ecological cooperation system, and implementing demonstration projects and case studies will drive the continuous progress and development of microgrid technology.

Prospect

Research limitations: CDQN training consumes resources and large systems will face challenges.The digital twin technology still needs to be further improved and optimized in model construction and parameter adjustment. Future research can consider combining other optimization methods and technologies to further improve the energy scheduling performance of microgrid. In the future, distributed reinforcement learning and parallel computing in deep reinforcement learning are used to improve training efficiency and system response speed. In the aspect of model construction and parameter adjustment, the data collection of real system and the verification of model accuracy are strengthened. At the same time, the application scope and feasibility of digital twin technology in microgrid system are further studied, and more suitable modeling methods and parameter adjustment strategies are explored to improve the prediction accuracy and practicability of digital twin model.In addition, as research progresses, it is expected that more testing scenarios will be included in the scope of the study. These test scenarios will cover various situations such as different seasons, load types, and energy structures to comprehensively evaluate the applicability and robustness of the proposed method. By testing and optimizing in different scenarios, the practicality and reliability of the method can be further improved.