Abstract
Event-based optimization (EBO) provides a general framework for policy optimization in many discrete event dynamic systems where decision making is triggered by events that represent state transitions with common features. Because the number of events can be defined by the user and usually increases linearly with respect to the system scale, EBO has good potential to address large-scale problems where the state space grows exponentially. However, in many practical systems, sensors are geographically distributed and connected to a central controller through imperfect communication channels. Therefore, events observed by the sensors may not reach the controller. Optimization methods of event-based policies in these cases have not yet been identified. In this paper, we consider this important problem and make three major contributions. First, we formulate a mathematical EBO model in which the communication between sensors and controllers is subject to random packet dropping. Second, we show that this EBO model can be converted to another EBO model with perfect communication. Then, the performance difference equation and the performance derivative equation for event-based policies are straightforward to develop. One gradient-based policy iteration algorithm is developed for problems where the state transition probabilities are explicitly known, while another for problems where they are unknown. Third, the performance of the algorithms and the impact of the packet dropping probability on policy performance are numerically demonstrated on a single-zone occupant level estimation problem in buildings.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cassandras C G, Lafortune S. Introduction to Discrete Event Systems. 2nd ed. New York: Springer, 2008
Cao X R. Basic ideas for event-based optimization of Markov systems. Discrete Event Dyn Syst, 2005, 15: 169–197
Cao X R. Stochastic Learning and Optimization-A Sensitivity-Based Approach. New York: Springer, 2007
Cassandras C G. The event-driven paradigm for control, communication and optimization. J Control Decision, 2014, 1: 3–17
Ren Z, Krogh B H. State aggregation in Markov decision processes. In: Proceedings of the 41st IEEE Confefence on Decision and Control, Las Vegas, 2002. 3819–3824
Cao X R, Ren Z, Bhatnagar S, et al. A time aggregation approach to Markov decision processes. Automatica, 2002, 38: 929–943
Xia L, Zhao Q C, Jia Q S. A structure property of optimal policies for maintenance problems with safety-critical components. IEEE Trans Automat Sci Eng, 2008, 5: 519–531
Jia Q S. A structural property of optimal policies for multi-component maintenance problems. IEEE Trans Automat Sci Eng, 2010, 7: 677–680
Bertsekas D P, Tsitsiklis J N. Neuro-Dynamic Programming. Belmont: Athena Scientific, 1996
Powell W B. Approxiamte Dynamic Programming: Solving the Curse of Dimensionality. New York: Wiley-Interscience, 2007
Bertsekas D P. Dynamic Programming and Optimal Control: Approximate Dynamic Programming. 4th ed. Nashua: Athena Scientific, 2012
Sutton R S, Barto A G. Reinforcement Learning: An Introduction. Cambridge: MIT Press, 1998
Guestrin C, Koller D, Parr R, et al. Efficient solution algorithms for factored MDPs. J Artif Intell Res, 2003, 19: 399–468
Åström K J, Bernhardsson B. Comparison of periodic and event based sampling for first-order stochastic systems. In: Proceedings of the 14th IFAC World Congress, Beijing, 1999. 301–306
Arzén K E. A simple event-based PID controller. In: Proceedings of the 14th IFAC World Congress, Beijing, 1999. 423–428
Tabuada P. Event-triggered real-time scheduling of stabilizing control tasks. IEEE Trans Automat Contr, 2007, 52: 1680–1685
Heemels W P M H, Johansson K H, Tabuada P. An introduction to event-triggered and self-triggered control. In: Proceedings of the 51st IEEE Conference on Decision and Control, Muai, 2012. 3270–3285
Zhang X M, Han Q L, Zhang B L. An overview and deep investigation on sampled-data-based event-triggered control and filtering for networked systems. IEEE Trans Ind Inf, 2017, 13: 4–16
Wu Z G, Xu Y, Lu R, et al. Event-triggered control for consensus of multiagent systems with fixed/switching topologies. IEEE Trans Syst Man Cybern Syst, 2018, 48: 1736–1746
Xia L, Jia Q S, Cao X R. A tutorial on event-based optimization-a new optimization framework. Discrete Event Dyn Syst, 2014, 24: 103–132
Cao X R, Chen H F. Perturbation realization, potentials, and sensitivity analysis of Markov processes. IEEE Trans Automat Contr, 1997, 42: 1382–1393
Jia Q S, Wen Z, Xia L. Event-based sensor activation for indoor occupant distribution estimation. In: Proceedings of the 12th International Conference on Control, Automation, Robotics, and Vision, Guangzhou, 2012. 240–245
Jia Q S, Shen J X, Xu Z B, et al. Simulation-based policy improvement for energy management in commercial office buildings. IEEE Trans Smart Grid, 2012, 3: 2211–2223
Jia Q S, Guo Y. Event-based evacuation in outdoor environment. In: Proceedings of the 24th Chinese Control and Decision Conference, Taiyuan, 2012. 33–38
Wang D X, Cao X R. Event-based optimization for POMDP and its application in portfolio management. In: Proceedings of the 18th IFAC World Congress, Milano, 2011. 3228–3233
Jia Q S, Wu J. On distributed event-based optimization for shared economy in cyber-physical energy systems. Sci China Inf Sci, 2018, 61: 110203
Guan X, Xu Z, Jia Q S, et al. Cyber-physical model for efficient and secured operation of CPES or energy internet. Sci China Inf Sci, 2018, 61: 110201
Zhong M, Cassandras C G. Asynchronous distributed optimization with event-driven communication. IEEE Trans Automat Contr, 2010, 55: 2735–2750
Jia Q S. Event-based optimization with lagged state information. In: Proceedings of the 31st Chinese Control Conference, Hefei, 2012. 2055–2060
Cao X R, Wang D X, Qiu L. Partial-information state-based optimization of partially observable markov decision processes and the separation principle. IEEE Trans Automat Contr, 2014, 59: 921–936
Sinopoli B, Schenato L, Franceschetti M, et al. Kalman filtering with intermittent observations. IEEE Trans Automat Contr, 2004, 49: 1453–1464
Zhang M, Shen C, Wu Z G, et al. Dissipative filtering for switched fuzzy systems with missing measurements. IEEE Trans Cybern, 2019, 1–10
Wang H T, Jia Q S, Lei Y L, et al. Estimation of occupant distribution by detecting the entrance and leaving events of zones in building. In: Proceedings of the 2012 IEEE International Conference on Multisensor Fusion and Integration, Hamburg, 2012. 27–32
Jia Q S, Wang H, Lei Y, et al. A decentralized stay-time based occupant distribution estimation method for buildings. IEEE Trans Automat Sci Eng, 2015, 12: 1482–1491
Acknowledgements
This work was supported in part by National Key Research and Development Program of China (Grant No. 2017YFC0704100), National Natural Science Foundation of China (Grant No. 61673229), Major Program of the National Natural Science Foundation of China (Grant No. 2018AAA0101600) and the 111 International Collaboration Project of China (Grant No. BP2018006). The authors would like to thank Professor Chiristos G. CASSANDRAS for comments and suggestions on early version of this work. All remaining errors are of course the sole responsibility of the authors.
Author information
Authors and Affiliations
Corresponding author
Supplementary File
Rights and permissions
About this article
Cite this article
Jia, QS., Tang, JX. & Lang, Z. Event-based optimization with random packet dropping. Sci. China Inf. Sci. 63, 212202 (2020). https://doi.org/10.1007/s11432-019-2702-x
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11432-019-2702-x