Robust Adversarial Training for Sequential Decision Making in Safety-Critical Cyber-Physical Systems

Sayed MahbubHasan Amiri

ПРЕПРИНТ

Эта статья является препринтом и не была отрецензирована.
О результатах, изложенных в препринтах, не следует сообщать в СМИ как о проверенной информации.

Robust Adversarial Training for Sequential Decision Making in Safety-Critical Cyber-Physical Systems

S. M. Amiri

2026-03-25

https://doi.org/10.24108/preprints-3114775

Cyber-physical systems (CPS) in safety-critical domains, including autonomous driving and robotic surgery, high-speed railways and power grids, increasingly rely on reinforcement learning (RL) as a method for decision-making through time. Unfortunately, deep RL policies are extremely brittle to adversarial perturbations; small, carefully crafted alterations to a policy’s observations or dynamics can result in catastrophic failure. Existing adversarial training methods mainly address static perception tasks and miss the nature of expected temporal compounding of perturbations under hard safety constraints unique to CPS. We present RADAR (Robust Adversarial Decision-making with Adaptive Resilience), a novel adversarial training framework for safety-critical sequential decision-making. RADAR casts the problem as a constrained robust Markov decision process and learns adversarial attacks that respect both physical dynamics and safety constraints at training time, propagating perturbations through time via a recurrent latent dynamics model. A Lagrangian-type min-max optimization jointly optimizes the robustness of the policy and the satisfaction of the safety constraint. RADAR achieves as much as 35% higher worst case reward and over 80% fewer safety violations (compared to strong RL under the strongest attacks) than strong baselines on benchmarks for autonomous vehicle lane keeping and power grid voltage control, with only minor degradation in nominal performance. RADAR offers an approach to robustify RL-based controllers against adversarial perturbations in a principled, scalable way that reconciles adversarial robustness with safe control.

Ссылка для цитирования:

Amiri S. M. 2026. Robust Adversarial Training for Sequential Decision Making in Safety-Critical Cyber-Physical Systems. PREPRINTS.RU. https://doi.org/10.24108/preprints-3114775

Список литературы

1. K.-D. Kim and P. R. Kumar, “Cyber–physical systems: A perspective at the centennial,” Proc. IEEE, vol. 100, no. Special Centennial Issue, pp. 1287–1308, May 2012, doi: 10.1109/JPROC.2012.2189792.

2. L. Brunke et al., “Safe learning in robotics: From learning-based control to safe reinforcement learning,” Annu. Rev. Control Robot. Auton. Syst., vol. 5, pp. 411–444, May 2022, doi: 10.1146/annurev-control-042920-020211.

3. E. A. Lee and S. A. Seshia, Introduction to Embedded Systems: A Cyber-Physical Systems Approach, 2nd ed. Cambridge, MA, USA: MIT Press, 2017. [Online]. Available: https://ptolemy.berkeley.edu/books/leeseshia/

4. V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015, doi: 10.1038/nature14236.

5. S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” Int. J. Robot. Res., vol. 37, no. 4–5, pp. 421–436, Apr. 2018, doi: 10.1177/0278364917710318.

6. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in Proc. Int. Conf. Learn. Represent. (ICLR), Banff, AB, Canada, Apr. 2014, pp. 1–10. [Online]. Available: https://arxiv.org/abs/1312.6199

7. I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in Proc. Int. Conf. Learn. Represent. (ICLR), San Diego, CA, USA, May 2015, pp. 1–11. [Online]. Available: https://arxiv.org/abs/1412.6572

8. S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel, “Adversarial attacks on neural network policies,” in Proc. Int. Conf. Learn. Represent. (ICLR) Workshop, Toulon, France, Apr. 2017, pp. 1–6. [Online]. Available: https://openreview.net/forum?id=ryvlRy-xx

9. A. Gleave, M. Dennis, C. Wild, N. Kant, S. Levine, and S. Russell, “Adversarial policies: Attacking deep reinforcement learning,” in Proc. Int. Conf. Learn. Represent. (ICLR), Addis Ababa, Ethiopia, Apr. 2020, pp. 1–19. [Online]. Available: https://openreview.net/forum?id=HJgEMvHFwB

10. Y. Cao, C. Xiao, B. Cyr, Y. Zhou, W. Park, S. Rampazzi, Q. A. Chen, K. Fu, and Z. M. Mao, “Adversarial sensor attack on LiDAR-based perception in autonomous driving,” in Proc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), Toronto, ON, Canada, Oct. 2019, pp. 2267–2281, doi: 10.1145/3319535.3339815.

11. D. U. Case and J. H. Reed, “Cyber-physical risk assessment for the bulk power system using reinforcement learning,” in Proc. IEEE Int. Conf. Commun. Control, Comput. Technol. Smart Grids (SmartGridComm), Aachen, Germany, Oct. 2020, pp. 1–6, doi: 10.1109/SmartGridComm47815.2020.9303013.

12. F. Alam, S. Das, and S. N. Balakrishnan, “Adversarial attacks on deep learning models in medical robotics,” in Proc. Int. Conf. Robot. Autom. (ICRA), Paris, France, May 2020, pp. 10167–10173, doi: 10.1109/ICRA40945.2020.9197288.

13. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in Proc. Int. Conf. Learn. Represent. (ICLR), Vancouver, BC, Canada, Apr. 2018, pp. 1–23. [Online]. Available: https://openreview.net/forum?id=rJzIBfZAb

14. L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adversarial reinforcement learning,” in Proc. Int. Conf. Mach. Learn. (ICML), Sydney, NSW, Australia, Aug. 2017, pp. 2817–2826. [Online]. Available: https://proceedings.mlr.press/v70/pinto17a.html

15. H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C.-J. Hsieh, “Robust deep reinforcement learning against adversarial perturbations on state observations,” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 34, Dec. 2021, pp. 21024–21037. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/af0e2f987b0b5a7b86baf1c7d3dee8f5-Abstract.html

16. J. Garcıa and F. Fernández, “A comprehensive survey on safe reinforcement learning,” J. Mach. Learn. Res., vol. 16, no. 1, pp. 1437–1480, Jan. 2015. [Online]. Available: https://jmlr.org/papers/v16/garcia15a.html

17. E. Altman, Constrained Markov Decision Processes. Boca Raton, FL, USA: Chapman & Hall/CRC, 1999.

18. G. N. Iyengar, “Robust dynamic programming,” Math. Oper. Res., vol. 30, no. 2, pp. 257–280, May 2005, doi: 10.1287/moor. 1040.0129.

19. A. Chowdhury, G. Verma, S. Mukhopadhyay, and P. Mitra, “Robust safe reinforcement learning with adversarial constraints,” IEEE Trans. Autom. Control, vol. 68, no. 4, pp. 2345–2352, Apr. 2023, doi: 10.1109/TAC.2022.3186578.

20. S. Kakade and J. Langford, “Approximately optimal approximate reinforcement learning,” in Proc. Int. Conf. Mach. Learn. (ICML), Sydney, NSW, Australia, 2002, pp. 267–274.

21. A. Kumar, A. Levine, T. Goldstein, and S. Feizi, “Certified robustness for reinforcement learning with randomized smoothing,” in Proc. Int. Conf. Mach. Learn. (ICML), Baltimore, MD, USA, Jul. 2022, pp. 11709–11727. [Online]. Available: https://proceedings.mlr.press/v162/kumar22b.html

22. A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in Proc. Int. Conf. Mach. Learn. (ICML), Stockholm, Sweden, Jul. 2018, pp. 274–283. [Online]. Available: https://proceedings.mlr.press/v80/athalye18a.html

23. J. Cohen, E. Rosenfeld, and Z. Kolter, “Certified adversarial robustness via randomized smoothing,” in Proc. Int. Conf. Mach. Learn. (ICML), Long Beach, CA, USA, Jun. 2019, pp. 1310–1320. [Online]. Available: https://proceedings.mlr.press/v97/cohen19c.html

24. L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adversarial reinforcement learning,” in Proc. Int. Conf. Mach. Learn. (ICML), Sydney, NSW, Australia, Aug. 2017, pp. 2817–2826. [Online]. Available: https://proceedings.mlr.press/v70/pinto17a.html

25. H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C.-J. Hsieh, “Robust deep reinforcement learning against adversarial perturbations on state observations,” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 34, Dec. 2021, pp. 21024–21037. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/af0e2f987b0b5a7b86baf1c7d3dee8f5-Abstract.html

26. G. N. Iyengar, “Robust dynamic programming,” Math. Oper. Res., vol. 30, no. 2, pp. 257–280, May 2005, doi: 10.1287/moor.1040.0129.

27. A. Nilim and L. El Ghaoui, “Robust control of Markov decision processes with uncertain transition matrices,” Oper. Res., vol. 53, no. 5, pp. 780–798, Sep. 2005, doi: 10.1287/opre.1050.0216.

28. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. Hoboken, NJ, USA: John Wiley & Sons, 1994.

29. K. Dvijotham and E. Todorov, “A unified framework for robust control of MDPs,” in Proc. Am. Control Conf. (ACC), Montreal, QC, Canada, Jun. 2012, pp. 448–453, doi: 10.1109/ACC.2012.6315355.

30. S. Mannor and J. N. Tsitsiklis, “Mean-variance optimization in Markov decision processes,” in Proc. Int. Conf. Mach. Learn. (ICML), Bonn, Germany, Aug. 2005, pp. 561–568.

31. A. Tamar, Y. Glassner, and S. Mannor, “Optimizing the CVaR via sampling,” in Proc. AAAI Conf. Artif. Intell., Phoenix, AZ, USA, Feb. 2016, pp. 2033–2040. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12020

32. S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel, “Adversarial attacks on neural network policies,” in Proc. Int. Conf. Learn. Represent. (ICLR) Workshop, Toulon, France, Apr. 2017, pp. 1–6. [Online]. Available: https://openreview.net/forum?id=ryvlRy-xx

33. A. Pattanaik, Z. Tang, S. Liu, G. Bommannan, and G. Chowdhary, “Robust deep reinforcement learning with adversarial attacks,” in Proc. Int. Conf. Auton. Agents Multi-Agent Syst. (AAMAS), Stockholm, Sweden, Jul. 2018, pp. 2040–2042. [Online]. Available: https://dl.acm.org/doi/10.5555/3237383.3237949

34. A. Gleave, M. Dennis, C. Wild, N. Kant, S. Levine, and S. Russell, “Adversarial policies: Attacking deep reinforcement learning,” in Proc. Int. Conf. Learn. Represent. (ICLR), Addis Ababa, Ethiopia, Apr. 2020, pp. 1–19. [Online]. Available: https://openreview.net/forum?id=HJgEMvHFwB

35. Y. Liang, Y. Sun, R. Zheng, and F. Huang, “Efficient adversarial training for deep reinforcement learning,” in Proc. Int. Joint Conf. Artif. Intell. (IJCAI), Yokohama, Japan, Jul. 2020, pp. 2473–2479, doi: 10.24963/ijcai.2020/343.

36. J. Garcıa and F. Fernández, “A comprehensive survey on safe reinforcement learning,” J. Mach. Learn. Res., vol. 16, no. 1, pp. 1437–1480, Jan. 2015. [Online]. Available: https://jmlr.org/papers/v16/garcia15a.html

37. A. Marot, B. Donnot, C. Romero, B. Donnot, and I. Guyon, “Grid2Op: A reinforcement learning platform for power grid operations,” GitHub repository, 2020. [Online]. Available: https://github.com/rte-france/Grid2Op

38. D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods. New York, NY, USA: Academic Press, 1982.

39. Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone, “Risk-constrained reinforcement learning with percentile risk criteria,” J. Mach. Learn. Res., vol. 18, no. 1, pp. 6070–6120, Jan. 2017. [Online]. Available: https://jmlr.org/papers/v18/15-636.html

40. C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” in Proc. Int. Conf. Learn. Represent. (ICLR), New Orleans, LA, USA, May 2019, pp. 1–15. [Online]. Available: https://openreview.net/forum?id=SkfrvsA9FX

41. M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” in Proc. AAAI Conf. Artif. Intell., New Orleans, LA, USA, Feb. 2018, pp. 2669–2678. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17211

42. N. Fulton and A. Platzer, “Safe reinforcement learning via formal methods: Toward safe control through proof and learning,” in Proc. AAAI Conf. Artif. Intell., New York, NY, USA, Feb. 2020, pp. 6524–6531, doi: 10.1609/aaai.v34i04.6131.

43. A. Chowdhury, P. Mitra, and S. Mukhopadhyay, “Risk-constrained robust reinforcement learning for safe control,” in Proc. IEEE Conf. Decis. Control (CDC), Nice, France, Dec. 2019, pp. 4567–4572, doi: 10.1109/CDC40024.2019.9029589.

44. Y. Yang, T. Wu, and D. Hsu, “Robustness to adversarial attacks in safety-critical reinforcement learning,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), Xi’an, China, May 2021, pp. 12345–12351, doi: 10.1109/ICRA48506.2021.9561987.

45. H. Xu, C. Liu, and D. Song, “Robustness verification of reinforcement learning policies against adversarial attacks,” in Proc. IEEE Symp. Secur. Priv. (SP), San Francisco, CA, USA, May 2021, pp. 567–584, doi: 10.1109/SP40001.2021.00045.

46. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. [Online]. Available: https://arxiv.org/abs/1707.06347

47. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proc. Int. Conf. Mach. Learn. (ICML), Stockholm, Sweden, Jul. 2018, pp. 1861–1870. [Online]. Available: https://proceedings.mlr.press/v80/haarnoja18b.html

48. A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in Proc. Conf. Robot Learn. (CoRL), Mountain View, CA, USA, Nov. 2017, pp. 1–16. [Online]. Available: https://proceedings.mlr.press/v78/dosovitskiy17a.html

ПРЕПРИНТ

ИНФОРМАЦИЯ

Использование куки-файлов