Λεπτομέρειες βιβλιογραφικής εγγραφής
| Τίτλος: |
On the Convergence of Modified Policy Iteration in Risk-Sensitive Exponential Cost Markov Decision Processes. |
| Συγγραφείς: |
Murthy, Yashaswini1 (AUTHOR) ymurthy@caltech.edu, Moharrami, Mehrdad2 (AUTHOR) moharami@uiowa.edu, Srikant, Rayadurgam3 (AUTHOR) rsrikant@illinois.edu |
| Πηγή: |
Operations Research. May/Jun2026, Vol. 74 Issue 3, p1425-1436. 12p. |
| Θεματικοί όροι: |
*Markov processes, *Dynamic programming, Robust control, Mathematical proofs, Cost functions, Reinforcement learning |
| Περίληψη: |
Many real systems, such as networks, finance, and safety-critical autonomy, must hedge against rare but costly events. Risk-sensitive control formalizes this idea by optimizing an exponential cost objective that prioritizes reliability over just average performance. Classical dynamic programming methods such as value iteration and policy iteration are well-understood in this risk-sensitive setting. However, modified policy iteration (MPI), which combines the strengths of both through partial policy evaluation, has lacked any theoretical understanding. This paper addresses this gap. It analyzes MPI for risk-sensitive Markov decision processes governed by a multiplicative Bellman equation, develops normalization and contraction tools suited to this setting, and proves both convergence and finite-time guarantees. The results provide a principled foundation for algorithms that combine computational efficiency with robustness, supporting the development of reinforcement learning methods that emphasize long-term reliability. Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI is well-studied in the context of discounted and average-cost Markov decision processes (MDPs). In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration are well-studied in the context of risk-sensitive MDPs, MPI is unexplored. To the best of our knowledge, we provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Because the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof that is quite different than existing results for discounted and risk-neutral average-cost problems as well as risk-sensitive value and policy iteration approaches. Funding: This work was supported by the National Science Foundation (NSF) Grants Division of Computing and Communication Foundations (CCF) [Grant 22-07547], the Division of Computer and Network Systems (CNS) [Grant 23-12714], and the Air Force Office of Scientific Research (AFOSR) [Grant FA9550-24-1-0002]. Supplemental Material: All supplemental materials, including the code, data, and files required to reproduce the results, are available at https://doi.org/10.1287/opre.2024.0818. [ABSTRACT FROM AUTHOR] |
|
Copyright of Operations Research is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) |
| Βάση Δεδομένων: |
Business Source Index |
