Academic Journal
On the Convergence of Modified Policy Iteration in Risk-Sensitive Exponential Cost Markov Decision Processes.
| Title: | On the Convergence of Modified Policy Iteration in Risk-Sensitive Exponential Cost Markov Decision Processes. |
|---|---|
| Authors: | Murthy, Yashaswini1 (AUTHOR) ymurthy@caltech.edu, Moharrami, Mehrdad2 (AUTHOR) moharami@uiowa.edu, Srikant, Rayadurgam3 (AUTHOR) rsrikant@illinois.edu |
| Source: | Operations Research. May/Jun2026, Vol. 74 Issue 3, p1425-1436. 12p. |
| Subject Terms: | *Markov processes, *Dynamic programming, Robust control, Mathematical proofs, Cost functions, Reinforcement learning |
| Abstract: | Many real systems, such as networks, finance, and safety-critical autonomy, must hedge against rare but costly events. Risk-sensitive control formalizes this idea by optimizing an exponential cost objective that prioritizes reliability over just average performance. Classical dynamic programming methods such as value iteration and policy iteration are well-understood in this risk-sensitive setting. However, modified policy iteration (MPI), which combines the strengths of both through partial policy evaluation, has lacked any theoretical understanding. This paper addresses this gap. It analyzes MPI for risk-sensitive Markov decision processes governed by a multiplicative Bellman equation, develops normalization and contraction tools suited to this setting, and proves both convergence and finite-time guarantees. The results provide a principled foundation for algorithms that combine computational efficiency with robustness, supporting the development of reinforcement learning methods that emphasize long-term reliability. Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI is well-studied in the context of discounted and average-cost Markov decision processes (MDPs). In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration are well-studied in the context of risk-sensitive MDPs, MPI is unexplored. To the best of our knowledge, we provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Because the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof that is quite different than existing results for discounted and risk-neutral average-cost problems as well as risk-sensitive value and policy iteration approaches. Funding: This work was supported by the National Science Foundation (NSF) Grants Division of Computing and Communication Foundations (CCF) [Grant 22-07547], the Division of Computer and Network Systems (CNS) [Grant 23-12714], and the Air Force Office of Scientific Research (AFOSR) [Grant FA9550-24-1-0002]. Supplemental Material: All supplemental materials, including the code, data, and files required to reproduce the results, are available at https://doi.org/10.1287/opre.2024.0818. [ABSTRACT FROM AUTHOR] |
| Copyright of Operations Research is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Business Source Index |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://www.jstor.org/openurl?issn=0030364X&date=2026&volume=74&issue=3&spage=1425 Name: JSTOR (all content) (s7799221) Category: fullText Text: Icon: https://imageserver.ebscohost.com/branding/jstor/iconJSTOR.gif MouseOverText: Full Text from JSTOR |
|---|---|
| Header | DbId: bsx DbLabel: Business Source Index An: 194583180 RelevancyScore: 1492 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1491.61413574219 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: On the Convergence of Modified Policy Iteration in Risk-Sensitive Exponential Cost Markov Decision Processes. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Murthy%2C+Yashaswini%22">Murthy, Yashaswini</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> ymurthy@caltech.edu</i><br /><searchLink fieldCode="AR" term="%22Moharrami%2C+Mehrdad%22">Moharrami, Mehrdad</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> moharami@uiowa.edu</i><br /><searchLink fieldCode="AR" term="%22Srikant%2C+Rayadurgam%22">Srikant, Rayadurgam</searchLink><relatesTo>3</relatesTo> (AUTHOR)<i> rsrikant@illinois.edu</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Operations+Research%22">Operations Research</searchLink>. May/Jun2026, Vol. 74 Issue 3, p1425-1436. 12p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Markov+processes%22">Markov processes</searchLink><br />*<searchLink fieldCode="DE" term="%22Dynamic+programming%22">Dynamic programming</searchLink><br /><searchLink fieldCode="DE" term="%22Robust+control%22">Robust control</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+proofs%22">Mathematical proofs</searchLink><br /><searchLink fieldCode="DE" term="%22Cost+functions%22">Cost functions</searchLink><br /><searchLink fieldCode="DE" term="%22Reinforcement+learning%22">Reinforcement learning</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Many real systems, such as networks, finance, and safety-critical autonomy, must hedge against rare but costly events. Risk-sensitive control formalizes this idea by optimizing an exponential cost objective that prioritizes reliability over just average performance. Classical dynamic programming methods such as value iteration and policy iteration are well-understood in this risk-sensitive setting. However, modified policy iteration (MPI), which combines the strengths of both through partial policy evaluation, has lacked any theoretical understanding. This paper addresses this gap. It analyzes MPI for risk-sensitive Markov decision processes governed by a multiplicative Bellman equation, develops normalization and contraction tools suited to this setting, and proves both convergence and finite-time guarantees. The results provide a principled foundation for algorithms that combine computational efficiency with robustness, supporting the development of reinforcement learning methods that emphasize long-term reliability. Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI is well-studied in the context of discounted and average-cost Markov decision processes (MDPs). In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration are well-studied in the context of risk-sensitive MDPs, MPI is unexplored. To the best of our knowledge, we provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Because the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof that is quite different than existing results for discounted and risk-neutral average-cost problems as well as risk-sensitive value and policy iteration approaches. Funding: This work was supported by the National Science Foundation (NSF) Grants Division of Computing and Communication Foundations (CCF) [Grant 22-07547], the Division of Computer and Network Systems (CNS) [Grant 23-12714], and the Air Force Office of Scientific Research (AFOSR) [Grant FA9550-24-1-0002]. Supplemental Material: All supplemental materials, including the code, data, and files required to reproduce the results, are available at https://doi.org/10.1287/opre.2024.0818. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Operations Research is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=bsx&AN=194583180 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1287/opre.2024.0818 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 12 StartPage: 1425 Subjects: – SubjectFull: Markov processes Type: general – SubjectFull: Dynamic programming Type: general – SubjectFull: Robust control Type: general – SubjectFull: Mathematical proofs Type: general – SubjectFull: Cost functions Type: general – SubjectFull: Reinforcement learning Type: general Titles: – TitleFull: On the Convergence of Modified Policy Iteration in Risk-Sensitive Exponential Cost Markov Decision Processes. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Murthy, Yashaswini – PersonEntity: Name: NameFull: Moharrami, Mehrdad – PersonEntity: Name: NameFull: Srikant, Rayadurgam IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: May/Jun2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 0030364X Numbering: – Type: volume Value: 74 – Type: issue Value: 3 Titles: – TitleFull: Operations Research Type: main |
| ResultId | 1 |