Academic Journal
Beyond Discounted Returns: Robust Markov Decision Processes with Average and Blackwell Optimality.
| Title: | Beyond Discounted Returns: Robust Markov Decision Processes with Average and Blackwell Optimality. |
|---|---|
| Authors: | Grand-Clément, Julien1 (AUTHOR) grand-clement@hec.fr, Petrik, Marek2 (AUTHOR) mpetrik@cs.unh.edu, Vieille, Nicolas3 (AUTHOR) vieille@hec.fr |
| Source: | Operations Research. Jul/Aug2026, Vol. 74 Issue 4, p2088-2112. 25p. |
| Subject Terms: | *Markov processes, Stochastic control theory |
| Abstract: | Robust Markov decision processes (RMDPs) have been studied extensively when the objective is the discounted return, but little is known for average optimality and Blackwell optimality. We show that average optimal policies can be chosen stationary and deterministic for sa-rectangular RMDPs, but perhaps surprisingly, we show that for s-rectangular RMDPs average optimal policies may not exist, and if they do exist, they may not be stationary. We also study Blackwell optimality for sa-rectangular RMDPs, where we show that approximately Blackwell optimal policies always exist, although exact Blackwell optimal policies may not exist. We provide a general sufficient condition for their existence. We then discuss the connection between average and Blackwell optimality, and we describe several algorithms to compute the optimal average return. Interestingly, our approach leverages the connections between RMDPs and stochastic games. Overall, our paper emphasizes the superior practical properties of distance-based sa-rectangular models over s-rectangular models for average and Blackwell optimality. Robust Markov decision processes (RMDPs) are a widely used framework for sequential decision-making under parameter uncertainty. RMDPs have been studied extensively when the objective was to maximize the discounted return, but little is known for average optimality (optimizing the long-run average of the rewards obtained over time) and Blackwell optimality (remaining discount optimal for all discount factors sufficiently close to 1). In this paper, we prove several foundational results for RMDPs beyond the discounted return. We show that average optimal policies can be chosen stationary and deterministic for sa-rectangular RMDPs but, perhaps surprisingly, we show that for s-rectangular RMDPs average optimal policies may not exist, and if they do exist, they may need to be history dependent (Markovian). We also study Blackwell optimality for sa-rectangular RMDPs, where we show that ϵ -Blackwell optimal policies always exist, although Blackwell optimal policies may not exist. We also provide a sufficient condition for their existence, which encompasses virtually any examples from the literature. We then discuss the connection between average and Blackwell optimality, and we describe several algorithms to compute the optimal average return. Interestingly, our approach leverages the connections between RMDPs and stochastic games. Overall, our paper emphasizes the superior practical properties of distance-based sa-rectangular models over s-rectangular models for average and Blackwell optimality. Funding: J. Grand-Clément was supported by Agence Nationale de la Recherche [Grant 11-LABX-0047]. M. Petrik was supported by NSF [Grants 2144601 and 2218063]. N. Vieille thanks the HEC foundation for support. Supplemental Material: All supplemental materials, including the code, data, and files required to reproduce the results, are available at https://doi.org/10.1287/opre.2023.0694. [ABSTRACT FROM AUTHOR] |
| Copyright of Operations Research is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Business Source Index |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://www.jstor.org/openurl?issn=0030364X&date=2026&volume=74&issue=4&spage=2088 Name: JSTOR (all content) (s7799221) Category: fullText Text: Icon: https://imageserver.ebscohost.com/branding/jstor/iconJSTOR.gif MouseOverText: Full Text from JSTOR |
|---|---|
| Header | DbId: bsx DbLabel: Business Source Index An: 195839140 RelevancyScore: 1492 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1491.61413574219 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Beyond Discounted Returns: Robust Markov Decision Processes with Average and Blackwell Optimality. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Grand-Clément%2C+Julien%22">Grand-Clément, Julien</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> grand-clement@hec.fr</i><br /><searchLink fieldCode="AR" term="%22Petrik%2C+Marek%22">Petrik, Marek</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> mpetrik@cs.unh.edu</i><br /><searchLink fieldCode="AR" term="%22Vieille%2C+Nicolas%22">Vieille, Nicolas</searchLink><relatesTo>3</relatesTo> (AUTHOR)<i> vieille@hec.fr</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Operations+Research%22">Operations Research</searchLink>. Jul/Aug2026, Vol. 74 Issue 4, p2088-2112. 25p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Markov+processes%22">Markov processes</searchLink><br /><searchLink fieldCode="DE" term="%22Stochastic+control+theory%22">Stochastic control theory</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Robust Markov decision processes (RMDPs) have been studied extensively when the objective is the discounted return, but little is known for average optimality and Blackwell optimality. We show that average optimal policies can be chosen stationary and deterministic for sa-rectangular RMDPs, but perhaps surprisingly, we show that for s-rectangular RMDPs average optimal policies may not exist, and if they do exist, they may not be stationary. We also study Blackwell optimality for sa-rectangular RMDPs, where we show that approximately Blackwell optimal policies always exist, although exact Blackwell optimal policies may not exist. We provide a general sufficient condition for their existence. We then discuss the connection between average and Blackwell optimality, and we describe several algorithms to compute the optimal average return. Interestingly, our approach leverages the connections between RMDPs and stochastic games. Overall, our paper emphasizes the superior practical properties of distance-based sa-rectangular models over s-rectangular models for average and Blackwell optimality. Robust Markov decision processes (RMDPs) are a widely used framework for sequential decision-making under parameter uncertainty. RMDPs have been studied extensively when the objective was to maximize the discounted return, but little is known for average optimality (optimizing the long-run average of the rewards obtained over time) and Blackwell optimality (remaining discount optimal for all discount factors sufficiently close to 1). In this paper, we prove several foundational results for RMDPs beyond the discounted return. We show that average optimal policies can be chosen stationary and deterministic for sa-rectangular RMDPs but, perhaps surprisingly, we show that for s-rectangular RMDPs average optimal policies may not exist, and if they do exist, they may need to be history dependent (Markovian). We also study Blackwell optimality for sa-rectangular RMDPs, where we show that ϵ -Blackwell optimal policies always exist, although Blackwell optimal policies may not exist. We also provide a sufficient condition for their existence, which encompasses virtually any examples from the literature. We then discuss the connection between average and Blackwell optimality, and we describe several algorithms to compute the optimal average return. Interestingly, our approach leverages the connections between RMDPs and stochastic games. Overall, our paper emphasizes the superior practical properties of distance-based sa-rectangular models over s-rectangular models for average and Blackwell optimality. Funding: J. Grand-Clément was supported by Agence Nationale de la Recherche [Grant 11-LABX-0047]. M. Petrik was supported by NSF [Grants 2144601 and 2218063]. N. Vieille thanks the HEC foundation for support. Supplemental Material: All supplemental materials, including the code, data, and files required to reproduce the results, are available at https://doi.org/10.1287/opre.2023.0694. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Operations Research is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=bsx&AN=195839140 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1287/opre.2023.0694 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 25 StartPage: 2088 Subjects: – SubjectFull: Markov processes Type: general – SubjectFull: Stochastic control theory Type: general Titles: – TitleFull: Beyond Discounted Returns: Robust Markov Decision Processes with Average and Blackwell Optimality. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Grand-Clément, Julien – PersonEntity: Name: NameFull: Petrik, Marek – PersonEntity: Name: NameFull: Vieille, Nicolas IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 07 Text: Jul/Aug2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 0030364X Numbering: – Type: volume Value: 74 – Type: issue Value: 4 Titles: – TitleFull: Operations Research Type: main |
| ResultId | 1 |