Academic Journal
Stochastic First-Order Methods for Average-Reward Markov Decision Processes.
| Τίτλος: | Stochastic First-Order Methods for Average-Reward Markov Decision Processes. |
|---|---|
| Συγγραφείς: | Li, Tianjiao1 (AUTHOR) tli432@gatech.edu, Wu, Feiyang2 (AUTHOR) fwu91@gatech.edu, Lan, Guanghui1 (AUTHOR) george.lan@isye.gatech.edu |
| Πηγή: | Mathematics of Operations Research (INFORMS). Nov2025, Vol. 50 Issue 4, p3125-3160. 36p. |
| Θεματικοί όροι: | *Markov processes, *Stochastic analysis, Reinforcement learning, University research |
| Περίληψη: | We study average-reward Markov decision processes (AMDPs) and develop novel first-order methods with strong theoretical guarantees for both policy optimization and policy evaluation. Compared with intensive research efforts in finite sample analysis of policy gradient methods for discounted MDPs, existing studies on policy gradient methods for AMDPs mostly focus on regret bounds under restrictive assumptions, and they often lack guarantees on the overall sample complexities. Toward this end, we develop an average-reward stochastic policy mirror descent method for solving AMDPs with and without regularizers and provide convergence guarantees in terms of the long-term average reward. For policy evaluation, existing on-policy methods suffer from suboptimal convergence rates as well as failure in handling insufficiently random policies due to the lack of exploration in the action space. To remedy these issues, we develop a variance-reduced temporal difference (VRTD) method with linear function approximation for randomized policies along with optimal convergence guarantees, and design an exploratory VRTD method that resolves the exploration issue and provides comparable convergence guarantees. By combining the policy evaluation and policy optimization parts, we establish sample complexity results for solving AMDPs under both generative and Markovian noise models. It is worth noting that when linear function approximation is utilized, our algorithm only needs to update in the low-dimensional parameter space, and thus can handle MDPs with large state and action spaces. Funding: T. Li and G. Lan were supported in part by the National Science Foundation [Grant CCF-1909298] and the Office of Navel Research [Grant N00014-20-1-2089]. [ABSTRACT FROM AUTHOR] |
| Copyright of Mathematics of Operations Research (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Βάση Δεδομένων: | Business Source Index |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://www.jstor.org/openurl?issn=0364765X&date=2025&volume=50&issue=4&spage=3125 Name: JSTOR (all content) (s7799221) Category: fullText Text: Icon: https://imageserver.ebscohost.com/branding/jstor/iconJSTOR.gif MouseOverText: Full Text from JSTOR |
|---|---|
| Header | DbId: bsx DbLabel: Business Source Index An: 189423727 RelevancyScore: 1451 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1451.22045898438 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Stochastic First-Order Methods for Average-Reward Markov Decision Processes. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Li%2C+Tianjiao%22">Li, Tianjiao</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> tli432@gatech.edu</i><br /><searchLink fieldCode="AR" term="%22Wu%2C+Feiyang%22">Wu, Feiyang</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> fwu91@gatech.edu</i><br /><searchLink fieldCode="AR" term="%22Lan%2C+Guanghui%22">Lan, Guanghui</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> george.lan@isye.gatech.edu</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Mathematics+of+Operations+Research+%28INFORMS%29%22">Mathematics of Operations Research (INFORMS)</searchLink>. Nov2025, Vol. 50 Issue 4, p3125-3160. 36p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Markov+processes%22">Markov processes</searchLink><br />*<searchLink fieldCode="DE" term="%22Stochastic+analysis%22">Stochastic analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Reinforcement+learning%22">Reinforcement learning</searchLink><br /><searchLink fieldCode="DE" term="%22University+research%22">University research</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: We study average-reward Markov decision processes (AMDPs) and develop novel first-order methods with strong theoretical guarantees for both policy optimization and policy evaluation. Compared with intensive research efforts in finite sample analysis of policy gradient methods for discounted MDPs, existing studies on policy gradient methods for AMDPs mostly focus on regret bounds under restrictive assumptions, and they often lack guarantees on the overall sample complexities. Toward this end, we develop an average-reward stochastic policy mirror descent method for solving AMDPs with and without regularizers and provide convergence guarantees in terms of the long-term average reward. For policy evaluation, existing on-policy methods suffer from suboptimal convergence rates as well as failure in handling insufficiently random policies due to the lack of exploration in the action space. To remedy these issues, we develop a variance-reduced temporal difference (VRTD) method with linear function approximation for randomized policies along with optimal convergence guarantees, and design an exploratory VRTD method that resolves the exploration issue and provides comparable convergence guarantees. By combining the policy evaluation and policy optimization parts, we establish sample complexity results for solving AMDPs under both generative and Markovian noise models. It is worth noting that when linear function approximation is utilized, our algorithm only needs to update in the low-dimensional parameter space, and thus can handle MDPs with large state and action spaces. Funding: T. Li and G. Lan were supported in part by the National Science Foundation [Grant CCF-1909298] and the Office of Navel Research [Grant N00014-20-1-2089]. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Mathematics of Operations Research (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=bsx&AN=189423727 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1287/moor.2022.0241 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 36 StartPage: 3125 Subjects: – SubjectFull: Markov processes Type: general – SubjectFull: Stochastic analysis Type: general – SubjectFull: Reinforcement learning Type: general – SubjectFull: University research Type: general Titles: – TitleFull: Stochastic First-Order Methods for Average-Reward Markov Decision Processes. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Li, Tianjiao – PersonEntity: Name: NameFull: Wu, Feiyang – PersonEntity: Name: NameFull: Lan, Guanghui IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 11 Text: Nov2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 0364765X Numbering: – Type: volume Value: 50 – Type: issue Value: 4 Titles: – TitleFull: Mathematics of Operations Research (INFORMS) Type: main |
| ResultId | 1 |