Academic Journal
On the Impossibility of Statistically Improving Empirical Optimization: A Second Order Stochastic Dominance Perspective.
| Title: | On the Impossibility of Statistically Improving Empirical Optimization: A Second Order Stochastic Dominance Perspective. |
|---|---|
| Authors: | Lam, Henry |
| Source: | Management Science (INFORMS); Sep2026, Vol. 72 Issue 9, p8215-8245, 31p |
| Subject Terms: | Stochastic dominance, Statistical accuracy, Mathematical regularization, Robust optimization, Empirical research, Stochastic programming, Mathematical analysis |
| Abstract: | When the underlying probability distribution in a stochastic optimization is observed only through data, various data-driven formulations have been studied to obtain approximate optimal solutions. We show that no such formulations can, in a sense, theoretically improve the statistical quality of the solution obtained from empirical optimization. We argue this by proving that the first order behavior of the optimality gap against the oracle best solution, which includes both the bias and variance, for any data-driven solution second order stochastically dominates that from empirical optimization as long as suitable smoothness holds with respect to the underlying distribution. We demonstrate this impossibility of improvement in examples ranging across regularized optimization, distributionally robust optimization, parametric optimization, and Bayesian generalizations. We also discuss the connections of our results to other perspectives in statistics and data-driven optimization and illustrate practical implications in choosing among data-driven formulations. This paper was accepted by J. George Shanthikumar, data science. Funding: This work was supported by the National Science Foundation Division of Information and Intelligent Systems [Grant 1849280] and Division of Civil, Mechanical, and Manufacturing Innovation [Grant 1834710]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.04482. [ABSTRACT FROM AUTHOR] |
| Copyright of Management Science (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Complementary Index |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://www.jstor.org/openurl?issn=00251909&date=2026&volume=72&issue=9&spage=8215 Name: JSTOR (all content) (s7799221) Category: fullText Text: Icon: https://imageserver.ebscohost.com/branding/jstor/iconJSTOR.gif MouseOverText: Full Text from JSTOR |
|---|---|
| Header | DbId: edb DbLabel: Complementary Index An: 197037851 RelevancyScore: 1162 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1162.42749023438 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: On the Impossibility of Statistically Improving Empirical Optimization: A Second Order Stochastic Dominance Perspective. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Lam%2C+Henry%22">Lam, Henry</searchLink> – Name: TitleSource Label: Source Group: Src Data: Management Science (INFORMS); Sep2026, Vol. 72 Issue 9, p8215-8245, 31p – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Stochastic+dominance%22">Stochastic dominance</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+accuracy%22">Statistical accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+regularization%22">Mathematical regularization</searchLink><br /><searchLink fieldCode="DE" term="%22Robust+optimization%22">Robust optimization</searchLink><br /><searchLink fieldCode="DE" term="%22Empirical+research%22">Empirical research</searchLink><br /><searchLink fieldCode="DE" term="%22Stochastic+programming%22">Stochastic programming</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+analysis%22">Mathematical analysis</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: When the underlying probability distribution in a stochastic optimization is observed only through data, various data-driven formulations have been studied to obtain approximate optimal solutions. We show that no such formulations can, in a sense, theoretically improve the statistical quality of the solution obtained from empirical optimization. We argue this by proving that the first order behavior of the optimality gap against the oracle best solution, which includes both the bias and variance, for any data-driven solution second order stochastically dominates that from empirical optimization as long as suitable smoothness holds with respect to the underlying distribution. We demonstrate this impossibility of improvement in examples ranging across regularized optimization, distributionally robust optimization, parametric optimization, and Bayesian generalizations. We also discuss the connections of our results to other perspectives in statistics and data-driven optimization and illustrate practical implications in choosing among data-driven formulations. This paper was accepted by J. George Shanthikumar, data science. Funding: This work was supported by the National Science Foundation Division of Information and Intelligent Systems [Grant 1849280] and Division of Civil, Mechanical, and Manufacturing Innovation [Grant 1834710]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.04482. [ABSTRACT FROM AUTHOR] – Name: Abstract Label: Group: Ab Data: <i>Copyright of Management Science (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edb&AN=197037851 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1287/mnsc.2024.04482 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 31 StartPage: 8215 Subjects: – SubjectFull: Stochastic dominance Type: general – SubjectFull: Statistical accuracy Type: general – SubjectFull: Mathematical regularization Type: general – SubjectFull: Robust optimization Type: general – SubjectFull: Empirical research Type: general – SubjectFull: Stochastic programming Type: general – SubjectFull: Mathematical analysis Type: general Titles: – TitleFull: On the Impossibility of Statistically Improving Empirical Optimization: A Second Order Stochastic Dominance Perspective. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Lam, Henry IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 09 Text: Sep2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 00251909 Numbering: – Type: volume Value: 72 – Type: issue Value: 9 Titles: – TitleFull: Management Science (INFORMS) Type: main |
| ResultId | 1 |