Academic Journal
Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria.
| Τίτλος: | Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria. |
|---|---|
| Συγγραφείς: | Joo, Joonhwi1 (AUTHOR) joonhwi.joo@utdallas.edu, Chiong, Khai X.1 (AUTHOR) khai.chiong@utdallas.edu |
| Πηγή: | Management Science (INFORMS). May2026, Vol. 72 Issue 5, p4450-4473. 24p. |
| Θεματικοί όροι: | *Decision theory, *Marketing strategy, *Monte Carlo method, *Business revenue, Effect sizes (Statistics), Control groups |
| Περίληψη: | Many firms conduct A/B tests to find a marketing action that improves a value of interest, such as revenue or profit. We develop the asymptotic minimax regret (AMMR) criterion, a practical decision-theoretic approach for choosing among binary marketing actions based on A/B tests. The AMMR is a general large-sample approximation of the minimax-regret criterion from a frequentist standpoint. Our method directly optimizes the decision-relevant metric, accounting for the product of the error probability and the associated magnitude of value loss. Implementing the AMMR decision rule is straightforward; it comprises simply comparing the standardized treatment-effect estimate to the AMMR-optimal decision threshold. The AMMR suggests selecting the treatment whenever the point estimate is positive, as this minimizes the maximum expected net loss from decision errors. A case study of a mobile game company's A/B testing with Monte Carlo validation demonstrates that the AMMR decision rule effectively selects the optimal marketing action and improves revenue across various data-generating processes. This paper was accepted by Raphael Thomadsen, marketing. Funding: K. X. Chiong gratefully acknowledges the financial support from the NEC Foundation of America. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.06590. [ABSTRACT FROM AUTHOR] |
| Copyright of Management Science (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Βάση Δεδομένων: | Business Source Index |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://www.jstor.org/openurl?issn=00251909&date=2026&volume=72&issue=5&spage=4450 Name: JSTOR (all content) (s7799221) Category: fullText Text: Icon: https://imageserver.ebscohost.com/branding/jstor/iconJSTOR.gif MouseOverText: Full Text from JSTOR |
|---|---|
| Header | DbId: bsx DbLabel: Business Source Index An: 193596747 RelevancyScore: 1492 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1491.61413574219 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Joo%2C+Joonhwi%22">Joo, Joonhwi</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> joonhwi.joo@utdallas.edu</i><br /><searchLink fieldCode="AR" term="%22Chiong%2C+Khai+X%2E%22">Chiong, Khai X.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> khai.chiong@utdallas.edu</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Management+Science+%28INFORMS%29%22">Management Science (INFORMS)</searchLink>. May2026, Vol. 72 Issue 5, p4450-4473. 24p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Decision+theory%22">Decision theory</searchLink><br />*<searchLink fieldCode="DE" term="%22Marketing+strategy%22">Marketing strategy</searchLink><br />*<searchLink fieldCode="DE" term="%22Monte+Carlo+method%22">Monte Carlo method</searchLink><br />*<searchLink fieldCode="DE" term="%22Business+revenue%22">Business revenue</searchLink><br /><searchLink fieldCode="DE" term="%22Effect+sizes+%28Statistics%29%22">Effect sizes (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Control+groups%22">Control groups</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Many firms conduct A/B tests to find a marketing action that improves a value of interest, such as revenue or profit. We develop the asymptotic minimax regret (AMMR) criterion, a practical decision-theoretic approach for choosing among binary marketing actions based on A/B tests. The AMMR is a general large-sample approximation of the minimax-regret criterion from a frequentist standpoint. Our method directly optimizes the decision-relevant metric, accounting for the product of the error probability and the associated magnitude of value loss. Implementing the AMMR decision rule is straightforward; it comprises simply comparing the standardized treatment-effect estimate to the AMMR-optimal decision threshold. The AMMR suggests selecting the treatment whenever the point estimate is positive, as this minimizes the maximum expected net loss from decision errors. A case study of a mobile game company's A/B testing with Monte Carlo validation demonstrates that the AMMR decision rule effectively selects the optimal marketing action and improves revenue across various data-generating processes. This paper was accepted by Raphael Thomadsen, marketing. Funding: K. X. Chiong gratefully acknowledges the financial support from the NEC Foundation of America. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.06590. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Management Science (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=bsx&AN=193596747 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1287/mnsc.2024.06590 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 24 StartPage: 4450 Subjects: – SubjectFull: Decision theory Type: general – SubjectFull: Marketing strategy Type: general – SubjectFull: Monte Carlo method Type: general – SubjectFull: Business revenue Type: general – SubjectFull: Effect sizes (Statistics) Type: general – SubjectFull: Control groups Type: general Titles: – TitleFull: Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Joo, Joonhwi – PersonEntity: Name: NameFull: Chiong, Khai X. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: May2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 00251909 Numbering: – Type: volume Value: 72 – Type: issue Value: 5 Titles: – TitleFull: Management Science (INFORMS) Type: main |
| ResultId | 1 |