Academic Journal
Decision-Tailored Uncertainty Quantification for Probability Estimates.
| Title: | Decision-Tailored Uncertainty Quantification for Probability Estimates. |
|---|---|
| Authors: | Smith, Zachary1 (AUTHOR) zach.smith@utexas.edu, Bickel, J. Eric1 (AUTHOR) ebickel@utexas.edu, Field Jr., Richard2 (AUTHOR) rvfield@sandia.gov, Darling, Michael3 (AUTHOR) mcdarli@sandia.gov |
| Source: | Decision Analysis (INFORMS). Sep2026, Vol. 23 Issue 3, p248-259. 12p. |
| Subject Terms: | *Uncertainty (Information theory), *Forecasting, *Decision making, *Statistics, Loss functions (Statistics), Statistical measurement, Machine learning, Statistical models |
| Abstract: | Proper loss functions are used in decision analysis to elicit probability estimates that reflect an individual's beliefs. In this paper, we consider a decision framework in which the probability estimates come from a statistical or machine learning model. We regard the probability estimates themselves as random variables with respect to the underlying distribution of the data set from which the estimates were derived. We derive a decision-tailored proper loss function, and we show that the generalized variance (based on a generalized bias–variance decomposition that we also derive in the paper) of the probability estimates under this decision-tailored loss quantifies the uncertainty in the probability estimates in a way that is relevant to the decision problem. In particular, points with low decision-tailored generalized variance correspond to points whose optimal decisions are robust with respect to the distribution of the probability estimate at that point, whereas high decision-tailored generalized variance corresponds to higher uncertainty under the distribution of the probability estimate. Funding: This work was supported by the Laboratory Directed Research and Development program at Sandia National Laboratories. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-NA0003525. [ABSTRACT FROM AUTHOR] |
| Copyright of Decision Analysis (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Business Source Index |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: bsx DbLabel: Business Source Index An: 196802360 RelevancyScore: 1515 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1515 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Decision-Tailored Uncertainty Quantification for Probability Estimates. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Smith%2C+Zachary%22">Smith, Zachary</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> zach.smith@utexas.edu</i><br /><searchLink fieldCode="AR" term="%22Bickel%2C+J%2E+Eric%22">Bickel, J. Eric</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> ebickel@utexas.edu</i><br /><searchLink fieldCode="AR" term="%22Field+Jr%2E%2C+Richard%22">Field Jr., Richard</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> rvfield@sandia.gov</i><br /><searchLink fieldCode="AR" term="%22Darling%2C+Michael%22">Darling, Michael</searchLink><relatesTo>3</relatesTo> (AUTHOR)<i> mcdarli@sandia.gov</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Decision+Analysis+%28INFORMS%29%22">Decision Analysis (INFORMS)</searchLink>. Sep2026, Vol. 23 Issue 3, p248-259. 12p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Uncertainty+%28Information+theory%29%22">Uncertainty (Information theory)</searchLink><br />*<searchLink fieldCode="DE" term="%22Forecasting%22">Forecasting</searchLink><br />*<searchLink fieldCode="DE" term="%22Decision+making%22">Decision making</searchLink><br />*<searchLink fieldCode="DE" term="%22Statistics%22">Statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Loss+functions+%28Statistics%29%22">Loss functions (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+measurement%22">Statistical measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+models%22">Statistical models</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Proper loss functions are used in decision analysis to elicit probability estimates that reflect an individual's beliefs. In this paper, we consider a decision framework in which the probability estimates come from a statistical or machine learning model. We regard the probability estimates themselves as random variables with respect to the underlying distribution of the data set from which the estimates were derived. We derive a decision-tailored proper loss function, and we show that the generalized variance (based on a generalized bias–variance decomposition that we also derive in the paper) of the probability estimates under this decision-tailored loss quantifies the uncertainty in the probability estimates in a way that is relevant to the decision problem. In particular, points with low decision-tailored generalized variance correspond to points whose optimal decisions are robust with respect to the distribution of the probability estimate at that point, whereas high decision-tailored generalized variance corresponds to higher uncertainty under the distribution of the probability estimate. Funding: This work was supported by the Laboratory Directed Research and Development program at Sandia National Laboratories. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-NA0003525. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Decision Analysis (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=bsx&AN=196802360 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1287/deca.2025.0421 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 12 StartPage: 248 Subjects: – SubjectFull: Uncertainty (Information theory) Type: general – SubjectFull: Forecasting Type: general – SubjectFull: Decision making Type: general – SubjectFull: Statistics Type: general – SubjectFull: Loss functions (Statistics) Type: general – SubjectFull: Statistical measurement Type: general – SubjectFull: Machine learning Type: general – SubjectFull: Statistical models Type: general Titles: – TitleFull: Decision-Tailored Uncertainty Quantification for Probability Estimates. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Smith, Zachary – PersonEntity: Name: NameFull: Bickel, J. Eric – PersonEntity: Name: NameFull: Field Jr., Richard – PersonEntity: Name: NameFull: Darling, Michael IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 09 Text: Sep2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 15458490 Numbering: – Type: volume Value: 23 – Type: issue Value: 3 Titles: – TitleFull: Decision Analysis (INFORMS) Type: main |
| ResultId | 1 |