Academic Journal
Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems
| Τίτλος: | Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems |
|---|---|
| Γλώσσα: | English |
| Συγγραφείς: | Alexander Kwako (ORCID |
| Πηγή: | Educational Measurement: Issues and Practice. 2026 45(2). |
| Διαθεσιμότητα: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 14 |
| Ημερομηνία έκδοσης: | 2026 |
| Τύπος εγγράφου: | Journal Articles Reports - Research |
| Descriptors: | Models, Confidence Testing, Scores, Scoring, Computer Assisted Testing, Man Machine Systems, Automation, Item Response Theory |
| DOI: | 10.1111/emip.70019 |
| ISSN: | 0731-1745 1745-3992 |
| Περίληψη: | In large-scale assessments, constructed response items are often scored using hybrid scoring systems, which combine human and automated scores. In this study, we augment automated scoring with confidence modeling to strategically route difficult-to-score responses for human review. We utilize "hybrid performance curves" to visualize the impact of routing on performance. Additionally, we propose several "hybrid scoring policies" for selecting optimal routing thresholds given practical constraints. Our findings reveal that hybrid scoring systems can achieve an overall performance that exceeds that of human- and automated-only systems. Moreover, the superior performance of the hybrid system is less expensive than a human-only system. These findings highlight the complementarity of human raters and automated scoring engines. Although current standards focus on the performance of human raters and automated scoring engines in isolation, we recommend that practitioners also report on the performance of the hybrid scoring system as a whole. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Αριθμός Καταχώρησης: | EJ1506976 |
| Βάση Δεδομένων: | ERIC |
| ISSN: | 0731-1745 1745-3992 |
|---|---|
| DOI: | 10.1111/emip.70019 |