Academic Journal
A transition evaluation model with probability-based effectiveness indicators-a new measurement model for problem-solving process data.
| Title: | A transition evaluation model with probability-based effectiveness indicators-a new measurement model for problem-solving process data. |
|---|---|
| Authors: | Wang P; Beijing Key Laboratory of Learning and Cognition, School of Psychology, Capital Normal University, Beijing, China., Han Y; Cognitive Science and Allied Health School, Beijing Language and Culture University, Beijing, China.; Institute of Life and Health Sciences, Beijing Language and Culture University, Beijing, China.; Key Laboratory of Language and Cognitive Science (Ministry of Education), Beijing Language and Culture University, Beijing, China., Liu H; Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education (Beijing Normal University), Faculty of Psychology, Beijing Normal University, No. 19 Xin Jie Kou Wai Street, Beijing, 100875, China. hyliu@bnu.edu.cn.; Research Center for Capacity Building in Educational Assessment and Evaluation (Beijing Higher Education Innovation Center for Philosophy and Social Sciences), Beijing, China. hyliu@bnu.edu.cn. |
| Source: | Behavior research methods [Behav Res Methods] 2026 Apr 16; Vol. 58 (5). Date of Electronic Publication: 2026 Apr 16. |
| Publication Type: | Journal Article |
| Language: | English |
| Journal Info: | Publisher: Springer Country of Publication: United States NLM ID: 101244316 Publication Model: Electronic Cited Medium: Internet ISSN: 1554-3528 (Electronic) Linking ISSN: 1554351X NLM ISO Abbreviation: Behav Res Methods Subsets: MEDLINE |
| Imprint Name(s): | Publication: 2010- : New York : Springer Original Publication: Austin, Tex. : Psychonomic Society, c2005- |
| MeSH Terms: | Problem Solving* , Models, Statistical*, Humans ; Probability ; Computer Simulation |
| Abstract: | Computer-based interactive tasks generate rich process data that capture respondents' problem-solving behaviors, particularly sequences of actions that trigger transitions between problem states. In recent years, the process-based measurement models analyzing transition sequences have emerged as a promising approach for estimating latent problem-solving ability. A fundamental step in developing these models is the predefinition of the transition effectiveness. However, existing effectiveness indicators are often limited to restricted value ranges (e.g., dichotomous or polytomous scales) and theoretical perspective of expert evaluation, thereby constraining the flexibility of process-based models. To address these limitations, this study introduces two probability-based indicators: state effectiveness INLINEMATH and transition effectiveness INLINEMATH . When validated using empirical data from the PISA 2012 problem-solving assessment, the probability-based effectiveness indicators exhibited a broader range of numerical values, enabling finer-grained discrimination among states and transitions. Subsequently, we developed the transition evaluation model (TEM), a process-based model that incorporates the transition effectiveness INLINEMATH to better differentiate transition characteristics. Simulation study demonstrated TEM's robust parameter estimation, satisfactory model-data fit, and high estimation accuracy across diverse conditions. In an empirical study, TEM outperformed three models, including the Sequential Response Model (SRM), the State Response Measurement Model (SRMM), and SRM with Polytomous Effectiveness Indicators (SRM-PEI) in terms of model-data fit, and yields more nuanced transition characteristic curves and interpretable ability estimates. These findings underscore the value of probability-based effectiveness indicators and TEM as advanced tools for analyzing complex problem-solving assessments. (© 2026. The Psychonomic Society, Inc.) |
| Competing Interests: | Declarations. Ethics approval: This empirical study used anonymized data from the publicly available PISA 2012 dataset administered by the OECD. The original PISA project obtained ethical approval and informed consent through national educational authorities in each participating country, following protocols. According to institutional and journal guidelines, no additional ethics approval was required. Consent to participate: All participants had previously provided consent under the original data collection procedures of the PISA project. Consent for publication: Not applicable. Conflicts of interest/Competing interests: No authors reported any financial or other conflicts of interest in relation to the work described. |
| References: | Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51. https://doi.org/10.1007/BF02291411. (PMID: 10.1007/BF02291411) Brooks, S. P., & Gelman, A. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434–455. (PMID: 10.1080/10618600.1998.10474787) Chen, Y. (2020). A continuous-time dynamic choice measurement model for problem-solving process data. Psychometrika, 85(4), 1052–1075. https://doi.org/10.1007/s11336-020-09734-1. (PMID: 10.1007/s11336-020-09734-1333468837826320) Fu, Y., Zhan, P., Chen, Q., & Jiao, H. (2024). Joint modeling of action sequences and action time in computer-based interactive tasks. Behavior Research Methods, 56(5), 4293–4310. https://doi.org/10.3758/s13428-023-02178-2. (PMID: 10.3758/s13428-023-02178-237429984) Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking & Reasoning, 7(1), 69–89. https://doi.org/10.1080/13546780042000046. (PMID: 10.1080/13546780042000046) Geisser, S., & Eddy, W. F. (1979). A predictive approach to model selection. Journal of the American Statistical Association, 74(365), 153–160. (PMID: 10.1080/01621459.1979.10481632) Gelfand, A. E., & Dey, D. K. (1994). Bayesian model choice: Asymptotics and exact calculations. Journal of the Royal Statistical Society. Series B, Statistical Methodology, 56, 501–514. (PMID: 10.1111/j.2517-6161.1994.tb01996.x) Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2014). Bayesian data analysis (3rd ed.). Chapman & Hall/CRC Press. Gelman, A., Meng, X.-L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6, 733–760. Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136. (PMID: 10.1214/ss/1177011136) Goldhammer, F., Hahnel, C., Kroehne, U., & Zehner, F. (2021). From byproduct to design factor: On validating the interpretation of process indicators based on log data. Large-Scale Assessments in Education, 9(1), 1–25. https://doi.org/10.1186/s40536-021-00113-5. (PMID: 10.1186/s40536-021-00113-5) Griffin, P., & Care, E. (2014). Assessment and teaching of 21st century skills: Methods and approach. Springer. Guttman, I. (1967). The use of the concept of a future observation in goodness-of-fit problems. Journal of the Royal Statistical Society. Series B, Methodological, 29(1), 83–100. https://doi.org/10.1111/j.2517-6161.1967.tb00676.x. (PMID: 10.1111/j.2517-6161.1967.tb00676.x) Han, Y., Ji, F., Wang, P., & Liu, H. (2025). Assessing multiple abilities through process data in computer-based assessments: The multidimensional sequential response model (MSRM). Behavior Research Methods. https://doi.org/10.3758/s13428-025-02658-7. (PMID: 10.3758/s13428-025-02658-741162688) Han, Y., Liu, H., & Ji, F. (2022). A sequential response model for analyzing process data on technology-based problem-solving tasks. Multivariate Behavioral Research, 57(6), 960–977. https://doi.org/10.1080/00273171.2021.1932403. (PMID: 10.1080/00273171.2021.193240334224276) He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with N-grams: Insights from a computer-based large-scale assessment. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 750–777). IGI Global. (PMID: 10.4018/978-1-4666-9441-5.ch029) He, S., & Cui, Y. (2025). A systematic review of the use of log-based process data in computer-based assessments. Computers & Education. https://doi.org/10.1016/j.compedu.2025.105245. (PMID: 10.1016/j.compedu.2025.105245) Jiao, H., He, Q., & Veldkamp, B. P. (2021). Process data in educational and psychological measurement. Frontiers in Psychology, 12, Article 793399. https://doi.org/10.3389/fpsyg.2021.793399. (PMID: 10.3389/fpsyg.2021.793399349251948677654) Koehler, E., Brown, E., & Haneuse, J. P. A. (2009). On the assessment of Monte Carlo error in simulation-based statistical analyses. The American Statistician, 63(2), 155–162. https://doi.org/10.1198/tast.2009.0030. (PMID: 10.1198/tast.2009.0030225449723337209) LaMar, M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67–88. https://doi.org/10.1007/s11336-017-9570-0. (PMID: 10.1007/s11336-017-9570-028447309) Levy, R., & Mislevy, R. J. (2016). Bayesian psychometric modeling. CRC Press. Liu, H., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, Article 1372. https://doi.org/10.3389/fpsyg.2018.01372. (PMID: 10.3389/fpsyg.2018.01372301231716085588) OECD. (2012). Literacy, numeracy and problem solving in technology-rich environments: Framework for the OECD Survey of Adult Skills. OECD Publishing. https://doi.org/10.1787/9789264128859-en. (PMID: 10.1787/9789264128859-en) OECD. (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems (volume V). PISA OECD publishing. https://doi.org/10.1787/9789264208070-en. (PMID: 10.1787/9789264208070-en) Patz, R. J., & Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24(4), 342–366. https://doi.org/10.3102/10769986024004342. (PMID: 10.3102/10769986024004342) Patz, R. J., & Junker, B. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178. https://doi.org/10.3102/10769986024002146. (PMID: 10.3102/10769986024002146) Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applies statistician. The Annals of Statistics, 12, 1151–1172. https://doi.org/10.2307/2240995. (PMID: 10.2307/2240995) Shu, Z., Bergner, Y., Zhu, M., Hao, J., & von Davier, A. A. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109–131. Shute, V. J., & Moore, G. R. (2017). Consistency and validity in game-based stealth assessment. In H. Jiao & R. W. Lissitz (Eds.), Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective (pp. 31–51). Information Age Publishing. (PMID: 10.1108/978-1-68123-931-620251003) Spiegelhalter, D. J., Best, N. G., Carlin, B. P., and Van der Linde, A. 1998. Bayesian deviance, the effective number of parameters, and the comparison of arbitrarily complex models. Technical report, MRC Biostatistics Unit. Stadler, M., Fischer, F., & Greiff, S. (2019). Taking a closer look: An exploratory analysis of successful and unsuccessful strategy use in complex problems. Frontiers in Psychology, 10, Article 777. https://doi.org/10.3389/fpsyg.2019.00777. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed., pp, 27–28). MIT Press. Thissen, D., & Steinberg, L. (1986). A taxonomy of item response models. Psychometrika, 51, 567–577. (PMID: 10.1007/BF02295596) Ulitzsch, E., He, Q., & Pohl, S. (2022). Using sequence mining techniques for understanding incorrect behavioral patterns on interactive tasks. Journal of Educational and Behavioral Statistics, 47(1), 3–35. (PMID: 10.3102/10769986211010467) von Davier, A. A., Mislevy, R. J., Hao, J., & (Eds.). (2022). Computational psychometrics: New methodologies for a new generation of digital learning and assessment: With examples in R and Python. Springer Nature. Wang, P., & Liu, H. (2024). Polytomous effectiveness indicators in complex problem-solving tasks and their applications in developing measurement model. Psychometrika, 89, 877–902. https://doi.org/10.1007/s11336-024-09963-8. (PMID: 10.1007/s11336-024-09963-838592619) Xiao, Y., He, Q., Veldkamp, B., & Liu, H. (2021). Exploring latent states of problem-solving competence using hidden Markov model on process data. Journal of Computer Assisted Learning, 37(5), 1232–1247. https://doi.org/10.1111/jcal.12559. (PMID: 10.1111/jcal.12559) Xiao, Y., & Liu, H. (2024). A state response measurement model for problem-solving process data. Behavior Research Methods, 56(1), 258–277. https://doi.org/10.3758/s13428-022-02042-9. (PMID: 10.3758/s13428-022-02042-936597007) Xiao, Y., & Liu, H. (2025). A mixture modeling approach to detect different behavioral patterns for process data. Fudan Journal of the Humanities and Social Sciences, 18(1), 79–113. https://doi.org/10.1007/s40647-024-00405-4. (PMID: 10.1007/s40647-024-00405-4) |
| Grant Information: | 2025M773450 China Postdoctoral Science Foundation; 24JZDW003 Ministry of Education Key Projects of Philosophy and Social Sciences Research |
| Contributed Indexing: | Keywords: Probability-based effectiveness indicators; Problem-solving assessment; Process data; State and transition effectiveness; Transition evaluation model |
| Entry Date(s): | Date Created: 20260416 Date Completed: 20260715 Latest Revision: 20260715 |
| Update Code: | 20260715 |
| DOI: | 10.3758/s13428-026-02994-2 |
| PMID: | 41991869 |
| Database: | MEDLINE |
| ISSN: | 1554-3528 |
|---|---|
| DOI: | 10.3758/s13428-026-02994-2 |