Academic Journal
A state response measurement model for problem-solving process data.
| Τίτλος: | A state response measurement model for problem-solving process data. |
|---|---|
| Συγγραφείς: | Xiao Y; Department of Educational Psychology, Faculty of Education, East China Normal University, Shanghai, China.; Faculty of Psychology, Beijing Normal University, Beijing, China., Liu H; Faculty of Psychology, Beijing Normal University, Beijing, China. hyliu@bnu.edu.cn.; Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, China. hyliu@bnu.edu.cn. |
| Πηγή: | Behavior research methods [Behav Res Methods] 2024 Jan; Vol. 56 (1), pp. 258-277. Date of Electronic Publication: 2023 Jan 03. |
| Τύπος έκδοσης: | Journal Article; Research Support, Non-U.S. Gov't |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: Springer Country of Publication: United States NLM ID: 101244316 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1554-3528 (Electronic) Linking ISSN: 1554351X NLM ISO Abbreviation: Behav Res Methods Subsets: MEDLINE |
| Imprint Name(s): | Publication: 2010- : New York : Springer Original Publication: Austin, Tex. : Psychonomic Society, c2005- |
| Ιατρικοί όροι (MeSH): | Problem Solving*, Humans ; Computer Simulation ; Bayes Theorem |
| Περίληψη: | In computer simulation-based interactive tasks, different people make different response processes to the same tasks, resulting in various action sequences. These sequences contain rich information, not only about respondents, but also about tasks. In this study, we propose a state response (SR) measurement model with a Bayesian approach for analyzing the process sequences, which assumes that each action made is determined by the individual's problem-solving ability and the easiness of the current problem state. This model is closer to reality compared with the action sub-model (referred to as DC model) of Chen's (2020) continuous-time dynamic choice (CTDC) measurement model that defines the easiness parameter only at the task level and ignores the task's process characteristics. The simulation study showed that the SR model performed well in parameter estimation. Moreover, the estimation accuracy of the SR model was quite similar to that of the DC model when state easiness parameters were equal within the task, but was much higher when within-task state easiness parameters were unequal. For the empirical data from the Program for International Student Assessment 2012, the SR model showed better model fit than the DC model. The estimates for state easiness parameters within each task were obviously different and made sense for characterizing task steps, further demonstrating the rationality of the proposed SR model. (© 2022. The Psychonomic Society, Inc.) |
| References: | Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723. (PMID: 10.1109/TAC.1974.1100705) Arieli-Attali, M., Ou, L., & Simmering, V. R. (2019). Understanding test Takers' choices in a self-adapted test: A hidden Markov modeling of process data. Frontiers in Psychology, 10, Article 83. https://doi.org/10.3389/fpsyg.2019.00083. (PMID: 10.3389/fpsyg.2019.0008330787889) Battauz, M. (2020). Regularized estimation of the nominal response model. Multivariate Behavioral Research, 55(6), 811–824. https://doi.org/10.1080/00273171.2019.1681252. (PMID: 10.1080/00273171.2019.168125231682150) Brooks, S. P., & Gelman, A. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7(4), 434–455. Buchner, A., & Funke, J. (1993). Finite-state automata: Dynamic task environments in problem-solving research. The Quarterly Journal of Experimental Psychology, 46(1), 83–118. (PMID: 10.1080/14640749308401068) Chen, Y. (2020). A continuous-time dynamic choice measurement model for problem-solving process data. Psychometrika, 85(4), 1052–1075. https://doi.org/10.1007/s11336-020-09734-1. (PMID: 10.1007/s11336-020-09734-1333468837826320) Chen, Y., Li, X., Liu, J., & Ying, Z. (2021). Item Response Theory — A Statistical Framework for Educational and Psychological Measurement. arXiv:2108.08604. https://arxiv.org/abs/2108.08604. Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking & Reasoning, 7, 69–89. (PMID: 10.1080/13546780042000046) Fox, J. P. (2010). Bayesian Item Response Modeling: Theory and Application. Springer. (PMID: 10.1007/978-1-4419-0742-4) Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–511. (PMID: 10.1214/ss/1177011136) Geisser, S., & Eddy, W. F. (1979). A predictive approach to model selection. Journal of the American Statistical Association, 74, 153–160. (PMID: 10.1080/01621459.1979.10481632) Gelfand, A. E., & Dey, D. K. (1994). Bayesian model choice: Asymptotics and exact calculations. Journal of the Royal Statistical Society, Series B, 56, 501–514. Gilholm, P., Mengersen, K., & Thompson, H. (2021). Bayesian Hierarchical Multidimensional Item Response Modeling of Small Sample, Sparse Data for Personalized Developmental Surveillance. Educational and Psychological Measurement, 81(5), 936–956. https://doi.org/10.1177/0013164420987582. (PMID: 10.1177/0013164420987582345658128377345) Greiff, S., Wüstenberg, S., & Avvisati, F. (2015). Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92–105. (PMID: 10.1016/j.compedu.2015.10.018) Greiff, S., Niepel, C., Scherer, R., & Martin, R. (2016). Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior, 61, 36–46. (PMID: 10.1016/j.chb.2016.02.095) Han, Y., Liu, H., & Ji, F. (2021). A Sequential Response Model for Analyzing Process Data on Technology-Based Problem-Solving Tasks. Multivariate Behavioral Research. Advance Online Publication. https://doi.org/10.1080/00273171.2021.1932403. He, Q., Borgonovi, F., & Paccagnella, M. (2019). Using Process Data to Understand Adults' Problem-Solving Behaviour in the Programme for the International Assessment of Adult Competencies (PIAAC): Identifying Generalised Patterns Across Multiple Tasks with Sequence Mining. OECD Education Working Papers (OECD Publishing). https://doi.org/10.1787/650918f2-en. He, Q., Borgonovi, F., & Paccagnella, M. (2021). Leveraging process data to assess adults' problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166, Article 104170. https://doi.org/10.1016/j.compedu.2021.104170. He, Q., & von Davier, M. (2015). Identifying feature sequences from process data in problem-solving items with n-grams. In L. A. van der Ark, D. M. Bolt, W.-C. Wang, J. A. Douglas, & S.-M. Chow (Eds.), Quantitative psychology research (pp. 173–190). Springer. https://doi.org/10.1007/978-3-319-19977-1_13. (PMID: 10.1007/978-3-319-19977-1_13) He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with ngrams: Insights from a computer-based large-scale assessment. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 750–777). IGI Global. (PMID: 10.4018/978-1-4666-9441-5.ch029) Junker, B. W., Patz, R. J., & VanHoudnos, N. M. (2016). Markov chain Monte Carlo for item response models. In W. J. van der Linden (Ed.), Handbook of item response theory, volume two: statistical tools (pp. 271–325). CRC Press. Kerr, D., Chung, G., & Iseli, M. (2011). The feasibility of using cluster analysis to examine log data from educational video games (CRESST Report No. 790). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Center for Studies in Education, UCLA. https://files.eric.ed.gov/fulltext/ED520531.pdf. Kim, J. S., & Bolt, D. M. (2007). Estimating item response theory models using Markov chain Monte Carlo methods. Educational Measurement: Issues and Practice, 26(4), 38–51. (PMID: 10.1111/j.1745-3992.2007.00107.x) König, C., Spoden, C., & Frey, A. (2020). An Optimized Bayesian Hierarchical Two-Parameter Logistic Model for Small Sample Item Calibration. Applied Psychological Measurement, 44(4), 311–326. http://doi.org/ https://doi.org/10.1177/0146621619893786. LaMar, M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67–88. https://doi.org/10.1007/s11336-017-9570-0. (PMID: 10.1007/s11336-017-9570-028447309) Levy, R., & Mislevy, R. J. (2016). Bayesian psychometric modeling. CRC Press. Liao, D., He, Q., & Jiao, H. (2019). Mapping background variables with sequential patterns in problem-solving environments: An investigation of US Adults' employment status in PIAAC. Frontiers in Psychology, 10, 646. https://doi.org/10.3389/fpsyg.2019.00646. (PMID: 10.3389/fpsyg.2019.00646309719866445889) Liu, H., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, Article 1372. https://doi.org/10.3389/fpsyg.2018.01372. OECD. (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems (Volume V). OECD Publishing. https://www.oecd-ilibrary.org/education/pisa-2012-results-skills-for-life-volume-v_9789264208070-en. Patz, R. J., & Junker, B. W. (1999a). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational Behavioral Statistics, 24(4), 342–366. https://doi.org/10.3102/10769986024004342. (PMID: 10.3102/10769986024004342) Patz, R. J., & Junker, B. W. (1999b). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational Behavioral Statistics, 24(2), 146–178. https://doi.org/10.3102/10769986024002146. (PMID: 10.3102/10769986024002146) Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: a didactic. Frontiers in psychology, 9, 2231. (PMID: 10.3389/fpsyg.2018.02231305327166265513) R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Rosenthal, J. S. (2011). Optimal proposal distributions and adaptive MCMC. In S. Brooks, A. Gelman, G. L. Jones, & X.-L. Meng (Eds.), Handbook of Markov Chain Monte Carlo (pp. 93–111). Chapman and Hall/CRC. Sclove, L. S. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333–343. (PMID: 10.1007/BF02294360) Shu, Z., Bergner, Y., Zhu, M., Hao, J., & von Davier, A. A. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109–131. Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van der Linde, A. (1998). Bayesian deviance, the effective number of parameters, and the comparison of arbitrarily complex models. Technical report, MRC Biostatistics Unit. Stadler, M., Hofer, S., & Greiff, S. (2020). First among equals: Log data indicates ability differences despite equal scores. Computers in Human Behavior, 111, 106442. https://doi.org/10.1016/j.chb.2020.106442. (PMID: 10.1016/j.chb.2020.106442) Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. (PMID: 10.1214/aos/1176344136) Tang, X., Wang, Z., He, Q., Liu, J., & Ying, Z. (2020). Latent Feature Extraction for Process Data via Multidimensional Scaling. Psychometrika, 85(2), 378–397. (PMID: 10.1007/s11336-020-09708-332572672) Tang, X., Wang, Z., Liu, J., & Ying, Z. (2021). An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology, 74(1), 1–33. https://doi.org/10.1111/bmsp.12203. (PMID: 10.1111/bmsp.1220332442346) Xiao, Y., He, Q., Veldkamp, B., & Liu, H. (2021). Exploring latent states of problem-solving competence using hidden Markov model on process data. Journal of Computer Assisted Learning. Advanced Online. Publication. https://doi.org/10.1111/jcal.12559. Zhan, P., & Qiao, X. (2022). Diagnostic Classification Analysis of Problem-Solving Competency Using Process Data: An Item Expansion Method. Psychometrika. Advanced Online Publication. https://doi.org/10.1007/s11336-022-09855-9. |
| Contributed Indexing: | Keywords: Measurement modeling; Problem-solving; Process data; State response model |
| Entry Date(s): | Date Created: 20230103 Date Completed: 20240118 Latest Revision: 20240906 |
| Update Code: | 20260130 |
| DOI: | 10.3758/s13428-022-02042-9 |
| PMID: | 36597007 |
| Βάση Δεδομένων: | MEDLINE |
καταχωρήστε σχόλιο πρώτοι!