Academic Journal
Προβολή σε EDS
Συνδεδεμένο Πλήρες Κείμενο

Joint modeling of action sequences and action time in computer-based interactive tasks.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Joint modeling of action sequences and action time in computer-based interactive tasks.
Συγγραφείς: Fu Y; School of Psychology, Zhejiang Normal University, Jinhua, China., Zhan P; School of Psychology, Zhejiang Normal University, Jinhua, China. pdzhan@gmail.com.; Intelligent Laboratory of Child and Adolescent Mental Health and Crisis Intervention of Zhejiang Province, Zhejiang Normal University, Jinhua, China. pdzhan@gmail.com., Chen Q; School of Psychology, Zhejiang Normal University, Jinhua, China., Jiao H; Human Development and Quantitative Methodology, University of Maryland, College Park, MD, USA.
Πηγή: Behavior research methods [Behav Res Methods] 2024 Aug; Vol. 56 (5), pp. 4293-4310. Date of Electronic Publication: 2023 Jul 10.
Τύπος έκδοσης: Journal Article; Research Support, Non-U.S. Gov't
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: Springer Country of Publication: United States NLM ID: 101244316 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1554-3528 (Electronic) Linking ISSN: 1554351X NLM ISO Abbreviation: Behav Res Methods Subsets: MEDLINE
Imprint Name(s): Publication: 2010- : New York : Springer
Original Publication: Austin, Tex. : Psychonomic Society, c2005-
Ιατρικοί όροι (MeSH): Problem Solving*/physiology, Humans ; Computer Simulation ; Time Factors ; Models, Statistical
Περίληψη: Process data refers to data recorded in computer-based assessments that reflect the problem-solving processes of participants and provide greater insight into how they solve problems. Action time, namely the amount of time required to complete a state transition, is also included in such data along with actions. In this study, an action-level joint model of action sequences and action time is proposed, in which the sequential response model (SRM) is used as the measurement model for action sequences, and a new log-normal action time model is proposed as the measurement model for action time. The proposed model can be regarded as an extension of the SRM by incorporating action time within the joint-hierarchical modeling framework and as an extension of the conventional item-level joint models in process data analysis. Results of the empirical and simulation studies demonstrated that the model setup was justified, model parameters could be interpreted, parameter estimates were accurate, and taking into account participants' action time further was beneficial for obtaining a deep understanding of participants' behavioral patterns. Overall, the proposed action-level joint model provides an innovative modeling framework for analyzing process data in computer-based assessments from the latent variable modeling perspective.
(© 2023. The Psychonomic Society, Inc.)
References: Albert, D., & Steinberg, L. (2011). Age differences in strategic planning as indexed by the Tower of London. Child Development, 82(5), 1501–1517. https://doi.org/10.1111/j.1467-8624.2011.01613.x. (PMID: 10.1111/j.1467-8624.2011.01613.x21679178)
Anderson, J. R., Funke, J., & Plata, G. (Eds.). (2007). Cognitive psychologic (6 Aufl.). Spektrum Akademischer Verlag http://www.gbv.de/dms/bs/toc/529836963.pdf.
Arieli-Attali, M., Ou, L., & Simmering, V. R. (2019). Understanding test takers' choices in a self-adapted test: A hidden Markov modeling of process data. Frontiers in Psychology, 10, 83. (PMID: 307878896372528)
Bergner, Y., & von Davier, A. A. (2019). Process data in NAEP: Past, present, and future. Journal of Educational and Behavioral Statistics, 44(6), 706–732.
Bergner, Y., Walker, E., & Ogan, A. (2017). Dynamic Bayesian network models for peer tutoring interactions. In A. A. von Davier, M. Zhu, & P. C. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 249–268). Springer.
Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51. https://doi.org/10.1007/BF02291411. (PMID: 10.1007/BF02291411)
Bolsinova, M., & Tijmstra, J. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71, 13–38. (PMID: 28635139)
Chen, Y. (2020). A continuous-time dynamic choice measurement model for problem-solving process data. Psychometrika, 85(4), 1052–1075. https://doi.org/10.1007/s11336-020-09734-1. (PMID: 10.1007/s11336-020-09734-1333468837826320)
De Boeck, P., & Jeon, M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102. (PMID: 307878916372526)
Eichmann, B., Goldhammer, F., Greiff, S., Pucite, L., & Naumann, J. (2019). The role of planning in complex problem solving. Computers & Education, 128, 1–12. https://doi.org/10.1016/j.compedu.2018.08.004. (PMID: 10.1016/j.compedu.2018.08.004)
Fox, J. P., Entink, R. K., & van der Linden, W. (2007). Modeling of responses and response times with the package cirt. Journal of Statistical Software, 20, 1–14.
Fox, J. P., & Marianti, S. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51(4), 540–553. (PMID: 27269482)
Frederiksen, N., Glaser, R., Lesgold, A., & Shafto, M. (1990). Diagnostic monitoring of skill and knowledge acquisition. Lawrence Erlbaum Associates.
Fu, Y., Chen, Q., & Zhan, P. (2023). Binary modeling of action sequences in problem-solving tasks: One- and two-parameter action sequence model. Acta Psychologica Sinica, 55(8), 1383–1396.
Gelfand, A. E., Dey, D. K., & Chang, H. (1992). Model determination using predictive distributions with implementation via sampling-based methods. Stanford Univ CA Dept of Statistics.
Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472.
Gelman, A., Meng, X.-L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6(4), 733–760.
Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608.
Guttman, I. (1967). The use of the concept of a future observation in goodness-of-fit problems. Journal of the Royal Statistical Society: Series B (Methodological), 29(1), 83–100.
Han, Y., Liu, H., & Ji, F. (2022a). A sequential response model for analyzing process data on technology-based problem-solving tasks. Multivariate Behavioral Research, 57(6), 960–977. https://doi.org/10.1080/00273171.2021.1932403. (PMID: 10.1080/00273171.2021.193240334224276)
Han, Y., & Wilson, M. (2022). Analyzing student response processes to evaluate success on a technology-based problem-solving task. Applied Measurement in Education, 35(1), 33–45.
Han, Y., Xiao, Y., & Liu, H. (2022b). Feature extraction and ability estimation of process data in the problem-solving test. Advances in Psychological Science, 30(6), 1393–1409.
Hao, J., Shu, Z., & von Davier, A. (2015). Analyzing process data from game/scenario-based tasks: an edit distance approach. Journal of Educational Data Mining, 7(1), 33–50.
Harding, S. M. E., Griffin, P. E., Awwal, N., Alom, B. M., & Scoular, C. (2017). Measuring collaborative problem-solving using mathematics-based tasks. AERA Open, 3(3), 1–19.
He, Q., Borgonovi, F., & Paccagnella, M. (2021). Leveraging process data to assess adults’ problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166, 104170.
He, Q., Liao, D., & Jiao, H. (2019). Clustering behavioral patterns using process data in PIAAC problem-solving items. In Theoretical and practical advances in computer-based educational measurement (pp. 189–212). Springer.
He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with N-grams: Insights from a computer-based large-scale assessment. In R. Yigal, F. Steve, & M. Maryam (Eds.), Handbook of research on technology tools for real-world skill development (pp. 749–776). Information Science Reference.
Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative problem-solving skills. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach(pp. 37–56). Dordrecht: Springer.
Hoffman, M. D., & Gelman, A. (2014). The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593–1623.
Klein Entink, R. H., Fox, J. P., & van der Linden, W. J. (2009a). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74(1), 21–48. (PMID: 20037635)
Klein Entink, R. H., van der Linden, W. J., & Fox, J.-P. (2009b). A box-cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62, 621–640. (PMID: 19187574)
LaMar, M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67–88. (PMID: 28447309)
Levy, R. (2019). Dynamic Bayesian network modeling of game-based diagnostic assessments. Multivariate Behavioral Research, 54(6), 771–794. (PMID: 30942094)
Liu, H., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372. (PMID: 301231716085588)
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. https://doi.org/10.1007/BF02296272. (PMID: 10.1007/BF02296272)
Mislevy, R. J. (2019). Advances in measurement and cognition. The ANNALS of the American Academy of Political and Social Science, 683(1), 164–182.
Molenaar, D., Tuerlinckx, F., & van der Maas, H. L. (2015). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68(2), 197–219. (PMID: 25109494)
Molenaar, D., Oberski, D., Vermunt, J., & De Boeck, P. (2016). Hidden Markov item response theory models for responses and response times. Multivariate Behavioral Research, 51(5), 606–626. (PMID: 27712114)
Naumann, J., & Goldhammer, F. (2017). Time-on-task effects in digital reading are non-linear and moderated by persons’ skills and tasks’ demands. Learning and Individual Differences, 53, 1–16. https://doi.org/10.1016/j.lindif.2016.10.002. (PMID: 10.1016/j.lindif.2016.10.002)
Newell, A., & Simon, H. A. (1972). Human problem solving (Vol. 104, No. 9). Prentice-Hall.
OECD. (2012). Literacy, numeracy and problem solving in technology-rich environments: Framework for the OECD Survey of Adult Skills. OECD Publishing. https://doi.org/10.1787/9789264128859-en.
OECD. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. OECD Publishing. https://doi.org/10.1787/9789264190511-en.
OECD. (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems (Volume V). OECD Publishing. https://www.oecd-ilibrary.org/education/pisa-2012-results-skills-for-life-volume-v_9789264208070-en.
OECD. (2017). PISA 2015 technical report. OECD Publishing. https://www.oecd.org/pisa/sitedocument/PISA-2015-technical-report-final.pdf.
Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9, 2231. (PMID: 305327166265513)
Reckase, M. (2009). Multidimensional Item Response Theory. Springer.
Rosen, Y. (2017). Assessing students in human-to-agent settings to inform collaborative problem-solving learning. Journal of Educational Measurement, 54(1), 36–53.
Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. The Annals of Statistics, 12(4), 1151–1172.
Scherer, R., Greiff, S., & Hautamäki, J. (2015). Exploring the relation between time on task and ability in complex problem solving. Intelligence, 48, 37–50.
Shu, Z., Bergner, Y., Zhu, M., Hao, J., & von Davier, A. A. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109–131.
Shute, V. J., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in games. MIT Press. https://doi.org/10.7551/mitpress/9589.001.0001. (PMID: 10.7551/mitpress/9589.001.0001)
Siddiq, F., Gochyyev, P., & Wilson, M. (2017). Learning in Digital Networks–ICT literacy: A novel assessment of students' 21st century skills. Computers & Education, 109, 11–37.
Simon, H. A., & Newell, A. (1971). Human problem solving: The state of the theory in 1970. American Psychologist, 26(2), 145–159. https://doi.org/10.1037/h0030806. (PMID: 10.1037/h0030806)
Stelter, A., Goldhammer, F., Naumann, J., & Rölke, H. (2015). Die automatisierung prozeduralen wissens. In J. Stiller & C. Laschke (Eds.), Eine analysebasierend auf prozessdaten (pp. 111–131). Peter Lang Edition.
Tang, S., Peterson, J. C., & Pardos, Z. A. (2016). Deep neural networks and how they apply to sequential education data. In Proceedings of the third (2016) ACM conference on learning@ scale. ACM (pp. 321–324).
Tang, X., Wang, Z., He, Q., Liu, J., & Ying, Z. (2020). Latent feature extraction for process data via multidimensional scaling. Psychometrika, 85(2), 378–397. (PMID: 32572672)
Tang, X., Wang, Z., Liu, J., & Ying, Z. (2021). An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology, 74(1), 1–33. (PMID: 32442346)
Ulitzsch, E., He, Q., & Pohl, S. (2022). Using sequence mining techniques for understanding incorrect behavioral patterns on interactive tasks. Journal of Educational and Behavioral Statistics, 47(1), 3–35.
Ulitzsch, E., He, Q., Ulitzsch, V., Molter, H., Nichterlein, A., Niedermeier, R., & Pohl, S. (2021). Combining clickstream analyses and graph-modeled data clustering for identifying common response processes. Psychometrika, 86, 190–214. (PMID: 335443008035117)
Ulitzsch, E., von Davier, M., & Pohl, S. (2020). Using response times for joint modeling of response and omission behavior. Multivariate Behavioral Research, 55(3), 425–453. (PMID: 31448968)
Van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.
Van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308.
Van der Linden, W. J., & Fox, J.-P. (2015). Joint hierarchical modeling of responses and response times. In W. J. van der Lin-den (Ed.), Handbook of Item Response Theory (Vol. 1). FL: Chapman & Hall/CRC.
Vista, A., Care, E., & Awwal, N. (2017). Visualising and examining sequential actions as behavioural paths that can be interpreted as markers of complex behaviours. Computers in Human Behavior, 76, 656–671.
Von Davier, A. A., Mislevy, R. J., Hao, J., & (Eds.). (2022). Computational psychometrics: New methodologies for a new generation of digital learning and assessment: With examples in R and Python. Springer Nature.
Wang, C., Chang, H., & Douglas, J. (2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66, 144–168. (PMID: 22506914)
Watanabe, S., & Opper, M. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11(12), 3571–3594.
Wilson, M., Gochyyev, P., & Scalise, K. (2017). Modeling data from collaborative assessments: learning in digital interactive social networks. Journal of Educational Measurement, 54(1), 85–102.
Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.
Xiao, Y., He, Q., Veldkamp, B., & Liu, H. (2021). Exploring latent states of problem-solving competence using hidden Markov model on process data. Journal of Computer Assisted Learning, 37(5), 1232–1247.
Xiao, Y., & Liu, H. (2023). A state response measurement model for problem-solving process data. Behavior Research Methods. https://doi.org/10.3758/s13428-022-02042-9.
Yamaguchi, K., & Fujita, K. (2022). Bayesian estimation of test engagement behavior models with response times. Psyarxiv. Retrieved from psyarxiv.com/379pr.
Yuan, J., Xiao, Y., & Liu, H. (2019). Assessment of collaborative problem solving based on process stream data: A new paradigm for extracting indicators and modeling dyad data. Frontiers in Psychology, 10, 369. (PMID: 308633446399305)
Zhan, P., Chen, Q., Wang, S., & Zhang, X. (2023). Longitudinal joint modeling for assessing parallel interactive development of latent ability and processing speed using responses and response times. Behavior Research Methods. Online First. https://doi.org/10.3758/s13428-023-02113-5.
Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71, 262–286. (PMID: 28872185)
Zhan, P., Jiao, H., Man, K., Wang, W.-C., & He, K. (2021). Variable speed across dimensions of ability in the joint model for responses and response times. Frontiers in Psychology, 12, 469196. (PMID: 338544548039373)
Zhan, P., Man, K., Wind, S., & Malone, J. (2022). Cognitive diagnosis modeling incorporating response times and fixation counts: Providing comprehensive feedback and accurate diagnosis. Journal of Educational and Behavioral Statistics, 47(6), 736–776.
Zhan, P., & Qiao, X. (2022). Diagnostic classification analysis of problem-solving competence using process data: An item expansion method. Psychometrika, 87, 1529–1547. (PMID: 35389193)
Zhan, S., Hao, J., & Davier, A. V. (2015). Analyzing process data from game/scenario-based tasks: An edit distance approach. Journal of Educational Data Mining, 7(1), 33–50.
Zhang, S., Wang, Z., Qi, J., Liu, J., & Ying, Z. (2023). Accurate assessment via process data. Psychometrika, 88, 76–97. (PMID: 35962849)
Zhu, M., Shu, Z., & von Davier, A. A. (2016). Using networks to visualize and analyze process data for educational assessment. Journal of Educational Measurement, 53(2), 190–211.
Contributed Indexing: Keywords: Action sequence; Action time; Item response theory; Joint modeling; Process data
Entry Date(s): Date Created: 20230710 Date Completed: 20240730 Latest Revision: 20241028
Update Code: 20260130
DOI: 10.3758/s13428-023-02178-2
PMID: 37429984
Βάση Δεδομένων: MEDLINE
Περιγραφή
ISSN:1554-3528
DOI:10.3758/s13428-023-02178-2