Academic Journal
Comparing Zero-Shot Large Language Model Prompting with Human Coding of Theory Concepts in Student Essays
| Τίτλος: | Comparing Zero-Shot Large Language Model Prompting with Human Coding of Theory Concepts in Student Essays |
|---|---|
| Γλώσσα: | English |
| Συγγραφείς: | Shelley Keith, Philip I. Pavlik, Kristen L. Stives, Laura Jean Kerr |
| Πηγή: | Journal of Educational Data Mining. 2026 18(1):286-317. |
| Διαθεσιμότητα: | International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM |
| Peer Reviewed: | Y |
| Page Count: | 32 |
| Ημερομηνία έκδοσης: | 2026 |
| Τύπος εγγράφου: | Journal Articles Reports - Research Tests/Questionnaires |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Artificial Intelligence, Natural Language Processing, Prompting, Writing Evaluation, Essays, Scoring, Automation, Computer Assisted Testing, Coding, Accuracy, College Students, Error Patterns, Bias, Criminology, Required Courses, Majors (Students), Computation, Correlation |
| ISSN: | 2157-2100 |
| Περίληψη: | Recent studies have explored the cost and time benefits of using artificial intelligence (AI), particularly large language models (LLMs), in coding student essays. While these models show promise, not enough is understood about the factors that affect how their qualitative coding performance compares to human coding. This study examines coding accuracy for content errors in college student essays on criminological theories by comparing human-coded results with outputs from four LLMs. We evaluated human-AI correlations, AI error, and AI bias across four LLMs, five prompt types, three theory content coding dimensions, and four criminological theories. Results indicate that LLM choice significantly influenced human-AI correspondence, with Claude Sonnet 4 exhibiting the best overall performance and GPT 4.1 Mini the worst. Prompt type had minimal impact on performance. Across models, error rates were lowest when identifying whether students listed a concept, and highest when assessing whether definitions were correct. LLMs performed better on concise theories than on more complex ones. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Αριθμός Καταχώρησης: | EJ1506614 |
| Βάση Δεδομένων: | ERIC |
| ISSN: | 2157-2100 |
|---|