Academic Journal
Automated scoring in the era of artificial intelligence: An empirical study with Turkish essays
| Τίτλος: | Automated scoring in the era of artificial intelligence: An empirical study with Turkish essays |
|---|---|
| Συγγραφείς: | Burak Aydın, Tarık Kışla, Nursel Tan Elmas, Okan Bulut |
| Πηγή: | Aydın, B, Kışla, T, Elmas, N T & Bulut, O 2025, ' Automated scoring in the era of artificial intelligence : An empirical study with Turkish essays ', System, vol. 133, 103784 . https://doi.org/10.1016/j.system.2025.103784 |
| Στοιχεία εκδότη: | Elsevier BV, 2025. |
| Έτος έκδοσης: | 2025 |
| Θεματικοί όροι: | Multilevel models, Automated scoring, Turkish essays, Rater reliability, Zero-shot with rubric, name=Linguistics and Language, Large language models, name=Educational science, name=Education, name=Language and Linguistics |
| Περιγραφή: | Automated scoring (AS) has gained significant attention as a tool to enhance the efficiency and reliability of assessment processes. Yet, its application in under-represented languages, such as Turkish, remains limited. This study addresses this gap by empirically evaluating AS for Turkish using a zero-shot approach with a rubric powered by OpenAI's GPT-4o. A dataset of 590 essays written by learners of Turkish as a second language was scored by professional human raters and an artificial intelligence (AI) model integrated via a custom-built interface. The scoring rubric, grounded in the Common European Framework of Reference for Languages, assessed six dimensions of writing quality. Results revealed a strong alignment between human and AI scores with a Quadratic Weighted Kappa of 0.72, Pearson correlation of 0.73, and an overlap measure of 83.5 %. Analysis of rater effects showed minimal influence on score discrepancies, though factors such as experience and gender exhibited modest effects. These findings demonstrate the potential of AI-driven scoring in Turkish, offering valuable insights for broader implementation in under-represented languages, such as the possible source of disagreements between human and AI scores. Conclusions from a specific writing task with a single human rater underscore the need for future research to explore diverse inputs and multiple raters. |
| Τύπος εγγράφου: | Article |
| Γλώσσα: | English |
| ISSN: | 0346-251X |
| DOI: | 10.1016/j.system.2025.103784 |
| Σύνδεσμος πρόσβασης: | http://www.scopus.com/inward/record.url?scp=105010969717&partnerID=8YFLogxK https://doi.org/10.1016/j.system.2025.103784 http://fox.leuphana.de/portal/de/publications/automated-scoring-in-the-era-of-artificial-intelligence(f284135e-c3a8-446c-be6e-35f049a4a7f7).html |
| Rights: | CC BY |
| Αριθμός Καταχώρησης: | edsair.doi.dedup.....4158268bf3b99a3c6b3b3fb15fce3800 |
| Βάση Δεδομένων: | OpenAIRE |
| ISSN: | 0346251X |
|---|---|
| DOI: | 10.1016/j.system.2025.103784 |