Academic Journal
Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming.
| Τίτλος: | Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming. |
|---|---|
| Συγγραφείς: | Cocchieri, Alessio1 (AUTHOR) a.cocchieri@unibo.it, Ragazzi, Luca1 (AUTHOR) l.ragazzi@unibo.it, Aguzzi, Gianluca1 (AUTHOR) gianluca.aguzzi@unibo.it, Frisoni, Giacomo1 (AUTHOR) giacomo.frisoni@unibo.it, Molfetta, Lorenzo1 (AUTHOR) lorenzo.molfetta@unibo.it, Moro, Gianluca1 (AUTHOR) gianluca.moro@unibo.it, Viroli, Mirko1 (AUTHOR) mirko.viroli@unibo.it |
| Πηγή: | Journal of Systems & Software. Dec2026, Vol. 242, pN.PAG-N.PAG. 1p. |
| Θεματικοί όροι: | *Object-oriented programming, Java programming language, Model validation, Language models, Benchmark problems (Computer science), Computer programming education, Computer software testing |
| Περίληψη: | Current code generation benchmarks largely overlook object-oriented programming (OOP) skills, leaving open whether large language models (LLMs) can effectively apply OOP principles to structured programming tasks. The dominance and permissiveness of Python in most benchmarks further obscure model weaknesses in core OOP principles. We introduce the Java Academic Benchmark (JAB), based on 103 authentic Java exams collected over a decade at a major university, together with 506 expert-written JUnit tests, to rigorously evaluate advanced Java programming competence with a strong focus on OOP. To complement execution-based scoring, we propose KODE, an LLM-as-a-Judge framework that assesses OOP adherence across four pedagogical dimensions. We evaluate 27 LLMs under two resolution strategies: single-attempt (one-shot) and agentic (iterative refinement with compiler and test feedback). Results reveal that, under our evaluation protocol, larger closed models match or surpass bachelor-level OOP students – especially under agentic resolution – while smaller open models lag behind. JAB's class-level design enables fine-grained error analysis, exposing recurring misconceptions. Editor's note: Open Science material was validated by the Journal of Systems and Software Open Science Board. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Systems & Software is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Βάση Δεδομένων: | Business Source Index |
καταχωρήστε σχόλιο πρώτοι!