Academic Journal
Προβολή σε EDS

Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming.
Συγγραφείς: Cocchieri, Alessio1 (AUTHOR) a.cocchieri@unibo.it, Ragazzi, Luca1 (AUTHOR) l.ragazzi@unibo.it, Aguzzi, Gianluca1 (AUTHOR) gianluca.aguzzi@unibo.it, Frisoni, Giacomo1 (AUTHOR) giacomo.frisoni@unibo.it, Molfetta, Lorenzo1 (AUTHOR) lorenzo.molfetta@unibo.it, Moro, Gianluca1 (AUTHOR) gianluca.moro@unibo.it, Viroli, Mirko1 (AUTHOR) mirko.viroli@unibo.it
Πηγή: Journal of Systems & Software. Dec2026, Vol. 242, pN.PAG-N.PAG. 1p.
Θεματικοί όροι: *Object-oriented programming, Java programming language, Model validation, Language models, Benchmark problems (Computer science), Computer programming education, Computer software testing
Περίληψη: Current code generation benchmarks largely overlook object-oriented programming (OOP) skills, leaving open whether large language models (LLMs) can effectively apply OOP principles to structured programming tasks. The dominance and permissiveness of Python in most benchmarks further obscure model weaknesses in core OOP principles. We introduce the Java Academic Benchmark (JAB), based on 103 authentic Java exams collected over a decade at a major university, together with 506 expert-written JUnit tests, to rigorously evaluate advanced Java programming competence with a strong focus on OOP. To complement execution-based scoring, we propose KODE, an LLM-as-a-Judge framework that assesses OOP adherence across four pedagogical dimensions. We evaluate 27 LLMs under two resolution strategies: single-attempt (one-shot) and agentic (iterative refinement with compiler and test feedback). Results reveal that, under our evaluation protocol, larger closed models match or surpass bachelor-level OOP students – especially under agentic resolution – while smaller open models lag behind. JAB's class-level design enables fine-grained error analysis, exposing recurring misconceptions. Editor's note: Open Science material was validated by the Journal of Systems and Software Open Science Board. [ABSTRACT FROM AUTHOR]
Copyright of Journal of Systems & Software is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Βάση Δεδομένων: Business Source Index
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://www.doi.org/10.1016/j.jss.2026.113033?
    Name: ScienceDirect (all content) (s7799221)
    Category: fullText
    Text: View record from ScienceDirect
    MouseOverText: View record from ScienceDirect
Header DbId: bsx
DbLabel: Business Source Index
An: 196801289
RelevancyScore: 1338
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 1337.99206542969
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Cocchieri%2C+Alessio%22">Cocchieri, Alessio</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> a.cocchieri@unibo.it</i><br /><searchLink fieldCode="AR" term="%22Ragazzi%2C+Luca%22">Ragazzi, Luca</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> l.ragazzi@unibo.it</i><br /><searchLink fieldCode="AR" term="%22Aguzzi%2C+Gianluca%22">Aguzzi, Gianluca</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> gianluca.aguzzi@unibo.it</i><br /><searchLink fieldCode="AR" term="%22Frisoni%2C+Giacomo%22">Frisoni, Giacomo</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> giacomo.frisoni@unibo.it</i><br /><searchLink fieldCode="AR" term="%22Molfetta%2C+Lorenzo%22">Molfetta, Lorenzo</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> lorenzo.molfetta@unibo.it</i><br /><searchLink fieldCode="AR" term="%22Moro%2C+Gianluca%22">Moro, Gianluca</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> gianluca.moro@unibo.it</i><br /><searchLink fieldCode="AR" term="%22Viroli%2C+Mirko%22">Viroli, Mirko</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> mirko.viroli@unibo.it</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Journal+of+Systems+%26+Software%22">Journal of Systems & Software</searchLink>. Dec2026, Vol. 242, pN.PAG-N.PAG. 1p.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *<searchLink fieldCode="DE" term="%22Object-oriented+programming%22">Object-oriented programming</searchLink><br /><searchLink fieldCode="DE" term="%22Java+programming+language%22">Java programming language</searchLink><br /><searchLink fieldCode="DE" term="%22Model+validation%22">Model validation</searchLink><br /><searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Benchmark+problems+%28Computer+science%29%22">Benchmark problems (Computer science)</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+programming+education%22">Computer programming education</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+software+testing%22">Computer software testing</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Current code generation benchmarks largely overlook object-oriented programming (OOP) skills, leaving open whether large language models (LLMs) can effectively apply OOP principles to structured programming tasks. The dominance and permissiveness of Python in most benchmarks further obscure model weaknesses in core OOP principles. We introduce the Java Academic Benchmark (JAB), based on 103 authentic Java exams collected over a decade at a major university, together with 506 expert-written JUnit tests, to rigorously evaluate advanced Java programming competence with a strong focus on OOP. To complement execution-based scoring, we propose KODE, an LLM-as-a-Judge framework that assesses OOP adherence across four pedagogical dimensions. We evaluate 27 LLMs under two resolution strategies: single-attempt (one-shot) and agentic (iterative refinement with compiler and test feedback). Results reveal that, under our evaluation protocol, larger closed models match or surpass bachelor-level OOP students – especially under agentic resolution – while smaller open models lag behind. JAB's class-level design enables fine-grained error analysis, exposing recurring misconceptions. Editor's note: Open Science material was validated by the Journal of Systems and Software Open Science Board. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Journal of Systems & Software is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=bsx&AN=196801289
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1016/j.jss.2026.113033
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 1
        StartPage: N.PAG
    Subjects:
      – SubjectFull: Object-oriented programming
        Type: general
      – SubjectFull: Java programming language
        Type: general
      – SubjectFull: Model validation
        Type: general
      – SubjectFull: Language models
        Type: general
      – SubjectFull: Benchmark problems (Computer science)
        Type: general
      – SubjectFull: Computer programming education
        Type: general
      – SubjectFull: Computer software testing
        Type: general
    Titles:
      – TitleFull: Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Cocchieri, Alessio
      – PersonEntity:
          Name:
            NameFull: Ragazzi, Luca
      – PersonEntity:
          Name:
            NameFull: Aguzzi, Gianluca
      – PersonEntity:
          Name:
            NameFull: Frisoni, Giacomo
      – PersonEntity:
          Name:
            NameFull: Molfetta, Lorenzo
      – PersonEntity:
          Name:
            NameFull: Moro, Gianluca
      – PersonEntity:
          Name:
            NameFull: Viroli, Mirko
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Text: Dec2026
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 01641212
          Numbering:
            – Type: volume
              Value: 242
          Titles:
            – TitleFull: Journal of Systems & Software
              Type: main
ResultId 1