Academic Journal
View in EDS
Linked Full Text

Evaluation of LLM Tools for Feedback Generation in a Course on Concurrent Programming

Bibliographic Details
Title: Evaluation of LLM Tools for Feedback Generation in a Course on Concurrent Programming
Language: English
Authors: Iria Estévez-Ayres (ORCID 0000-0002-1047-5398), Patricia Callejo (ORCID 0000-0001-6124-6213), Miguel Ángel Hombrados-Herrera (ORCID 0000-0002-8254-8795), Carlos Alario-Hoyos (ORCID 0000-0002-3082-0814), Carlos Delgado Kloos (ORCID 0000-0003-4093-3705)
Source: International Journal of Artificial Intelligence in Education. 2025 35(2):774-790.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 17
Publication Date: 2025
Document Type: Journal Articles
Reports - Evaluative
Education Level: Higher Education
Postsecondary Education
Descriptors: Artificial Intelligence, Natural Language Processing, Computer Mediated Communication, Technology Uses in Education, Educational Technology, Feedback (Response), Computer Science Education, Programming, College Students, Error Patterns, Student Evaluation, Evaluation Methods, Accuracy
DOI: 10.1007/s40593-024-00406-0
ISSN: 1560-4292
1560-4306
Abstract: The emergence of Large Language Models (LLMs) has marked a significant change in education. The appearance of these LLMs and their associated chatbots has yielded several advantages for both students and educators, including their use as teaching assistants for content creation or summarisation. This paper aims to evaluate the capacity of LLMs chatbots to provide feedback on student exercises in a university programming course. The complexity of the programming topic in this study (concurrency) makes the need for feedback to students even more important. The authors conducted an assessment of exercises submitted by students. Then, ChatGPT (from OpenAI) and Bard (from Google) were employed to evaluate each exercise, looking for typical concurrency errors, such as starvation, deadlocks, or race conditions. Compared to the ground-truth evaluations performed by expert teachers, it is possible to conclude that none of these two tools can accurately assess the exercises despite the generally positive reception of LLMs within the educational sector. All attempts result in an accuracy rate of 50%, meaning that both tools have limitations in their ability to evaluate these particular exercises effectively, specifically finding typical concurrency errors.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1488271
Database: ERIC
FullText Links:
  – Type: other
    Url: https://resolver.ebsco.com:443/public/rma-ftfapi/ejs/direct?AccessToken=4301A74A06AB0FABBC0C&Show=Object
Text:
  Availability: 0
CustomLinks:
  – Url: https://dx.doi.org/doi:10.1007/s40593-024-00406-0
    Name: EDS - Springer Nature Journals (s7799221)
    Category: fullText
    Text: View record at Springer
Header DbId: eric
DbLabel: ERIC
An: EJ1488271
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evaluation of LLM Tools for Feedback Generation in a Course on Concurrent Programming
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Iria+Estévez-Ayres%22">Iria Estévez-Ayres</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-1047-5398">0000-0002-1047-5398</externalLink>)<br /><searchLink fieldCode="AR" term="%22Patricia+Callejo%22">Patricia Callejo</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-6124-6213">0000-0001-6124-6213</externalLink>)<br /><searchLink fieldCode="AR" term="%22Miguel+Ángel+Hombrados-Herrera%22">Miguel Ángel Hombrados-Herrera</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-8254-8795">0000-0002-8254-8795</externalLink>)<br /><searchLink fieldCode="AR" term="%22Carlos+Alario-Hoyos%22">Carlos Alario-Hoyos</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-3082-0814">0000-0002-3082-0814</externalLink>)<br /><searchLink fieldCode="AR" term="%22Carlos+Delgado+Kloos%22">Carlos Delgado Kloos</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-4093-3705">0000-0003-4093-3705</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Artificial+Intelligence+in+Education%22"><i>International Journal of Artificial Intelligence in Education</i></searchLink>. 2025 35(2):774-790.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 17
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Mediated+Communication%22">Computer Mediated Communication</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Technology%22">Educational Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Science+Education%22">Computer Science Education</searchLink><br /><searchLink fieldCode="DE" term="%22Programming%22">Programming</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Error+Patterns%22">Error Patterns</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s40593-024-00406-0
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1560-4292<br />1560-4306
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The emergence of Large Language Models (LLMs) has marked a significant change in education. The appearance of these LLMs and their associated chatbots has yielded several advantages for both students and educators, including their use as teaching assistants for content creation or summarisation. This paper aims to evaluate the capacity of LLMs chatbots to provide feedback on student exercises in a university programming course. The complexity of the programming topic in this study (concurrency) makes the need for feedback to students even more important. The authors conducted an assessment of exercises submitted by students. Then, ChatGPT (from OpenAI) and Bard (from Google) were employed to evaluate each exercise, looking for typical concurrency errors, such as starvation, deadlocks, or race conditions. Compared to the ground-truth evaluations performed by expert teachers, it is possible to conclude that none of these two tools can accurately assess the exercises despite the generally positive reception of LLMs within the educational sector. All attempts result in an accuracy rate of 50%, meaning that both tools have limitations in their ability to evaluate these particular exercises effectively, specifically finding typical concurrency errors.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1488271
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1488271
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s40593-024-00406-0
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 17
        StartPage: 774
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Computer Mediated Communication
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Educational Technology
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Computer Science Education
        Type: general
      – SubjectFull: Programming
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Error Patterns
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Accuracy
        Type: general
    Titles:
      – TitleFull: Evaluation of LLM Tools for Feedback Generation in a Course on Concurrent Programming
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Iria Estévez-Ayres
      – PersonEntity:
          Name:
            NameFull: Patricia Callejo
      – PersonEntity:
          Name:
            NameFull: Miguel Ángel Hombrados-Herrera
      – PersonEntity:
          Name:
            NameFull: Carlos Alario-Hoyos
      – PersonEntity:
          Name:
            NameFull: Carlos Delgado Kloos
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1560-4292
            – Type: issn-electronic
              Value: 1560-4306
          Numbering:
            – Type: volume
              Value: 35
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: International Journal of Artificial Intelligence in Education
              Type: main
ResultId 1