Academic Journal
Προβολή σε EDS

Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study.
Συγγραφείς: Wang, Ye, Xue, Weihao, Huang, Qiao, Jiang, Bo, Zhang, Hua
Πηγή: Journal of Software: Evolution & Process; Jan2025, Vol. 37 Issue 1, p1-20, 20p
Θεματικοί όροι: Language models, ChatGPT, Computer software development, Empirical research, Application program interfaces
Περίληψη: As software development grows increasingly complex, application programming interface (API) plays a significant role in enhancing development efficiency and code quality. However, the explosive growth in the number of APIs makes it impossible for developers to become familiar with all of them. In actual development scenarios, developers may spend a significant amount of time searching for suitable APIs, which could severely impact the development process. Recently, the OpenAI's large language model (LLM) based application—ChatGPT has shown exceptional performance across various software development tasks, responding swiftly to instructions and generating high‐quality textual responses, suggesting its potential in API recommendation tasks. Thus, this paper presents an empirical study to investigate the performance of ChatGPT in query‐based API recommendation tasks. Specifically, we utilized the existing benchmark APIBENCH‐Q and the newly constructed dataset as evaluation datasets, selecting the state‐of‐the‐art models BIKER and MULAREC for comparison with ChatGPT. Our research findings demonstrate that ChatGPT outperforms existing approaches in terms of success rate, mean reciprocal rank (MRR), and mean average precision (MAP). Through a manual examination of samples in which ChatGPT exceeds baseline performance and those where it provides incorrect answers, we further substantiate ChatGPT's advantages over the baselines and identify several issues contributing to its suboptimal performance. To address these issues and enhance ChatGPT's recommendation capabilities, we employed two strategies: (1) utilizing a more advanced LLM (GPT‐4) and (2) exploring a new approach—MACAR, which is based on the Chain of Thought methodology. The results indicate that both strategies are effective. [ABSTRACT FROM AUTHOR]
Copyright of Journal of Software: Evolution & Process is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Βάση Δεδομένων: Complementary Index
FullText Links:
  – Type: other
Text:
  Availability: 0
Header DbId: edb
DbLabel: Complementary Index
An: 183853574
RelevancyScore: 984
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 983.694702148438
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wang%2C+Ye%22">Wang, Ye</searchLink><br /><searchLink fieldCode="AR" term="%22Xue%2C+Weihao%22">Xue, Weihao</searchLink><br /><searchLink fieldCode="AR" term="%22Huang%2C+Qiao%22">Huang, Qiao</searchLink><br /><searchLink fieldCode="AR" term="%22Jiang%2C+Bo%22">Jiang, Bo</searchLink><br /><searchLink fieldCode="AR" term="%22Zhang%2C+Hua%22">Zhang, Hua</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: Journal of Software: Evolution & Process; Jan2025, Vol. 37 Issue 1, p1-20, 20p
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22ChatGPT%22">ChatGPT</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+software+development%22">Computer software development</searchLink><br /><searchLink fieldCode="DE" term="%22Empirical+research%22">Empirical research</searchLink><br /><searchLink fieldCode="DE" term="%22Application+program+interfaces%22">Application program interfaces</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: As software development grows increasingly complex, application programming interface (API) plays a significant role in enhancing development efficiency and code quality. However, the explosive growth in the number of APIs makes it impossible for developers to become familiar with all of them. In actual development scenarios, developers may spend a significant amount of time searching for suitable APIs, which could severely impact the development process. Recently, the OpenAI's large language model (LLM) based application—ChatGPT has shown exceptional performance across various software development tasks, responding swiftly to instructions and generating high‐quality textual responses, suggesting its potential in API recommendation tasks. Thus, this paper presents an empirical study to investigate the performance of ChatGPT in query‐based API recommendation tasks. Specifically, we utilized the existing benchmark APIBENCH‐Q and the newly constructed dataset as evaluation datasets, selecting the state‐of‐the‐art models BIKER and MULAREC for comparison with ChatGPT. Our research findings demonstrate that ChatGPT outperforms existing approaches in terms of success rate, mean reciprocal rank (MRR), and mean average precision (MAP). Through a manual examination of samples in which ChatGPT exceeds baseline performance and those where it provides incorrect answers, we further substantiate ChatGPT's advantages over the baselines and identify several issues contributing to its suboptimal performance. To address these issues and enhance ChatGPT's recommendation capabilities, we employed two strategies: (1) utilizing a more advanced LLM (GPT‐4) and (2) exploring a new approach—MACAR, which is based on the Chain of Thought methodology. The results indicate that both strategies are effective. [ABSTRACT FROM AUTHOR]
– Name: Abstract
  Label:
  Group: Ab
  Data: <i>Copyright of Journal of Software: Evolution & Process is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edb&AN=183853574
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/smr.2765
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 20
        StartPage: 1
    Subjects:
      – SubjectFull: Language models
        Type: general
      – SubjectFull: ChatGPT
        Type: general
      – SubjectFull: Computer software development
        Type: general
      – SubjectFull: Empirical research
        Type: general
      – SubjectFull: Application program interfaces
        Type: general
    Titles:
      – TitleFull: Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wang, Ye
      – PersonEntity:
          Name:
            NameFull: Xue, Weihao
      – PersonEntity:
          Name:
            NameFull: Huang, Qiao
      – PersonEntity:
          Name:
            NameFull: Jiang, Bo
      – PersonEntity:
          Name:
            NameFull: Zhang, Hua
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Text: Jan2025
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 20477473
          Numbering:
            – Type: volume
              Value: 37
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Software: Evolution & Process
              Type: main
ResultId 1