Λεπτομέρειες βιβλιογραφικής εγγραφής
| Τίτλος: |
GeoPyEval: An Automated Evaluation Framework for Python Code Generation Capabilities of Large Language Models in Geospatial Domains. |
| Συγγραφείς: |
Wu, Shaowen1 (AUTHOR), Xie, Lutong1 (AUTHOR), Hou, Shuyang1 (AUTHOR) whuhsy@whu.edu.cn, Chen, Guanyu2 (AUTHOR), Jiao, Haoyue2 (AUTHOR), Liu, Ziqi1 (AUTHOR), Guan, Xuefeng1 (AUTHOR), Wu, Huayi1 (AUTHOR) |
| Πηγή: |
Transactions in GIS. Jun2026, Vol. 30 Issue 4, p1-28. 28p. |
| Θεματικοί όροι: |
*Benchmarking (Management), Python programming language, Geoinformatics, Language models |
| Περίληψη: |
Despite the increasing adoption of large language models (LLMs) for Python geospatial code generation, no automated evaluation framework exists for this domain. We propose GeoPyEval, the first function‐level framework in this field, featuring an expert‐in‐the‐loop, automated execution design with three core modules: (a) GeoPyEval‐Bench, constructed from expert‐verified LLM‐generated content, covering 18 mainstream geospatial libraries with 1074 unit test tasks; (b) an expert‐configurable submission program for invoking LLMs to perform code generation and execution; and (c) a judging program for correctness evaluation. We systematically assessed 12 mainstream LLMs available as of November 2025 regarding accuracy, resource consumption, operational efficiency, and error type logs. Results show that GPT‐5 achieved the highest accuracy with a Pass@5 of 64.28%. GeoPyEval extends existing LLM evaluation frameworks to Python geospatial code generation, establishing a novel benchmarking standard for this domain. [ABSTRACT FROM AUTHOR] |
|
Copyright of Transactions in GIS is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) |
| Βάση Δεδομένων: |
Business Source Index |