Academic Journal
Can large language models generate geospatial code?
| Τίτλος: | Can large language models generate geospatial code? |
|---|---|
| Συγγραφείς: | Hou, Shuyang, Shen, Zhangxiao, Liang, Jianyuan, Jiao, Haoyue, Zhao, Anqi, Qing, Yaxian, Peng, Dehua, Gui, Zhipeng, Guan, Xuefeng, Xiang, Longgang, Wu, Huayi |
| Πηγή: | Geo-Spatial Information Science; Apr2026, Vol. 29 Issue 2, p753-787, 35p |
| Θεματικοί όροι: | Geographic information systems, Evaluation methodology, Code generators, Language models, Geographic information system software, Benchmark problems (Computer science) |
| Περίληψη: | As large language models increasingly exhibit hallucinations such as refusal to respond, generation of non-executable code, and poor readability in geospatial code generation tasks, establishing a systematic and quantifiable evaluation framework has become essential for advancing their application in GIS. This paper introduces the GeoCode-Eval framework, the first comprehensive evaluation framework for LLMs in geospatial code generation. Grounded in three dimensions – cognition and memory, understanding and interpretation, and innovation and creation – the framework addresses eight competency levels, including platform and tool cognition, functional knowledge, dataset recognition, information extraction, and various code-related tasks. To support this, the GeoCode-Bench benchmark was developed, consisting of 5,000 multiple-choice questions, 1,500 true/false questions, 1,500 fill-in-the-blank questions, and 1,000 coding tasks. Using six indicators, namely executability, accuracy, readability, location correctness, content correctness, and summary completeness, the study evaluates twelve representative models spanning four categories, including DeepSeek-Coder-V2 and GeoCode-GPT (7B). A combination of analytical methods, including entropy weighting, the coefficient of variation, skewness, and kurtosis, is applied to examine model capability distribution, indicator distribution, code type characteristics, and error type patterns. Results show consistent performance in tool cognition and code summarization, while significant performance gaps persist in code generation, completion, and correction. Common errors include data type and syntax issues. This study provides a quantifiable foundation for the evaluation of capabilities and future optimization of LLMs in geospatial code generation, thereby extending the application boundaries of LLMs in GIS and offering valuable insights into the development of evaluation methodologies for LLM applications in other vertical domains. [ABSTRACT FROM AUTHOR] |
| Copyright of Geo-Spatial Information Science is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Βάση Δεδομένων: | Complementary Index |
καταχωρήστε σχόλιο πρώτοι!