Λεπτομέρειες βιβλιογραφικής εγγραφής
| Τίτλος: |
Bilingual analysis of logical coherence and accuracy in large language model performance on high school math word problems: Russian vs. English. |
| Συγγραφείς: |
Sirgebayeva, Aigerim, Yergazy, Yersaiyn, Jaxylykova, Assel, Pak, Alexandr |
| Πηγή: |
PeerJ Computer Science; Jun2026, p1-29, 29p |
| Θεματικοί όροι: |
Word problems (Mathematics), Logical fallacies, Cohesion (Linguistics), Statistical accuracy, Russian language, Language models, English language |
| Περίληψη: |
Nowadays, large language models (LLMs) have demonstrated impressive performance in solving math word problems (MWPs). However, there are currently challenges on their reasoning abilities, particularly on problems demanding advanced reasoning, with notable inconsistencies observed across different languages. This article conducts a thorough evaluation of the reasoning capabilities of LLMs like Gemini-1.5-Pro, Llama 3.1-406B-Instruct, Llama 4-Maverick-Instruct, Mistral-8x22B-Instruct-v0.1, ChatGPT-3.5 Turbo, ChatGPT-4, ChatGPT-4o on math word problems (MWPs) in Russian and English languages. Understanding the bilingual performance of LLMs in solving MWPs is crucial for assessing their potential and limitations in a global context. Despite their ability to generate correct answers, the coherence in logical inference and their accuracy is uncertain, particularly with complex math word problems. In response to this challenge, we introduce the QazUNTv2 dataset, featuring high school math word problems (MWPs) in Russian and English, along with categorized labels that represent sections in mathematics, such as algebra, probability theory, and logic, and meticulously crafted manual solutions. Further, solutions were obtained through LLMs for comparison purposes. To evaluate them against the dataset's reference versions, we applied ROUGE-1 and ROUGE-2 (Recall-Oriented Understudy for Gisting Evaluation) and BERTScore (Bidirectional Encoder Representations from Transformers Score) metrics. We conducted a comparative analysis of LLM behavior when solving tasks from different mathematical domains mentioned before. The study examines the differences in LLM performance when solving mathematical problems in English and Russian, highlighting the impact of linguistic structure on reasoning accuracy. We identified seven primary types of reasoning errors and quantitatively assessed error rates across models. The results show that ChatGPT-3.5 Turbo exhibited the highest error rate in both languages and differs significantly from all other models (p < 0.001), while Gemini 1.5 Pro and GPT-4 Turbo demonstrated greater accuracy, emphasizing the need for improved logical inference in LLMs. Additionally, our findings reveal notable cross-linguistic differences in model responses, with Russian answers being more concise than English ones and incorrect responses tending to be more verbose than correct ones. These patterns suggest that linguistic structure influences reasoning efficiency, while verbosity may serve as an indicator of model uncertainty. [ABSTRACT FROM AUTHOR] |
|
Copyright of PeerJ Computer Science is the property of PeerJ Inc. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) |
| Βάση Δεδομένων: |
Complementary Index |