Academic Journal
Attempts to investigate what words trivia experts focus on in quiz questions: Human-AI comparison through "LLM-as-a-judge" approach.
| Title: | Attempts to investigate what words trivia experts focus on in quiz questions: Human-AI comparison through "LLM-as-a-judge" approach. |
|---|---|
| Authors: | Shirasuna M; Faculty of Informatics, Shizuoka University, Hamamatsu-shi, Shizuoka, Japan., Yoshida Y; Faculty of Informatics, Shizuoka University, Hamamatsu-shi, Shizuoka, Japan. |
| Source: | PloS one [PLoS One] 2026 Sep 22; Vol. 21 (9), pp. e0358332. Date of Electronic Publication: 2026 Sep 22 (Print Publication: 2026). |
| Publication Type: | Journal Article; Comparative Study |
| Language: | English |
| Journal Info: | Publisher: Public Library of Science Country of Publication: United States NLM ID: 101285081 Publication Model: eCollection Cited Medium: Internet ISSN: 1932-6203 (Electronic) Linking ISSN: 19326203 NLM ISO Abbreviation: PLoS One Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: San Francisco, CA : Public Library of Science |
| MeSH Terms: | Games, Recreational*/psychology , Large Language Models* , Electronic Data Processing* , Processing Speed* , Judgment*, Japan ; Psycholinguistics ; Humans ; Male ; Female ; Adult |
| Abstract: | Computational capacity and knowledge of humans are more limited than those of large language models (LLMs). However, in buzzer quizzes, human trivia experts can often identify the correct answer even from insufficient information such as only a few words. Investigating how they can make fast and accurate judgments is expected to highlight new characteristics of human intelligence, but little is known about that. In this exploratory and case-based analysis, we predicted that trivia experts and LLMs would differ in which words/phrases in a question are important for identifying the answer, and compared experts' performance with LLMs' performance in Japanese trivia questions through an LLM-as-a-judge approach: We regarded LLMs as evaluators and then used their outputs as comparative tools for experts' evaluations. First, we constructed a quiz question processing system that tokenized question texts based on morphological analysis and then numerically evaluated the importance of each token using GPT-4o/GPT-4.1. Then, we conducted a behavioral experiment wherein actual trivia experts were asked to numerically evaluate the importance of each token, just as LLMs had performed. As a result, trivia experts treated a variety of words/phrases as important, even if each word/phrase did not appear to be strongly associated with the answer. Their evaluation scores tended to accumulate faster than those of the LLMs. This may indicate that trivia experts can make inductive inferences faster (e.g., finding a common concept even from few items). More advanced question-answering systems may be designed by applying trivia experts' cognitive processes to LLMs' information processing, and our findings may provide a scaffolding toward such goals. (Copyright: © 2026 Shirasuna, Yoshida. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.) |
| Competing Interests: | The authors declare no competing interests. |
| References: | Psychon Bull Rev. 2024 Aug;31(4):1867-1879. (PMID: 38347367) Annu Rev Psychol. 2011;62:451-82. (PMID: 21126183) Proc Natl Acad Sci U S A. 2025 Oct 21;122(42):e2518443122. (PMID: 41082665) Innovation (Camb). 2026 Jan 09;7(6):101253. (PMID: 42254963) Mem Cognit. 2019 Feb;47(2):292-298. (PMID: 30324558) Cogn Sci. 2018 Sep;42(7):2364-2396. (PMID: 30101508) Sci Rep. 2025 Sep 2;15(1):32290. (PMID: 40897780) Trends Cogn Sci. 2020 Nov;24(11):873-883. (PMID: 33041198) Perspect Psychol Sci. 2024 Sep;19(5):839-848. (PMID: 37522323) Behav Brain Sci. 2019 Feb 04;43:e1. (PMID: 30714890) Sci Rep. 2025 Jul 10;15(1):24956. (PMID: 40640311) Q J Exp Psychol (Hove). 2020 Jul;73(7):1092-1100. (PMID: 32238037) Mem Cognit. 2026 Feb;54(2):375-400. (PMID: 40694202) |
| Entry Date(s): | Date Created: 20260922 Date Completed: 20260923 Latest Revision: 20260924 |
| Update Code: | 20260924 |
| PubMed Central ID: | PMC13596818 |
| DOI: | 10.1371/journal.pone.0358332 |
| PMID: | 42771653 |
| Database: | MEDLINE |
| ISSN: | 1932-6203 |
|---|---|
| DOI: | 10.1371/journal.pone.0358332 |