Treffer: Large Language Models’ Ability to Assess Main Concepts in Story Retelling: A Proof-of-Concept Comparison of Human Versus Machine Ratings.

Purpose: Despite an abundance of manual, labor-intensive discourse analysis methods, there remains a dearth of clinically convenient, psychometrically robust instruments to measure change in real-world communication in aphasia. The Brief Assessment of Transactional Success (BATS) addresses this gap while developing automated methods for analyzing story retelling discourse. This study investigated automation of main concept (MC) analysis of stories by comparing scores from three large language models (LLMs) to those of human raters. Method: After watching/listening to each of the eight short video/audio BATS stimuli and retelling each story, 96 persons with aphasia (PWA; n = 48 female) engaged in topic-constrained conversations over Zoom with 94 familiar and 107 unfamiliar conversation partners (CPs). CPs then retold each story as coconstructed during their conversations with PWA. Audio files from the resulting 1,760 story retells were transcribed using Python and AssemblyAI’s speech-totext application programming interface. Each MC was first scored by human raters for presence, accuracy, and completeness. Raters used a semiautomated application, MainConcept. For each transcript, an MC composite ratio score was obtained. We evaluated three state-of-the-art LLMs: two proprietary models, GPT-4 and GPT-4o, and one open-source model, Llama-3-70B. The interrater reliability between each LLM versus human MC scoring was assessed via the Pearson correlation coefficient and reliability coefficients based on the generalizability theory (G-theory). Results: The Pearson correlation coefficients indicate strong positive linear relationships between LLM and human MC scores. G-theory reliability coefficients also indicate reliable scoring between LLM and human scoring across the spectrum of participants and conditions. Conclusions: This promising proof-of-concept study affirms the reliability of three LLMs in evaluating BATS story retell MCs and justifies ongoing investigation into their use. Providing clinicians and clinical researchers with automated tools for analyzing discourse without the need for prohibitively labor-intensive manual scoring could be a paradigm shift, potentially revolutionizing the aphasia intervention landscape. [ABSTRACT FROM AUTHOR]

Copyright of American Journal of Speech-Language Pathology is the property of American Speech-Language-Hearing Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)