• Comparative Analysis of Human...

Treffer: Comparative Analysis of Human and Large Language Model Performance in Pharmacology Multiple-Choice Questions

Title:
Comparative Analysis of Human and Large Language Model Performance in Pharmacology Multiple-Choice Questions
Authors:
Rodriguez, Ricardo, Huet, Stéphane, Favre, Benoît, Rouvier, Mickael
Contributors:
Laboratoire Informatique d'Avignon (LIA), Avignon Université (AU)-Centre d'Enseignement et de Recherche en Informatique - CERI, Traitement Automatique du Langage Ecrit et Parlé (TALEP), Laboratoire d'Informatique et des Systèmes (LIS) (Marseille, Toulon) (LIS), Aix Marseille Université (AMU)-Université de Toulon (UTLN)-Centre National de la Recherche Scientifique (CNRS)-Aix Marseille Université (AMU)-Université de Toulon (UTLN)-Centre National de la Recherche Scientifique (CNRS), Groupe d’Étude en Traduction Automatique/Traitement Automatisé des Langues et de la Parole (GETALP), Laboratoire d'Informatique de Grenoble (LIG), Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes (UGA)-Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP), Université Grenoble Alpes (UGA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes (UGA)-Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP), Université Grenoble Alpes (UGA), Galia Angelova, Maria Kunilovskaya, Marie Escribe, Ruslan Mitkov, ANR-23-IAS1-0005,MALADES,Grands modèles de langue adaptables et souverains pour le domaine médical français(2023)
Source:
15th Recent Advances in Natural Language Processing Conference (RANLP 2025). :1022-1029
Publisher Information:
CCSD; ACL, 2025.
Publication Year:
2025
Collection:
collection:UNIV-AVIGNON
collection:UGA
collection:UNIV-TLN
collection:CNRS
collection:UNIV-AMU
collection:INPG
collection:LIG
collection:LIG_TDCGE_GETALP
collection:LIA
collection:LIS-LAB
collection:UGA-EPE
collection:ANR
collection:INCIAM
collection:ANR-IA-23
collection:ANR-IA
collection:TEST-UGA
Subject Terms:
[INFO.INFO-CL]Computer Science [cs], Computation and Language [cs.CL], [INFO.INFO-AI]Computer Science [cs], Artificial Intelligence [cs.AI]
Subject Geographic:
Varna, Bulgaria
Original Identifier:
HAL: hal-05377384
Document Type:
Konferenz conferenceObject<br />Conference papers
Language:
English
Relation:
info:eu-repo/semantics/altIdentifier/doi/10.26615/978-954-452-098-4-117
DOI:
10.26615/978-954-452-098-4-117
Access URL:
https://hal.science/hal-05377384
https://hal.science/hal-05377384v1/document
https://hal.science/hal-05377384v1/file/main_hal.pdf
Rights:
info:eu-repo/semantics/OpenAccess
URL: http://creativecommons.org/licenses/by/
Accession Number:
edshal.hal.05377384v1
Database:
HAL

Weitere Informationen

In this article, we study the answers generated by a selection of Large Language Models to a set of Multiple Choice Questions in Pharmacology, and compare them to the answers provided by students, to understand which questions in this clinical domain are difficult for the models when compared to humans and why. We extract the internal logits to infer probability distributions and analyse the main features that determine the difficulty of questions using statistical methods. We also provide an extension to the FrenchMedMCQA dataset, with pairs of question-answers in pharmacology, enriched with student response rate, answer scoring, clinical topics, and annotations on question structure and semantics.