Treffer: Toward diagnosis of semantic errors in Python programming platforms for beginners

International audience ; Reliable semantic error analysis of student codes can be used to improve both the learning and teaching experience, through a better understanding of students' codes. We propose a novel approach that integrates automatic semantic error detection in student codes with the construction of a solution space for analyzing students' programming trajectories. This paper addresses the challenge of automatically annotating codes with predefined error tags and assisting the construction of a solution space, leveraging generative AI based on Large Language Models (LLMs) to streamline these processes and save time for experts. To achieve this, we carried out two studies. The first study is focused on annotating student erroneous codes with error tags. Kappa analysis was employed to measure the agreement between human-annotated codes and AI-generated annotations, assessing the relevance and accuracy of the LLMs in error detection. The results revealed agreements upto a Cohen's kappa coefficient of 0.39. The second study examined the construction of a solution space by analyzing the overlap between AI-generated and real student erroneous codes. This evaluation aimed to determine the capacity of the LLMs in generating erroneous codes used for constructing the solution space. The results showed that, on average, 68% of the analyzed students' erroneous codes were covered by the AI-generated codes. The results of the first study are mitigated compared to the second ones but the combination of these two approaches could appear as positive to compensate for each other and improvements are ongoing investigation.