Treffer: Feedback with BERT: When Detecting Students’ Misconceptions Becomes Automatic

When learning a new programming language, students can benefit from feedback on their work, to get a better overview of their level, strengths and shortcomings. However, giving personalised feedback on misconceptions is complex due to a lack of means or resources; the design of automated tests and feedback is cumbersome and timeconsuming. Our work aims to overcome some of these limitations by enabling automatic feedback thanks to a machine learning model. We developed a multi-label classification architecture following latest advances in natural language processing. By using code embeddings, i.e. generated vectors on students’ code submissions, our system allows to detect specific misconceptions occurring in code snippets, and provide predefined feedback based on these classes. To control the classes and enable the training of our deep neural network, we developed an approach inspired by DeepBugs. The training instances are mutants of original students’ submissions, where the injected modifications are representative of a set of 14 misconceptions we selected. Our model obtained f1-score values up to 72.9% when predicting an evaluation dataset of students’ mistakes. We also highlight limits of our current mutation labelling technique and improvements to be conducted as further work.