Treffer: PABLO and PYRITE: Helping Novices Debug Python Code Through Data-Driven Fault Localization and Repair

As dynamically-typed languages grow in popularity, especially among beginning programmers, novices have an increased need for scalable, helpful feedback for fixing their bugs.Localization and repair can be ambiguous: not all repairs which prevent the program from crashing are equally useful for beginners. We propose scalable approaches for fault localization and repair for dynamic languages that are helpful for debugging and generalize to handle a wide variety of errors commonly faced by novice programmers. We base our approach on a combination of static, dynamic, and contextual features, guided by machine learning. We evaluate on over 980,000 diverse real user interactions across four years from the popular PythonTutor.com website, which is used both in classes and by non-traditional learners. We find that our approach is scalable, general, and quite accurate: up to 77% of these historical novice users would have been helped by our top-three localization responses, compared to 45% for the default interpreter, and we successfully synthesize repairs to 76% of our historical buggy programs. We also conducted two human studies. Participants preferred our localization approach to the baseline (p = 0.018), and found it additionally useful for bugs meriting multiple edits. Participants found our repairs to contain helpful information beyond the baseline in 45% of programs.