Treffer: Letter Recognition for Handwriting on Embedded System Using a Machine Learning Model

This thesis investigates the development and deployment of a lightweight machine learning model for real-time handwritten letter recognition on a highly resource-constrained embedded system, the ESP32-CAM microcontroller. The project explores two convolutional neural network (CNN) architectures: a quantized version of the pre-trained MobileNetV2 and a custom-designed SimpleCNN. Both models were adapted for efficient inference within the ESP32-CAM’s strict memory and processing limits. The system uses the EMNIST Letters dataset, with preprocessing and data augmentation to enhance model generalization across varying handwriting styles. The MobileNetV2 model achieved an F1 score of 87.4% with an inference time of 294 ms, while the SimpleCNN model reached 92.8% accuracy in a simulated environment before hardware deployment. On the other hand, the deployed model only utilizes around 20% of the ESP32-CAM's available RAM... there is potential to use a model up to five times larger, which could improve recognition performance. While the system shows that accurate handwritten letter recognition is feasible on embedded platforms, its scope is limited to uppercase letters and balanced datasets due to platform and training constraints. The results underscore the potential for using embedded deep learning in portable, assistive, and educational devices, provided that limitations in model size, training time, and dataset complexity are carefully managed. Keywords: Handwriting recognition, embedded systems, ESP32-CAM, convolutional neural networks (CNN), MobileNetV2, EMNIST dataset, model quantization, real-time inference, machine learning.