Single sensor Wrist-Worn contact detection system for upper limb prosthetics using deep learning

Abstract Upper-limb prosthesis users usually lack comprehensive feedback or touch. This significantly hinders their ability to interact with their environment and lead normal lives. This research involves the development and evaluation of a low-cost, non-invasive and easy to implement, wearable touch detection system. The system seeks to restore a sense of touch by interpreting arm movements through a wrist-worn 6-axis Inertial Measurement Unit (IMU). The system is intended to be compatible with a wide range of upper-limb prostheses to give sensory feedback to the user when paired with existing haptic motor setups. Training and testing data were obtained from ten subjects performing a series of paired touch and non-touch activities. To perform real-time touch event classification from the IMU data, multiple lightweight machine learning models were developed and evaluated, including Logistic Regression, Support Vector Machines, Random Forest, and a 1D-CNN. The best model, a 1D-CNN model, achieved 99.1% accuracy and an F1-score of 0.85 in classifying touch events of various types. The model also maintained an accuracy of 82% at an augmentation intensity of 0.05, used to mimic intense real-world sensor noise. The model learned features directly from windowed time-series data and was trained using a combination of a partial Synthetic Minority Over-sampling Technique (SMOTE) and a Focal Loss function to prioritize the minority "touch" class. Model results improved when trained on initial contact instances rather than the full contact duration. While the 1D-CNN established the upper limit of detection accuracy, a lighter Logistic Regression model was successfully deployed on an embedded microcontroller as a real-time proof of concept for the feasibility of using deep learning on IMU data for real-time sensory substitution.