An explainable multimodal deep learning approach for stress detection in emotion-aware systems

Abstract We propose a continuous stress-detection pipeline that integrates wavelet-based heart-rate-variability (HRV) features with behavioural signals in a staged ensemble. The method computes time-resolved HRV spectral indices (VLF/LF/HF) using wavelet decomposition and incorporates them with behavioural descriptors before training a Random-Forest-based ensemble on the fused feature set. Post-hoc SHAP analysis yields both dataset-level importance rankings and instance-level attributions, enabling clinicians and end users to interpret the rationale underlying each prediction. Evaluated on the WESAD dataset (N=15; ECG sampled at 700 Hz; session duration ≈35–40 min), the system achieved 83.0% accuracy on a held-out test split. SHAP consistently highlighted LF and HF approximate power as dominant predictors, while the wavelet plots revealed short-term spectral fluctuations aligned with well-known stress episodes, reinforcing the physiological validity of the reported findings.