Emotion recognition in human robot collaboration for multimodal approaches, real-world challenges and future directions

Abstract Emotion recognition is fundamental to building socially intelligent robotic systems capable of effective and adaptive Human-Robot Interaction (HRI) and Collaboration (HRC). This literature review synthesizes recent advances from 2015 to 2025, covering 42 empirical studies focused on speech, facial, and multimodal emotion recognition approaches tailored for robotic contexts. We provide a modality-wise classification of methods, highlight key deep learning architectures and signal processing strategies, and analyze their performance across diverse robotic platforms and environments. Multimodal systems accounted for over 50% of the studies, reflecting the field’s shift toward fusion-based robustness. Across the literature, reported accuracies range from 68 to 96% under controlled conditions, with real-world deployments typically facing 4-27.5% performance degradation (mean: 8.6% ± 3.0% for speech systems, 14.4% ± 7.1% for facial systems, computed across studies in Table 1 reporting paired lab and real-world scores). Despite notable progress, challenges persist in deploying emotion-aware systems under real-world conditions due to acoustic variability, sensor limitations, and a lack of generalizable models. We discuss practical mitigation techniques, including domain adaptation, personalization, and multimodal fusion, and outline future research directions toward ethical, real-time, and context-sensitive emotion recognition in HRC.