Explainable Artificial Intelligence in Healthcare: Current Landscape, Challenges, and Future Directions

ABSTRACT Background and Aims Artificial Intelligence (AI), particularly Machine Learning (ML) and Deep Learning (DL), is transforming healthcare by enabling improved diagnosis, prognosis, and personalized treatments. However, the opacity of many AI models operates as “black boxes,” limiting interperability, clinician trust, and real‐world adoption. Explainable Artificial Intelligence (XAI) has emerged to address these limitations by providing transparent and actionable insights. This systematic review aims to synthesize the current evidence on XAI in healthcare, mapping AI models to XAI techniques, domains, and clinical applications. Methods A systematic search was conducted across six databases (Elsevier, Springer, Taylor & Francis, Semantic Scholar, ACM, and IEEE Xplore) for peer‐reviewed published between 2017 and 2025. After duplicate removal and title/abstract screening, full texts were evaluated against predefined inclusion/exclusion criteria, following PRISMA guidelines. Data extraction included AI model types, XAI techniques, healthcare domains, study design, validation methods, and ethical/regulatory reporting. Results Seventy studies were included, spanning oncology (40%), cardiology (21%), infectious diseases (14%), neurology (11%), and clinical decision support systems (13%). Deep learning models (CNN, RNN, LSTM, and Transformers) were most frequently applied (76%), followed by tree‐based models (Random Forest, XGBoost, Decision Trees; 24%). SHAP (54%) and LIME (30%) were the most commonly used XAI techniques, with Grad‐CAM (23%) and attention mechanisms (20%) applied mainly in imaging and sequence‐based tasks. Only 12 studies explicitly addressed ethical or regulatory considerations. Hybrid interpretable models and human‐centered designs are emerging trends, but real‐world validation and standardized interpretability metrics remain limited. Conclusion XAI enhances transparency, clinician trust, and decision‐making in healthcare AI applications, yet challenges persist, including inconsistent validation, underdeveloped ethical/regulatory frameworks, and lack of standardized interpretability measures. Future work should focus on hybrid, clinically validated XAI models, comprehensive ethical compliance, and user‐centered, domain‐specific implementations to ensure safe and effective integration into clinical practice.