Classification of health product defect reports by deep learning

Abstract Quality defects in substandard medicines represent a threat to public health. Rapid and accurate identification of these defects is critical to prioritise the cases and implement regulatory measures. However, the development of surveillance systems to classify reports of health product defects remains a largely unmet need in medicine safety monitoring. The objective of this study is to implement an AI system to support the classification and prioritization of health product defect reports. To develop a deep learning system for the classification of health product defect reports, 13,830 reports collected between 2010 and 2021 were used. The reports were labelled by a panel of pharmacovigilance experts into 21 categories following standardised medical terminology. Our system harnesses state-of-the-art language algorithms that extract rich textual features to classify the reports. The functionality of the system is enhanced with explainable features that provide interpretability and actionable insights to decision-makers. Our system achieves top-1, top-2, and top-3 accuracies of 86%, 93%, and 96%, respectively. There is a statistically significant positive correlation between sample size and top-1 (Pearson’s r: 0.643; 95% CI: [0.2921, 0.8411]; p-value: 0.0016), top-2 (Pearson’s r: 0.735; 95% CI: [0.4439, 0.8856]; p-value: 0.0001), and top-3 (Pearson’s r: 0.635; 95% CI: [0.2808, 0.8374]; p-value: 0.0020) performance metrics. Likewise, model accuracy is positively correlated with confidence scores (Pearson’s r: 0.927; 95% CI: [0.8253, 0.9703]; p-value < 0.00001). A feature analysis reveals that the most influential words in model decision are conceptually and semantically related to their respective product defect categories. Our model has been validated with prospective data. The developed classification system allows for standardisation in case triage, and potentially improves case prioritisation and processing workflows, leading to more prompt response for quality defects with high public health impact.