The performance of Covidence: An artificial intelligence-based tool for title and abstract screening in a breast cancer evidence-based clinical practice guideline

Background: Clinical practice guidelines (CPGs) support evidence-based care but are time-consuming to develop. We aimed to compare artificial intelligence (AI)-assisted versus manual title and abstract screening (Stage I) in Covidence using data from a published breast-cancer CPG. Methods: This systematic review (SR) included 8,774 articles identified through a medical literature search, after duplicate removal. Three article subsets (n = 500, 1,000, and 2,000) were randomly selected from 8,774 articles to perform 30, 30, and 10 trials, respectively, independent Stage I AI-assisted trials. The primary outcome of each trial is workload savings achieved through AI-assisted identification of 95% and 100% relevant articles (e.g., sensitivity), and 100% of finally-included articles. The secondary outcome is missed finally-included articles when the sensitivity of 95% was reached for each subset. Results: At 95% sensitivity, 100% relevant articles and 100% finally-included articles were identified, median (minimum, maximum) workload savings were 40.7% (4.4%, 59.4%), 25.0% (0.4%, 55.2%), and 57.6% (6.2%, 76.4%) for n = 500; 38.3% (6.2%, 54.0%), 17.3% (0.0%, 39.1%), and 63.9% (0.4%, 77.5%) for n = 1,000; 16.6% (10.8%, 41.8%), 4.4% (0.3%, 20.9%), and 17.9% (0.8%, 64.6%) for n = 2,000, respectively. Covidence’s performance does not improve as the size of the subsets increases for a CPG with multiple complicated research questions. A potential positive correlation between the proportion of relevant articles in initial training of Covidence and workload savings at Stage I across all 70 trials. At 95% sensitivity, five trials missed one article (n = 500); two trials missed two articles and one trial missed one article (n = 1,000); and one trial missed three articles and five trials missed one article (n = 2,000). Conclusion: AI-assistance in Covidence for Stage I screening showed promise and pitfalls in the SR for a breast cancer CPG on a complex topic. Further prospective research is needed to better understand the performance of AI-assistance in Covidence and intricacies of CPG topics.