Pipeline evaluation of a state-of-the-art AI algorithm for detection of focal cortical dysplasia: insights into potential failure sources

Abstract Purpose MELD Graph is a state-of-the-art artificial intelligence (AI) model for automated detection of focal cortical dysplasia (FCD), but its performance remains limited, highlighting the need to investigate which aspects of the pipeline affect its accuracy. Methods A retrospective failure-mode analysis of the MELD Graph pipeline was performed in 242 subjects, with model predictions and FreeSurfer segmentations reviewed to classify errors as segmentation-associated or algorithm-related. FCD imaging features salient to humans were quantified, with statistical associations examined for both MELD Graph detection and focal FreeSurfer segmentation failure. Results MELD Graph demonstrated overall performance similar to previously published non-harmonized results, achieving a sensitivity of 69%, specificity of 44%, and positive predictive value (PPV) of 75%. Focal FreeSurfer segmentation failures were associated with 21% of false negative patients, 25% of false positive clusters in patients, and 16% of false positive clusters in controls. Following manual cortical segmentation correction and rerunning of MELD Graph, 67% of the segmentation-associated missed lesions were detected, and segmentation-associated false positive clusters were reduced or eliminated in 75% of controls with such clusters. Higher conspicuity on T1-weighted images was associated with MELD Graph detection, whereas greater conspicuity on T2-FLAIR images relative to T1 was associated with detection failure. Non–bottom-of-sulcus lesion location, higher human conspicuity measures, and low T1 image quality were positively associated with focal FreeSurfer segmentation failures. Conclusion FreeSurfer segmentation failures are a significant potential source of error in the MELD Graph pipeline. FCD imaging features salient to humans and image quality were also associated with variability in algorithm performance. Robust cortical segmentation and stronger integration of T2-FLAIR imaging features may be beneficial for automated FCD detection tools. Clinical trial registration Not applicable. This study is a retrospective analysis of previously acquired open-source imaging datasets and does not constitute a clinical trial.