Interobserver and intraobserver reliability for 2 grading systems for gastric ulcer syndrome in horses

Abstract Background Grading of equine gastric ulcer syndrome (EGUS) is undertaken in clinical and research settings, but the reliability of EGUS grading systems is poorly understood. Hypothesis/Objectives Investigate interobserver and intraobserver reliability of an established ordinal grading system and a novel visual analog scale (VAS), and assess the influence of observer experience. Animals Sixty deidentified gastroscopy videos. Methods Six observers (3 specialists and 3 residents) graded videos using the EGUS Council (EGUC) system and VAS. Observers graded the videos three 3 for each system, using a cross‐over design with at least 1 week between each phase. The order of videos was randomized for each phase. Methods Interobserver and intraobserver reliability were estimated using Gwet's agreement coefficient with ordinal weights applied (AC2) for the EGUC system and the intraclass correlation coefficient (ICC) for the VAS. Results Using the EGUC system, interobserver reliability was substantial for squamous (AC2 = 0.69; 95% confidence interval [CI], 0.57‐0.80) and glandular mucosa (AC2 = 0.72; 95% CI, 0.70‐0.75), and intraobserver reliability was substantial for squamous (AC2 = 0.80; 95% CI, 0.71‐0.90) and glandular mucosa (AC2 = 0.80; 95% CI, 0.74‐0.86). Interobserver reliability using the VAS was moderate for squamous (ICC = 0.64; 95% CI, 0.31‐0.96) and poor for glandular mucosa (ICC = 0.35; 95% CI, 0.06‐0.64), and intraobserver reliability was moderate for squamous (ICC = 0.74; 95% CI, 0.62‐0.86) and glandular mucosa (ICC = 0.56; 95% CI, 0.39‐0.72). Conclusions and Clinical Importance The EGUC system had acceptable intraobserver and interobserver reliability and performed well regardless of observer experience. Familiarity and observer experience improved reliability of the VAS.