Toward Well-Connected Retina Segmentation: A Fully Differentiable Endpoint Connectivity Loss (DECL)

Despite achieving high volumetric overlap, pixel-based deep learning segmentation approaches do not ensure true vascular connectivity, leading to fragmentations structurally analogous to Branch Retinal Artery Occlusions. This fundamental deficiency means that even segmentations achieving high pixel-wise overlap metrics (such as the Dice Score) do not guarantee or improve true vascular connectivity, rendering them unreliable for quantitative analysis and subsequent clinical risk stratification of high-risk systemic events, such as stroke. To resolve this, we introduce the Differentiable Endpoint Connectivity Loss (DECL), a novel, fully differentiable loss function that directly optimizes vascular continuity by precisely targeting errors in vessel endpoints. DECL leverages a differentiable soft-skeletonization module to impose two specialized penalties: a Normalized Count Penalty (<inline-formula> <tex-math notation="LaTeX">$\mathbf {L_{c}}$ </tex-math></inline-formula>), which rigorously penalizes the discrete difference in the number of predicted versus true endpoints (<inline-formula> <tex-math notation="LaTeX">$\Delta N_{p}$ </tex-math></inline-formula>), and a Normalized Distance Loss (<inline-formula> <tex-math notation="LaTeX">$\mathbf {L_{d}}$ </tex-math></inline-formula>), which penalizes the Euclidean distance between the weighted centers of the true and predicted endpoints. In comparative experiments across four datasets (DRIVE, CHASE DB1, STARE, and OCTA-500), we tested DECL against established methods including Dice CE, clDice, cbDice, and SAC Loss. DECL and its hybrid variant, <inline-formula> <tex-math notation="LaTeX">$\text {DECL}_{\text {clDice}}$ </tex-math></inline-formula>, consistently recorded the highest mean Dice Score and clDice scores. Specifically, <inline-formula> <tex-math notation="LaTeX">$\text {DECL}_{\text {clDice}}$ </tex-math></inline-formula> improved the mean clDice score from 0.839 to <inline-formula> <tex-math notation="LaTeX">$\mathbf {0.851}$ </tex-math></inline-formula> on CHASE DB1 and from 0.836 to <inline-formula> <tex-math notation="LaTeX">$\mathbf {0.842}$ </tex-math></inline-formula> on DRIVE, while minimizing Betti Number Error (<inline-formula> <tex-math notation="LaTeX">$\beta $ </tex-math></inline-formula>-Error) and Variation of Information. The DECL method establishes a robust, mathematically justified framework for generating high-fidelity vascular maps, essential for accurate biomarker quantification and reliable clinical assessment.