BTSTNet: A Beamforming-Based Target Separation Transformer Network for Passive SONAR

Passive SONAR has attracted significant attention in covert operations, enabling vessel detection and classification through spectrogram analysis of received signals without active transmission. However, in realistic ocean environments, interference from multiple vessels and ambient noise make accurate analysis highly challenging. As deep learning-based source separation models rely on time–frequency representations, they often fail in complex environments, making spatial cues crucial for accurate target separation. Therefore, we propose BTSTNet, a beamforming-based transformer network that performs spatially aware target separation from complex acoustic mixtures. BTSTNet processes multi-channel beamformed inputs as a joint space–time–frequency representation and separates the target signal from complex mixtures in the target direction. We integrate a hybrid transformer encoder–decoder architecture with a novel Cross-Channel Aggregation Head (CCA-Head) to model spatial dependencies across channels. In addition, we construct a new beamforming-based passive SONAR mixture dataset that simulates diverse underwater conditions. Experimental evaluation on the constructed simulation dataset indicates that BTSTNet achieves superior separation performance compared to existing deep learning-based source separation models.