ANN-Based Equalization of Polarization Mixing and MZM Nonlinearity for Dual-Polarization Self-Coherent Transceivers in Free Space Optical Communications

Free-space optical (FSO) communication using dual-polarization self-coherent (DP-SC) transceivers is a promising solution for achieving high data rates and enhanced spectral efficiency in the next optical wireless systems generation. However, conventional digital signal processing (DSP) techniques, particularly those based on matrix inversion (MI), are highly sensitive to polarization-induced impairments under atmospheric turbulence. This paper proposes a novel DSP framework that leverages artificial neural networks (ANNs) to recover the transmitted message symbols by processing the in-phase and quadrature components extracted from the three-photodetector (3PD) photocurrents, by passing the need for channel estimation and matrix operations. The ANN is trained to learn the nonlinear mapping from photocurrents to original symbols, effectively compensating for polarization rotation, phase shifts, and Mach-Zehnder modulator (MZM) nonlinearities. Monte Carlo simulations demonstrate that the proposed ANN-based approach achieves significant lower bit error ratio (BER), higher Q-factor, and greater robustness compared to MI-based DSP. This efficient data-driven framework enables real-time, low-latency signal recovery, making it well-suited for adaptive and turbulence-resilient FSO communication links. The obtained results underline the importance of addressing polarization scattering and modulator nonlinearities, which are effectively mitigated by the proposed ANN approach. This work focuses on ANN-assisted mitigation of MZM nonlinearity and polarization-induced mixing in a dual-polarization self-coherent FSO receiver, while other FSO impairments such as pointing errors and scintillation are not explicitly modeled.