Hybrid attention optimized hierarchical multiscale transformer architecture for image super-resolution

Abstract The transformer architecture has demonstrated significant performance in image super-resolution (SR). However, for existing Transformer-based models, there are common drawbacks. They often fall short in local feature modeling and limited feature representation capabilities. When it comes to reconstructing high-resolution (HR) images, these deficiencies become more prominent, resulting in the poor restoration of fine details. In order to resolve the existing issues, we propose the Hierarchical Multiscale Transformer Architecture Based on Hybrid Attention (HMT). Notably, this architecture has the ability to effectively capture the fine-grained interactions between local image features and other regions, which in turn enables it to generate details that are clearer and more coherent. Specifically, we introduce the Fourier Mamba Synergy (FMSA) module, which adopts a dual branch hierarchical architecture design to achieve cross-domain feature collaboration and efficient long-range dependency modeling by constructing Fourier Spectrum Modulation (FSM) and Mamba Spatial Mixing (MSM). In addition, we introduced Dynamic Convolutional Attention (DCA), which separates feature branches based on channel splitting strategy and achieves multi-scale feature enhancement through long-range dependency capture and instance weight adaptation. Finally, we designed a multi-scale encoding and decoding architecture and dynamic fusion mechanism. HMT exceeds HGFormer by 0.07dB and 0.08dB in PSNR metrics at a scaling factor of 4 on BSD100 and Manga109 datasets respectively. Extensive experiments on public datasets show good performance in both objective metrics and visual quality. The code can be available at https://github.com/lingyuyan2014/HMT.