Instance Prompt Meets Dynamic Adapter: Continual Test-Time Adaptation for Urban Scene Image Segmentation

Recently, to enhance the ability of deployed models to adapt to continually changing environments, the continual test-time adaptation (CTTA) for urban scene image (USI) segmentation has been preliminarily studied. CTTA aims to improve the performance of models while continually adapting to multiple different USI domains that arrive online in sequence. However, due to the lack of effective solutions for error accumulation and catastrophic forgetting, existing methods fail to achieve satisfactory performance. To this end, this article proposes a novel CTTA method featuring instance prompts and dynamic adapters. The key innovations include the following. First, an instancewise prompt generator based on the depth anything model is proposed, which can provide more precise prompt information at different levels and effectively alleviate error accumulation. Second, a dynamic adapter module is proposed, where the short- and long-term adapters learn domain-specific and domain-invariant knowledge, respectively, to quickly adapt to new domains while avoiding catastrophic forgetting. Finally, an uncertainty estimation strategy with multiple predictions is designed to further optimize the CTTA of USIs. Two typical CTTA scenarios, i.e., the continual discrepancies in geographical regions and weather conditions, are established using autonomous aerial vehicle images and street-view images, respectively. Extensive experiments indicate that the proposed method outperforms existing methods by a sizeable margin, with an improvement of at least 1.8%–2.1%.