Geometric Prior-Guided Federated Prompt Calibration

Federated Prompt Learning (FPL) enables efficient collaborative adaptation of large-scale models, but it remains vulnerable to data heterogeneity: locally optimized prompts may become biased, thereby weakening the aggregated global model. Existing methods mainly improve aggregation or regularization, but they do not directly address this source of local training bias. To address this issue, we propose Geometry-Guided Text Prompt Calibration (GGTPC), a framework that equips clients with a global geometric prior for more reliable local calibration. The prior captures the shape of the global data distribution through covariance-derived statistics. The server reconstructs it from aggregated local statistics, without exchanging raw data. While these statistics alone do not provide formal privacy guarantees, standard mechanisms such as differential privacy or secure aggregation can be applied on top. Based on this prior, clients employ a Geometry-Prior Calibration Layer (GPCL) during training to better align local feature distributions with the global structure. On the label-skewed CIFAR-100 dataset (<inline-formula> <tex-math notation="LaTeX">$\beta $ </tex-math></inline-formula>=0.1), GGTPC surpasses the strongest baseline by 2.09% in Top-1 accuracy, and the Top-1 accuracy gain increases to 9.00% under extreme skew (<inline-formula> <tex-math notation="LaTeX">$\beta $ </tex-math></inline-formula>=0.01). When used as a plug-and-play module on the domain-skewed Office-Home dataset, GGTPC improves FedAvg by 4.45% in average accuracy. Across different forms of heterogeneity, GGTPC delivers consistent improvements, making it a lightweight add-on for resource-constrained federated vision systems.