Stabilizing updates in differentially private stochastic gradient descent with buffered rejection

Abstract Differentially private stochastic gradient descent is a standard algorithm for training deep models on sensitive data, but under tight privacy budgets it must add large noise to every step, which slows convergence and reduces accuracy. Selective update methods for differential private stochastic gradient descent reject updates that fail a noisy validation test and save privacy cost, but each decision still relies on a single noisy signal and remains unstable. We propose a differential private training algorithm that combines a buffered rejection mechanism with a phased parameter decay strategy for stochastic gradient descent. In each iteration, the proposed algorithm maintains two candidate updates, evaluates their privately perturbed loss improvements, and applies a local preferential choice. This buffered comparison spends privacy budget on directions that are more likely to be beneficial. The phased decay strategy tracks validation accuracy and gradually adjusts the noise multipliers, learning rate, and rejection threshold to match the current training stage. Experiments on MNIST, Fashion-MNIST, CIFAR-10, and IMDb with identical privacy budgets show that the proposed algorithm consistently improves test accuracy over the standard differential private stochastic gradient descent and the selective update based differential private stochastic gradient descent, typically by 0.5–2 percentage points, and converges faster at the same privacy level. Membership inference evaluations report area under the ROC curve values close to 0.5, indicating that these gains do not weaken empirical privacy.