Adversarially Robust Distillation with Intermediate Sample Variance Regularization

Abstract

The widespread deployment of deep neural networks in safety critical applications has highlighted their severe vulnerability to adversarial examples, which are inputs crafted with imperceptible perturbations designed to mislead the model. While adversarial training remains the most effective defense mechanism, the massive computational cost and immense memory requirements of adversarially trained overparameterized models prohibit their deployment on resource constrained edge devices. Adversarially robust knowledge distillation has emerged as a promising solution to compress these large robust teacher models into smaller, efficient student models while preserving adversarial resilience. However, current robust distillation paradigms often struggle with intermediate feature collapse and catastrophic variance shift, where the student model fails to mimic the robust feature representations of the teacher under adversarial conditions. In this paper, we propose Adversarially Robust Distillation with Intermediate Sample Variance Regularization, a novel framework that enforces structural consistency between the teacher and student networks by directly constraining the variance of intermediate feature maps. By minimizing the discrepancy in sample variance across corresponding network stages, our method prevents the student model from developing excessively high variance activations that amplify adversarial perturbations. Through extensive evaluation on standard image classification benchmarks, we demonstrate that our approach significantly enhances both standard accuracy and adversarial robustness compared to baseline distillation methods. Furthermore, we provide a detailed analysis of the feature space dynamics, confirming that intermediate sample variance regularization effectively stabilizes the learning trajectory and induces tighter feature clustering.