Pre-training exploits public datasets to pre-train an advanced machine
learning model, so that the model can be easily tuned to adapt to various
downstream tasks. Pre-training has been extensively explored to mitigate
computation and communication resource consumption. Inspired by these
advantages, we are the first to explore how model pre-training can mitigate
noise detriment in differentially private federated learning (DPFL). DPFL is
upgraded from federated learning (FL), the de-facto standard for privacy
preservation when training the model across multiple clients owning private
data. DPFL introduces differentially private (DP) noises to obfuscate model
gradients exposed in FL, which however can considerably impair model accuracy.
In our work, we compare head fine-tuning (HT) and full fine-tuning (FT), which
are based on pre-training, with scratch training (ST) in DPFL through a
comprehensive empirical study. Our experiments tune pre-trained models
(obtained by pre-training on ImageNet-1K) with CIFAR-10, CHMNIST and
Fashion-MNIST (FMNIST) datasets, respectively. The results demonstrate that HT
and FT can significantly mitigate noise influence by diminishing gradient
exposure times. In particular, HT outperforms FT when the privacy budget is
tight or the model size is large. Visualization and explanation study further
substantiates our findings. Our pioneering study introduces a new perspective
on enhancing DPFL and expanding its practical applications.
外部データセット
ImageNet-1K
CIFAR-10
CHMNIST
Fashion-MNIST
参考文献
Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security
Deep learning with differential privacy
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, L. Zhang
Published: 2016
Proceedings of the AAAI Conference on Artificial Intelligence
A Generalized Shuffle Framework for Privacy Amplification: Strengthening Privacy Guarantees and Enhancing Utility
Please tell me more: Privacy impact of explainability through the lens of membership inference attack
Han Liu, Yuhao Wu, Zhiyuan Yu, Ning Zhang
Published: 2024
IEEE Transactions on Dependable and Secure Computing
An Optimized Sparse Response Mechanism for Differentially Private Federated Learning
J. Ma, Y. Zhou, L. Cui, S. Guo
Published: 2024
arxiv
被引用数 1
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas
Published: 2016.2.18
Modern mobile devices have access to a wealth of data suitable for learning
models, which in turn can greatly improve the user experience on the device.
For example, language models can improve speech recognition and text entry, and
image models can automatically select good photos. However, this rich data is
often privacy sensitive, large in quantity, or both, which may preclude logging
to the data center and training there using conventional approaches. We
advocate an alternative that leaves the training data distributed on the mobile
devices, and learns a shared model by aggregating locally-computed updates. We
term this decentralized approach Federated Learning.
We present a practical method for the federated learning of deep networks
based on iterative model averaging, and conduct an extensive empirical
evaluation, considering five different model architectures and four datasets.
These experiments demonstrate the approach is robust to the unbalanced and
non-IID data distributions that are a defining characteristic of this setting.
Communication costs are the principal constraint, and we show a reduction in
required communication rounds by 10-100x as compared to synchronized stochastic
gradient descent.