These labels were automatically added by AI and may be inaccurate. For details, see About Literature Database.
Abstract
Parameter-efficient fine-tuning (PEFT) has emerged as a practical solution
for adapting large language models (LLMs) to custom datasets with significantly
reduced computational cost. When carrying out PEFT under collaborative learning
scenarios (e.g., federated learning), it is often required to exchange model
updates (or gradients) across parties. These gradients, even with limited
dimensions, can cause severe breach of data privacy. Recent works have shown
that both contextual prefixes and personally identifiable information (PII) can
be exposed through gradients. However, \emph{simultaneously} and
\emph{accurately} recovering both components from the same training instance
remains infeasible due to the following challenges: 1) limited number of PEFT
parameters; 2) high-dimensional token spaces; and 3) large batch sizes. We
propose ReCIT, a novel privacy attack that addresses all challenges, and
achieves recovery of \emph{full} private data from PEFT gradients with high
fidelity. Specifically, ReCIT proposes to enhance the memorization capability
of the pre-trained model through malicious fine-tuning with Personal Notes;
ReCIT also proposes a novel filter-based token extraction technique and a token
pairing mechanism, to accurately reconstruct tokens from the training sequences
with large batch sizes. Extensive evaluations show that ReCIT consistently
outperforms state-of-the-art gradient inversion and memorization-based attacks
across different PEFT paradigms. It achieves up to 10$\times$ higher PII
recovery rates and remains effective across varying batch sizes, especially in
settings where prefix reconstruction is intractable for conventional
approaches. These findings highlight an urgent need to reassess the privacy
guarantees of PEFT, especially in decentralized or shared training
environments.