These labels were automatically added by AI and may be inaccurate. For details, see About Literature Database.
Abstract
Large Language Models (LLMs) deployed in enterprise settings (e.g., as
Microsoft 365 Copilot) face novel security challenges. One critical threat is
prompt inference attacks: adversaries chain together seemingly benign prompts
to gradually extract confidential data. In this paper, we present a
comprehensive study of multi-stage prompt inference attacks in an enterprise
LLM context. We simulate realistic attack scenarios where an attacker uses
mild-mannered queries and indirect prompt injections to exploit an LLM
integrated with private corporate data. We develop a formal threat model for
these multi-turn inference attacks and analyze them using probability theory,
optimization frameworks, and information-theoretic leakage bounds. The attacks
are shown to reliably exfiltrate sensitive information from the LLM's context
(e.g., internal SharePoint documents or emails), even when standard safety
measures are in place.
We propose and evaluate defenses to counter such attacks, including
statistical anomaly detection, fine-grained access control, prompt sanitization
techniques, and architectural modifications to LLM deployment. Each defense is
supported by mathematical analysis or experimental simulation. For example, we
derive bounds on information leakage under differential privacy-based training
and demonstrate an anomaly detection method that flags multi-turn attacks with
high AUC. We also introduce an approach called "spotlighting" that uses input
transformations to isolate untrusted prompt content, reducing attack success by
an order of magnitude. Finally, we provide a formal proof of concept and
empirical validation for a combined defense-in-depth strategy. Our work
highlights that securing LLMs in enterprise settings requires moving beyond
single-turn prompt filtering toward a holistic, multi-stage perspective on both
attacks and defenses.