These labels were automatically added by AI and may be inaccurate. For details, see About Literature Database.
Abstract
Watermarking techniques for large language models (LLMs) can significantly
impact output quality, yet their effects on truthfulness, safety, and
helpfulness remain critically underexamined. This paper presents a systematic
analysis of how two popular watermarking approaches-Gumbel and KGW-affect these
core alignment properties across four aligned LLMs. Our experiments reveal two
distinct degradation patterns: guard attenuation, where enhanced helpfulness
undermines model safety, and guard amplification, where excessive caution
reduces model helpfulness. These patterns emerge from watermark-induced shifts
in token distribution, surfacing the fundamental tension that exists between
alignment objectives.
To mitigate these degradations, we propose Alignment Resampling (AR), an
inference-time sampling method that uses an external reward model to restore
alignment. We establish a theoretical lower bound on the improvement in
expected reward score as the sample size is increased and empirically
demonstrate that sampling just 2-4 watermarked generations effectively recovers
or surpasses baseline (unwatermarked) alignment scores. To overcome the limited
response diversity of standard Gumbel watermarking, our modified implementation
sacrifices strict distortion-freeness while maintaining robust detectability,
ensuring compatibility with AR. Experimental results confirm that AR
successfully recovers baseline alignment in both watermarking approaches, while
maintaining strong watermark detectability. This work reveals the critical
balance between watermark strength and model alignment, providing a simple
inference-time solution to responsibly deploy watermarked LLMs in practice.