Adaptive Reward-Poisoning Attacks against Reinforcement Learning

Authors: Xuezhou Zhang, Yuzhe Ma, Adish Singla, Xiaojin Zhu
Published: 2020-03-27 | Updated: 2020-06-22

Source: https://arxiv.org/abs/2003.12613

PDF: https://arxiv.org/pdf/2003.12613

AIにより推定されたラベル

強化学習攻撃 バックドア攻撃 Q-Learningアルゴリズム

※ こちらのラベルはAIによって自動的に追加されました。そのため、正確でないことがあります。
詳細は文献データベースについてをご覧ください。

Abstract

In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward r_t into r_t + δ_t at each step, with the goal of forcing the RL agent to learn a nefarious policy. We categorize such attacks by the infinity-norm constraint on δ_t: We provide a lower threshold below which reward-poisoning attack is infeasible and RL is certified to be safe; we provide a corresponding upper threshold above which the attack is feasible. Feasible attacks can be further categorized as non-adaptive where δ_t depends only on (s_t, a_t, s_t + 1), or adaptive where δ_t depends further on the RL agent’s learning process at time t. Non-adaptive attacks have been the focus of prior works. However, we show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps polynomial in state-space size |S|, whereas non-adaptive attacks require exponential steps. We provide a constructive proof that a Fast Adaptive Attack strategy achieves the polynomial rate. Finally, we show that empirically an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.