Adaptive Reward-Poisoning Attacks against Reinforcement Learning

Authors: Xuezhou Zhang, Yuzhe Ma, Adish Singla, Xiaojin Zhu
Published: 2020-03-27 | Updated: 2020-06-22

Source: https://arxiv.org/abs/2003.12613

PDF: https://arxiv.org/pdf/2003.12613

Labels Predicted by AI

強化学習攻撃 バックドア攻撃 Q-Learningアルゴリズム

Please note that these labels were automatically added by AI. Therefore, they may not be entirely accurate.
For more details, please see the About the Literature Database page.

Abstract

In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward r_t into r_t + δ_t at each step, with the goal of forcing the RL agent to learn a nefarious policy. We categorize such attacks by the infinity-norm constraint on δ_t: We provide a lower threshold below which reward-poisoning attack is infeasible and RL is certified to be safe; we provide a corresponding upper threshold above which the attack is feasible. Feasible attacks can be further categorized as non-adaptive where δ_t depends only on (s_t, a_t, s_t + 1), or adaptive where δ_t depends further on the RL agent’s learning process at time t. Non-adaptive attacks have been the focus of prior works. However, we show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps polynomial in state-space size |S|, whereas non-adaptive attacks require exponential steps. We provide a constructive proof that a Fast Adaptive Attack strategy achieves the polynomial rate. Finally, we show that empirically an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.