Abstract
Machine learning is a key tool for Android malware detection, effectively
identifying malicious patterns in apps. However, ML-based detectors are
vulnerable to evasion attacks, where small, crafted changes bypass detection.
Despite progress in adversarial defenses, the lack of comprehensive evaluation
frameworks in binary-constrained domains limits understanding of their
robustness. We introduce two key contributions. First, Prioritized Binary
Rounding, a technique to convert continuous perturbations into binary feature
spaces while preserving high attack success and low perturbation size. Second,
the sigma-binary attack, a novel adversarial method for binary domains,
designed to achieve attack goals with minimal feature changes. Experiments on
the Malscan dataset show that sigma-binary outperforms existing attacks and
exposes key vulnerabilities in state-of-the-art defenses. Defenses equipped
with adversary detectors, such as KDE, DLA, DNN+, and ICNN, exhibit significant
brittleness, with attack success rates exceeding 90% using fewer than 10
feature modifications and reaching 100% with just 20. Adversarially trained
defenses, including AT-rFGSM-k, AT-MaxMA, improves robustness under small
budgets but remains vulnerable to unrestricted perturbations, with attack
success rates of 99.45% and 96.62%, respectively. Although PAD-SMA demonstrates
strong robustness against state-of-the-art gradient-based adversarial attacks
by maintaining an attack success rate below 16.55%, the sigma-binary attack
significantly outperforms these methods, achieving a 94.56% success rate under
unrestricted perturbations. These findings highlight the critical need for
precise method like sigma-binary to expose hidden vulnerabilities in existing
defenses and support the development of more resilient malware detection
systems.