Botnets are increasingly used by malicious actors, creating increasing threat
to a large number of internet users. To address this growing danger, we propose
to study methods to detect botnets, especially those that are hard to capture
with the commonly used methods, such as the signature based ones and the
existing anomaly-based ones. More specifically, we propose a novel machine
learning based method, named Recurrent Variational Autoencoder (RVAE), for
detecting botnets through sequential characteristics of network traffic flow
data including attacks by botnets. We validate robustness of our method with
the CTU-13 dataset, where we have chosen the testing dataset to have different
types of botnets than those of training dataset. Tests show that RVAE is able
to detect botnets with the same accuracy as the best known results published in
literature. In addition, we propose an approach to assign anomaly score based
on probability distributions, which allows us to detect botnets in streaming
mode as the new networking statistics becomes available. This on-line detection
capability would enable real-time detection of unknown botnets.