Despite all the advantages associated with Network Intrusion Detection Systems (NIDSs) that utilize machine learning (ML) models, there is a significant reluctance among cyber security experts to implement these models in real-world production settings. This is primarily because of their opaque nature, meaning it is unclear how and why the models make their decisions. In this work, we design a deep learning-based NIDS, ExpIDS to have high decision tree explanation fidelity, i.e., the predictions of decision tree explanation corresponding to ExpIDS should be as close to ExpIDS's predictions as possible. ExpIDS can also adapt to changes in network traffic distribution (drift). With the help of extensive experiments, we verify that ExpIDS achieves higher decision tree explanation fidelity and a malicious traffic detection performance comparable to state-of-the-art NIDSs for common attacks with varying levels of real-world drift.