Network intrusion detection (NID) is an essential defense strategy that is used to discover the trace of suspicious user behaviour in large-scale cyberspace, and machine learning (ML), due to its capability of automation and intelligence, has been gradually adopted as a mainstream hunting method in recent years. However, traditional ML based network intrusion detection systems (NIDSs) are not effective to recognize unknown threats and their high detection rate often comes with the cost of high false alarms, which leads to the problem of alarm fatigue. To address the above problems, in this paper, we propose a novel neural network based detection system, DualNet, which is constructed with a general feature extraction stage and a crucial feature learning stage. DualNet can rapidly reuse the spatial-temporal features in accordance with their importance to facilitate the entire learning process and simultaneously mitigate several optimization problems occurred in deep learning (DL). We evaluate the DualNet on two benchmark cyber attack datasets, NSL-KDD and UNSW-NB15. Our experiment shows that DualNet outperforms classical ML based NIDSs and is more effective than existing DL methods for NID in terms of accuracy, detection rate and false alarm rate.