In this paper, we present an operational system for cyber threat intelligence
gathering from various social platforms on the Internet particularly sites on
the darknet and deepnet. We focus our attention to collecting information from
hacker forum discussions and marketplaces offering products and services
focusing on malicious hacking. We have developed an operational system for
obtaining information from these sites for the purposes of identifying emerging
cyber threats. Currently, this system collects on average 305 high-quality
cyber threat warnings each week. These threat warnings include information on
newly developed malware and exploits that have not yet been deployed in a
cyber-attack. This provides a significant service to cyber-defenders. The
system is significantly augmented through the use of various data mining and
machine learning techniques. With the use of machine learning models, we are
able to recall 92% of products in marketplaces and 80% of discussions on forums
relating to malicious hacking with high precision. We perform preliminary
analysis on the data collected, demonstrating its application to aid a security
expert for better threat analysis.