Deep-net Sites (dark-net). This scenario refers to deep-net sites, which are widely used for underground communication. “The Onion Router” (Tor) is a free software dedicated to protect the privacy of its users by obscuring traffic analysis as a form of network surveillance (Matusitz 2008). The network traffic in Tor is guided through a number of volunteer-operated servers (also called “nodes”), which encrypt the information. It blindly passes on neither registering where the traffic came from nor where it is headed, disallowing any tracking. Effectively, this allows not only for anonymized browsing (the IP-address revealed will only be that of the last node), but also for circumvention of censorship. This enables the possibility of create groups of discussion from terrorists through anonymous communication.

Dark Markets. This scenario is related to users who advertise and sell their wares on marketplaces. Dark-marketplaces provide a new avenue to gather information about the cyber threat landscape. Through marketplaces is possible to sell goods and services related to malicious hacking, drugs, pornography, weapons and software services. Only a small fraction of products (13% in our collected data to date) are related to malicious hacking. Vendors often advertise their products on forums to attract attention towards their goods and services.

Dark Forums. In this context, user-oriented platforms have the sole purpose of enabling communication. Dark forums provide the opportunity for the emergence of a community of like-minded individuals – regardless of their geophysical location. The administrators set up the dark forums with communication safety for their members in mind. Whereas the structure and the organization of Darknet-hosted forums might be very similar to more familiar web-forums, the topics and concerns of the users vary distinctly. Some forums address malicious hackers feature discussions on programming, other on hacking, other on cyber-security and so on. Specific threads are dedicated to security concerns like privacy and online-safety – topics which plug back into and determine the structures and usage of the platforms.

In order to improve the level of security in the above mentioned scenarios, different and innovative mechanisms should be combined, should be used. Network monitoring has been extensively used for security, forensics and anomaly detection, whose main objective is to identify malicious activities based on traffic patterns and to trigger alerts. Those alerts are often processed by security experts who can rely on advanced log correlation engines or SIEM (Security information and event management) for incident respond purposes (OCR 2017) with manual investigation or confirmation as a second step. In this case the application of system monitoring to Dark-net represents an effective method to analyze malicious activities on networks including the Internet.

If participated and monitored, these communities can provide law enforcement and security agencies with information and intelligence to be employed for identifying and preventing potential on-going or planned cyber-criminal activities. In other words, they can offer insight on cyber-criminal individual profiles, their connections, behaviour, capabilities, as well as intentions.

Through monitoring darknet for example is indeed possible to understand and (i) the current status of ongoing activities, (ii) what the trend in terms of “products on the market” is, (iii) how long a product stay on the market. Then on the basis of such information to investigate about origin of products, how such product will be used and so on. Three main ways to enable monitoring are described in the following and they are based on: Crawlers and Parsers, Machine Learning approaches and virtual HUMINTs (Meegahapola, Alwis, Heshan, Mallawaarachchi, Meedeniya and Jayarathna 2017; Nunes et al. 2016; Dragos 2012).

Crawler and Parsers. They are programs designed to traverse the website and retrieve HTML documents. Topic based crawlers have been used for focused crawling where only webpages of interest are retrieved. More recently, focused crawling was employed to collect forum discussions from darknet. Different crawlers for different platforms (markets/forums) identified by experts, have been designed, due to the structural difference and access control measures for each platform. Each crawler addresses design challenges like accessibility, unresponsive server, repeating links creating a loop etc. to gather information regarding products from markets and discussions on forums.

Machine Learning approach. This is a combination of supervised and semi-supervised methods. Supervised methods include the well-known classification techniques of Naive Bayes (NB), random forest (RF), support vector machine (SVM) and logistic regression (LOG-REG). However, supervised techniques required labelled data, and this is expensive and often requires expert knowledge. Semi-supervised approaches work with limited labelled data by leveraging information from unlabelled data.

Virtual HUMINT. This approach aims at collecting tactical/operational intelligence from the information generated by members of the virtual communities. Basically, it consists in establishing and operating an intelligent virtual identity (avatar) to gain trust from and create long-term relationships with the members of the participated/monitored communities, as well as recruit, handle, manipulate and control them with the purpose of collecting information. When it comes to virtual HUMINT collection by intelligence authorities, the operations usually involve establishing an online alias/personality (including social media accounts) and membership of relevant web forums, etc. Targets are then befriended (or the targets befriend the alias). Interactions with the targets may be informed by a combination of analysis of signals intelligence (SIGINT), monitoring of the target’s online behaviour, and intelligence gathered offline / “on-the-ground” (Dhami 2011). Main elements of virtual HUMINT collection by private companies are of a similar nature: they include the fake ID (the creation of many fake digital personas on many platforms to engage hackers), the joining of a “gang” (the attempt to gain access to closed forums, chat rooms and cybercrime marketplaces), the bait and switch (offering seemingly purloined data or seeking to buy such data), as well as flirting and flattering (playing to the usual human vulnerabilities) (Hirschauge 2015). Evidently, HUMINT, whether it is on- or offline, has some general limitations as well. It requires a rather great deal of resources, extensive training and supplies (Koren 2015). Source credibility and reliability are two other points that are mentioned when it comes to the limits of HUMINT (Sullivan 2012). Furthermore, the lists of limitations with regard to HUMINT collection include the following aspects: a) HUMINT is dependent on the subjective interpersonal capabilities of the individual rather than on the abilities to operate collection equipment; b) information in response to specific requirements can only be collected if sources are available and identified that have that information, which might not always be the case; c) HUMINT collection takes a significant amount of time to develop; e) a potential lack of language proficiency in certain situations by the collector can significantly slow down collection efforts (US Army 2006). If cyber criminals know of intelligence agencies and/or private companies successfully conducting virtual HUMINT collection, its sheer existence can serve as a deterrent by forcing criminals to operate more cautiously. Furthermore, since virtual HUMINT collection is conducted online, HUMINT officers operate in a significantly less dangerous environment than in non-virtual cases of HUMINT. They can interact and develop relationships with criminals and sources from the comfort of their own office or home (Koren 2015). HUMINT seems also being especially effective in the case of criminals (rather than spies or terrorists), because in this field there is a larger pool of candidates that are likely to provide information for monetary incentives or be susceptible to recruitment under false pretenses, and their awareness of and “training” against intelligence activities might be lower (Magee 2010).