Using online detection and analysis is increasingly important as part of the overall threat assessment process. But as Internet traffic grows, so does the complexity of monitoring the growing data traffic volumes. This is especially true for the “deep web”, which is much larger than the “surface web”, which can be accessed using standard search engines such as Google, Bing and Yahoo! By combining tried and true threat assessment methodologies with specialist insight in deep web detection and monitoring, security providers can gain significant insight to detect, analyze and prevent threats from persons of interest.
In this blog, we’ll take a quick look at the state of the Internet today—including its hidden parts—and the methods and processes employed in order to guarantee clients the highest level of security.
The Surface Web, the Deep Web, and the Darknet
Internet and social media use continue to grow exponentially. As of 2015, Internet Live Stats has recorded over 1 billion unique websites and over 3 billion Internet users worldwide. There are now over 1.3 billion active Facebook users. By 2018, it is expected that half of the world’s 7 billion people will have Internet access – at home.
As impressive as these numbers are, they don’t even take the deep web into account.
Before digging into this further, we need to clarify some terms. we typically refer to as “the Internet” includes the “surface” web and the “deep” web.
- The Surface Web (also known as Visible Web, or Indexable Web) is what most people use and know. It includes all content that is by traditional search engines. Search engines like Google and Bing rely on programs called crawlers to build databases of the Web, essentially mapping its structure for easy, searchable access.
- The Deep Web (a.k.a Deepnet, Hidden Web, or Invisible Web) includes all content not indexed or findable by search engines. This includes private websites and forums, private email servers and more. Most of this content is innocuous. It’s extremely hard to measure the size of the Deep Web due to its very nature. Estimates about its size range from 500 times bigger than the surface web (a very low number, according to many experts) to several thousands of times bigger. One estimate puts the number of individual documents on the Deep Web at nearly 550 billion. By comparison, the Surface Web contains around one billion.
- The Darknet is one small part of the Deep Web. Often referred to as a single entity (“the Darknet”), it is in fact comprised of many independent servers, each of which is a “darknet” in its own right. The purpose of a darknet is to exchange data or content privately between trusted peers. Encryption, Virtual Private Networks and software like Tor are employed (if not outright required) in order to access a darknet. While the Darknet as a whole has received vast media attention for its shady content, it is also routinely used by journalists, sources, dissidents, or people who simply wish to keep their exchanges fully hidden from governments and corporations. Governments themselves occasionally rely on darknet nodes to share sensitive material.
Security and the Deep Web/Darknet
Simply monitoring social media platforms and surface content is insufficient. In order to effectively detect and thwart potential threats, it is crucial to gather intelligence on both the deep web and various darknets.
With so much data and information exchange—and therefore, potential threats—taking place beneath the surface web, not taking the deep web into account is not an option. It would be similar to investigating a house in order to secure it and choosing to ignore multiple rooms.
The Darknet’s main emphasis is security and anonymity, especially due to the encryption methods and methods of payment (Bitcoin or other crypto-currencies) being used, but it is certainly not foolproof. The most obvious example of it failing is the now-defunct Silk Road. Known as an online black market, this darknet site could be used to purchase illegal drugs, weapons and other criminal services while remaining fully anonymous—or so its creators hoped. The Silk Road, as well as other marketplaces such as Cloud 9 and Hydra, have been shut down by multiple international law enforcement groups, proving that the deep web can effectively be monitored—although it’s not an easy task.
Threat analysis in the deep web
What’s the difference between doing threat analysis on the surface web compared to the deep web?
- Deep web monitoring scans much more than the surface web—including the dark web, to find out what is being said about brands, companies, prominent people, etc.
- Deep web analysis is a specialty field that requires technology and human analytical expertise to provide actionable intelligence. Crawlers and automated tools by themselves are not enough.
- It’s important to search and monitor without leaving a trace, so that persons of interest do not become suspicious. Deep web monitoring first detects a person of interest, then follows that person. Should patterns change, or threats escalate, this can be monitored and acted upon. For example: a P.O.I. is first detected in Europe as a threat to a person in California, then travels to California. Protection providers can be informed of the development and take suitable responsive measures.
- There are many channels being monitored. Security professionals will look into credentials integrity (to see if a company was breached and if its data is circulating on any black market), private forums, and real-time communication streams such as chat channels. They’ll also be on the lookout for malware and unpatched exploits being spread.
Applying best practice threat analysis methods to online intelligence
So what methods do security professionals use to monitor deep parts of the Web, and why?
In many cases, they rely on the same key observations and principles developed by the Secret Service’s “Exceptional Case Study Project” – which was completed long before Internet use became so widespread. Some of the key points include:
- Attacks (including threats, plots, near-attacks, and assassinations) are “the products of understandable and often discernable processes of thinking and behavior”. They are not spontaneous acts, but are planned. The implications when it comes to online intelligence are clear: attackers use the Internet to research and plan their actions. Therefore, their patterns and processes can be detected online.
- There is no single, clear attacker profile, but many attackers had indicated interest in attacking a prominent figure beforehand, had interests in militant or radical ideas, almost all had histories of grievances and resentment toward prominent figures. Therefore, attackers can come from a wide variety of backgrounds, which means that a broad-scope investigation is necessary. It’s also important to correlate a potential attacker’s affiliations with radical/militant ideas or groups. This can be done by analyzing a would-be attacker’s intent, the target he/she is interested in, and his/her expressions of grievances.
- Attacks are a means to a goal, a way to solve a problem. Detecting perceived problems/grievances can lead to detecting motivation and likelihood of attack—giving security professionals a greater awareness of potential danger ahead of time.
- It’s important to keep in mind that those who make threats are not necessarily those that pose threats. Actual attackers rarely broadcast their intentions directly, but may do so indirectly. Therefore, not only do threats have to first be detected, they then must be monitored in order to differentiate between levels and types of threats. The monitoring is an on-going process. One must analyze the evolution of likelihood of attack to determine whether a threat will eventually manifest itself in a concrete manner.
AS Solution has good experience with deep web analytics as part of its intel and embedded intel operations. What about you – what are your experiences?