The hardest part of navigating the Deep Web is simply knowing where to start.
The sites hosted on the Tor network, which anonymizes both users’ traffic and websites’ location, cannot be accessed through the regular Internet and aren’t directly indexed by Google. However, there are a number of search engines and directories to help you crawl your way through the dark. As with Google and Bing, each comes with distinct pros and cons.
Juha Nurmi is a 26-year-old developer living in Finland and working at a technical university as a researcher. He just returned from Google’s Summer of Code, where he worked to improve the functionality of Ahmia.fi, one of the most representative and up-to-date Tor directories and search engines. It’s available both on the clearnet (the Internet you’re used to using), and as a hidden service (a website hosted on the Tor network).
“There’s a current trend to make the Tor network more accessible and user-friendly,” Nurmi said. “My motivation to write search engine software is to support human rights, such as privacy and freedom of speech.”
The Ahmia project is part of the Hermes Center for Transparency and Digital Human Rights, a nonprofit that supports the Tor network and works on online anonymity systems, such as Globaleaks and Tor2web.
Nurmi explained that there are basically three characteristics that distinguish search engines on the Deep Web from those on the clearnet—all of which make it far more difficult to provide accurate and timely results.
“There’s a current trend to make the Tor network more accessible and user-friendly.” —Juha Nurmi
“First, the linking between onion sites”—hidden sites on the Tor network with the top-level domain suffix .onion—“is thin; as a result, algorithms using the backlinks aren’t working very well,” he said. “Second, it takes time to crawl everything because Tor is slow. Lastly, onion sites are replacing their addresses all the time.”
Ahmia has indexed about 1,229 hidden websites on the Tor network. That’s not including sites running other services, like Internet relay chat (IRC) or the instant-messaging service Jabber. That also doesn’t include the considerable number of websites that don’t publicly advertise their existence, thus making it virtually impossible for Ahmia to find them. Ahmia also filters out child pornography sites, of which there are plenty lurking in the Deep Web.
Even still, Ahmia’s study of the most popular hidden services on the Tor network paints an unflattering picture of the Deep Web. The most popular destination by is Pinkmeth, a revenge-porn site where users upload sexually explicit material without consent from its subjects. The site started on the regular Internet, but like leaked celebrity nude photos of the past few weeks, it’s since retreated to the Deep Web, a step removed from authorities. In terms of popularity, PinkMeth’s followed by the bulletin board Anonymous BBS; Hidden Wiki, a well-known directory collecting hundreds of links to other hidden websites; and the forum IntelExchange.
Here’s what the searches on Ahmia look like in word cloud form:
Ahmia is far from the only service looking to improve search capabilities on the Deep Web.
Torsearch.se, developed by systems engineer Chris MacNaughton, actually works on the clearnet, and as such, it’s one of the faster search engines around.
“The way I’m capable of indexing onions is by using Nutch, an Apache project originally built by Yahoo to handle crawling the Internet, to crawl onions through Tor,” MacNaughton explained to The Kernel. “When I detect that a user is not using Tor, I will modify the onion links to point to onion.to, a system using the Tor2web proxy.”
Ahmia’s study of the most popular hidden services on the Tor network paints an unflattering picture of the Deep Web.
By modifying the onion links, any hidden website can be viewed with a normal browser. In short, Tor2web serves as a proxy network that lets Internet users access anonymous servers, even if they are not using Tor Browser. Doing so only protects the publishers, not readers, however, since their behavior is not being anonymized through Tor. It’s a system that’s used to make the content from hidden sites reach a wider audience.
Torch is another old search engine for onion websites. It needs to be accessed through Tor Browser. It claims to index 147,776 onion pages. There’s also an advanced search form where it’s possible to sort results for types of documents and other parameters. Similarly sleek and only available on Tor, TorFind offers a search engine for I2P, a decentralized and anonymous network that’s optimized for the file-sharing service BitTorrent.
Grams is another new entry to the Deep Web. As its name suggests, it’s used primarily as a gateway to black markets like Silk Road 2, the successor to the infamous marketplace seized by the FBI last year. Grams lets people search for products and listings across different marketplaces, including Agora, Evolution, and Pandora (not to be confused with the music service).
However, as with many early search engines on the Deep Web, the user experience on Grams is not always smooth. Most of the markets it searches routinely suffer temporary outages, be it from distributed denial-of-service attacks or routine security issues. That adds yet another layer of complexity, not to mention the fact that most of the markets require users’ registration, which makes it difficult to jump from Grams to specific listings.
What Grams lacks in ease of use it makes up for in ingenuity and ambition. It’s increased functionality since launching earlier this year, now allowing users the ability to search for the name of a vendor (who could be present on more than one marketplace) or even for that person’s PGP key, a digital signature of sorts that increases security for email communication.
Some of the best places to find up-to-date information on the Tor network marketplaces aren’t even search engines.
All You’re Wiki is the single best entry point for users both new and experienced diving into the Deep Web. Acting as a comprehensive Deep Web directory, it provides a well-maintained and long list of hundreds of links and explainers to help you find your way.
What Grams lacks in ease of use it makes up for in ingenuity and ambition.
In fact, wikis have long been among the most important search tools on the Deep Web. The Hidden Wiki, an uncensored directory—meaning unlike All You’re Wiki and Ahmia, its links include material like child porn— is a seven-year-old enterprise that may be among the most popular and longstanding institutions in Deep Web history.
Then there’s DeepDotWeb, an editorial resource readily available on the surface Internet that lists all of the underground bazaars with tips and news.
There’s a whole other world out there. You just have to know the right search engines to find it.
Photos via DragonLord878/Flickr (CC BY-SA 2.0) and James Lee/Flickr (CC BY 2.0) | Remix by Rob Price