Knowing all possible web paths in the world is the initial step for making a search engine (SE). By means of SE one can analyze the web for the material he/she likes. In normal Domain Name System, each TLD provider (Top Level Domain) can sell or release list of all its domains. As an example .com TLD can sell or release all the domains which are end with “.com“. But the problem is more complicated in TOR (or other hidden service providers). In this post I will talk about my tool named Onion Harvester and how to find initial points for finding hidden services to be crawled.
But they have restriction for seeing results. For example SHODAN lets for unregistered users to view just 1 page (10 results). If you registered, the limitation is 5 pages. But again restriction!
So what to do?