Web crawler

A web crawler is the program a search engine uses to discover and download the web: start from some known pages, fetch them, pull out every link, and add new links to a to-do list called the frontier. At scale that means billions of URLs and thousands of fetchers working in parallel. What makes it interesting is politeness, since hammering one website with parallel requests is effectively an attack, so the frontier is organised per host, respecting robots.txt and spacing out requests to each site. The other big concern is not fetching the same page twice, which takes URL normalisation and a huge “already seen” set, often a Bloom filter, plus defences against traps like infinitely generated calendar pages.