distributed web crawler/indexer
Distributed web scraping/crawling platform including indexing for searching later
