Crawler Overview

Companywell is a data aggregation company that uses information that is available to the public and combines it into an easy to use platform and API. Our crawling technology is built very similar to Google’s search engine crawlers, but instead of building a search engine we’re using our crawlers to build company and people datasets.

With our crawlers we take the raw crawling data and build relationships between companies and people to build a perfect profile of data.

We crawl a lot of data. Here’s a live counter of our stats.

LIVE METRICS
  • ... Lifetime Page Count
  • ... Monthly Pages Indexed
  • ... Pages Crawled Today

Technical Details

Similar to Google, our crawler will only access public web pages and anything that is protected behind a secure page will not be indexed.