Crawling and bulk access

The database is open under the ODbL 1.0. Search engines, researchers, AI assistants and agents are all welcome. This page says how to get the data efficiently, so that one reader cannot slow the site down for everyone else. The terms for automated access are the agreement; this page is a guide to them.

Use these first

  1. Bulk data. Ask for the county or state you need, in CSV, JSON or HSDS, from the license page and we send it as a file. Every dataset we publish is listed in /data.json.
  2. The API and MCP server. Documented in llms.txt and on Developers. Without a key you get 100 requests a day per IP address. A free key gives 2,000, and the paid plans more.
  3. The sitemap. /sitemap.xml lists every public page: the main pages, and each jurisdiction, agency, body, person, service and place, with the date it last changed. Fetch only what changed since your last visit. For whole datasets, the API and the bulk data are faster than following links page by page.

If you crawl pages

The daily page allowance for AI crawlers

The API has its own daily limits, sent with every response in x-ratelimit-* headers.

AI systems

Reading a page to answer a question, the way a person or an assistant would, needs nothing more than following robots.txt. Credit the data by citing the record's page. Access at scale, such as building a dataset, training a model, or keeping a copy current, goes through the API with your own key or the bulk data, on a plan or a written agreement.

Need more?

To agree a bulk or training arrangement, tell us on the Support page before you crawl harder.