Crawling and bulk access
The database is open under the ODbL 1.0. Search engines, researchers, AI assistants and agents are all welcome. This page says how to get the data efficiently, so that one reader cannot slow the site down for everyone else. The terms for automated access are the agreement; this page is a guide to them.
Use these first
- Bulk data. Ask for the county or state you need, in CSV, JSON or HSDS, from the license page and we send it as a file. Every dataset we publish is listed in /data.json.
- The API and MCP server. Documented in llms.txt and on Developers. Without a key you get 100 requests a day per IP address. A free key gives 2,000, and the paid plans more.
- The sitemap. /sitemap.xml lists every public page: the main pages, and each jurisdiction, agency, body, person, service and place, with the date it last changed. Fetch only what changed since your last visit. For whole datasets, the API and the bulk data are faster than following links page by page.
If you crawl pages
- Identify yourself honestly: a user agent that names your crawler and gives a URL or email to contact you.
- Follow robots.txt. Sign-in, account and edit pages are excluded; they hold no data.
- Keep a modest pace, and slow down when responses slow down.
-
On a 402 or 429 response, wait for the number of seconds in its
Retry-Afterheader before trying again.
The daily page allowance for AI crawlers
- Each named AI crawler, such as GPTBot or ClaudeBot, reads 2,500 pages a day free, counted across all of its IP addresses.
- The count is kept on each of our servers, so the total in a day can be a small multiple of that.
-
Past the allowance, pages answer
402 Payment RequiredwithRetry-Afterand links to the plans and the terms. - This page, the license, the terms, pricing, support and Developers always answer, so a crawler can read why it was stopped.
- Signed-in users are not counted.
The API has its own daily limits, sent with every response in x-ratelimit-*
headers.
AI systems
Reading a page to answer a question, the way a person or an assistant would, needs nothing more than following robots.txt. Credit the data by citing the record's page. Access at scale, such as building a dataset, training a model, or keeping a copy current, goes through the API with your own key or the bulk data, on a plan or a written agreement.
Need more?
To agree a bulk or training arrangement, tell us on the Support page before you crawl harder.