Research · Web archives
Official government sites in Common Crawl
As of 2026-11-06. Crawls CC-MAIN-2026-39 and CC-MAIN-2026-43.
Common Crawl is a free, open repository of web pages that anyone can use. Jurisdictional builds on it to read public information that governments publish, without adding load to their servers. We measured how much of the official U.S. government web below the federal level is in two recent crawls.
48.0%
We checked 35,447 official domains and hosts of cities, counties, special districts, school districts, states, territories, and tribal governments. 17,012 of them (48.0%) had at least one HTML page in either crawl. The other 18,435 had none, as of these two crawls.
What we found
Having pages doesn't mean having many. In CC-MAIN-2026-43, the median domain with pages had 22, and 3,490 of 16,044 had 5 or fewer.
| What the crawls show for a domain | Domains | Share |
|---|---|---|
| HTML pages in either crawl | 17,012 | 48.0% |
| No trace: no page, no error, and no robots.txt request | 10,693 | 30.2% |
| robots.txt answered 401 or 403 | 2,053 | 5.8% |
| robots.txt disallows the crawler from the home page | 534 | 1.5% |
| robots.txt answered with a server error or 429 | 388 | 1.1% |
| Pages answered with a refusal or server error | 1,246 | 3.5% |
| Pages only redirected elsewhere | 673 | 1.9% |
| Other or not explained | 2,848 | 8.0% |
A domain with no pages isn't always a gap, because some have nothing to crawl.
So we made one plain web request to 9,731 domains with no HTML in either crawl, nearly all of them .gov.
4,858 answered as working sites, in the checks made on 2026-11-02.
The rest were not working sites, redirected elsewhere, or could not be classified.
For those 4,858 working sites, the crawls show:
| What the crawls show |
Working .gov sites
|
Share |
|---|---|---|
| No trace | 1,719 | 35.4% |
| robots.txt answered 401 or 403 | 967 | 19.9% |
| robots.txt disallows the crawler from the home page | 213 | 4.4% |
| robots.txt answered with a server error or 429 | 114 | 2.3% |
| Pages answered with a refusal or server error | 605 | 12.5% |
| Pages only redirected elsewhere | 12 | 0.2% |
| Other or not explained | 1,228 | 25.3% |
How robots.txt matters
A robots.txt file tells crawlers what a site would like them to skip. Common Crawl follows it. Some sites don't serve the file at all to the crawler, and answer the request for it with a refusal.
In CC-MAIN-2026-43, 88.8% of domains that served robots.txt normally had HTML pages (13,702 of 15,432). Among domains that answered 401 or 403 and never served it, 8.9% did (212 of 2,374). CC-MAIN-2026-39 shows the same pattern: 87.6% and 8.0%.
For about one working site in four in the table above, robots.txt is the closest cause we can see: refused, or disallowing the crawler. A refusal may not be deliberate, since firewalls and hosting services can answer on a site's behalf. Of the 534 domains whose robots.txt disallows the crawler, 386 name CCBot and 148 disallow all crawlers.
Why it matters
These pages are public records: meeting notices, contact details, services, and rules. A site missing from a shared repository is also missing from whatever gets built on it, including search tools, research, and archives. The largest group of working sites shows no sign the crawl ever reached them, so this is mostly a question of being found.
Method
- Official list: the CISA .gov registry of 2026-10-27 for local, state, territorial, and tribal types, plus non-.gov hosts from Jurisdictional's published records. We checked non-.gov hosts added in this version by reading the site, or from agency records where a page could not be read.
- Crawl data: Common Crawl's public index for each crawl: its warc, robotstxt, and crawldiagnostics parts.
- HTML: an HTTP 200 capture detected as HTML.
- Robots files: we read 5,673 robots.txt files for the 2,008 domains that served one and had no HTML. A domain counts as disallowing the crawler only if every file read for it refuses the home page.
- One cause per domain: the first match in the table's order, and a domain that served robots.txt on any host is judged by that file.
- Working site: one plain web request from one network that returned a site, not an error, parked page, or default server page.
Limits
- This is a snapshot of two crawls. A site may appear in other crawls, and each crawl visits only part of the web.
- "No trace" is the absence of a record in these crawls, not a statement about the site.
- The tables show what the crawl recorded, which suggests a cause but doesn't prove one. We have not seen the crawler's own logs.
- robots.txt is judged for the home page only.
- The official list is what we can verify today, so it is neither complete nor a count of governments. A government can have several domains.
- The check for working sites covers .gov domains only.
- The 1,228 "other" sites are unexplained. Many were allowed by robots.txt and left no refusal.
- We publish counts, not lists of domains.
Data and credit
Download:
crawl-coverage-by-type-2026-11-06.csv
, counts by government type.
Generated by mcc coverage-causes
in the municipal-common-crawl project (PLAN S16).
The data comes from Common Crawl. This project isn't affiliated with Common Crawl, and we're grateful for its work.
Suggested citation
Jurisdictional. 2026. "Official Government Sites in Common Crawl." Data as of November 6, 2026. https://jurisdictional.org/research/official-sites-in-common-crawl.