AIOInsights research · 2026-11-01
The AI Blocked Census: how many California business websites shut out AI.
When a customer asks ChatGPT, Claude or Perplexity for a recommendation, the assistant can only use what it is allowed to read. AIOInsights checked 34,519 California business websites listed in OpenStreetMap and found that 3.3% of the 24,908 reachable sites (816) disallow at least one named major AI crawler in robots.txt, and 1.1% disallow every crawler.
Measured separately, by a different test: 8.7% (2,177 sites) blocked a request that claimed to be GPTBot. That request came from AIOInsights servers, not OpenAI's network, so a wall that checks crawler IP addresses can block it while letting the real GPTBot in. It is not added to the figures above.
Is your website on the wrong side of this? Run the free AIOInsights check: about 20 seconds, no account, and it says in plain words whether AI crawlers can read your site.
By type of business
| Type of business | Websites reachable | Disallow a named AI crawler | Block AI search | Block every crawler | Blocked a request that claimed to be GPTBot |
|---|---|---|---|---|---|
| Financial offices | 47 | 8.5% | 4.3% | 4.3% | 12.8% |
| Real estate offices | 256 | 8.2% | 1.6% | 0.0% | 10.9% |
| Financial advisors | 43 | 4.7% | 7.0% | 7.0% | 7.0% |
| Gyms and fitness | 729 | 4.3% | 1.0% | 0.8% | 10.7% |
| Other offices | 2,548 | 4.1% | 2.3% | 1.3% | 10.8% |
| Other trades | 623 | 4.0% | 1.0% | 0.6% | 9.5% |
| Auto repair and dealers | 863 | 3.7% | 1.3% | 1.0% | 15.9% |
| Shops and retail | 7,373 | 3.2% | 1.2% | 0.8% | 8.4% |
| Hotels and lodging | 603 | 3.2% | 0.2% | 0.2% | 7.3% |
| Doctors and clinics | 1,339 | 3.1% | 0.7% | 0.4% | 11.4% |
| Restaurants, cafes and bars | 8,130 | 3.1% | 1.0% | 0.9% | 6.7% |
| Law offices | 391 | 2.8% | 0.3% | 0.3% | 7.9% |
| Contractors and trades | 453 | 2.4% | 0.7% | 0.4% | 6.0% |
| Dentists | 630 | 2.2% | 6.2% | 5.7% | 12.9% |
| Other businesses | 373 | 2.1% | 1.1% | 0.8% | 10.7% |
| Insurance agencies | 172 | 1.7% | 2.3% | 0.6% | 4.1% |
| Veterinarians | 224 | 0.4% | 4.5% | 4.0% | 12.5% |
| Accountants | 60 | 0.0% | 16.7% | 16.7% | 21.7% |
| Tax advisors | 50 | 0.0% | 10.0% | 10.0% | 16.0% |
By city
Cities with at least 25 reachable business websites in the census.
| City | Websites reachable | Disallow a named AI crawler | Block AI search | Block every crawler | Blocked a request that claimed to be GPTBot |
|---|---|---|---|---|---|
| San Diego | 1,234 | 3.2% | 0.9% | 0.5% | 6.8% |
| San Francisco | 1,219 | 2.5% | 0.7% | 0.6% | 6.6% |
| Los Angeles | 1,154 | 3.1% | 1.0% | 0.6% | 7.2% |
| San Jose | 973 | 3.5% | 2.4% | 2.0% | 10.3% |
| Fresno | 763 | 4.2% | 2.1% | 1.7% | 10.4% |
| Santa Barbara | 683 | 2.8% | 0.4% | 0.3% | 6.4% |
| Berkeley | 446 | 3.1% | 0.9% | 0.0% | 8.1% |
| Pasadena | 410 | 2.0% | 2.4% | 2.4% | 8.0% |
| Oakland | 276 | 3.6% | 1.8% | 1.4% | 7.2% |
| Goleta | 221 | 4.5% | 0.9% | 0.5% | 10.0% |
| Sacramento | 211 | 2.4% | 0.5% | 0.0% | 9.5% |
| Simi Valley | 200 | 2.0% | 2.0% | 2.0% | 7.0% |
| Sunnyvale | 183 | 3.8% | 2.7% | 2.2% | 12.0% |
| Santa Cruz | 176 | 4.0% | 1.1% | 0.6% | 8.0% |
| West Hollywood | 163 | 4.9% | 1.8% | 1.2% | 7.4% |
| Santa Clara | 158 | 2.5% | 1.9% | 1.3% | 10.1% |
| Mountain View | 147 | 1.4% | 0.7% | 0.7% | 4.1% |
| Palo Alto | 130 | 2.3% | 0.0% | 0.0% | 5.4% |
| Burbank | 124 | 3.2% | 0.8% | 0.8% | 4.8% |
| Long Beach | 118 | 0.8% | 0.8% | 0.8% | 6.8% |
| Newport Beach | 117 | 4.3% | 0.0% | 0.0% | 4.3% |
| Orange | 116 | 6.9% | 3.4% | 2.6% | 9.5% |
| Lake Forest | 115 | 5.2% | 0.9% | 0.0% | 7.0% |
| Redwood City | 113 | 1.8% | 0.9% | 0.9% | 10.6% |
| San Luis Obispo | 110 | 3.6% | 1.8% | 1.8% | 6.4% |
| San Gabriel | 107 | 0.9% | 0.0% | 0.0% | 11.2% |
| Campbell | 107 | 4.7% | 0.9% | 0.9% | 11.2% |
| Milpitas | 101 | 3.0% | 3.0% | 3.0% | 13.9% |
| Alameda | 100 | 4.0% | 1.0% | 1.0% | 10.0% |
| El Cajon | 99 | 6.1% | 2.0% | 2.0% | 12.1% |
| Camarillo | 98 | 3.1% | 3.1% | 2.0% | 14.3% |
| Alhambra | 96 | 1.0% | 1.0% | 0.0% | 5.2% |
| Modesto | 95 | 3.2% | 4.2% | 2.1% | 7.4% |
| Turlock | 95 | 2.1% | 5.3% | 5.3% | 13.7% |
| Beverly Hills | 90 | 2.2% | 1.1% | 1.1% | 7.8% |
| Irvine | 88 | 3.4% | 2.3% | 2.3% | 9.1% |
| San Mateo | 86 | 2.3% | 1.2% | 1.2% | 5.8% |
| Palm Springs | 86 | 2.3% | 0.0% | 0.0% | 4.7% |
| Arcadia | 79 | 0.0% | 5.1% | 5.1% | 5.1% |
| Sebastopol | 76 | 3.9% | 1.3% | 1.3% | 11.8% |
| Santa Monica | 75 | 5.3% | 1.3% | 0.0% | 10.7% |
| Redding | 75 | 2.7% | 4.0% | 4.0% | 13.3% |
| Glendale | 74 | 4.1% | 0.0% | 0.0% | 8.1% |
| Anaheim | 74 | 4.1% | 2.7% | 2.7% | 10.8% |
| Riverside | 73 | 4.1% | 0.0% | 0.0% | 9.6% |
| Gilroy | 70 | 2.9% | 2.9% | 1.4% | 15.7% |
| Ventura | 69 | 5.8% | 2.9% | 1.4% | 11.6% |
| Chico | 68 | 4.4% | 0.0% | 0.0% | 13.2% |
| Cupertino | 66 | 4.5% | 0.0% | 0.0% | 10.6% |
| Santa Rosa | 65 | 7.7% | 1.5% | 0.0% | 15.4% |
| Fremont | 62 | 3.2% | 1.6% | 1.6% | 9.7% |
| Rancho Cucamonga | 61 | 4.9% | 0.0% | 0.0% | 8.2% |
| Los Gatos | 60 | 3.3% | 0.0% | 0.0% | 10.0% |
| Los Altos | 59 | 1.7% | 0.0% | 0.0% | 6.8% |
| Temecula | 59 | 3.4% | 1.7% | 1.7% | 5.1% |
| Elk Grove | 59 | 5.1% | 3.4% | 1.7% | 16.9% |
| Poway | 58 | 3.4% | 1.7% | 1.7% | 6.9% |
| Carlsbad | 56 | 7.1% | 0.0% | 0.0% | 14.3% |
| Chula Vista | 55 | 3.6% | 1.8% | 1.8% | 9.1% |
| Escondido | 55 | 10.9% | 3.6% | 1.8% | 21.8% |
| Concord | 55 | 7.3% | 0.0% | 0.0% | 10.9% |
| Livermore | 54 | 3.7% | 0.0% | 0.0% | 9.3% |
| Bakersfield | 54 | 1.9% | 1.9% | 1.9% | 14.8% |
| La Jolla | 53 | 0.0% | 0.0% | 0.0% | 5.7% |
| Santa Ana | 53 | 3.8% | 1.9% | 1.9% | 13.2% |
| Corona | 53 | 7.5% | 3.8% | 3.8% | 11.3% |
| La Mesa | 51 | 2.0% | 0.0% | 0.0% | 5.9% |
| Fontana | 50 | 10.0% | 2.0% | 0.0% | 20.0% |
| Thousand Oaks | 50 | 0.0% | 4.0% | 4.0% | 6.0% |
| San Clemente | 50 | 6.0% | 2.0% | 2.0% | 4.0% |
| San Rafael | 49 | 0.0% | 2.0% | 2.0% | 4.1% |
| South Pasadena | 48 | 0.0% | 0.0% | 0.0% | 8.3% |
| Morgan Hill | 47 | 0.0% | 0.0% | 0.0% | 12.8% |
| Torrance | 47 | 4.3% | 0.0% | 0.0% | 12.8% |
| Redlands | 46 | 2.2% | 4.3% | 4.3% | 13.0% |
| Napa | 46 | 0.0% | 0.0% | 0.0% | 6.5% |
| Salinas | 45 | 0.0% | 2.2% | 2.2% | 11.1% |
| San Marino | 45 | 2.2% | 2.2% | 2.2% | 11.1% |
| Walnut Creek | 44 | 0.0% | 0.0% | 0.0% | 11.4% |
| Shafter | 44 | 9.1% | 0.0% | 0.0% | 15.9% |
| Danville | 44 | 2.3% | 0.0% | 0.0% | 11.4% |
| Albany | 43 | 0.0% | 0.0% | 0.0% | 4.7% |
| Visalia | 42 | 0.0% | 0.0% | 0.0% | 9.5% |
| Costa Mesa | 42 | 2.4% | 0.0% | 0.0% | 7.1% |
| Lemon Grove | 42 | 4.8% | 0.0% | 0.0% | 4.8% |
| Paso Robles | 42 | 2.4% | 2.4% | 2.4% | 9.5% |
| Sherman Oaks | 41 | 4.9% | 0.0% | 0.0% | 9.8% |
| Isla Vista | 41 | 2.4% | 2.4% | 2.4% | 14.6% |
| Roseville | 40 | 0.0% | 0.0% | 0.0% | 2.5% |
| San Bernardino | 40 | 0.0% | 0.0% | 0.0% | 12.5% |
| Eureka | 39 | 5.1% | 0.0% | 0.0% | 7.7% |
| Monterey Park | 39 | 5.1% | 7.7% | 5.1% | 10.3% |
| Sonoma | 39 | 2.6% | 2.6% | 2.6% | 5.1% |
| Mission Viejo | 39 | 5.1% | 0.0% | 0.0% | 5.1% |
| Santa Paula | 39 | 2.6% | 0.0% | 0.0% | 7.7% |
| Menlo Park | 38 | 5.3% | 2.6% | 0.0% | 7.9% |
| Monterey | 38 | 2.6% | 0.0% | 0.0% | 15.8% |
| Pacifica | 38 | 2.6% | 0.0% | 0.0% | 10.5% |
| Burlingame | 37 | 0.0% | 0.0% | 0.0% | 8.1% |
| South Lake Tahoe | 37 | 10.8% | 0.0% | 0.0% | 16.2% |
| Dublin | 37 | 8.1% | 0.0% | 0.0% | 8.1% |
| Nevada City | 36 | 0.0% | 0.0% | 0.0% | 11.1% |
| Clovis | 36 | 2.8% | 0.0% | 0.0% | 5.6% |
| Downey | 36 | 5.6% | 0.0% | 0.0% | 11.1% |
| Fullerton | 35 | 2.9% | 0.0% | 0.0% | 8.6% |
| Montecito | 35 | 11.4% | 0.0% | 0.0% | 5.7% |
| Laguna Niguel | 34 | 0.0% | 0.0% | 0.0% | 0.0% |
| Arcata | 33 | 0.0% | 0.0% | 0.0% | 6.1% |
| Victorville | 33 | 6.1% | 0.0% | 0.0% | 18.2% |
| Huntington Beach | 33 | 9.1% | 0.0% | 0.0% | 12.1% |
| Woodland Hills | 33 | 3.0% | 0.0% | 0.0% | 6.1% |
| San Marcos | 32 | 12.5% | 0.0% | 0.0% | 6.3% |
| Folsom | 32 | 3.1% | 0.0% | 0.0% | 3.1% |
| Tustin | 32 | 3.1% | 6.3% | 6.3% | 3.1% |
| San Pedro | 31 | 0.0% | 0.0% | 0.0% | 12.9% |
| Vacaville | 31 | 0.0% | 0.0% | 0.0% | 6.5% |
| Oxnard | 30 | 6.7% | 0.0% | 0.0% | 16.7% |
| Del Mar | 30 | 6.7% | 0.0% | 0.0% | 13.3% |
| Encinitas | 30 | 3.3% | 3.3% | 3.3% | 13.3% |
| Monrovia | 30 | 0.0% | 0.0% | 0.0% | 6.7% |
| Martinez | 30 | 3.3% | 0.0% | 0.0% | 13.3% |
| Northridge | 30 | 0.0% | 0.0% | 0.0% | 6.7% |
| Vista | 29 | 10.3% | 6.9% | 3.4% | 6.9% |
| Venice | 29 | 3.4% | 0.0% | 0.0% | 3.4% |
| Pleasant Hill | 29 | 3.4% | 0.0% | 0.0% | 10.3% |
| Pleasanton | 29 | 0.0% | 6.9% | 6.9% | 17.2% |
| Lakeside | 28 | 7.1% | 0.0% | 0.0% | 17.9% |
| Emeryville | 28 | 0.0% | 0.0% | 0.0% | 7.1% |
| North Hollywood | 28 | 0.0% | 0.0% | 0.0% | 3.6% |
| Garden Grove | 27 | 0.0% | 0.0% | 0.0% | 0.0% |
| Temple City | 26 | 0.0% | 0.0% | 0.0% | 7.7% |
| Stockton | 26 | 3.8% | 7.7% | 3.8% | 15.4% |
| Santee | 26 | 0.0% | 0.0% | 0.0% | 3.8% |
| San Leandro | 26 | 0.0% | 3.8% | 3.8% | 7.7% |
| Oceanside | 25 | 12.0% | 0.0% | 0.0% | 16.0% |
| Van Nuys | 25 | 0.0% | 0.0% | 0.0% | 24.0% |
| Hermosa Beach | 25 | 0.0% | 0.0% | 0.0% | 4.0% |
| Pismo Beach | 25 | 4.0% | 4.0% | 4.0% | 8.0% |
How the census was taken
AIOInsights took every business in California that lists a website in OpenStreetMap (map data as of 2026-11-01; contributors credited under the Open Database License), kept one record per website, and checked each site between 2026-11-01 and the publish date. For each site it read /robots.txt with the same rules the free AIOInsights check uses, and loaded the homepage twice: once as a browser and once identifying as GPTBot, OpenAI's crawler. A site counts as reachable when the browser visit loads; only reachable sites are in the percentages.
- Disallows a named AI crawler (the headline): robots.txt names a major AI crawler (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and their kin) and disallows the whole site. This is the site's own published instruction, read directly.
- Blocks AI search: robots.txt shuts out the crawlers AI assistants use to answer live questions (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot), or every crawler.
- Blocks every crawler:
User-agent: *withDisallow: /, which also shuts out Google. - Blocked a request that claimed to be GPTBot: reported on its own, never added to the headline. The homepage request carried GPTBot's user-agent but came from AIOInsights servers, not OpenAI's network. The site answered it with a bot challenge or a 401/403 while letting the browser visit in. Bot walls that verify crawler IP addresses turn that request away and still let the real GPTBot in, so this counts sites that refuse an unverified GPTBot request, not sites that block OpenAI.
This census publishes totals only. It never names a business. Any business can check its own site free, with no email, at aioinsights.com.