AIOInsights research · 2026-11-01

The AI Blocked Census: how many California business websites shut out AI.

When a customer asks ChatGPT, Claude or Perplexity for a recommendation, the assistant can only use what it is allowed to read. AIOInsights checked 34,519 California business websites listed in OpenStreetMap and found that 3.3% of the 24,908 reachable sites (816) disallow at least one named major AI crawler in robots.txt, and 1.1% disallow every crawler.

24,908California business websites checked and reachable
3.3%disallow at least one named major AI crawler in robots.txt (816 sites)
1.1%disallow every crawler in robots.txt (263 sites)
1.4%block the crawlers AI assistants use to answer live questions

Measured separately, by a different test: 8.7% (2,177 sites) blocked a request that claimed to be GPTBot. That request came from AIOInsights servers, not OpenAI's network, so a wall that checks crawler IP addresses can block it while letting the real GPTBot in. It is not added to the figures above.

Is your website on the wrong side of this? Run the free AIOInsights check: about 20 seconds, no account, and it says in plain words whether AI crawlers can read your site.

Check my website

By type of business

Type of businessWebsites reachableDisallow a named AI crawlerBlock AI searchBlock every crawlerBlocked a request that claimed to be GPTBot
Financial offices478.5%4.3%4.3%12.8%
Real estate offices2568.2%1.6%0.0%10.9%
Financial advisors434.7%7.0%7.0%7.0%
Gyms and fitness7294.3%1.0%0.8%10.7%
Other offices2,5484.1%2.3%1.3%10.8%
Other trades6234.0%1.0%0.6%9.5%
Auto repair and dealers8633.7%1.3%1.0%15.9%
Shops and retail7,3733.2%1.2%0.8%8.4%
Hotels and lodging6033.2%0.2%0.2%7.3%
Doctors and clinics1,3393.1%0.7%0.4%11.4%
Restaurants, cafes and bars8,1303.1%1.0%0.9%6.7%
Law offices3912.8%0.3%0.3%7.9%
Contractors and trades4532.4%0.7%0.4%6.0%
Dentists6302.2%6.2%5.7%12.9%
Other businesses3732.1%1.1%0.8%10.7%
Insurance agencies1721.7%2.3%0.6%4.1%
Veterinarians2240.4%4.5%4.0%12.5%
Accountants600.0%16.7%16.7%21.7%
Tax advisors500.0%10.0%10.0%16.0%

By city

Cities with at least 25 reachable business websites in the census.

CityWebsites reachableDisallow a named AI crawlerBlock AI searchBlock every crawlerBlocked a request that claimed to be GPTBot
San Diego1,2343.2%0.9%0.5%6.8%
San Francisco1,2192.5%0.7%0.6%6.6%
Los Angeles1,1543.1%1.0%0.6%7.2%
San Jose9733.5%2.4%2.0%10.3%
Fresno7634.2%2.1%1.7%10.4%
Santa Barbara6832.8%0.4%0.3%6.4%
Berkeley4463.1%0.9%0.0%8.1%
Pasadena4102.0%2.4%2.4%8.0%
Oakland2763.6%1.8%1.4%7.2%
Goleta2214.5%0.9%0.5%10.0%
Sacramento2112.4%0.5%0.0%9.5%
Simi Valley2002.0%2.0%2.0%7.0%
Sunnyvale1833.8%2.7%2.2%12.0%
Santa Cruz1764.0%1.1%0.6%8.0%
West Hollywood1634.9%1.8%1.2%7.4%
Santa Clara1582.5%1.9%1.3%10.1%
Mountain View1471.4%0.7%0.7%4.1%
Palo Alto1302.3%0.0%0.0%5.4%
Burbank1243.2%0.8%0.8%4.8%
Long Beach1180.8%0.8%0.8%6.8%
Newport Beach1174.3%0.0%0.0%4.3%
Orange1166.9%3.4%2.6%9.5%
Lake Forest1155.2%0.9%0.0%7.0%
Redwood City1131.8%0.9%0.9%10.6%
San Luis Obispo1103.6%1.8%1.8%6.4%
San Gabriel1070.9%0.0%0.0%11.2%
Campbell1074.7%0.9%0.9%11.2%
Milpitas1013.0%3.0%3.0%13.9%
Alameda1004.0%1.0%1.0%10.0%
El Cajon996.1%2.0%2.0%12.1%
Camarillo983.1%3.1%2.0%14.3%
Alhambra961.0%1.0%0.0%5.2%
Modesto953.2%4.2%2.1%7.4%
Turlock952.1%5.3%5.3%13.7%
Beverly Hills902.2%1.1%1.1%7.8%
Irvine883.4%2.3%2.3%9.1%
San Mateo862.3%1.2%1.2%5.8%
Palm Springs862.3%0.0%0.0%4.7%
Arcadia790.0%5.1%5.1%5.1%
Sebastopol763.9%1.3%1.3%11.8%
Santa Monica755.3%1.3%0.0%10.7%
Redding752.7%4.0%4.0%13.3%
Glendale744.1%0.0%0.0%8.1%
Anaheim744.1%2.7%2.7%10.8%
Riverside734.1%0.0%0.0%9.6%
Gilroy702.9%2.9%1.4%15.7%
Ventura695.8%2.9%1.4%11.6%
Chico684.4%0.0%0.0%13.2%
Cupertino664.5%0.0%0.0%10.6%
Santa Rosa657.7%1.5%0.0%15.4%
Fremont623.2%1.6%1.6%9.7%
Rancho Cucamonga614.9%0.0%0.0%8.2%
Los Gatos603.3%0.0%0.0%10.0%
Los Altos591.7%0.0%0.0%6.8%
Temecula593.4%1.7%1.7%5.1%
Elk Grove595.1%3.4%1.7%16.9%
Poway583.4%1.7%1.7%6.9%
Carlsbad567.1%0.0%0.0%14.3%
Chula Vista553.6%1.8%1.8%9.1%
Escondido5510.9%3.6%1.8%21.8%
Concord557.3%0.0%0.0%10.9%
Livermore543.7%0.0%0.0%9.3%
Bakersfield541.9%1.9%1.9%14.8%
La Jolla530.0%0.0%0.0%5.7%
Santa Ana533.8%1.9%1.9%13.2%
Corona537.5%3.8%3.8%11.3%
La Mesa512.0%0.0%0.0%5.9%
Fontana5010.0%2.0%0.0%20.0%
Thousand Oaks500.0%4.0%4.0%6.0%
San Clemente506.0%2.0%2.0%4.0%
San Rafael490.0%2.0%2.0%4.1%
South Pasadena480.0%0.0%0.0%8.3%
Morgan Hill470.0%0.0%0.0%12.8%
Torrance474.3%0.0%0.0%12.8%
Redlands462.2%4.3%4.3%13.0%
Napa460.0%0.0%0.0%6.5%
Salinas450.0%2.2%2.2%11.1%
San Marino452.2%2.2%2.2%11.1%
Walnut Creek440.0%0.0%0.0%11.4%
Shafter449.1%0.0%0.0%15.9%
Danville442.3%0.0%0.0%11.4%
Albany430.0%0.0%0.0%4.7%
Visalia420.0%0.0%0.0%9.5%
Costa Mesa422.4%0.0%0.0%7.1%
Lemon Grove424.8%0.0%0.0%4.8%
Paso Robles422.4%2.4%2.4%9.5%
Sherman Oaks414.9%0.0%0.0%9.8%
Isla Vista412.4%2.4%2.4%14.6%
Roseville400.0%0.0%0.0%2.5%
San Bernardino400.0%0.0%0.0%12.5%
Eureka395.1%0.0%0.0%7.7%
Monterey Park395.1%7.7%5.1%10.3%
Sonoma392.6%2.6%2.6%5.1%
Mission Viejo395.1%0.0%0.0%5.1%
Santa Paula392.6%0.0%0.0%7.7%
Menlo Park385.3%2.6%0.0%7.9%
Monterey382.6%0.0%0.0%15.8%
Pacifica382.6%0.0%0.0%10.5%
Burlingame370.0%0.0%0.0%8.1%
South Lake Tahoe3710.8%0.0%0.0%16.2%
Dublin378.1%0.0%0.0%8.1%
Nevada City360.0%0.0%0.0%11.1%
Clovis362.8%0.0%0.0%5.6%
Downey365.6%0.0%0.0%11.1%
Fullerton352.9%0.0%0.0%8.6%
Montecito3511.4%0.0%0.0%5.7%
Laguna Niguel340.0%0.0%0.0%0.0%
Arcata330.0%0.0%0.0%6.1%
Victorville336.1%0.0%0.0%18.2%
Huntington Beach339.1%0.0%0.0%12.1%
Woodland Hills333.0%0.0%0.0%6.1%
San Marcos3212.5%0.0%0.0%6.3%
Folsom323.1%0.0%0.0%3.1%
Tustin323.1%6.3%6.3%3.1%
San Pedro310.0%0.0%0.0%12.9%
Vacaville310.0%0.0%0.0%6.5%
Oxnard306.7%0.0%0.0%16.7%
Del Mar306.7%0.0%0.0%13.3%
Encinitas303.3%3.3%3.3%13.3%
Monrovia300.0%0.0%0.0%6.7%
Martinez303.3%0.0%0.0%13.3%
Northridge300.0%0.0%0.0%6.7%
Vista2910.3%6.9%3.4%6.9%
Venice293.4%0.0%0.0%3.4%
Pleasant Hill293.4%0.0%0.0%10.3%
Pleasanton290.0%6.9%6.9%17.2%
Lakeside287.1%0.0%0.0%17.9%
Emeryville280.0%0.0%0.0%7.1%
North Hollywood280.0%0.0%0.0%3.6%
Garden Grove270.0%0.0%0.0%0.0%
Temple City260.0%0.0%0.0%7.7%
Stockton263.8%7.7%3.8%15.4%
Santee260.0%0.0%0.0%3.8%
San Leandro260.0%3.8%3.8%7.7%
Oceanside2512.0%0.0%0.0%16.0%
Van Nuys250.0%0.0%0.0%24.0%
Hermosa Beach250.0%0.0%0.0%4.0%
Pismo Beach254.0%4.0%4.0%8.0%

How the census was taken

AIOInsights took every business in California that lists a website in OpenStreetMap (map data as of 2026-11-01; contributors credited under the Open Database License), kept one record per website, and checked each site between 2026-11-01 and the publish date. For each site it read /robots.txt with the same rules the free AIOInsights check uses, and loaded the homepage twice: once as a browser and once identifying as GPTBot, OpenAI's crawler. A site counts as reachable when the browser visit loads; only reachable sites are in the percentages.

This census publishes totals only. It never names a business. Any business can check its own site free, with no email, at aioinsights.com.