Audit cleaning-company sites for AI crawler access
An agent used MCP tools to read robots.txt rules for 14 AI crawlers and check 8 page tags on 18 cleaning-company sites; 9 returned their home page.
Done with a custom-built or unnamed agent
- What they did
- The authors took domains with "cleaning" in the name from the Tranco top-1M list and filtered them down to 18 service firms. For each site they called two Site Check tools. One read robots.txt, headers and redirects for 14 named AI crawlers. The other checked 8 page tags. Site selection and filtering were done by hand from page titles.
- How it went
- 9 of 18 sites returned their home page. Of those 9, one blocks 12 of the 14 crawlers, 7 lack a share image, and only 2 have all 8 tags. The other 9 sites gave a 403, a CDN error or no answer.
- Worth knowing
- A refused or failed fetch doesn't show what AI bots receive, and firewall or CDN rules can block bots that robots.txt allows. Treat 9 sites as a snapshot, not a survey.
Try it yourself with Claude Code
Here is a list of [number] company websites: [list of domains]. For each one, fetch robots.txt and report whether it allows or blocks major AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Also check the homepage for meta tags that affect AI or search access, and flag any site that returns its homepage instead of a real robots.txt. Finish with a single table of results and a short list of the biggest gaps.
Want help getting this working? Join the Vibe Coding Academy ↗ Run by Alex Finn, who also runs this site.
Source: tools.openkrill.app · Oct 11, 2026 · Submitted by OpenKrill
Related use cases
Sell services through an agent that takes payment and delivers
An agent runs a transaction flow, creates Stripe checkout links, verifies payment before delivering a service, and asks the owner on Telegram to approve refunds or disputes.
Custom agent
Find and score leads from social post engagers
A Cowork agent scrapes LinkedIn post engagers via Apify, enriches via Apollo, scores them against an ideal customer, and drafts outreach for the best fits.
Investigate customer errors and draft tickets in support
Cursor's support team uses Cursor with MCP to pull Datadog logs, search past cases, and draft a Linear bug ticket and customer reply.
Automate event social posting across platforms
Claude Code connected social channels to drive event registrations, working well on Twitter and Reddit and somewhat on LinkedIn.
Find stale commercial listings and email brokers
A Hermes bot scrapes lease listings, filters by days on market and zoning, calculates commissions and emails brokers.
Handle Shopify support tickets, orders and refunds
An AI agent built on Claude and GPT cancels orders, tracks shipments, creates return labels and processes refunds for Shopify merchants' support tickets (vendor listing).
Get 5 like this every morning
The best things people got an AI agent to do, each with the prompt to try it.