Crawl documentation into clean Markdown for AI agents
The author built a documentation crawler for their AI agents that pulled 60 pages of clean Markdown in 49 seconds.
Done with a custom-built or unnamed agent
- What they did
- The author built pulpie-mcp, an MCP server with four tools: fetch, save, crawl and list. It runs a local extraction model behind a backend that starts automatically. Claude was asked to crawl a 60-page docs site. It found pages through robots.txt and sitemaps, used four concurrent workers, and wrote Markdown files with frontmatter to disk instead of returning them to the context window.
- How it went
- Total time was about 49 seconds: roughly 10 for backend cold start and 37 for crawling and saving 60 pages. Untested on client-rendered pages, which may come back empty. PDFs aren't supported.
- Worth knowing
- The model is licensed CC BY-NC 4.0 (non-commercial), and it has only been tested on Linux with an NVIDIA GPU, using about 420 MB of VRAM.
Try it yourself with Claude Code
Build a crawler that takes the documentation site [docs URL], follows its internal links, and saves each page as clean Markdown (no nav or footer) in a folder called docs-md, one file per page. Respect robots.txt and rate limit politely. Done when you have crawled the whole site and given me a count of pages saved and a list of any that failed.
Source: DEV · Oct 5, 2026
Five of these in your inbox every morning
The best things people got an AI agent to do, each with the prompt to try it.
More like this
- Claude
Build a C compiler with a team of parallel agents
Anthropic's engineering write-up on tasking Opus 4.6 agent teams with building a C compiler.
- Custom agent
Add agentic forgetting to an agent's memory
Someone implemented a rudimentary version of Tim Kellogg's agentic forgetting approach and reports it works impressively even in its simplest form.
Configure a local AI lab and build a knowledge graph by chat
Users had Hermes Agent set up a local AI stack (liteLLM, Postgres, Prometheus, Grafana) and install Gbrain as an MCP server to build a personal knowledge graph.
Migrate a legacy product stack with Claude Code
An engineer ran a proof-of-concept that consolidated 4 repos into 2, moved MySQL to PostgreSQL, upgraded frameworks and replaced the auth model, in about 15 work days.
Build a Pong game and website from scratch
A tester asked Devin to create a basic Pong-style game and a website, and it finished both in under 20 minutes.
Deliver a software feature autonomously (partial)
Claude computer use produced 70 files for a planned feature but the code didn't work, so the author went back to manual work.