Run bulk web-extraction jobs with cheap batch-API agents
A developer uses coding agents for daily work and sets up cheap agents-SDK jobs on batch APIs for large runs like visiting 5000 websites to extract data, plus recurring cloud agent jobs. ·
Done withClaude CodeCodexMeta Muse
- What they did
- For large extraction runs, such as visiting 5000 websites and pulling something from each, the developer builds a cheap job with an agents SDK instead of using a main coding agent. The model is Muse, Deepseek, or GPT-6-luna, called through its batch API. Recurring work goes to cloud agents. Dotfiles and agent context live in a GitHub repo so harnesses and providers are easy to switch.
- How it went
- The source gives no results or numbers for the bulk runs themselves. It only describes the setup and why it was chosen: cost. Nothing is said about accuracy or failures.
- Worth knowing
- Batch APIs on cheaper models keep large runs inexpensive, but the comment gives no actual costs or turnaround times.
Try it yourself with Claude Code
I have a list of [number] websites in [file name or path]. Write a script that visits each one and extracts [the data fields you want, e.g. pricing, contact email, product list] into a single CSV, running it as a batch job to keep costs low. Test it on 10 sites first and show me the results and an estimated cost before starting the full run; you are done when the CSV has one row per site with failures logged separately.
Want help getting this working? Join the Vibe Coding Academy ↗ Run by Alex Finn, who also runs this site.
Source: HN · Nov 4, 2026
Related use cases
Turn tickets into merged pull requests
Pulls tasks from GitHub Issues or Linear, then runs Claude Code or Codex in isolated Kubernetes pods to carry each one through to a PR.
Build retro games with a child using Claude Code
An author builds games like a space shooter and an animal hospital game with his 7-year-old, using pixel art assets and a spec drafted via Claude's question tool.
Extract receipt data for freelancer bookkeeping
An app built with Claude Code extracts vendor, date, total, tax and line items from receipts, including foreign-currency ones at historical exchange rates, for freelancers and small businesses.
Run persistent financial research with an agent
LangAlpha is a Claude Code-style agent for investing research that keeps persistent workspaces and memory so analysis such as models and comps can be updated across sessions.
Custom agent
Turn any URL into a structured Notion note
An n8n workflow fetches a submitted URL, strips it to readable content, has OpenAI summarise it into structured fields, and saves a Notion note.
Survey AI product names with a swarm of search agents
The author had a swarm of Claude agents run 4,654 searches to catalog AI bot names and analyze naming trends by gender, industry and year.
Get 5 like this every morning
The best things people got an AI agent to do, each with the prompt to try it.