Find and fix flaky tests in a broken CI pipeline overnight
Claude monitors CI pass rates, identifies worst flaky tests, reproduces them and proposes fixes overnight; pipeline got 20 minutes faster and pass rate rose 35%, with some bad fixes discarded.
Done withClaude Code
- What they did
- The author gave Claude read access to CI artifacts and had it build tooling to track the rolling 30-day pass/fail rate. It picked the worst flaky tests, hypothesized test versus production causes, and narrowed to minimal repros. Without a deterministic repro, it guessed a fix and re-ran tests overnight to judge it statistically. The author reviewed PRs by day.
- How it went
- Over a handful of weeks the pipeline got 20 minutes shorter and the pass rate rose 35%, from roughly 40% on master. Some of Claude's proposed solutions were bad and had to be thrown away.
- Worth knowing
- Plan to review every PR yourself; the author discarded some bad fixes, so overnight runs need daytime human checking.
Try it yourself with Claude Code
Look at the last [number] CI runs for [repo name] and rank the tests by how often they flake. For the worst [number], reproduce the failure locally, find the cause, and propose a fix on a separate branch. Run each fix [number] times to confirm it's stable, discard any fix that doesn't hold, and summarize what worked. Don't merge anything without showing me the diff first.
Source: HN · May 1, 2026
Five of these in your inbox every morning
The best things people got an AI agent to do, each with the prompt to try it.
More like this
Turn tickets into merged pull requests
Pulls tasks from GitHub Issues or Linear, then runs Claude Code or Codex in isolated Kubernetes pods to carry each one through to a PR.
Manage an inbox and build courses and budgets with Claude Code
A non-technical user filters their inbox to emails needing replies, turned 15 homeschool PDFs into interactive narrated courses, and built a budget dashboard.
Build a 2D platformer game without coding
A content creator with no coding experience used Claude Code to build a black-and-white wave-combat platformer in HTML canvas and played it live.
- Claude
Build a C compiler with a team of parallel agents
Anthropic's engineering write-up on tasking Opus 4.6 agent teams with building a C compiler.
Build and test a simple endpoint project with Devin
A reviewer had Devin set up a project, write code, install dependencies, test endpoints and add error handling in about 4 minutes on a simple task.
Build a booking and payment app for tattoo artists
A former programmer used Replit Agent over a weekend to build a booking app with calendar, Google OAuth, Stripe deposits and notifications, though time zones and UI needed back-and-forth.