Detect flaky tests and find their root causes
A QA-focused post on using Strands Agents with Gemini to detect flaky tests and trace their root causes. ·
Done withGemini
- What they did
- A Python tool parses JUnit XML results and scores each test by the share of commits where it both passed and failed. It writes JSON reports. A Strands Agents agent running on Gemini 2.5 Flash has two tools: one fetches reports for a chosen period, the other exports a Markdown analysis. The agent reads the error messages, infers likely root causes, and writes recommendations.
- How it went
- On generated sample data, 12 of 109 tests were flagged from 19,644 parsed results, and the agent produced a root-cause report. The author says it gives a first look only, and infrastructure failures can look like flakiness.
- Worth knowing
- Tests only get flagged when the same commit has both passing and failing runs. Results also need commit_sha and run_id properties, and the setup assumes GitHub Actions.
Try it yourself with Gemini
Look at the test suite in [path to repo] and CI logs from [CI system or log folder]. Find tests that pass and fail inconsistently, rerun suspicious ones [20] times to confirm they are flaky, and for each one identify the likely root cause such as timing, shared state, ordering, or network. Finish with a ranked report of flaky tests with a suggested fix for each, and don't change any code until I approve.
Want help getting this working? Join the Vibe Coding Academy ↗ Run by Alex Finn, who also runs this site.
Source: DEV · Nov 10, 2026
Related use cases
Orchestrate a multi-phase RAG migration
A walkthrough of using Gemini CLI to coordinate a complex, multi-phase cloud migration project.
Solve open math problems autonomously
DeepMind's Aletheia agent on Gemini Deep Think was evaluated on 700 open Erdős problems and autonomously solved four.
Write and send AI newsletters automatically
An n8n agent finds a trending AI topic, writes a newsletter with Gemini, formats it and emails it to subscribers without manual work.
Build a C compiler with a team of parallel agents
Anthropic's engineering write-up on tasking Opus 4.6 agent teams with building a C compiler.
Monitor tasks and open pull requests with long-running agents
A commenter runs a persistent script that watches tasks and uses Cursor to open pull requests, building a documentation summarizer and a data pipeline.
Turn GitHub issues into draft pull requests with long-running agents
The author used Dots for read-heavy long-horizon tasks like reading GitHub issues and making PRs, which were typically poor but gave a starting point.
Get 5 like this every morning
The best things people got an AI agent to do, each with the prompt to try it.