Audit reproducibility of your published papers
A researcher used Claude to check the code and data of every journal paper they authored in the last five years and found significant reproducibility issues. ·
Done withClaude
- What they did
- Mike Frank used Claude to audit the computational reproducibility of every journal paper he had co-authored in the last five years. The code and data for these papers were already publicly available, so the audit worked from those open materials. The post doesn't give the mechanics. They are in the linked blog post.
- How it went
- The audit found significant reproducibility issues, even though the papers' code and data were openly available. The post doesn't give counts or describe the specific problems.
- Worth knowing
- The papers' code and data had to be public for this audit to work. The post gives no cost or time figures, so check the linked blog post for those.
Try it yourself with Claude
I have the code and data for my published papers in [folder path or repository links]. For each paper, try to set up the environment, run the analysis, and compare your results to the figures and tables reported in the paper at [path to paper PDFs]. Give me a report listing each paper as reproduced, partially reproduced, or failed, with the specific causes such as missing files, hardcoded paths, or unpinned dependencies. Do not modify my original files, and work in a copy.
Want help getting this working? Join the Vibe Coding Academy ↗ Run by Alex Finn, who also runs this site.
Source: Bluesky · Nov 10, 2026
Related use cases
Build a C compiler with a team of parallel agents
Anthropic's engineering write-up on tasking Opus 4.6 agent teams with building a C compiler.
Produce a finished music release from one chat
Claude Code wrote lyrics, generated and mastered the song, made cover art and a 10-second video via an MCP connector in about 6 minutes; clip joined with ffmpeg outside.
Submitted by Aetherwave Studio
Manage and launch paid ad campaigns via an ads MCP
The poster uses an ads MCP that reports on and manages or launches campaigns, with scheduled routines checking other data sources to make decisions.
Find connections across 100 non-fiction books
Claude Code browses a personal library with CLI tools and surfaces trails of related excerpts across books.
Generate deep research reports into Notion
An n8n workflow turns a topic into up to three Tavily searches, compiles a Markdown report with an agent, and pushes it to a Notion page.
Analyze 170k LLM logs overnight
A user left a Gemini agent running overnight to analyze 170k LLM logs with the llm CLI and a free model, and it finished the job despite limits.
Get 5 like this every morning
The best things people got an AI agent to do, each with the prompt to try it.