Test Devin on integrations, migration and security review
Answer.AI tried Devin on an observability integration, an nbdev migration and a vulnerability review; it found some real issues, hallucinated others, and failed the migration.
- What they did
- Answer.AI gave Devin, which works through Slack in its own Docker environment with a browser, editor and shell, about 20 real tasks. These included generating and uploading synthetic data to Braintrust, migrating a Python project to nbdev, and a security review of a repo under 700 lines. People checked the output and often took over.
- How it went
- The Braintrust integration was abandoned and redone in Cursor. The nbdev migration failed on basic setup. The security review found some real issues but also false positives and hallucinated ones. Overall 14 failures, 3 successes and 3 inconclusive.
- Worth knowing
- Devin pressed on with impossible tasks for days. The authors suggest a single focused LLM call for something like a small-repo security review.
Try it yourself with Devin
Take on three tasks in my repo [repo name]: add an integration with [observability tool], migrate [a module or tool] to [target tool or version], and review the code for security vulnerabilities. For each one, tell me how you verified your results, and clearly mark any finding you could not confirm. Open separate pull requests for the work and do not merge anything.
Source: answer.ai · Jan 8, 2025
Five of these in your inbox every morning
The best things people got an AI agent to do, each with the prompt to try it.
More like this
Keep coding agents running autonomously for long stretches
Cursor's write-up on scaling long-running autonomous coding.
Rewrite a README after a rearchitecture
Devin scanned a rearchitected codebase and delivered a new README as a PR with accurate setup and API details, but struggled with design rationale and purpose.
Grade Cursor on a real data science project
A data scientist tested Cursor on a real project: weak on pandas, data cleansing and scraping, strong at prototyping.
- Claude
Build a C compiler with a team of parallel agents
Anthropic's engineering write-up on tasking Opus 4.6 agent teams with building a C compiler.
- ClaudeJules
Pair-program with LLMs on personal projects
The author uses Claude web and Jules (Gemini with GitHub integration) as pair-programming partners, navigating while coding locally and submitting back.
Lint, fix, commit and push code from the terminal
Scout runs a linter, fixes a syntax error, re-runs it, reviews staged changes, writes a commit message and pushes to a new branch, shown as a walkthrough.