Try Devin on research tasks (mostly failed)
Answer.AI had Devin fetch an author's recent papers and research transcript summarization; it got stuck parsing HTML and returned generic results, rated a failure.
Done withDevin
- What they did
- Answer.AI gave Devin, which works through Slack in its own Docker environment with a browser, editor and shell, two tasks. One was grabbing an author's 25 most recent Google Scholar papers with Playwright, skipping paywalls. The other was researching transcript summarization with accurate timestamps. They then rated the results.
- How it went
- Both failed. On the scraping task Devin got stuck parsing HTML and went to sleep. On the research task it returned related but unhelpful material and code that missed the core problem.
- Worth knowing
- Across 20 tasks Devin had 14 failures and 3 successes. One research success, a Discord bot, worked because the goal was loose pseudocode-level guidance.
Try it yourself with Devin
Research [author or topic] by finding their [number] most recent papers and writing a one-paragraph summary of each, with links to the sources. Then give me a short overview of the main themes across them. If a page can't be read, tell me which one and why instead of guessing, and stop once you have a cited summary I can check.
Source: answer.ai · Jan 8, 2025
Five of these in your inbox every morning
The best things people got an AI agent to do, each with the prompt to try it.
More like this
Hand narrow code changes to Devin for pull requests
A team used Devin for a month on small fixes like chart formatting and API edge cases, with tests and PRs; a complex full-stack task went poorly.
Generate internal dashboards and data migration scripts
A backend engineer says their team uses Devin for internal dashboards and migration scripts, treating output as a starting point that saves about four hours of boilerplate.
Build a Pong game and website from scratch
A tester asked Devin to create a basic Pong-style game and a website, and it finished both in under 20 minutes.
Find connections across 100 non-fiction books
Claude Code browses a personal library with CLI tools and surfaces trails of related excerpts across books.
Summarize 20 articles into a cited document
An agent extracted key insights from 20 articles, found common themes and produced a cited summary; the write-up estimates 18 minutes versus hours by hand.
Build a personal research wiki from papers and articles
Cursor Agent writes and maintains interlinked markdown pages from user-supplied sources, summarizing and cross-referencing, with Obsidian used to browse.