agentusecasesAll 965 use cases
Games & Simulations

Play Pokémon Blue with an agent

The Gemini Plays Pokémon agent reads game RAM and screenshots, turns actions into button presses, and uses pathfinder and boulder-puzzle sub-agents.

Done withGemini

What they did
An independent developer, Joel Zhang, built a harness that gives Gemini 2.5 Pro a subset of Game Boy RAM as text, overlaid on a screenshot. Gemini picks actions that become button presses. The harness summarizes history every 100 turns and runs a critic instance every 25 turns. It also provides pathfinder and boulder-puzzle sub-agents, which are prompted Gemini instances.
How it went
The agent beat Pokémon Blue in 813 hours on run 1, then in 406.5 hours on a fully autonomous second run. It looped on long contexts, hallucinated items from other games, and handled raw pixels poorly.
Worth knowing
The model struggled to read raw screen pixels, so the RAM-to-text translation was needed. Behavior also degraded once context grew well past 100k tokens.

Try it yourself with Gemini

Help me build an agent that plays [game, e.g. Pokémon Blue] in an emulator at [emulator path or setup]. It should read the game state from [RAM or screenshots], decide on an action, and send it as button presses, with a simple helper for [subtask, e.g. pathfinding]. You're done when it can play for [number] steps on its own and log what it decided at each step.

Read the original ↗

Source: arxiv.org · Undated

Five of these in your inbox every morning

The best things people got an AI agent to do, each with the prompt to try it.

More like this