Play a button-clicking game via a browser agent
A developer wired Gemini's Computer Use model to Playwright in Java and had it play a simple button-clicking game to verify click-coordinate scaling.
Done withGemini
- What they did
- Guillaume Laforge built a Java loop around Gemini's computer-use preview model and Playwright's headless Chromium, with a 1000x1000 viewport. The model got a screenshot after every action and returned click actions with coordinates on a normalized 1000x1000 grid. His code scaled those to the viewport and clicked. He built a simple game where a red button moves after each click, and told the agent to click it 10 times and read the score.
- How it went
- The agent clicked the moving button and reported a final score of 10, which confirmed the click handling and coordinate scaling worked. The source mentions no failures in this test.
- Worth knowing
- Send a fresh screenshot to the model after every action, and wait about a second first so the page has finished rendering. Otherwise the model sees a blank or half-drawn page.
Try it yourself with Gemini
Use browser control to open [URL of a simple web game] and play it by looking at screenshots and clicking, for about [number] moves. Report whether your clicks landed where you intended, and note any mismatch between screenshot coordinates and actual click positions. Stop when you've finished the moves and summarize how it went.
1 more account of this
Source: glaforge.dev · Nov 3, 2025
Five of these in your inbox every morning
The best things people got an AI agent to do, each with the prompt to try it.
More like this
Race three models to build a Counter-Strike clone
InstantDB had Codex, Opus, and Gemini each try to build a multiplayer shooter.
Edit existing video clips with plain-English instructions
A creator tested Gemini Omni Flash over 36 hours, editing clips in place from typed instructions while preserving lighting and camera angle, with variable quality.
Solve open math problems autonomously
DeepMind's Aletheia agent on Gemini Deep Think was evaluated on 700 open Erdős problems and autonomously solved four.
Play RollerCoaster Tycoon
Ramp Labs put Claude Code in RollerCoaster Tycoon.
Play Doom with computer-use agents as an informal benchmark
The OpenClaw team used computer-use agents to play Doom on remote machines, treating game performance as an informal way to improve the computer-use workflow.
Test an iPhone game and report what passed
Cognition had Devin on GPT-6 Astra test the game Otter Run in a simulator, returning a recording and a report of checks passed and untested areas.