Fill out forms from images with a form-filling agent
The author used Gemini to build an agent that plans the fields in a form image and matches them to bounding boxes, for a friend who wanted forms filled automatically. ·
Done withGemini
- What they did
- The author said a friend wanted JPEG/PNG forms filled automatically, so they reused prior work with OmniParser and built an annotated dataset from IRS and OPM forms with clearly defined fields. They used Gemini to identify what fields were present in a document and match those fields to bounding boxes, noting GPT-4o plus YOLO could have done the same job.
- How it went
- The post presents it as a working approach for automating tedious form filling, especially in manual supply-chain workflows. It does not give accuracy numbers, speed, or failure cases in the thread text shown.
- Worth knowing
- The main catch is that they first compiled their own annotated dataset from real forms, rather than relying on prompting alone.
Try it yourself with Gemini
Build me a small tool that takes an image of a form at [file path], detects each blank field and its label, and draws bounding boxes where answers should go. Then fill the fields using the details in [my info file], and save the completed form as a new image at [output path]. Show me the result for review before treating it as done, and stop once the filled form lines up with every field.
Want help getting this working? Join the Vibe Coding Academy ↗ Run by Alex Finn, who also runs this site.
Source: Reddit · Apr 1, 2025
Related use cases
Orchestrate a multi-phase RAG migration
A walkthrough of using Gemini CLI to coordinate a complex, multi-phase cloud migration project.
Build a match-3 game from a single prompt
An author prompted Gemini 3 Pro once to produce a working Candy Crush-style game with a shareable link, and says a non-technical friend did too.
Use Grok as the narrative engine of a game
The poster built a game where Grok generates the story narration, and says it outperformed Gemini and GPT for narrative output.
Get a second opinion on an MRI
One person's account of using Claude Code to analyze their own MRI.
Manage calendar and tasks by text with an assistant
A review of Lindy as an iMessage assistant that reschedules meetings and schedules from screenshots, with reported failures like hidden invites and unsent emails.
Maintain a home media library and handle personal projects
Commenters say Grok Bot manages a Plex library with backups, deduplication, thumbnails, and collections, and helped build a Tesla HUD display after finding old purchase emails.
Get 5 like this every morning
The best things people got an AI agent to do, each with the prompt to try it.