A new way to use the web – Google’s Gemini 2.5 Computer Use model can navigate websites by interpreting screenshots and responding with function calls.
Built‑in safety – Every action passes through a safety layer to prevent prompt‑injection, unauthorized purchases or risky clicks.
Early praise – Testers like Poke.com and Autotab report faster execution and smarter context handling, making the agent 50 % quicker than rivals.
Introduction – reaction from early testers
“Watching this thing use my computer felt like magic,” said a developer from a startup that tested Google’s new Gemini 2.5 Computer Use model. “It scrolled through a form, filled out shipping info and even clicked the right drop‑down menu.” The clip quickly went viral on Hacker News, where the top comment read, “Is this the end of Zapier?”
Google’s new model, announced in October 2025, aims to do something few AI systems have attempted: use software the way humans do. Instead of returning text to the user and expecting them to copy and paste, the model sees a screenshot of a website, interprets the user’s intent, and decides what to click, type or scroll next.
How it works
The Gemini 2.5 Computer Use model runs on top of the Gemini 2.5 Pro foundation. It takes three inputs: the user’s request, the URL and a screenshot of the current page. The model processes these and returns a function call describing an action such as click(x,y), type(text) or scroll(amount). Once the action executes, a new screenshot is captured and the loop begins again.
This interactive loop allows the agent to complete multi‑step tasks: fill out forms, sign into accounts, check order statuses or compose emails. Google stresses that the system is optimized for web UIs and can adapt to simple mobile screens, though it does not yet control desktop operating systems.
Safety first
Letting an AI click around your browser raises obvious concerns. Google says that a per‑step safety service evaluates every proposed action and blocks anything that might cause harm. The agent’s system prompt also includes guardrails that restrict purchases over a certain threshold, limit access to sensitive websites (like financial or medical accounts) and prevent the AI from inputting personal data without explicit permission.
Google warns users about prompt injection – malicious web content that tries to trick the model into misbehaving. Safety checks look for suspicious patterns and stop the agent from copying untrusted data into text boxes or running arbitrary scripts. If the model encounters a high‑risk situation, it will ask the user for confirmation or hand control back entirely.
Early testers report speed and context gains
During closed trials, partners like Poke.com and Autotab praised the model’s efficiency. Poke, a social media scheduling tool, reported that Gemini 2.5 Computer Use enabled their agent to complete tasks 50 % faster than competitors. Autotab’s founder said the model “improved context parsing by 18 %,” meaning it made fewer mistakes when filling out complex forms. Google’s own payments team used the model to run UI tests and found it successfully rehabilitated over 60 % of failing tests.
These results suggest that controlling a UI through screenshots – rather than by directly calling API endpoints – can sometimes be more robust. Real web pages contain unexpected pop‑ups, cookie notices and dark‑mode glitches. By seeing exactly what the user sees, the model can adapt.
Why it matters
The release arrives amid growing competition for agentic AI – systems that can act on behalf of users. OpenAI, Anthropic and start‑ups like Adept have all shown prototypes that order groceries, configure spreadsheets or run shell commands. Google’s entry comes months after some of these competitors but claims better safety and performance.
If AI models can reliably use existing UIs, companies might not need to expose APIs for every feature. An agent could book a flight even if the airline has no developer portal. For users, it could mean delegating repetitive tasks – scheduling deliveries, submitting expense reports or signing up for classes – to an assistant that understands context.
Limitations and open questions
Despite the hype, Google admits limitations. The current model only works in Chrome and on websites. It doesn’t yet handle desktop applications like Excel or Photoshop. The safety service can sometimes be overly cautious, requiring the user to intervene frequently. And because the model relies on screenshots, performance may degrade on pages with heavy animations or endless scrolling.
Another challenge is accessibility. Many websites change layout frequently or display dynamic content based on location or user profile. The model may need constant fine‑tuning to handle these variations. Furthermore, relying on a screenshot loop means privacy – the AI sees everything on the screen. Google says screenshots are processed locally and only minimal metadata is sent, but sceptics are wary.
Future outlook
Google plans to expand the model to mobile apps and eventually to an entire operating system. The company is also exploring a business version where enterprises can define custom UI flows and integrate with internal tools. This could transform how employees handle repetitive tasks like HR forms or invoice approvals.
For the broader ecosystem, the launch highlights a shift toward agents that operate on existing interfaces rather than waiting for developers to build APIs—a direction also visible in OpenAI’s ChatGPT Apps SDK strategy. If successful, this paradigm could democratize automation, letting users point an AI at any website and say, “Do this for me.”
