A demo answers a simple question: can an agent do this task? Production asks a harder one: will it do this task every time, across thousands of runs, at a cost that makes sense?
Moving from the first question to the second is where most of the real engineering in computer-use agents happens.
Scale changes what you notice
At one run a day, a one-in-a-thousand failure is invisible. At ten thousand runs a day, it happens ten times before lunch. Scale turns edge cases into daily events and exposes problems that never show up in a demo:
- Variance. The same task can take different paths on different runs, and small differences compound.
- Drift. The systems an agent operates change — new layouts, renamed fields, extra steps — often without notice.
- Concurrency. Many runs at once can collide on rate limits, sessions, and shared records.
- Cost. A few extra model calls per run are trivial once, and significant at volume.
Evaluations are the roadmap
The teams that scale well treat evaluation as a core product function, not a research exercise.
They build evaluation sets from real runs, including the failures. They keep regression suites for every workflow they support, so a fix in one place doesn’t quietly break another. They track results per customer environment, because the same workflow can behave differently in two companies’ systems.
Most importantly, they name their failure modes and count them. A list of the top ten reasons jobs fail, with numbers next to each, is one of the most useful documents an agent company can have.
You can’t improve a completion rate you don’t measure.
Observability for agents
When something goes wrong at scale, you need to understand it quickly. Agent products need their own kind of observability:
- A timeline for every run, with screenshots or recordings at key steps.
- A clear record of which decisions the agent made, and why.
- Failures grouped by cause, so patterns stand out.
- Alerts when completion rates drop or a workflow starts behaving differently — often the first sign that a target system has changed.
The economics of a completed job
At volume, the unit that matters is the cost per completed job — including retries, verification, and the human time spent on exceptions. It is the number to compare against the cost of doing the work by hand, and it improves with every point of reliability.
That framing also clarifies trade-offs. A more capable model on a hard step may be cheaper overall if it prevents retries. A cheaper model on a simple step may be all you need. Good systems choose deliberately, step by step.
Operations is a feature
Running ten thousand jobs a day is an operational challenge as much as a technical one. Queues, scheduling, back-pressure when a target system is slow, priority for urgent work, service-level commitments, and well-designed review queues for the jobs that need a person — these are all part of the product customers experience.
From possible to dependable
The first wave of computer-use agents proved that this is possible. The next wave will prove that it is dependable. That is a harder, less glamorous problem, and it is the one we are most excited about.
If you are working on it, we want to hear from you.