The Computer-Use Agent pilot

Problem

Computer-use agents are a category everyone can describe and very few people can evaluate. The model operates a screen. The training data has to capture why each action happened and whether the goal was ever reached, neither of which a click log tells you. Nobody hands you a labeled corpus for that.

Deccan AI needed to know whether this was a real offering or an interesting demo. My job was to answer that end to end, and to answer it in a form a client could act on rather than a memo saying the space looks promising.

How I thought about it

The failure mode for a pilot like this is starting with the pipeline. It's the fun part, so people build it first, and then the market research becomes a search for evidence that the thing they already built is what somebody wants.

I ran it the other way. Market research first: who is buying computer-use data, what shape are they buying it in, what does an acceptable deliverable look like on their side. Only then did the pipeline design start, and it started from the deliverable and worked backwards.

The second thing I decided early was that the annotation platform had to be scoped as a technical requirement, not a preference. Screen-based trajectory annotation has real constraints around capture fidelity and around how you review a session without watching all of it. If the platform can't support the review workflow, the QC cost eats the margin, and you find that out at scale rather than in the pilot.

The third decision was about what a step is. Screen actions are a terrible unit of labeling on their own, because a click carries no intent and an agent that clicks the right thing for the wrong reason looks identical in the log to one that understood the task. So the taxonomy had to work at the level of intent, with screen actions attached underneath. That choice drove almost everything else: the schema, the review workflow, how long a session takes to annotate, and therefore what the work costs to run.

Architecture

The pilot was built as a chain from capture to deliverable. Trajectory capture produced screen state and action sequences. An annotation layer added intent and step-level labels against a taxonomy I designed. A validation layer applied rubric-based scoring and automated checks for the failures that are cheap to detect programmatically, like broken action sequences or trajectories that never reach a terminal state. The output was a packaged sample deliverable with documentation of the schema, the rubric, and the QC process, so a client evaluating it could see how it was made rather than only what it held.

What I built

I ran the market research and wrote the offering definition. I designed the ML and data pipeline. I wrote the technical requirements for the annotation platform, in the form the platform team could build against. And I packaged the sample deliverable, which is the part that turns a pilot into a conversation with a buyer.