About Associate Product Manager, Agentic Pod (RL-Gym) · Deccan AI · Hyderabad

Amit Thakur

I build the environments where frontier AI learns.

01 / 03

One system, four layers

I don't treat data, models, evaluation, and product as four separate jobs. Most of the failures I've watched happen live in the seams between them: a rubric no two annotators apply the same way, or a reward signal that quietly scores fluency instead of correctness. So I work the whole stack, and I've spent real time at every layer of it.

L1 · Data

The environment is the data. If the world an agent acts in isn't real enough, nothing downstream is worth measuring.

I architected synthetic enterprise databases as RL-Gym environments: multi-domain Snowflake schemas with real referential integrity, so an agent has to navigate state that behaves like a company's actual systems rather than a toy fixture. Before that I owned multimodal LLM and VLM dataset products end to end at Deccan AI, from roadmap through annotation design, which is where I learned that dataset quality is mostly a question of how well the instructions were written.

L2 · Models

I don't train frontier models. I study how they break, which is a different skill and one that transfers across labs.

At Turing, on a Google Gen AI project, I read through more than 5,000 agent responses, tracing reasoning chains and tool-call sequences to find where behavior actually came apart; that work cut tool-misuse errors by roughly 20% and improved prompt success by about 12%. I also benchmarked agent architectures against each other, ReAct versus planner-executor setups, and got task success up around 30% by picking the right one for the task shape.

L3 · Evaluation

A score you can't defend is a number, not a signal.

I've built LLM-as-judge and reward pipelines that grade agent reasoning and tool-use trajectories at scale, along with the task verifiers and difficulty rubrics that define what pass and fail mean in the first place. The parts that make it hold up are unglamorous: golden datasets, and calibration rounds with annotators before a single production batch ships.

L4 · Product

Research that doesn't ship is a slide deck with good citations.

I took the Computer-Use Agent pilot from market research through pipeline design and a packaged sample deliverable, and it landed active client interest. On the multimodal side I ran resource planning and PnL across freelance and internal teams, and I was in the room for solution design and pricing, which changes how you scope technical work.

02 / 03

Experience

Associate Product Manager, Agentic Pod (RL-Gym)

May 2026 – present

Deccan AI

Technical direction for the non-coding agentic RL-Gym program, and the task-design and QC standards used across annotation and QC teams.

Set the design standard for multi-turn, multi-server tool-use environments that frontier labs train and evaluate agents against; built LLM-as-judge and reward pipelines, task verifiers, and difficulty rubrics that define the pass/fail criteria.
Led a Computer-Use Agent pilot end to end, from market research through pipeline design and a packaged sample deliverable, producing active client interest.
Architected synthetic enterprise databases as gym environments, and ran benchmarks across datasets and agent architectures whose findings drove roadmap and dataset-quality calls.
Ran the annotation review process and reviewer management for a BrowserGym web-agent pilot (sample task creation), setting prompt-quality standards and screening tasks for reward hacking.

What changedMy output is now other people's throughput. The standards I write get applied by teams I don't sit with, so ambiguity in a rubric is a bug that ships.

AI Operations Associate → Associate Product Manager

Oct 2025 – May 2026 · promoted Mar 2026

Multimodal Pod, Deccan AI

Multimodal LLM and VLM dataset products end to end, plus the delivery organization behind them.

Owned roadmap, workflows, rubrics, and annotation design; defined QC processes and resource planning across freelance and internal teams, including scaling and PnL.
Benchmarked agent architectures such as ReAct and planner-executor, improving task success by about 30%; built early LLM evaluation pipelines with golden datasets.
Built multi-turn validation frameworks and Jira-integrated QC question sets, and partnered with clients on solution design and pricing.

What changedI went from optimizing a pipeline to owning whether a program makes money. Once you carry PnL, you scope technical work differently, and you get much more honest about what a pilot is really for.

AI Model Evaluator (part-time, via Soul AI)

Apr – Aug 2025

Deccan AI's expert network

Expert evaluation work alongside the Turing contract.

Evaluated model outputs against task rubrics as a working evaluator, not a designer of the process.
The evaluation work led directly to a full-time offer from Deccan AI.

What changedSitting on the annotator side of a rubric is the fastest way to learn which instructions are actually followable. I design rubrics differently because of it.

Applied LLM Engineer (contract)

Jan – Oct 2025

Turing · Google Gen AI project

Failure analysis and preference evaluation on LLM agent behavior.

Analyzed 5,000+ agent responses with rubric-based scoring and RLHF-style preference evaluation, improving prompt success by around 12%.
Debugged agent failures by tracing reasoning chains and tool-call sequences, reducing tool-misuse errors by roughly 20%.
Ran diff-based comparison of execution traces against golden trajectories to find alignment gaps.

What changedReading thousands of trajectories by hand taught me what a useful rubric looks like, because I kept hitting the ones that weren't.

Data Engineer

Sept 2024 – Feb 2025

TELUS Digital AI

Production ETL for AI training data operations.

Built and ran Airflow pipelines processing 20,000+ records a day.
Added automated validation into the pipeline, and cut processing time by 30% through parallelization.

What changedI stopped thinking about data quality as a cleanup step and started building the checks into the pipeline itself. That habit is the direct ancestor of how I do QC now.

Data Analyst

Oct 2023 – Aug 2024

ArcadeDigitech

Forecasting, variance analysis, and reporting for business planning.

Built forecasting and variance models that improved budgeting accuracy by about 15%.
Rebuilt recurring reporting in Power BI with automation, cutting prep time roughly 70%.

What changedMost of the value in an analysis is in how fast someone can act on it, not how clever the model is.

03 / 03

Positions I'll defend

Evaluation is an environment problem

Most eval debates are about judges. Better prompts, better ensembles, better calibration. That's the second-hardest problem. The hardest one is that most benchmarks put the agent in a world too clean to fail in interesting ways. If your environment has one server, no stale state, and tools that always return, you're measuring instruction-following and calling it agency. I get more signal from making the environment harder than from making the judge smarter, and it's usually cheaper.

Reward design is taste, formalized

Everyone wants reward signals to be objective. In practice you are encoding a judgment about what counts as a good trajectory, and pretending otherwise just hides the judgment somewhere harder to audit. So I make it explicit: task verifiers for the parts that are checkable against state, rubrics for the parts that aren't, and a difficulty tier so the signal isn't flat across easy and hard tasks. The honest version of reward design is writing down your taste and letting people argue with it.

Synthetic worlds need real integrity

The thing that makes a synthetic environment useful isn't scale, it's consequence. When I build enterprise databases as gym environments, the referential integrity is the whole point: an agent that writes a bad record should break something downstream, the way it would in a real company. Loose synthetic data teaches agents that actions are free. I'd rather ship one multi-domain schema where state actually constrains behavior than ten thousand independent tasks with no world behind them.

Annotation is a systems discipline

People talk about human data like it's a labor line item. It's a distributed system with humans as the compute, and it fails the way distributed systems fail: inconsistent state and quiet degradation under load. So I treat it that way. Instructions are the API, calibration rounds are integration tests, QC question sets are monitoring, and a rubric change is a breaking change that needs a rollout plan. The teams that get this produce data two grades better at the same cost.

AGI arrives through boring infrastructure

I'm not sure capability is the bottleneck anymore. What I see day to day is that we can build models that are far better than our ability to tell whether they did the job. Every step toward more general agents makes the measurement problem harder, not easier, because the tasks get longer and the failure modes get more subtle. My bet is that a lot of the remaining distance gets covered by unglamorous work: building the environments and verifiers nobody gets credit for.

From pilot to product, honestly

A pilot's job is to find out whether a thing is real, and most pilots are built to prove that it already is. I've run enough of them to prefer the version where the sample deliverable is packaged well enough that a client could put it into production tomorrow, and where the market research happens before the pipeline design instead of after. If the pilot can't survive a pricing conversation, the research wasn't finished. Shipping is a research result.