AI program & delivery lead · RL environments · LLM evaluation · India
Amit Thakur
I build the environments where frontier AI learns.
Work
Selected work.
Three case studies. Everything else follows.
Deccan AI · Agentic Pod · May 2026 – now
Agentic RL-Gym
Multi-server tool-use environments where frontier agents train, with the task and QC standards the teams work from.
Case studyDeccan AI · 2026
Computer‑Use Agent pilot
From market research to a packaged sample deliverable: pipeline design, platform requirements, an intent-level taxonomy.
Case studyDeccan AI · Oct 2025 – now
LLM-as-judge at scale
Reward and judge pipelines scoring reasoning and tool use, over synthetic enterprise data with real referential integrity.
Case studyEverything else · 2023 → now
- 2026 Camera-motion reasoning benchmarkin progress A CameraBench-aligned benchmark for how multimodal models reason about camera movement, from rubric and taxonomy design to pilot data collection. Deccan AI · Agentic Pod
- 2026 BrowserGym web-agent pilot Annotation review process and reviewer management for sample task creation, with prompt-quality standards and screening for reward hacking. Deccan AI · Agentic Pod
- Oct 2025 – May 2026 Multimodal LLM / VLM dataset products Roadmap, workflows, rubrics and annotation design end to end; QC processes, resource planning and PnL across freelance, contractor and internal teams. Deccan AI · Multimodal Pod 15–20 people day to day, 50 at peak
- 2025 – 2026 Agent architecture benchmarks ReAct versus planner–executor setups benchmarked per task shape, with early LLM evaluation pipelines built on golden datasets. Deccan AI · Multimodal Pod ~30% higher task success
- Jan – Oct 2025 Agent failure analysis, Google Gen AI project 5,000+ agent responses scored with rubrics and RLHF-style preference evaluation; reasoning chains and tool-call sequences traced against golden trajectories. Turing ~12% prompt success, ~20% fewer tool-misuse errors
- Sep 2024 – Feb 2025 Airflow ETL for AI training data Production pipelines with automated validation, parallelised. TELUS Digital AI 20k+ records a day, 30% faster
- Oct 2023 – Aug 2024 Forecasting and reporting automation Forecasting and variance analysis for budgeting, with Power BI dashboards and automated reporting. ArcadeDigitech +15% budgeting accuracy, 70% less prep time
Experience
From analyst to program lead, 2023 to now.
- May 2026 – now 4 mo
Associate Product Manager, Agentic Pod · RL-Gym lead
Deccan AI
Technical direction for the non-coding RL-Gym: task environments, evaluation standards, new-offering pilots.
- Mar 2026 – May 2026 2 mo
Associate Product Manager, Multimodal Pod
Deccan AI
Owned multimodal LLM and VLM dataset products end to end; client solution design and pricing.
- Oct 2025 – Mar 2026 5 mo
AI Operations Associate, Multimodal Pod
Deccan AI
Annotation strategy, QC processes and resource planning across freelance, contractor and internal teams.
- Apr 2025 – Aug 2025 4 mo · part-time
AI Model Evaluator (part-time)
Soul AI, Deccan AI expert network
Expert evaluation of model outputs against task rubrics, alongside the Turing contract.
- Jan 2025 – Oct 2025 9 mo
Applied LLM Engineer (contract)
Turing · Google Gen AI project
Failure analysis and preference evaluation across 5,000+ agent responses.
- Sep 2024 – Feb 2025 5 mo
Data Engineer
TELUS Digital AI
Airflow ETL for AI training data, 20k+ records a day with automated validation.
- Oct 2023 – Aug 2024 10 mo
Data Analyst
ArcadeDigitech
Forecasting, variance analysis and Power BI reporting for business planning.
About
I don't treat data, models, evaluation, and product as four separate jobs.
- L1 · Data
The environment is the data. If the world an agent acts in isn't real enough, nothing downstream is worth measuring.
- L2 · Models
I don't train frontier models. I study how they break, which is a different skill and one that transfers across labs.
- L3 · Evaluation
A score you can't defend is a number, not a signal.
- L4 · Product
Research that doesn't ship is a slide deck with good citations.
Signals
Things I've said in public.
we built VoxClash: a blind arena where listeners pick the better of two voices, enough picks turn into an Elo score, and every vote gets tagged with what actually went wrong, a flat line, an odd pause, a mispronunciation. Not just a ranking but the reason behind it.Read on LinkedIn ↗ Latent reasoning from a 150M-parameter model · Aug 2026
The hottest result in AI right now came from a 150M parameter model. Pathway, a lab betting on post-transformer architectures, published ARC-AGI-1 results for BDH-CQ this month. The model doesn't write a chain of thought. It reasons in latent space, iterating on the solution internally and only producing output when it's done. The numbers surprised a lot of people in AI community : → 29.5% — pass@2 on the ARC-AGI-1 public eval set → $0.0007 — inference cost per task → 0.85 — H200 GPU-seconds per task → 11× — cheaper than GPT-5.6 Luna (Low), even after OpenAI's 80% price cut. For context: ARC-AGI was built specifically to resist memorization. Frontier models throw thousands of reasoning tokens at it. This model is smaller than BERT. Two questions worth arguing about: 1. Does reasoning need to be verbalized at all? 2. If latent reasoning scales, what happens to interpretability when there's no trace left to read?Read on LinkedIn ↗ Agents that don't trust their own map · Aug 2026
Give an AI agent a map and it will still open every door. A benchmark published last week (ScrambleToolBench) tested something most tool-use benchmarks skip: can an agent figure out unfamiliar tools purely by trial and error? The authors scrambled tool names and schemas so prior knowledge was useless. The agent has to build its own map of what each tool does. The models did fine at the discovery part. The failures showed up later, when the environment shifted: → Instead of tracing the change deductively, agents fell back to exhaustive search → They showed "belief inertia", clinging to an outdated map instead of updating it → More test-time reasoning made it worse. Extra thinking tokens amplified the brute-force search → Persistent memory reduced compounding errors but didn't close the gap The part that stings: when your agent re-explores everything for the tenth time, that isn't thoroughness. It doesn't trust what it already learned. Paper: arxiv.org/abs/2608.02358Read on LinkedIn ↗ Does a robot need an LLM in the control loop? · Aug 2026
Does a robot really need a billion-parameter LLM just to pick up a cup? A new paper, TurboVLA (from HUST and Huawei), made me question that assumption. Many recent VLA architectures rely on a large LLM in the control loop. This not only enhances the reasonability aspect but also results in increased response times and a higher demand for GPUs. TurboVLA takes a different approach by using a vision encoder, a lightweight BERT encoder, and a small cross-attention module instead of an LLM in the control loop. Here are the results on the LIBERO benchmark: • 97.7% success rate (vs 96.9% for π0.5) • 0.2B parameters (~6% the size of π0.5) • 32 Hz control frequency (vs 11 Hz) • 0.9 GB VRAM (vs 12.8 GB) on a single Nvidia RTX 4090 TurboVLA also performed well on real hardware, outperforming π0.5 on all four manipulation tasks after being trained on just 65 demonstrations per task. What impressed me most was the ablation study. When language input was removed, success on goal-directed tasks dropped from 97.4% to 11.6%. That's the key takeaway to me. Language is still essential for understanding tasks, but a large LLM doesn't necessarily need to be part of the control loop. If this approach continues to improve, it could make robot policies faster, more efficient, and much easier to deploy on local hardware. Paper: https://lnkd.in/dseiah3FRead on LinkedIn ↗ Coding agents fail on alignment, not capability · Jul 2026
Your AI coding agent isn't dumb. It just doesn't understand you. A recent study examined 20,574 real developer interactions with AI coding tools (Cursor, Claude Code, Copilot and others). One of the biggest studies of how these systems really behave in the wild. The unexpected result: most failures were not capability failures. The system was able to generate the code. The failures were mainly caused by alignment issues. The real problem: 38% → Broke developer rules 27% → Misunderstood developer intent 23% → Misreported progress (said work was done when it wasn’t) 18% → Actual coding mistakes Fix rates were low: Only 9% of failures were fixed 91% of fixes required developer feedback The system fixed itself just 3% of the time And code-level errors are decreasing while constraint violations and misleading "done!" reports are increasing. The problem is no longer one of model intelligence. It's one of communication. Paper: arxiv.org/abs/2605.29442Read on LinkedIn ↗