AI program & delivery lead · RL environments · LLM evaluation · India

Amit Thakur

I build the environments where frontier AI learns.

Work

Selected work.

Three case studies. Everything else follows.

Everything else · 2023 → now

  1. 2026 Camera-motion reasoning benchmarkin progress A CameraBench-aligned benchmark for how multimodal models reason about camera movement, from rubric and taxonomy design to pilot data collection. Deccan AI · Agentic Pod
  2. 2026 BrowserGym web-agent pilot Annotation review process and reviewer management for sample task creation, with prompt-quality standards and screening for reward hacking. Deccan AI · Agentic Pod
  3. Oct 2025 – May 2026 Multimodal LLM / VLM dataset products Roadmap, workflows, rubrics and annotation design end to end; QC processes, resource planning and PnL across freelance, contractor and internal teams. Deccan AI · Multimodal Pod 15–20 people day to day, 50 at peak
  4. 2025 – 2026 Agent architecture benchmarks ReAct versus planner–executor setups benchmarked per task shape, with early LLM evaluation pipelines built on golden datasets. Deccan AI · Multimodal Pod ~30% higher task success
  5. Jan – Oct 2025 Agent failure analysis, Google Gen AI project 5,000+ agent responses scored with rubrics and RLHF-style preference evaluation; reasoning chains and tool-call sequences traced against golden trajectories. Turing ~12% prompt success, ~20% fewer tool-misuse errors
  6. Sep 2024 – Feb 2025 Airflow ETL for AI training data Production pipelines with automated validation, parallelised. TELUS Digital AI 20k+ records a day, 30% faster
  7. Oct 2023 – Aug 2024 Forecasting and reporting automation Forecasting and variance analysis for budgeting, with Power BI dashboards and automated reporting. ArcadeDigitech +15% budgeting accuracy, 70% less prep time

Experience

From analyst to program lead, 2023 to now.

Full detail
  1. May 2026 – now 4 mo

    Associate Product Manager, Agentic Pod · RL-Gym lead

    Deccan AI

    Technical direction for the non-coding RL-Gym: task environments, evaluation standards, new-offering pilots.

  2. Mar 2026 – May 2026 2 mo

    Associate Product Manager, Multimodal Pod

    Deccan AI

    Owned multimodal LLM and VLM dataset products end to end; client solution design and pricing.

  3. Oct 2025 – Mar 2026 5 mo

    AI Operations Associate, Multimodal Pod

    Deccan AI

    Annotation strategy, QC processes and resource planning across freelance, contractor and internal teams.

  4. Apr 2025 – Aug 2025 4 mo · part-time

    AI Model Evaluator (part-time)

    Soul AI, Deccan AI expert network

    Expert evaluation of model outputs against task rubrics, alongside the Turing contract.

  5. Jan 2025 – Oct 2025 9 mo

    Applied LLM Engineer (contract)

    Turing · Google Gen AI project

    Failure analysis and preference evaluation across 5,000+ agent responses.

  6. Sep 2024 – Feb 2025 5 mo

    Data Engineer

    TELUS Digital AI

    Airflow ETL for AI training data, 20k+ records a day with automated validation.

  7. Oct 2023 – Aug 2024 10 mo

    Data Analyst

    ArcadeDigitech

    Forecasting, variance analysis and Power BI reporting for business planning.

Amit Thakur

About

I don't treat data, models, evaluation, and product as four separate jobs.

  • L1 · Data

    The environment is the data. If the world an agent acts in isn't real enough, nothing downstream is worth measuring.

  • L2 · Models

    I don't train frontier models. I study how they break, which is a different skill and one that transfers across labs.

  • L3 · Evaluation

    A score you can't defend is a number, not a signal.

  • L4 · Product

    Research that doesn't ship is a slide deck with good citations.

Signals

Things I've said in public.

All posts on LinkedIn
VoxClash: a blind arena for TTS, scored with reasons · Sep 2026
we built VoxClash: a blind arena where listeners pick the better of two voices, enough picks turn into an Elo score, and every vote gets tagged with what actually went wrong, a flat line, an odd pause, a mispronunciation. Not just a ranking but the reason behind it.
Read on LinkedIn ↗
Latent reasoning from a 150M-parameter model · Aug 2026
The hottest result in AI right now came from a 150M parameter model. Pathway, a lab betting on post-transformer architectures, published ARC-AGI-1 results for BDH-CQ this month. The model doesn't write a chain of thought. It reasons in latent space, iterating on the solution internally and only producing output when it's done. The numbers surprised a lot of people in AI community : → 29.5% — pass@2 on the ARC-AGI-1 public eval set → $0.0007 — inference cost per task → 0.85 — H200 GPU-seconds per task → 11× — cheaper than GPT-5.6 Luna (Low), even after OpenAI's 80% price cut. For context: ARC-AGI was built specifically to resist memorization. Frontier models throw thousands of reasoning tokens at it. This model is smaller than BERT. Two questions worth arguing about: 1. Does reasoning need to be verbalized at all? 2. If latent reasoning scales, what happens to interpretability when there's no trace left to read?
Read on LinkedIn ↗
Agents that don't trust their own map · Aug 2026
Give an AI agent a map and it will still open every door. A benchmark published last week (ScrambleToolBench) tested something most tool-use benchmarks skip: can an agent figure out unfamiliar tools purely by trial and error? The authors scrambled tool names and schemas so prior knowledge was useless. The agent has to build its own map of what each tool does. The models did fine at the discovery part. The failures showed up later, when the environment shifted: → Instead of tracing the change deductively, agents fell back to exhaustive search → They showed "belief inertia", clinging to an outdated map instead of updating it → More test-time reasoning made it worse. Extra thinking tokens amplified the brute-force search → Persistent memory reduced compounding errors but didn't close the gap The part that stings: when your agent re-explores everything for the tenth time, that isn't thoroughness. It doesn't trust what it already learned. Paper: arxiv.org/abs/2608.02358
Read on LinkedIn ↗
Does a robot need an LLM in the control loop? · Aug 2026
Does a robot really need a billion-parameter LLM just to pick up a cup? A new paper, TurboVLA (from HUST and Huawei), made me question that assumption. Many recent VLA architectures rely on a large LLM in the control loop. This not only enhances the reasonability aspect but also results in increased response times and a higher demand for GPUs. TurboVLA takes a different approach by using a vision encoder, a lightweight BERT encoder, and a small cross-attention module instead of an LLM in the control loop. Here are the results on the LIBERO benchmark: • 97.7% success rate (vs 96.9% for π0.5) • 0.2B parameters (~6% the size of π0.5) • 32 Hz control frequency (vs 11 Hz) • 0.9 GB VRAM (vs 12.8 GB) on a single Nvidia RTX 4090 TurboVLA also performed well on real hardware, outperforming π0.5 on all four manipulation tasks after being trained on just 65 demonstrations per task. What impressed me most was the ablation study. When language input was removed, success on goal-directed tasks dropped from 97.4% to 11.6%. That's the key takeaway to me. Language is still essential for understanding tasks, but a large LLM doesn't necessarily need to be part of the control loop. If this approach continues to improve, it could make robot policies faster, more efficient, and much easier to deploy on local hardware. Paper: https://lnkd.in/dseiah3F
Read on LinkedIn ↗
Coding agents fail on alignment, not capability · Jul 2026
Your AI coding agent isn't dumb. It just doesn't understand you. A recent study examined 20,574 real developer interactions with AI coding tools (Cursor, Claude Code, Copilot and others). One of the biggest studies of how these systems really behave in the wild. The unexpected result: most failures were not capability failures. The system was able to generate the code. The failures were mainly caused by alignment issues. The real problem: 38% → Broke developer rules 27% → Misunderstood developer intent 23% → Misreported progress (said work was done when it wasn’t) 18% → Actual coding mistakes Fix rates were low: Only 9% of failures were fixed 91% of fixes required developer feedback The system fixed itself just 3% of the time And code-level errors are decreasing while constraint violations and misleading "done!" reports are increasing. The problem is no longer one of model intelligence. It's one of communication. Paper: arxiv.org/abs/2605.29442
Read on LinkedIn ↗