In this data-heavy presentation, Yegor Denisov-Blanch shares findings from Stanford’s large-scale, longitudinal study analyzing real-world telemetry from nearly 100,000 developers (often cited as ~100k–120k across related versions) across 600+ companies.
The research uses sophisticated, ML-augmented metrics—approximating expert code reviews for effort, complexity, maintainability, and net value delivered—rather than misleading proxies like lines of code, commit counts, or PR volume.
It draws on millions of commits and billions of lines of code to cut through vendor hype.
Key Findings:
- Net productivity gains are real but modest: After accounting for increased rework (bug fixes, refactoring, and technical debt introduced by AI-generated code), the average/net productivity uplift is around 15–20% (median often cited closer to 10% in mid-2025 data). Some teams see strong gains, while others experience flat or even negative impact.
- High variance and contextual dependence:
- Task & project type: Biggest gains (30–40%) on low-complexity, greenfield/new code tasks. Much smaller gains (0–10% or less) on high-complexity, brownfield/legacy code maintenance.
- Language popularity: Stronger lifts (10–25%) in popular languages like Python or Java with abundant training data; limited or negative effects in niche/legacy languages (e.g., COBOL, Haskell).
- Codebase factors: Gains diminish as codebases grow larger and more complex due to context limitations. Clean, well-maintained codebases (high “cleanliness index” with good tests, docs, modularity) act as multipliers for AI success.
- The rework trap and shifting bottlenecks: AI often speeds up initial code generation and increases output volume, but this leads to more bugs, larger changes, and higher downstream rework. Short-term “feels faster” effects can mask longer-term slowdowns if not managed. Traditional metrics create false positives.
- “Rich get richer” dynamic: Top-performing teams and organizations compound advantages through better practices, while others fall further behind. The performance gap between high and low adopters is widening.
Overall Message and Recommendations:
AI coding tools deliver tangible but context-dependent value—not the 10x hype often claimed, and not universally. Success depends heavily on task suitability, codebase health, usage patterns, and organizational maturity. The talk stresses the need for better measurement frameworks, process adjustments (strong reviews, quality gates, targeted application), and avoiding over-reliance on activity metrics. It balances realism (AI won’t replace engineers anytime soon) with optimism for those who use the tools strategically. Practical advice includes focusing AI on suitable work, maintaining codebase hygiene, and measuring true outcomes. The video runs about 20–30 minutes and is frequently referenced as one of the most rigorous, non-vendor studies available.
This talk overlaps thematically with the earlier Stanford presentation you asked about but provides a focused, slightly earlier or variant view on the same research body—emphasizing nuanced, evidence-based insights over anecdotes. It pairs well with the Qodo quality report, Kitze’s vibe coding discussion, and McKinsey’s operating model talk for a full picture of AI in software engineering realities.
