IOV LABSThe Productivity Mirage: Why AI Tool Adoption Can Lower Measured Productivity
Abstract
AI tools have exploded and users feel faster, but the most rigorous measurement points the other way: in METR's 2025 RCT, skilled developers estimated they were 20% faster with AI while they were measured 19% slower, a 39-point gap. We call this gap the productivity mirage, and we argue it is not a METR anomaly but one recurring structure across domains. We synthesize three original IOV Labs measurements (The Completion Illusion, The Vibe Tax, The RAI Pipeline) with the external METR RCT under one frame: tools lower the cost of producing output, so work feels faster, but realized productivity is decided in connection, verification, and control, which tools do not provide. We quantify the same gap in three domains, general development 39 points, vibe-coding security 30 points (vulnerability 20 to 50 percent), agent completion 9.2 points (self-report 100 percent versus actual 90.8 percent). Then a transparent time-budget model shows that compounding these independently measured leaks turns a felt +43% into a measured -18%, a magnitude we did not tune but that coincides with METR's independent -19%. Finally a counterexample: a pipeline with control layers (routing, grounding, completion checks, governance) cuts cost 66% and recovers the measured figure to +27%. The conclusion is methodological, AI productivity is a systems problem, not a tool-choice problem. This is a synthesis and a model, not a new experiment, the leak sizes are measured and the compounding is a model with stated assumptions.
Keywords
- productivity
- AI adoption
- METR
- perception gap
- verification
- control layers
- completion illusion
- vibe coding
- synthesis
- reproducibility