IOV LABSThere are more AI tools than anyone can try, and using them feels fast. So the natural assumption is that adoption raises productivity. The most rigorous measurement we have says the opposite, and the gap is large enough to be worth explaining rather than dismissing.
The mirage
In METR's 2025 randomized controlled trial, experienced open-source developers using AI estimated they were 20 percent faster. Measured against the clock, they were 19 percent slower. Not given no tools, given current tools and timed: a 39-point gap between what they felt and what happened. We call this gap the productivity mirage.
The claim of this study is that the mirage is not a METR quirk. It is one recurring structure, and it shows up wherever we look.
One frame: tools only lower the cost of producing
Split any piece of work into producing (the draft, the code, the document) and turning that into an outcome (connecting it to the real task, verifying it is correct, controlling the risk). AI tools make producing dramatically cheaper, so the work feels fast. But realized productivity is decided in the second half, and tools do not give you the second half. Worse, when producing gets cheap and voluminous, there is more to verify and fix downstream. The mirage is exactly this gap.
The same gap in three domains
We had already measured the second half in three separate studies. Put side by side, they are the same shape. In general development, METR's 39-point gap. In vibe-coding security (The Vibe Tax), asking only for speed raised the vulnerability rate from 20 to 50 percent, and the scanner caught none of the truly vulnerable code, a 30-point gap between feeling done and being safe. In agent completion (The Completion Illusion), agents self-reported 100 percent done while actually landing 90.8 percent, a 9.2-point gap between claimed and real.
Compounding the leaks reproduces the reversal
If these are one phenomenon, the leaks we measured should compound into the METR reversal. We built a transparent time-budget model: a task is one unit, AI speeds the producing substep, and each measured leak (false completion 7.4%, security rework 30 points, plus grounding and context-switching) adds rework. With the produce speedup alone the model feels plus 43 percent. Compound the measured leaks and it lands at minus 18 percent. We did not tune to that number; it coincides with METR's independently measured minus 19 percent. Our leaks are the right size to account for the reversal. This is a consistency argument, not a proof of cause, and we say so.
The fix is a system, not more caution
If the mirage is the absence of control rather than a property of the tool, then adding control should recover the measured figure with the same tool. Our RAI Pipeline study is that counterexample. Put routing, grounding, completion verification, and governance in the system layer and the same time-budget model moves from minus 18 to plus 27 percent, while cost drops 66 percent. The point is not a better individual tool. It is building the connection, verification, and control layers that turn produced output into outcome. This is exactly the conclusion The Vibe Tax and The Completion Illusion each reached on their own: the answer is not "be more careful," it is systemic.
Takeaway. The bottleneck in AI productivity is not the tool. Tools already made producing cheap, which is why everyone feels faster. Measured productivity comes from connection, verification, and control, and if you do not build those as a system, the feeling rises while the number falls. The next competition is not about choosing tools; it is about the system that turns tools into outcomes.
A synthesis and a model, not a new experiment. The four leak sizes are measured (three by us, one by METR); the compounding is a transparent model with stated assumptions. RAI is a product the author runs, disclosed, and used here only as the counterexample, which the model reproduces without it.