The control arm
The comparison group nobody is running — the same work done the old way, so you could actually tell whether the new way is faster.
Also called control group
Every claim about AI productivity is a claim about a difference, and a difference requires two measurements. Almost nobody has two.
The problem is not ignorance of research methods. Running a control arm means doing the same work the slow way on purpose. Developers do not volunteer for that, and managers do not want to pay for it. The result is a pile of confident numbers in both directions that no one can falsify.
I do not run the control arm either. I feel several times faster than I was two years ago, but I have no evidence that would survive a hostile reading. I would rather admit that than add another unsupported number.
See also Verification.