← All thoughts

Tech & AI

Stale Code Comments Now Cost Money With AI Agents

Bar chart of extra steps, tokens and wait time in coding agent sessions with an unprompted detour

On November 6 two Google engineers, Sam Saccone and Madhuri Garikapati, published a short study on the Google Developers Blog called Agent Distraction. They looked at traces from Google's internal coding agents across thousands of engineers and hundreds of millions of agent steps. They used one narrow signal: inline TODO and FIXME comments that point to a bug ticket. The ticket shows whether the bug is still open, so they could tell whether an agent was doing the work it was asked to do or following an old note it found in a file.

The first numbers look small. Only 13% of sessions exposed an agent to one of these comments and agents ignored the comment in 94% of encounters. In the other 6% the agent took some action on the referenced bug. Most of that was unrequested. Google says 83% of those actions came without the developer asking about the bug.

What Google found when agents ran into bug-referencing TODO comments
MeasureResult
Sessions exposed to a TODO or FIXME bug reference13%
Encounters where the agent acted on the bug6%
Share of those actions the developer did not ask for83%
Unprompted detours aimed at closed tickets41%
Unprompted detours aimed at open bugs outside the task57%
Open-bug detours where the agent edited code39%

Source: Google Developers Blog, with a second summary from InfoQ.

The cost is the part I would pay attention to. Google controlled for developer, codebase complexity, task type, model tier, files read and several other factors. After that, a session with an unprompted detour still used 70% more steps and 77% more tokens than one without, with 105% more wait time for the developer. This is an observational study inside one company and Google has not published the raw data. Even so, I don't think the direction is surprising to anyone who uses agents every day.

Bar chart showing that coding agent sessions with an unprompted detour used 70% more steps, 77% more tokens and 105% more wait time
Estimated extra cost per session when a Google coding agent chased a bug reference the developer had not asked about. Data from Google's Agent Distraction study.

Why this looks familiar to me

I use agents for a large share of the code in my own SaaS projects and the behavior Google describes matches what I see. An agent reads several files to plan a change, finds a note about something unrelated and treats the note as part of the assignment. A human developer skims past an old comment in a second or two. An agent can turn the same comment into a side branch of work that I then have to review and usually throw away.

I have run tech sites for close to 20 years and the publishing side has gone through a similar change. An old article with outdated instructions did little damage when it sat deep in the archive and almost nobody opened it. Once search engines and AI answers began pulling from the whole archive, those old pages started to matter again. Codebases are in the same position now. Text that no person read for years is read by an agent on nearly every run.

Google is making the same point from another direction

On Tuesday Chrome released Modern Web Guidance 1.0, a free set of skills that coding agents load while writing web code. Version 1.0 adds a modernization workflow that scans HTML, CSS, JavaScript and package.json for legacy patterns and then walks the agent through safe upgrades. Firefox and Edge are joining the project. Search Engine Journal checked the public eval dashboard on Tuesday morning and found guided runs passing 88% of automated checks against 56% for unguided runs. That is Google grading its own tool, so I would treat the gap as an upper bound.

The two releases cover different problems, but both rest on the same idea. What an agent reads in and around a repository has a large effect on the code it produces. Old comments pull it toward work nobody asked for. Old patterns in its training data pull it toward code that browsers no longer need.

What I expect next

I expect the major coding tools to ship stale-context checks as a default feature within the next year. These would flag comments that point to closed tickets, docs that no longer match the code and agent instruction files that describe a project as it used to be. Cleanup work that teams kept postponing because it looked cosmetic will get done, mostly by agents. Google's own cleaner removed thousands of obsolete references over six months and more than 95% of its changes were accepted by human reviewers.

For a small team the first step is cheap. Search the repository for TODO and FIXME comments, compare them with the issue tracker and delete the ones that are resolved. I would do that before paying for another round of tokens on a large agent task. Google's figures come from one very large codebase and I would like to see a smaller company publish the same measurement from its own traces. I expect the result to point the same way.