Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study
arxiv.org
As autonomous coding agents see rapid adoption, their evaluation has primarily focused on task completion rates holding the target codebase fixed. This leaves a critical question unanswered: does the structural and stylistic quality, or ``cleanliness'' of the underlying code affect an agent's ability to navigate and modify it? To isolate the effect of code cleanliness from agent capability, we introduce an evaluation protocol built around minimal pairs: repositories that match on architecture, dependencies, and external behaviour, but differ on static-analysis rule violations and cognitive complexity. The pairs are constructed in both directions, by agent pipelines that either degrade a clean repository or clean a messy one. We author 33 tasks across six such pairs, evaluated through hidden tests at the application's public surface. Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate. However, it substantially alters the agent's operational footprint: agents working on cleaner code use 7 to 8% fewer tokens and reduce file revisitations by 34%. Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development, shaping the computational cost and navigational efficiency of coding agents. Code cleanliness joins model choice, harness, and prompting as a factor that materially affects agent behaviours.
  • ExtremeDullard@piefed.socialEnglish
    2·
    1 month ago

    Yes: I experimented with this. I stripped out all the comments in one of my Python libraries and asked the model to explain what the library does. And what do you know: all of the sudden, instead of giving me a nice overview of the library and its functionality, it hallucinated a bunch of half-baked bullshit.

    I will add this: run the model a few times on the codebase and will fuck it up enough to degrade its own performances on it.

  • tangeli@piefed.socialEnglish
    1·
    1 month ago

    The International Obfuscated C Code Contest should run a separate competition for code obfuscated from the perspective of LLMs. Cracking Complexity: ChatGPT’s Exploration of Enigmatic C Code from the IOCCC is a start, but there should be a stream that rewards code inscrutable to LLMs specifically.

    It is inconceivable that all expressions of an algorithm can be equally well used, maintained and elaborated by LLMs, including at the same cost. The question is not whether the ‘cleanliness’ of code makes a difference but rather, which aspects of coding style make code most amenable to development, use and maintenance using LLMs and how to configure LLMs to produce such code. Bonus for identifying those aspects that also make it more comprehensible to humans.