AI Coding Assistance Produced No Clear Speed Gain but Lower Quiz Scores
Computer scientist Károly Zsolnai-Fehér argues that AI coding assistance can weaken learning when it substitutes for understanding rather than supporting it. In the small study he examines, mostly junior developers using an AI assistant finished a Python task only marginally faster—a difference that was not statistically significant—but scored 17 points lower on a subsequent quiz, with the largest weakness in debugging. His recommendation is to automate work already understood and use AI as a tutor for unfamiliar tasks.
The speed gain was modest; the learning gap was not
Károly Zsolnai-Fehér contrasts broad warnings that AI is “rotting your brain” with a narrower question: what happens when people use an AI coding assistant while learning a new technical skill?
The study he discusses split 52 mostly junior software engineers into an AI-assisted group and a hand-coding group. Participants worked with the Python library Trio, completed a short coding task, and then took a quiz. The AI users finished roughly two minutes sooner on average—about 23 minutes rather than 25, an 8% difference—but the task-time result was not statistically significant.
The quiz produced a clearer separation. Developers who wrote code by hand averaged 67%; those who used AI averaged 50%. The difference, nearly two letter grades, was statistically significant, with a reported effect size of Cohen’s d = 0.738 and p = 0.01.
| Measure | AI group | No-AI group | Reported result |
|---|---|---|---|
| Task time | About 23 minutes | About 25 minutes | p = 0.391; not statistically significant |
| Post-task quiz | 50% | 67% | Cohen's d = 0.738; p = 0.01 |
Zsolnai-Fehér therefore qualifies the productivity finding: AI may have made participants faster, but the experiment did not establish that it did. The stronger result within this study is that AI-assisted participants performed worse afterward on an assessment of the material.
The study design separates completing an assignment from retaining the understanding needed to work independently. Participants first completed a 10-minute warm-up coding task, then a 35-minute Trio task, followed by a 25-minute post-task quiz and a five-minute survey. Zsolnai-Fehér characterises the result cautiously: the AI-assisted group scored lower on the post-task quiz.
Debugging exposed the cost of delegation
The average quiz difference matters, but Károly Zsolnai-Fehér emphasizes where it was largest: debugging questions. The AI group particularly underperformed when asked to identify that code was incorrect and explain why it failed.
For Zsolnai-Fehér, that pattern identifies a practical risk in delegating code generation. AI can produce code, but when something goes wrong, the user still has to diagnose and repair the failure.
If you rely on the AI too much, sooner or later, you will lose the capability to fix something that goes wrong.
A displayed chart categorizes several modes of AI use as either higher- or lower-skill-development interactions. Generation-then-comprehension, hybrid code-and-explanation interactions, and conceptual inquiry are placed in the higher-skill grouping. AI delegation, progressive AI reliance, and iterative AI debugging are placed in the lower-skill grouping.
Two plotted examples make the distinction concrete without establishing a universal relationship between speed and learning. Generation-then-comprehension is shown at a 24-minute completion time and an 86% skills-quiz score. AI delegation is shown at 19.5 minutes and 39%. The chart’s framing is about the character of the interaction: whether the user is using the system to develop and check understanding, or progressively handing responsibility for the work to it.
Use AI to accelerate known work and interrogate unfamiliar work
Károly Zsolnai-Fehér does not argue for abandoning coding assistants. His distinction is between automating work whose underlying reasoning is already understood and using AI as a tutor when the work is unfamiliar.
First, use AI to automate or speed up things already understood. In that role, he says, it is excellent: the system can reduce effort on tasks whose underlying reasoning the user already possesses.
Second, when working with something unfamiliar, ask questions. The intended role is tutor rather than replacement. Zsolnai-Fehér’s advice is to use the assistant for conceptual inquiry and explanation while keeping the learner mentally engaged.
Third, when code breaks, try to fix it before asking AI to intervene. Then ask it to explain what was missed. That sequence preserves an attempt to diagnose the problem and turns the model’s response into feedback on the user’s reasoning.
The decision rule is not simply whether to use AI. It is whether the interaction leaves the user responsible for understanding. Generation may complete a task; questions, explanations, and a first attempt at debugging are the practices Zsolnai-Fehér associates with learning from it.
The result is a warning, not a general verdict
Károly Zsolnai-Fehér is explicit about the study’s limits: it involved 52 mostly junior developers, one Python library, one short task, and one quiz. The assistant was chat-based, not a full agentic coding system.
He says a full agentic system would probably make the observed difference larger, but presents that as his expectation rather than an experimental result. The study is not, in his terms, an ultimate study or the final word on AI and coding skills.
Its usefulness lies in making a specific trade-off visible. The apparent time advantage was not statistically established, while the post-task learning gap was—and debugging was the area of greatest weakness. For someone learning a new tool or library, Zsolnai-Fehér’s recommendation is to keep AI in a tutoring role: use it to clarify concepts and explain missed reasoning, rather than as a substitute for producing, checking, and repairing code.