← All posts

AI

Nobody carries the code: why testing-after breaks under AI-accelerated development

What erodes under AI-accelerated development is human understanding, and no amount of inspection restores it after the fact. Comprehension debt, intent debt, the generative ratification loop, and one testable proposal for preventing all three.

Vitaly Sharovatov

Vitaly Sharovatov

Nov 17, 202626 min read
Nobody carries the code: why testing-after breaks under AI-accelerated development

Vitaly Sharovatov, with Liliia Abdulina, Head of QA at JetBrains. This research was opened by Liliia and developed with her in the open. The analysis, evaluation and proposal below are Vitaly's writing from that work. Intent debt, and the triangle-of-debt framing it belongs to, was introduced by Anupam Krishnamurthy. The full research and its discussion are public at github.com/BeyondQuality/beyondquality/discussions/28.

This paper is Part One of Qase Perspectives on AI in Testing, the booklet we hand out at events. Part Two is Maksim Koutun's Test management for AI coding agents. Download the full booklet as a PDF.

The wall teams are hitting

Many companies are hitting the same wall: as teams adopt agentic AI, code output accelerates and existing testing, manual and automated, cannot cope with the volume. Testing-after straining under scale is not a new problem, but AI has multiplied it past the industry's previous coping mechanisms.

Two symptoms:

  1. Traditional "testing-after" cannot keep up when code is produced at a much higher pace. Verifying that new code does what it was supposed to and doesn't break anything else takes time, and by the time testers finish one task, the inspection queue has grown faster than anyone can clear it.

  2. Traditional "testing-after" can't test properly. To verify a feature, or build an automated test capable of that verification, humans need two types of understanding: why and how the code was built. Since code production is delegated to agents, comprehension debt grows: nobody carries the deep understanding of how the code works. Intent debt grows alongside it: people lose the why, the problem this code solves and the tradeoffs accepted.

One premise underlies everything that follows. Whether code is "good enough" depends on the business problem it solves, the consequences of failure, the product's lifecycle stage, and the tradeoffs the team has consciously accepted. A prototype and a payment system have different quality requirements even if the code looks identical. Any assessment operating only on the code surface can catch superficial issues at best.

What the old model quietly provided

Before reasoning about what agents changed, we need a picture of the process before them.

  1. Decision: humans agree on what needs to be built (idea, hypothesis, PRD, specification, whatever you call it). Decisions are always incomplete; humans fill the gaps using shared context, domain knowledge, and tacit understanding, and that shared understanding is load-bearing.
  2. Implementation: a human designs and builds from that agreement. It is also a learning loop: the developer asks the PM more questions, clarifies the PRD with the tester, notices gaps the spec didn't anticipate, and learns the product better. Some of that learning flows back into the decision, some stays tacit.
  3. Testing: verifies that the implementation matches the decision, and produces learning: that the spec was ambiguous, that the user's real workflow didn't match the PRD, that two features interact unexpectedly. That learning flows back to step 1. Testing thus has two functions: verdict (pass/fail) and learning.

In all three steps, humans carry business domain understanding, accumulating it for free by doing their jobs.

Why doing-the-work produces learning. The mechanism is well established in cognitive science. The generation effect (Slamecka & Graf, 1978; meta-analysed across 310 experiments by McCurdy et al., 2020) shows that actively producing information yields far better retention than passively receiving it. Bjork's desirable difficulties framework (1994, 2020) formalises the principle: the cognitive effort of doing something yourself is the learning, not a cost to be eliminated.

Domain understanding also includes transferable professional heuristics, what Howden called the "oracle concept": testers reason by analogy, drawing on how Java or Scala handle a concept when they test its Kotlin equivalent. That intuition accumulates across a career, not within a project.

The risk knowledge that actually drives testing decisions is always project-specific, and it sits in three layers:

  1. Strategic risk knowledge (how to assess risk, build a risk register, map testing to business exposure): transferable, available to experienced humans. This is where the economics of testing framework operates.
  2. Population-level risk weighting (payments are risky, settings pages aren't): a useful starting point, potentially miscalibrated for any specific project. A tester who lived through a payment disaster elsewhere arrives overcautious about payments while the real risk sits in the export module.
  3. Local risk calibration (in THIS project, the export module is where the real risk lives): only comes from engagement with this codebase, this team, this product. The implicit learning from doing the work.

Knowledge could still fade as people quit or transferred, but the rate was bounded: new joiners absorbed the same context through participation, so the system replenished what it lost.

Testing-after was already at its ceiling

For simplicity, I see two types of companies. To understand how their QA approaches scale, and why agents make that difference matter more, model QA costs at the team level.

Proactive QA companies first. Humans design the processes within steps 1-3 so that each step is more likely to produce something of good quality: Kaizen, zero bug policy, pair/mob programming, TDD, shift-left testing. Testing is present too, as appraisal, but the balance of effort shifts towards prevention over appraisal, and pair and mob programming act as shared understanding amplifiers.

Adding n teams, the hidden superlinear costs are the cross-team ones. Tests share a CI pipeline, so run time grows with total test count and flake probability compounds: if each of t tests has independent flake probability p, the chance of at least one flake per run is 1-(1-p)^t, which approaches 1 quickly. Team A's change breaks team B's tests, and investigation cost grows with the number of team pairs, O(n²).

Actual cost function: O(n + εn²), with n teams and ε the cross-team coordination coefficient. Proactive practices (clear module boundaries, APIs, contracts) exist specifically to keep ε small. It is never zero: their "linearity" means the quadratic coefficient is small, not absent.

Then the reactive QC companies. Little to no proactive QA: quality is controlled, not assured. Testing happens after development and struggles to cope, and dev and tester mental models drift apart, so testing becomes "guess what the developer meant". Three costs scale superlinearly:

  • Rework feedback loop: defects found → rework → rework introduces new defects at rate r, so total effort ≈ O(n / (1-r)). Since r rises with system complexity, which rises with n, the denominator shrinks as n grows and can blow up as r approaches 1.
  • Automated test maintenance: writing tests may be O(n), but maintenance grows with interconnectedness. Each new feature can break existing tests through changed assumptions, not bugs, and that fragility scales with cross-module dependencies, O(n²) in the worst case.
  • Manual testing: combinatorial state explosion. Each team adds features that interact with existing ones, and testing the interactions grows faster than the feature count.

Cost is also O(n + εn²), but ε is much larger with nothing suppressing the quadratic term, and the rework loop adds a multiplicative 1/(1-r(n)) on top: O((n + εn²) / (1-r(n))). The difference between the two types is small ε and no rework multiplier vs. large ε compounded by rework.

Scaling walls also appear inside testing itself. Before AI, deterministic testing repeatedly hit combinatorial walls, and each time the industry invented a paradigm that accepted incompleteness:

ParadigmYearDimensionThe wallResponse
Fuzzing1988Input spaceCan't enumerate all inputs to a programSample randomly from input space, observe what breaks
Property-based testing2000Scenario spaceCan't enumerate all combinations of valid inputs and statesSpecify invariants, let the framework generate cases automatically
Chaos engineering2011Interaction/failure spaceCan't enumerate all failure combinations in distributed systemsInject random failures in production, observe emergent behaviour

This is the critical setup for the AI transition: the pre-AI testing world was already evolving toward probabilistic approaches because deterministic enumeration couldn't scale. But every new paradigm still assumed that humans understood the system deeply enough to design the tests, interpret the results, and act on them.

What agents actually changed

  1. Decision: humans still decide what to build, but the specification now serves two audiences. The agent has no shared understanding, no history with the codebase, no tacit knowledge, which makes the specification simultaneously more critical (the agent works only from what it's given) and more insufficient.

  2. Implementation: agents produce code dramatically faster, but several things break.

    • Comprehension inversion: before, the implementer understood the code best; now the implementer has no persistent understanding. Nobody has the deep understanding the author traditionally had. Not more code, but code with less understanding per line.
    • Comprehensible artifact degrades: AI-generated code is syntactically fluent, but its intent derives from the prompt rather than from design reasoning. It may look idiomatic while being subtly wrong.
    • Shared understanding doesn't accumulate: a human builds a mental model over months; an agent starts roughly fresh each session.
  3. Testing: three functions are stressed differently.

    • Verdict is overwhelmed: more code to verify, the same or fewer humans to do it.
    • Learning is broken: the agent has no persistent mental model to update, so the loop from testing back to implementation is degraded and the same bug class can recur indefinitely.
    • Feedback to decision still works: humans still learn from testing, but the loop that improved implementation quality over time is gone.
    • AI-generated tests have a bootstrap problem: if AI generates both code and tests, they can share blind spots, passing precisely because they share a misunderstanding of the spec.
    • Failure modes are novel: human bugs sit in predictable places (off-by-one, missing edge cases), while AI-generated code "reads right" while implementing subtly wrong logic. Existing heuristics may miss that.

The comprehension inversion is empirically supported. The generation effect predicts that accepting agent-generated code removes the act that produces retention, and experiments confirm it. Bastani et al. (2025, PNAS) found students using GPT-4 performed 17% worse on assessments once access was removed. Shen & Tamkin (2026, Anthropic) found engineers who delegated code generation scored 50% on comprehension versus 67% for those who coded manually, while those who used AI for conceptual questions kept theirs. METR (Becker et al., 2025) found developers using AI took 19% longer on their own repos while believing they were 20% faster, a perception gap that is itself evidence the debt accumulates invisibly.

Business domain understanding is qualitatively degraded, both the "how" and the "why". The agent produces correct-looking implementations without building a mental model that persists across sessions, and it has no knowledge of this business domain, this risk profile, or this team's accepted tradeoffs. These are the two debts: comprehension debt (the "how") and intent debt (the "why").

The analysis identified eight factors that change when agents enter development: one quantitative, seven qualitative.

#FactorPre-AIWith agents
1Code volumen/aAccelerated (quantitative change)
2Comprehension per line of codeConstantDegraded: nobody deeply understands agent-written code
3Feedback loop (testing → implementation)WorksSeverely degraded: persistence gives information recall, not judgment updating
4Shared understandingFills specification gapsAbsent on the agent side, so gaps stay unfilled
5Test independenceHuman testers bring an independent perspectiveAI tests can share the code's blind spots
6Failure modesPredictable, heuristic-detectableNovel: plausible-looking but subtly wrong
7Business domain understandingHumans carry risk profile, product lifecycle, tradeoffs, consequences of failure, plus transferable heuristics (oracle concept)LLMs pattern-match broadly but lack consequence-grounded calibration, contextual exception recognition, and project-specific knowledge. Open research question
8Intent preservationHumans carry the "why": business rationale, design decisions, accepted tradeoffsEroding: agents don't retain intent, and the humans who decided rotate, forget, or leave. Manifests when decisions need making, not when things break

Row 1 is quantitative; rows 2-8 are qualitative. This is why "just scale testing-after" has a structural ceiling: it addresses row 1 alone.

One study points the other way. Borg et al. (2026), "Echoes of AI", ran a two-phase experiment with 151 professional developers (phase 1 added features with or without GitHub Copilot, phase 2 had new developers evolve that code without AI) and found no significant difference in comprehensibility or maintainability. Our analysis predicts a different result for agentic AI than for autocomplete. Mode of authorship: with autocomplete the human is still the author, reading and integrating each suggestion, so the generation effect still operates; in agentic mode the agent produces blocks autonomously and the human reviews at coarser grain or not at all. Statistical power: phase 2 had n=75 with power ~0.17 for small effects. Until a comparable study runs against agentic AI, the prediction stands as a prediction.

The economics move the wrong way

Given these two debts, what happens to the economics of defects? Pre-AI, Boehm's cost-of-change curve showed that fixing a defect grows exponentially more expensive from requirements through design, code, testing, and production, because the later you find a flaw, the more work products are built on top of it.

In the AI era, code generation and rewriting become cheap; an agent regenerates a function in seconds. That might suggest the curve flattens. It doesn't. The cost driver shifts from rework volume to comprehension debt:

ActivityPre-AIAI era
Writing codeExpensive (human time)Cheap (agent generates it)
Understanding codeFree (the author understands it)Expensive (nobody deeply understands agent-written code)
Rewriting codeExpensive (human time)Cheap (agent regenerates it)
Finding what to rewriteModerate (humans trace through code they understand)Very expensive (comprehension gap)

The coding section of the curve collapses, but "idea → specification" becomes MORE critical (wrong ideas amplified at machine speed) and "testing → production" steeper (diagnosis harder). Total defect cost may rise even as the cost of writing code falls.

Return to the two company types, now with agents in the loop. In proactive QA companies, prevention practices partially break. TDD still works if the human writes the tests and the agent implements, but that requires understanding the system, so the specification problem returns. Pair programming with an agent is fundamentally different: it doesn't push back on design and brings no independent domain knowledge. But these companies are better positioned, because their mindset is already "embed quality into the process", so they experience this as adapting their practices rather than as everything breaking.

Agents then move every term in both cost models the wrong way at once. Effective n increases, because agents multiply code output per team, so even a small ε starts to bite. ε itself increases, because coordination assumed humans could read each other's code. In reactive QC companies, where ε is already large, r(n) increases too: the humans doing rework understand the surrounding code less well, so the 1/(1-r(n)) denominator shrinks faster. The compounding is multiplicative: the superlinear costs accelerate.

The safety net now ratifies

Pre-AI, both debts accumulated whenever people quit, transferred, or forgot. The industry response evolved into layered compensation mechanisms:

  • PRDs, feature specs, roadmaps, OKRs, strategy decks captured the why at the business level: what problem the product solves, what tradeoffs were accepted, what success looks like.
  • Wiki pages, decision logs, meeting notes captured the why of operational decisions: who decided what, when, on what evidence.
  • Architecture Decision Records (ADRs) captured the why behind architectural choices, persisting intent across team changes.
  • Code documentation and inline comments captured the how alongside the code.
  • Automated tests captured behavioural intent executably: specifications that fail when the system stops doing what was intended.
  • Onboarding material and runbooks captured operational knowledge.

These mechanisms shared a critical property: they were human-authored. Each originated from someone who had done the work of deciding, building, or operating the system, so each was a product of human learning. When the code drifted, people caught it by comparing against these artifacts, and new joiners received the team's understanding first, the code second.

In the AI era, this property is reversed. When AI-generated code becomes the source from which artifacts are generated, each artifact derives from code surface rather than from human learning. If the implementation is wrong, the documentation describes the wrong behaviour as intentional, the tests pass on it, and the next iteration extends it. Readers cannot catch the drift, because artifacts and implementation now agree by construction. The mechanism does not merely fail; it ratifies the drift.

We name this the generative ratification loop:

AI implements wrong → AI-generated tests pass on the wrong → AI-generated docs describe the wrong as intentional → next iteration builds on the wrong as if it were correct.

Each step ratifies the previous step's errors because all steps share the same source. The signal that would catch the wrong implementation, a human noticing "this isn't what we wanted", is absent at every step.

The loop sits in a broader pattern: self-referential AI loops without an external corrective signal compound errors. The prominent instance is model collapse (Shumailov et al., 2024, Nature), where training LLMs on data generated by previous LLMs causes "irreversible defects in the resulting models, in which tails of the original content distribution disappear". Our loop propagates errors through artifact chains instead, but the meta-pattern holds: when AI feeds AI with no human-authored anchor of intent, quality degrades.

If the lifecycle bridge is to be rebuilt, the artifacts that span it cannot be derived from agent-generated code alone. They need an anchor of human-authored intent, and where that anchor lives is the design space the research question opens.

Everything shipping against this problem is appraisal

Three broad families of response exist:

  • Direction 1: improve inspection. Better automated review, smarter static analysis, richer tooling for reviewers. Evaluates the code after the fact.
  • Direction 2: improve the AI-generated test layer. Better test generation, better oracles, mutation testing. Evaluates behaviour after the fact.
  • Direction 3: change the construction process so the debts don't accumulate at all. Prevention rather than appraisal.

What follows assesses one representative product per appraisal Direction. Direction 3 is the subject of the proposal, and has no mature product to evaluate.

The Economics of Testing research frames testing as an investment within the Cost of Quality (CoQ) framework, which sorts quality costs into prevention, appraisal, internal failure, and external failure. These map onto our models:

CoQ CategoryMaps to
Prevention costsWhat proactive QA companies invest in to keep ε small
Appraisal costsTesting-after (Direction 1/2 activities)
Internal failure costsThe rework feedback loop (the r(n) factor)
External failure costsThe escape rate, what gets past testing-after

The economic insight: prevention costs less than appraisal, which costs less than failure.

For Direction 1, the representative product is Claude Code Review, from Anthropic (announcement, docs).

What it proposes: multi-agent automated PR review. When a pull request opens, specialised agents analyse the code simultaneously, each targeting a class of issue: logic errors, boundary conditions, API misuse, authentication flaws, project conventions. A verification step attempts to disprove each finding before it is posted as an inline comment. $15-25 per review.

What it gets right: parallelized analysis covers more issue classes at once than one human reviewer could, the falsification step reduces noise (the main failure mode of such tools), and project-file configuration lets teams encode some domain context.

What it does not address: can we reason about code quality without understanding the business domain, the risk profile, and the lifecycle stage? Only superficially. Automated review catches syntax errors, known vulnerability patterns, and generic logic issues, but not whether the code does the right thing, the most important dimension of quality. Code that passes every check and solves the wrong problem is zero-quality code.

Evolution path and its risks: one could strengthen this by feeding the reviewer richer context (bug trackers, PRDs, Slack threads, commit history), and if the product evolves that way, that's welcome. Two risks come with it, though. Multi-channel noise: the noise level of an individual message is uncharacterisable from the message alone, and separating signal from noise requires a risk register to say what matters. Testing theatre: a reviewer that "checked Jira, read the PRDs, and consulted Slack" looks comprehensive enough that nobody asks whether we are testing the right things at the right investment level. Either way, better-informed testing-after is still testing-after.

Effect on cost model: it does not reduce the linear term. It adds a per-PR cost that scales with code volume, so the faster agents generate code, the more reviews are needed. It does not affect ε, because coordination requires understanding rather than inspection, nor r(n), because the comprehension gap is unaddressed. At $25 per review and 20 PRs/day, a team pays ~$10K/month for inspection alone: the trust tax made visible.

Assessment: primarily Direction 1. Addresses row 1 and partially row 5. Does not address rows 2, 3, 4, 6, 7, or 8.

For Direction 2, the representative product is JiTTests, from Meta (Meta Engineering Blog).

What it proposes: replace maintained test suites with Just-in-Time Tests, LLM-generated per code change. New code lands → the system infers change intention → creates mutants (intentionally faulty code) → generates and runs tests to catch the faults → assessors filter true positives.

What it gets right: it eliminates the test maintenance scaling problem, so there is no superlinear maintenance cost as n grows, and per-change testing is adaptive, each test tailored to what actually changed. Mutation testing also partially sidesteps comprehension by asking "did this change break observable behaviour?".

But eliminating maintained suites also eliminates one of the few places domain knowledge is encoded in machine-readable form. A test that exists because the team learned users misuse a feature in a specific way is organisational memory crystallised into code. Generating tests per-change from scratch discards that knowledge along with the maintenance cost, and the tradeoff deserves explicit evaluation.

There is a sharper version of the same critique. JiTTests generates tests from the code change, so if the change is wrong, the tests pass on the wrong behaviour and ratify it as intentional. They cannot catch a drift from the team's intent, having access only to the code, and a behavioural-regression catch requires that the previous behaviour was correct.

Effect on cost model: it reduces the linear term (no per-team test maintenance) and may reduce ε slightly (no shared suite to conflict over). It does not affect r(n) or the rework multiplier, because the comprehension gap is unaddressed. Meta has published no cost figures for the inference involved.

Assessment: the strongest possible Direction 2 approach. Addresses row 1, partially rows 2, 5, 6, misses rows 3, 4, 7, 8. It makes "inspect after the fact" as good as it can be, which clarifies where the remaining problems lie: specification, learning, shared understanding, domain context.

Together, the two evaluations show that appraisal optimisation hits a structural ceiling. Both address row 1 and partially a few others, but miss the qualitative rows precisely because those rows are the debts. More appraisal improves throughput; none of it is prevention.

What any answer has to satisfy

The shape of any solution depends on the asymmetry between the two debts, which are not equally non-negotiable.

Intent is absolute. Without the "why", you cannot define success, assess the product, or decide what to build next. Lose the intent and you are not building a product; you are generating code. Intent debt manifests when decisions need making, and no amount of comprehension compensates then.

Comprehension is conditional. You need enough to supervise (catch when agents go wrong), intervene (debug when things break), and evolve (make architectural decisions). You may not need line-by-line understanding of generated code. Experience with AI-assisted pairing suggests teams can operate with partial comprehension and full intent.

Degraded comprehension is a cost of speed and risk: debugging is slower and regressions likelier, but decisions remain possible. Lost intent is a cost of direction: you do not know whether to keep, change, or rebuild. The first is recoverable, the second compounds.

The research question follows. Is there a configuration of people, process, and AI agents that can:

  • prevent intent debt from accumulating during construction
  • keep comprehension sufficient for supervision, intervention, and evolution
  • produce human-authored anchors for the lifecycle artifacts that compensate for knowledge loss (docs, ADRs, tests, onboarding)
  • keep those anchors coherent as the system is maintained, extended, and teams rotate

Only one family of response can satisfy all four: prevention at construction time. Appraisal operates after the debt has accumulated and cannot, by construction, prevent it, though it remains useful as a complementary layer.

One candidate, offered as a hypothesis

What follows is one Direction 3 hypothesis, not the only one. It addresses three of the four conditions through a single process change, collaborative building. The fourth, maintaining the anchors over turnover and extension, is partially addressed by session-captured inputs feeding artifact generation, but remains open.

Pre-AI, the three amigos model was a planning meeting: discuss requirements, then split up and build separately, with all the information loss sequential handoffs entail. AI changes this in two ways. The skill barrier drops: implementation no longer requires everyone to be a developer, so anyone with domain knowledge can participate, provided an engineer keeps the agent technically on track. And speed changes: with an agent, code emerges at a pace that keeps everyone engaged rather than idle while one person types.

The hypothesis: when two or more people collaborate at small iteration grain while delegating implementation to an agent, three things change at once.

  • Intent debt does not accumulate during construction. The people making the decisions are present while the thing is built. Intent is never delegated during the session, so it cannot be lost there. (Lifecycle persistence is a separate problem.)
  • Comprehension debt is structurally minimised. The engineer helps decide what to build, sees each small step emerge in the context of decisions just made, and is in dialogue with the agent. Their comprehension comes from participating in the decisions, not from reading code, consistent with Shen & Tamkin's (2026) finding that conceptual engagement preserves comprehension while delegation does not.
  • Lifecycle artifacts have a human-authored anchor. The session produces clean inputs of intent, risk, and design rationale. Artifacts generated from those inputs carry the why from the human side of the session rather than being reverse-engineered from AI-generated code.
  • Granularity is the load-bearing parameter. All three claims depend on the iteration grain staying small enough that the team decides step by step: what to build next, and whether each result is right. They are not writing the code, the agent does that; they are guiding it and judging at each step. An iteration is that small when it produces one coherent commit, a change a person can capture in a hand-written message covering the why and the how. That message is the record of both debts, and it is self-checking: when no one can write a coherent message, the change was too big or too tangled. Prompt the agent to plan and execute large autonomous blocks instead, and the mode reverts to delegation: intent is handed off in one shot, comprehension never builds, and the dynamics return to the solo-with-AI baseline.

The baseline is three roles plus the agent, each contributing what the others cannot.

Product person (or domain expert): carries the "why". Knows the business problem, the user needs, the tradeoffs the team has consciously accepted.

Engineer: participates in incremental construction rather than reviewing code afterwards (code review is a weak appraisal method, particularly at high change volume). Helps decide what to build, dialogues with the agent ("now handle this edge case", "why did you do it this way?"), and tracks whether the implementation is heading in the right direction.

QA engineer: contributes risk thinking before and during construction.

AI agent: writes the code from the team's collective guidance. A single agent by design: one stream of work is what three people can watch and steer together, and watching it together keeps intent shared.

What the ensemble has to hold, the why and the how, is not confined to the feature. It extends to how the change fits the wider system, which the diversity of the three roles brings out.

In this model the QA engineer is not testing. They are preventing, at two layers.

Strategic layer, at the framing of the work. Before or as the session opens, the QA engineer maps the work to the risk register and compliance landscape. What categories of risk does this feature touch (data handling, financial transactions, security boundaries, regulatory scope)? What compliance frameworks apply (GDPR, PCI-DSS, SOC 2)? What is the consequence-of-failure profile, what verification strategy fits it, and what quality bar suits the product's lifecycle stage? These questions shape what gets built.

Tactical layer, during construction.

  1. Risk thinking. "What if the user submits twice?" "This payment flow needs to handle concurrent requests". These become immediate instructions to the agent, not bug reports weeks later.
  2. Challenging assumptions. "The spec says X but users actually do Y". The QA engineer questions what the developer and product person take for granted.
  3. Edge cases as specifications. The tester's edge cases become building instructions: TDD driven by risk thinking rather than developer intuition.
  4. User behaviour knowledge. "Users misuse the export feature for regulatory reporting, not the convenience case you are building for". This shapes implementation in real time.

This is what "QA engineer" always meant. Quality assurance: assuring quality is built in. The industry drifted into using QA for the person who tests after dev is done, which is quality control. The model makes the name accurate again.

The session also has the side effect mob programming practitioners have long observed: each role's expertise becomes visible to the others by exposure. The product person develops risk intuitions, the engineer absorbs business context, the QA engineer picks up architectural reasoning. Over time each may develop a flatter T, deep in their primary specialism with usable competence in the others. That partly addresses the 5:1 ratio problem, since low-risk features may not need a dedicated QA engineer present, and makes the session's cost partly an investment in cross-skilling rather than overhead. Whether T-shaping actually appears is empirically testable.

What the session leaves behind

Collaborative building is prevention, and prevention does not eliminate appraisal: integration testing, system-level behaviour, and emergent properties at scale still need verification after the fact.

The key idea, still a hypothesis: the collaborative session produces exactly the inputs that make agent-generated tests good. During the session the team produces:

  • Business rules articulated by the product person. "Refunds over $500 require manager approval". Acceptance criteria in natural language, in the agent's context.
  • Edge cases raised by the QA engineer. "What happens when the payment gateway times out?" Test cases grounded in real risk thinking, not code patterns.
  • Risk assessments. "This payment flow is critical, test it exhaustively. The settings page is low-risk". This is the local risk calibration (layer 3) that population-level AI test generation cannot provide.
  • Design decisions and their rationale. "We chose eventual consistency because the latency budget is 50ms". Constraints tests should respect, not flag as bugs.

All of it sits in the agent's context, so it can generate tests grounded in actual intent, actual risk assessment, and actual edge cases from domain experts. Such tests differ qualitatively from tests generated from code alone: they have input-stage independence (the QA engineer's risk thinking is independent of the code, addressing row 5 at the input), they encode intent (row 8), and they reflect local risk calibration (row 7).

A residual asymmetry on row 5: the inputs are independent of the code, but the output is still LLM-generated, so if the same model family generates code and tests, model-level blindspots remain. Independence is restored in what gets tested, not in how the tests are written. Nor does this replace all appraisal, since integration, performance, and chaos testing operate above the feature.

The same logic generalises beyond tests, because AI generates downstream artifacts well from clean intent. The session's outputs can feed intent docs (PRD-like: the business problem, what success looks like, the tradeoffs accepted) that anchor the why; comprehension docs (ADR-like: the design decisions, what alternatives were rejected) that anchor the how; and onboarding material, since new joiners need both. All three docs are human-verified, being the anchors the third condition demands.

Generation runs in one direction, from the anchor to the artifacts. The intent and comprehension docs hold the team's reasoning, and the team owns and verifies them. Code, tests, and onboarding material are drafted by the agent from those docs and verified by the team in turn. The dividing line from the ratification loop is not whether a human or the agent does the typing; it is the source. Generated from a human-authored anchor, an artifact carries intent forward. Generated from another code-derived artifact, it ratifies what was there.

This matters specifically because we are QA. Quality begins long before code is written, and onboarding is itself a quality investment: a new joiner who learns the system correctly, including the why, makes better decisions later, even when their work is guiding agents.

The economics of three people building one thing

The objection to team-based building is that it is expensive to have several people working on one thing. The comparison should not be headcount on a feature. It should be total cost of delivery:

Solo developer + AICollaborative team + AI
Implementation speedFastComparable (AI still writes the code)
Intent debtAccumulates (solo developer may not fully understand the "why")Prevented at construction time (product person present)
Comprehension debtAccumulates (generation effect, delegation destroys retention)Minimised (engineer participates in incremental construction)
Test qualityAI-generated from code patterns (shared blind spots)AI-generated from team's intent, risks, and edge cases
ReworkHigh (wrong features, missed edge cases discovered late)Low (risks addressed during construction)
Diagnosis costHigh (nobody understands the code)Low (engineer participated in building, has context)

The model shifts effort from appraisal and failure (expensive) to prevention (cheaper). Whether total cost is lower needs measurement, but the economic structure favours prevention.

That argument is about dollar cost. It does not address the logistical cost of synchronous collaboration: timezone alignment, calendar overhead, the opportunity cost of pulling three people off other work. That constraint is real even when the rework math favours the collaborative model, and whether async variants exist (staged collaboration, pre-session capture followed by review, partial attendance) is an open question. The comparison also leaves out tokens and inference, deliberately: a full cost decomposition belongs with the economics of testing work.

What would show this is wrong

The hypothesis predicts that collaborative building reduces comprehension and intent debt relative to solo-with-AI development, by enough to offset the operational cost of multi-person sessions. Three families of outcome variables:

  • Comprehension debt proxies. Time to diagnose a fault in code produced under each condition. Correctness of fault localisation when team members debug their own team's code. Ability to predict behaviour on novel inputs without re-reading the code.
  • Intent debt proxies. Ability of team members, including those who joined later, to articulate why each major design choice was made. Alignment of system behaviour with stated business intent over time.
  • Total cost of delivery. Implementation time plus rework plus diagnosis plus maintenance, per feature or per sprint.

At minimum the arms are solo-developer-with-AI versus collaborative-team-with-AI, with a third arm (solo developer without AI) to separate team configuration from AI itself. Iteration grain should be manipulated within the collaborative arm to find where the model collapses into delegation.

Two things stay constant across the collaborative arm, or the experiment measures more than one variable at once. Artifact form: differing formats would confound the collaboration mechanism with the format, so fix a set drawn from common practice (conventional commits, an ADR template, a PRD-style template, the team's test framework). Team composition: require all three roles, defined by what is present rather than by title.

Feature-level measurement is tractable but underestimates lifecycle effects; sprint-level captures rework but not turnover; quarter-level captures both but is operationally hard. A staged programme beats a single-shot study.

The hypothesis would be falsified by any of the following:

  • Collaborative-with-AI shows no meaningful difference from solo-with-AI on the comprehension and intent proxies.
  • The difference exists but is too small to justify the operational cost of the sessions.
  • The gain at creation is real but does not survive into maintenance, meaning the proposal addresses creation and not lifecycle.

What stays open even if the hypothesis holds

  • The 5:1 ratio problem. One tester, five developers. The tester cannot join five concurrent sessions. Two possible mitigations: reorganise into fewer concurrent workstreams, each with a mini-team; or the tester's leverage rises because collaborative building produces less rework. Both need modelling.
  • What is the minimum viable team composition in production? The experiment fixes all three roles; production may not need them. With the product role defined by context and authority rather than title, many teams already run as developer plus QA with partial PM presence. Which degraded compositions preserve the result is a follow-up study.
  • Does this work at scale? The examples are small teams and single features. How does the model apply to platform-level work, cross-team coordination, or large architectural changes?
  • How to capture the session outputs? How much of the intent, risk, and rationale should be persisted, in what format, with what tooling, under what discipline? This determines whether the ratification loop can be broken at lifecycle scale. The direction is concrete: capture session outputs, generate artifacts from them, measure the lifecycle outcomes.
  • How to measure the difference? Metrics that specifically capture comprehension and intent debt are themselves an open research question.
  • What happens when the agent becomes more reliable? Does the engineer's role change? Does the minimum viable team shrink?
  • Does collaborative building substitute for agent learning? At the moment of creation, plausibly yes: the intent and comprehension that agent learning would provide are already present in the humans guiding the session. Across the lifecycle, a candidate mechanism exists but needs building and testing: session capture produces clean human-authored intent, from which AI generates the artifacts that carry the bridge across turnover. Lifecycle persistence is no longer "no mechanism exists" but "needs validation".

This paper advances a hypothesis. It has not been demonstrated at organizational scale. If your team runs the experiment described in "What would show this is wrong", we would like to hear what you found, whether it supports the proposal or not. Write to Torben Robertson, Head of Content, at trobertson@qase.io.

Read Part Two, Test management for AI coding agents, or download the full booklet.

Ship quality software faster with Qase