Rebuilding the development loop around agents

The situation

A B2B healthtech company: 60 engineers, an eleven-year-old Rails and React monolith, ~1.4M lines, and a board that had asked what the company's AI strategy was. The existing answer was 60 code-completion licences and no measurement of any kind.

The actual engineering problems, which is where we started instead:

• Median PR cycle time was 4.1 days, and review latency — PR opened to first substantive comment — was 19 hours. Nothing was slow because engineers typed slowly.
• Escaped defects clustered brutally: 61% of production incidents over two years originated in four modules, all with test coverage under 20%, all authored by people who had left.
• The billing module was the biggest of these. Nobody would change it. Work routed around it, which is how you end up with three places that calculate a proration.
• Onboarding to first meaningful PR averaged 5.5 weeks.

What got built

1. A review agent trained on their own incidents

Generic AI review comments on style and get muted within a fortnight. We mined two years of incident reports, postmortems and revert commits, and derived twenty-three concrete failure patterns this team specifically ships — N+1s introduced in serializers, missing tenant scoping on new queries (the one that matters most in multi-tenant healthcare), background jobs without idempotency keys, migrations that lock a large table, PHI reaching a log line.

The reviewer checks for those and stays silent otherwise. It posts at most three comments per PR, each one citing the specific past incident the pattern caused. Engineers argued with the design in week one and were converted by the citations: it is very hard to dismiss a comment that says "this is the pattern that caused INC-2024-11."

It has no merge rights and never will. It is a reviewer, not an approver.

2. Characterisation tests before touching the scary module

The billing module could not be safely refactored because nobody knew what it did. So the first agent work was not refactoring — it was writing characterisation tests: tests that assert current behaviour, correct or not, to create a safety net.

Method: instrument production for two weeks to capture real input/output pairs for the module's public surface (scrubbed of PHI at the boundary), generate tests from those pairs, and have an engineer review each generated test for whether the captured behaviour was actually intended. That review was the valuable part. It surfaced four long-standing bugs, one of which had been silently under-billing a class of customers for three years.

Coverage on the module went from 17% to 74% in five weeks. The team then did the refactor themselves, which was always the point.

3. Spec-to-PR, deliberately narrow

An agent that takes a well-formed ticket and produces a draft PR with tests, running in CI. We scoped it hard: it only runs on tickets tagged as belonging to a pattern that already exists in the codebase — a new endpoint on an existing resource shape, a new report on the existing reporting framework, a field addition through the standard stack. It does not do novel design and it does not touch the four high-incident modules at all.

Roughly 30% of the backlog qualified. Of the drafts it produced, 46% were merged after human review and edit; the rest were closed or rewritten, and closure reasons are captured to improve the templates. We report the 46% rather than the 30% because the honest number is the one that survives review.

4. Comprehension for humans

Grounded question-answering over the repo, the ADRs, the wiki and the full git history — the history being the part everyone forgets and the part that answers "why". "Why does this module do its own retry instead of using the shared client?" is answered by a 2019 commit message and a linked incident, not by a paraphrase of the code. This was the single most used tool of the four, by a wide margin, and it moved onboarding-to-first-PR from 5.5 weeks to 2.5.

5. The policy document

Their enterprise customers' security questionnaires had started asking about AI in the SDLC, and the company had no honest answer. We wrote one: which agents exist, what identity each runs as, what credentials each holds (all short-lived, all scoped, none with production data access), that generated code is marked in commit trailers, that no agent can approve or merge, what is logged, and how it is audited quarterly. Two of their customers asked for it verbatim.

What broke

Where it landed

Escaped defect rate is the number we care most about and it is also the one that needs longer than twelve weeks to claim honestly — the trend was down, the sample was not yet large enough to state as a result, and we said so in the final report rather than putting a number on a chart. That is the standard we hold ourselves to on every engagement, and it is why the tables on this page are shorter than a vendor's would be.