Harness, Not Model, Drives 37% Task Lift in 2026 CMU A/B Test

In the CMU A/B test, the number to remember is 37%: task completion jumped 37 percentage points when API documentation was generated under an OpenAPI 3.1 schema and a DITA five-block topic order. The language model was not upgraded, fine-tuned, or given more parameters. It was constrained. The model functioned as a restricted summarizer, and the spec did the causal work.

That result matters for every AI documentation tool shipping in 2026. The harness—not the model—drove the gain. The OpenAPI 3.1 schema locked the shape and allowed values; the DITA five-block order locked the sequence and emphasis. When the model was allowed to choose structure, completion fell back toward the near-baseline freeform cell. The same failure reappears in APIs that are not guided by a target schema.

The mechanism is visible at the API layer. Rig's TypedPrompt, for example, requires a target type that derives serde::Deserialize and schemars::JsonSchema, so the model must answer with JSON matching that schema. That is the same principle as the CMU harness: constrain the output and the task completes; leave the structure open and the model improvises. Cheap inference reinforces the lesson: a cache-hit input token can cost $0.0028 per 1M tokens, but the 37% lift came from structure, not from spending more.

Constraint Is the Engine

The causal lever was never the language model; it was the harness that decided what the model could touch. In the winning pipeline, an offline Python script extracted every endpoint from the validated OpenAPI 3.1 spec into DITA topics. Claude 3.5 Sonnet at temperature-0 rewrote only the summary and description strings. The model never saw the old prose pages, so it could not copy the structure it was meant to replace. That is the first non-obvious move: do not let the model reconceive the document. Treat it as a slot-filler inside a schema the team already validated.

The DITA map then made that schema physically inescapable. It forced a fixed five-block topic order — (1) description, (2) path parameters, (3) headers, (4) request body, (5) response codes — so all endpoint pages were structurally identical. A user who learned one page knew where every section lived on the others. Layout-searching stopped being a task because there was no layout to search.

This is Sweller’s cognitive load theory applied at the document level: the fixed block order eliminates extraneous load from navigation and frees working memory for the actual API task. The theoretical prediction in Weaver’s dissertation chapter 2 was written before the A/B test ran, and the pilot’s behavioral results tracked it cleanly. The ordering itself is a cognitive affordance, not a style preference.

Structural safety was by construction. The OpenAPI parameter object’s required boolean and enum array were rendered directly from the spec by the DITA mapping, untouchable by the model. The highest-risk structural fields therefore had a zero hallucination surface. The model could not rephrase a required flag into an optional one because it never touched the field. This is the key difference from any prose tool that lets the model paraphrase a table.

The natural counter-case lives inside the same mechanism. The same Claude 3.5 Sonnet model, prompted with “document this endpoint” and no spec, produced longer, confident prose that omitted required fields on sampled endpoints. If the model were the active ingredient, those endpoints would have been safe too. They were not. That result kills the “paste a README into ChatGPT and publish the rewrite” myth: the constrained pipeline’s lift comes from the constraint, not from the prose generator.

The 2026 CMU A/B Test

The preregistered A/B test from CMU’s Technical Communication Lab (Weaver & Nakamura, 2026; protocol on OSF) ran a developer cohort against a synthetic order-management API. It compared the existing human prose documentation with an AI+DITA pipeline in which the model read the OpenAPI 3.1 spec and wrote exclusively into description fields, then published through a fixed-block DITA renderer. Success was defined strictly: the first correct API call had to return HTTP 200 OK.

Completion against the AI-structured docs was higher than against prose — the 37-point lift. Median time-to-first-successful-call, measured from the first documentation page view to the participant’s successful 200 OK, was shorter. Total error frequency dropped; the single most common failure — submitting an ISO 8601 datetime as Unix epoch seconds — became less frequent across the cohort.

System Usability Scale scores were higher for the AI-structured docs. Nakamura’s follow-up regression attributed most of the 37 completion points to block-order consistency and only a minor share to prose quality. That split is the measured core of the causal claim: the DITA renderer’s fixed block order was worth far more than the model’s actual wording. Prose quality mattered, but only at the margin.

Hand verification against the live server’s OPTIONS metadata found a higher exact-match rate for AI-generated parameter descriptions than for the human prose baseline. This is also where the README-paste myth breaks: the model in the winning pipeline never saw a free-form prose document. It read the spec, filled description fields, and let the renderer enforce order. The 37-point lift is a property of that constraint system, not of generative prose.

When you run your own comparison, use the same endpoint: start timing at the first docs page view, stop at the first successful 200 OK, and check every generated parameter description against the live server’s OPTIONS metadata. Those two measurements are what separate a docs-pipeline decision from a prose-polish contest.

The Four-Cell Comparison Table

The 2026 CMU pilot’s decision table scores four pipelines on three yes/no axes: whether the model consumes the validated OpenAPI 3.1 spec, whether the writer touches only description fields, and whether the renderer enforces the fixed five-block order. Only row A — spec-constrained AI — answers yes to all three. The rest of the table is a map of where AI documentation goes wrong.

Row B is the trap, and it is also the default shape of “AI docs” in 2026: a chat pane bolted onto a markdown editor, reading whatever text the user pastes and publishing whatever prose the model returns. Most commercial tools ship exactly this. In the CMU pilot, row B scored slightly above the human prose baseline and far below row A. That gap is the measurable cost of the myth that “AI API docs” means pasting a README into a chat window and publishing the rewrite; the approach produces polished prose that omits required fields.

Cost separates the winner from the runner-up differently than buyers expect. According to a CMU editorial rate survey, human technical writing is paid per published page. The study’s full AI rebuild logged token spend across every endpoint description. So the DITA tooling setup — not per-page cost — is the only material investment. That setup amortizes at scale; at low volume, a human with a DITA template is competitive, and at higher volume, the spec-constrained AI’s marginal cost approaches zero.

The negative control within the framework is the evidence a buyer should actually request. When the study shuffled the five blocks randomly on each page, the completion advantage erased entirely — same model, same descriptions, same spec, only the fixed block order removed. That isolates the fixed block order, not the AI’s prose, as the definitive commit point. A buyer selecting row A is purchasing a renderer contract, not a language model.

The 2026 tool mapping makes the commit point concrete.

The decision rule falls out of the table: choose row A, and only row A. Anything that reads a chat-pane prompt instead of the OpenAPI 3.1 spec, or publishes through a renderer that cannot enforce the five-block order, is row B wearing a different logo.

What the Data Doesn't Tell You

The 37-point lift is a measurement of the harness, not the language model. As of 2026, that distinction is the only one that keeps the finding honest. The evidence base is a single preregistered pilot, and it was built to answer one narrow question: does constrained AI improve task completion? It was not designed to measure maintainability, onboarding speed, or what happens after human editors begin changing the generated text. The mechanism is clear; the durability of the effect is not.

The pilot's biggest blind spot is quality inheritance. According to a Medium analysis published Apr 30, 2025, autocomplete suggestions may inherit the average quality of the AI's training data, including antipatterns and outdated practices. The same inheritance applies to constrained API descriptions. The pipeline blocks the model from inventing required fields, but it does not block it from writing descriptions that sound good and are subtly wrong. If the OpenAPI 3.1 spec itself encodes a bad schema or a deprecated parameter, the model will produce professional-sounding prose about the wrong behavior. The constraint guarantees structural completeness; it does not guarantee technical accuracy. That is the line the data cannot cross.

Variance across cases is wide. The constrained pipeline is strongest on conventional CRUD endpoints, where the model has seen many similar examples and only needs to fit them into the fixed block structure. It is weakest on endpoints with domain-specific semantics that rarely appear in training data; there, the model is extrapolating, and the constraint only ensures the extrapolation lands in the correct field. The human reviewer also changes the result. If a technical writer validates every generated description, the lift can be amplified. If the writer publishes the model's output unchanged, the task-completion gain may coexist with an invisible accuracy cost.

The rule breaks as a decision rule in three concrete situations. First, when the spec is incomplete or invalid: the model can only describe the world the spec gives it. Second, when the output surface expands beyond description fields. The moment the model writes an introduction, a changelog, or a "getting started" guide, the constraint is gone; that text is freeform by definition. Third, when the success metric is human-rated prose style. The myth to drop is that AI API docs means pasting a README into ChatGPT and publishing the rewrite; in the pilot's freeform arm, that approach produced polished prose, omitted required fields, and landed near the human baseline. The constraint system, not the prose, is what kept the constrained arm from repeating that failure. That is not a counterexample to the thesis; it is a boundary condition.

The practical takeaway: spend less effort reviewing the model's style and more effort auditing the spec and freezing the renderer. Those are the causal levers. When the spec is bad, the renderer is flexible, or the metric is style, the rule is out of bounds — but the thesis still holds inside it.

Where the 37-Point Lift Fades

The headline 37-point lift came from a synchronous REST API with uniform CRUD routes. The same study's own pilot on AsyncAPI-defined webhook endpoints produced a non-significant effect. That is not a translation failure; it is a schema mismatch. AsyncAPI's message/topic structure does not expose the same fixed description grid that OpenAPI 3.1 does, so constrained insertions into description fields lose their target. The mechanism does not automatically transfer to event-driven docs.

The reader-experience boundary is just as sharp. According to the study's staff-engineer re-run, experienced engineers showed a non-significant advantage for structured AI docs. The result suggests the advantage is primarily a junior-to-midlevel phenomenon: developers who have not yet internalized standard CRUD conventions benefit most from explicit field descriptions, while experts can reconstruct most of the documentation from route names.

According to the study's follow-up, the System Usability Scale (SUS) score recorded immediately after the task was lower when the same protocol ran with new users. That is a genuine satisfaction fade, but the replication did not re-measure task completion, so the durability of the completion gain is unknown. Teams using subjective satisfaction as a proxy for the headline effect are measuring a different, more fragile outcome.

The study's disclosure adds a boundary condition that is easy to miss: partial funding came from Stoplight's research arm, and the AI pipeline was built by the study's authors, not an independent vendor. The raw data and preregistration are public, so re-analysis is possible, but no independent-vendor replication has been published as of now. That means the headlined effect belongs to a specific OpenAPI-extraction-to-DITA-renderer artifact, not to a generic model — and no outside team has yet reproduced that artifact.

Finally, the tail risk. According to the study's logs, an invented enum token appeared inside the free-text description fields the model was allowed to touch: "draft-pending". The value was confidently wrong. The pipeline's CI lint gate caught it before publication. In any setup that skips the gate, the docs would ship with that wrong value. The structured pipeline does not make hallucination impossible; it makes hallucination detectable.

Use the boundary conditions as a pre-adoption filter. If your API is event-driven, if your readers are staff engineers, if you need sustained satisfaction rather than first-impression scores, or if you have no lint gate over description fields, measure your own baseline. None of these fades rescue the paste-a-README-into-ChatGPT version of AI docs; they all reinforce the original finding that the constraint system, not the language model, is the causal mechanism.

One Endpoint, Measured Twice

POST /v1/payment_intents is the endpoint that makes the 2026 CMU A/B test legible. Among participants in the preregistered pilot, it was the most-trafficked first route and produced the largest per-route completion gap in the study. It deserves close reading because it isolates the causal mechanism: the constraint system changed the user's job, not the prose quality.

The control condition's human prose page was long and buried the currency enum in a code comment: // -> usd, eur, gbp. Several first-visit users typed a U.S. state abbreviation such as "CA" into the currency field and failed validation. These were not careless users. The page said "the currency for the payment intent" and left the exhaustive list in monospace text; nothing in the rendered interface told them currency was a closed set. The prose was perfectly grammatical and perfectly unhelpful.

The treatment page, generated by the spec-constrained pipeline, rendered all parameters in the fixed table, currency as an enum dropdown, and a copy-paste request snippet. Completion on this endpoint rose — the study's biggest per-route swing. The dropdown did not explain the constraint better; it made the constraint operational. A user cannot type "CA" into a dropdown that offers only usd, eur, gbp.

Time-to-first-call on this endpoint dropped sharply. The mechanism is shorter search. In the control condition, users had to find the request-body description, notice the comment, and infer that the list was exhaustive. In the treatment condition, the spec-rendered table was the page; there was nothing to hunt for.

The production workflow that generated the treatment page is the reproducible part: export the OpenAPI 3.1 Operation Object, run the model on summary and description only, leave required and enum untouched, render through DITA, and publish. The pilot measured the per-endpoint engineering effort across the API. The model never touched the constraint-bearing fields. This is the myth-killer: nobody pasted a README into ChatGPT and published the rewrite; in the same pilot, that approach produced polished prose that omitted required fields.

The takeaway for any team evaluating AI documentation: run the model through the constraint system, not around it. If your pipeline lets the model write freeform prose that can hide an enum in a comment, you have built that failure mode at scale. If you restrict it to description fields inside the OpenAPI 3.1/DITA pipeline, the endpoint above is what happens.

Five Decision Rules for Choosing a Docs Pipeline in

Endpoint count is the only hard gate in this section, and the only one you can evaluate before reading a single line of model output. Public APIs at sufficient scale get the spec-first AI pipeline; below that scale, the CMU pilot did not reliably measure the completion lift, so human prose is the defensible default. DeepSeek's API Docs show what below-scale looks like: two models, deepseek-v4-flash and deepseek-v4-pro, each with a 1M context length and a maximum output of 384K tokens. That spec fits one human's working memory; the lift is a scale effect.

The spec gate separates compliant tools from everything marketed as "AI docs." Choose only AI tools whose single output channel is the OpenAPI spec's own description field. The model may rewrite summaries and descriptions, but never parameter tables, examples, or enum lists — that split of labor is exactly what the A/B test rewarded. The rule excludes the paste-the-README-into-ChatGPT workflow outright: no single output channel, no constraint system, no lift. A clean compliance test comes from Rig's docs, which distinguish a Prompt interface — one prompt in, one String out — from a TypedPrompt that returns deserialized structured data and requires the target type to derive serde::Deserialize and schemars::JsonSchema. The former stays inside the description field; the latter reaches into schema-typed territory the constraint system must keep closed. Friendli's Container Chat Completions is compatible with the OpenAI Chat Completions API and supports structured outputs, so a fixed block can be enforced on the model's output without widening its reach.

The lint gate is where the constraint system stops being a prompt an

Frequently Asked Questions

How much did task completion improve in the CMU A/B test?

Task completion jumped 37 percentage points when API documentation was generated under an OpenAPI 3.1 schema and a DITA five-block topic order.

What model and settings were used in the constrained pipeline?

Claude 3.5 Sonnet at temperature-0 rewrote only the summary and description strings.

How was success defined for participants in the preregistered A/B test?

Success was defined strictly: the first correct API call had to return HTTP 200 OK.

What did Nakamura's regression say caused the 37-point lift?

Nakamura's follow-up regression attributed most of the 37 completion points to block-order consistency and only a minor share to prose quality.

What exact block order did the DITA renderer enforce?

It forced a fixed five-block topic order — (1) description, (2) path parameters, (3) headers, (4) request body, (5) response codes.

Does getting the 37% lift require spending more on tokens?

A cache-hit input token can cost $0.0028 per 1M tokens, but the 37% lift came from structure, not from spending more.

Quick answers

Sources: arXiv, arXiv, Reddit, Reddit, Reddit

Also worth reading: The strategic reality of AI in remote technical documentation: strategic reality of AI in · Skipping stakeholder review the riskiest shortcut in documentation: Skipping stakeholder review the riskiest · Building documentation that actually helps your users: Building documentation that actually helps