Why AI Pilots Stall in Production (and the Four Controls That Fix It)

AI pilots rarely fail at the model. They fail at the four controls that make a model usable by an organisation: an evaluation suite that tells you whether a change helped, an access model that decides who and what the system may touch, a cost ceiling that keeps unit economics sane, and an owner who is accountable when quality drifts at 2am.

Everything else — the framework, the vector database, the prompt, the choice between three near-identical models — is downstream of those four. Teams that skip them do not usually get a dramatic failure. They get a slow, expensive stall: the pilot works in the room, nobody can prove it works in the business, and the budget quietly moves elsewhere.

The demo gap

A demo and a production system are not the same artefact at different stages of polish. They are different kinds of system, judged by different criteria.

The gap is not closed by a better model. It is closed by instrumenting the system so that the organisation can answer four questions: is it good, is it allowed, is it affordable, and who is responsible.

Control one: evaluation

Without an evaluation set, every change is a guess. A new prompt, a different chunk size, a cheaper model, an added tool — each one might improve the system or quietly break a case nobody tested. Teams without evaluation cannot tell the difference, so they stop changing anything, or they change everything and hope.

The fix is less elaborate than vendors suggest. Start with 50 to 100 real cases drawn from actual use, not invented examples. Include the awkward ones: the query with two possible answers, the document with a table, the request that should be refused, the input in Swahili mixed with English, the case where the correct response is "I don't know".

Then decide what "good" means for each category. For a retrieval system that might be whether the right passage appeared in the top five results. For an extraction task it might be field-level accuracy. For a support assistant it might be a rubric scored by a human or a second model, with a sample reviewed manually each week.

Two rules make evaluation useful rather than decorative:

• Run it in CI. An eval that lives in a notebook is run once. An eval that blocks a merge is run every time, which is the only way it changes behaviour.
• Track it over time. A single score tells you little. A trend line tells you whether the last three changes helped, and when a model provider's silent update moved your quality.

There is a second, cheaper form of evaluation that most teams skip: production sampling. Once a week, pull 30 real interactions and read them. Not scored, not aggregated — read. You will find failure modes your test set never imagined, usually within twenty minutes.

Control two: permissions

Data access is where AI projects meet reality. A pilot often runs on a copy of the database with generous permissions because it was faster. That copy becomes the reason the system cannot ship, or worse, the reason it ships with a data leak.

The questions to settle early are not technical trivia. They are organisational:

• Whose data is this, and on what basis are we processing it? Under Kenya's Data Protection Act 2019, personal data needs a lawful basis, a defined purpose and a retention period. "It was in the database" is not a basis.
• Does the model provider see the data, and where does it go? Cross-border transfer is a design decision with legal consequences. Some providers offer regional processing; some do not. Know which one you are using.
• Can a user retrieve something they should not see? Retrieval systems are especially exposed here. If the index contains HR documents and the assistant serves all staff, the assistant will eventually quote a salary. Access control has to be enforced at retrieval time, not by hoping the prompt behaves.
• What is logged, for how long, and who can read it? Logging is necessary for debugging and audit. It is also a second copy of sensitive data if left unmanaged.

The pattern that works is boring: per-request identity, least-privilege tool access, filtered retrieval, and an audit trail that records which user asked what and which sources were used. It is the same discipline as any other enterprise system, applied to a component that is unusually good at sounding confident about things it should not say.

This matters more as agents gain the ability to take actions. A model that can only read is a privacy problem. A model that can send an email, issue a refund or change a record is an operational risk with a permission model attached. The security community has spent the last year documenting exactly how quickly that goes wrong when tool access is granted casually; the protocol layer is standardising faster than the governance around it.

Control three: cost

AI features have a marginal cost that traditional software does not. A database query is approximately free at the margin. A model call is not. A retrieval pipeline that makes five model calls per request is five times the cost of one that makes one, and an agent that loops is unbounded until you bound it.

The failure mode is predictable. The pilot runs at 200 requests a day and costs almost nothing. It rolls out to 4,000 users, someone adds a retry, context grows, a second model is added for quality, and the monthly bill arrives as a surprise that nobody can attribute to a feature.

Three controls prevent this:

• Measure cost per unit of work, not per month. Cost per resolved ticket, per document processed, per active tenant. That is the number a finance director can reason about, and the one that tells you whether the feature has a business.
• Set hard ceilings. Token limits per request, iteration limits per agent, spend alerts per environment, and a circuit breaker that degrades gracefully rather than failing open. A system that can spend without limit will.
• Design for cheap paths first. Cache what is repeated, route simple requests to smaller models, trim context, constrain output length, batch where the workflow allows. Most teams can cut spend substantially before touching quality, because most of the spend is on avoidable tokens.

The uncomfortable question to ask early is whether the feature is cheaper than the human process it replaces. If a support assistant costs more per ticket than the ticket's margin, better evaluation will not save it.

Control four: ownership

Every production system needs someone who notices when it degrades. For conventional software, that person is usually obvious: the team that owns the service, the on-call rota, the dashboard. For AI, ownership often dissolves into a committee — the data team owns the pipeline, the product team owns the prompt, the platform team owns the deployment, and nobody owns the answer quality.

The symptoms are recognisable:

• Nobody can say what percentage of answers were correct last week.
• Model quality complaints arrive through informal channels, not alerts.
• A provider deprecates a model and the first person to notice is a customer.
• Prompt changes are made directly in production by whoever has access.
• There is no record of which model version produced which decision.

Ownership means four concrete things: a named person accountable for quality, a dashboard that shows quality, latency, error rate and cost together, a change process for prompts and models that is treated like any other deployment, and a documented fallback for when a provider is unavailable or a model is retired.

That last one is easy to postpone and expensive to improvise. Model deprecations are announced with notice, but the migration still takes weeks if your prompts, evals and cost model are tuned to one provider.

What "done" should mean

A pilot is ready to scale when you can answer these without a meeting:

1. What is the evaluation score, and how has it moved in the last month?
2. What happens when the model is wrong — who sees it, and what do they do?
3. Which users can access which data, and how is that enforced?
4. What does one successful task cost, and what is the monthly ceiling?
5. Which model and version is in production, and what is the fallback?
6. Who is paged when quality drops or the provider fails?
7. What is logged, for how long, and who can read it?

If three or more of those answers are "we'd need to check", the system is still a pilot regardless of how many users it has.

A pre-scale checklist

Print it, argue about it, and do not scale until every line has an owner.

• 50–100 real evaluation cases, versioned, running in CI.
• A weekly sample of production interactions reviewed by a human.
• Retrieval filtered by user permissions, not by prompt instruction.
• A data map: what is collected, why, where it is processed, and when it is deleted.
• Per-request identity and least-privilege tool access for any agent.
• Hard token, iteration and spend limits per environment.
• Cost per unit of work tracked on the same dashboard as quality.
• Prompt and model changes deployed through review, with a rollback path.
• A named owner and an escalation route.
• A documented fallback for provider outage and model retirement.

None of this is exotic. It is the ordinary discipline of running software, applied to a component that is probabilistic, externally hosted and unusually persuasive. The teams that stall are rarely the ones with the weakest models. They are the ones that treated the model as the project.

If you have a pilot that works in a demo and not in the business, we can review the architecture and tell you which of the four controls is missing first. That is usually a two-week assessment, not a rebuild.