Running an AI Feature in Production: Plan for the Model to Change

A feature that passed every test in March behaves differently in September. Nobody changed the code. The prompts are the same, the retrieval is the same, and the test suite that ran green in March still runs green. Something upstream moved, and the first reliable signal was a customer noticing.

This is the ordinary condition of running AI inside a product, not an exotic failure. Almost every organisation building on a foundation model is operating a dependency it does not control, cannot freeze indefinitely and did not specify. That is a workable position. It is only workable deliberately.

The risk supervisors expect to grow

The Bank of England and FCA survey of AI in UK financial services asked firms which risks they expected to increase most over the following three years. The answers were not public perception or model accuracy. They were third-party dependencies, model complexity, and embedded or “hidden” models — three descriptions of one underlying problem, which is that behaviour the firm relies on is produced somewhere the firm cannot see.

The same survey found 75% of firms already using AI, foundation models accounting for 17% of use cases, and only 34% of firms reporting a complete understanding of the AI they use.

The Stanford HAI 2026 AI Index Report recorded 362 documented AI incidents in 2025, up from 233 in 2024. Deployment is running ahead of the operational practice around it.

Six practices that keep a feature reliable

1. Pin the version, and know how long you can hold it

Calling a provider’s default endpoint means accepting whatever is behind it on any given morning. Naming a specific model version converts an invisible change into a scheduled one. The pin is not the whole answer, because versions are retired — so the number worth knowing is not just which version you are on, but how much notice the provider gives before it goes away, and how long your evaluation and re-approval cycle takes. If the second number is larger than the first, that gap is the actual risk, and it is a procurement problem rather than an engineering one.

2. Keep an evaluation set that represents your work

Public benchmarks describe general capability. They say nothing about whether a model still handles your document formats, your product names, your edge cases and your regulatory language. A useful evaluation set is drawn from real work, labelled by people who know what a correct answer looks like, frozen so results stay comparable over time, and weighted towards the cases that actually hurt — the ambiguous ones, the ones with missing information, the ones where the right response is to decline.

A hundred well-chosen cases beat ten thousand scraped ones. The cost is in the labelling, and it is paid once.

3. Treat a model upgrade as a release, not a notification

A new version arriving is not an event that happens to you; it is a change you choose to accept. Run the evaluation set against the candidate, compare against the pinned version, look at the differences case by case rather than only at the aggregate score, and decide. Aggregate parity routinely hides a category that has become materially worse — the average holds because something else improved.

The NIST AI Risk Management Framework describes this as ongoing activity across the lifecycle rather than a one-off qualification. The practical form is a gate somebody owns and can refuse to open.

4. Instrument the output, not only the uptime

Standard monitoring reports that the service responded. It does not report that the responses became less useful. Add signals that track substance: refusal and fallback rates, output length distribution, the proportion of responses that cite a source where one is required, human override rates, and the rate at which users abandon the flow. None of these is a quality measure by itself, but together they move before complaints do. A change in the distribution is worth investigating even when every individual response looks reasonable.

5. Keep a fallback that is not the same model

Resilience means the feature degrades rather than disappears. Depending on the workload that might be a second provider, a smaller self-hosted model, a deterministic rules path, a cached response, or an honest message that routes the user to a person. What matters is that the fallback is exercised on a schedule. An untested fallback is a belief, not a control, and it usually fails on the day it is first genuinely needed.

6. Put evaluation rights in the contract

Much of this is unavailable unless it was bought. Notice periods before a version is retired, the right to evaluate a candidate before it becomes the default, disclosure of material changes to model behaviour, data handling and retention terms, service continuity, and exit assistance all belong in the agreement. The Artificial Intelligence Playbook for the UK Government covers buying and implementing AI products alongside building them, and the procurement half is the one more often skipped. A good pilot price is poor value if the organisation cannot later test, operate or replace the service.

Who owns it

These practices fail quietly when nobody is accountable for them, because none is urgent on any particular day. ISO/IEC 42001, the management-system standard for AI, offers a frame worth borrowing without certifying against it: defined responsibilities, documented objectives, controlled change, and review over time.

Translated into a product team, that is four named owners. Someone owns the evaluation set and keeps it current. Someone owns the upgrade decision and can say no. Someone owns the live signals and looks at them on a cadence. Someone owns the vendor relationship and the contract terms. These can be four hats on two people. They cannot be nobody.

Where to start

If none of this exists yet, the first week is not an architecture exercise. Write down which models the product calls, at which versions, through which accounts. Pull fifty real cases from the last month and have someone label the correct output. Run them against what is live today and keep the result. That is a baseline, and it takes a week.

Everything else is easier once a baseline exists, because every later question — did this upgrade help, did that prompt change work, is the feature getting worse — becomes a comparison instead of an argument.

If you are putting an AI feature into a product and want the operating practice designed alongside it, that is product and platform work, and it is usually scoped as a delivery sprint.