How to run an AI pilot that actually reaches production

Almost every company we speak to has run an AI pilot. Many have run several. A vendor builds a chatbot over a weekend, the demo goes well, someone senior says “this is exciting”, and then nothing happens. Six months later the pilot is still a pilot.

This is not because the models are not good enough. For most business tasks they are. Pilots stall because they are designed to answer the wrong question. A demo answers “can AI do something impressive with our data?” The answer is almost always yes. A useful pilot answers a harder question: “will this system do this specific job, at this quality, inside our workflow, at a cost we can carry, safely enough that we will switch it on for real?”

If you design the pilot around that second question from the first day, it has a real chance of reaching production. Here is how.

Why pilots stall

When we look at pilots that never shipped, the same reasons come up.

• No agreed definition of success. The pilot “looked good”, but nobody wrote down what good meant, so nobody can say it passed.
• Clean data that is not your data. The pilot ran on a tidy sample. Production data has scanned pages, missing fields, Hinglish and the odd photo of a document taken on a moving bus.
• No path into the real systems. The demo sat in a separate window. Production means reading from and writing to your LOS, Tally, SAP or helpdesk, with permissions and an audit trail.
• The hard parts were left for later. Security review, data protection, human hand-off, monitoring and cost limits all take time. If they start after the pilot, they become the reason the project dies.
• Nobody owned it. The pilot belonged to an innovation team. The operations head whose team would use it was never asked.

Every one of these is a design problem. None is a model problem.

Pick a workflow, not a technology

Do not start with “we want to use generative AI”. Start with a piece of work that costs you time or money today and that you can describe in one sentence. For example: “Our claims team spends too long finding information in reimbursement files.” Or: “We make thousands of reminder calls a month that follow the same script.”

A good first workflow has four things:

1. Volume. It happens often enough that improving it matters.
2. A clear right answer. A person can look at the output and say whether it is correct.
3. Data you can get. You can export real examples without a six-month approval process.
4. An owner. A named person whose team does the work today and wants it done better.

Be honest about whether AI is the right tool at all. Some problems are solved faster by a rule, a better form or a shared spreadsheet. If that is the answer, take it.

Agree the metric and the evaluation set before you build

This is the single most important step, and the one most often skipped.

Before anyone writes code, agree three things in writing:

• The business metric. What changes for the business if this works? Time per file, share of calls completed without a person, days to close the month. One number, maybe two.
• The quality bar. How accurate must the output be, measured how? For document extraction it might be field-level accuracy. For a support agent it might be the share of answers a reviewer rates as correct and complete.
• The safety line. The thing that must never happen. A clinical answer to a patient. A call outside permitted hours. An incorrect payment match.

Then build an evaluation set: a few hundred real examples, with the correct answer for each, checked by the people who do the work. This is your exam paper. The pilot is scored against it, every change is scored against it, and in production it becomes the test you rerun every week.

An evaluation set feels like extra work. It is the work. Without it, every improvement is a guess and every debate is an opinion.

Use real data from day one

Run the pilot on the messiest data you have, not the cleanest. If one in five of your documents is a phone photo, one in five of your test documents should be too. If customers write in Hinglish, test in Hinglish.

This will make the first results look worse. That is the point. You want to find the hard cases in week two, not in the first week of production.

Real data also means real data protection. Agree from the start where the data will live, who can see it, and what is redacted before it reaches a model. In India that usually means deploying in your own cloud account in an Indian region, and making sure no data is used to train anyone’s model.

Build the boring parts early

The parts of an AI system that decide whether it ships are rarely the model. They are:

• Integration. Reading from and writing to your real systems, even if only read-only during the pilot.
• Human hand-off. What happens when the AI is not sure, and how a person picks up with full context.
• Logging. Every input, output and action recorded, with personal data redacted, so you can see what happened and why.
• Guardrails. Limits on what the system can say and do, tested against attempts to break them.
• Cost controls. Usage caps and alerts so a surge in volume does not become a surge in your bill.

Put a thin version of each into the pilot. They do not need to be finished, but they need to exist. Then the step from pilot to production is a matter of hardening, not starting again.

Decide the go or no-go rule in advance

Before the pilot starts, write down what happens at the end:

• Go: the metric and quality bar are met and the safety line was not crossed. We move to the production build, with the budget and timeline already outlined.
• Adjust: close but not there. We narrow the scope (one document type instead of five, one call type instead of three) and rerun for a fixed period.
• Stop: the gap is too wide. We write up what we learned and stop spending.

A pilot that can fail is a pilot worth running. “Stop” is a perfectly good outcome if it arrives in six weeks and costs a fixed amount, rather than drifting for six months.

Plan the handover and the running costs

Production is not the end. Models change, your data changes, your customers ask new questions. A system that was accurate in March can drift by September without anyone noticing.

So before you go live, decide:

• Who watches quality, how often, and against which evaluation set.
• Who fixes things when quality drops, and how fast.
• What it costs to run each month: model usage, hosting, monitoring and the people involved. Get a realistic estimate from real pilot volumes, not from a pricing page.
• Who trains your team to use it, and to know when not to trust it.

If none of this has an owner, the system will slowly stop being used, and you will be back where you started.

A one-page pilot checklist

Print this and bring it to your next AI meeting.

The short version

A pilot is not a demo. It is a small, fixed-price experiment with a written definition of success, real data and the hard parts already started. Design it that way and the question at the end is not “was that exciting?” but “do we switch it on?”

That is the way we run every engagement at Envisstack: Discover → Prove → Build → Run. If you have a pilot that has stalled, or one you want to get right the first time, we are happy to look at it with you.