Pilot vs POC: how to choose and run the right test

How to plan a proof of concept

A good POC answers one question with evidence, quickly and cheaply. Use this plan.

1. Write the hypothesis as one sentence. Example: "Our support documentation lets a retrieval-based assistant answer at least 80 of 100 sample questions correctly." The threshold is yours to set; agree it with the business owner.
2. Fix the time box. Two to six weeks is typical. If the question needs longer, it is probably two questions.
3. Define the test set before you build. Collect the 100 questions, the 20 invoices or the five integration flows in advance, so nobody tunes the test to the result.
4. Use safe data. Masked, synthetic or public data in an isolated cloud account. No production credentials.
5. Record cost and effort. Note what it took to make the POC work. This becomes the estimate for the real build.
6. End with a written decision. Go, no-go or "go if", with the open risks the pilot must test.

Typical POC success measures

• Functional: does it do the job on the test set (accuracy, completeness, error rate)?
• Technical: response time, throughput, compatibility with your identity provider, network and data formats.
• Effort: hours and skills needed, licences, cloud cost of the test environment.
• Economic, if you run a proof of value: projected cost per transaction against today's cost.

How to plan a pilot

A pilot is a small production service. Treat it like one: it needs an owner, support, monitoring and an exit plan.

1. Pick the pilot group. One team, site, region or customer segment. Include typical users, not only enthusiasts.
2. Set the duration from the business cycle. If month-end, payroll or seasonal peaks matter, the pilot must include one.
3. Make it production-grade where it matters. Real identity and access control, logging, backup and a support route. POC shortcuts do not carry over.
4. Measure a baseline first. Record today's handling time, error rate or cost so you can compare.
5. Agree exit criteria. For example: adoption above an agreed share of the group, no priority-1 incidents for four weeks, user satisfaction at or above today's level, support load the service desk can absorb.
6. Prepare the rollout or rollback. Know in advance what the next stage costs and who approves it, and how you switch back if the pilot fails.

Typical pilot success measures

• Adoption: share of the pilot group actively using the solution each week.
• Outcome: change in handling time, error rate, throughput or cost compared with the baseline.
• Operations: incidents, support tickets, alert noise, restore tests passed.
• People: user feedback, training time needed, workarounds observed.

Running AI pilots and POCs

AI projects add risks that a normal software test does not catch. Plan for them explicitly.

• Evaluate on your data, not the demo. Language models can give fluent but wrong answers. Build a test set from real questions and score the answers by hand.
• Test in a pilot what users actually do. Real users phrase things differently from the project team. Log prompts and answers (with appropriate data-protection controls) and review samples weekly.
• Use a risk framework. The NIST AI Risk Management Framework and its Generative AI Profile (NIST AI 600-1) give a structure for listing and measuring AI risks during testing.
• Check the EU AI Act early. If you are the provider of a high-risk AI system listed in Annex III, testing it in real-world conditions outside a regulatory sandbox falls under Article 60 of the AI Act. Conditions include a real-world testing plan submitted to and approved by the national market surveillance authority, informed consent from test subjects, and a duration of no more than six months, extendable once by six months with notice.
• Plan the path to production. Many AI POCs stall because nobody budgeted for data pipelines, monitoring and model updates. Our MLOps services cover that step.

Budget and timeline: how to size each test

There is no standard price for a POC or pilot; it depends on scope. Size each one against the decision it supports.

If you buy the solution from a supplier, write the POC or pilot into the procurement. The Cabinet Office guidance recommends testing the approach with the market and including any pilot in the service specification, because suppliers can advise on realistic timescales and commercial models.

Common mistakes and how to avoid them

• Endless POC. Scope grows, the end date moves and nobody decides. Fix: one hypothesis, a fixed end date and a decision meeting in the calendar on day one.
• POC code in production. Shortcuts taken for speed become security holes. Fix: plan a rebuild between POC and pilot.
• Hand-picked pilot users. Enthusiasts make any tool look good. Fix: include typical and sceptical users.
• No baseline. Without today's numbers you cannot show improvement. Fix: measure before the pilot starts.
• Pilot without support. Users hit problems and nobody answers. Fix: name a support route and owner, and monitor the pilot like a live service.
• Success with no next step. The pilot works, but there is no budget for rollout. Fix: get the rollout budget approved in principle before the pilot starts.

How Opsio helps

Opsio plans and runs POCs and pilots for AI, data and cloud projects on AWS and Azure. We set up isolated test accounts for POCs, production-grade landing zones for pilots, and the monitoring and support needed to run a pilot as a real service. Talk to our AI consulting team about scoping your next test, or read what a POC means in AI.

Frequently asked questions

Should I do a POC or a pilot first?

Do the POC first when the technology is unproven for your setup. Go straight to a pilot when the technology is already proven elsewhere in a similar environment and the remaining risk is adoption or operations.

Can a pilot replace a POC?

Only when technical feasibility is already proven. Otherwise a pilot exposes real users and data to a solution that may not work, which costs more to undo than a short POC.

How do you measure the success of a pilot?

Compare the pilot group against a baseline you measured before it started: adoption, handling time, error rate, cost per transaction, incidents and user feedback. Agree the thresholds before the pilot begins.

How long should an AI pilot run?

Long enough to cover a normal business cycle and enough real usage to judge answer quality, usually one to several months. Under the EU AI Act, real-world testing of a high-risk AI system by its provider is limited to six months, extendable once by six months.

What comes after a successful pilot?

A staged rollout: fix what the pilot found, update training and support, then expand group by group while you keep measuring against the same success criteria.

Related reading: What Is a Pilot Light Disaster Recovery Strategy.