Why your AI demo worked and your AI implementation didn't

The prototype was brilliant. The AI understood the task, produced exactly what you wanted and made everyone wonder why you hadn't automated it years ago. Then you tried putting it into the actual business.

I've seen versions of this happen throughout my career in technology. Someone builds a demonstration. It works beautifully.

Everyone gets excited. Then you try connecting it to the real business and discover the difficult part wasn't actually the clever bit. AI has made this particularly easy to experience.

You can build an impressive prototype incredibly quickly now. Give an AI a carefully chosen customer enquiry and it writes a great response. Give it a clean document and it extracts exactly the information you wanted.

Connect it to some sample data and suddenly it looks as though you've automated an entire process. Then Monday morning arrives. A real customer sends an enquiry containing three different questions.

The CRM record is out of date. The price list has changed. The customer's attachment won't open.

One of the systems is unavailable. The customer is an exception to the normal rule. And apparently Susan has been doing this particular process differently for the last six years because of something that happened in 2020.

Welcome to implementation.

The demo proves the AI can do something

That's useful. It just doesn't prove the business can use it reliably. Those are two different tests.

A demo asks: Can AI perform this task? Implementation asks:

Can AI perform this task repeatedly, using our information, inside our systems, with our exceptions, without creating more problems than it solves? The second question is considerably harder.

A demo gets the nice version of the problem

When we test an idea, we naturally start with something manageable. Here's a customer enquiry. Here's the product information.

Here's what the response should achieve. Go. The AI does brilliantly.

But real processes aren't normally that clean. The information might be incomplete. The customer may use terminology you weren't expecting.

Two internal sources may disagree. The request may fall between two categories. There may be something unusual about the account.

The normal response might be inappropriate. The person dealing with it knows all of this because they've been doing the job for eight years. The AI doesn't.

That's why "it worked in the demo" and "it works in the business" are very different milestones.

The first problem is usually context

AI needs to know enough about the situation to make a useful decision. A demo conveniently gives it that information. A production system has to find it.

Imagine you're building AI to help answer sales enquiries. To produce a genuinely useful response, it might need: the customer's message,

their previous correspondence, current products, current pricing,

availability, delivery information, the customer's existing account,

the salesperson responsible, and the company's rules about what can be offered. Where does all of that live?

One system? Probably not. Now we're no longer solving an AI problem.

We're solving a systems problem.

Then you discover the data isn't quite what everyone thought

This tends to happen quickly. The CRM is the official customer record. Except salespeople sometimes don't update it.

The product database contains current information. Except for those three products. The shared drive contains the latest documents.

Except half the team still uses copies saved locally. The website has the correct price. Except existing customers have different pricing.

Humans navigate this mess using experience. They know which source to trust. They remember the exceptions.

They ask somebody when something looks odd. When you introduce AI, all of those hidden workarounds become visible. The AI needs a way to know what the people already know.

Then come the exceptions

Most processes look simple when described at a high level. Customer asks for refund → check order → issue refund. Easy.

Then somebody who actually handles refunds explains: Unless it's over £200. Unless it's a subscription.

Unless the product has already been replaced. Unless the order came through a particular distributor. Unless the customer is outside the UK.

Unless it's more than 30 days old. Unless the account is flagged. Unless...

This doesn't mean the process can't be automated. It means you haven't understood the process until you've understood the exceptions.

"It depends" is one of the most important things you can hear

When I'm trying to understand a process and somebody says: It depends. I want to know what it depends on.

That's where the actual business logic often lives. What makes this case different from that one? Which information changes the decision?

Which exceptions can be turned into rules? Which require judgement? Which should simply go to a person?

This is much more useful than asking somebody to describe the happy path from beginning to end. The happy path is usually easy. Businesses spend a surprising amount of their time dealing with everything that didn't follow it.

The process on paper may not be the process people actually use

This isn't unusual. A company has an official process. Then people discover quicker or better ways of getting the job done.

Over time, those workarounds become the real process. Perhaps somebody exports a report because the system's built-in reporting isn't useful. Maybe staff message one particular person before approving something.

Maybe a spreadsheet exists purely because two systems don't communicate. Maybe someone manually checks something the written process doesn't mention. If you automate the documented process without understanding the real one, you may automate the wrong thing.

This is why I like talking to the people actually doing the work. They know where it breaks.

Integration suddenly matters

The AI demonstration probably involved one model and one piece of information. The real process might involve: a CRM,

email, accounting software, cloud storage,

a website, an internal database, a supplier portal,

and a spreadsheet. The AI now needs some way of moving through that environment. Sometimes that's an API.

Sometimes normal automation. Sometimes an AI agent. Increasingly, it may involve AI using the browser or computer interface itself.

Often it will be a combination. The clever model in the middle may actually be the easiest part.

Then permissions appear

Your prototype probably had access to exactly the information you gave it. Production is different. Can the AI read every customer record?

Can it see financial information? Can it access employee data? Can it edit the CRM?

Can it send email? Can it issue a refund? Can it publish something?

Can it delete anything? The moment AI moves from producing an answer to taking an action, these questions become part of the implementation. Capability tells you what the AI could do.

Permissions determine what it actually gets to do.

Reliability has a different meaning in a business

Suppose your prototype succeeds nine times out of ten. That sounds quite good. Now suppose the process runs 1,000 times a week.

That's potentially 100 cases requiring attention. Whether that's acceptable depends entirely on what the process is. If the AI is categorising internal research documents and the occasional one lands in the wrong category, perhaps that's manageable.

If it's issuing payments, probably not. This is why "the AI is 90% accurate" doesn't tell me very much on its own. I want to know:

90% accurate at what? What happens during the other 10%? Can we detect the failure?

What does it cost? Can we reverse it? Does a human need to check?

The acceptable reliability level belongs to the process, not the technology.

The failures aren't always obvious

This is particularly important with generative AI. Traditional software often fails loudly. An error message appears.

The process stops. AI can fail much more politely. It gives you an answer.

The answer looks sensible. The workflow continues. But one assumption was wrong.

That's potentially harder to deal with because the system may not realise it has failed. And neither may the next system in the chain.

One wrong answer can become five wrong actions

Imagine an AI agent incorrectly identifies a customer as eligible for a refund. It updates the CRM. Issues the refund.

Emails the customer. Notifies accounts. Changes the order status.

Updates a report. The original mistake wasn't necessarily enormous. But the workflow amplified it.

This is one of the big differences between AI that generates and AI that acts. As we give AI more ability to take action, we need to think about how errors propagate.

This is where checkpoints matter

You don't need a human reviewing everything. But you do need to decide where review has value. Perhaps the AI can:

read the enquiry, retrieve the customer information, check the relevant policy,

prepare the decision, draft the response, and then stop.

A person approves. The system completes the rest automatically. Once the process has run reliably for long enough, perhaps some categories no longer need approval.

You can increase autonomy gradually. That's very different from building the most autonomous version on day one and hoping it behaves.

Then there's the human bit

This gets overlooked surprisingly often. You can build an excellent AI system and still have a terrible implementation because nobody actually uses it. Perhaps the team doesn't trust it.

Perhaps they don't understand what it does. Perhaps it adds another system they have to check. Perhaps it saves the company time but makes one person's job more annoying.

Perhaps staff quietly continue using the old process. Perhaps nobody explained what happens when the AI is wrong. Perhaps the people whose work you're changing weren't involved in designing it.

Technology doesn't enter a business in a vacuum. It lands in somebody's working day.

If the AI creates another inbox, you've probably gone wrong

This is one of my favourite tests. Suppose you automate a process and the result is: Here's another dashboard somebody needs to check every morning.

Have we actually improved anything? Sometimes yes. Often we've simply moved the work.

The objective shouldn't be to insert AI into the process. It should be to remove unnecessary work from the process. That's a very different measure of success.

Saving time isn't enough if checking eats it all

Imagine AI produces a report in five minutes instead of the two hours it previously took somebody. Brilliant. Except someone now spends an hour and a half checking every figure because they don't trust it.

The theoretical saving is enormous. The real saving isn't. This is one reason AI productivity can be difficult to measure.

You need to count: the work removed, the checking added,

the mistakes corrected, the exceptions escalated, and any new administration the system creates.

The number that matters is the net improvement to the process.

Sometimes the AI is solving the wrong problem

This happens because AI is exciting. Someone sees an impressive capability and asks: Where can we use this?

I prefer turning that around. Start with: Where is the work getting stuck?

Where are people waiting? Copying? Chasing?

Checking? Searching? Re-entering information?

Writing the same thing repeatedly? Switching between systems? Fixing avoidable mistakes?

Then ask what would improve it. Sometimes the answer is AI. Sometimes automation.

Sometimes integration. Sometimes changing the process. Sometimes deleting the process entirely.

The technology should come second.

A boring implementation that works is better than an amazing demo

The best AI implementation may not look particularly impressive. Nobody watches it generate a beautiful presentation. There isn't a humanoid avatar.

It doesn't have a name. It quietly: reads an incoming document,

extracts six pieces of information, checks them against a system, updates a record,

flags the two cases it isn't sure about, and saves somebody 40 minutes every morning. That's a success.

Not because the AI looks intelligent. Because somebody no longer has to spend 40 minutes doing something that didn't require them.

Start smaller than the demo makes you want to

AI demos encourage ambition. If the system can apparently do the entire process, why not automate the entire process? Because the objective isn't to discover the maximum amount the AI can theoretically do.

It's to build something reliable enough that people actually want to use it. I'd start with a contained piece. Something repeated often enough to matter.

Something where success is measurable. Something with information you can access. Something where mistakes can be detected.

Something where the consequences of a mistake are manageable. Then run it. Watch what breaks.

Fix it. Expand. That sounds less revolutionary.

It's also how useful systems tend to get built.

Measure the before as well as the after

This is another mistake businesses make. They implement AI and then ask whether it helped. Helped compared with what?

Before changing the process, understand: How long does it currently take? How often does it happen?

Where does it normally fail? How many people touch it? How much rework is there?

How long are customers waiting? What does a mistake cost? Then you have something to compare.

Otherwise, "AI made this faster" can become little more than a feeling.

The real implementation checklist

Before moving an AI prototype into a real business process, I'd want answers to these questions: The process Do we understand what actually happens, including exceptions?

The information Does the AI have the right context, and do we know which sources to trust? The systems

Can it reliably interact with everything the process depends on? The permissions Can it access only what it needs and take only the actions we've authorised?

The failures What can go wrong, how will we notice, and what happens next? The humans

Where is judgement still valuable, and do the people using the process understand it? The measurement What are we trying to improve, and how will we know whether we did?

If those questions don't have answers, the problem probably isn't that you need a better AI model.

The clever bit is becoming easier

This may be one of the biggest changes AI creates for technology projects. Building the impressive prototype is getting extraordinarily fast. AI itself can help write the code.

Create interfaces. Connect services. Analyse data.

Test ideas. That lowers the cost of experimentation enormously. It's brilliant.

But it can also create the illusion that because the prototype took two days, the production system should take three. The last 20% is still full of: edge cases,

security, permissions, testing,

monitoring, integration, data,

failure handling, and humans. The prototype proves the idea.

Implementation proves the system.

AI doesn't remove the boring work of building good systems

It changes it. It accelerates parts of it. It gives us capabilities we didn't have before.

And it makes some things possible that previously wouldn't have been worth building. But the fundamentals haven't disappeared. You still need to understand the problem.

You still need reliable information. You still need sensible processes. You still need to know what happens when something fails.

And you still need the thing you're building to make someone's working day better. So if your AI demo was amazing and the implementation has become frustrating, that doesn't necessarily mean the AI failed. It may simply mean you've reached the part where the real work starts.

Where to go next

• Don't start with AI. Start with the work. How to find worthwhile AI opportunities by looking at where work actually gets stuck.
• AI is getting smarter. Your business data might be what holds it back. Why better AI makes your internal information, documents and systems more important.
• The important question isn't what AI can do. It's what you should let it do. A practical framework for deciding how much authority an AI workflow should have.
• AI agents aren't digital employees. Here's what they actually are. Where agents fit inside real business processes and where humans still matter.

Related: Don't start with AI. Start with the work..