Why AI Pilots Stall After a Good Demo, and How to Price the Job Before You Approve the Next One

If you run a company of a few hundred people and approved an AI pilot this year, you probably did it after a demo that worked. Months later the pilot is neither live nor stopped, and nobody can say whether the job got cheaper.

Meanwhile you keep hearing that 95% of AI pilots fail. The studies behind that number count different failures, and most of what they blame is something a demo was never built to show.

A demo proves a task can be done once, under conditions someone chose. A pilot has to answer a business question: is this job cheaper on our own work, and as accurate?

So price the job first, test the five things the demo held fixed on your own files, and give the pilot a date and a result that end it, either way.

None of this says whether any vendor's tool is good; a demo run on documents the vendor chose can't tell you that either way.

When we take a job like this on at Qvery, we start from the client's own files and from the person who handles the exceptions today, we agree the result in writing before we price it, and anything the AI agent would change in another system waits for a person's approval.

The rest of this piece is that discipline, written so you can hold any vendor to it, us included.

The Famous Failure Rates Count Different Failures, and None of Them Prices Your Pilot

Here is who each one counted, and what it counted as failure.

None of these studies counted companies of 100 to 500 people on their own. The smallest band in IDC's playbook for Lenovo was 250 to 999 employees, 338 of its 2,920 decision-makers, and MIT draws its large-company line at $100 million in annual revenue.³¹

The figures that follow projects toward production agree on a shape rather than a rate: plenty of starts, fewer finishes, and the drop between proof of concept and production. MIT's own chart shows where it happens.

Task-specific tools, custom or vendor-sold, fall from 20% of organizations piloting to 5% implementing.¹ That stall between the pilot and the work is this piece's subject, and it's the one failure you can see coming before you pay for it.

A Demo Shows the Task Done Once, Under Conditions Someone Chose

Ask the same studies why projects stop. IDC's respondents put data quality first and integration third.³ S&P Global's named privacy, security and cost.² Of the five root causes RAND heard from 65 practitioners, only one, problems too hard for AI, says the AI couldn't do the job.⁵

The rest is something a demo never has to face: your data, your systems, your people's time, and what the job is worth.

A demo answers whether this tool can do this task. A pilot has to answer a different question: will it do this task on our work, every day, inside our systems, with our people owning the misses, after the model changes?

A CIO in MIT's interviews had seen "dozens of demos this year" and found maybe one or two of them useful; the rest were "wrappers or science projects."¹

The running example is lease abstraction at a property management or commercial real estate firm.

The demo abstracts a handful of leases; your files hold amendments that change the base rent, side letters, scans of copies of copies, and abstracts that have to land in the lease administration system, where a missed renewal date costs money.

Put five questions to any demo before the pilot starts:

• Whose documents ran, and who chose them?
• How many times did each one run, and what was the output checked against?
• Who was watching, and what happened when something came out wrong?
• Which of your systems did it read from and write to, and where did your documents go?
• Which model ran, and what happens when it changes?

Some demos already answer one or two: a vendor who ran your leases three times against your own abstracts has answered the first two. Count what was answered, and treat the rest as open.

Price the Job Before the Pilot, Because Every Question the Demo Left Open Adds Hours

Start with the business. Every figure below is an assumption for the example; put your own in its place.

Say your firm abstracts 100 leases a quarter. One practitioner with more than twenty years in retail and office real estate describes a good abstract as hours of work, because the financial terms, the critical dates and the covenants have to be right.

Call it three hours: 300 hours a quarter, most of it one lease administrator's time.

The demo's version: the AI agent drafts each abstract with every field cited to its page and clause, and your administrator checks each draft in half an hour. That's 50 hours, a saving of 250 hours a quarter.

Multiply by what an hour of your administrator costs: that's the most the hours can be worth before the vendor's fees and the set-up, and it turns into cash only if those hours go to work you'd otherwise pay for.

Now add the work the demo didn't show.

The saving drops from 250 hours to 165, and each row answers one of the five questions. Which documents ran and how the output was checked are the 60 hours of re-reading; which systems, the 25 of re-keying; which model, the 15 of re-testing.

Who was watching decides whether the 35 hours of checking hold: practitioners in tax and legal work report checking that took longer than doing the work by hand.

One thing has no hours in it: what an error costs, such as a renewal option lost because an amendment moved the notice date. Write your own figure down before the pilot starts; it sets how many errors the pilot can accept.

The same case prices the stall: a pilot that runs two quarters without a decision has spent the vendor's fee and your administrator's hours checking outputs on the side, and the 165 to 250 hours a quarter are neither banked nor ruled out.

The same rows fit a carrier invoice checked against its rate confirmation, a vendor contract checked against your playbook, or a supplier invoice matched to its purchase order; only the documents and the systems change.

Your Hardest Files Decide the Price, So Let the Person Who Fixes Them Pick the Ones to Test

The biggest row in the case, re-reading the hard files, is invisible in a demo that ran on clean ones.

In commercial real estate, one buyer of lease abstracts describes old leases scanned so many times that text recognition gives up on them, and an appraiser was once handed an extension written on a signed notebook page, with no term, no rent, and no original lease.

Ask whoever fixes the exceptions today for last quarter's hardest leases: the amended ones, the scans, the side letters, the one they had to call the landlord about. Don't tell the vendor which are which, and don't let the vendor add or swap any.

That set shows how the tool fails on hard files. A count of last quarter's amended, scanned and side-letter leases shows how often, and the two together price the re-reading before you pay for a pilot.

If the vendor's answer is to leave the hard files out, price that before the pilot starts: it's the share of the job that stays with your people. Offered during the pilot as the fix for a failure, it's a reason to stop and re-price the job without them.

Here's what an AI agent for lease abstraction would have to do on those files, written so you can brief whoever builds it: a short version of mapping the workflow before you buy an AI agent, proposed for the pilot, with no product behind it.

1. 01

   What it reads

   Each lease with its amendments and side letters, in date order, so a later amendment overrides the base lease for the dates it covers.

2. 02

   What starts it

   A new or amended lease saved to the property's folder, plus a quarterly pass for critical dates in the next twelve months.

3. 03

   The rule it checks

   Your abstract template: every field filled from a cited page and clause, or left blank and flagged when it can't cite one.

4. 04

   What it produces

   A draft abstract with its citations, and a list of the fields it couldn't fill.

5. 05

   What a person approves

   Each draft before anything is written to the lease administration system, and every critical date one by one.

6. 06

   How it's counted

   Review minutes per abstract, fields corrected at approval, and critical dates caught or missed.

Score Every Run Against a Right Answer, and Run the Same Leases Again When the Model Changes

A demo usually shows a run that worked. A pilot starts with the right answers: last quarter's abstracts, the ones your administrator checked against their leases. Score every run against them field by field, and let the administrator settle any field where the old abstract is the one in error.

Then run each test lease three times and count the leases that are right every time. Agreement between runs isn't accuracy. A lease that comes out wrong the same way on every run looks consistent, and it's the worse failure, because nothing flags it.

Researchers set five models to their most repeatable settings (temperature zero, a fixed seed) and ran them ten times on eight tasks; accuracy moved by up to 15% from run to run.⁷ The chart shows GPT-4o's share of questions right on at least one run against right on all ten.⁷

A demo that re-runs a stumble until it comes out right is showing you the share right on at least one run. Your administrator gets one run of each lease, whichever comes out, so the share that matters is the one right on all ten: the questions that came out right on every run tested.

These are 2024 models on exam questions: read them as the reason to measure repeat runs on your own leases, never as a rate for any product.

How many leases is your call. By the rule of three, zero serious errors in N independent documents still leaves room for an error rate of about 3 in N, at 95% confidence: about 5% after 60 leases, about 3% after 100, for leases like the ones tested.⁸

Three runs of one lease still count as one document.

The model will also change under you. Researchers at Stanford and UC Berkeley put the same questions to GPT-4 in March and June 2023, and its accuracy at telling prime numbers from composite ones went from 84% to 51%.⁹

Vendors also retire models on their own schedules: OpenAI gives at least six months' notice for a generally available model and much shorter notice for previews, two weeks in its example; Anthropic gives at least 60 days.¹⁰¹¹

OpenAI has told developers that certain GPT-4o and GPT-4.1 snapshots leave its API on January 14, 2027, and GPT-4o is the model in the repeat-run chart above.¹⁰

I don't know how often a model change breaks a setup that worked, and I haven't seen anyone publish that number.

So keep the scored test leases and re-run them on every change, which is the 15 hours in the case. And put it in the contract: the model named, notice before it changes, and the right to re-run your test leases before a switch reaches your live work.

The Person Who Catches the Mistakes and the System It Writes To Belong Inside the Pilot

In the demo, the vendor's engineer caught each stumble. In production your lease administrator does, and the 35 and 60 hours in the case are their hours.

So the owner is whoever runs the process today, not a project manager and not the vendor. The pilot runs in their normal week, they log the minutes they spend checking and correcting, and their hours go into the plan before it starts. If nobody has those hours, the pilot isn't ready.

The owner who counts the minutes is also the person who says what good enough means.

The write into your systems is the other half. A demo shows an abstract on a screen; production needs it on the right record in the lease administration system. MIT's buyers put it bluntly: "If it doesn't plug into Salesforce or our internal systems, no one's going to use it."¹

The write is also where a miss reaches people: one leasing office describes a maintenance chatbot that left residents believing it had created work orders it never created.

So the pilot writes drafts into a copy of the system, or into the live one only after approval, and someone checks what landed against what was approved.

Ask before it starts who builds that connection and who pays for it; a pilot that skips it moves the 25 re-keying hours into production, where nobody counted them.

Get the Terms for Your Documents in Writing Before the First Upload

Before the first lease goes to a vendor, get five answers in writing: where documents are stored, how long they're kept, whether they're used to train any model, who else processes them (the model's provider included), and how they're deleted when you stop.

Privacy and security were the risks S&P Global's respondents named most, at 38% each.²

For a law firm it's a duty: ABA Formal Opinion 512 requires a client's informed consent, not boilerplate in an engagement letter, before client information goes into a self-learning AI tool that could disclose it.¹²

Financial institutions under the FTC Safeguards Rule must take reasonable steps to choose providers that can safeguard customer information, require it by contract, and assess them periodically, based on the risk they present.¹³

Read the Vendor's Proposal for What It Showed, What It Said, and What It Skipped

An assistant such as Claude or ChatGPT can do the first pass. The prompt below has it answer the five questions, plus one about your side, from the vendor's proposal and your demo notes only.

Each answer comes back quoted and sorted as shown, stated or not stated, with a draft pilot condition for everything not shown on your own documents. Remove client names, prices, and anything under NDA first, and use only an assistant your company has approved for work documents.

I am deciding whether to approve an AI pilot. Below are two documents: the vendor's proposal and my own notes from the demo. Client names, prices and anything under NDA have been removed.

Answer the six questions below from these two documents only, and sort each answer into one of three:
- SHOWN: my demo notes record it happening in front of us, with the documents named and a count (how many documents, how many runs, how many came out right).
- STATED: the vendor says it, in the proposal or out loud as my notes record, but my notes do not record it being shown.
If part of an answer was shown and part was not, sort the parts separately.
- NOT STATED: neither document says it.

1. Whose documents was the demo run on, and who chose them?
2. How many times was each document run, and what were the outputs checked against?
3. Who was watching during the demo, and what happened when an output was wrong?
4. Which of our systems does it read from and write to, was any of them connected in the demo, and where do our documents go (storage, how long it is kept, use for training, who else processes it, deletion when we stop)?
5. Which AI model does it run on, and what happens when that model is changed or retired?
6. Who on our side reviews its outputs once it is live, how many hours a week that takes, and who answers for an error?

First, list every claim in the proposal that my notes contradict or do not support, quoting the proposal, and my notes where they contradict it.

Then Table 1: Question | Answer in one sentence | SHOWN, STATED or NOT STATED | Exact words (the sentence from the proposal or my notes the answer rests on, quoted word for word; "none" if NOT STATED).

Table 2, one row for every question where my notes do not show it working on our own documents and systems: Condition the pilot must meet | How we test it on our own documents | Result that stops the pilot.

Rules: quote only words that appear in the two documents. Do not judge whether the product is good. Do not fill a gap with what vendors usually do; a gap stays NOT STATED. If the proposal and my notes disagree, say so and sort the answer by my notes.

You get the claims your notes contradict or don't support, a table with every answer sorted and quoted, and draft conditions; check the quotes against the two documents first.

We tried it on made-up lease abstraction proposals, on Claude and on an OpenAI model. Every quote could be found word for word in the two documents, and once the prompt asked for contradictions first, both assistants led with the proposal's "100% accuracy" against the notes' 26 of 30.

Treat its conditions as a first draft (in our test, one assistant suggested accepting a wrong output that came right on the second try): the hours, the cost of an error, and the result that stops the pilot have to come from you.

Write Each Question the Demo Left Open Into the Pilot as a Condition, With a Date to Stop

Each row below says what the pilot must do, what settles it, and the result that stops it, for the lease case.

Then measure the job the same way before and during the pilot: the business case from earlier, with your numbers in it.

Write all of it into the pilot's statement of work: the end date, acceptance tied to the conditions, what happens to your documents and the set-up if it stops, and the terms after the pilot. Run the pilot long enough to meet a normal month's exceptions, and no longer.

Don't approve one on a demo that ran on documents you didn't supply, and don't extend a pilot that missed its condition. Missing the condition is a result: as tested, the job doesn't do what the pilot was written to prove.

If your pilot has already stalled, put the five questions to it. Each one it can answer on your own documents, with a number, is done; each one it can't becomes a condition for a rescoped pilot with a date, or the reason to stop now.

Someone in IT at a law firm of about 120 users described its attorneys picking a demo apart and the firm deciding to wait another year. Waiting a year is also a decision.

This week, ask your lease administrator, or whoever does the job your pilot covers, for the ten hardest documents of last quarter and the answers they checked. Under the document terms you have in writing, send them to the vendor, ask for three runs of each with nobody watching, and score what comes back.

That's the first number the demo never gave you.