# Test Logistics AI Software on Your Own Loads Before You Buy It

> Give every vendor on your shortlist the same week of your own loads, keep the correct answers to yourself, and count the wrong fields each tool didn't flag. That count, and the minutes your team spends on what it did flag, decide whether the software pays.

By Vlad Shvets, Partner @ Qvery · Published 2026-10-29 · Updated 2026-10-29 · 12 min read
Canonical: https://qvery.ai/blog/logistics-ai-software-evaluation/

In a freight brokerage or 3PL of a few hundred people, a load comes with four or five documents: the customer's tender, the carrier's rate confirmation, the BOL, the POD, the carrier's invoice.

Each shipper and carrier sends them in its own layout, and whatever doesn't arrive by EDI, someone on your team reads and types into the TMS.

Software that reads those documents for you is now sold by [a long list of vendors](https://qvery.ai/blog/best-freight-broker-software/), each with a demo that works and an accuracy figure in the nineties. You have to pick one, or none, and the demo can't tell you which.

So I set the accuracy figures vendors in this category publish against what a single load needs to be right.

The short version: buy the tool that is wrong least often **without telling you**, measured on your own week, at a review cost below what keying costs you today, and let it write into your TMS one field at a time.

This is the comparison you run before you sign anything; [the two-week pilot that follows a contract](https://qvery.ai/blog/ai-pilot-plan/) has its own method. And this piece names no winner.

## Price Last Week's Paperwork Before You Compare Any Vendor

Count from records and a stopwatch, not from memory. Take last week's loads, the documents each one needed, and a timed sample of your team keying them: twenty loads across two people is enough to see the spread.

Note who does the keying and what a wrong field costs to fix once it's in the system.

Vendor examples put manual rate confirmation entry at about **12 minutes a load**.[^1][^2] Use that only until your own timing replaces it.

Run the case with that figure, every number an assumption except the wage. Say your team keys **4,000 loads a month** at 12 minutes each. That's **800 hours a month**. At the US median wage for cargo and freight agents, **$25.13 an hour**,[^3] it's **$20,104 a month** in wages, before benefits.

That's the ceiling. No tool can give back more than the keying costs, and every minute your team spends checking the tool's work comes out of it.

This is the kind of job we build AI agents for at Qvery: one stack of documents that repeats, a field list your team already knows, and a check against the TMS that never changes. Whoever builds it, the test below is the same.

## A Vendor's Accuracy Figure Averages Fields, but Your Team Fixes Whole Loads

One vendor describes a rate confirmation reader that pulls "all 30+ fields" from the document and calls **95% field extraction** "table stakes".[^2] That's a figure **per field**, and a load with one wrong field is a load someone has to touch.

If each field is right 95% of the time and the misses fall independently, a 30-field document comes out with every field right only about one time in five. At 99% per field, it's about three times in four.

**Share of documents with every field right, if misses fall independently**

| Right per field | 30 fields all right | 12 fields that matter all right |
|---|---:|---:|
| 95% | 21.5% | 54.0% |
| 98% | 54.5% | 78.5% |
| 99% | 74.0% | 88.6% |
| 99.5% | 86.0% | 94.2% |

The arithmetic is an illustration, not a forecast. Real errors bunch up on the bad scans and the odd layouts, so on your files the share of clean loads can land above or below it. Only **your files** say which way.

A tool's figure also moves between a test set and real mail.

One vendor published both for its quote-request reader: **98.8%** field accuracy on a 14-email test set, then **88.5%** on pickup, **80.8%** on drop and **89.4%** on weight across 104 real emails in its pilots.[^4] The page doesn't say why the numbers differ, and it's one vendor's own report.

**One Vendor's Own Figures: Its Test Set Against 104 Real Emails** (Field accuracy (%))

The vendor reports 98.8% field accuracy on its 14-email test set, and 80.8% to 89.4% on three fields across 104 real emails.

| Label | Value |
|---|---:|
| Test set (14 emails): key load fields | 98.8% |
| Real emails (104): weight | 89.4% |
| Real emails (104): pickup | 88.5% |
| Real emails (104): drop | 80.8% |

Field extraction accuracy that FasterQuotes reports for its own quote-request reader: across key load variables on a 14-email US test set, and by field on 104 real RFQ emails in its live pilots. The vendor gives no reason for the difference. Source: FasterQuotes: What to Ask an AI Automation Vendor in Logistics (2026)

So ask every vendor the same question: on our files, what share of loads comes out with **no wrong field** among the fields that change something? A per-field average is not an answer to it.

## Send Every Vendor the Same Files From Your Own Week, and Keep the Right Answers

The test pack is one set of files from your own work, sent unchanged to every vendor on the shortlist:

- **Loads from a normal week**, drawn at random, with every document each one needed: tenders in each customer's layout, rate confirmations from several carriers, BOLs and PODs including the phone photos nobody can read, carrier invoices with accessorials. A couple of hundred loads is a workable start, and [the pilot plan](https://qvery.ai/blog/ai-pilot-plan/) shows how the count sets what a clean result can prove.
- **Last quarter's hard cases**, the ones your team still remembers: a multi-stop, a rate revised after booking, a tender sent twice, an amendment buried in a reply chain, a cancellation.
- **An unseen layout**, at least one, such as a new shipper's tender or a carrier you started using this month.

The unseen layout tells you what kind of reader you're buying.

A tool built on templates learns one layout at a time, and Microsoft's own documentation for its template models says variation in a document's visual structure lowers accuracy and that each variation needs its own model; its other model type is built to read one kind of document across many layouts.[^5]

The new shipper's tender shows you which kind is reading your mail, whatever the brochure says. Before you pay for one, it's worth settling [which steps need a model at all](https://qvery.ai/blog/ai-agents-vs-rpa/).

**What Reddit says**

> Every carriers format was different, sometimes even between different dispatchers at the same company.
> ([r/FreightBrokers · Jun 2026](https://www.reddit.com/r/FreightBrokers/comments/1u9ah22/where_ai_is_actually_working_in_brokerage_vs/oshr14y/))

> Their needs are substantially different than others in the 'industry'. I put industry in quotes because it's really 15,000 micro industries standing on each others shoulders in a trench coat.
> ([r/FreightBrokers · Oct 2026](https://www.reddit.com/r/FreightBrokers/comments/1xbtl88/automation_and_ai_in_freight_broking_back_office/19rhpt6npcwn5/))

Then write down the right answers. For each load in the pack, take what your team keyed into the TMS after corrections, export it, and **check it against the documents**, because the TMS has mistakes in it too.

Where the TMS and the paper disagree, or the tender, the rate con and the amendment disagree, the person who owns that rule decides which wins, and you write the rule down once. No vendor ever sees your right answers.

Split the pack in two. The vendor gets one part to set up on: the fields you want, your TMS's field names, a few layouts to learn. The other part you keep back until the scoring run.

Keep back **whole layouts**, never half of one, so a tool that memorized a customer's tender can't score on that customer.

Every vendor gets **the same files** and the same one-page instructions, and no vendor adds, swaps, or drops one. Run it before anything is signed, in the vendor's own environment.

![A test pack of a normal week's loads, hard cases and an unseen layout splits into set-up and kept-back files; each vendor gets the same files, scored against answers only you hold.](https://qvery.ai/blog/logistics-ai-software-evaluation/illustration-the-pack.webp)

*No vendor sees the right answers, and none gets a file the others didn't.*

A vendor that won't run your files before a contract is asking you to run the test after you've paid.

Two tools reading the same files is the only way their disagreements show, and the disagreements are where your right answers earn their keep: a vendor's confidence about a disputed field tells you nothing, and the person who owns the rule at your company settles it.

## Count the Wrong Fields Each Tool Didn't Flag, Starting With the Rate and the Appointment

These tools mark the fields they're unsure of for a person to check, usually by a confidence score.

Microsoft's documentation defines a confidence of 0.95 as right **19 times in 20** and says confidence can decide whether a value is accepted automatically or sent for review.[^6]

Read the other way: if the score means what it says, a field accepted at 0.95 is wrong one time in twenty, and nobody looks at it.

![An AI tool reads a rate confirmation: one field right, one wrong and flagged for review, one wrong and unflagged, which goes straight into the TMS and is found later.](https://qvery.ai/blog/logistics-ai-software-evaluation/illustration-two-paths.webp)

*Every review screen shows the left column. The test counts the right one.*

So score each field of the files you kept back one of five ways: **right**, **right but flagged**, **wrong and flagged**, **wrong and not flagged**, or **missing**. A document the tool returned nothing for stays in the count as missing, which means a person keys it.

The wrong fields it didn't flag are the number that decides, because they reach the TMS, the carrier, or the invoice with nobody reading them first. What each one costs depends on the field:

**What an unflagged wrong field costs, by field**

| Field | If it's wrong and nobody sees it | Who finds out |
|---|---|---|
| Rate, fuel surcharge, accessorials | You pay the carrier or bill the customer the wrong amount | Billing, at the invoice or the dispute |
| Pickup and delivery dates and times | The truck misses the appointment | The driver, at the dock |
| Pickup and delivery addresses | The truck goes to the wrong place | The driver |
| Carrier name and MC number | The load is booked and paid to the wrong company | Whoever pays carriers |
| Customer reference or PO number | The invoice comes back unmatched and payment waits | The customer's payables team |
| Commodity, weight, equipment | The wrong truck, or the wrong paperwork at pickup | The driver or the shipper |

A wrong PO number delays a payment; a wrong rate or a wrong MC number costs money that may not come back.

The counting is a spreadsheet job, and a chat assistant such as Claude or ChatGPT can do it for you once each vendor sends back its output.

The prompt below takes one vendor's output for the files you kept back and your right answers for the same loads, makes both sides comparable, and labels every field.

**The scoring prompt**

```
You are checking an AI tool's output against the correct entries for the same freight loads. Two files are attached.
1. VENDOR.csv: one row per load and field, with the columns load_id, field, value, flagged (yes or no). A document the tool returned nothing for appears with an empty value.
2. KEY.csv: one row per load and field, with the columns load_id, field, correct_value.

First, put both files in the same form: dates as YYYY-MM-DD and times as 24-hour HH:MM; amounts as numbers with two decimals and no currency sign; addresses in capitals with street types abbreviated (ST, AVE, RD); MC numbers and reference numbers with spaces and dashes removed. A difference that disappears after this is not an error.

Then label every load_id and field in KEY.csv with exactly one of:
- RIGHT: matches, not flagged
- RIGHT BUT FLAGGED: matches, flagged yes
- WRONG FLAGGED: does not match, flagged yes
- WRONG NOT FLAGGED: does not match, flagged no
- MISSING: empty or absent

Return:
A. Every row that is not RIGHT: load_id, field, the tool's value, the correct value, the label.
B. A count of each label for each field.
C. The number of loads with no wrong or missing field, and the number of loads with at least one WRONG NOT FLAGGED on any of these fields: rate, fuel surcharge, accessorials, pickup date and time, delivery date and time, pickup address, delivery address, carrier name, MC number, customer reference or PO number.

Never guess a correct value. List any field that appears in VENDOR.csv but not in KEY.csv separately, unscored.
```

You get one table per vendor. **Check five loads** by hand against the documents before you trust the counts: where your labels and the assistant's differ, fix the normalizing rules and run it again.

Then compare the vendors on line C, the loads with an unflagged error on a field that costs money or a missed truck.

Which of your freight documents deserve an agent first is a separate decision, and the logistics playbook walks a load's paperwork from the rate request to the invoice.

Industry Playbook: https://qvery.ai/industry-playbooks/ai-agents-for-logistics/

## Review Minutes per Load Decide Whether the Purchase Pays

The other number is the time your team spends on what the tool flags.

Have the person who keys loads today work each vendor's review screen on the files you kept back, and time it: minutes per flagged load, false flags included, plus a fair estimate of what each unflagged error costs to find and fix later.

Two tools with the same pitch come apart on the same assumptions as before, 4,000 loads at 12 minutes of keying each:

**Two tools on the same 4,000 loads, every input an assumption**

| Per load and month | Tool A | Tool B |
|---|---:|---:|
| Loads flagged for review | 30% | 10% |
| Minutes per flagged load | 4 | 4 |
| Wrong fields not flagged, per load | 0.10 | 0.30 |
| Minutes to find and fix each one later | 25 | 25 |
| Minutes per load | 3.7 | 7.9 |
| Hours a month back, against 800 today | 553 | 273 |
| Wages a month back, at $25.13 an hour | $13,905 | $6,869 |

Tool B looks better in the demo: it asks for help a third as often. It costs more than **twice the minutes** per load, because the errors it doesn't flag come back later, and slower.

The case counts only minutes; a wrong rate or a missed appointment can cost money on top, which makes B worse, not better.

The other extreme fails too. A tool that flags every load never lets a wrong field through unchecked, and it saves you nothing.

Put the tool's own costs beside the hours: its fees, the set-up, mapping it to your TMS fields, the upkeep of the connection, and the weeks you run both ways. Those are your line items to fill in, and the hours back are the most they can be worth.

Hours back are capacity unless you'd otherwise hire for them, and the minutes in the table are a named person's minutes, so make sure that person has them.

The same count works on a tool built for you instead of bought: an agent built around your own documents faces the same files and the same numbers before anyone relies on it.

Have Us Build the Agent That Keys Your Loads, Tested on Your Own Files: https://calendly.com/qvery-vlad/30min

## Let a Field Skip Review Only if Every Error on It Was Flagged

A field may go into the TMS without a person only if the tool never got it **wrong without flagging** it on the files you kept back, and a mistake in it can be undone. Everything else arrives as a draft a person approves before the load goes live.

Zero misses in a small set still leaves room for misses. By the rule of three, no misses in 60 loads is consistent with a true rate as high as 3 in 60, **about 5%**.[^7]

So write the size of the set into the rule itself: no unflagged error on the rate in 200 loads, never just no unflagged error.

Put the tool's actions through the pack as well:

1. **Send the amendment thread** A customer's reply chain that changes the delivery date and then cancels. Does the tool change the booking, cancel it, or draft the change for a person?
2. **Check everything it sends** A rate confirmation carries the carrier's pay. If the tool emails documents, the shipper must never receive one.
3. **Give it a carrier's invoice** It can prepare the match against the rate con; [the check itself](https://qvery.ai/blog/audit-carrier-invoice-ai/) has a method of its own, and a person approves the payment.
4. **Send one tender twice** See whether you get a second load. Then correct a field by hand, send a later document for the same load, and see whether your correction survives.

A tool's confidence decides when it should stop and ask; what it may do alone is set by what a mistake would cost, the line we draw [for a forwarder's shared inbox](https://qvery.ai/blog/freight-forwarding-automation/) too. The pack gives you the numbers to draw it field by field.

## Write the Test Into the Contract, and Run It Again After Every Change

The winner's numbers on the files you kept back become the condition for going live, in writing. Then the contract covers what changes after.

> 📌 **What the contract carries** 
> - The same files run again after any change to the model, the software version, or the TMS connection, with notice in writing.
> - A second set the vendor never sees, refreshed from new weeks, because a vendor who knows the test can tune to it.
> - Which TMS fields it writes, and who repairs the connection when your TMS updates, at whose cost.
> - Where your documents are stored, whether they train anything, and how they're deleted: [the five answers to get in writing](https://qvery.ai/blog/why-ai-pilots-fail/) apply unchanged.
> - Your data and the field mappings you built leave with you.
> - A customer on your TMS you can call, ideally one shipping freight like yours.

This week, pull last week's loads and their TMS entries, check the entries against the paper, and mark the layouts you'll keep back. Then write the one page every vendor gets. If the count shows your hours sit somewhere other than keying, you've learned it before paying anyone to find out.

Pick the Freight Paperwork Your Team Hands to AI First: https://calendly.com/qvery-vlad/30min

## Frequently asked questions



### How do I test logistics AI software before buying it?

Build one test pack from a week of your own loads, plus last quarter's hard cases and at least one layout the vendor has never seen. Give every vendor the same files and instructions, keep part of the pack back for scoring, and compare their output with your own checked TMS entries.

### What accuracy should AI reach on BOLs or rate confirmations?

There is no safe single figure. Ask for the share of loads with no wrong field, on your files, and count the wrong fields the tool didn't flag on the rate, the appointment, the addresses and the carrier. A field goes into the TMS without review only if it had none.

### Can AI enter freight orders into our TMS automatically?

Field by field, once it has earned it on your files. Until then, have the tool draft the load and a person approve it. Test what it does with a tender sent twice and with a later document for a load someone already corrected.

### Who reviews AI exceptions in logistics?

The person who keys the loads today, working the vendor's review screen. Time their minutes per flagged load during the test: those minutes decide whether the tool pays.

### How many loads should the test pack hold?

Enough that a clean result means something. With no misses, 60 loads rule out an error rate above about 5% at 95% confidence; ruling out 1% takes about 300.

## Sources

[^1]: Laneproof, "Rate Confirmation Automation Software: What Brokers Need" (2026), https://www.laneproof.com/blog/rate-confirmation-automation-software
[^2]: Debales AI, "AI Agents for Freight Brokers: The Complete 2026 Guide" (2026), https://debales.ai/blog/ai-agents-for-freight-brokers-complete-2026-guide
[^3]: U.S. Bureau of Labor Statistics, "Occupational Employment and Wage Statistics: Cargo and Freight Agents (43-5011), median hourly wage, May 2025" (2026), https://data.bls.gov/timeseries/OEUN000000000000043501108
[^4]: FasterQuotes, "What to Ask an AI Automation Vendor in Logistics: 15 Essential Questions" (2026), https://fasterquotes.io/blog/what-to-ask-ai-automation-vendor-logistics
[^5]: Microsoft Learn, "Document Intelligence custom models" (2026), https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/train/custom-model
[^6]: Microsoft Learn, "Interpret and improve accuracy and confidence scores" (2026), https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept/accuracy-confidence
[^7]: JAMA (Hanley JA, Lippman-Hand A), "If nothing goes wrong, is everything all right? Interpreting zero numerators" (1983), https://doi.org/10.1001/jama.1983.03330370053031