# Why Call Center Quality Scores Disagree

> A call center quality score adds up two kinds of line: script lines a recording can settle and judgment calls a person weighs. Script lines move when the client's script changes; judgment calls move with whoever listens. Split them, and AI agents can check the script lines on every scored call.

By Vlad Shvets, Partner @ Qvery · Published 2026-11-20 · 14 min read
Canonical: https://qvery.ai/blog/why-call-qa-scores-disagree/

A contact-center outsourcer's client programs usually come with a monthly quality score, and the client reads it as proof that the calls went well.

Your quality assurance (QA) analysts build it by scoring a handful of calls per rep on the client's scorecard, the checklist of what a rep must say and do. The average goes into the monthly review.

That score also moves for reasons that have nothing to do with the call.

- The same greeting passed one month and lost points the next.
- The client changed its script, and the evaluation form still scores the old lines.
- Two analysts score one calibration call and land on different totals.
- The quality score stayed high while the client's complaints rose.
- One scored call, zeroed for a barking dog, dragged down a rep's whole month.

Scoring the calls costs money before anyone argues about the result. Take a program of 100 reps scored on 12 calls each a month, which is what one state's call-center contract asks for: 1,200 evaluations.

At 15 minutes each, the low end of one QA vendor's estimate for a five-minute call, your analysts spend **300 hours a month** scoring. That is about **$171,828 a year** at $47.73 an hour with benefits, before calibration sessions, where analysts score the same calls to line up their scoring, and disputes.

> At Qvery, we help outsourcers spend less defending quality scores, with custom AI agents that check every scored call against the client's current script and quote each miss.

Your analysts keep every line that needs a person: whether the rep owned the problem, and whether the caller left helped.

## A Quality Score Adds Up Two Kinds of Line

An analyst scoring a call works down the client's form one line at a time. Some lines are settled by the recording: the sentence was said or it was not, and it came before or after the step it has to precede. Others need a person to weigh how the call went.

**One scorecard, its lines sorted into the two kinds**

| Script lines: the recording settles them | Judgment calls: a person weighs them |
|---|---|
| The recording notice was read | The rep showed empathy |
| The required disclosure, word for word | The rep owned the problem |
| The notice came before the identity questions | The issue was resolved |
| No banned phrase, such as a promised refund | The call took no more effort than it needed |
| This month's closing line | The rep's tone fit the caller |

Both columns are scored by the same person, on the same form, into one number. When that number moves, the total cannot tell you which column moved it, and the two columns move for different reasons.

## Script Lines Move When the Rule Moves

A script line should be the steadiest thing on the form. Reps describe the opposite: the same greeting scored 100 one month and lost points the next, because the reviewer wanted “I am happy to assist you” in it.

Another rep lost points for “What else can I help you with today?” in place of “Is there anything else I can help you with?”

Behind most of these swings sits one unanswered question: does this line need the exact words, or only the meaning?

Some lines on a client's script must be read word for word, others can be said naturally, and reps and reviewers often disagree about which is which. In one office the rule on verifying a caller's details changed back and forth between “verify everything” and “verify only when necessary.”

Script changes make it worse. When a client rewrites its script, the evaluation form has to follow, and QA analysts say themselves that it often lags: quality was not told about the change at once, or the form has not been updated yet. Reps then lose points for lines the new script dropped.

**What Reddit says**

> Last month I received a 100 on my graded call ... How does anyone know what they are doing wrong if the QA agents are inconsistent?
> ([r/CallCenterWorkers · May 2024](https://www.reddit.com/r/CallCenterWorkers/comments/1d0tep1/q_a_is_conflicting/))

> That doesn't warrant an error or point deduction in an evaluation unless the script or your policy and procedure were changed.
> ([r/CallCenterWorkers · May 2024](https://www.reddit.com/r/CallCenterWorkers/comments/1d0tep1/comment/l5pqm09/))

> Sometimes changes happen to scripting or a specific process and QA isn't notified right away, or they've been told about the change but the corresponding changes to the evaluation forms haven't been made yet.
> ([r/callcentres · May 2025](https://www.reddit.com/r/callcentres/comments/1kh47ge/comment/mr42k8l/))

What these swings share is a missing record: which version of which client's words applied on the day of the call, and whether each line needs the exact words.

A full-time reviewer gets through 20 to 30 calls a day, by one QA vendor's count, often across several client programs. Nobody holds every client's latest wording in their head at that pace.

Calibration does not fix it either. Getting two analysts to agree on a script line scored against last month's script is like holding a meeting to agree on what a receipt says. Someone should read the receipt, and for a script line the receipt is the recording and the dated script.

## Judgment Calls Move With Whoever Listens

The judgment calls are where analysts differ most. Two good analysts can hear the same call and disagree about whether the rep owned the problem. That is what calibration is for.

Most operations run it, by COPC's survey of more than 900 contact-center executives.

- **89%** have a calibration process for the people who score calls
- **86%** call their calibration process effective

Source: COPC Inc.: Global Benchmarking Series 2022, Contact Center Quality Assurance (2022), 900+ executives surveyed in 2021

The same survey published no figure for how often two scorers agree.

The sample makes each disagreement count. SQM Group, which sells QA software, reports that most reps are scored on **4 to 10 calls** a month; the state contract above asks for 12. With that few, one call carries the month.

Reps describe a dog barking on a call that was otherwise perfect and earned a zero, and a month that hangs on two scores of 60 and 55. On stretched teams, reviewers describe playing calls at one and a half to three times normal speed to keep up.

> 💡 **Where calibration time pays** On the judgment calls, where good analysts weigh the same call differently. On script lines it is wasted, because a recording and a dated list already hold the answer.

## A Green Score Can Sit Over Unhappy Customers

The same scorecard can pass a call that left the customer unhappy, and the companies that sell call scoring report it themselves.

- **81%** of reps' QA scores do not track their customer satisfaction (CSAT) or first-contact resolution, by SQM Group's own research

Source: SQM Group, a QA software and survey vendor: Call Center Quality Assurance, Automate It With AI and Post-call Surveys (2026)

Only **19%** of managers in SQM's research strongly agree that their QA program helps CSAT, and in COPC's survey **22%** of executives said they do not analyze how their QA data relates to customer satisfaction.

Part of the reason is the weighting. Script lines are easy to tick, and reps describe forms that dock heavily for a missed scripted line while barely rewarding a fixed problem. Reps say it plainly: you can follow every step in the script and still leave the customer with the same problem.

A survey answer is noisy too. The trouble for an outsourcer is that the client reads both numbers and asks which one is true, and the outsourcer is the one who signed the quality score.

The state contract makes this concrete: the average of the scored calls is the month's service level score, and the state's own staff may run their own independent quality reviews of the same work.

> When a client asks why your quality score and its complaints disagree, the answer has to come from the recording, not from whoever scored the call. Split the score before that conversation happens. Have the lines a recording settles checked against the script that applied that day, and put your analysts' calibration on the judgment calls. Then a score that moves has a reason you can show.
>
> Vlad Shvets, Partner @ Qvery

## How Qvery's AI Agents Check the Script Lines

We build these agents around your client programs, connected to the contact center software you already run. Each finished recording passes to the agents, sorted by client program, and from there they work on their own while reps keep taking calls as they do now.

The agents write the call out with times and with who is speaking, then check it against that client's list in the version in force on the day of the call. That covers the recording notice, each required disclosure, the closing, the banned phrases, and the order wherever the list sets one.

They read the client's change notices too, so a call on the 14th is checked against the script that applied on the 14th.

Each line comes back matched, different, or not found, with the minute and the rep's own words. A different line says what changed: a price, a date, or a clause left out. A line counts only when the rep says it, never when the customer repeats it.

![Scored calls and the client’s dated list go to Qvery’s AI agents, which mark each line matched, different or not found with a quote; your analyst replays each miss and scores the judgment calls](https://qvery.ai/blog/why-call-qa-scores-disagree/illustration-scored-call-lines.webp)

*The agents settle the lines a recording can settle, against the script in force on the day of the call; your analyst replays every miss and keeps every judgment call.*

Your analyst opens the call with the script lines already marked and spends the minutes on the judgment calls. A “not found” is a question, never a fail: crosstalk or a bad line can hide a sentence, so an analyst replays the call before the line counts against anyone.

Most floors already run some speech analytics, software that searches recordings for words and phrases, and 79% of the executives in COPC's survey said they use it in QA. If yours already catches missing phrases well, keep it.

What is usually missing is the client's list as a dated record: which wording applied on which day, which lines need the exact words and which only the meaning, and a quote with the minute your quality lead can put in front of a client.

You can see it on your own work: send one recorded call and the statements your client requires, and the agents send back each line marked, timed and quoted.

Free report: outsourcing/call-compliance-audit

On the example from the top of this page, your analysts score 1,200 calls a month at 15 minutes each. With the script lines marked, an analyst still hears the call once through and scores the judgment calls; we assume that takes 9 minutes. That hands back **1,440 hours** a year.

The agents run for about $400 a month at that volume, an estimate: speech-to-text and checking at about 5 cents a call, plus hosting and upkeep of each client's list.

After that running cost, the year comes to **$63,931 saved**. Each scored call costs $7.49 instead of $11.93, and every $10,000 of the build is earned back in 1.9 months.

Type your own program's numbers over the example: the calls your analysts score in a month, the minutes each takes, and what an analyst's hour costs.

(An interactive calculator on the page: roi-call-quality-scoring.)

Those hours are worth most where you would otherwise hire: a new client program that needs QA coverage, or a client asking for a bigger sample. Checking every recorded call, not only the scored ones, costs cents a call and is priced on your volume.

Call scoring is also one of three jobs the Outsourcing Playbook prices for an outsourcer of about 200 people, beside coaching built from scratch and the monthly client scorecards rebuilt by hand.

Industry Playbook: https://qvery.ai/industry-playbooks/ai-agents-for-outsourcing/

## What the Agents Never Decide

A finding describes a line in a call, never the person who took it. The agents never score a rep, never pass or fail a call, and never treat “not found” as “not said” until an analyst has replayed it.

Nothing goes to your client unless your quality lead sends it, and the evidence file records who replayed each difference and what they concluded.

Recordings carry your clients' customers, so the data processing agreement names everyone who can hear them, along with the transcription service and the model provider and where each runs. You review that list before signing.

It stays short enough for a client's security team to read, which matters because many client contracts limit subcontractors. If your recorder pauses while customers pay, no card number reaches the agents; if it does not, long strings of digits are masked in the transcript. Qvery AI OÜ is an EU company and signs that agreement.

> The list is the part that takes judgment to build. For every line we mark whether the client wants the exact words or only the meaning, and the date each version took effect. If we get a line wrong, the agents get it wrong the same way on every call, and that shows up in week two, when we set their results beside your analysts' scores on calls they already scored.
>
> Piotr Babel, Partner @ Qvery

During setup we measure how often the agents' findings agree with your analysts on the same calls, and you see that figure before you rely on it.

Let Qvery Take the Script Lines Off Your Quality Team: https://calendly.com/qvery-vlad/30min

## The First Program Runs on Calls Your Analysts Already Scored

What we need from you is one client's script with its change notices, a month of calls your analysts already scored, and the disputes from that month. Week one goes to connecting your recordings and loading that client's lines, each marked for exact words or meaning only.

In the second week the agents check the calls your analysts scored, and your quality lead replays every place the two disagree. Each replay lands on one of three answers: the rule was unclear, the listener missed it, or the agents did. Most often it is the rule, and a line on the list gets reworded.

That week shows you how much of the score was the rule and not the call.

From then on the agents check that program's calls every day. Each further program needs its own list and a few days of parallel checks beside your analysts, who keep scoring their usual sample in the meantime.

## Frequently asked questions



### Can the AI agents score empathy or whether the issue was resolved?

No. They fill only the facts in the transcript: the required lines, the order they came in, and the banned phrases. Empathy, ownership, and resolution need a person's judgment, so they stay on your analysts' scorecard. Both kinds of result can still share one report for the client.

### What about accents, crosstalk, or a dog barking in the background?

Calls are transcribed in whatever language they were held in, accents included. A stretch the transcript cannot make out is marked unclear with its start and end, and any line that depends on it goes to an analyst to replay. A barking dog makes a stretch unclear; it does not fail a line by itself.

### Do separate channels for the rep and the caller help?

Yes. Recordings with the rep and the caller on separate channels give the cleanest labels for who said what, so we ask for them where your system can record that way. On a single mixed channel, the agents name a speaker only where the words make it clear and mark the rest as speaker unknown.

### Which contact center software do the agents work with?

They read recordings from systems outsourcers commonly run, such as Genesys Cloud, NICE CXone, Five9, Verint, and Talkdesk. Without a direct connection, a nightly export to a shared folder does the job a day later. Results land in a spreadsheet, your QA tool, or a ticket per flagged call, and your clients' own systems are never touched.

### What happens to our QA analysts?

They keep the calls that need them. The script lines stop eating their week, so their time goes to the judgment calls, to coaching, and to the programs that never had enough coverage. When you win a new client program, that time covers it before you post a job for another analyst.

### Can a rep dispute what the agents found?

Yes, and the dispute starts from the minute and the words, because every difference is replayed by an analyst before it counts. If the rep used different words and the client allows the meaning, the line is marked that way from then on. In the first weeks, most disagreements your quality lead settles end in a rewording on the list.

### Our client lets reps paraphrase some lines. Does that still work?

Yes. Each line on the list is marked as exact words or meaning only, taken from the client's contract or script. A changed price, a missing clause, or a disclosure read too late is still flagged on a meaning-only line. Each client program keeps its own list, so one client's strict wording never spills into another's.

### What happens when a client changes its script in the middle of a month?

Your team edits that client's list the day the notice arrives, with the date the change takes effect. Calls from before that date are still checked against the old wording, so no rep is marked down for following the script that applied that day. The notice stays in the evidence file, so when the client asks why a score moved, the date and the wording are there.

### What if the agents show that some of our past scores were wrong?

That is your quality lead's call, case by case. The agents write findings and never send anything to a client. Some outsourcers bring a month's evidence file to the next business review; others answer a client's spot request with a single timed quote. Either way you decide with the recording in front of you, before the client asks.

### We already run speech analytics. Why custom AI agents?

Keep what works: speech analytics that flags a missing phrase is a good start. Custom agents add each client's list as a dated record, an exact-words or meaning-only mark on every line, a check of the order, and a quote with the minute. Built around your programs, they follow each client's change notices instead of waiting for someone to edit a rule.

### Does $400 a month cover every call we take?

It covers the calls your analysts score today, about 1,200 a month in the example, plus upkeep of each client's list. Writing out and checking a call costs cents, so the bill follows your volume. Checking every recorded call on the floor is possible, and we price it on your own numbers before anything is built.
