A Validity Check for Outbound Call Quality Samples

Research question and evidence scope

Research question: does the quality sample represent the calls the campaign actually makes, and can two reviewers apply the rubric consistently? The unit is one sampled call and its review record. The method examines coverage, selection, rubric clarity, and the link between the score and customer outcomes.

Start with the population. Which campaigns, shifts, agents, call types, and outcomes are eligible? A sample drawn only from completed sales calls excludes the calls that were abandoned, refused, or misdispositioned. Those may be the most important calls to review.

Define the selection rule. Random sampling is better than convenience, but it can still underrepresent small queues or rare adverse cases. Use stratification for campaign, shift, and outcome, plus a targeted sample of complaints and correct stops. Report the frame, the exclusions, and the minimum count per stratum.

Operating scenario and evidence

Write observable criteria. Replace professional tone with specific behaviors: the opening identifies the business, the purpose is stated, the verification occurs before disclosure, the opt-out language is read, and the disposition matches the call. Cite the timestamp or line for every score.

Separate compliance, accuracy, customer treatment, and outcome. A call can be compliant and still unpleasant, or friendly and inaccurate. Do not collapse the dimensions into one number unless the business has agreed the weighting and the consequences.

Method and review criteria

Have two reviewers score the same sample independently. Compare exact and adjacent agreement. Discuss every material disagreement and record whether the rubric, the evidence, or the reviewer's product knowledge caused it. A low agreement rate means the score is not ready for a personnel or vendor decision.

Compare the scores with customer outcomes: repeat contacts, complaints, opt-outs, and later conversions. If the highest-scoring calls have the same complaint rate as the lowest, the rubric may be measuring style rather than quality. The comparison does not prove causation, but it can reveal a rubric that rewards the wrong behavior.

Include adverse cases and correct stops. A caller who ends a call because the person asked not to be contacted should score well on the boundary criterion. If the rubric rewards only completed sales, it will teach persistence over respect.

Authority and practical implications

Report the sample size, the population, the selection rule, the exclusions, the score distribution, and the agreement result. Segment by campaign and by risk. A single average can hide a serious failure in one call type.

The method cannot prove that a score predicts long-term performance or legal compliance. It tests whether the sample and the rubric support a defensible review. Use it to improve the process, not to certify a provider.

Reviewer calibration is not a one-time event. New reviewers, new scripts, and new campaigns can change the interpretation. Recalibrate after a material change and keep the agreement result. A rubric that was reliable last quarter may not be reliable now.

Limitations, conclusion, and sources

The decision record should show the population, the frame, the sample size, the strata, the exclusions, the agreement, the score distribution, and the actions. It should separate the score from the consequence. A low score can lead to coaching, a script fix, or a data repair. The record should name which one and why.

Add a targeted sample of calls that the dashboard labels as short, unresolved, or abandoned. Those calls are often excluded from quality review even though they carry the customer's worst experience. Review them with the same rubric and report them separately.

The operational use is to fix the rubric, the sample frame, or the coaching process. If the scores do not predict any customer outcome, the rubric may be measuring style. Keep the score as one input and show the limitation beside it.

Report the sample coverage by campaign and shift. A team that is rarely sampled may look better or worse than it is. If a queue is too small to sample independently, combine it with a similar queue and say so. Do not present a precise score for a group that produced three calls.

The operational use is to fix the frame, the rubric, or the consequence. A score that is used for a vendor decision needs a larger and more representative sample than one used for a weekly coaching conversation. State the intended use before choosing the sample size.

Sources checked October 22, 2026: U.S. GAO, Assessing Data Reliability | https://www.gao.gov/products/gao-20-283g | NIST Cybersecurity Framework 2.0 | https://www.nist.gov/cyberframework | National Do Not Call Registry, Business Guidance | https://telemarketing.donotcall.gov/. These sources provide regulatory, privacy, and data-reliability context. They do not certify this protocol or any provider's performance, and they do not replace advice from qualified counsel.