The Waterfall Order Playbook: Known-Answer Test to Per-Segment Cut Lines

This playbook is for the recruiting or revenue operations lead who already pays for several contact-data sources and has to decide, segment by segment, which source to query first, which to query next, where to stop, and which value wins when two sources disagree. It picks up where the single-source accuracy audit leaves off. Over about five working days, it turns a stratified sample of your own records, checked against a hidden set of known answers, into a written provider order for each segment. Each order includes the cost per incremental correct contact at every position, a cut line set in advance, and a tie-break rule that two people would apply the same way.

What a per-segment waterfall order has to contain

A per-segment waterfall order is a one-page document per segment with four parts. It lists the sources in sequence, gives the measured cost per incremental correct contact at each position with a confidence interval, marks where the waterfall stops, and states the rule for settling conflicting values. If any of the four is missing, the order is a preference, not a standard.

Terms used throughout:

• Waterfall enrichment - querying contact-data sources one after another, where each source only sees the records that every earlier source failed to fill.
• Position - a source's place in that sequence. P1 sees every record. P3 sees only the records that P1 and P2 missed.
• Residual - the set of records still unfilled when they reach a given position.
• Incremental correct contact - a contact that a position adds, that no earlier position supplied, and that matches the hidden known answer.
• Known-answer set - records whose correct email, phone and role you have confirmed first-party, kept away from everyone running the lookups.
• Cut line - the cost per incremental correct contact, fixed before testing, above which a position is dropped.
• Tie-break rule - the written procedure for choosing between two different values for the same field.

Why set the order per segment rather than once for everything? The people behind a single job title change with geography. In Refolk's index of professional profiles, a 25-profile sample of US results for Revenue Operations director and manager titles returned 19 Directors of RevOps. The matching UK sample returned 15 RevOps Managers. The title family is the same but the seniority mix is different, and seniority affects which sources find people. One published waterfall guide puts it plainly: "The right order changes by geography and seniority". A single global order averages across segments where sources behave differently, and so it fits none of them.

Why one global order by cost per hit picks the wrong sequence

Ranking sources globally by cost per hit rewards sources that return many values, not sources that return correct ones. It also treats gains at positions 3 and 4 as real when the sample is too small to tell them apart from noise. The published advice does not even agree on the starting principle.

Most vendor-written guides share three defects:

1. "Verified" is counted as correct. A verifier confirms that a mailbox accepts mail. It cannot tell you whether the right person reads it.
2. Late positions are reported from samples too small to measure them. A 3-point gain at P3 measured on 100 records cannot be told apart from zero for any practical purpose.
3. Charges on misses are left out of the cost. Two sources with the same list price can differ tenfold in real cost per correct contact.

There is also a ceiling that no ordering gets past. In a 288-contact benchmark of seven finder tools, the tools claimed 92.4% found. That fell to 88.5% once the results were verified, and all seven combined reached 89.6% live-verified coverage. Positions 5 and later are competing for roughly the last 10 points.

Stacking sources still pays. One audit of 1.3 million lookups found that combining providers raised coverage from 35.2% to 52.8%. In a 5,000-contact, 14-tool benchmark, coverage ranged from 41.3% to 87.1%, while accuracy ranged separately from 66.3% to 99.1%. Coverage and accuracy are separate measures, and an order built on only one of them will be wrong.

A verifier that passes an address has told you a mailbox exists, not that the right person reads it.

How many records each segment needs

Plan on roughly 400 to 1,100 known-answer records per segment. The question is not whether a late position adds anything. It is whether the lower bound of its gain clears your cut line. If a segment's whole population is smaller than that, sample most of it or pool it with a segment where the sources behave the same way.

The 95% margin of error for a proportion is z × √(p(1 - p) ÷ n), with z = 1.96. Published margin tables quote it at p = 50%, which is the worst case in absolute points. Late waterfall positions work at small rates. There the absolute margin looks modest, but the margin relative to the estimate is huge.

I derived the p = 3% columns from the same formula. Halving the margin takes four times the sample. The commonly published sample sizes are 385 records for ±5 points and 1,068 for ±3 points.

Use a Wilson score interval rather than the simple formula. Wilson is a confidence-interval method that stays sensible near 0%. Under it, any non-zero count separates from zero, so "is the gain real?" is the wrong test. The right test is whether the interval's lower bound clears the rate your cut line implies. No published guide sets a per-segment sample size for waterfall positions. Treat 400 to 1,100 as a planning range, and check your own lower bounds once the grid is scored.

Small segments and the finite-population correction

When a segment's population is small, the required sample gets smaller, but it becomes most of the population. Adjust the base figure of 385 with n = n0 ÷ (1 + (n0 - 1) ÷ N), where N is the population.

The populations are counts from Refolk's index. The required sample sizes are derived. The US segment is 7.9 times the size of the UK segment under the same title definition, and the orchestration-tool segment is 4.8 times the size of the contact-data-platform segment.

This has two consequences. First, you cannot test the UK segment on its own at ±5 points without sampling 58% of everyone in it, and you will rarely hold known answers for that many people. Second, both tool-skill segments are too small to test on their own at all. Pool segments by how sources behave on them (domain type, geography, seniority), not by how your org chart splits territories, and record the pooling in the pre-registration.

Most teams guess at N, and this is where the guess hurts. Describe each segment to Refolk in plain English. The count of matching profiles is the N the correction needs, and it tells you before Day 1 which segments have to be pooled.

How to calculate cost per verified contact at each position

Cost per incremental correct contact at a position is everything you paid at that position divided by the correct contacts it added that no earlier position supplied. Wherever a source charges for lookups that found nothing, those charges go in the numerator too.

No formal formula has been published. One practitioner says only that the numerator has to include credits burned on lookups that found nothing. This is the working version I use:

• Cost at position k = (spend at k + validation cost at k) ÷ known-answer-confirmed correct contacts added at k.
• Spend at k, for a source that charges on every lookup = records reaching k × price per lookup.
• Spend at k, for a source that charges only on results = hits × price per hit.

Confirm the billing rules for each source and each field before Day 2:

• One large sales platform's waterfall documentation says some sources charge only when they return data, while others may use credits on a lookup even when they find no email or phone.
• The same platform's bulk enrichment charges 1 credit for demographics or email, plus 8 credits if a mobile number is returned. Price is per field, not per record.
• One orchestration platform says that if the entire waterfall returns nothing, no credits are used. Practitioners report it has charged for empty results in the past, so check this against your own test invoice.
• Some waterfall aggregators charge only for data they find.
• The orchestration platform's safe-to-send-only setting cuts list size by 20 to 30%, which changes both the cost and the count of usable contacts further down the waterfall.

The billing model can matter more than the list price:

The inputs are hypothetical. The billing mechanics come from the published platform documentation described above.

A source that charges on a miss pays for every record that reaches it, so its cost per correct contact depends on its conditional hit rate. That is its hit rate on the residual it actually sees, not on the full list. With any cut line between $0.48 and $1.43, the same source passes or fails purely on that conditional rate.

To compare against the cut line, turn it into a rate. The break-even incremental correct rate at position k is cost per record reaching k divided by the cut-line value. Keep the position only if the Wilson lower bound of its incremental correct rate (correct contacts added ÷ records reaching k) is at or above that break-even rate. This is the same as saying that cost calculated at the lower bound must be at or under the cut line.

What counts as a correct contact

A returned value counts as correct only when it matches the hidden known answer on both the contact detail and the current role. A deliverable address is not the same as a correct one. An address on a catch-all domain stays unconfirmed until the known answer says otherwise.

A catch-all domain is one whose mail server accepts mail for any address, so a verifier cannot tell a real mailbox from a guess. Catch-all domains are common:

• One study found 22.8% of unique B2B domains behave as catch-all.
• Another puts the figure at 15% to 28% of B2B domains.
• In one 5,000-contact benchmark, 36% of the sample sat on catch-all domains.

Use these five scoring categories, and fix them in the pre-registration:

The bounce evidence shows why catch-all hits cannot be scored as correct. Unscored addresses on catch-all domains hard-bounce at roughly 8% to 15% when mailed blind. In one head-to-head test, two finders' catch-all results bounced at 3.2% and 17.2%. On 8,000 catch-all addresses, one verifier gave a verdict on only 355 (4.4%). By contrast, addresses that SMTP checks classed as valid bounced rarely: 17 of them, or 0.4% of the valid sample, in one benchmark.

The method this playbook uses comes from a smaller test: 500 contacts whose correct email was already known, with only name and domain sent to each tool. Nobody has published a rate for "deliverable but wrong person or out of role" beyond what bounce testing catches. You will produce that rate for your own segments as a by-product of this test. No published standard exists for scoring catch-all hits as unconfirmed either. It is this playbook's rule, and you should write it into your pre-registration so nobody relaxes it after seeing the numbers.

The five-day procedure

The procedure takes five working days and four people: an ops lead, an analyst, an ops engineer and a second rater. Run every source on every sampled record, then work out the best order offline instead of testing orders live.

Notes on the steps that people most often shortcut:

• Why a full grid instead of a waterfall. A live waterfall only tells you how P3 performs in the order you happened to run. The full grid lets you replay any order offline and see how P3 performs as P2. It costs more on Day 2, because sources that charge on a miss bill for every record, so budget for that explicitly.
• Choosing among candidate orders. For each segment, pick the order that keeps the most confirmed-correct contacts at positions that pass the cut line. If two orders tie, take the one with the lower total spend.
• Known-answer sources. A recent reply, a connected call, or signed paperwork confirms identity and role. A record that only passed a verifier at some point is not a known answer.
• Rater agreement. Set in the pre-registration what share of the double-scored subset must match, so you are not deciding after the fact whether the scoring was good enough.

Segment definitions and pooling:
  Segment A: ________  Population N: ____  Sample n: ____
  Segment B: ________  Population N: ____  Sample n: ____
  Pooled because sources behave alike on: ________
Sources under test, with billing model per field:
  Source 1: hit-only / charged on miss  Price: ____ per ____
  Source 2: hit-only / charged on miss  Price: ____ per ____
Inputs sent to every source: name + company or domain only
Known-answer evidence accepted: recent reply / connected call / signed paperwork
Maximum known-answer age: ____ days
Answer file custodian (not running lookups): ________
Scoring categories: confirmed-correct / deliverable-wrong or out-of-role /
  catch-all-unconfirmed / invalid / not found
Cut line: $____ per incremental correct contact
Interval method: Wilson, 95%; position kept only if cost at lower bound <= cut line
Rater agreement required on double-scored subset: ____%
Tie-break rule version: ____
Re-test date: ________
Signed: ________ (ops lead)  ________ (analyst)  Date: ________

Once you have pooled segments, the sampling frame matters as much as the sample size. Start from a population you can name and count, so each segment's sample is a known share of something real.

The tie-break rule for conflicting values

When two sources return different values for the same field, work through a fixed sequence of tests, each with a stated fallback for when data is missing. Accept the rule only after two people, resolving the conflicting rows independently, reach the same answers.

Expect a lot of these rows. Verifiers alone disagree on about 10% to 15% of catch-all addresses, and in one 8,000-address test of hard cases, 663 addresses (8.3%) were contested. The documented approaches each fail in a predictable way:

Base the rule on field-specific preference measured in your own test. That is the one input this playbook gives you that no published guide has. Use recency and majority vote only as fallbacks, each with a defined default for when its input is missing.

Field: ________   Segment: ________   Rule version: ____
1. Drop any value scored invalid by verification.
2. Prefer a value verified deliverable over a catch-all-unconfirmed value.
3. Prefer the source with the higher measured confirmed-correct rate
   for THIS field in THIS segment (from the test grid): ________ over ________.
4. If rates are within the overlap of their Wilson intervals, prefer the value
   with the later verification date. A value with no date counts as oldest.
5. If three or more sources returned values, take the value two or more agree on.
6. Otherwise take the value from the earlier waterfall position and flag the record
   for manual review.
Disputed cases (added after two-rater test):
  Case 1: ________  Resolution: ________
Two-rater agreement on conflicting rows: ____ of ____ (____%)

To measure agreement, give both raters the same list of conflicting rows and the written rule, and nothing else. Count the rows where they chose the same value. Every row where they split becomes a numbered disputed case with its resolution written into the rule. Re-run the rule on the remaining rows until agreement meets the threshold you pre-registered.

How enrichment waterfall tests go wrong

Waterfall tests usually fail in seven ways. Each one makes a late or cheap source look better than it is, and each has a tell you can spot in your own grid and a check that catches it.

1. "Verified" scored as correct. Catch-all hits inflate coverage.
   • Tell: a suspiciously high hit rate on enterprise segments.
   • Check: split every position's results by catch-all flag, and count only known-answer matches.
2. Wrong person counted as correct. The address is deliverable but belongs to someone who has left the role. One vendor's decay study found that in disagreements, the contact details were right while "the role was out of date."
   • Tell: low bounce rates alongside poor reply rates from the right title.
   • Check: compare the role field to the known answer, not just the email.
3. Residual denominator ignored. P3 is measured only on the records P1 and P2 missed. At 70% upstream fill, a 400-record sample leaves 120 records for P3, which sits close to the n = 100 row of the margin table.
   • Tell: a P3 gain reported as a share of the full sample.
   • Check: report the number of records reaching every position alongside its gain.
4. Known-answer leakage. If you send the known email or phone as an input, a source can echo it back and score a perfect match.
   • Tell: a source matches known answers at a rate above its own published coverage.
   • Check: audit the request log so that inputs are limited to name plus company or domain.
5. Stale known answers. At the commonly quoted 2.1% a month, 6.2% of answers go stale in a quarter (derived). That is as large as a typical P3 gain.
   • Tell: a source "misses" on answers that are months old and gets the newer value right.
   • Check: date-stamp every answer and enforce the maximum age you pre-registered.
6. Order chosen from a self-published benchmark. Two vendors each ran their own benchmark and each ranked itself first, with results that contradicted each other.
   • Tell: the order matches whichever vendor's report someone read last.
   • Check: treat vendor benchmarks as hypotheses. Some publish row-level CSVs and an open-source lookup benchmark exists on GitHub, so you can rerun them, but your own known-answer set decides.
7. Ambiguous tie-break. "Most recent wins" breaks down as soon as a source returns no timestamp.
   • Tell: two people resolve the same conflicting row differently.
   • Check: measure two-rater agreement on every conflicting row before publishing.

Refolk helps with the second failure. The known-answer role check needs a current title, and comparing each disputed record against its live profile in Refolk shows whether the person or only the source has moved.

Keeping the waterfall order current

Re-test on the date written into the published order, and sooner if a trigger fires. No published re-test cadence for waterfall order exists, so base yours on how fast your segment's contacts decay and how small the smallest gain you kept is.

The decay evidence is weak and inconsistent, and you should know how weak before relying on any figure:

• The widely quoted 2.1% a month traces back to a database decay simulation that credits it to a study with no linkable source.
• One contact-data vendor's cohort study measured about 1% a month in job changes. Its three-month window showed 0.50% a month, half the longer-run figure, which the study puts down to detection lag.
• The same vendor reports 12.25% of US sales leaders changed roles over twelve months and 25.67% over twenty-four.
• An analyst estimate puts business-data decay near 3% a month.

The estimates differ by about two times. Job changes take months to show up in datasets, so short re-tests understate staleness. Do not shorten the cycle just because a quick re-check looked clean.

Re-test before the scheduled date when:

• a source changes its billing model, its per-field pricing, or what it charges on a miss;
• a source's production hit rate at its position drifts outside the interval you measured;
• a segment's population moves enough to change whether it must be pooled;
• you add, drop, or renew a source.

At each re-test, refresh the known-answer set before running anything. Old answers bias the result toward whichever source last updated.