# The Outreach Split-Test Verdict: Ship, Keep Testing, or Kill

*Given send, delivery, and reply counts for two outreach variants, run three validity gates and reach a Ship, Keep Testing, or Kill verdict a second analyst would match.*

- Canonical URL: https://www.refolk.ai/guides/outreach-split-test-verdict
- Pillar: Process, data, and compliance
- Format: Framework
- Published: 2026-10-24
- Last reviewed: 2026-10-24
- Reading time: 25 min
- Keywords: how many sends per variant for outreach a/b test, is my reply rate difference statistically significant, recruiting outreach message a/b test, how to read a cold email split test, positive reply rate definition, pooling a/b tests across small campaigns

## Key takeaways

- At 95% confidence and 80% power, detecting a 50% lift takes about 363 sends per arm at a 17% reply baseline but 1,471 sends per arm at a 5% baseline.
- A 150-candidate search split 75/75 cannot detect even a doubling of a 17% reply rate, which needs about 103 sends per arm.
- Counting auto-replies doubled the reply rate in a 6-million-email benchmark, from 1.6% to 3.2% per prospect, so the reply definition has to be locked before the first send.
- In Refolk's index, exact-title recruiting operations profiles number 142 in the US and 18 in the UK, so a test aimed at one specialist audience in one country is underpowered unless it is pooled across strata with a Cochran-Mantel-Haenszel test.
- Read an outreach test 7 days after the last counted touch, not the first send, because the first three touches carried 95.1% of email replies in one reply-position study.
- A Keep Testing verdict always states the extra sends per arm: the required sample size minus the current one, for example 63 more when 300 per arm are in and 363 are needed.

This is the standard I use to decide whether a winning outreach message is real before a whole team starts sending it. It is for recruiting operations, revenue operations, and anyone who will be asked how the test was run. It is built for low volumes: a 150-candidate search or a 300-account sequence, not a send of thousands. It gives you three validity gates, a sample-size table, a locked reply definition, a readout window, and a verdict rule that a second analyst working from the same counts will reproduce.

## What does the outreach test verdict decide?

The verdict decides whether the gap between two outreach messages is real enough and big enough to change what everyone sends. It returns Ship, Keep Testing, or Kill, and it returns nothing until the test has passed three validity gates.

Terms used throughout:

- **Variant**: one version of the message. **Arm**: the group of people who received a given variant. The message you send today is the control arm.
- **Baseline**: the reply rate you expect the control arm to get, taken from your own history on that channel.
- **Minimum detectable lift (MDL)**: the smallest improvement over baseline, relative to baseline, that would be worth rolling out. You fix it before the first send.
- **Stratum**: a subgroup you split within, such as one role, one region, or one account segment. The plural is strata.

The three verdicts:

- **Ship**: the difference is statistically significant and the observed lift is at least the MDL. Roll the variant out.
- **Kill**: the upper end of the 95% confidence interval falls below the MDL. Even the most optimistic reading of the evidence is smaller than the lift you said was worth having. Keep the control.
- **Keep Testing**: neither condition holds. This verdict always comes with a number: the extra sends per arm, equal to the required sample size minus the current one.

#### From two variants to one verdict

1. **Randomized split** - People are assigned to arms by chance within each role or segment
2. **Deliverability parity** - Both arms reach inboxes at comparable rates
3. **Locked reply definition** - One numerator and one denominator, fixed before sending
4. **Score against the MDL** - p-value and confidence interval compared with the pre-set lift
5. **Verdict** - Ship, Keep Testing with stated extra sends, or Kill

*A test that fails any of the first three gates gets no verdict at all, because scoring a broken test produces a confident wrong answer.*

A failed gate does not give you a weak verdict. It gives you no verdict. Either fix the defect and re-run the affected arm, or record the test as void. Both are better than shipping a winner that came from a spam filter.

## How many sends per variant does an outreach A/B test need?

The number of sends per variant depends on two inputs: your baseline reply rate and the MDL. At a 17% baseline you need about 363 sends per arm to detect a 50% lift. At a 5% baseline you need 1,471.

These figures come from the standard two-proportion formula: two-sided, 95% confidence, 80% power.

| Baseline | Lift (target rate) | Sends per arm | Expected control replies per arm |
|---|---|---|---|
| 5% | 25% (6.25%) | 5,333 | 267 |
| 5% | 50% (7.5%) | 1,471 | 74 |
| 5% | 100% (10%) | 435 | 22 |
| 17% | 25% (21.25%) | 1,344 | 228 |
| 17% | 50% (25.5%) | 363 | 62 |
| 17% | 100% (34%) | ~103 | 18 |

The sample-size tables that rank in search mostly come from sales sending tools, and they assume a cold-email baseline of roughly 3 to 5 percent. One large sending-platform benchmark puts the average cold-email reply rate at 3.43%, with the top quarter of campaigns at 5.5% or more. At those baselines a modest lift takes thousands of sends to detect.

I checked one published figure against the formula: roughly 2,200 sends per variant to detect a move from 5% to 7%. It matches. Not every published table does. At least one widely cited figure for a 20% lift at a 3.43% baseline cannot be reproduced from its own stated inputs. Recompute any table before you build a test on it.

Messages on professional networks start much higher. A recruiting platform measured a 17.0% per-message reply rate on LinkedIn, and in its matched data 16.6% replied on LinkedIn against 4.4% on email. That changes the test size a lot. Required sends scale roughly with one over the square of the absolute difference. A 50% lift on 17% is 8.5 points, while a 50% lift on 5% is 2.5 points, so the same relative lift costs about 4x fewer sends on the higher baseline.

What this means at real volumes:

- **A 150-candidate search split 75/75** cannot detect even a doubling of a 17% reply rate. That needs about 103 per arm.
- **A 300-account email sequence split 150/150** at a 5% baseline can detect only about a tripling. By my calculation, 5% to 15% needs about 140 per arm.
- **No single low-volume campaign** can detect a 25% lift on either channel. The table needs 1,344 per arm even at the higher baseline.

Set the MDL from the decision, not from the volume you happen to have. Ask what lift would justify rewriting templates and retraining the team. If the answer is 50% and the campaign supplies 75 per arm, the honest conclusion before you send anything is that this campaign alone cannot answer the question. Pooling, covered below, exists for exactly this case. Take the baseline from your last control arm on the same channel. Treat the 5% and 17% rows as anchors, not as your numbers.

## The three validity gates

Each gate protects one assumption that the significance test makes without telling you. The first is that the arms differed only in the message. The second is that both arms had the same chance of being read. The third is that a reply meant the same thing in both. Each gate is a yes-or-no check, and all three must pass before you score.

### Gate 1: Was the split randomized?

**What it proves:** any difference between the arms comes from the message, not from who received it.

**How to pass:** assign people by chance within each stratum, so that arm sizes in every stratum differ by no more than 1. Send both arms over the same dates, from the same senders, using the same method.

**What it looks like when it lies:**

- **Hand-picked arms.** Recruiters who choose who gets which variant tend to put warm candidates into the version they like. Profiles marked "Open to Work" are about 35% more likely to respond, so the arm with more of them wins whatever the copy says.
- **Random assignment, unequal sending.** InMails sent individually see response rates roughly 15% higher than InMails sent in bulk. If one arm goes out one at a time and the other goes out in bulk, the sending method has become the variant.
- **Different dates.** If one arm lands in a holiday week, the calendar becomes the variant.

**Check:** compare the mix of profile attributes in each arm, such as Open to Work share, seniority, region, and employer type. If they are visibly unbalanced, re-randomize before sending.

### Gate 2: Did both arms have deliverability parity?

**What it proves:** both arms had the same chance of being seen, so any gap reflects the copy and not the filtering.

**How to pass:** compare bounces and delivered counts for each arm every day, and watch the spam rate. Google's guidance is to keep the user-reported spam rate below 0.1% and to prevent it from ever reaching 0.3%. Bulk senders above 0.3% lose eligibility for mitigation and only regain it once their spam rate stays below 0.3% for 7 consecutive days.

**Limits of the evidence:** Google publishes no bounce-rate threshold and no inbox-placement threshold. There is also no public standard for how far apart two arms' delivered rates can be before the comparison is invalid. Write your own tolerance into the pre-registration before you send, so it cannot be bent later to rescue a result. Spam-rate dashboards generally report by sending domain, not by message variant. Treat that figure as a ceiling on both arms together, and get per-arm parity from bounce and delivered counts.

**What it looks like when it lies:** an arm that tripped a filter looks exactly like weak copy. As a diagnostic only, compare opens from non-Apple mail clients. Apple Mail accounts for roughly 47% of all email opens and pre-fetches tracking pixels. One publisher's data showed total open rates jumping from 22.6% to 40.5% after Apple's privacy change. If non-Apple opens fall in one arm only, suspect delivery. Never use opens as the outcome.

**On LinkedIn:** there is no spam folder, but there is an account-level guardrail. Recruiter warns users whose response rate falls below 13% on 100 or more InMails sent within a 14-day period. A weak arm sent at volume can push the shared account toward that line and hurt both arms.

### Gate 3: Is there one locked reply definition?

**What it proves:** the numerator and the denominator meant the same thing in both arms and did not move while replies came in.

The definition I use as the primary metric is **unique human first replies divided by delivered messages**. Each person counts once. Auto-responders and out-of-office notes are excluded from the numerator, and hard bounces are excluded from the denominator. Positive replies are tagged separately and reported as a secondary metric. A **positive reply** is a human reply that shows interest in a conversation, such as agreeing to a call or asking for details. Decide in writing, before sending, how you will classify edge cases like "not now, try me later".

The definition changes the result more than the copy does. Here is one dataset scored four ways:

| Definition | Rate |
|---|---|
| All replies per prospect | 3.2% |
| Human replies per prospect | 1.6% |
| Human replies per email sent | 0.8% |
| Positive replies per prospect (derived) | ~0.23% |

The data comes from a 6-million-email benchmark. The positive row applies that benchmark's ratio of one positive in every seven human replies.

**2x - How much counting auto-replies inflates the reply rate**

In a 6-million-email benchmark, about 49,000 out-of-office replies arrived against 48,000 human ones, so including them doubled the rate.

Auto-replies distort the result in a particular way. Out-of-office volume rises with list size and with the time of year, not with message quality. An arm that happens to land in a vacation week collects more of them and looks more engaging than it is.

Published benchmarks also disagree on the denominator, so never compare your test rate to a benchmark until you have matched definitions:

- One sending platform's annual report counts every reply, follow-ups included, divided by total emails sent.
- The same platform's how-to guide uses unique human replies divided by delivered emails.
- One agency counts unique replies divided by emails sent, excluding auto-replies and bounce notifications.

No major benchmark I found uses unique positive first replies over delivered as its headline rate. Positive replies run at about one in seven human replies, so at these volumes they are too few to carry a verdict on their own. Report them, but decide on the human-reply metric unless you pre-registered otherwise.

> **Rule:** Lock the definition before the first send
>
> Write the numerator, the denominator, what counts as human, and what counts as positive into the pre-registration. Any change after the first reply arrives voids the test.

## When should you pool A/B tests across small campaigns?

Pool when no single role or segment can reach the required sends per arm. When you do, pool only with a stratified test that compares the arms inside each stratum before combining them. At recruiting and ABM volumes, pooling is the only way to get enough power.

The audience you are targeting puts a hard ceiling on your sample. In Refolk's index of professional profiles, exact-title matches for recruiting operations number 142 in the US and 18 in the UK. Revenue operations titles in the US number 352. Exact-title counts miss people with adjacent titles, so read them as orders of magnitude.

| Segment in Refolk's index | Count |
|---|---|
| Recruiting ops, US | 142 |
| Recruiting ops, UK | 18 |
| Revenue ops, US | 352 |
| US to UK recruiting ops (derived) | 7.9x |
| US RevOps to US recruiting ops (derived) | 2.5x |

**18 - Exact-title recruiting operations profiles in the UK, in Refolk's index**

Split evenly, that gives 9 people per arm, nowhere near the roughly 103 per arm needed to detect even a doubling of a 17% reply rate.

Here is the arithmetic. US recruiting ops splits 71/71, below the 103 per arm a doubling needs at 17%. US RevOps splits 176/176, which is enough for a doubling but short of the 363 needed for a 50% lift. Any test aimed at one specialist audience in one country is underpowered from the start.

Pooling is how you get past this. If you run the same variant pair across four roles of 150 people each, you have 300 per arm, close to the 363 needed for a 50% lift at 17%. That test lands at Keep Testing with 63 extra sends per arm, which is a usable answer.

### Why pooling must be stratified

Response rates differ by role and segment. The UK recruiting-ops profiles in Refolk's index sit at employers as different as Anthropic, Jane Street, QuantCo, and British Airways, and there is no reason to expect a quant firm's recruiting team to reply at the same rate as an airline's. If one arm happens to hold more people from the high-responding stratum, a single combined table credits the copy. As one statistics reference puts it, pooling data where the prevalence of the characteristic varies between replicates "can quite simply give the wrong answer".

The **Cochran-Mantel-Haenszel (CMH) test** solves this. It combines evidence from one 2x2 table per stratum into a single summary test and a common odds ratio, so each stratum is compared only with itself. Before trusting the common odds ratio, check that the per-stratum odds ratios are similar, using the Woolf test. If they point in opposite directions, do not pool. The variant works differently by segment, so report each stratum on its own.

The slow part of planning a pooled test is sizing the strata: counting how many people actually fit each role-and-region cell before you commit sends to it. [Refolk](/) takes a description in plain English and returns the matching people across LinkedIn, GitHub, and the open web. That lets you see whether a stratum holds 18 people or 180 before you write the pre-registration, not after the first wave has gone out.

I ran this search: `Recruiting operations managers in the UK at AI labs or quant trading firms` - [see the full result list](https://www.refolk.ai/s/d89txjkjsy).

*Returns named people with current titles and employers, which is the stratum you would size and then randomize before splitting.*

The profile fields that come back with the list also feed the Gate 1 balance check, so you can compare the mix of each arm before anyone sends a message.

**Stratified CMH score in R, one 2x2 table per stratum**

```
# Rows = arm (A, B). Columns = replied, not replied.
# Values fill column by column, so each line is A replied, B replied, A not, B not.
counts <- array(c(
  A1_replied, B1_replied, A1_not, B1_not,
  A2_replied, B2_replied, A2_not, B2_not,
  A3_replied, B3_replied, A3_not, B3_not,
  A4_replied, B4_replied, A4_not, B4_not
), dim = c(2, 2, 4),
   dimnames = list(arm = c("A", "B"),
                   outcome = c("replied", "not_replied"),
                   stratum = paste0("S", 1:4)))
mantelhaen.test(counts)
# Record the p-value and the 95% CI on the common odds ratio.
# Compare that CI with the MDL expressed as an odds ratio.
```

*Replace each placeholder with your counts, where "not" means delivered minus replied; add a block per stratum and set the last dim to the number of strata.*

## When should you read the results?

Read the test 7 days after the last counted touch. For most sequences that means 7 days after the third touch, not 7 days after the first send. Reading earlier undercounts slow responders, and reading repeatedly inflates false positives.

What the timing evidence shows:

- **InMail:** 65% of responses arrive within 24 hours and 90% within one week. This comes from a secondary citation, and I could not find the original LinkedIn data.
- **Speed by channel:** in one recruiting platform's data, the median email reply landed in under 4 hours, against 8.4 hours on LinkedIn.
- **Email within 24 hours across a full sequence:** no public figure. If you need it, measure it on your own last campaign.

Follow-ups carry a large share of replies, so a readout that ignores them is biased toward whichever variant has the stronger opener.

#### Where email replies come from, by touch

| Stage | Figure | Note |
| --- | --- | --- |
| First email | 61.7% | share of all replies |
| Follow-up 1 | 20.9% | share of all replies |
| Follow-up 2 | 12.5% | share of all replies |

*In one reply-position study the first three emails carried 95.1% of all replies, so the readout clock starts at the third touch.*

Other sources put the first email's share lower. A sending platform's benchmark says the first email captures 58% of replies, and an agency found 41.4%. Interested replies may lean earlier: in the reply-position study, 52.8% of 123 interested replies came from the first email. That sample is small, so don't build on it.

Published guidance on run time disagrees too, with one source saying wait 5 to 7 business days and another saying 7 to 10. My rule is 7 calendar days after the last counted touch, written as a fixed date in the pre-registration. If you want more margin, extend the window before sending, never after looking.

> **Watch out:** Peeking turns noise into winners
>
> If you check an ongoing test ten times, what you read as 1% significance is really 5%. If you must look early, pre-register a sequential method, such as Evan Miller's sequential procedure or Etsy's threshold based on the minimum detectable effect, instead of glancing at the dashboard.

## How do you score the test and reach a verdict?

Run a two-proportion test, or CMH if you pooled. Compute the 95% confidence interval for the difference in reply rates, and compare both the p-value and the interval with the MDL converted to percentage points. Once the gates have passed, the verdict follows mechanically from where the interval sits.

First, convert the MDL to points: multiply the baseline by the relative MDL. At 17% with a 50% MDL that is 8.5 points. At 5% with a 100% MDL it is 5 points. For a pooled CMH test, express the MDL as an odds ratio instead: the target rate's odds divided by the baseline's odds.

Then score:

1. **Count each arm:** delivered messages and unique human first replies, taken from the blind classification.
2. **Compute each arm's rate** and the difference between them.
3. **Compute the 95% confidence interval** for the difference. With small counts, a score-based or exact interval is more reliable than the simple normal approximation. Whichever you choose, name it in the pre-registration so the second analyst uses the same one.
4. **Place the interval** on the matrix below and read off the verdict.

#### Reading the confidence interval against the MDL

Horizontal axis runs from Upper CI bound below the MDL to Upper CI bound at or above the MDL. Vertical axis runs from Lower CI bound at or below zero to Lower CI bound above zero.

| Quadrant | What it means |
| --- | --- |
| Real but too small | Kill, because the gain is real but below the lift worth rolling out |
| Significant, MDL in reach | Ship if the observed lift reaches the MDL, otherwise Keep Testing |
| No case | Kill, because even the optimistic end misses the MDL |
| Unresolved | Keep Testing, with the extra sends per arm stated |

*The verdict depends on where both ends of the interval sit, never on the point estimate alone.*

### How many more sends a Keep Testing verdict needs

Extra sends per arm equal the required n from the pre-registration minus the current n per arm. In the pooled example, 363 minus 300 gives 63 more sends per arm, which go out under the same randomization, dates rule, and definition.

If the remaining audience cannot supply those sends, record that as the finding: at this MDL, with this audience, the question cannot be answered, so the control stays. Do not quietly lower the MDL mid-test to make the numbers fit. A new MDL means a new test with its own pre-registration.

> **Tip:** Report the interval, not just the p-value
>
> A p-value tells the team whether there is a difference. The interval tells them how big it might be, and the size is what decides whether the rollout is worth the retraining.

Two analysts holding the same pre-registration and the same counts will reach the same verdict. Every judgement that could bend the result (baseline, MDL, definition, interval method, tolerance, readout date) was made before any data existed.

## Step-by-step: running the outreach test verdict

The procedure takes about four hours of ops and analyst time across the life of the test, plus the sending days. Follow it in order. Each step has a "done" condition, and you should not start the next step until it is met.

#### Outreach test verdict procedure

1. **Pre-register the test** - The ops lead writes the baseline, MDL, locked reply definition, sends per arm, deliverability tolerance, and readout date in a dated document nobody can edit.
2. **Randomize the split within each stratum** - Assign people by chance inside each role or segment, so that arm sizes differ by no more than 1 per stratum and the profile mix is comparable.
3. **Send both arms over the same dates** - Use the same senders, the same sending method, and the same days for both arms.
4. **Run the deliverability gate daily** - Compare bounces and delivered counts per arm, keep spam rate under 0.3%, and confirm the delivered gap stays within the pre-set tolerance.
5. **Wait out the readout window** - Look at nothing until 7 days after the last counted touch, usually the third.
6. **Classify replies blind** - A second person tags each reply as human or auto and as positive or not, without seeing the arm, until the two classifiers agree.
7. **Score against the minimum detectable lift** - Run a two-proportion or CMH test and compare the p-value and 95% interval with the MDL in points.
8. **Record the verdict** - Ship, Kill, or Keep Testing, and for Keep Testing state the extra sends per arm as required n minus current n.

The pre-registration is the document that makes the verdict reproducible. Copy this form, fill every line before the first send, and store it somewhere edits are logged or blocked.

**Outreach test pre-registration**

```
TEST ID =
Date registered =
Owner =
Channel (email / InMail / other) =
Control message = (paste in full)
Variant message = (paste in full)
Strata (role, region, or segment) =
Baseline reply rate (from last control arm on this channel) =
Minimum detectable lift (relative) =
MDL in points (baseline x MDL) =
Sends per arm required (95% confidence, 80% power) =
Sends per arm available, by stratum =
Primary metric = unique human first replies / delivered
Excluded from numerator = auto-replies, out-of-office, bounce notices, repeat replies from the same person
Excluded from denominator = hard bounces
Positive reply means =
Not positive (edge cases decided in advance) =
Randomization method =
Send dates (both arms) =
Senders and sending method (same for both arms) =
Deliverability tolerance (max gap in delivered rate between arms) =
Spam-rate ceiling = below 0.3%, target below 0.1%
Last counted touch =
Readout date (7 days after last counted touch) =
Analysis = two-proportion test / CMH across strata
Interval method =
Verdict rule = Ship if significant and lift >= MDL; Kill if upper CI bound < MDL; otherwise Keep Testing, extra sends per arm = required n minus current n
```

*Fill every line before the first send, date it, and store it where it cannot be edited; a blank line is a decision you will be tempted to make later.*

## How outreach tests go wrong

Most false winners come from the test, not the copy. The usual culprits are a hand-picked split, an arm that hit a spam filter, auto-replies in the count, opens used as the metric, daily peeking, naive pooling, or an early readout. Each one produces a specific false signal and has a specific check.

1. **Split not randomized.** False signal: recruiters hand-pick the arms, so one arm holds more warm profiles, and "Open to Work" profiles are about 35% more likely to respond. Check: compare the profile-attribute mix of each arm before sending, and confirm the assignment was made by chance within each stratum.
2. **Unequal deliverability.** False signal: one arm hits a filter, and the copy takes the blame for a delivery problem. Check: per-arm bounces and delivered counts, the spam rate, and opens from non-Apple clients only, used as a diagnostic.
3. **Auto-replies counted as replies.** False signal: auto-replies make up roughly half of everything in the reply folder, so the rate looks about twice as high, and the arm that landed in a vacation week looks best. Check: re-score after excluding auto-replies and see whether the gap survives.
4. **Open rate used as the metric.** False signal: if half your opens are machines, and machines open both variants 100% of the time, the measured difference shrinks toward zero, and small real differences disappear. Check: confirm the outcome is replies and opens appear only as a delivery diagnostic.
5. **Peeking.** False signal: checking every day and stopping when the gap looks good. Ten peeks turn a nominal 1% significance level into 5%. Check: was the readout date set in advance, and did anyone report numbers before it?
6. **Naive pooling across segments.** False signal: one combined table where one arm happens to hold more of a high-responding stratum, so segment mix looks like copy quality. Check: per-stratum results alongside a CMH test, plus a Woolf check that the per-stratum odds ratios agree.
7. **Early readout.** False signal: reading before the window closes undercounts slow responders and favors the variant with the stronger first touch. Check: the readout date is at least 7 days after the last counted touch.

> A variant that went to spam is a delivery result, not a copy result, and it gets no verdict.

Failure modes 1, 3, and 6 are the dangerous ones because they produce clean-looking significance. The p-value is real, but it measures the wrong thing. If a result looks too good for the change you made, work through this list before you announce it.

## Before you roll out the winner

Run this check before any variant goes to the whole team, then keep the inputs current, because baselines and provider rules drift.

#### Verdict sign-off

- [ ] A dated, uneditable pre-registration exists and every line was filled before the first send.
- [ ] Arm sizes differ by no more than 1 within every stratum, and the profile mix in each arm is comparable.
- [ ] Both arms went out over the same dates, from the same senders, using the same sending method.
- [ ] Both arms stayed under a 0.3% spam rate, and the gap in delivered rates stayed within the pre-registered tolerance.
- [ ] The numerator excludes auto-replies and repeat replies, and the denominator is delivered messages.
- [ ] Replies were classified by a second person who did not know the arm, and the two classifiers agree.
- [ ] The readout happened no earlier than 7 days after the last counted touch, with no earlier looks at reply rates.
- [ ] If the variant was pooled, per-stratum results were checked for agreement before the CMH result was used.
- [ ] The verdict record states the p-value, the 95% interval, the MDL in points, and, for Keep Testing, the extra sends per arm.

To keep the standard current:

- **Re-derive the baseline every test.** Use the last control arm on the same channel, not a published average. Once a winner ships, it becomes the new control, and its rate becomes the next baseline.
- **Re-check provider rules before each campaign.** Mailbox providers publish and revise their bulk-sender requirements: Google's sender guidelines FAQ covers spam-rate thresholds, and Microsoft enforces its own bulk-sender requirements. LinkedIn's InMail response-rate warning can change too. Read the current version rather than relying on the figures here.
- **Re-measure your own reply latency.** The 7-day window rests partly on a secondary InMail figure. Once you have two or three campaigns, check what share of your own replies arrived within 7 days of each touch, and lengthen the window if yours run slower.
- **Re-size strata before pooling.** Audiences change as people move jobs. Re-count each role-and-region cell before you plan sends, so the sample sizes in the pre-registration reflect the people who actually exist today.

## Frequently asked questions

### How many sends per variant do I need for an outreach A/B test?

At 95% confidence and 80% power, you need about 363 sends per arm to detect a 50% lift from a 17% baseline, which is typical of professional-network messages. From a 5% cold-email baseline you need 1,471 per arm. Detecting a doubling takes about 103 and 435. Take your baseline from your own control history and fix the minimum lift worth acting on before you send. If one campaign can't supply the sends, pool the same variant pair across roles with a stratified test.

### Is my reply rate difference statistically significant?

You can only ask that once the test has passed three validity gates: a randomized split, deliverability parity, and one locked reply definition. Then run a two-proportion test, or a Cochran-Mantel-Haenszel test if you pooled across strata. Read the 95% confidence interval for the difference and compare it to your pre-set minimum detectable lift. Significance alone doesn't earn a Ship verdict. The observed lift also has to reach the lift you said was worth rolling out.

### What is a positive reply rate and how should I define it?

Positive reply rate is the share of delivered messages that drew a human reply showing interest, such as agreeing to a call or asking for details. Published benchmarks use different denominators, and none I found uses positive first replies over delivered as its headline rate. Write your own definition before the first send and track it separately from the main human-reply metric. Expect roughly one positive reply for every seven human replies.

### Should auto-replies count in my reply rate?

No. In a 6-million-email benchmark, counting auto-replies doubled the reply rate, from 1.6% to 3.2% per prospect, because out-of-office notes roughly matched human replies one for one. How many auto-replies you get depends on list size and time of year, not on message quality, so counting them adds noise that can hide a real difference in copy. Exclude them in the locked definition, and have a second person classify replies without knowing which arm they came from.

### Can I pool A/B tests across several small campaigns?

Yes, as long as every campaign tests the same two variants and you analyze them with a stratified method such as Cochran-Mantel-Haenszel. That method compares the arms inside each role or segment before combining. First check that the per-stratum odds ratios are similar, using the Woolf test. Pooling everything into one table can give the wrong answer when reply rates differ by segment. Four 150-person roles give 300 per arm, close to the 363 needed for a 50% lift at 17%.

### How long should I wait before reading an outreach test?

Wait at least 7 days after the last counted touch, which usually means after the third touch, not the first send. About 90% of InMail responses arrive within a week, and in one reply-position study the first three emails carried 95.1% of replies. Put the readout date in the pre-registration and don't check reply rates before then. Ten peeks can turn what looks like 1% significance into 5%.

---

*From the Refolk guide library. I revise these guides rather than replacing them, so the current version is always at https://www.refolk.ai/guides/outreach-split-test-verdict*