Refolk
TeardownProcess, data, and compliance

The Reply-Drop Teardown: One Halved Sequence to a Proven Root Cause

Take one sequence whose replies halved, run seven ordered isolation passes with the counts written down, and name a single proven cause inside one working day.

21 min readLast reviewed November 20, 2026Read as Markdown

Key takeaways

  • In a 1,400-recipient sequence split 700/700, a reply drop from 6% to 3% is real at the sequence level (z ≈ 2.71, p ≈ 0.007), but the same halving inside a 175-recipient Outlook cell is noise (z ≈ 1.35, p ≈ 0.18).
  • Provider and domain cuts in small sequences have to rest on categorical evidence, such as 5.7.515 strings in bounce logs or a seed landing in Junk, not on reply-rate p-values.
  • The rule that opens and replies falling together means deliverability fails because open tracking is unreliable, and switching tracking on or off can move reply rates by itself: one large benchmark shows 7.4% replies without open tracking against 4.4% with it.
  • Copy is the same across every sending domain, so a reply drop concentrated on one domain cannot be a copy problem, and rewriting copy before the domain cut only adds a confounder.
  • Microsoft counts its 5,000-a-day threshold per visible From domain and Gmail bulk-sender status is permanent once reached, so check historical peaks and shared From domains, not one sequence's daily volume.
  • In Refolk's index, 468 people in the United States list Email Deliverability as a skill but only 15 hold a deliverability title, so most teams running this diagnosis have no named owner for the log and seed passes.

This guide takes one outreach sequence whose replies halved in three weeks and follows it from the dashboard number to one signed-off cause. It is for recruiting operations, revenue operations, and anyone who has to defend how the diagnosis was done. It covers seven isolation passes in a fixed order, the counts recorded at each one, the wrong turn the team took, and the evidence that rules out the four causes that were not to blame.

The worked case: what does a halved sequence look like?

The case is one sequence sent to 1,400 recipients: 700 in the window before the drop and 700 in the window after. Replies fell from 6% to 3%, which is 42 replies down to 21. At that size the halving is statistically real (z ≈ 2.71, p ≈ 0.007). That is the only thing the dashboard number proves.

The case is a composite, built so you can check every count by hand. A few terms first:

  • Reply rate: replies divided by the recipients who were sent step one, counted separately for each window.
  • Window: a fixed range of send dates that you compare with another range.
  • Two-proportion z-test: the standard test of whether two rates differ by more than chance. Here it is two-sided, at a 0.05 significance level and 80% power.

There are five suspects, and the whole job is to convict one of them and clear the other four with evidence:

  1. The report: the number changed because the definition, the denominator or the tracking changed.
  2. Reply routing: replies arrived, but the tool never logged them.
  3. Inbox placement: mail went to junk or was rejected at some providers or from some domains.
  4. List drift: the people being emailed changed. This is also called segment-mix drift, meaning a shift in title, seniority, company size or source between windows.
  5. Copy: the message itself stopped working.

The dashboard number tells you there is a drop. It does not tell you where. Every cut you make to find out where makes the cell smaller, and cell size decides what a cut can prove.

Baseline → halvedn per window700/700 sequence?175/175 Outlook cell?
8% → 4%~553YesNo
6% → 3%~749Yes (z 2.71)No (z 1.35)
4% → 2%~1,141NoNo
2% → 1%~2,318NoNo

The "n per window" column is the size you need for an 80% chance of detecting the halving before you look. The case's 700 is slightly under the ~749 that a drop from 6% to 3% calls for, but the observed counts still clear the bar once the drop has happened. These values are computed with the standard formula. Nobody has published an outreach-specific figure.

Check where your baseline sits. One large cold-email benchmark of 31M emails puts the average sequence reply rate at 4.5%. Sequences of 500 or more recipients averaged 2.4%, and the best sequences, at 21 to 50 recipients, reached 6.2%. If your baseline is 2% to 4%, the table says a 700-per-window sequence cannot even show a halving reliably. In that case the first question is whether you have a drop at all.

Each cut shrinks what the numbers can prove

  1. Whole sequence
    1,400

    both windows together

  2. One window
    700

    a halving from 6% is detectable

  3. Gmail and other cell
    525

    below the ~749 a halving needs

  4. Outlook-facing cell
    175

    z ≈ 1.35, p ≈ 0.18, noise

A halving that is significant across 1,400 recipients becomes noise once you split it down to a 175-recipient provider cell.

Why does the "opens and replies fell together" rule no longer work?

The rule assumes open counts measure humans opening mail. They have been unreliable since Apple's Mail Privacy Protection launched, and many teams now switch open tracking off. Worse, turning tracking on or off can move the reply rate by itself, so the rule can blame the wrong cause in either direction.

Benchmarks from outreach tools show how large that effect is:

  • Campaigns without open tracking had a 7.4% reply rate against 4.4% with it, a 68% gap, in one large cold-email benchmark.
  • A second benchmark found that turning off open tracking more than doubled reply rates, 2.36% against 1.08%.
  • The same second source found 0.87% against 0.21% replies without and with tracked links.

Two consequences follow. First, if anyone changed a tracking setting between your windows, that change is a suspect in its own right and belongs in the measurement pass. Second, the false positive this rule produces is the expensive one: opens look flat while mail is landing in junk, so the team decides the copy has gone stale. I found no published figure for what share of opens are machine opens, so don't try to correct opens with a ratio. Use recipient-provider reply splits and bounce strings instead.

The isolation passes, in order

Run seven passes in a fixed order. The cheapest passes, the ones that give a yes or no answer, come first, and copy comes last. My time estimates add up to about six to seven hours, which fits one working day. These estimates are not measured.

Seven passes from dashboard number to named cause

  1. Check the measurement
    Freeze the reply definition and denominator, recompute raw reply counts against the prior 30 days, and log any tracking-setting change between windows.
  2. Check reply routing
    Confirm reply-to, forwarding, mailbox connections and auto-reply filters, and count replies sitting in mailboxes that the tool never logged.
  3. Split by sending inbox and domain
    Tabulate sends, replies and bounces per sending domain, and search the bounce logs for 5.7.515 and other 5.7.x strings.
  4. Split by recipient provider
    Group recipients by MX into Google, Microsoft and other, and record the reply rate per provider per window with its z-test result.
  5. Run seed placement
    Send the live message through the live path at nearly the same time, and record the folder per provider for each sending domain.
  6. Compare segment mix
    Compare title, seniority, company size and source mix between the two windows before blaming copy.
  7. Read copy by step, last
    Compute reply rates by step within the unaffected provider and domain cells only.
PassOwnerMy estimate
1 to 2: measurement, routingRevOps, ops or IT1 hour
3 to 4: domain and provider splitsRevOps2 hours
5: seed placementDeliverability owner1 to 2 hours
6 to 7: segment mix, copySourcing, outreach owner2 hours

Published orders disagree. One ranking audit puts measurement first and reply handling last. Another goes mailbox health, then targeting, then copy, and skips measurement entirely. I put routing second because it costs 30 minutes and settles a whole suspect with a single count. Anything that settles a suspect that cheaply should run before anything that only weighs evidence.

Pass by pass: the case with its counts and forks

Each pass below gives what was run, what came back, and the fork you would take if your own case comes back differently. Write your counts in the same order. The written record is what lets someone else sign off the cause.

Pass 1: measurement

Freeze the definition. Decide whether auto-replies, out-of-office messages and unsubscribe requests count as replies. Decide whether the denominator is recipients sent or recipients delivered. Then follow the practitioner advice to check raw reply count, not reply rate, against the prior 30 days.

  • Case result: the definitions matched across windows, no tracking setting changed, and the raw count fell from 42 to 21. The drop is in the replies, not in the report.
  • Fork: if tracking changed between windows, stop. Re-baseline on a window with the current setting before going further, because a gap like 7.4% against 4.4% can come from the setting alone.

Pass 2: reply routing

Check the reply-to address on every sending inbox, forwarding rules, any mailbox that has disconnected from the outreach tool, and auto-reply filters that might be swallowing real answers.

  • Case result: none found. Every reply in the mailboxes had been logged.
  • Fork: if you find unlogged replies, add them back and re-run pass 1. A disconnected mailbox can look exactly like a domain-level placement problem, so this pass has to come before pass 3.

The wrong turn: copy changed partway through the bad window

Before any of these passes ran, the outreach owner had rewritten the step-one copy partway through the bad window. It looked like the obvious fix. What it actually did was split the bad window into two copy versions. Pass 7 would now have to separate them by send date, and any later recovery could no longer be credited cleanly. Note the date of any mid-window change in your own record, because you need it in pass 7.

Pass 3: sending inbox and domain

Build the per-domain table, then run the log search. Search your bounce logs for the error string "5.7.515". That is the code Microsoft uses to reject high-volume mail that fails its sender requirements. Search for other 5.7.x strings too: Gmail's temporary 4.7.x errors escalate to permanent 5.7.x rejections.

  • Case result: the reply drop was concentrated on one sending domain, whose recipient list leaned towards Outlook. The overall bounce rate looked normal, the classic "replies down but bounces normal" pattern, because junked mail never bounces. The log search found 5.7.515 strings on that domain only.
  • Fork: if the sequence alone sends far below 5,000 a day, ask what else sends from the same visible From domain. Microsoft counts per From domain, not per sequence or per tool. In the case, the suspect domain was the company's main From domain, which other systems also sent from.

Sources disagree on whether Outlook junks first and then rejects with 5.7.515 if problems persist, or rejects outright. Either way, the absence of 5.7.515 does not clear a domain. Junking leaves no string in the log.

Pass 4: recipient provider

Group recipients by MX record, the DNS entry that says which provider receives a domain's mail. Use three groups: Google, Microsoft and other. Then compute each cell's reply rate per window.

  • Case result: the Outlook-facing cell was 175 per window, and its halving gave z ≈ 1.35, p ≈ 0.18. That is noise. The Gmail-and-other cell of 525 per window is also under the ~749 a halving needs.
  • What it proves: only that the split is consistent with pass 3. It does not prove anything on its own. If you report "Outlook replies down 50%" from this cell, you are reporting chance.
  • Fork: if the drop is spread evenly across providers, placement becomes less likely and passes 6 and 7 carry more weight.
A 175-recipient cell cannot prove a halving. A 5.7.515 string in the bounce log can.

Pass 5: seed placement

A seed test sends mail to test inboxes you control at each provider, to see which folder it lands in. The documented method is the same production message, through the same sending path, at nearly the same time as the real campaign.

  • Case result: Outlook seeds from the suspect domain landed in Junk. Outlook seeds from the other domains, and all Gmail seeds, landed in the inbox.
  • How it was reported: as a folder result per provider and per domain, never as a placement percentage. Suped's caution applies: if four of five seeds at one provider reach the inbox, that is 80% of those five test accounts, not 80% of your recipients.
  • Fork: if seeds land in the inbox but pass 3 found rejection strings, trust the strings. Seeds have no engagement history, and nobody has published how often they disagree with real recipients.

Pass 6: segment-mix drift

Compare equivalent cohorts before changing copy or increasing volume. Tabulate title, seniority, company size and source for both windows. A sourcing change can halve replies on perfectly healthy infrastructure.

  • Case result: the mix matched across windows, so list drift is ruled out.
  • Fork: if one window pulled heavily from a new source or a more senior band, rebuild a matched cohort before going further.

If the list came from a saved plain-English search in Refolk, the query text is the segment definition. You can tell whether the cohort changed by checking whether the query changed, without rebuilding the mix from exports.

Pass 7: step-level copy

Compute reply rates by step inside the unaffected domain and provider cells only, and split the bad window by the date of the rewrite.

  • Case result: the unaffected cells were too small to prove a copy effect either way, and they did not show the collapse the suspect domain showed. That ranks copy last. It does not clear copy forever.
  • Decision: the rewrite stays parked until the domain is fixed and a clean window is measured.

The verdict

The cause was inbox placement on one Outlook-facing domain. The evidence that rules out each of the other four:

  • Report: the definitions matched, tracking was unchanged, and raw counts fell from 42 to 21.
  • Routing: no unlogged replies in any mailbox.
  • List drift: the cohort mix matched across windows.
  • Copy: copy was identical across every sending domain, so a drop concentrated on one domain cannot be a copy effect.

That last argument settles the case, and it needs no statistics. It is categorical evidence (present or absent, not a rate), which is the only kind small cells can carry.

Reading where the drop sits against the hard evidence

Bounce strings or seeds in junkNo hard evidence
Placement on one domain or provider
Pause that domain, fix it, re-seed before resuming
Sender-wide placement problem
Check shared From domains and historical volume peaks
Probably noise at this cell size
Re-check routing, then gather more sends before acting
Report, list drift or copy
Run passes 6 and 7 on matched cohorts
Drop sits in one cellDrop spread across all cells
Concentration plus categorical evidence convicts placement; spread without evidence sends you to segment mix and copy.
Cut table for passes 3 and 4
Cell | Window | Sent | Replies | Reply rate | 5.7.515 count | Other 5.7.x count | Seed folder | Signal or noise
domain-a | before |  |  |  |  |  |  |
domain-a | after  |  |  |  |  |  |  |
domain-b | before |  |  |  |  |  |  |
domain-b | after  |  |  |  |  |  |  |
Google MX | before |  |  |  |  |  |  |
Google MX | after  |  |  |  |  |  |  |
Microsoft MX | before |  |  |  |  |  |  |
Microsoft MX | after  |  |  |  |  |  |  |
Other MX | before |  |  |  |  |  |  |
Other MX | after  |  |  |  |  |  |  |

One row per sending domain, then one row per recipient provider; write "noise" in the last column for any cell under the Table B size.

Who should own the seed and log passes?

Passes 3 and 5 need someone who can read SMTP bounce strings and run a seed test properly, and very few people hold that job title. In Refolk's index of professional profiles, 15 people in the United States hold a deliverability-specific title. 468 list "Email Deliverability" as a skill, and none of the 10 most common titles in that group contains the word "deliverability".

31.2x
US professionals claiming the Email Deliverability skill per deliverability title-holder
In Refolk's index, 468 people claim the skill against 15 who hold the title; Founder is the most common title among the skill-claimers.
Refolk's index queryCount
Deliverability titles, US15
Deliverability titles, UK2
"Email Deliverability" skill, US468

The US title pool is 7.5 times the UK one. The US title-holders' top employers include Marigold, Proofpoint, Mailchimp, Resend, OneSignal and Hightouch. The two in the UK sit at Oracle and AI Global Media. My inference, which I have not measured, is that most teams diagnosing a reply drop have no named deliverability owner, and that this is why the copy rewrite happens first: the outreach owner is the only person in the room who can change something.

If you need to hire or contract that owner, searching by title alone finds the 15 and misses the 468. Refolk searches skills and employers together, so you can ask for both groups at once.

Which provider rules and dashboards can confirm a placement drop?

On Outlook, a rejection shows up in your own SMTP logs as 5.7.515, so you can diagnose an Outlook-facing domain from logs alone. Gmail placement loss is mostly invisible now that Postmaster Tools no longer shows domain and IP reputation. Know what each provider publishes before you trust any dashboard.

ProviderVolume triggerComplaint ceilingFailure response
Gmail~5,000/day to personal Gmail, permanent0.3% (target 0.1%)4.7.x, then 5.7.x permanent
Yahoo/AOLNot published0.3%, on inbox-delivered mailFiltering or rejection
Outlook.com5,000/day per From domainNot publishedJunk, then 550 5.7.515 (sources conflict)

A bulk sender, in Gmail's definition, is one that sends close to 5,000 or more messages to personal Gmail accounts in 24 hours. That status is permanent once reached. Microsoft's rule covers any domain sending 5,000 or more a day to Outlook.com, Hotmail.com and Live.com, counted per domain in the visible From address. Microsoft publishes no complaint-rate number in these requirements. That point is unverified, so check Microsoft's high-volume sender announcement directly. Yahoo publishes no volume threshold. Some secondary sources say 5,000, but Yahoo itself does not.

SurfaceKeyed byBlind spot
Postmaster Tools v2 (Gmail)DomainNo reputation charts; "No data" at low volume
SNDS (Outlook.com)IP, per dayNeeds control of the IP space; nothing below ~100/day per IP
JMRP (Outlook.com)IPComplaining recipient's address is redacted
Yahoo CFLDKIM domainGives a complaint count, not a rate

Notes that change how you read these:

  • Postmaster v2 volume floor: the compliance dashboard needs 5,000 messages a day to Gmail. Below that, Google publishes no minimum. Practitioners estimate about 100 a day for any data and about 1,000 a day for a reliable spam chart. Data is daily and can take up to 7 days to recover. Gmail's Feedback Loop data covers only @gmail.com recipients.
  • Gmail mitigation: eligibility returns once the spam rate stays below 0.3% for 7 consecutive days.
  • SNDS on shared infrastructure: because SNDS is per IP and requires you to prove you control the address space, senders on shared Workspace or Microsoft 365 IPs get nothing from it. That is my inference from how it works. The portal has been in transition and unreliable, so check its current state before relying on it.
  • JMRP timing: feedback starts within as little as 72 hours of enrolment, too slow for a one-day diagnosis.
  • Yahoo's denominator: Yahoo calculates spam rate on mail delivered to the inbox, which you cannot observe. When more mail goes to spam, the denominator shrinks, and the complaint rate can rise with no new complaints.
  • Yahoo unsubscribes: these must be honoured within 2 days. Late handling shows up in complaints, not replies.

How this diagnosis goes wrong

Most failed diagnoses have the right data and read it wrongly. Each failure mode below gives what the misleading evidence looks like and what to check instead.

  1. Trusting the opens rule. Looks like: opens flat, replies down, so the team calls it copy. The lie: with tracking off or Apple MPP inflating opens, flat opens say nothing about placement. Check instead: provider-split replies and bounce strings.
  2. Reading a provider split as proof. Looks like: "Outlook replies halved." The lie: at 175 per window, z ≈ 1.35 and p ≈ 0.18. Check instead: hard evidence, meaning 5.7.515 counts and a seed in Junk, not reply-rate p-values.
  3. Trusting empty dashboards. Looks like: Postmaster shows "No data" or a spam rate of 0. The lie: you are below an unpublished volume floor, or mail is being spam-foldered. Check instead: daily volume to Gmail first.
  4. Assuming SNDS covers you. Looks like: SNDS shows nothing, so the IP looks fine. The lie: SNDS is per IP and needs control of the address space, and shared mailboxes produce no data. Check instead: who owns the egress IP, the address your mail actually leaves from.
  5. Trusting seeds as a population estimate. Looks like: "92% inbox placement." The lie: seeds have no engagement history, and one source illustrates a 90% seed reading while real subscribers see 60% spam-folder rates. That is an illustration, not a measured rate. Suped warns that a larger panel does not justify a population margin of error. Check instead: report seeds per provider and per domain only.
  6. Taking the copy-first wrong turn. Looks like: a rewrite ships in the bad window and replies wobble. The lie: you have added a confounder while the real cause keeps running. Check instead: park copy until passes 1 to 6 are recorded.
  7. Missing segment drift. Looks like: clean infrastructure, falling replies, so the team calls it "copy fatigue." The lie: a sourcing change can halve replies by itself. Check instead: mix by title, seniority, company size and source.
  8. Underestimating bulk status. Looks like: "We only send 300 a day, the rules don't apply." The lie: Gmail bulk status is permanent once reached, and Microsoft counts every stream sharing the From domain. Check instead: historical peaks and every system sending from that domain.

How do you sign off the cause and keep the check current?

Sign off only when one suspect has positive evidence against it and each of the other four has a written reason it is ruled out. Then put the passes on a schedule, because provider rules and dashboards change without notice.

Reply-drop root-cause sign-off note
Sequence:
Windows compared: [before dates] vs [after dates]
Sends per window: before ___ / after ___
Replies per window: before ___ / after ___
Whole-sequence z and p: z ___ , p ___
Reply definition and denominator (frozen):
Tracking settings changed between windows: yes / no (detail)
Unlogged replies found in mailboxes: ___
Domain carrying the drop:
5.7.515 count by domain:
Other 5.7.x count by domain:
Provider cells and labels (signal or noise):
Seed results per provider per domain:
Segment mix equivalent: yes / no (attach mix table)
Copy changes during windows, with dates:
Named cause:
Evidence against each other suspect:
- Report:
- Routing:
- Placement:
- List drift:
- Copy:
Remediation and date of clean re-measure:
Signed off by:

Fill every line; "not checked" is an allowed answer but blocks sign-off.

To keep the check current, re-read the sources rather than relying on the values in this guide: Yahoo Sender Best Practices, Microsoft's high-volume sender announcement, and Google's Postmaster Tools documentation. Re-check three things each time: the volume trigger, the failure response code, and what each dashboard still shows. If any has changed, update Table A and the log search strings in pass 3 before the next diagnosis.

Before you call the cause proven

  • The reply definition and denominator are identical across both windows, and any tracking change is logged.
  • Raw reply counts have been compared against the prior 30 days, not only rates.
  • Every sending mailbox has been checked for unlogged replies, and the count is written down.
  • A per-domain table exists with sends, replies, bounces, and 5.7.515 and 5.7.x counts.
  • Every provider cell is labelled signal or noise against the Table B sample sizes.
  • Seed results are reported per provider and per domain, with no population percentage.
  • Segment mix by title, seniority, company size and source matches across windows, or the difference is explained.
  • Copy was read only inside unaffected cells, and any mid-window rewrite is dated.
  • The sign-off note names one cause and gives written evidence against each of the other four.

Questions practitioners ask

Why did my cold email reply rate drop suddenly when bounces look normal?

Mail moved to the junk folder never bounces, so your bounce rate can stay flat while replies fall. Outlook may junk mail from a domain before it starts rejecting with 550 5.7.515, though sources disagree on which comes first. Split sends and replies by sending domain, search the bounce logs for 5.7.515 and 5.7.x strings, then run a seed test per domain. A drop that sits on one domain points at placement, not copy.

How do I check if my emails are going to spam in Outlook or Gmail?

Run a seed test: send the same production message through the same sending path at nearly the same time as the real campaign, to test inboxes at each provider. For Outlook, also search your bounce logs for 5.7.515. Gmail's Postmaster Tools v2 no longer shows domain or IP reputation and shows nothing at low volume. A spam rate that drops to 0 after being high can mean your mail now goes to spam.

Is a reply rate drop a deliverability problem or a targeting problem?

You can tell by where the drop sits. A placement problem shows up on particular sending domains or recipient providers, with hard evidence like rejection strings or seeds in junk. A targeting problem shows up evenly across domains, alongside a change in the title, seniority, company size or source mix between windows. Rule out measurement and reply routing before either, because both are cheap and give a yes or no answer.

How many recipients do I need before a halved reply rate is statistically real?

It depends on the baseline. With a two-sided 0.05 test at 80% power, a halving from 6% to 3% needs about 749 recipients per window, from 4% to 2% about 1,141, and from 2% to 1% about 2,318. A 700/700 sequence falling from 6% to 3% still comes out significant, at z ≈ 2.71. A 175-recipient provider cell does not, so read those cells for hard evidence only.

Can Google Postmaster Tools or Microsoft SNDS diagnose a cold outreach drop?

Usually not. Postmaster Tools v2 dropped the reputation dashboards and shows no data below a minimum volume Google doesn't publish. Practitioners estimate about 100 messages a day for any data and about 1,000 for a reliable spam chart. SNDS reports per IP and requires you to prove you control the IP space, which senders on shared mailbox infrastructure cannot do. Bounce logs and seeds are the practical surfaces.

Should I rewrite the copy when outreach replies fall?

Not until every other cause has been checked and written down. Changing copy while mail goes to spam wastes the rewrite and adds a confounding variable, so you can no longer tell which change moved the number. Copy is identical across sending domains, so a drop concentrated on one domain cannot be a copy problem. Read copy last, by step, inside the unaffected cells only.

Try it on the search you came here for

Stop building boolean strings. Just describe the person.

Type one sentence. I plan the search, read GitHub, public LinkedIn and Crunchbase records, and the open web as it is right now, and hand back a ranked list with the reason next to every name.

  1. 01Describe them

    One plain sentence. Role, city, stack, stage, whatever matters to you.

  2. 02I read the web live

    GitHub, public LinkedIn and Crunchbase records, the open web. Not a database that went stale last quarter.

  3. 03You read the shortlist

    Ranked, with the reasoning under every name. Open a profile, ask a follow-up, narrow it down.

  • No boolean, no filters, no seat to buy. One box.
  • Read at search time, so a profile updated yesterday counts today.
  • Every step visible as it runs, every name with its reason.

500 free credits on sign-up. No card, no demo call. See real searches.

Read next