# The Target Split Test, Two Job Segments Sent in Matched Batches to a Verdict

*Set up two job targets as matched arms, size them from your own interview rate, send on a controlled schedule, and stop on a written rule with a dated shift, split, or hold.*

- Canonical URL: https://www.refolk.ai/candidates/guides/target-split-test-job-segments
- Pillar: Reading the market
- Format: Playbook
- Published: 2026-11-01
- Last reviewed: 2026-11-01
- Reading time: 26 min
- Keywords: which job title gets more interviews, test two job targets applications, how many applications to know if a target is working, is it my resume or the market interview rate, callback rate by job title level test, should I apply down a level test

## Key takeaways

- A conventional test needs about 20 interviews in the weaker arm, which means 432 applications per arm at a 5 percent baseline to detect a doubling, and 1,138 per arm at a 2 percent baseline.
- In Huntr's tracked data, send day, month, and remote status each move interview rates by about 1.56x. That is as large as the +50 percent lift a target test is trying to detect, so all three have to be balanced between the arms.
- Career-page applications reached an interview 6.87 percent of the time in Huntr's data, against 1.95 percent on LinkedIn. Limiting both arms to career sites removes a 3.52x confounder and also raises the baseline your sample size depends on.
- In Refolk's index, only 52 percent of a 25-profile sample of US Program Managers carried that exact title, so an arm defined by a title keyword pulls in mixed roles and has to be defined by what the posting actually says.
- Under Evan Miller's sequential rule, written down before the first send, you declare a winner only when one arm leads by 2 times the square root of the pre-set total N. If total interviews reach N first, you stop with no winner.
- Re-running a naive significance test after every new result produced a false positive in 47.8 percent of simulated streams, which is why checking a job-search test every day manufactures winners.

This guide settles a targeting question with real applications instead of guesswork: title A or title B, your current level or one down, this sector or metro or that one. It is for a job seeker who has a working resume and a few weeks of tracked applications. It ends in a dated decision to shift, split, or hold your target, with the two-by-two table that backs it.

## What a target split test answers

A target split test sends real applications to two defined job segments under the same conditions and compares how often each one earns an interview. It answers a market question (which segment wants you), not a formatting question (which resume layout reads better).

Three terms carry the method:

- **Arm**: one of the two segments under test, defined by a written inclusion rule, for example "senior product manager, fintech, hybrid or on-site, applied through the employer's career site."
- **Baseline rate**: your own interviews divided by applications over a recent tracked period on one channel. Everything else is sized from this number.
- **Verdict**: the dated decision at the end. Shift means you move your main target to the winning arm. Split means you keep sending to both. Hold means you keep your current target and look elsewhere for the problem.

Most public "A/B test your job search" advice tests resume versions, and the samples it suggests are far too small. One common recommendation is to send each version "to at least 20 similar job listings before drawing conclusions." At a 5 percent baseline, 20 sends expect one interview. A 2-to-0 result at that size is noise that looks like a winner. Kickresume's 200-application experiment sent its versions "in phases rather than head-to-head", so the calendar changed between conditions and each phase ran in a different market.

The design here borrows from the largest field experiment on hiring callbacks, by Kline, Rose and Walters, who sent 83,643 applications to 11,114 jobs. Three of their rules carry over to one person:

1. **Distinct decision-makers.** They spread each firm's vacancies across counties to "minimize the chances that multiple sampled job vacancies were being managed by the same individual." For you, that means at most one application per company across both arms.
2. **Waves with both conditions in each.** Their sampling "was organized in a series of 5 waves," and every wave carried both conditions. For you, both arms get sends every week.
3. **A design written down before sending.** They worked from a registered design, which means the plan was fixed before any results arrived. For you, that is a dated document.

One rule does not carry over. They sent matched pairs of applications to the same job. You are one person with one name, so each posting gets at most one application from you.

The test cannot tell you why a gap exists. A gap means the market responded differently to you in two segments under matched conditions. It does not say whether the cause is pay band, keyword fit, competition, or a hiring freeze in one sector. It also answers "is it my resume or the market" only partly. If both arms come in below your baseline, the target is probably not the problem.

> **Watch out:** Twenty sends per arm is not a test
>
> At a 5 percent baseline, 20 applications per arm expect one interview each, so any split you see is chance. Before you read a single count, check your sends against the sizing table in the next section.

## How many applications each arm needs

Each arm needs enough applications that the weaker arm should produce about 20 interviews. At a 5 percent baseline, that is 432 applications per arm to detect a doubling under conventional settings, or 248 under relaxed settings.

Two definitions make the table readable. **Lift** is how much better the challenger arm has to be for the test to reliably detect it: "2x" means double your baseline, "+50%" means one and a half times. **Conventional settings** mean a two-sided test with a 5 percent false-alarm rate and 80 percent power. Power is the chance of detecting a real gap of the stated size. **Relaxed settings** mean a one-sided test at 10 percent with the same power: you accept more false alarms and only ask whether the challenger beats the current target.

| Baseline, lift | Per arm, conventional | Per arm, relaxed | Control-arm interviews (conventional) |
|---|---|---|---|
| 2%, 2x | 1,138 | 654 | 23 |
| 2%, +50% | 3,822 | 2,196 | 76 |
| 5%, 2x | 432 | 248 | 22 |
| 5%, +50% | 1,468 | 843 | 73 |
| 10%, 2x | 197 | 113 | 20 |
| 10%, +50% | 683 | 393 | 68 |

Every figure comes from the standard two-proportion sample-size formula. Look at the last column first. Whatever the baseline, a conventional doubling test needs about 20 or more interviews in the control arm, and a +50 percent test needs roughly 70. That is the useful way to plan: decide how many interviews you need to see, then work back to how many applications that takes at your rate.

> Size the test in interviews, not applications. Decide how many you need to see, then work back to sends.

### When the number is more than you can send

If the per-arm count is more than about 8 weeks of your sending capacity, you have three honest options:

- **Widen the contrast.** Test two segments different enough that a doubling is plausible, such as two sectors rather than two neighbouring titles. Then use the 2x rows.
- **Accept relaxed settings.** This roughly halves the sample, at the cost of more false winners. Say so in the verdict.
- **Accept that a gap smaller than the one you sized for will not show up.** A test sized for a doubling cannot see a +50 percent gap. If it ends with no winner, report that as the finding, not as a failure.

### Two things that move your baseline

**Channel moves the baseline a lot.** Across 1.24 million applications on 25 job sites in Huntr's tracked data, applications through a company's own careers page reached an interview 6.87 percent of the time, against 2.92 percent on Indeed and 1.95 percent on LinkedIn. One of Huntr's quarterly reports found employer career sites had 1.7x the interview rate of job boards. Limiting both arms to career sites moves many job seekers from the 2 percent rows of the table toward the 5 percent rows, which cuts the sends needed for a doubling from 1,138 to 432 per arm.

**Volume may push it down.** In one Huntr quarter, users with 11 to 20 applications had an 8.12 percent per-application rate, against 2.86 percent for users with 100 or more. Another quarter showed 9.25 percent against 2.58 percent. Huntr itself cautions that its results "reflect correlations rather than proof," and the cause is not established. Matched arms protect the comparison, because both arms are affected the same way, but a test that forces high volume may lower both rates and stretch the sample further. Re-measure your pooled rate partway through so the verdict uses the rate you actually had during the test.

Do not compare your baseline with employer-side figures. CareerPlug's recruiting metrics report put the average applicant-to-interview conversion at 15 percent, but that is counted per applicant at small employers and is not comparable with a job seeker's own log.

> **Tip:** Fix the channel to career sites before sizing
>
> Applying through employer career sites removes the largest confounder in the data and raises the baseline your sample size is calculated from. Leave referrals out of both arms too: a referral rate that circulates secondhand is far higher than any cold channel, and one referral in one arm can swing the count.

## Which variables to hold fixed between the arms

Hold channel, send day, season, work mode, posting age, and company the same across both arms. In Huntr's tracked data, send day, month, and work mode each move interview rates by about 1.56x, and channel by up to 3.52x. Any one of them left unbalanced can create a gap as large as the one you are testing for.

| Variable | High | Low | Ratio |
|---|---|---|---|
| Send day (Tue vs Sat) | 3.9% | 2.5% | 1.56x |
| Month (Nov vs Jan) | 4.2% | 2.7% | 1.56x |
| Work mode (hybrid/on-site vs remote) | 5.3% | 3.4% | 1.56x |
| Channel (career page vs LinkedIn) | 6.87% | 1.95% | 3.52x |
| Volume (11 to 20 vs 100+ apps) | 8.12% | 2.86% | 2.84x |

Rates are from Huntr's published research. The ratios are derived. The work-mode figures are marked preliminary in their source.

**1.56x - Gap from send day, month, or remote status alone in Huntr's data**

That is larger than the +50 percent lift a target test is trying to detect, so an unbalanced weekday mix can produce a fake winner.

#### What the test holds fixed, outermost first

1. **Season** - Both arms run in the same weeks, never one after the other
2. **Channel** - Employer career site only, in both arms
3. **Send day** - Weekdays only, equal counts per arm on each send day
4. **Posting** - 48 hours old or less at send, same work-mode rule in both arms
5. **Company** - One application per company across both arms combined

*Each layer is a known confounder, and every application in both arms has to pass all five before it counts.*

### Channel

If one arm's jobs are mostly reachable through LinkedIn Easy Apply and the other's through career pages, the channel alone can produce a 3.52x gap. Write "career site only" into both inclusion rules. If a posting has no career-site route, it does not enter either arm.

### Send day and season

In Huntr's data, Tuesday had the highest recorded interview rate at 3.9 percent per application, and Saturday and Sunday the lowest at 2.5 and 2.6 percent. January converted at 2.7 percent, the lowest month, and November reached 4.2 percent. You cannot control the season, but you can make sure both arms experience the same one. Run them at the same time and interleave them, as Kline, Rose and Walters did within every wave, rather than in phases. Send on weekdays only, with equal counts per arm on each send day.

### Posting age

Hold posting age at 48 hours or less. Treat this as a control, not as a way to raise your odds. The only rigorous timing evidence is NBER working paper w32320 by Davis and Samaniego de la Parra, which links 125 million applications to vacancies on Dice.com. Postings there are "often only two or three days, with a median of seven days" (mean 9.4), with "almost half of applications flowing to openings posted in the past 48 hours." An earlier version of the paper found 41 percent of applications arrive within 48 hours and 56 percent within 96.

That research measures how applications flow in. It does not measure how posting age changes your chance of an interview, and that effect has not been established publicly. A widely repeated claim of an "8x" advantage within 96 hours reports its sample as both 1,600 and over 160,000 applications and publishes no method, so I leave it out. Kickresume's small test found response rates "stayed inside a narrow band" whatever the posting age. The 48-hour rule is there so one arm does not end up full of stale postings while the other gets fresh ones.

### One application per company

Huntr finds per-application rates fall to 1.85 percent when someone sends 8 or more applications to one company. Repeat applications also bunch responses inside one employer, so one recruiter's decision counts several times. Apply to each company once across both arms combined. On Dice.com, recruitment and staffing firms "account for two-thirds of all postings," so decide in your design whether agency postings are excluded, or included with each agency counted as one company. Apply the same choice to both arms.

### Work mode and country

Huntr's preliminary figures put remote-only applications at 3.4 percent against 5.3 percent for hybrid and on-site, and own-country applications at 6.2 percent against 4 percent for positions abroad. If one arm is mostly remote and the other mostly on-site, you are testing work mode, not the target. If your question is metro versus metro across a border, the country gap is part of what you are measuring. Name it as part of the contrast rather than treating it as noise.

## How to define an arm so it holds the same jobs

Define each arm by what the posting says, covering title family, level, sector, metro, and work mode, so a stranger could sort any posting into A, B, or neither. A rule built on the title string alone lets mixed roles into the arm, and they dilute the comparison.

Title strings are unreliable even among people. In Refolk's index of professional profiles, here is how three segments compare:

| Segment | Headcount | Exact-title share in sample | Multiple vs US PM |
|---|---|---|---|
| Product Manager, US | 67,494 | 88% | 1.00x |
| Program Manager, US | 139,984 | 52% | 2.07x |
| Product Manager, UK | 15,749 | 80% | 0.23x |

The exact-title share comes from a 25-profile sample per segment, and the multiples are derived.

**52% - Share of a US Program Manager sample in Refolk's index whose title exactly matched**

Nearly half the people a "Program Manager" keyword catches hold some other title, and postings that use the word are likely just as mixed.

That last point is my inference from profile data, not a measurement of postings, so check it yourself. Pull 10 postings that your keyword search returns for each arm and sort them against the written rule. If fewer than most of them fit, the rule needs tightening or the keyword needs replacing.

### Headcount is not demand

Refolk's index shows 2.07x more people in the US matching Program Manager than holding the Product Manager title, and the UK Product Manager pool is 0.23x the US one. None of that tells you your interview odds. A bigger pool of people with a title can mean more openings, more competition, or both. That is why the question has to be settled by sending applications.

### Writing a level test

"Should I apply down a level" is the most common version of this test, and the one where title strings mislead most. A "Senior" title at one company can be a mid-level job at another. Define level by the scope in the posting: who the role reports to, what it owns, team size, budget, and years asked for. Write those markers into each rule. Keep sector, metro, and work mode identical across the two arms so that level is the only thing that differs.

Sorting postings into arms by content is the slow part at this volume. [Refolk](/candidates) scores how well each posting fits your own history, which gives you a consistent first read on scope and level before you check the posting against the written rule yourself.

## The split test procedure, step by step

The procedure runs from baseline to verdict in seven steps. Setup takes about two days, and the sending phase runs for as many weeks as your sizing says.

#### Target split test, baseline to verdict

1. **Measure your baseline** - Divide interviews by applications over your last 8 to 12 weeks of tracked applications on one channel, ideally career sites. Done is one rate with its date range, in about an hour.
2. **Define the two arms** - Write an inclusion rule for each arm covering title family, level, sector, metro, and work mode, judged on what the posting says, with the channel limited to career sites. Done is two rules a stranger could apply, in about a day.
3. **Size the arms** - Look up the per-arm count for your baseline and target lift in the sizing table. If it is more than about 8 weeks of capacity, widen the contrast or accept relaxed settings; done is a per-arm target and an end date.
4. **Write the rules down before the first send** - Record the stopping rule, response codes, response window, and the shift, split, or hold decision rule. Done is a document dated before your first send.
5. **Send in matched batches** - Each weekday, send equal numbers to each arm, alternate which arm goes first, use only postings 48 hours old or less, and apply to each company once across both arms. Done is equal counts per arm every week.
6. **Code every response** - Log each response under the fixed codes the day it arrives, and close each application when its window ends. This takes minutes per day.
7. **Read the result and call it** - Check the stopping rule weekly, never daily. At the stop, write a dated verdict of shift, split, or hold with the two-by-two table.

> **Rule:** The design document is dated before the first send
>
> The stopping rule, the response codes, the window, and the decision rule are all fixed before any result exists. Once results come in, you may only add entries to the change log, and every change is reported in the verdict.

The design document is the registration. Copy it, fill it in, and date it.

**Split test design document**

```
TARGET SPLIT TEST          Registered on: [date]
Question: Do [arm A] or [arm B] postings earn me more interviews?

BASELINE
Interviews / applications: [x] / [y] = [rate]%
Channel: employer career site     Period: [start] to [end]

ARM A INCLUSION RULE
Title family:
Level (scope markers from the posting, not the title):
Sector:
Metro / country:
Work mode:
Exclusions:

ARM B INCLUSION RULE
Title family:
Level (scope markers from the posting, not the title):
Sector:
Metro / country:
Work mode:
Exclusions:

SHARED CONTROLS
Channel: employer career site only, no referrals
Posting age at send: 48 hours or less
Send days: weekdays only, equal counts per arm, alternate which arm goes first
Companies: one application per company across both arms
Staffing agencies: [excluded] or [included, each agency counted as one company]
Resume: same tailoring process for both arms

SIZING
Target lift: [2x or +50%]     Settings: [conventional or relaxed]
Sends per arm: [n]     Weekly capacity: [n]     Planned end: [date]

STOPPING RULE (tick one)
[ ] Sequential: N = [total interviews]. Winner when one arm leads by [2 x sqrt(N), rounded up]
    (one-sided) or [2.25 x sqrt(N), rounded up] (two-sided). If A + B reaches N first: stop, no winner.
[ ] Fixed: stop at [n] sends per arm, run Fisher's exact test once at alpha [0.05 or 0.10].

RESPONSE CODING
Codes: I, V, S, R, N     Primary count: strict (I)     Secondary: broad (I + V + S)
Response window: [days] after each send, identical for both arms

DECISION RULE
Winner, pooled rate at or above baseline -> SHIFT to the winner
Winner, pooled rate below baseline -> SHIFT, then repair materials
No winner, pooled rate at or above baseline -> SPLIT
No winner, pooled rate below baseline -> HOLD, look upstream

CHANGE LOG
[date] [change] [reason]
```

*Fill every bracket before the first send, then date it; after that, only the change log at the bottom may grow.*

### One send day

Each send day follows the same small routine. The point is that, whatever the arms differ on, they never differ on the day, the order, the posting age, or the company.

#### One send day in a matched batch

1. **Pull postings** - Career-site postings 48 hours old or less that fit either arm's written rule
2. **Screen companies** - Drop any company already used in either arm
3. **Balance** - Take equal numbers for A and B, capped by the arm with fewer matches
4. **Alternate order** - Arm A goes first on odd send days, arm B first on even days
5. **Log** - Record arm, company, posting date, send date, and weekday before moving on

*The arm with fewer qualifying postings that day sets the batch size for both arms.*

Tailoring has to be the same process in both arms, or the resume becomes a second variable. Refolk tailors the resume and cover letter to each posting from the same underlying history, so both arms get identical treatment and a few hundred tailored sends do not take over your evenings.

Keep one row per application in a log. These columns are enough to run every check in the failure-mode section.

**Send and response log, one row per application**

```
send_date,weekday,arm,first_in_batch,company,posting_title,posting_date,posting_age_hours,channel,work_mode,country,rule_check_ok,response_code,response_date,days_to_response,window_close_date
```

*Paste the header into a spreadsheet; add a row at send time and fill the response columns as they arrive.*

## How to read the counts and call the verdict

Read the counts only through the stopping rule you wrote down: either Evan Miller's sequential threshold on interview counts, or a fixed sample size followed by one Fisher's exact test. Then place the result on the verdict grid and write shift, split, or hold with the date.

### Why not just check every day

Miller's own warning is direct: "stopping an A/B test early because the results are statistically significant is usually a bad idea." In one public simulation, re-running a naive significance test after every new observation produced a false positive in 47.8 percent of streams. Etsy's engineering team has written about the same peeking problem in product testing. In a job search the temptation is worse, because every interview feels like news. Look weekly, and act only on the pre-set threshold.

### Option 1: Miller's sequential rule

Miller's procedure counts successes, which here means interviews, so it fits the plan of thinking in interviews:

1. Before the first send, choose N, the total number of interviews across both arms at which the test ends.
2. Keep a running count of interviews in each arm.
3. If one arm's lead reaches 2√N, stop and declare it the winner. For a two-sided test, use about 2.25√N.
4. If the two counts together reach N first, stop and declare no winner.

As arithmetic: with N set at 64, the square root is 8, so a winner needs a 16-interview lead one-sided, or 18 two-sided. Miller's rule compares raw counts, so it only means something if both arms received the same number of sends. That is the reason the batches are matched. When you set N, check it against the last column of the sizing table: if your N implies far fewer interviews than that column, the test is too small to see the gap you care about.

### Option 2: fixed sample, then Fisher's exact test

Send the per-arm count from the sizing table, let every window close, and fill in the two-by-two table: interviews and non-interviews for each arm. Run Fisher's exact test once, which "can be employed to compare two binomial distributions," at the alpha you wrote down. One test, at the end, and nothing read in between. No one has published a Bayesian stopping rule calibrated for job search, so I do not recommend improvising one.

### Coding responses

No public source says how to code each response type. Huntr counts whether an application "led to at least one interview" without publishing what it accepts as one, and Kline, Rose and Walters scored any "contact" (24 percent of their applications were contacted). This is the coding scheme I use. Report both the strict and broad counts.

| Code | Response | Strict | Broad |
|---|---|---|---|
| I | Live screen or interview with a person scheduled | Counts | Counts |
| V | One-way video: recorded answers, no interviewer present | No | Counts |
| S | Automated or AI screening step | No | Counts |
| R | Rejection | No | No |
| N | Nothing by window close | No | No |

The response window is the number of days after each send during which a response counts. Choose it from your baseline log: long enough to cover most of the first responses you have had in the past. Apply it per application, counted from that application's send date, so an arm that started a day later is not cut short.

### The verdict grid

#### Reading the verdict

Horizontal axis runs from No winner declared to Winner declared. Vertical axis runs from Pooled rate below baseline to Pooled rate at or above baseline.

| Quadrant | What it means |
| --- | --- |
| Split | Both segments respond at your usual rate, so keep sending to both and double your posting pool |
| Shift | Move your main target to the winning arm; if the winner is your current target, this is a hold |
| Hold and look upstream | The target is not the problem, so check resume, channel, and volume before retesting |
| Shift and repair | Move to the winner, then fix the materials, since both arms underperform your baseline |

*The stopping rule decides left or right, and your pooled rate against your own baseline decides top or bottom.*

The bottom-left quadrant is the closest this test gets to answering "is it my resume or the market." Two arms that both fall short of your own baseline, with no gap between them, point away from the target and toward something both arms share. Write the verdict as one line with the date, the quadrant, the two-by-two table, the strict and broad counts, and any entries from the change log.

## How the split test goes wrong

Most failed split tests do not fail on the arithmetic. They fail because something other than the target differed between the arms, or because someone read the counts before the rule allowed it. Each failure below produces a winner that is not real, and each has a check you can run on the log in minutes.

1. **Arms that are too small.**
   - Looks like: at 20 sends per arm on a 5 percent baseline, each arm expects 1 interview, and a 2-to-0 count reads as a clear winner.
   - Check: compare your sends per arm against the sizing table before reading any count. If you are under it, the verdict says "underpowered" and nothing else.
2. **Peeking.**
   - Looks like: an arm "wins" on the day you happened to look. Naive daily re-testing produced false positives in 47.8 percent of simulated streams.
   - Check: act only on the pre-written Miller threshold or the fixed sample size. Your design document's date must be earlier than your first send.
3. **Uneven send days or months.**
   - Looks like: one arm sent mostly on Tuesdays and the other on weekends shows a 1.56x gap with no real difference between targets. The same happens if one arm ran in January and the other in November.
   - Check: count sends by weekday for each arm. The counts should match within each week, and both arms should span the same weeks.
4. **Channel drift.**
   - Looks like: one arm's jobs are only reachable through LinkedIn Easy Apply, so the channel alone creates a gap of up to 3.5x.
   - Check: the share of career-site applications must be the same in both arms. Under this design, that share is 100 percent.
5. **Arms defined by title string.**
   - Looks like: only 52 percent of the US Program Manager sample in Refolk's index carried that exact title, so a keyword-defined arm fills with mixed roles and its rate describes no single target.
   - Check: audit 10 postings per arm against the written inclusion rule and record how many pass.
6. **Repeat applications to one company.**
   - Looks like: one employer's recruiter responds to three of your applications in the same arm, and one decision counts three times. Repeat sends also drag the per-application rate toward the 1.85 percent Huntr sees at 8 or more applications per company.
   - Check: count unique companies per arm. It should equal sends per arm, with no company appearing in both arms.
7. **Inconsistent coding.**
   - Looks like: one-way video invites counted as interviews in one arm only, which inflates that arm.
   - Check: score each arm under both the strict and the broad definition and report both. If the winner changes between them, the result rests on how you coded responses, not on the market.
8. **Late responses.**
   - Looks like: the arm that started later is still collecting responses when you count, so it looks worse than it is. Or responses that arrive after the window get counted for one arm and not the other.
   - Check: close both arms using the same window, counted from each application's own send date, and leave out anything that arrives later from both.

> **Watch out:** Unbalanced work mode is the quiet one
>
> Remote-only applications ran at 3.4 percent against 5.3 percent for hybrid and on-site in Huntr's preliminary figures. If one arm's postings skew remote, you are testing work mode under another name, so put the same work-mode rule in both arms.

## Keeping the verdict current

A verdict describes the market as it was during the weeks you sent. Date it, name the date you will next check your baseline, and rerun the test when the conditions it depended on change.

What to do after each verdict:

- **After a shift**, your old baseline no longer applies. Measure a fresh baseline in the new target over the next 8 to 12 weeks of career-site sends, because the next test will be sized from it.
- **After a split**, keep both arms in your weekly sends and keep logging them separately. The ongoing log will show you if one arm starts to fade.
- **After a hold**, change one upstream thing at a time (resume, channel mix, or volume), and measure the baseline again before running another target test.
- **When the season turns**, remember that a test run entirely in a low month and one run entirely in a high month are not comparable. Compare verdicts only within similar months.
- **When your pooled rate moves** well away from the baseline you sized from, re-look up the per-arm count for the next test instead of reusing the old one.

#### Before you call the verdict

- [ ] The design document is dated earlier than the first send in the log.
- [ ] Each arm's send count meets the sizing table figure for your baseline and lift, or the verdict says it is underpowered.
- [ ] Sends per weekday are equal across the two arms within every week, and both arms span the same weeks.
- [ ] Every application in both arms went through an employer career site.
- [ ] Every posting was 48 hours old or less on the day it was sent.
- [ ] No company appears more than once across both arms combined.
- [ ] Ten postings per arm were audited against the written inclusion rule, with the pass count recorded.
- [ ] Both arms were closed using the same response window, counted from each application's own send date.
- [ ] The verdict reports the two-by-two table and both the strict and broad interview counts.
- [ ] The verdict is dated, names its quadrant, lists any change-log entries, and sets the next baseline check date.

## Frequently asked questions

### How many applications do I need to know if a job target is working?

Plan in interviews, not applications. A conventional test for a doubling needs roughly 20 interviews in the weaker arm. That works out to about 432 applications per arm at a 5 percent baseline, 197 at 10 percent, and 1,138 at 2 percent. Relaxed settings (one-sided, 0.10) cut those numbers to 248, 113, and 654. If your number is more than about eight weeks of sending capacity, widen the contrast between the targets or accept the relaxed settings.

### Is it my resume or the market if I am not getting interviews?

A split test can partly separate the two. If both arms come in below your own baseline and neither wins, the target is probably not the problem. Keep the target and look upstream at your resume, channel, and volume. If one arm clearly wins under your stopping rule, the market is treating the two segments differently. The test tells you that a segment responded under matched conditions, not why.

### Should I apply down a level, and how do I test it?

Run your current level and one level down as two arms. Define each by the scope described in the posting, not the title string, and send both through career sites on matched weekdays. Keep sector, metro, and work mode the same in both arms. Size the test from your baseline, write the stopping rule first, and only shift down if the lower arm wins under that rule. A no-winner result means stepping down did not get you more interviews.

### Can I compare two job titles with 20 applications each?

No. At a 5 percent baseline, 20 sends expect one interview per arm, so a 2-to-0 result is well within chance. The smallest conventional test in the sizing table, a doubling from a 10 percent baseline, still needs 197 applications per arm. Twenty sends can catch a broken resume or a dead channel if nothing at all comes back, but they cannot rank two targets.

### Does applying within 48 hours of a posting going up get more interviews?

That has not been shown publicly. Research on 125 million Dice.com applications finds that almost half of applications go to openings posted in the past 48 hours and that the median posting lasts seven days, but it measures how applications flow in, not your odds of an interview. A widely quoted '8x' claim reports inconsistent sample sizes and no published method. Use the 48-hour rule to keep the arms comparable, not to boost your odds.

### Does a one-way video interview count as an interview in the test?

Count it in the broad tally but not the strict one, and report both. No public source defines how to code response types, and Huntr does not publish what it counts as an interview. A one-way video, where you record answers to set questions with no interviewer present, is a screening step. If you count it in only one arm, that arm looks better than it is, so apply the same codes to both.

---

*From the Refolk guide library. I revise these guides rather than replacing them, so the current version is always at https://www.refolk.ai/candidates/guides/target-split-test-job-segments*