# The People-Search Bake-Off Playbook: Filled Reqs to a Scored Verdict

*Run any people-search tool against five to ten filled reqs and report recall, precision with intervals, fact accuracy, and cost per qualified profile in one verdict table.*

- Canonical URL: https://www.refolk.ai/guides/people-search-tool-bake-off
- Pillar: Process, data, and compliance
- Format: Playbook
- Published: 2026-11-13
- Last reviewed: 2026-11-13
- Reading time: 22 min
- Keywords: how to evaluate an AI sourcing tool, AI sourcing tool pilot test plan, compare candidate search tools accuracy, sourcing tool precision test, retrospective test placed candidates sourcing tool, people search tool bake-off

## Key takeaways

- Five to ten filled reqs graded at precision at 20 can only separate two sourcing tools whose precision differs by about 15 points; a 5-point gap needs roughly 46 briefs.
- In Refolk's index, a plain title query for UK sourcers returned at least 5 off-function profiles in 25 (20%), against 2 in 25 (8%) in the US, because "sourcing" is also a procurement job family.
- One vendor's own people-search benchmark moved from 63.3% to 86.2% precision while its answered-query count fell from 1,399 to 1,323, so a vendor-published score is not evidence about your reqs.
- Grade a pooled, de-labelled top 20 from every tool, and do not start full grading until two graders reach a Cohen's kappa of at least 0.61 on about 30 calibration profiles.
- Score recall on non-hired finalists separately from recall on hires, because a past hire who now lists your company can be found by the employer keyword alone.
- A published buyer cut line for sourcing tools is no more than 20 to 25% irrelevant results in the top 20, where irrelevant means wrong seniority, wrong geography, or wrong function.

This is the protocol for testing a people-search or AI sourcing tool against your current process before you buy or renew it. It's for recruiting operations, revenue operations, and whoever finance or legal will ask how the tool was chosen. It ends in one comparison table and a stated verdict: adopt, extend the pilot, or drop.

The test runs on your own history. You take five to ten reqs you've already filled, and the hire and finalists on each are the known answers. It covers retrieval quality only. How to order contact-data enrichment sources is in the Waterfall Order Playbook, and compliance logging for the screening stage belongs in your own evidence trail.

## What does a people-search bake-off measure?

A bake-off checks whether a tool returns the right people for briefs where you already know the answer, and what each right person costs. Every tool runs on the same frozen briefs and gets the same blind grades, and you report five things for each one.

A few terms first:

- **Recall@20**: the share of known-answer people who appear anywhere in a tool's top 20 results.
- **Precision at 10 / at 20 (P@10, P@20)**: the share of the top 10 or top 20 results that blind graders mark relevant.
- **Known-answer set**: past reqs where you already know who was right, namely the hire and the finalists.
- **Cost per qualified profile**: licence cost divided by the number of profiles graded relevant.

| Metric | What it proves | How it lies |
|---|---|---|
| Recall@20, hires | The tool can find the person you placed | Hire now lists your company, so the employer keyword does the work |
| Recall@20, finalists | It finds the shortlist, not one lucky name | Few finalists per req, so one hit moves the rate a lot |
| P@10 and P@20 | Graders would contact most of what it returns | A vague rubric makes precision depend on who graded |
| Profile-fact accuracy | Employer, title, and location are current | Right person, wrong current employer |
| Cost per qualified profile | What a usable result costs | A tool returning few results looks cheap per seat |

One published buyer check is worth adding to the hire line: does the tool put the candidate you actually placed in its **top 5** results? It's a blunt test and easy to explain to a hiring manager.

## Why can't a vendor benchmark replace your own test?

Published people-search scores move depending on who runs them, which queries they use, and when. None of those queries are your reqs. Treat vendor and academic benchmarks as ideas for building a rubric, not as evidence for a purchase.

Here's what the public record shows:

- One search provider's people benchmark uses 1,400 synthetic role-based queries distilled from 10,000 historical queries. For role-based discovery it uses an LLM judge that fetches each result page and checks it against the query criteria, because there's no single correct answer.
- That provider's own published precision moved from **63.3% to 86.2%** between two runs, and the answered-query count fell from **1,399 to 1,323**. When a query goes unanswered, it drops out of the denominator, so the score covers a smaller and possibly easier set.
- Other vendors self-report precision on a competitor's benchmark, sometimes several points ahead of each other. Nobody outside the vendor ran those numbers.
- PeopleSearchBench, an academic benchmark of 119 multilingual queries across four scenarios, got its automated judgments to agree with human annotators at a **Cohen's kappa of 0.84**. Cohen's kappa measures how far two graders agree beyond what chance would produce.

The kappa result is the useful part. PeopleSearchBench got there by splitting each query into criteria that can be checked one at a time. The grading rubric later in this guide uses the same design.

> Borrow a benchmark's rubric design, never its score: neither the queries nor the denominator are yours.

## How many reqs and graded results do you need?

Five to ten filled reqs can separate two tools only when their precision differs by about 15 points or more. Smaller gaps need far more graded results than most teams have filled reqs to supply. I couldn't find a published sample-size rule specific to sourcing, so the figures below come from the standard two-proportion formula.

| Precision gap | Graded per tool (95%) | Graded per tool (90%) | Briefs at P@20 (95%) |
|---|---|---|---|
| 0.80 vs 0.65 | 136 | 107 | 7 |
| 0.80 vs 0.70 | 290 | 229 | 15 |
| 0.85 vs 0.80 | 906 | 714 | 46 |

*Derived: two-sided test, 80% power, results assumed independent.*

**46 - briefs needed at P@20 to separate a tool scoring 0.85 from one scoring 0.80**

Derived from the standard two-proportion formula at 95% confidence and 80% power; seven briefs only detect a 15-point gap.

The independence assumption is optimistic. The 20 results for one brief are correlated, because a brief that suits a tool lifts all 20 of them. So the real sample you need is larger than the table shows. To handle this, resample whole briefs (a bootstrap over briefs) when you build intervals, rather than treating 200 results as 200 independent coin flips.

Use these two interval methods:

- **Wilson interval** for each tool's precision. It does better than the textbook normal approximation when samples are small and rates are close to 0 or 1, which is the usual case here.
- **Newcombe interval** for the difference between two tools. It combines the two Wilson intervals. If the Newcombe interval for "candidate minus incumbent" includes zero, you haven't shown a difference.

If you want to size the test before running it, an open-source Wilson sample-size tool (AssureSizeWilson) will do it. For reference, one worked clinical protocol needed 107 per arm for an 80% chance that the half-width of a 90% interval stays at or below 0.1 at a rate of 0.75. That's the right order of magnitude for a single tool's P@20.

### Where the cut lines come from

One published buyer threshold says to accept **no more than 20 to 25% irrelevant results in the top 20**. Below that, the tool isn't saving meaningful time. It defines irrelevant as wrong seniority, wrong geography, or wrong function. In P@20 terms, the bar is roughly 0.75 to 0.80.

A second source tells buyers to have a recruiter score the top 20 blind and sets a "below 25%" cut line, but it doesn't say which metric that applies to. Don't adopt it until you've written down which number it refers to.

> **Rule:** Pre-register before the first run
>
> Write down the metrics, cut lines, sample size, and decision rule, date the sheet, and change nothing after any tool has run. A cut line set after you've seen the results won't hold up when finance asks.

## What has to be signed before any data reaches a vendor?

Two things: a DPIA completed before processing, and written processing instructions to each vendor that ban reuse of your data for training. Get both done before any brief, name, or answer key leaves your systems.

The UK ICO's audit of AI recruitment tools is the most concrete public guidance. It made 296 recommendations, all accepted or partially accepted. Four findings apply directly to a bake-off:

- DPIAs should be done before processing, not after.
- Recruiters must give their AI provider clear and detailed processing instructions.
- Some providers repurposed candidate personal data to train, test, and maintain their own tools. The ICO considered that unlikely to be lawful.
- The ICO said a tool being "at least better than random" would usually not be sufficient. Your bake-off results are therefore also your accuracy evidence, so keep them.

Under the EU AI Act, recruitment systems are listed in Annex III, and Article 26 sets out deployer duties:

- **Article 26(4)**: input data must be relevant and sufficiently representative where the deployer controls it. Your frozen briefs are input data.
- **Article 26(6)**: logs must be kept for at least six months.
- **Article 26(7)**: workers must be informed before deployment.

The application date for the Annex III recruitment duties has been postponed by amending legislation. Check the AI Act Service Desk page for Article 26 rather than relying on a date you've read elsewhere. Whether a pilot or a retrospective bake-off counts as "deployment" isn't settled, so get counsel's view in writing. Keeping every export and screenshot for at least six months is a cheap default either way.

## The bake-off procedure, step by step

The whole job is eight steps. It takes three to four working weeks at most, and most of that is the legal gate and the grading. Run the steps in order, because each one protects the next from contamination.

#### The people-search bake-off

1. **Clear the legal gate** - The ops lead and the DPO complete a DPIA and send each vendor processing instructions that ban reuse of your data for training. About 1 week; done when the DPIA is signed.
2. **Build the known-answer set** - Pull 5 to 10 reqs filled in the last 12 months; the answer key is the hire plus every finalist who reached the final stage. 1 to 2 days; done when the key is sealed away from graders.
3. **Freeze the briefs** - Use the intake note or job description dated before the first finalist interview and the role's original title, and flag hires who now list your company as a separate stratum. About half a day per brief; done when briefs are dated and hashed.
4. **Pre-register metrics and cut lines** - Write down metrics, cut lines, sample size, and decision rule before any run. Half a day.
5. **Run every tool on the same day** - Paste identical brief text into every tool and the incumbent process on one day, and export the top 20 with screenshots. 1 day.
6. **Pool the results and grade blind** - Merge all top 20s, remove duplicates, strip tool labels; two graders calibrate on about 30 profiles and must reach kappa of 0.61 or higher before full grading. 2 to 4 days.
7. **Check profile facts** - Sample 30 to 50 profiles per tool at random and verify employer, title, location, and last-update date against live sources. About 1 day.
8. **Compute and decide** - Report recall@20 of hires and finalists, P@10 and P@20 with Wilson intervals, the Newcombe difference against the incumbent, and cost per qualified profile, then apply the cut lines.

### Steps 2 and 3: keeping hindsight out of the answer key

Practitioners commonly take roles filled in the last year, run them through the tool, and check whether the placed candidate shows up near the top. Some also compare the tool's top results against everyone they actually interviewed and hired. The weak point is **hindsight leakage**: the brief or the profile now contains information that didn't exist at search time.

I found no published procedure for controlling this. The controls below are my own design, not an established standard:

- **Date-stamp the brief.** Use the intake note or job description as it stood before the first finalist interview. Later versions usually describe the person you hired.
- **Use the original title.** If the role was re-titled to match the hire, use the old title.
- **Judge "found" against the past profile.** A hit counts if the tool returned the person and the profile as it stood at search time would have matched the brief.
- **Separate the current-employee group.** A hire who now lists your company may be found only through that employer keyword. Tools that exclude current employees will miss these hires by design. Report this group on its own line.
- **Hash the briefs.** Run any standard checksum on each final brief file so nobody can quietly edit the text between tools.

Finalists who weren't hired are your cleanest signal. They never joined you, so nothing about your company leaks into their profiles.

### Step 5: identical text, one day

Live web sources change from day to day. Two tools run a week apart are searching two different snapshots of the web. Run everything on the same day and screenshot each top 20 as you export it.

The text must also be identical across tools. Boolean-only tools force you to translate the brief into a query, and that translation is a second variable you can't control. Its quality depends on who wrote it. [Refolk](/) takes the frozen brief as plain English, so the hashed text goes in unchanged. For Boolean tools, have one person write each string from the brief alone, record it in the pre-registration annex, and accept that the incumbent's score includes that person's skill.

### Step 7: fact accuracy is a separate grade

I found no published audit of employer, title, or location accuracy for sourcing databases. The nearest proxies are contact-data decay figures. One source puts decay at about 2.1% a month, which compounds to 22.5% a year. Published title-change rates conflict badly: one source says 15 to 25% a year and another says 65.8%. Don't build a cut line on either.

The method is simple. Take a random sample, verify each record by hand, and record the error rate. One source reports that a 300-row audit takes about two hours, so 30 to 50 profiles per tool is an afternoon. Keep this grade apart from relevance. A profile can be the right person and still show the wrong current employer.

## How should blind pooled grading work?

Merge every tool's top 20 into one de-duplicated, unlabelled pool and have two calibrated graders judge every profile in it. This is **pooling**, borrowed from information-retrieval evaluation, where the top-ranked results from each system are combined and the whole set is judged.

#### From exports to blind grades

1. **Export** - Top 20 per tool per brief, same day, with screenshots
2. **Merge** - Combine every tool's results and remove duplicate people
3. **Blind** - Strip tool labels and shuffle the order within each brief
4. **Calibrate** - Two graders score about 30 profiles; rewrite the rubric if kappa is below 0.61
5. **Grade** - Grade the full pool against the rubric, function before title
6. **Unblind** - Re-attach tool labels and compute metrics per tool

*Graders never see which tool returned a profile, and full grading waits until two graders agree at kappa 0.61 or higher on a calibration set.*

TREC is the long-running US benchmark programme for search evaluation. One TREC track set its pool depth at 25. Depth 20 here matches the published buyer threshold, and it keeps grading inside the 2 to 4 days the procedure allows.

### Grader agreement: what counts as good enough

The usual scale (Landis and Koch) calls a kappa of 0.41 to 0.60 moderate and 0.61 to 0.80 substantial. That scale is contested: McHugh argues it's too lenient and treats anything below 0.60 as inadequate. Requiring 0.61 or higher before full grading satisfies both views.

Two practical adjustments:

- **Grade binary for the headline numbers.** In one study, two-level grades reached a kappa of 0.63 while four-level grades averaged 0.60 weighted. Record a four-level grade if you want it, but compute precision on relevant / not relevant.
- **Report raw agreement as well.** Kappa tends to understate agreement on the rare category. If one tool returns almost nothing relevant, kappa can look poor even when the graders mostly agree.

**Blind grading rubric for one pooled profile**

```
Profile ID (tool label removed):
Brief ID:
C1 Function: does this person do the brief's job family? (Y / N / Can't tell)
   Judge the work described, not the title text. Procurement "sourcing" = N.
C2 Seniority: does the level match the brief? (Y / N / Can't tell)
C3 Geography: is the person in the brief's location or stated remote scope? (Y / N / Can't tell)
C4 Must-have from brief: ______ (Y / N / Can't tell)
C5 Must-have from brief: ______ (Y / N / Can't tell)
Grade: RELEVANT only if C1, C2 and C3 are Y and no must-have is N. Otherwise NOT RELEVANT.
Optional detail grade: 3 strong / 2 relevant / 1 marginal / 0 wrong
Would you contact this person for this req? (Y / N)
Grader initials:            Minutes spent:
```

*Replace C4 and C5 with the brief's own must-haves; keep every criterion checkable on its own.*

### Why graders judge function, not title

Matching on job titles fails in different ways in different markets. In Refolk's index of professional profiles, a title query for sourcers returned at least 5 off-function profiles in a sample of 25 in the UK: procurement, billing, tutoring, and category-lead roles. The same query returned 2 in 25 in the US. "Sourcing" is also a procurement job family, and UK employers use it that way.

**20% - of a UK sourcer title sample in Refolk's index sat in the wrong function**

At least 5 of 25 profiles were procurement, billing, tutoring, or category-lead roles, from a title query alone.

| Market | Off-function | Rate | Wilson 95% CI |
|---|---|---|---|
| US | 2/25 | 8% | 2.2% to 25.0% |
| UK | 5/25 | 20% | 8.9% to 39.1% |

*Counts from Refolk's index samples; intervals derived.*

The intervals overlap, so 25 profiles can't prove the UK rate is higher than the US rate. That's the sample-size lesson in small form. The direction is still a reason to include at least one UK or homonym-prone req in your known-answer set. The pools are uneven too: Refolk's index holds 5,730 US profiles with sourcer titles against 476 in the UK, about 12 to 1. Thin markets are where weak retrieval shows.

A tool that takes the exclusion in the brief's own words should do better on these briefs. Refolk accepts a plain-English exclusion directly, so you can test whether the tool honours it rather than whether someone wrote the right NOT clause.

I ran this search: `Senior sourcers in the UK who source machine learning engineers, excluding procurement roles` - [see the full result list](https://www.refolk.ai/s/713jz134mw).

*Returns UK sourcing specialists who recruit ML engineers, with procurement-family "sourcing" titles filtered out, ready to grade against the rubric above.*

## How does a people-search bake-off go wrong?

Most bad verdicts come from seven failures, and each one has a specific check. Read this list before you pre-register, because several of the checks have to be planned in before any tool runs.

1. **Hire-recall leakage.** A past hire now lists your company, and a tool finds them only through the employer keyword, which inflates hire recall. *Check:* score recall on non-hired finalists separately and report the current-employee group on its own line.
2. **Homonym titles.** "Sourcing Specialist" is also a procurement title. False positive: the title matches but the person works in the wrong function. *Check:* graders judge function from the work described (rubric C1), never from the title text.
3. **Underpowered verdict.** Someone declares a winner on 5 briefs with a 10-point gap. Dataset C shows that gap needs about 15 briefs. *Check:* no winner unless the Newcombe interval for the difference excludes zero.
4. **Low grader agreement.** A kappa below 0.6 means the rubric is ambiguous and precision depends on which grader scored the profile. *Check:* calibrate before full grading, rewrite the rubric on failure, and report raw agreement next to kappa.
5. **Vendor-run benchmarks.** Self-reported runs drift, and the count of answered queries changes between runs (1,399 against 1,323 in one case). *Check:* run the evaluation yourself and record how many briefs each tool returned results for.
6. **Stale profile facts.** A profile is relevant for the role but shows the wrong current employer. Retrospective tests make this worse: with contact data decaying about 22.5% a year, a known answer from 12 months ago has probably moved, which can penalise a tool whose data is fresher than your answer key. *Check:* the Step 7 sample, graded apart from relevance, and the past-profile rule for "found".
7. **Live web drift.** Tools run on different days search different snapshots. *Check:* same-day runs with screenshots, archived with the pre-registration sheet.

> **Watch out:** The tool that finds your hire may only be finding your logo
>
> If hire recall is high and finalist recall is low, assume leakage until you've proved otherwise. Finalists who never joined you are the only known answers your company name can't contaminate.

## How do you turn the numbers into adopt, extend, or drop?

Two questions decide the verdict. Does the tool clear the pre-registered cut lines? Is its difference from the incumbent real, meaning the Newcombe interval excludes zero in its favour? Cost per qualified profile settles cases where precision ties. Recall and fact accuracy act as vetoes.

#### The verdict grid

Horizontal axis runs from Difference vs incumbent includes zero to Difference excludes zero in tool's favour. Vertical axis runs from Misses cut lines to Meets cut lines.

| Quadrant | What it means |
| --- | --- |
| Good enough, not proven better | Extend the pilot, or adopt only if cost per qualified profile is clearly lower |
| Good enough and proven better | Adopt, unless a recall or fact-accuracy veto fires |
| Not good enough, not better | Drop |
| Better than incumbent, still below the bar | Extend with a live pilot and re-examine whether the incumbent also fails the bar |

*Clearing the cut lines and beating the incumbent are separate tests, and a tool needs both to be adopted on retrieval evidence alone.*

Decision rules to copy into your pre-registration:

- **Adopt** when P@20 meets the cut line, the Newcombe interval excludes zero in the tool's favour (or precision ties and cost per qualified profile is clearly lower), finalist recall is no worse than the incumbent's, and fact accuracy meets your locally set floor.
- **Extend the pilot** when the tool clears the bar but the difference interval includes zero, or when it beats the incumbent but both miss the bar. One published alternative is a live pilot of at least three to four weeks across two live reqs, after baselining on the last twenty closed reqs. A live pilot also adds briefs toward the counts in Dataset C.
- **Drop** when the tool misses the cut lines and isn't distinguishable from the incumbent, or when the vendor won't accept processing instructions that ban training on your data.

**Bake-off pre-registration and report sheet**

```
Bake-off ID:                     Date sealed:
Ops owner:                       DPIA signed by DPO on:
Processing instructions (no training reuse) accepted by: [tool] on [date]
Tools under test:                Incumbent process:
Briefs: [n] reqs filled in last 12 months; checksums listed in annex
Answer key: hire + finalists per req; sealed by [name], not a grader
Current-employee stratum: [n] hires now listing our company
Pool depth: top 20 per tool; all runs on [date]; screenshots archived
Primary metric: P@20, relevant = right function, seniority, geography
Secondary: P@10; recall@20 hires; recall@20 non-hired finalists; fact accuracy (30 to 50 per tool); cost per qualified profile
Cut lines:
  P@20 at or above 0.75 (no more than 25% irrelevant in top 20)
  Placed candidate in top 5 on at least [x] of [n] briefs
  Fact accuracy at or above [local floor]
Grader gate: kappa at or above 0.61 on ~30 calibration profiles; raw agreement reported
Intervals: Wilson per tool; Newcombe for difference vs incumbent; bootstrap over briefs
Decision rule: Adopt / Extend / Drop as written above

REPORT
Tool | Recall@20 hires | Recall@20 finalists | P@10 (95% CI) | P@20 (95% CI) | P@20 diff vs incumbent (95% CI) | Fact accuracy | Cost per qualified profile | Verdict
Incumbent |  |  |  |  |  n/a |  |  |
Tool A |  |  |  |  |  |  |  |
Tool B |  |  |  |  |  |  |  |
Kappa at calibration: ____  Raw agreement: ____  Briefs with zero results, per tool: ____
```

*Fill and date everything above the report table before any run; fill the table only after unblinding.*

Cost per qualified profile needs a fixed numerator. Pick annual licence cost or pilot-period cost, write it on the sheet, and use the same basis for every tool and for the incumbent's sourcing hours.

## How do you keep the verdict current?

Treat the bake-off as a reusable asset. Keep the sealed answer key, the hashed briefs, and the rubric, and re-run them at every renewal. Each newly filled req moves you toward the brief counts that can detect smaller gaps.

Three habits keep it honest over time:

- **Grow the set.** Add each filled req's frozen brief and answer key as it closes. Ten briefs can only detect large gaps, and a set heading toward 46 can catch a 5-point difference.
- **Re-check the tool, not the memory of it.** Vendors change models and indexes between renewals. A verdict from last cycle is a hypothesis for this one, so re-run it on the same day and under the same rules.
- **Re-check the legal footing.** Confirm that processing instructions are still accepted. Look up the current Annex III application date on the AI Act Service Desk, and get counsel's view again if your use has moved from test toward deployment.

For the narrower question of how fast your own data goes stale, the Contact Decay Playbook builds a refresh interval from a dated sample.

#### Before you send the verdict

- [ ] The DPIA was signed and processing instructions banning training reuse were accepted before any data left your systems.
- [ ] Every brief predates the first finalist interview, uses the original title, and has a recorded checksum.
- [ ] The answer key was sealed by someone who did not grade.
- [ ] Metrics, cut lines, sample size, and decision rule were dated before the first run.
- [ ] All tools ran on the same day with identical brief text, and every top 20 has a screenshot.
- [ ] Graders reached a kappa of 0.61 or higher on the calibration set, and raw agreement is reported alongside it.
- [ ] Recall is reported separately for hires, non-hired finalists, and the current-employee group.
- [ ] Every precision figure has a Wilson interval and every winner claim has a Newcombe interval that excludes zero.
- [ ] Fact accuracy comes from a random sample of 30 to 50 profiles per tool, graded apart from relevance.
- [ ] All exports, screenshots, and sheets are archived for at least six months.

## Frequently asked questions

### How many past reqs do I need to evaluate an AI sourcing tool?

Five to ten filled reqs is enough to catch a large difference and nothing smaller. At precision at 20, about 7 briefs can detect a 15-point gap (0.80 against 0.65) at 95% confidence and 80% power. A 5-point gap (0.85 against 0.80) needs about 46 briefs. If the tools look close after ten reqs, you cannot call a winner yet. Extend the pilot or add reqs as they close.

### What precision should a sourcing tool hit in the top 20 results?

One published buyer threshold says no more than 20 to 25% of the top 20 should be irrelevant. That works out to precision at 20 of roughly 0.75 to 0.80. Irrelevant means wrong seniority, wrong geography, or wrong function. Treat this as a starting cut line, write your own into the pre-registration sheet before any run, and say which metric it applies to.

### Can I trust a vendor's published people-search benchmark score?

Not as evidence for your own reqs. One vendor's own published precision moved from 63.3% to 86.2% between runs while its answered-query count fell from 1,399 to 1,323. Other vendors self-report scores on a competitor's benchmark. The queries aren't yours, and unanswered queries quietly leave the denominator. Use those benchmarks for rubric ideas and run the test on your own reqs.

### What if the person we hired now shows our company on their profile?

Then a tool may find them only because it matches your company name, and that inflates hire recall. Put those hires in a separate group and use the original title and the brief as it stood before the hire. Report recall on non-hired finalists on its own line. Tools that exclude current employees will miss that group by design, so don't count those misses against them.

### Do I need a DPIA before piloting a people-search tool?

Assume yes. The UK ICO's audit of AI recruitment tools said DPIAs should be done before processing, not after it. It also said recruiters must give the AI provider clear, detailed processing instructions. Whether a pilot counts as deployment under the EU AI Act hasn't been settled, so get counsel's written view and keep your bake-off records at least six months.

### Can an LLM grade the results instead of human reviewers?

Possibly, if you check it against humans first. The academic PeopleSearchBench reached a Cohen's kappa of 0.84 with human annotators by splitting each query into criteria it could check one at a time. On your own pool, have two humans grade the calibration set, run the model on the same profiles, and only use it if its agreement with the humans clears the same 0.61 bar.

---

*From the Refolk guide library. I revise these guides rather than replacing them, so the current version is always at https://www.refolk.ai/guides/people-search-tool-bake-off*