Exa's 1,400-Query Benchmark Just Ended the "Just Ask" Sourcing Pitch

Every AI sourcing vendor now sells the same sentence: describe the candidate in plain English, get a shortlist. Exa Labs just published a 1,400-query benchmark that scores whether those shortlists are actually correct, and the spread is brutal. One tool clears 90%, one lands at 79%, and the rest sit under 30%.

That gap changes the buying question for 2026. It is no longer "does it take natural language?" (they all do). It is "what percentage of your top ten are real matches?"

What Exa's People Search Benchmark actually measures

The Exa People Search Benchmark is an open-source evaluation of 1,400 synthetic, role-based candidate queries that scores each tool on whether its top-10 results actually match the query. It is hosted on Hugging Face and GitHub, and the scoring is deliberately simple: for every query, each of the top 10 returned candidates is checked against the original criteria.

Exa Labs built the query set by analyzing 10,000 historical people searches from exa.ai and clustering them into three dominant patterns:

• Role-based executive search
• Skill and role discovery filtered by location
• Specific-individual lookup

The queries are short and realistic ("senior payroll specialist in boston"), with structured metadata for role_function, seniority, and geography. That structure is what makes the benchmark reproducible: any vendor can run their own harness against the same 1,400 prompts and publish a comparable number.

The reason this benchmark matters more than the last dozen G2 categories is that it measures the exact thing buyers actually care about, which is top-10 precision on ambiguous natural-language queries. Time saved, database size, and InMail acceptance are all still useful, but none of them tell you whether the fifth candidate in a shortlist is a real match or a hallucinated one.

The scoreboard: one tool at 93.5%, most under 30%

Per Metaview, which published the first vendor comparison, the reported spread on Exa's 1,400-query set is enormous: Metaview Sourcing at 93.5%, the next-best tested competitor at 79%, Exa's own agent at 63%, and the rest under 30%.

93.5%

Two things about that scoreboard are worth staring at.

First, the tool that published the benchmark is not the tool that won it. Exa's own agent scored 63% on Exa's own eval. That is the single most credible thing about the whole exercise: if Exa were gaming the query set, its agent would be at the top, not in fourth place. The benchmark clearly measures agentic reasoning and ranking, not just raw index quality (Exa itself indexes 1B+ professional profiles refreshed weekly).

Second, "under 30%" is not a bug in the losing tools. It is the ceiling of a keyword-plus-filter architecture running against ambiguous prompts. LinkedIn's own MUSE research paper concedes the point: traditional search fails when descriptions get complex, because keyword matching misses synonyms and attribute filters cannot capture abstract skills. Tools that translate a natural-language prompt into a Boolean string inherit Boolean's precision cap. The pretty UI does not raise the ceiling.

The tool that published the benchmark is not the tool that won it. That is the single most credible thing about the whole exercise.

Why "top 10 precision" is a harder test than it sounds

Top-10 precision is punishing because the underlying talent pools are small enough that a wrong-city or wrong-seniority match is a hard miss, not a rounding error. This is where index-level data changes how you read the score.

In Refolk's index of professional profiles, here is what the US supply actually looks like for three benchmark-style queries:

The Senior Payroll pool is 3.5x larger than the Senior Backend pool. That matters because the harder the query (smaller pool, more location constraints, more seniority nuance), the more brutally top-10 precision penalizes a system that pads results with adjacent titles. A 30%-precision tool asked for "senior backend engineer in Brooklyn" is not returning three good candidates and seven okay ones. It is often returning three good candidates and seven mid-level engineers in New Jersey.

This is the exact gap Refolk is built to close: you describe the person in plain English, including the messy constraints ("Rust experience, shipped a database, currently in a Series B or later, US-based"), and Refolk returns a ranked shortlist across GitHub, LinkedIn, and the open web instead of a Boolean approximation of what you asked for.

The four vendors buyers actually cross-shop, and how the pitches differ

Buyers evaluating natural-language sourcing in 2026 are usually looking at four tools with meaningfully different architectures: LinkedIn Hiring Assistant, Juicebox, the older Boolean cohort (hireEZ, Findem, Fetcher), and the newer benchmark-topping entrants like Metaview and Exa. The pitches sound identical. The math underneath is not.

• LinkedIn Hiring Assistant. Sells time saved, not precision. LinkedIn claims a 65% InMail acceptance rate from Hiring Assistant sourced candidates versus 39% from manual sourcing, and 1.5 hours saved per role. It runs on LinkedIn's graph only, currently in English, German, and French, with additional languages rolling out in 2026. It is blind to GitHub and open-web signal.
• Juicebox / PeopleGPT. Natural-language search over 800M+ profiles, plus 24/7 autonomous sourcing agents. Sold on coverage and automation. Tested in the academic PeopleSearchBench alongside Exa, Lessie, and Claude Code.
• hireEZ, Findem, Fetcher. Older cohort. hireEZ and Findem are strong specialists; Fetcher is a done-for-you service. All were built pre-LLM and retrofitted a natural-language layer over Boolean plumbing, which is exactly the architecture the Exa benchmark punishes.
• Metaview. Pivoted from AI notetaker to sourcing agent; posted the 93.5% score. Sourcing tiers run $100 per user per month for 200 sourced profiles and $300 for unlimited. Self-reported number, harness promised on GitHub but not yet public.

Two pitches are worth separating in your head. "We save 1.5 hours per role" (LinkedIn) is a productivity claim. "We return correct candidates 93.5% of the time on a 1,400-query eval" (Metaview) is a quality claim. Only the second one is testable, and only the second one predicts whether the tool will still be useful when your query is weird.

Precision beats database size, and the math is finally defensible

Coverage is a floor, not a ceiling. Juicebox pitches 800M profiles, Exa indexes 1B+, but the academic PeopleSearchBench (119 queries across Lessie, Exa, Juicebox, and Claude Code) found clear architectural tradeoffs, with data source diversity mattering only if ranking is good.

The math is straightforward. A 30%-precision tool over 1B profiles returns roughly 3 usable candidates in a top-10 shortlist. A 90%-precision tool over 200M profiles returns roughly 9. Coverage helps only after ranking is solved. In practice most sourcers will refuse to run a second query after seeing seven junk results in the first one, so low precision compounds into low usage, which compounds into a shelfware contract at renewal.

This is also why the vendor landscape is bifurcating. Tools built on a Boolean core with a natural-language veneer cannot get past the precision cap without rebuilding the retrieval stack. Tools built around agentic reasoning plus semantic retrieval have headroom. The Exa benchmark is, effectively, a diagnostic that tells you which side of that line a vendor is on.

How to run the benchmark on the vendor pitching you

Do not accept a self-reported number. The harness is reproducible and the queries are public. Any vendor claiming a top score should be willing to rerun the eval, live, on either the public 1,400 or a subset of your own real queries.

A defensible buyer eval in 2026 looks like this:

1. Ask the vendor to run their tool against 50 of your last real reqs, blind, and score top-10 precision yourself.
2. Ask them for their score on Exa's public 1,400-query set. If they refuse or delay, treat that as the answer.
3. Pull 20 queries from Exa's Hugging Face dataset and run them yourself during the demo. Watch what happens on the "senior payroll specialist in boston" type prompts, not the softball ones.
4. Check whether the top-10 candidates are actually reachable (real emails, current employer accurate, GitHub linkable). Precision on a stale profile is worth zero.
5. Compare cost per verified match, not cost per seat. Per PeopleSearchBench, web verification alone runs $0.75 per query and full evaluation cost $89.40, so verification is the hidden line item.

The whole point of Refolk is to make step 3 the entire product loop: you ask in plain English, you see the shortlist, you see why each person is in it, and you re-ask until it is right.

The buyer takeaway for 2026

52% of talent leaders plan to deploy autonomous sourcing agents in 2026. The market is buying faster than any independent quality baseline can catch up, which is exactly why a public benchmark matters right now.

64 pts

Three things to hold onto:

• Natural language is table stakes. Precision on top-10 is the actual product.
• Self-reported scores are the new G2 review. Make the vendor rerun the eval on your queries.
• Index size is a floor. A 30%-precision tool over a billion profiles ships fewer usable candidates than a 90%-precision tool over a fraction of that.

The Exa benchmark did not create the precision gap in AI sourcing. It just made it impossible to hide. If you are cross-shopping tools this quarter, that is the yardstick to hold every demo against.

FAQ

Is the Exa People Search Benchmark actually independent?

The benchmark is open-source and published by Exa Labs, but the query set is public on Hugging Face and GitHub, the scoring method is transparent, and Exa's own agent only scored 63% on it. That last fact is the strongest evidence of independence: if Exa were gaming the eval, they would not have published a scoreboard that puts their own product in fourth place behind Metaview at 93.5% and the next-best tool at 79%.

Why do most AI sourcing tools score under 30%?

Because most were built before LLMs and layered a natural-language input over a Boolean retrieval core. Keyword-plus-filter architectures have a hard precision ceiling on ambiguous queries. LinkedIn's own MUSE research says the same thing: keyword matching misses synonyms and attribute filters cannot capture abstract skills. Adding a natural-language front end does not raise that ceiling.

Does database size matter more than precision?

No. A 30%-precision tool over 1B profiles returns roughly 3 usable candidates per top-10 shortlist. A 90%-precision tool over 200M returns roughly 9. Coverage is a floor, not a moat. The academic PeopleSearchBench found data source diversity helps only when ranking is already strong, so precision compounds and coverage does not.

How should I actually test a vendor before I buy?

Run the vendor against 50 of your real reqs, score top-10 precision yourself, and ask them to reproduce their Exa benchmark score live during the demo. Pull 20 queries from Exa's public Hugging Face dataset and watch how the tool handles the terse, ambiguous ones like "senior payroll specialist in boston." Then verify the top candidates are actually reachable. If a vendor will not rerun the eval on your queries, that is your answer.