Exa's 1,400-Query People Search Benchmark: What the 49-Point Cliff Means

Exa quietly open-sourced a 1,400-query people search benchmark, and within weeks Metaview's blog post claiming 93.5% precision, with Juicebox at 79% and everyone else under 30%, started showing up in RFPs and vendor calls. If you are evaluating AI sourcing tools right now, this benchmark is going to land on your desk.

Here is what it actually measures, why the gap is a cliff rather than a rounding error, and how to design a 20-query private eval that reproduces the methodology on the roles you actually hire for.

What the Exa People Search Benchmark actually is

The Exa People Search Benchmark is a public, open-source retrieval test of 1,400 synthetic role-based queries, distilled from 10,000 historical exa.ai people queries and scored purely on precision of returned profiles against role, function, seniority, and geography. It lives at github.com/exa-labs/benchmarks, with a Hugging Face mirror.

The queries look like this: "senior payroll specialist in boston". Each one carries structured fields:

• role_title (the job title being searched)
• role_function (the broader function bucket)
• role_seniority (entry, mid, senior, staff, manager, director)
• geo_name (city, metro, country)
• geo_type (the granularity of that geo)

Exa built it alongside two sibling benchmarks in the same repo: Company Search (~800 queries) and Publication Retrieval (1,866 queries). The people index Exa itself runs against covers 1B+ public profiles with a hybrid retrieval system, weekly refresh, and fine-tuned embeddings for role, skill, and company queries. That last detail matters, and I will come back to it.

The benchmark is not a workflow test. It does not measure outreach reply rate, enrichment accuracy, ATS integration, or whether a tool can chain "left Stripe in the last 6 months and now contributes to a Rust repo" into a single query. It measures one thing: given a plain-English role plus geo, how many of the returned profiles are actually that role in that geo.

The scoreboard, and why the gap is a cliff

Metaview's public claim is 93.5% precision on the benchmark, with Juicebox second at 79%, Exa's own agent at 63%, and the rest of the tested field under 30%. The gap between #2 and the pack is 49 points or more.

49+

Why the cliff exists

The under-30% tail is not a model quality problem. It is a data source problem. Most AI sourcing tools sit on top of a general web index (or a thin LinkedIn scrape) and ask a language model to interpret the results. On a query like "senior payroll specialist in boston," a general index returns job postings, listicles, company payroll pages, and the occasional actual profile. Precision collapses because most of what comes back is not a person.

Exa itself flags this in its comparison pages: vertical people, company, and publication indexes are structurally different from general retrieval. Tools that route through general web indexes cannot win this test. Tools with a proprietary, weekly-refreshed people index sit in a different regime entirely.

That is the mechanism behind the cliff. You are not looking at "some vendors are 3x better than others." You are looking at two different product categories that happen to share a marketing surface.

The three things the benchmark under-tests

The Exa benchmark is honest about what it is: a role + geo retrieval test on synthetic queries. It under-tests the harder buyer workflows that show up in actual recruiter briefs.

1. Multi-constraint stacks. "Senior backend engineer in Berlin who left Klarna in the last 12 months and has Rust in a pinned GitHub repo" is four constraints, not two. The benchmark tests two.
2. Signal beyond title and geo. GitHub activity, patent authorship, conference talks, tenure math, "openness-to-move" scoring (Entelo's old pitch), security clearances (SeekOut's 3.7M+ cleared pool), all invisible to a role + geo test.
3. Named-individual and skill-discovery patterns. Exa's own clustering identified three usage patterns in the source 10,000 queries: role-based, skill/role discovery, and specific-individual lookup. Only role-based made it into the 1,400. The other two categories, which are half the real workload, are not scored.

A vendor that scores 93.5% on this benchmark can still miss badly on a constraint-stacked brief. The correct read of the scoreboard is "these tools have a real people index" rather than "this tool will find the person my hiring manager wants."

This is the exact gap Refolk is built for: you describe the person in plain English across GitHub, LinkedIn, and the open web, and get a ranked shortlist back, so a constraint-stacked query runs as one prompt instead of four Boolean sweeps.

Metaview's number is a self-reported run

Metaview's 93.5% is a self-reported run of a competitor's benchmark, using Metaview's own code path against Exa's public queries. Nobody has independently replicated it. That is not a knock. Exa uses the same posture on its own vs.-Parallel and vs.-Tavily pages, where Exa Agent High scores 74% on BrowseComp at $0.50 versus Parallel Task Ultra 8x at 58% for $2.40. Vendor-run benchmarks are how this category talks about itself right now.

The correct buyer reflex is not "trust the blog post." It is "rerun the harness with our API keys, on our roles." Exa open-sourced the whole thing precisely so you can.

Vendor benchmarks are marketing until you rerun them with your own API keys on your own roles.

There is already a second benchmark with a different verdict

A second, peer-reviewed people search benchmark exists, and buyers citing "the Exa benchmark" in evals often do not know about it. PeopleSearchBench (arXiv 2603.27476) evaluates Lessie, Exa, Juicebox, and Claude Code (Sonnet 4.6) on 119 queries across four scenarios, using a decompose-then-verify pipeline with Cohen's κ=0.84 human agreement.

It measures three dimensions absent from Exa's benchmark:

• Relevance Precision (are the returned profiles actually the target?)
• Effective Coverage (how much of the true set did you find?)
• Information Utility (is the returned data useful for the next step?)

Results cap at 15 per query, scoring is padded nDCG plus qualified-result thresholds, and the leader beats the second system by 18.5 points. Different test, different regime, different leaderboard. If a vendor cites one benchmark and not the other, ask why.

How to design your own 20-query private eval

The right response to a public benchmark landing in your inbox is to build a private one. Here is a 20-query template that reproduces Exa's methodology and then stress-tests the workflows Exa's benchmark ignores.

The 20-query split

• 8 role + geo queries (mirror the Exa methodology). Pick eight roles you actually hire for. Format them like "staff platform engineer in Austin" or "senior clinical research associate in Cambridge, MA." Score on precision at 25 results.
• 5 constraint-stacked queries (test the real workload). Add tenure math ("left in the last 6 months"), company origin ("ex-Stripe"), stack signal ("Rust in a pinned repo"), or credential filters ("PhD in molecular biology"). Score on precision at 10.
• 4 skill/role discovery queries (the pattern Exa dropped). "Who at Series B fintechs in London has built a fraud model from scratch?" These have no clean ground truth, so score on top-5 usefulness with two internal raters.
• 3 named-individual lookups ("find me the person who wrote X paper / gave Y talk / runs Z open-source project"). Score binary: found or not.

The scoring rules

• Two raters per query, blind to which tool returned which result.
• Precision only. Do not score coverage on this pass, because coverage requires a ground-truth set you almost certainly do not have.
• Log latency and cost per query. Exa's own vs.-Parallel page bundles cost framing for a reason: a tool that is 5 points better at 10x the price is not better.
• Rerun quarterly. Indexes drift, models change, and last quarter's winner is not this quarter's.

Why one benchmark moved the whole category in weeks

The dedicated sourcer population is small enough that a single benchmark blog reaches most of it inside a month. In Refolk's index of professional profiles, only 1,493 people in the US currently hold titles like "Sourcer," "Talent Sourcer," or "Sourcing Recruiter," concentrated at Meta, MongoDB, Microsoft, Amazon, and Gartner, with the SF Bay Area as the top hub.

The broader technical recruiter pool is roughly 14x that size. In Refolk's index, 21,131 profiles across the US and UK hold "Technical Recruiter" or "Technical Sourcer" titles, clustered in Seattle, SF Bay, Austin, and Boston. Dedicated sourcers are about 7% of the technical recruiter population.

1,493

That is why one Metaview blog post moved the buying conversation for an entire category. The eval-running population is small, deeply connected on LinkedIn and in a handful of Slacks, and hungry for defensible artifacts to bring to procurement. A public benchmark is exactly that artifact. It is also why the counter-move (a private 20-query eval, rerun quarterly) is worth the two afternoons it takes to build.

Metaview itself claims 4,000+ organizations run sourcing through its platform, which is the buyer population most exposed to the benchmark's framing. If you are one of them, or evaluating against them, the 20-query template above is the fastest way to move from "trust the leaderboard" to "trust the roles you actually filled."

FAQ

Is the Exa People Search Benchmark independent?

No. Exa built the benchmark, open-sourced the harness, and runs its own agent on it. That is closer to a well-documented industry test than a peer-reviewed one. The mitigating factor is that the harness and dataset are fully public at github.com/exa-labs/benchmarks, so any vendor or buyer can rerun it. The academic alternative, PeopleSearchBench (arXiv 2603.27476), uses a different 119-query set and three-dimensional scoring, and produces a different leaderboard.

Does a 93.5% score mean Metaview will find the people I need?

Not necessarily. The benchmark measures precision on synthetic role + geo queries like "senior payroll specialist in boston." It does not test multi-constraint stacks, GitHub signal, tenure math, security clearances, or named-individual lookups, which are half of what real recruiter briefs ask for. A high benchmark score means the vendor has a real people index. It does not mean the vendor handles your hardest queries. Include at least 5 constraint-stacked queries in your private eval.

What makes the tail score under 30%?

Data source, not model quality. Tools routing through a general web index return job postings, listicles, and company pages alongside profiles, which collapses precision on a strict role + geo test. Tools with a proprietary, refreshed people index sit in a different retrieval regime. The 49-point cliff between #2 and the pack is the visible boundary between those two categories.

How often should I rerun a private benchmark?

Quarterly, at minimum. People indexes refresh on different cadences (Exa is weekly), models change under vendors' hoods without version numbers, and pricing shifts. A vendor that led your eval in Q1 can slip in Q3 without any announcement. Log latency and cost per query alongside precision, and keep the same 20-query set stable across runs so you are measuring the tool rather than the test.