Bespoke Labs Raised $40M for Agent Evals. The Title Barely Exists.
Bespoke Labs' $40M round exposed a hiring gap: the "evals engineer" title barely resolves on LinkedIn. Here is how to source the real pool.
Bespoke Labs closed a $40M Series A on July 6, 2026 to build simulation and evaluation environments for AI agents. Every founder and recruiter reading the announcement is now writing a JD for an "evals engineer," and almost none of them will find one through a LinkedIn title search.
VC dollars moved to the operating layer (evals, simulation, RL environments) about eighteen months faster than the talent taxonomy did. If you source on titles, you pull HVAC reliability testers. If you source on signals, you pull the people who wrote Terminal-Bench.
Why Bespoke's round matters for anyone hiring evals engineers
Bespoke Labs' $40M round is the cleanest recent proof that the operating layer under agents is now a real category, and it is pulling every applied-AI company into the same tiny hiring pool. Wing VC led the Series A; 8VC led the earlier seed. The angel list reads like a talent map: Jeff Dean, Tristan Handy (dbt Labs), Spiros Xanthos (Resolve AI), Dheeraj Pandey (DevRev), plus angels from Anthropic, OpenAI, and Meta.
A few facts to anchor on:
- Bespoke is a ~40-person company, founded in 2024 by Mahesh Sathiamoorthy and Alex Dimakis.
- They ship Terminal-Bench (used by Anthropic, OpenAI, and Google DeepMind for agent evaluation), the OpenThoughts reasoning dataset, and GEPA, a Genetic Evolutionary Policy Algorithm for prompt and policy tuning.
- They added 13 team members in June 2026 alone, including a Head of Data Ops from Grammarly and a Data Ops PM from AWS.
- The round is one of several in the same thesis: Mechanize (~$9M angel, April 2025), Prime Intellect ($21M total), HUD ($16M Series A).
That is a lot of new headcount hunting the same profile. The AI agents market itself is projected to grow from $7.8B in 2025 to $52.6B by 2030, and evaluation infrastructure is the bottleneck sitting under all of it.
What "evals engineer" actually means
An evals engineer owns the quality lifecycle of an AI system: dataset curation, scorer design, LLM-judge calibration, regression tracking, and the simulation environments agents run against before production. It is not ML research. It is not backend. It is not QA. It is the seam between all three.
The title barely existed in 2023 and only started appearing formally in enterprise postings in late 2025. That two-year lag between "role exists in practice" and "role exists as a title" is exactly why title-based sourcing fails here.
Two flavors of the role, which matter for how you filter:
- Product evals. Regression suites, LLM-as-judge pipelines, offline replay, agent trajectory scoring. This is what Cursor, Harvey, Sierra, Perplexity, Decagon, and Cognition hire in their first ten engineers.
- Capability and alignment evals. Red-teaming, dangerous-capability probes, model behavior evals. Anthropic, OpenAI, Google DeepMind, Mistral, and xAI run dedicated teams here. OpenAI's Engineering Manager, Evals (API) posting sits at $325K to $405K.
A PhD is a weak filter for the first flavor and a strong one for the second. If your JD does not split those, your funnel will not either.
The title filter is worse than useless. It is actively misleading.
In Refolk's index of US professional profiles, only 38 people carry a title matching "Evaluation Engineer" or "AI Evaluation," and 7 of the top 10 employers are aviation, HVAC, and automotive companies. The clean AI-evals title pool is 5.
| Slice (Refolk index, US) | Count | Top employers | Notes |
|---|---|---|---|
| Title contains "Evaluation Engineer" or "AI Evaluation" | 38 | GE Aviation (4), Hussmann (3), Dell (2), Honeywell (2), Wayve (1) | Majority are aviation and automotive QA roles, not AI |
| Title = "AI Coding-Agent Evaluation" (exact-ish) | 5 | ServiceNow, Intuit, Salesforce, DataAnnotation, iD Tech | The clean-title pool |
| Noise-to-signal ratio in title search | ~7.6x | - | 38 divided by 5 |
| Bay Area share of clean-title pool | 3 of 5 (60%) | - | Geographic concentration mirrors funding |
The mechanism is boring and important: the phrase "evaluation engineer" pre-existed AI by decades in industrial QA. A Boolean search treats a GE Aviation turbine test lead and a Sierra agent-evals engineer as the same person. They are not.
Sourcing evals engineers on title is like sourcing kernel engineers by searching "kernel." You get chefs.
What to search instead: seven signals that actually resolve the pool
Search for delivery evidence, not credentials. Evals engineering hires on repos, deployed agents, and shipped eval suites, so the best signals live on GitHub and in benchmark leaderboards, not in headline titles.
The seven signals I would build a search around:
- Terminal-Bench contributors, forkers, and issue authors. Bespoke's own team is largely defined by this repo.
- SWE-bench and SWE-bench Verified submissions and PRs.
- GAIA, τ-bench, and WebArena contributors. These are the canonical agent benchmarks.
- GEPA users and forks. Anyone tuning prompts or policies with a genetic algorithm has already picked a side.
- OpenThoughts dataset contributors.
- Braintrust in a profile or README. Braintrust publicly hires eval-focused engineering roles, so its users are pre-qualified on the stack.
- JD keywords that show up in real evals postings: "LLM-as-judge," "trajectory scoring," "offline replay," "regression suite for agents," "red-team harness," "reward model calibration."
This is the exact gap Refolk closes. You describe the person in plain English ("US engineers who have contributed to Terminal-Bench or SWE-bench in the last 18 months and shipped an LLM product") and get a ranked shortlist across GitHub, LinkedIn, and the open web, instead of a Boolean string that returns turbine testers.
The comp arbitrage nobody is pricing correctly
Two title strings in the same market are worth $86K apart for the same human, and neither one is "evals engineer." Levels.fyi shows "AI Software Engineer" at a $245K median and "AI Engineer" at $159K.
If you are hiring, this is a gift. If you are sourcing, it is a trap: candidates who let themselves get labeled as "AI Engineer" are hidden from anyone searching the higher-priced string. The people are the same. The keywords are not.
The practical move:
- Search both strings, plus "member of technical staff," plus the benchmark signals above.
- Do not price your offer to the title the candidate happens to carry. Price it to the work they have shipped.
- If you are on the candidate side, retitle your LinkedIn headline before your next round of outreach. It moves the median.
The adjacent pools that convert (and cost less than poaching Anthropic)
The two adjacent pools that convert into strong evals engineers are senior QA and SDET engineers with Python and one shipped LLM project, and MLOps engineers who own model-monitoring pipelines. Both are cheaper than a frontier-lab poach and both already own the quality-lifecycle muscle the role requires.
Why they convert:
- Senior QA and SDET. They already think in test harnesses, flaky-test triage, regression baselines, and dataset drift. Add LLM-as-judge and trajectory scoring and you have an evals engineer. The Python bar matters. The "one shipped LLM project" bar matters more.
- MLOps and ML platform. They already own monitoring, drift detection, and dataset versioning. The gap is agent-specific: trajectory replay, tool-call scoring, sandboxed environments.
- Applied research engineers with a benchmark PR. Cheaper than a full research hire, better calibrated on eval design than a pure infra engineer.
Frontier labs poach each other on this profile and you will not win that auction. You do not need to fish in that pond. You need to find the SDET at a mid-stage fintech who spent Q2 shipping an internal agent and wants to do it full-time. That is another place Refolk earns its keep: "senior SDETs in the US who have starred or contributed to a benchmark repo and list Python as a primary language" is a plain-English query, not a five-line Boolean.
What a real Bespoke-competitive search looks like
The winning search for an agent-evals engineer in July 2026 is a compound query on signal, not title, filtered by geography and stage. Here is the shape I would use for a founder trying to hire the next five people onto an eval infra team.
The layers, in order:
- Benchmark evidence. Contributor or issue author on Terminal-Bench, SWE-bench, GAIA, τ-bench, or WebArena in the last 18 months.
- Stack evidence. Public repo or profile mention of Braintrust or comparable eval platforms.
- Deployment evidence. At least one shipped LLM-powered product, agent, or internal tool.
- Adjacency filter. Currently or recently at an applied-AI company (Cursor, Harvey, Sierra, Perplexity, Decagon, Cognition), a frontier lab, or a mid-stage SaaS shipping agents.
- Geography. Bay Area weighted (60% of the clean pool sits there), NYC and Seattle secondary.
- Adjacent-pool expansion. Senior SDETs and MLOps engineers with one LLM shipping credit, added as a second tier.
Run that and you get a shortlist that is not gated by whether the candidate's current title happens to contain the word "evaluation." Run it as a LinkedIn Boolean and you get 38 profiles, 7 of them at aviation and HVAC companies.
The Bespoke round is a preview of the next four quarters. Every company funded in the operating-layer thesis (Mechanize, Prime Intellect, HUD, plus whoever is announced next) is going to write a JD that reads the same. Whoever figures out the signal-based search first hires the pool. Whoever waits for the title to catch up hires nobody.
FAQ
What is an evals engineer, in one sentence?
An evals engineer owns the quality lifecycle of an AI system: dataset curation, scorer design, LLM-as-judge calibration, agent trajectory scoring, and the simulation environments a product runs against before it ships. The role sits between ML research, backend, and QA, and it barely existed as a formal title before late 2025.
Why does searching "Evaluation Engineer" on LinkedIn return the wrong people?
Because "evaluation engineer" is a decades-old industrial QA title. In Refolk's index, 38 US profiles carry it, and 7 of the top 10 employers are GE Aviation, Hussmann, Honeywell, and other aviation, HVAC, and automotive firms. Only one of the top employers is a clean AI company (Wayve). Title search gives you turbine testers. The real pool is discoverable through benchmark contributions and stack keywords, not titles.
What benchmarks should I search for to find real agent-evals engineers?
Terminal-Bench (maintained by Bespoke Labs, used by Anthropic, OpenAI, and Google DeepMind), SWE-bench and SWE-bench Verified, GAIA, τ-bench, and WebArena. Contribution to any of these, plus a shipped LLM product, is a stronger signal than any LinkedIn title. GEPA users and OpenThoughts contributors are additional high-signal populations.
Which adjacent roles convert into evals engineering the fastest?
Senior QA and SDET engineers with strong Python and at least one shipped LLM project, and MLOps engineers who already own model monitoring and drift detection. Both groups already think in test harnesses, regression baselines, and quality pipelines, which is most of the evals role. They are meaningfully cheaper than poaching from a frontier lab.