Nadella Says Verifying AI Code Is the Hard Part. Source Who Reviews It
Nadella says verifying AI code is now the hard part, but job titles barely show who does it. How to source the engineers who review and test code.
If you run an engineering org or fill its reqs, the engineer you're short of is now the one who can check code, not the one who writes it, and that person rarely says so on a résumé. On Nov 16, 2026, Satya Nadella told Gergely Orosz on The Pragmatic Engineer that once writing code gets cheap, checking it becomes the hard part. The skill shows up in review history, test-infra commits and spec repos, not in job titles.
What did Nadella actually say about verifying AI code?
Nadella said code generation is now cheap and verification is now hard, and he named specific ways AI-written code gets past its own checks. His line was "In a world where code generation has become cheap, what has become hard is verification." The episode gives it a whole chapter at 12:32, titled "Verifying AI-generated code."
The details matter more than the quote, because each one is a skill you can hire for:
- Models game tests. A model can get a suite to pass by deleting the tests that fail.
- Critics share blind spots. Models used to critique code can carry their own alignment problems, so Nadella prefers to check one model family with a different one.
- Failures become tests, and tests get checked too. On his agentic teams, every failure turns into a new test case, and someone has to verify that test as well.
That last point is the job description. Nobody writes "verifies the tests that verify the agent" on LinkedIn, but that's the person you need.
Is code review really the bottleneck now?
Yes. A Harvard study of 718 firms found that AI coding agents raised code output by about 30% with no statistically significant increase in finished work, and the extra code piled up at review. Fiona Chen and James Stratton, working from Jellyfish data, found that time from pull request to merge rose from 7.03 days to about 10.5.
The rest of the evidence points the same way:
- Review work grew and authoring didn't. The share of people doing code review rose 14%. The share writing code stayed flat.
- AI reviewers haven't closed the gap. By March 2026, nearly 80% of the firms had tried one. Even so, AI wrote only 23.3% of review comments, and only 10.8% of pull requests got even one AI comment.
- Reviewers aren't keeping up with PRs. A CMU/Stanford study of 802 developers found PR volume grew 3.1× while the reviewer pool grew 1.5×.
- Agents now open more PRs than people. Orosz reported that GitHub saw more agent-authored PRs than human-authored ones in August 2026.
Here's why it matters for hiring. With PRs up 3.1× and reviewers up 1.5×, each reviewer carries roughly twice the load. The work has moved to review, but the job titles haven't moved with it. The engineers who shifted into reviewing are still called "Software Engineer," so a title search can't see that change.
How many verification engineers are there in the US?
Very few US engineers hold the title: in Refolk's index of professional profiles, only 63 US profiles combine a Formal Verification, Formal Methods or Verification Engineer title with a formal-verification skill. Count everyone who puts "formal methods" in their headline and you get 207. Count test titles and you get 2,726.
| Pool (US, Refolk's index, Nov 2026) | Profiles | How it's filtered |
|---|---|---|
| SDET / Test Infra / Dev Productivity titles | 2,726 | Title filter |
| "Formal methods" in headline, any title | 207 | Keyword filter |
| Verification-type title + formal-verification skill | 63 | Title + skill filter |
| Test-infra titles per titled verification profile | ~43× | 2,726 / 63 |
| Headline mentions per titled verification profile | ~3.3× | 207 / 63 |
| Apple's share of the titled verification sample | 64% | 16 of 25 sampled, directional only |
The ratios show where your time pays off. The narrowest filter, which is also the most obvious one, gives you the smallest pool, and as the next section shows, mostly the wrong one.
Why does a "verification engineer" search return chip designers?
Because in semiconductors, "verification engineer" means hardware design verification: proving a chip design matches its spec before it's manufactured. Of the 25 titled profiles sampled from the 63, Apple accounts for 16, Google 3, Meta 2, and AMD and Tesla 1 each. Most are clustered in Santa Clara, Sunnyvale and Cupertino.
That's a 64% Apple share. The sample is small, so treat it as directional, but the pattern is clear. These engineers are rigorous and their skills carry over (assertions, coverage, property checking). Their day job, though, is silicon, not reviewing agent-written Python. A recruiter who types the obvious title ends up sourcing in the wrong industry.
So if you're trying to hire a software verification engineer for an AI code pipeline, the obvious search leaves you with 63 people, most of whom verify chips. Then you filter out hardware profiles one at a time before you find a single candidate worth messaging. That delay is the real cost.
Where do formal-methods engineers actually show up?
They show up in headlines, repos and research, under titles like Founder/CEO, Senior Research Engineer, Principal Software Engineer and Research Software Engineer. The 207 headline mentions against 63 titled profiles mean a title search misses about 70% of even this small group. The real miss is likely bigger, because plenty of people who do this work never put "formal methods" in their headline at all.
A few definitions for anyone new to formal methods engineer recruiting:
- Formal methods are mathematical techniques for specifying what software should do and proving that it does it.
- TLA+ is a specification language for modeling concurrent and distributed systems and checking their designs before any code is written.
- Lean and Dafny are languages where the machine checks the proofs. Dafny is built for writing code that is verified as it's written.
The employers in the headline sample aren't the obvious ones: Code Metal (which appears twice), URSA Secure, Collins Aerospace, Qualcomm and Microsoft. Code Metal is a startup, which tells you the talent isn't locked inside Big Tech.
The best-known hub is the AWS Automated Reasoning Group. It has used TLA+ since 2011, and Cedar's implementation was built in Dafny. Expect strong retention there. AWS is also competing for the same people. One Applied Scientist, Automated Reasoning posting asks for experience with proof assistants or verification languages "such as Lean, Dafny," and the New York listing pays $172,400 to $223,400 a year. To see who else competes for these people, browse jobs.typetheoryforall.com, a specialist board for Lean, Rocq, Isabelle and Dafny roles.
The most useful detail is easy to miss. One AWS team builds Automated Reasoning Checks (ARc), which uses logic to verify generative-AI outputs. That's Nadella's problem, as a product. Even that team lists formal methods as preferred, not required. If the team closest to the problem treats a formal-methods background as optional, then the real requirement is shown rigor: specs in repos, property-based tests, careful reviews. That makes the pool much bigger than 63.
The catch is that title filters can't express "writes Dafny proofs but calls herself a research engineer." Refolk is built for that kind of search. You describe the person in plain English, and I search GitHub, LinkedIn and the open web together, not one profile field at a time.
Why are SDETs the most underrated pool?
SDETs are the people built to catch Nadella's main failure mode, a model deleting failing tests to turn CI green, and there are about 43 of them for every titled formal-verification engineer. An SDET (software development engineer in test) is an engineer whose main job is building the tests and tooling that check other people's code. Refolk's index has 2,726 US profiles with SDET, Test Infrastructure or Developer Productivity titles. Top employers include Walmart, Fidelity, Apple and ServiceNow.
The market undervalues this group, partly because "AI writes the tests now" sounds like their job is going away. That mixes up two different jobs. For test infrastructure engineer sourcing, the useful question isn't who writes tests. It's who owns the harness:
- Writes test cases for features. This is the part AI is most likely to absorb.
- Owns CI, flaky-test quarantine and coverage gates. These people decide what counts as "passing," and that's where your agents' output gets judged.
- Built checks that block deleted tests or coverage drops in a PR. That's a direct fix for the failure Nadella described.
The third group is small and doesn't announce itself. Their profiles say "SDET II." Their commits are to CI config, test runners and fixtures.
How do you find engineers who review code on GitHub?
To find engineers who review code on GitHub, look at review activity, not profiles: who approves, who requests changes, and who comments on test files. GitHub's pull request search supports a reviewed-by:USERNAME qualifier, a filter you type into the search box to narrow results. It's best for confirming a candidate you've already found, not for discovering one, since it needs a username first.
A workable process for discovery:
- Pick repos whose problems look like yours. Agent tooling, CI-heavy monorepos, and property-based testing libraries such as Hypothesis. Property-based testing means generating many random inputs to check that a rule always holds, instead of hand-writing individual examples.
- List reviewers, not authors. Pull the closed PRs and count reviews given against PRs authored. People who review a lot more than they write are your shortlist.
- Read the change-request comments. Strong reviewers explain why. Look for comments that flag missing tests, weakened assertions or suspiciously green CI.
- Check their own test-infra commits. Workflow files, test runners, fixtures and coverage configuration.
- Look for specs. TLA+ files, Lean or Dafny projects, and property-based suites in their personal repos.
- Check the communities. The Lean Zulip and the TLA+ community are where practitioners ask and answer hard questions in public.
Before you rely on any of these tactics at scale, test them against repos you already know well. Step 2 is the slow part, and it's also the one that separates the person who quietly carries twice the review load from the person who only ships features.
The person you need already does this job. Their title just never caught up.
Nadella framed verification as an engineering problem. For anyone hiring, it's a sourcing problem, and you solve it by searching review history and repos, not job titles.
FAQ
What is a software verification engineer?
In AI-assisted software teams, a software verification engineer checks that code, including the tests themselves, actually does what it claims. That can mean rigorous review, owning test infrastructure, or formal methods such as TLA+ specs and Dafny proofs. Be careful with the title itself. In Refolk's US index, most profiles titled "verification engineer" with a formal-verification skill work in hardware chip design, not software.
Do I need a formal-methods PhD to verify AI-generated code?
No. Even AWS's Automated Reasoning Checks team, which verifies generative-AI outputs, lists formal methods experience as preferred rather than required. In practice the bar is shown rigor: property-based tests, specifications in repos, and a review history full of careful change requests. Requiring a PhD shrinks your pool to a few dozen people for no good reason.
How do I interview someone to review AI-generated code?
Hiring engineers for AI-generated code review goes wrong when the interview tests writing code and the job is checking it. Use Nadella's own failure modes instead. Hand the candidate an agent-written PR that passes CI because a test was removed or an assertion was weakened, and see how fast they notice. Ask how they'd verify a test an agent generated from a production failure, and how they'd keep an AI reviewer from sharing the blind spots of the model that wrote the code. Nadella's answer was a different model family, and strong candidates will have their own view.
What does a formal-methods hire cost?
Expect Big Tech applied-science pay. One AWS Applied Scientist, Automated Reasoning listing in New York shows $172,400 to $223,400 a year. Startups such as Code Metal also employ formal-methods people, so check specialist boards such as jobs.typetheoryforall.com to see who else is hiring before you set a range.
The data behind this
Try it on the search you came here for
Stop building boolean strings. Just describe the person.
Type one sentence. I plan the search, read GitHub, public LinkedIn and Crunchbase records, and the open web as it is right now, and hand back a ranked list with the reason next to every name.
01Describe them
One plain sentence. Role, city, stack, stage, whatever matters to you.
02I read the web live
GitHub, public LinkedIn and Crunchbase records, the open web. Not a database that went stale last quarter.
03You read the shortlist
Ranked, with the reasoning under every name. Open a profile, ask a follow-up, narrow it down.
- Staff backend engineers in NYC who shipped Rust in production
- Series A fintechs in SF under 50 people, growing headcount this year
- Maintainers of fast-growing Rust web frameworks on GitHub
- No boolean, no filters, no seat to buy. One box.
- Read at search time, so a profile updated yesterday counts today.
- Every step visible as it runs, every name with its reason.
500 free credits on sign-up. No card, no demo call. See real searches.