The Agent-Builder Competency Read: Interview, Probe, or Pass

You have a candidate's public work open and you need to decide one thing: can this person design and operate a production LLM agent, or can they only wire up a demo wrapper? This guide is for engineering managers, technical founders, developer-relations leads, and technical sourcers who screen agent builders from GitHub and the open web. It gives you a scoring rubric tied to verifiable public artifacts and a defensible interview, probe, or pass verdict with the evidence attached.

The reason this standard is needed: existing hiring guides count stars, frameworks, and demos. Those are the signals that lie most often. The scarce skill, the one that separates an agent engineer from a prompt author, is reliability discipline, and it is invisible to a keyword scan. You have to open the code.

What separates a production agent builder from a wrapper author

A production builder leaves behind a running service, reproducible models, a written eval methodology, and reliability machinery in the code. A wrapper author leaves behind notebooks, a single accuracy number, and a README full of frameworks. The difference is visible in the artifacts if you know where to look.

Reviewers of AI-engineer profiles consistently contrast two eras of work. The notebook era is a .ipynb file with inline output and a copied tutorial. The production signal is a running service, a model versioned and reproducible from a script, ingestion and cleaning and validation as separate testable steps, a documented retrieval pipeline with a stated chunking strategy and retrieval metrics, and a written eval methodology with baselines and failure cases.

The separating signals, the ones a wrapper author almost never produces, are traces, tool-integration code, retry and guardrail logic, and a clear path to inference. If those are present and genuine, you are looking at someone who has operated an agent, not just prompted one.

The surface layer is where star-counting and framework-listing live, and it is the least predictive. Spend your minutes on the two top layers.

Why framework fluency is the wrong thing to score

Framework fluency is abundant; reliability discipline is rare. Scoring for the first finds you the weakest candidates faster, because evaluation, observability, cost control, and guardrails are the scarce skills and none of them show up in a keyword match.

The pool data makes the abundance concrete. In Refolk's index of professional profiles, framework keywords are common enough that filtering on them barely narrows the field.

Counts from Refolk's index, same LLM-agent keyword; multiples are derived. The LangChain row uses an LLM skill filter, so treat it as indicative rather than exact. The point stands either way: hundreds of people name these tools. That is not a filter, it is a starting population.

The candidates who list every framework and call every project cutting-edge are frequently the weakest, precisely because the dimensions that matter are not in the README. Score the invisible layers instead.

The five scoring dimensions and what each proves

Score five dimensions, each on a 0 to 3 scale, each tied to an artifact you can point a URL at. The rubric below is constructed for agent engineering specifically. No public source defines numeric thresholds against agent dimensions, so treat the thresholds as a defensible convention you can tune, not a published standard.

Each dimension carries a specific burden of proof. Evaluation rigor proves statistical judgement: a single accuracy number, especially one computed on data the model already saw, does not. Production reliability proves the person has operated the thing under load, where it fails differently than in a demo. Claim honesty is the cheapest to fake and the most telling when genuine, because a candidate who publishes a harness that produces a lower number is more trustworthy than one quoting a higher number with nothing behind it.

A note on pattern vocabulary, because it tells you whether someone has read the field. The dominant vocabulary is Anthropic's. Anthropic worked with dozens of teams building LLM agents and found the most successful implementations use simple, composable patterns rather than complex frameworks. The five named patterns are prompt chaining, routing, parallelization, orchestrator-worker, and evaluator-optimizer. ReAct and Plan-and-Execute are the loop-level patterns. Orchestrator-worker is the multi-agent one and is also the most expensive.

A candidate who cut a multi-agent design to a single loop for reliability is showing judgement, not a gap.

This is why restraint reads as a positive signal. A candidate who deliberately did not use an agent where a script would do, or who collapsed an orchestrator-worker design into one loop to make it reliable, is demonstrating engineering maturity. The expensive pattern avoided on purpose is worth more than the complex pattern shipped for show.

Grading evaluation claims against real benchmarks

Grade the methodology, not the number, because the number is gameable. A candidate who links their eval harness and states the setting is more credible than one quoting a higher bare score, and the gap between those two is where you win or lose the hire.

The two benchmarks that appear in strong work are SWE-bench, for coding agents, and tau-bench, for tool-agent-user interaction. Know their shapes so you can tell a real run from a vague claim.

SWE-bench full is 2,294 real GitHub issues from popular open-source repositories. SWE-bench Verified is a curated subset of 500 tasks filtered by human annotators and stratified from Easy, under 15 minutes, to Very Hard, over 4 hours, by human-estimated solve time. tau-bench proposed pass^k to evaluate reliability over multiple trials, which matters because consistency is the thing that breaks in production.

The numbers to anchor your expectations: on the original SWE-bench, the best model at the time, Claude 2, resolved only 1.96% of issues with BM25 retrieval. Even state-of-the-art function-calling agents like gpt-4o succeed on under 50% of tau-bench tasks and are inconsistent, with pass^8 under 25% in retail. So treat high scores with suspicion and ask for the harness.

When you find a candidate who publishes a real run with its setting, note the reliability metric. A pass^k on tau-bench or a pass@3 on SWE-bench Verified tells you they understand that an agent that works once is not an agent that works.

Finding these people by hand means opening repo after repo. Refolk lets you ask for the artifact directly, in plain English, across the public GitHub graph and professional profiles, so your screening starts from candidates who already clear the dimension you care about most.

The procedure: from a profile to a scored verdict

Run the seven steps in order. The first two are sourcer work and take roughly 25 minutes; the scoring steps are EM work and take roughly 75 minutes for a serious candidate. Identity comes first, then depth, because a verdict built on a fork or an agent-authored repo is worthless.

Some sourcing guides put channel selection before artifact review. The GitHub-profile sources treat artifact depth as the first real screen after identity, and that is the order I use here, because a channel decision made before you know whether the work is real wastes the channel.

For the identity step, verification detail matters. A repo is classified AI-authored when a Co-Authored-By trailer in any commit or PR references a known AI tool, which is the most reliable single indicator. Detected tools include Claude Code, GitHub Copilot, ChatGPT, Devin, Codex, Aider, Cline, Roo Code, Augment, Continue.dev, Gemini, and Windsurf. Forks are flagged by GitHub's "forked from" banner directly. AI-created MCP servers are detected by querying up to 10,000 commits, 30 recent PRs, and the recursive file tree.

Turning five scores into a verdict

With each of the five dimensions on a 0 to 3 scale, the maximum is 15. Weight artifact depth and claim honesty as gates rather than points: a candidate who scores zero on either cannot clear the top band, because an honest account of weak work beats a dishonest account of strong work.

Map the totals to verdicts like this. A score of 11 to 15 with both gates passed is an interview: strong evidence across depth, evaluation, and reliability. A score of 6 to 10, or a strong score gated down by one missing dimension, is a probe: enough signal to justify a 30-minute technical screen before you commit an interview loop. A score of 0 to 5, or any fork or agent-authored repo passed off as original, is a pass. Attach the evidence URLs to whichever verdict you record so the decision survives review.

How this read goes wrong: failure modes and false positives

The failure modes below are where a confident verdict turns out wrong. Each has a cheap check that defuses it. Treat this section as the most load-bearing part of the rubric, because a standard that produces false positives is worse than no standard.

• Star-counting. A high-star repo tells you about marketing, not engineering. The candidates who list every framework and call every project cutting-edge are frequently the weakest. Check: open the code, not the README.
• Forked demo passed as original. A high-star repo may be a fork. Check the "forked from" banner and the person's actual commit count inside the fork.
• Agent-authored commits credited to the human. Co-Authored-By trailers reveal this, but GitHub counts them only when the email is verified; an arbitrary noreply address renders the trailer but will not link to a profile. Check: read the trailers and ask the candidate to walk a commit live.
• Single accuracy number as evaluation. A single number, especially one computed on data the model already saw, does not demonstrate statistical judgement. Check: demand a train/validation/test split and a stated baseline.
• Polished output hiding a broken side effect. A timeout means the result may be unknown; it does not prove a write failed. Check: look for idempotency keys and read-after-write verification, not just a success message.
• "Cost-aware" with no instrumentation. A claim of cost control with no per-trace cost tracking. Check: grep for token and cost logging or an observability integration such as an OpenTelemetry-native tracer.
• Benchmark number without methodology. Resolution rate dropped from 12.47% to 5.49% when only correct fixes were counted. Check: confirm the eval harness and setting are linked.
• Context decay invisible in short demos. Demos pass at four or five turns and fail at 15 to 20. Production exposed the failure when sessions that usually took four or five turns stretched to 15 or 20. Check: look for context-length-labeled traces.

The last one has strong supporting evidence. In 1,500-plus agent runs traced with an agent-reliability toolkit, failure rates accelerated as context grew. The four documented agent failure modes to probe for are malformed tool calls, context saturation, unbounded loops, and text-versus-tool ambiguity. A candidate whose code handles these is one who has watched an agent fail in production.

The single most discriminating probe sits inside the "polished output" failure mode. For any transactional agent, compare the requested change with the resulting state, check identifiers and amounts and permissions, and verify a retried action did not create a duplicate. A polished receipt cannot repair a transaction that never committed or committed twice. Verifying that a retried action did not duplicate a charge is the test a convincing demo cannot pass, which makes it the reliability question worth saving for the interview.

Keeping the read current and sourcing the next batch

The rubric is evergreen, but the inputs drift: new AI coding tools appear, benchmarks get revised, and pattern vocabulary shifts. Re-check the mechanism, not the specific value. Confirm your Co-Authored-By tool list against current detectors, confirm the benchmark sizes against their source repositories, and keep scoring the invisible dimensions regardless of which framework is in fashion this quarter.

Before you record a verdict, run the checklist.

On sourcing, size your market before you set your filter. In Refolk's index, the United States holds 349 LangGraph-with-LLM-agent profiles against 34 in the United Kingdom, 15 of them in London.

Both counts from Refolk's index for the same skill and keyword; the roughly 10x ratio is derived. The lesson for a thin market: a UK-only search filtered on a framework name will exhaust fast. Widen to "ships production agents" phrasing rather than a framework keyword, because small absolute pools mean framework-name filters over-constrain. The same phrasing shift helps everywhere, since the reliability dimensions you actually score are not framework-specific. Refolk lets you ask for the behaviour, not the keyword, so the candidates who come back are already the ones worth opening the code for.