Scoring Agentic-Development Skill From a Candidate's Public Repos

You have a candidate's GitHub open and one question to answer before you spend an interview slot: are they genuinely good at building with AI coding agents, or just riding the hype? This guide gives engineering managers, technical founders, and sourcers a fixed set of dimensions to score agentic-development skill from public repository evidence, and a rule for turning that score into advance, probe-in-interview, or pass.

This is not a fraud check and not a slop detector. Those ask what is fake or low quality. This measures a positive skill: how well someone instructs, orchestrates, and supervises coding agents. And it runs before first contact, so you can triage a pipeline without scheduling a live assessment.

What "agentic-development skill" actually means here

Agentic-development skill is the ability to instruct, orchestrate, and supervise AI coding agents so their output ships safely, not the ability to invoke a tool. The distinction matters because tool access is now common while supervision is scarce.

Most AI-generated pull requests receive no review at all, and the reviews that do happen are largely dominated by AI agents rather than humans. That is the crux. If nearly nobody supervises agent output, then a candidate who visibly does - scoped diffs, added tests, real human iteration - is demonstrating the rare part of the job. Tool presence is table stakes. Oversight is the skill.

So the whole guide leans on one hierarchy: read oversight first, config quality second, tool presence last.

Tool presence is table stakes. Visible supervision is the skill worth an interview slot.

The five dimensions and what each one proves

Score every candidate on five dimensions. Each has a positive signal you can find in public traces and a tell for when that signal is lying to you.

The dimensions, in the order of how much they should move your decision:

1. Oversight evidence - human review, fix follow-ups, tests added after generation. Proves the candidate closes the loop on agent output. Lies when green CI is mistaken for review.
2. Commit and PR discipline - scoped diffs, message-to-diff fidelity, resolved imports. Proves the candidate controls what the agent emits. Lies when a large diff is split cosmetically but never actually reviewed.
3. Config quality - substantive AGENTS.md or CLAUDE.md that targets real agent error classes. Proves deliberate setup. Lies when the file is a default template.
4. Config-to-use fidelity - committed rules match what CI and scripts actually do. Proves the config is live, not decorative. Lies when the file claims practices no workflow reflects.
5. Tool-presence markers - Co-Authored-By trailers, tool skill claims. Proves a tool touched the work. Lies constantly: toggleable, not retroactive, and absent by choice.

Notice the weighting is inverted from intuition. The easiest thing to find, a filename, proves the least. The hardest thing to read, iteration history, proves the most.

Which public artifacts to read, and how each one misleads

The artifacts worth pulling are repo-root context files, commit trailers, PR bodies, and review threads. Each proves something narrow, and each has a documented way of lying.

Context files are the anchor. AGENTS.md is an open standard: a markdown file holding rules about the project - architecture, tool docs, testing practices, and constraints whose absence leads to agent errors. Vendor variants carry similar content under different names: CLAUDE.md, .cursor/rules, .clinerules, .github/copilot-instructions.md, and GEMINI.md. Their presence tells you the repo is set up for agents. It does not tell you the setup is any good.

Commit-level artifacts come from the tooling defaults. Claude Code appends a Co-Authored-By: Claude trailer to commits and a "Generated with Claude Code" line to PR bodies by default, and GitHub creates a ghost contributor entry so Claude's avatar appears in the contributors graph. Useful, but the setting only applies going forward - trailers in existing commits are never changed - so absence proves nothing, and team members can override the setting locally.

The table's discipline is worth internalizing: never read the middle column without the right column. Every artifact here has a failure mode built in.

One platform detail helps you read review threads correctly. GitHub now shows contributor role labels - First-time contributor, Contributor, Member - inline in the pull request list view for public repositories. That lets you see at a glance whether a candidate is a core member iterating on their own work or a drive-by contributor, which changes how you weigh their review activity.

Reading commit and PR discipline

Score PR discipline with three cheap checks: scope, import safety, and message-to-diff fidelity. A candidate who passes all three is signaling the exact discipline that avoids the AI merge tax.

The documented review heuristics, drawn from a practitioner checklist and the message-code-inconsistency literature:

• Scope. Filter the diff by declared scope. Any out-of-scope file should trigger a split. A PR that claims one thing and touches ten unrelated files is a supervision failure.
• Import safety. Verify imports resolve and no top-level input/output runs on import. Import-time side effects are a classic agent-introduced bug that green CI can hide.
• Message-to-diff fidelity. Commit messages must map to the diff within roughly a 20% margin, otherwise reject. This is the highest-yield single filter you have.

Fidelity earns its place with hard numbers. High message-code-inconsistency PRs take 3.5 times longer to merge and waste around 40 hours per PR. A candidate whose messages track their diffs is demonstrating the discipline that removes that tax. Also watch for the shallow tells: an empty diff, a description too brief for the code changed, or template markers like [WIP] and generic phrases.

One caveat on that ceiling: a 400-plus line diff can be a legitimate generated migration. The question is not the size but whether it was broken into commits a human could actually review. Size alone does not convict.

Calibrating against absence: base rates you must respect

Absence of agent signals is weak evidence today, and it is weakest of all for agents specifically. Before you penalize a blank repo, weigh it against how few engineers have left agent traces yet.

The prevalence picture, from the 2025 Stack Overflow survey:

Read the split carefully. General AI use is near-universal, so a total absence of any AI signal is mildly meaningful. But a majority still avoid agents entirely, so absence of agentic traces is not meaningful at all - it confuses adoption lag with lack of skill. A strong engineer may simply work in a codebase or team that has not adopted agents.

This is also why the tool-presence layer sits at the bottom of the hierarchy but still carries weak positive weight. In Refolk's index, 3,606 US professionals list Claude Code as a skill against 1,195 for Cursor, and Germany shows 572 for Claude Code, roughly one-sixth the US count. Naming a specific agent tool is a self-selecting early-adopter marker, so its presence is a stronger-than-average positive signal in less-saturated markets - but its absence remains no signal at all.

Running that search by hand means cross-referencing profile skills against repo contents across the public GitHub graph. Refolk does the join for you, so the pipeline you triage is already the one worth reading.

The scoring procedure

Follow this procedure end to end for about an hour per candidate. It moves from cheap collection to the judgement call, and every step has a clear "done" state so you can stop when the evidence is decisive.

A note on step order. The practitioner checklist runs scope first and then reads the diff; the message-code-inconsistency research recommends description-versus-diff heuristics before reading the diff at all. Both work. Pick the order that fits how you read, and stay consistent so your scores are comparable across candidates.

The per-PR discipline step is efficient by design. The checklist takes about 10 to 12 minutes for a typical four-commit PR and catches at least four of the six error families a human-only checklist would miss. Sample two or three PRs rather than auditing everything.

Converting the score to a decision

Map the composite to one of three outcomes on two axes: how strong the oversight evidence is, and how much of it there is. The matrix below is the judgement call the whole guide builds toward.

The bands below are guide-original. No published, validated rubric maps public repo traces to an agentic-skill score, so treat these thresholds as a defensible starting point built on documented heuristics, not an industry standard. Tune them to your bar.

Oversight evidence (0-3): human review, fix follow-ups, post-gen tests
Commit/PR discipline (0-3): scoped diffs, message-to-diff within ~20%, resolved imports
Config quality (0-2): substantive AGENTS.md, not default template
Config-to-use fidelity (0-1): rules match CI steps and scripts
Tool-presence markers (0-1): trailers or named tool skill (weak positive only)

ADVANCE: 7+ with oversight >= 2
PROBE IN INTERVIEW: 4-6, or high tool presence with oversight <= 1
PASS: below 4 with clear supervision failures (not mere absence)

Write the one-line rationale every time. "Advance: two PRs with scoped diffs, human review threads, tests added after generation" is defensible. "Advance: has an AGENTS.md" is not.

How this scoring goes wrong

The failure modes below are where a careful reviewer still reaches the wrong verdict. Give them more weight than the happy path, because each one has burned someone.

• Committed-but-unused config. A polished AGENTS.md that no CI step or command reflects. Cross-reference the file's claims against .github/workflows and the actual scripts before crediting it.
• Trailer absence misread as no-AI. Trailers are toggleable and never retroactive, and members can override the setting locally. Judge diff cadence and PR bodies, not the presence of a trailer.
• Green-CI halo. The diff looks clean, CI is green, messages read well, yet the PR slips a subtle bug: a mock-heavy test, a wrong API signature, an import-time side effect. Run the scope and import audit anyway.
• AI-detector overreach. Detectors flag clean senior code and non-native-English developers as AI. One vendor reports 96.2% accuracy with a 0.3% false-positive rate on 40-plus-line snippets, but even favorable numbers do not justify a hiring verdict. Use the score as a signal, never a decision.
• Review-metric illusion. Agent-mediated review inflates "reviewed" counts without human oversight. Confirm a human left substantive, non-agent review comments before counting a PR as supervised.
• Message-code inconsistency. A PR claims phantom changes. Verify the diff is not empty and the description length matches the change size.
• Oversized-diff penalty applied blindly. A 400-plus line diff may be a legitimate generated migration. Check whether it was split into reviewable, single-scope commits before penalizing.
• Base-rate blindness. Penalizing absence when a majority of developers still do not use agents. Treat absence as inconclusive.

Verify before you decide

Run this checklist before you write any candidate's rationale. It catches the mistakes that turn a defensible score into a guess.

Keeping the standard current

The evidence surface moves faster than most hiring rubrics, so re-check the mechanisms rather than the values. Three things will drift.

First, the AGENTS.md ecosystem. The spec was formalized as an open standard in August 2025, led by OpenAI with Google, Cursor, and Factory, then donated to the Linux Foundation's Agentic AI Foundation in December 2025. Since August 2025 more than 60,000 open-source projects have adopted it and more than twenty AI coding tools support it. As adoption climbs, the presence of an AGENTS.md loses signal value and content quality becomes the only thing worth reading.

Second, the base rates. Agent adoption sits at a minority today, which is why absence is inconclusive. When the majority crosses into daily agent use, absence will start to mean something. Recheck the developer survey before you trust the calibration step.

Third, the artifact defaults. Tool vendors change how attribution works - one attribution config landed in a specific Claude Code release and deprecated an older flag - so the exact trailer format and its defaults will shift. Confirm what the current tooling emits before reading any trailer as a fixed signal.

The most durable part of this standard is its spine: read oversight first, because supervision is the scarce skill, and never let a filename or a detector score make the decision that belongs to your reading of the work.