Scoring Staff-Level Design Judgment From Public Writing

You are looking at a staff or principal engineering candidate and you want to know whether they can actually reason about system design, not whether they ship a lot of commits. This guide is for engineering managers, technical founders, developer-relations leads, and technical sourcers who need to grade a stranger's published design record before booking a screen. It gives you a per-dimension framework: where to find the artifacts, how to confirm the person drove them, how to score each dimension, and how to roll that into a verdict of staff-level judgment, mid-level execution, or borrowed credit.

The rest of the engineering scoring library reads code: commits, repos, review history, profile authenticity. This one reads the written design record - public RFCs, ADRs, design docs, and talks - because that is where staff and principal ability shows and where the prolific-committer heuristic goes blind. A great committer can be a mid-level executor. The design writing is where you see whether they can frame a problem, weigh alternatives, and own the operational reality.

Why score the written record instead of the commit history

The written design record is the only public place where staff-level judgment leaves a legible trace, because staff is a different job from senior, defined by influence and design reasoning rather than throughput. Commit volume measures execution; a merged RFC with a defended trade-off measures the judgment you are hiring for.

Two facts from Refolk's index make artifact-level scoring worth the effort. The senior IC pool is large but its design skill is thinly distributed, so title and skill-tag filtering will not separate your candidates for you.

That number is the case for reading the artifact rather than the profile. A skill tag is self-asserted; a merged KEP with a live trade-off argument is not. When the tag pool is this thin, the score has to come from the writing.

There is a second reason the written record beats the profile: attribution. Proposal venues record authorship structurally. Kubernetes built its enhancement process to track the chain of custody for a proposed change from conception through implementation, so who drove a design is recorded rather than claimed. A blog byline is a claim. A PR author field is a fact.

Where the attributable design artifacts live

The most reliable artifacts are version-controlled proposal documents, because their authorship is recorded in metadata and their history is recorded in commits. Company blogs and talk archives are weaker, because bylines and speaker credit are self-asserted and must be corroborated against a repo trail.

Rank your venues by how hard they are to fake:

KEPs are small structured design documents in which the author states the motivation, the design and its alternatives, the risks, and the tests, submitted and updated through a pull-request workflow on the k/enhancements repo. Rust RFCs are markdown files merged by PR, and the RFC number equals the PR number on merge, so the document and its full edit history sit in one place. Python PEPs carry a mandatory Author: header of real names plus an optional Sponsor: field that records which core developer is sponsoring the proposal; if an author is already a core developer, no sponsor is needed. Architecture Decision Records, in the Nygard format popularized in a 2011 post, are dated, numbered files living in-repo with four sections - Status, Context, Decision, Consequences.

Start your search in the KEP browser, which filters by SIG, stage, and author, in rust-lang/rfcs, and in the Python PEP index. Then widen to in-repo ADR directories and, last, to blogs and talk archives - which you treat as pointers to artifacts, not as evidence on their own.

The five dimensions and what each one proves

Score every driver-authored document on five dimensions: problem framing, alternatives considered, explicit trade-offs, operational maturity, and cross-team influence. Each dimension proves a specific thing, and each has a tell for when it lies.

Named guidance converges on what a strong document does. Google's design-doc guidance says a design doc should clarify the problem being solved, act as a discussion platform to refine the design, explain the reasoning behind decisions and trade-offs, and list alternative designs and why they were not chosen. Rust's own process warns that RFCs which do not present convincing motivation, demonstrate understanding of the design's impact, or are honest about drawbacks and alternatives tend to be poorly received. Those two sources give you the load-bearing markers.

Published senior rubrics agree on where the weight sits. Here is what the named rubrics assign:

One graded rubric puts Trade-offs and Bottlenecks at 25% and Detailed Design at 35%. Another states that trade-off analysis is the highest-weighted dimension because it is the strongest signal for engineering maturity: junior engineers pick technologies, senior engineers analyze trade-offs. Operational maturity is a separate 20% section, and most candidates give it one line, when the standard is failure mode plus user consequence plus mitigation.

What each dimension proves, and how it lies

• Problem framing. Proves the candidate starts from the problem, not the solution. The reference standard is three paragraphs at most: what the problem is, why the project is necessary, and how it fits the technical or product strategy. It lies when the framing is a restatement of the chosen solution's benefits.
• Alternatives considered. Proves they investigated a solution space. It lies as template theater: an "Alternatives" heading with no real comparison against the chosen design.
• Explicit trade-offs. Proves engineering maturity, and it is the hardest dimension to fake because a real trade-off leaves a visible argument in the PR thread. It lies when drawbacks are absent or disingenuous.
• Operational maturity. Proves end-to-end ownership: how they deploy, monitor, scale, and evolve the system. It lies by omission, appearing as a single line about reliability.
• Cross-team influence. Proves the Will Larson axis - leadership without authority. It lies when the author never engages the people the design affects.

Junior engineers pick technologies. Senior engineers analyze trade-offs, and the argument is right there in the thread.

At staff level the operational bar rises: the reader is checking whether the candidate can own a system end-to-end, not just architect it but face its operational reality. Will Larson's four staff archetypes - Tech Lead, Architect, Solver, Right Hand - are a reminder that influence, not more code, is the job. Score cross-team influence against whichever archetype the role targets.

The procedure

Work the artifacts in order, from location to a leveled verdict. The whole pass runs about two and a half to three hours per candidate, most of it in the scoring and thread-reading steps.

A note on order: the sourced rubrics disagree here. Some score requirements and problem framing first; others weight depth and trade-offs highest regardless of where they appear in the document. I score in the order above because authorship must be settled before any grade means anything, but I let the trade-off dimension dominate the final level.

How to confirm the person actually drove the design

Confirm authorship from the merge and approval trail, not the byline, because the merge trail records who wrote what while a byline only records who is credited. This is the single most important step, because a score attached to the wrong person is worse than no score.

Use the metadata to separate the three roles. The document's author is the PR opener and the Author: or authors: field. Reviewers and approvers sit in their own metadata fields. Commenters appear only in the thread. A commenter can read like an owner if you only skim the discussion, so anchor on the fields.

Then read the commit history to see the individual contribution. In Rust, edits are made as new commits to the pull request and authors are told not to squash or rebase after the commits are visible, so the history shows who wrote what and when. Kubernetes verifies the chain explicitly: the enhancements subteam confirms all PRs are merged, reaches out to KEP authors to understand which PRs are part of the KEP, and updates status. That is a paper trail you can follow.

For merged proposals, isolate the original contribution. Merged KEPs and RFCs reflect the subteam, not one mind, and RFCs rarely go through the process unchanged, especially as alternatives and drawbacks are surfaced. Read the early revisions of the PR to see what the candidate actually proposed before the committee reshaped it. The first draft plus the author's thread responses is the individual; the final merged text is the consensus.

Once you can find the merged proposals fast, the bottleneck moves to reading them. Locating the candidates in the first place is where a plain-English search saves the most time.

Refolk turns the location step from a manual repo crawl into one query, so you spend your hours on scoring rather than searching.

How this goes wrong: the false positives to guard against

The failure modes all share one shape: something looks like driver-level design judgment and is not. Guard against each with a specific check, because the polished final artifact is designed to hide exactly these gaps.

Byline is not author. Ghost-writing is a real, marketed service. Vendors sell converting senior engineers' internal notes, research findings, and GitHub documentation into readable, authoritative content, and one promises the audience will never know it is ghost-written. The mechanism means blog prose quality is uncorrelated with the named person's judgment. The false positive is fluent writing with no defense under scrutiny. Check for a PR or commit trail and whether the person answers hard questions in the thread.

Template theater and consensus laundering. A KEP or RFC can be template-filled with an empty alternatives section, and a merged proposal reflects the subteam rather than one mind. For template theater, confirm the alternatives are genuinely contrasted with the chosen design. For consensus laundering, read the early revisions to recover the original contribution.

Recency drift. A strong document from six years ago may predate the level the candidate now claims. The same behavior reads differently by level: the rubric is read against the target level, and a single behavior can be a pass at mid-level and a flag at staff. Check dates against the target level, always.

Skill-tag inflation and talk-is-not-design. Because only about 11.7% of the senior IC pool even lists Distributed Systems, a self-listed skill is not evidence of judgment - score the artifact, not the profile. And conference speaker credit is self-asserted; abstract reviewers can usually tell when someone is bluffing, but a hiring reader cannot from the abstract alone. Watch the recorded talk and follow any linked artifact into a repo.

Aggregating into a verdict and setting your own thresholds

Roll the per-document scores into one of three verdicts - staff-level judgment, mid-level execution, or borrowed credit - read against the candidate's target level, and abstain when the evidence is thin. No public source sets a minimum count or recency window, so those thresholds are your standard to set explicitly and defend, not a sourced fact.

Use the trade-off and operational dimensions as the deciding axes, because they carry the most rubric weight and are the hardest to fake:

Here is the rubric skeleton I hand to the person doing the read. Fill one per document, then aggregate.

Candidate:
Artifact URL + date:
Authorship (driver / co-author / commenter):
Problem framing (0-3): ____ evidence: "..."
Alternatives considered (0-3): ____ evidence: "..."
Explicit trade-offs (0-5): ____ evidence: "..."
Operational maturity (0-4): ____ evidence: "..."
Cross-team influence (0-3): ____ evidence: "..."
Thread defense under pushback (yes / partial / went quiet):
Verdict: staff / mid / borrowed credit / insufficient

Weight trade-offs and operational maturity most heavily, mirroring the published rubrics. A document that scores high on framing and alternatives but flat on trade-offs and operations is mid-level execution, not staff. A document where the author went quiet the moment their trade-offs were challenged is a warning regardless of how polished the text is - the discussion thread carries more staff signal than the doc.

My own thresholds, stated as this guide's standard: require at least three driver-authored substantive artifacts, with at least one dated inside the window where the candidate claims their target level, and require a defended trade-off in at least one of them before you call the verdict staff-level. Below that floor, return insufficient evidence rather than force a score. State your floor in the write-up so a reviewer can argue with it.

Keeping the read current

Re-run the read when the candidate's target level changes or when new artifacts land, because the same design writing scores differently against different levels. A document that is a clean pass for a senior role can be a flag for a principal one, so the level you are hiring for is an input, not a constant.

The supply picture is the reason to keep the pipeline warm rather than react. In Refolk's index there are 32,368 Staff Software Engineers and 31,108 Principal Software Engineers in the US, roughly 0.96 Principal per Staff, against 2,135 Staff engineers in the UK - the US pool is about fifteen times larger.

Those counts move as titles change, so re-pull them when you build a slate rather than trusting a cached number. The mechanism to re-check is a fresh title-count query, not a memorized figure.

Two habits keep the standard honest over time. First, when a venue changes its process, update your attribution check - the Rust final comment period runs ten calendar days and is open at least five business days, and if that window or the metadata schema changes, your authorship read has to follow. Second, revisit your quantity and recency thresholds after a few hires: if the engineers who passed your bar are performing, the floor is right; if you are over-abstaining on strong people, loosen it and say why. The framework is only as good as the thresholds you are willing to defend out loud.