The Collaboration Score From Public GitHub Review History

Before you schedule an interview, you want to know whether a senior or staff engineering candidate actually works well with other humans: whether they take feedback, review others' code without wounding them, and drive a technical decision instead of just rubber-stamping it. This guide is for engineering managers, technical founders, developer-relations leads, and technical sourcers who need that read from a public footprint alone. It turns a candidate's public GitHub review comments, issue threads, and discussions into a defensible collaboration score across five named dimensions, with the evidence behind each score and the signals that would overturn it.

Most GitHub evaluation stops at green squares and star counts. Those measure output, not conduct. Collaboration is the signal hiring managers call decisive and least-examined, and it hides in the one place candidates rarely curate: how they talk to other engineers in review.

Why review history beats the commit graph

Review behavior is the least-gamed collaboration signal in a public footprint, and it predicts organizational influence better than commits do. One influence study reports that code review activity predicts network centrality far more powerfully than commit activity, with an R^2 of 0.74. The authors frame it as a transition from individual production toward collaborative coordination, which is exactly the behavior you are trying to price at the staff level.

There is a second reason to prefer it. Candidates optimize commit graphs for recruiters. Almost nobody curates their review comment history, so it stays closer to genuine behavior. When you read how someone phrases a change request or answers a first-time contributor, you are watching them work, not watching a portfolio.

The trap is reading the wrong quantity. Approval count feels like engagement but carries almost no information, because a large share of review comments are noise to begin with. Content, not count, is the only reliable signal. Everything below is built to read content and to resist the structural illusions that count creates.

What GitHub exposes and how to pull it

GitHub exposes several distinct collaboration artifacts, and all of them are readable for public data without authentication. A pull request review is a group of comments carrying a state - APPROVED, CHANGES_REQUESTED, or COMMENTED - plus an optional body. Inline review comments are a separate object: comments made on a portion of the diff during a review, distinct from commit comments and plain issue comments. Review requests are retrievable too, with one wrinkle: once a requested reviewer submits a review, they stop being a requested reviewer and their work returns from the List reviews operation instead.

The cleanest per-user path is GraphQL's ContributionsCollection. Its PullRequestReviewContributions field returns review activity on repositories where the user has push access. Two caveats matter for scoring, and both create false negatives you must plan around. There is no way to tell if the user previously had push access, so repositories where they no longer hold maintainer rights will not show. And the by-repository variant only returns public repositories or your own reviews, even with an authenticated org token. Note also that REST and GraphQL are not interchangeable; they overlap like a Venn diagram rather than mirroring each other, so expect to use both.

Rate limits set your extraction budget. Pull the whole footprint once, cache it, and score offline.

There is also a secondary limit of no more than 900 read requests per minute to a single REST endpoint and no more than 100 concurrent requests. For a single candidate this never bites. For a batch, authenticate and throttle.

The five dimensions and what each score means

Score collaboration across five dimensions, each grounded in an observable artifact rather than a vibe. The anchor for constructive versus destructive behavior is well established: destructive criticism is feedback that is both inconsiderate and nonspecific, while constructive criticism stays considerate and gives specific improvement suggestions. Google's engineering practices, the de facto standard with over 21,000 stars, add the operational rule: make comments about the code and never about the developer, and strike a balance between pointing out problems and giving direct guidance, because letting the developer decide helps them learn.

Score each dimension on a simple three-point scale: 2 for clear positive evidence, 1 for present-but-thin, 0 for absent or negative. Absent is not the same as negative. A candidate with no mentorship evidence scores 0 on that dimension only because it is unmeasured, and you say so in the confidence note rather than penalizing them for it.

Reading each dimension

Feedback handling shows in the candidate's own PRs: a reviewer asks for a change, and you can see whether the candidate argues productively, revises, and thanks the reviewer, or goes silent and merges anyway. Review quality turns on specificity. A comment that names the failing case and suggests a fix scores; a wall of "Nit:" style comments does not, because nit-only review means the person skipped design review. The "Nit:" prefix is itself a documented signal that a comment is non-blocking. Decision leadership lives in issue threads and Discussions where the candidate frames options, cites tradeoffs, and lands a call, not just in approvals.

Constructive versus destructive: the signals that decide it

The single hardest judgement is separating rigor from hostility, and it turns on two observable things: whether the comment targets the code or the person, and whether a change request carries cited rationale or bare rebuke. Constructive criticism stays considerate while being specific; destructive criticism is inconsiderate and nonspecific. That distinction is scorable line by line.

Use these tells when you read a comment:

• Targets the code, not the developer. "This loop reallocates on every iteration" scores. "You clearly did not test this" does not.
• Specific and actionable. Names the case, suggests a direction. Vague disapproval is the destructive-criticism signature.
• Balances problems with guidance. Points out the issue and offers a path, which is what helps the author learn.
• Labels non-blocking feedback. "Nit:", "Optional:", "Consider:", and "FYI:" tell the author what is mandatory versus preference.
• Change requests cite reasons. CHANGES_REQUESTED with rationale is rigor; CHANGES_REQUESTED as a bare pronouncement is gatekeeping.

Positive comments are not just nicer; they correlate with faster resolution times and stronger team relationships, which is the outcome you are hiring for. Practitioner surveys agree that constructive, polite suggestions are always preferred over harsh or toxic ones.

Rigor cites its reasons. Gatekeeping pronounces judgment. The difference is whether the comment teaches or just holds the line.

Now weight what you read. Because comment quality varies so widely, the raw distribution of comments in the wild sets your prior for how much a small sample can carry.

Read those numbers as a warning about sample size. If a third to nearly half of all comments are non-useful, a three-comment footprint tells you nothing. It also means one toxic-sounding line in a large body of civil work is closer to the base rate than to a pattern. Judge conflict conduct over a thread, never over a single line, because people cannot reliably detect sarcasm or gauge directness in written language.

Finding people who visibly lead technical discussion is the friction this framework hits first, because most footprints are review-and-approve with no leadership signal. Refolk lets you ask for that behavior in plain English across the public GitHub graph rather than paging through profiles hoping to spot it.

The scoring procedure

Work in a fixed order, and read comment content before you count anything. There is a genuine disagreement in practice here: some practitioners treat approval-without-comment as the first metric to compute, while the academic literature implies you must read comment content before any count is meaningful. Sequence content-reading first. Counting rubber-stamps is cheap and can happen in parallel, but the score comes from what the comments say.

For the noise filter in step four, use structure before semantics. A bare approval is an APPROVED state with an empty or near-empty body. Bot filtering is a known pattern; auto-generated PRs and their reviews add nothing and should be dropped before a human reads anything.

How this goes wrong: failure modes and false positives

This is the section to read twice, because the collaboration read is easier to get confidently wrong than right. Each failure mode below has a false-positive shape and a check that catches it.

• Rubber-stamp approvals. A high approval count reads as engagement but is empty. A developer with many approvals and no comment bodies is rubber-stamping. The Node.js glossary even names it RSLGTM, approving without doing a full review. Check structurally: compute the ratio of substantive-body approvals to bare approvals before reading a word.
• Authority-inflated LGTM. A short approval from a code owner is a real review; the same text from a drive-by reviewer is a formality. Scoring the comment without resolving who owns the touched files systematically over-credits drive-by reviewers. Check file ownership before weighting any short approval.
• Lapsed-maintainer invisibility. A strong reviewer reads as inactive because past maintainer repos do not return via ContributionsCollection. Check org membership and archived work; do not score absence as a negative.
• Tone misread across text and cultures. Terse or direct phrasing can scan as hostile when it is not, and readers cannot reliably detect sarcasm in writing. Check the whole thread and the author's later replies, not one line.
• Nit inflation. A reviewer who leaves only style nits looks engaged but skipped design review. Check the proportion of "Nit:"-prefixed comments against substantive design and architecture comments.
• Thin sample. With 34.5% to 44.47% of comments non-useful, a three-comment sample is statistically empty. Require a documented minimum volume window and treat anything below it as anecdote, flagged in the confidence note.
• Gatekeeper mistaken for rigor. A gatekeeper holds the line with a bar that is too high and pronounces judgment rather than sharing knowledge. Check whether CHANGES_REQUESTED comes with cited rationale or a bare rebuke.
• Selection bias toward maintainers. Only maintainers accumulate large public review counts. Strong individual contributors who review privately score as zero. Collaboration can be genuinely invisible on public data; when it is, say so and route to a reference instead.

The two failure modes that flip a score's sign are authority context and thin sample. Both are cheap to check and both are routinely skipped. Resolve file ownership and confirm volume before you trust any dimension score.

What the talent math tells you about where to spend this

Run the full read on scarce, high-stakes candidates, and screen the rest more cheaply. The framework is thorough by design and costs roughly two hours per candidate across extraction and scoring, so reserve it for seniority where each person is rare enough to justify the time.

The scarcity is real at the staff level. In Refolk's index of professional profiles, staff engineers are only about 15.2% of the US senior-plus-staff pool, and the geographic distribution is steep.

The US has roughly 28.9 times as many staff software engineers as Germany. When a pool is that thin, a per-person review-history read is proportionate: you are not screening a funnel of thousands, you are pricing a handful of people you cannot easily replace. For a broad senior funnel, the same math argues the other way. Use the structural rubber-stamp check and a quick review-quality skim as a filter, and reserve the full five-dimension read for finalists.

Ship the read and keep it honest

Before you call the score final, verify it against the checklist below. The goal is a one-page dossier a hiring manager can read in two minutes and defend in a debrief: scores, quoted evidence, and the specific signal that would flip each dimension.

Candidate handle:
Footprint: reviews pulled, issue threads, discussions (with date range)
Sample size and volume-window status:

Feedback handling  [0-2] : evidence link | overturns if:
Review quality     [0-2] : evidence link | overturns if:
Conflict conduct   [0-2] : evidence link | overturns if:
Mentorship         [0-2] : evidence link | overturns if:
Decision leadership[0-2] : evidence link | overturns if:

Rubber-stamp ratio (bare / total approvals):
Authority context checked? (file ownership resolved):
Invisible-reviewer risk (lapsed maintainer repos):
Net read (one sentence):

Keep the read current the same way you built it. The API states, rate limits, and contribution fields drift, so re-check the retrieval mechanics against the official docs rather than trusting a cached script. The published base rates for non-useful comments are stable enough to keep as your sample-size prior, but there is no cited minimum review count for a defensible score, so treat your volume threshold as a house convention and revisit it as you see how it performs. When a candidate's public footprint is genuinely too thin to score, the honest output is "unmeasured on public data," which routes you to a back-channel reference or a work sample instead of manufacturing a number you cannot defend.