The Repo-Corpus Readiness Standard: Ship, Patch, or Rebuild

You have pulled a set of repositories and their contributors to back a claim about a technology's adoption or its talent pool. Before that number goes into a briefing, someone has to decide whether the corpus is clean and representative enough to ship. This guide is the gate: a single pass/fail rubric that fuses sampling adequacy, bot filtering, identity deduplication, and mandatory bias disclosure, written so two analysts scoring the same corpus reach the same verdict.

It sits downstream of the sizing playbook, which covers how to build a repo talent-pool estimate, and downstream of the provenance reference, which covers reading individual signals. This is the definition of done. Adopt it as team policy and no repo-backed claim leaves the building without a recorded verdict of Ship, Patch, or Rebuild.

What "ready" means: the three verdicts

A corpus is ready to ship when its sampling frame is unbiased, its contributor list is deduplicated and bot-filtered, and its coverage and skew are measured and disclosed. Anything short of that is either a Patch, if the defect is a post-hoc filter you can re-run, or a Rebuild, if the frame itself is skewed.

The core distinction is where the defect lives. A biased frame poisons everything drawn from it, so no amount of downstream cleaning rescues it. A missed fork or an untagged bot is a filter you forgot to apply, and you can apply it now without collecting anything new.

The most common way analysts get this wrong is treating a biased frame as a patchable problem. If you built the frame by taking the top-starred repositories, you did not collect a small clean sample; you collected a sample that is structurally unlike the population. That is a Rebuild, and the rest of this guide explains why.

Why a star-gated frame fails "is my github dataset representative"

Selecting repositories by star count keeps the famous projects and silently discards most qualifying ones, so a star-gated corpus is unrepresentative by construction rather than by size. This is the single most important thing to internalise before you grade any corpus.

The evidence is direct. A stargazers-based classifier using a threshold of 10 stars posts high precision at 97% but an inversely proportional recall of just 32%. In plain terms: almost everything it keeps is a real engineered project, but it throws away roughly two-thirds of the real engineered projects that exist. Your talent pool or adoption count is then drawn from a third of the population that happens to be popular. That is popularity bias baked into the frame.

By contrast, the reaper method, which classifies repositories across seven dimensions rather than one popularity proxy, reaches 82% precision and 83% recall. The recall gap between 32% and 83% is the whole argument. A method that keeps most of the population is a defensible frame; a method that keeps a popular sliver is not.

The reaper dimensions are worth knowing because they give you a defensible inclusion rule that does not lean on stars: Community, Continuous integration, Documentation, History, Issues, License, and Unit testing. You do not have to run reaper itself, which was measured across 1,857,423 repositories over more than 200 nodes and more than a month. You can approximate it with a non-fork, recent-activity, commit-count rule. What you may not do is substitute star count and call the result representative.

Read that table as the difference between precision and recall. A star gate is precise: what it keeps is real. But recall is what governs representativeness, and on recall the star gate collapses. Recall for BoDeGHa is not published in the cited source, so I do not assert one; that is a limit of the evidence, not a hidden strength.

The four gates every corpus must clear

A corpus ships only if it clears four gates in order: an unbiased frame, fork exclusion and inclusion filtering, identity deduplication, and bot filtering with manual top-N review. Each gate proves something specific, and each has a characteristic way of lying.

Gate one, the frame. Proves the corpus can generalise. It lies when it looks complete because the famous repos are all present, while most of the population is missing. Check by recomputing coverage against an unfiltered frame.

Gate two, inclusion. Proves each repo is a real project, not a fork or a dead personal dump. It lies through fork inflation: of 6.8M public repositories, 3.0M, or 44%, are forks, and the majority of projects are personal and inactive. A "big adoption" number is often just forks and dead repos counted as independent projects.

Gate three, dedup. Proves your headcount is people, not accounts. It lies through identity inflation, where one person with three email addresses becomes three contributors. Check by comparing the individual count after merging to the raw identity count.

Gate four, bots. Proves your contributors are humans. It lies in both directions, which the next section covers in full.

Recall governs representativeness, and on recall a star-gated frame collapses from 83 percent to 32.

Deduplicating github contributor identities and filtering bots as one pass

Merge identities first, then flag bots, because the same record carries both. This ordering is not arbitrary. In CHAOSS GrimoireLab's SortingHat, the canonical Individual record stores a bot flag alongside the canonical name, email, country, and gender. If you merge first, a bot spread across three identities becomes one flagged individual; if you flag first, you risk tagging or missing three fragments of the same automation.

Mechanically, SortingHat consolidates raw identities from Git, GitHub, Jira, and Slack into a single canonical Individual, deriving that individual's primary key from the SHA-1 hash of its first attached identity and re-parenting all merged identity records to it. That gives you the stable per-person key you need before any count is trustworthy.

For bot filtering, the name-keyword rule is the simplest documented method and it fails in both directions. Given dependabot[bot], felix, John Smith, and Abbot, only dependabot[bot] should come out as a bot. But a substring match on "bot" flags Abbot as a false positive, and a custom bot named like a person escapes entirely. Renovate publishes as app/renovate and Dependabot pull requests come from dependabot[bot], so known automation is catchable by identifier, but unknown automation is not.

Model-based detection does better but is imperfect. A model trained on pull request and issue comments reached 0.77 precision on the reference dataset, and retraining on git commit messages lifted precision to 0.80. Combining the name rule with the model is what pushes accuracy up. Neither alone is enough.

Here is the trap in one line: your single most active "human" contributor is the account most likely to be undetected automation, because that is precisely where the detectors fail. Skip the manual top-N review and your headline number rests on the weakest part of the pipeline.

When you need to validate a repo-derived count against something outside your own pipeline, Refolk lets you ask for the same population in plain English and compare the two figures. If your corpus says 400 Go backend contributors and the index returns a wildly different order of magnitude, that gap is a signal to re-open one of the four gates before the briefing goes out.

The readiness procedure

Run the corpus through these seven steps in order and record the output of each. The verdict at the end is only reproducible if every step left an artefact another analyst can inspect.

Two notes on ordering. First, sources disagree on whether to filter bots before or after dedup; the SortingHat model merges identities first and flags bots in the same profile pass, and that is the ordering I recommend above for the reason given earlier. Second, coverage is computed near the end because you can only measure how much of the population your cleaned corpus resembles once the corpus is actually clean.

How coverage and language choice distort a claim

Report a coverage percentage, not a repo count, because the count tells the reader nothing about representativeness. Sample coverage is the percentage of the population that is similar to your sample, and it is the honest substitute for a classical margin of error, which is not established for repository sampling.

The scale of the gap is easy to underestimate. Major software-engineering conferences were found to cover just 9.15% of the Ohloh population. If curated academic venues cover under a tenth of the population, a corpus built from a handful of flagship repos covers far less, and a single-repo claim covers almost none. That is why single-repo generalisation is a failure mode: the coverage is tiny even when the repo is important.

The second distortion is the language you chose. A single-language corpus can swing a market size several-fold depending only on which language you picked, before any bias in the frame. Refolk's index makes the gap concrete.

Geography compounds it. The same index shows the Rust pool concentrates heavily by country, which any bias table has to disclose.

Read these two tables together. If your corpus is US Rust repositories, your claim rests on a pool of 3,387 in Refolk's index, roughly a fifth of the US Go pool and 3.5 times the German Rust pool. None of that is wrong to report, but all of it must be disclosed, because the reader of your briefing will otherwise assume the number is a market when it is a slice.

How this goes wrong: the failure modes

The failure modes below are why the standard exists. Each one has a verdict attached, because the point of a gate is to sort recoverable defects from disqualifying ones. Learn to name them on sight.

Star-gated frame (Rebuild). Looks representative because the top repos are famous; really it drops most of the population, with recall of 32% at threshold 10. Check by recomputing coverage against an unfiltered frame. If the frame was star-gated, stop and re-sample.

Fork double-counting (Patch). A big adoption number is inflated because 44% of repos are forks. Check that the fork flag is excluded and look for identical commit histories. Recoverable by re-filtering.

Substring bot filter (Patch). Names like Abbot or Talbot dropped as bots, or a custom bot with a human name kept. Check the flagged list against the documented dependabot and Abbot test case, and add a model to the name rule. Recoverable.

Top-contributor bots survive (Patch, but load-bearing). The most active "human" is actually automation, because detectors fail on the 20 most active contributors per project. Check by manually inspecting the top 20 per repo. Recoverable only if you actually do the manual review.

Identity inflation (Patch). One person counted as three via multiple emails, overstating the pool. Check by running the identity merge and comparing the individual count to the raw identity count. Recoverable.

Inactive-repo padding (Patch). Counts include dead personal repos, because most projects are personal and inactive. Check by applying recent-activity and commit-count thresholds. Recoverable.

Single-repo generalisation (Rebuild). A claim rests on one flagship repo, so coverage is tiny. Check by reporting the coverage percentage, not the repo count. If coverage is negligible, the frame is too narrow to fix by filtering.

The pattern is consistent. Frame-level problems, star gating and single-repo generalisation, are rebuilds because you cannot filter your way out of a skewed selection. Everything else is a patch, because it is a filter you can re-run on data you already have.

The ship checklist and keeping the standard current

Before you attach a verdict, run this checklist. Every item must be a recorded artefact, not a memory, because reproducibility between two analysts is the entire point of a standard.

This corpus was sampled from [population] using a [probabilistic / stratified] frame,
not gated on stars. It covers [X]% of the defined universe. Fork repositories were
excluded ([N] removed). Contributor identities were merged, reducing [raw identity
count] identities to [individual count] people. Bots were filtered using a name rule
plus a model, with manual review of the top 20 contributors per repo. Known skew:
the corpus over-represents [language / geography / popularity band], and the pool
size is sensitive to language choice ([note the multiple across languages]).

To keep this standard current, re-check the mechanisms rather than the values. Bot identifiers change: Dependabot publishes as dependabot[bot] and Renovate as app/renovate today, but new automation appears constantly, so re-audit your name rule against fresh known-bot accounts each cycle. Model precision improves over time; the 0.80 commit-message figure is a floor to beat, not a ceiling to trust. And coverage is only meaningful against the universe you defined, so revisit the universe whenever the population you are claiming about shifts. The verdicts stay fixed. The evidence you feed them does not.