The Commit-Mining Pipeline: Repos to a Handoff-Ready Contact Table

This is the standing procedure for turning a set of target GitHub repositories into a deduplicated, minimized contact table a recruiter or seller can work without surprises. It is for recruiting and revenue operations, and for anyone answerable for how the data was gathered. Public scrapers stop at dumping raw commit emails, leaving you with noreply strings, bot authors, fork noise, duplicate identities, and no minimization record. This guide takes the extraction clean end to end, with the stage-by-stage filters, resolution keys, compliance gate, and quality-control counts that make the output safe to hand off and re-run on a schedule.

What this pipeline produces, and what it refuses to produce

The deliverable is one canonical row per person, carrying only the fields outreach needs, backed by a stage ledger and a dated lawful-basis reference. Everything else that a raw scrape would hand you - noreply placeholders, [bot] authors, fork-inherited duplicates, and the same person split across three commit emails - is removed or collapsed on the way, with reason codes retained.

The reason to run it as a pipeline rather than a one-off query is repeatability. A clean table you cannot reproduce next quarter is a liability, because you cannot show how it was built when a data subject or a regulator asks. Every stage here records what it dropped and why, so the output is both a contact list and its own audit trail.

A note on scope. This guide covers extraction, resolution, minimization, and verification. It does not cover sequencing or deliverability recovery once you start sending, and it does not tell you whether a given person is worth contacting. It gets you a table you can trust.

Where the email actually lives on GitHub

The reliable source of a contributor's email is the commit object, not the profile. A call to the commits endpoint, for example curl https://api.github.com/repos/torvalds/linux/commits, returns .commit.author.email, .commit.author.name, and a resolved login where GitHub can match it. That is the field the pipeline reads in bulk.

The profile endpoint is a trap. GET /users/:login exposes an email field, but for the vast majority of users that field is null. If you build your extraction around the profile, you will pull almost nothing. The commits endpoint is where the signal is, which is also why identity resolution and the lawful-basis question land on you rather than on a tidy populated field.

Rate limits and pagination govern how long a pull takes, and pagination is the real constraint. The table below is the budget you plan against.

Pagination caps at 100 items per page, so a single large repository can swallow your whole hourly budget before a second repo is touched. Retrieving the commit history of torvalds/linux needs more than 8,000 requests. The practical consequence is that repo selection governs runtime more than token count does. Pick deep monorepos deliberately, and page them off-peak.

Reading and resolving the GitHub noreply commit email

A GitHub noreply address is a privacy placeholder GitHub substitutes for a user's real email, and its numeric prefix is the single most useful key in this whole pipeline. The format is documented and date-dependent. Accounts created after 18 July 2017 get ID+USERNAME@users.noreply.github.com, where the number is the account's source ID. Accounts created before that date, if they enabled privacy, get USERNAME@users.noreply.github.com with no ID.

That numeric prefix resolves two ways. You can extract the user ID and make an API call with it, or extract the login username and use the user API, cross-checking against the source ID. The ID form is rename-proof: because commits made with the private noreply email are linked to the GitHub account rather than to an email or username, the ID reconciles a person across display-name and email changes that would otherwise look like different contributors.

The noreply numeric ID is the only rename-proof key, because GitHub links commits by account, not by string.

The legacy username-only form is the quiet failure. It has no numeric ID, so an ID-based resolver silently skips it. Branch on the presence of a plus sign before you resolve: if the local part contains a +, parse the ID; if it does not, you are in legacy territory and must resolve by login alone and flag the row. Skipping that branch is how a real contributor disappears from the output without anyone noticing.

This step does not discard noreply rows. It enriches them. A noreply address with a resolved ID is often a better identity anchor than a raw personal email, even though you cannot mail it directly. You will still need a deliverable address from elsewhere in the commit history, but the ID keeps all of that person's aliases attached to one row.

The bot, CI, and fork filter

Bots and fork noise inflate your counts and, worse, plant non-humans in a contact list. The literal [bot] suffix in a login is the primary marker, paired with a noreply address, for example github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>. Drop any login ending in [bot], plus known CI and service addresses, before you count anything.

Co-authors are a separate and sharper trap. Co-authored-by is a GitHub convention layered on git's trailer mechanism, written as Co-authored-by: name <name@example.com>. The false positive to watch for: when a human squash-merges a Dependabot pull request, GitHub appends Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>. If you read co-author trailers naively, that bot reads as a human contributor on an otherwise legitimate commit.

Two defenses matter. Match the [bot] suffix and the service sign-off, not just the author field, so a bot riding a human's squash-merge is caught. And anchor trailer matches to the start of the line, so a Co-authored-by string quoted inside a commit body does not count as a real trailer. Fork noise is simpler: fork-inherited commits duplicate upstream authors, so de-duplicate by commit SHA across the whole repo set before you count yield.

Quarantine, do not delete. Every row the filter removes should go to a holding set with a reason code - bot_suffix, ci_address, coauthor_bot, fork_sha_dup. You will want those counts for the stage ledger, and you will want to re-inspect them the first time a filter behaves unexpectedly on a new repo.

Collapsing GitHub contributors into one person each

Identity resolution is where you decide that three rows are one human, and it has a strict precedence order. GitHub's own Contributors page groups by account and reports one entry per person, while local git output splits the same person because git groups by the name-plus-email string. Your pipeline must behave like the Contributors page, not like local git.

The precedence, strongest first:

1. Noreply numeric ID. The account-level key. Three different name and email pairs can be one account, and the numeric prefix is that account's id. Group on this before anything else.
2. Resolved login. Where you recovered a login but no stable ID, group on the login.
3. Normalized exact email. Same email, case-insensitive, is one person.
4. Local-part plus name heuristic. Same email local-part plus the same name across different domains, as a last resort.

That last rule is dangerous and needs a floor. Mailmap tooling skips email local-parts shorter than 8 characters because short local-parts collide across unrelated people and produce false merges. Enforce that 8-character floor, and never merge on name alone - two different engineers who share a first name will otherwise fold into one row.

The payoff is real, not cosmetic. In one worked mailmap example, a repository collapsed from 31 unique author identities to 23, a reduction of about 26%. An un-deduped pull over-counts the addressable pool by roughly a third before you send a single message, which throws off every downstream yield figure and wastes verification budget on duplicates.

Build a mailmap artifact as you go: one canonical row per person with the alias list attached. That artifact is reusable. Next time you re-run the pipeline on the same repo set, it collapses the known aliases immediately and you only resolve new contributors.

When your target is an identity rather than a repo - a specific kind of maintainer you want to reach across repos - this resolution work is exactly what a plain-English search layer removes. Refolk runs the commit graph and the resolution underneath, so you can ask for people the way you would describe them to a colleague instead of assembling and de-duplicating the pull yourself.

The minimization and lawful-basis gate

Minimization is a technical filter and lawful basis is a documentation artifact, and you need both before the table hits disk. For scraped personal emails the operative basis is legitimate interest, recorded in a Legitimate Interest Assessment. A current-quality LIA has six parts: a stated purpose, identification of the legitimate interest, a necessity test, a balancing test against data subject rights, a description of safeguards, and a conclusion. Each section must be specific to this scrape - regulator inquiries routinely call out boilerplate LIAs as evidence of bad faith.

Large-scale scraping also triggers a Data Protection Impact Assessment. The EDPB adopted Guidelines 03/2026 on web scraping in July 2026, its first guidance dedicated to the practice, which signals that scraping is now assessed on its own terms rather than by analogy. Treat GDPR compliance here as largely a documentation discipline: the defensible pipeline attaches a dated, scrape-specific LIA to the stored table rather than relying on a filter setting alone.

Minimization must happen at extraction, not after. Filtering at analysis time - storing full profiles and trimming them later - still breaches minimization, because the excess data existed on disk. Extract only the fields outreach needs and discard the rest before anything is written.

The geographic skew is why this gate matters even for a small list. In Refolk's index of professional profiles, Software Engineers with an Open Source skill number 574 across the US and Germany combined, split 513 in the US and 61 in Germany. The German slice is barely a tenth of the pool, but it is exactly the slice that most sharpens the LIA and DPIA obligation. Minimization effort is disproportionate to volume, so do not let the raw count fool you into treating documentation as optional.

The US pool runs 8.4 times the German one (513 against 61), yet the smaller slice carries the heavier compliance weight. Plan the LIA scope around who is in the list, not how many.

Verification before handoff

Verification is the stage where scraped lists quietly fail, so it is non-negotiable before handoff. Run every remaining address through a verifier and suppress invalids. The target is a total bounce rate under 2% in a modern sending program, with the best-maintained lists under 1%. The legacy "under 5%" standard is obsolete.

Recruitment and B2B sending already carry a baseline of decay: hard bounce rates commonly run 0.6 to 1.5% from candidate-database aging alone. Scraped commit emails that skip verification push toward the unverified range, where cold lists routinely bounce 10 to 30%, crossing the throttling line on the first send. The bands below tell you what each verified result means.

One verifier behavior defeats the whole stage: catch-all, or accept-all, domains. A verifier marks these as valid because the mail server accepts everything, but the specific mailbox may not exist. Treat catch-all results as unverified, not deliverable, or bounce risk survives into the handoff and shows up as throttling weeks later.

Run it in order: the eight-stage procedure

Here is the pipeline start to finish, with owner and rough duration on each stage. Run it in this order; each stage depends on the clean output of the one before it.

How this goes wrong: failure modes and false positives

Most of this pipeline's value is in catching the ways it silently misreports. Each failure mode below has a specific check that defeats it. The costly ones are not the loud errors but the quiet miscounts - a person dropped, a bot shipped, two people merged into one.

Two more deserve their own paragraph because they do not fit a one-line check.

Minimization after the fact. Storing full profiles and trimming them later still breaches minimization, because the surplus data was on disk. The only fix is to filter fields at extraction, before the write. If your architecture pulls everything and prunes in a later job, the compliance record is already broken no matter how clean the final table looks.

Verification theater. A verifier that marks accept-all domains as valid gives you a green result with live bounce risk underneath. Treat catch-all as unverified rather than deliverable, and the theater stops. A list that passed verification but still bounces at 4% usually failed on exactly this point.

Keep it current: the QC ledger and re-run schedule

The job is not done when the table is handed off; it is done when the handoff carries its own evidence and a schedule to refresh it. Record yield and reject counts at every stage and attach them to the deliverable, so the recruiter or seller receives a table plus a stage ledger they can read.

Run this checklist before you call the pipeline complete.

Keep the artifacts. The mailmap collapses known aliases on the next run, so a scheduled re-run only resolves new contributors rather than rebuilding from scratch. The dated LIA is your compliance record; refresh it when the scrape's purpose or scope changes, and re-check the EDPB web-scraping guidance periodically, since that is the authority a regulator will apply. Re-run the verification stage each cycle regardless, because address decay in recruitment and B2B lists is continuous and a table that bounced under 2% last quarter will drift above it if left to age.

The one time-sensitive input here is the bounce benchmark. The current line is under 2% total, with the obsolete "under 5%" standard retired. If you inherit a pipeline built to the old standard, re-tune verification to the tighter band before you send, because the gap between 2% and 5% is the gap between a deliverable list and a throttled sender.