The Commit-Log Cleanup Pipeline: Raw Author Emails to a Deduplicated Contact File

This is the mechanical build-and-dedup stage that sits before lawful clearance and CRM grading. It is for recruiting and revenue operators answerable for how a commit-mined contact list was gathered, and it delivers a repeatable pipeline that turns a raw dump of git author and committer emails into a deduplicated, provenance-stamped file where one person maps to exactly one record. Everything here is procedure: extract, normalise, strip machine rows, collapse identities, flag the low-value ones, and hand off a file someone downstream can audit.

The public guidance stops short. The top results tell you to run git log and grep for the @ sign, then leave you with a raw list full of bots, noreply masks, and the same person under five addresses. This guide picks up where that stops.

What a clean commit-mined contact file actually is

A finished file has one row per human contributor, each keyed to a durable identity, with every machine and low-value address either quarantined or flagged rather than silently dropped, and source provenance on every row. That is the whole target, and every step below exists to move a raw extract toward it.

The raw material is deceptively messy. A single contributor commits under a work laptop address, a personal address, a noreply mask, and a typo'd variant, sometimes all within one repository. Meanwhile bots and CI accounts push commits that look human until you inspect the author name. In a study of 275 popular GitHub projects, nearly half of contributors committed once and accounted for under 2% of commits, so a raw extract is dominated by drive-by and automated rows. The cleanup is not a polish pass. It is most of the work.

Treat the output as an input to a later stage, not a finished asset. This pipeline stops at a deduplicated, flagged, provenance-stamped file. Whether any given row may lawfully be contacted is a separate question handled by the clearance stage that reads this file. Your job here is to make that trail auditable.

The address classes you are routing

Every address in the extract belongs to one of a small number of classes, and each class has a fixed action: keep and use, keep as a key only, quarantine, or flag. Getting the routing right is the spine of the pipeline.

The two GitHub noreply formats are the trickiest because they split on a date. GitHub documents it precisely: accounts created after July 18, 2017 get an ID-based noreply address in the form ID+USERNAME@users.noreply.github.com, while accounts created before that date that enabled privacy earlier get the legacy USERNAME@users.noreply.github.com. A regex that matches only the ID form will let every legacy address through as if it were a contactable human address.

The patterns come straight from GitHub's documentation and the Dependabot and Actions issue trackers; the actions are this guide's mapping. Note that an ID noreply address is never contactable but is still valuable: the number in front of the plus sign is the most durable key you have.

Why the user ID is the only rename-proof key

The numeric GitHub user ID inside an ID-based noreply address is the one identifier that survives a username change, which makes it the correct primary key for collapsing many emails into one person. Everything else in a git commit is weaker.

Git's name and email fields are free-text strings. As the documentation puts it plainly, the settings for your name and email are just strings and you can set whatever you want. That means Jane and jane are two distinct authors until you normalise, and a name like "Jane Doe" is not a reliable join key at all. Usernames are better but still mutable: a legacy username-based noreply address breaks attribution when the contributor renames, because those commits stop mapping to the account. ID-based noreply commits are linked directly to the GitHub account, not to an email or username, so they continue to map through the user ID part of the address no matter how often the handle changes.

The practical rule: when an ID-based noreply address is present, key the person on that number. When it is not, fall back to name-and-email co-occurrence, and treat the resulting cluster as provisional rather than certain.

Bots and role addresses: what each filter proves

The bot filter removes automated rows that would otherwise pollute identity clusters, and the role filter removes human-looking addresses that deliver but never reply. They solve different problems, and confusing them is a common error.

Bots follow documented, repeatable patterns. Dependabot commits as dependabot[bot] with the author email 49699333+dependabot[bot]@users.noreply.github.com, and historically signed off as support@github.com or, earlier, support@dependabot.com. GitHub Actions configures itself as github-actions[bot] with the email 41898282+github-actions[bot]@users.noreply.github.com. The general rule is to drop any author whose name contains [bot], any NNN+*[bot]@users.noreply.github.com address, and support@github.com.

Role addresses are a different animal. They are real, deliverable inboxes like info@, sales@, support@, admin@, and hr@, plus noreply@, contact@, and postmaster@. A verifier will often pass a role address on a healthy domain through syntax, MX, and mailbox checks and still return a role verdict, because deliverability and value are separate questions. The signal to watch is not bounce but reply. In one audited case, removing 1,400 role-based rows from a list of 12,000 lifted reply rate from 0.9% to 4.1%, a 4.5x gain, even though every one of those rows delivered successfully.

There is a domain-reputation argument on top of the reply-rate one. Role addresses averaged a 2.33% bounce rate across industries against an ideal under 1%, and if role rows make up a tenth of a list while generating complaints at four times the normal rate, they contribute 40% of complaint volume from that tenth. Google's spam-complaint threshold is 0.3%, above which you lose deliverability mitigation. So the role filter protects the whole sending domain, not just the flagged rows.

Role filtering is a reply-rate lever, not a bounce lever: the addresses deliver fine and still cost you the reply.

The pipeline, step by step

Run these seven steps in order per batch of repositories. The timings assume a single operator working from a checked-out set of repos; the identity-collapse step is the one that varies most with repo count and contributor overlap.

The extract command is the load-bearing part of step one. Use --pretty=format: so git renders each commit using only the placeholders you specify and adds nothing else, which is what makes the output machine-readable. Author email is %ae, committer email %ce, and the mailmap-resolved variants are %aE and %aN.

git log --pretty=format:'%H%x09%an%x09%ae%x09%cn%x09%ce%x09%aI'

One order-of-operations choice is genuinely open. Some operators pull author and committer as separate rows and dedupe later; others keep one row per commit and collapse afterwards. Either works, because you dedupe on the identity key rather than the row. Keeping both the author and committer fields is worth the width: it lets you catch a CI account that committed another engineer's authored work, which is exactly the kind of contamination the bot filter is meant to remove.

For the collapse step, git shortlog -se lists contributors with commit counts and emails, which surfaces the duplicate addresses you then map. The .mailmap file at the top level of a repository maps author and committer names and emails to canonical identities, and git check-mailmap lets you test whether a given identity is already mapped. Build the mailmap keyed on the user ID first, and only reach for name-and-email co-occurrence where no ID exists.

Once the pipeline runs, the value of the output scales with how scarce the underlying talent is. In Refolk's index, Rust software engineers are far thinner on the ground than the raw noise in a commit extract might suggest, which is exactly when a clean file beats a bought one.

How thin the target list really is

The reason a clean commit-mined file is worth the work is supply. In Refolk's index, the pool of engineers for a scarce language in a smaller market is small enough that every duplicate and every bot row you fail to remove is a meaningful fraction of your usable list.

Counts are from Refolk's index; the ratio is derived from 610 divided by 88. The same asymmetry shows up within a single market when you compare languages, which tells you how much a dirty extract can distort a headcount if you never dedupe.

Counts are from Refolk's index; Go is 4.5x more common than Rust in the US, derived from 2,732 divided by 610. Read these two tables together and the operating rule is clear: the rarer the skill and the smaller the market, the more a clean, deduplicated commit-mined file is worth relative to buying a saturated list, because you cannot afford to lose real people to sloppy dedup or drown them in bot rows. When the search is well specified, Refolk returns the named people directly, which is the faster path than mining raw and cleaning up after.

How this goes wrong

The pipeline fails in specific, repeatable ways, and each has a cheap check that catches it before handoff. Treat this section as the acceptance test the file has to pass.

• Regex misses legacy noreply. A \d+\+ anchor catches only ID-based addresses, so username-only noreply slips through as contactable. Check: run both patterns and count residual @users.noreply.github.com rows; it should be zero.
• Bot hiding behind a human name. A CI job configured with a name like Buildkite but pushing under a real engineer's login pollutes identity clusters if you filter on name alone. Check: flag any (login, name) pair that appears below a co-occurrence threshold, and inspect it.
• Over-collapsing shared addresses. A communal address like bugs@company.xx maps many humans to one record, and a contributor missing from .mailmap still keeps that shared email. Check: reject any mailmap merge where one email spans multiple distinct GitHub IDs.
• Role address that passes verification. A role address on a healthy domain often passes syntax, MX, and mailbox checks yet is still low-value. Check: match the local-part against the generic-prefix list independently of the deliverability verdict.
• Deliverable but wrong destination. In the audited case, every role row delivered and almost none replied. Check: segment reply rate by address class, not just bounce rate, so the difference is visible.
• Case-sensitivity splits one person. Un-normalised Jane and jane inflate the person count. Check: confirm every email is lowercased before dedup runs.
• Stale address counted as live. An address valid years ago may now bounce. Check: attach a last_seen commit date and flag anything past the cutoff.

The two failure modes that cost the most are the legacy-noreply miss and over-collapsing on a shared address, because both are silent. The noreply miss inflates your contactable count with addresses that will never reach a person, and the over-collapse quietly merges several real engineers into one record you will never notice is wrong. Both are cheap to prevent and expensive to find after handoff.

Before you call it done

Run this checklist against the output file. If any item fails, the file is not ready to hand off, no matter how clean it looks.

Keeping the pipeline current

The parts of this pipeline most likely to drift are the bot patterns and the noreply date rule, so re-check them against source rather than trusting a cached copy. GitHub's email addresses reference is the authoritative spec for the two noreply formats and the July 18, 2017 cutoff; the Dependabot and GitHub Actions issue trackers are where the exact bot author and sign-off addresses are documented, and those have already changed once, when Dependabot moved from support@dependabot.com to support@github.com.

Two operational notes on freshness. First, whether privacy is GitHub's default is genuinely not established publicly: one practitioner write-up claims the noreply rewrite is the default, a community thread argues the setting is off by default, and the only documented fact is that enterprise managed-user accounts created after August 1, 2024 are unverified by default. Do not build any masking-rate assumption into the pipeline; treat every account's masking state as observed, not assumed. Second, if you widen the source beyond checked-out repos, the GH Archive dataset has recorded GitHub's public event firehose hourly since 2011, with PushEvents carrying commit author name and email, which is a way to backfill history without cloning. The same cleanup steps apply to that extract unchanged.

Re-run the acceptance checklist every time you add a repo source or change the identity-collapse logic. The pipeline is only as trustworthy as its last audit, and the whole point of quarantining rather than deleting is that the audit stays possible.