The Repo Collection-Route Verdict: API, Archive, Clone, or Decline

A teammate wants repository data for a sourcing or prospecting list: contributors, commit history, activity or dependency files. Before anyone writes a script, someone has to decide how that data gets collected. This guide is for the recruiting ops or revenue ops lead who owns that decision and has to defend it later. It gives you a six-dimension scoring model that puts every request on one of four routes, along with a written rationale that a second reviewer can reproduce.

What does the collection-route verdict decide?

The verdict decides which channel the repository data comes through: the authenticated API, the public event archive, a local clone, or no route at all. It is a gate in front of every repo data job. It comes before any mining pipeline runs and before any contact is cleared for outreach.

The four routes:

• Authenticated API. GitHub's REST API or its GraphQL API, called with a token tied to a named user or app. (GraphQL is a query language where one request asks for exactly the fields it names. It is billed in points rather than requests.)
• Public event archive. GH Archive is a public record of GitHub's Events API, which is the feed of public activity such as pushes, pull requests and issues. It is published as hourly JSON downloads and as a BigQuery dataset that updates every hour.
• Local clone. A full git copy of the repository, read on your own machine. One git operation fetches the whole history.
• Decline. The request does not go ahead in its current form, and the record says why.

This used to be a choice of convenience. Now it decides whether the fields exist at all, because three platform mechanics changed:

1. GitHub trimmed Events API payloads, with full rollout on October 7, 2025. Push events no longer carry the commits array that held commit SHAs, author names, emails and messages.
2. Stricter unauthenticated limits, announced on May 8, 2025, now cover cloning over HTTPS, anonymous REST calls and downloads from raw.githubusercontent.com.
3. GH Archive has hashed email addresses since 2016-08-11. It is built by polling the Events API, so it inherited the payload trim as well.

Both of the 2025 changes cite scraping or abuse, and both push bulk collection onto authenticated access. People often miss what follows from that: the token you collect with is now part of how the data was gathered, whether or not you write it down.

What does each route still return?

Commit identity (who wrote which commit, with name and email) now comes only through the authenticated API or a clone. The Events API and the archive still return activity and identifiers. For events from October 7, 2025 onward, they no longer return commit authors.

A check on real events after the trim found PushEvent carrying only before, head, push_id, ref and repository_id. The commits, head_commit, pusher and compare fields were gone. PullRequestEvent kept only id, number, url, base and head. The author_association field was removed from these event types: pull request, review, review comment, issue, issue comment, commit comment and discussion. GitHub says it will keep supplying the identifiers so you can fetch the details through the standard REST API.

This table leaves three questions open:

• GraphQL versus REST. GraphQL is expected to return the same fields as REST, but nobody has documented that field by field. Pull ten known commits both ways and diff them before you rely on GraphQL.
• What a clone actually captures. I could not source any documentation on how a clone records squash-merged pull requests, Co-authored-by trailers, .mailmap aliases or force-pushed history. Test each one on a repository whose history you already know.
• Archive hashing. The hashing algorithm behind archive emails is not public. Treat those identities as opaque.

How much rate budget does each route cost?

On the authenticated API you pay one request per page, plus one per detail lookup, out of an hourly budget set by the identity you collect with. A clone costs one git operation. The archive costs storage and query spend instead of API calls. Here is the budget for each identity:

App installations scale with size. An installation covering more than 20 repositories gets another 50 requests per hour for each repository. One on an organisation with more than 20 users gets another 50 per hour for each user. Neither can go above 12,500.

The hourly numbers have two more ceilings on top of them:

• Secondary limits. GitHub staff describe these as no more than 900 points per minute for REST and 2,000 points per minute for GraphQL. A job whose hourly total fits can still fail on a burst. A user token's 5,000 requests per hour averages about 83 a minute, so the per-minute cap only bites when you run requests in parallel. That is exactly what people do to finish faster.
• The search ceiling. Search returns at most 1,000 results per query, which is 10 pages of 100. Code search requires authentication, allows 10 requests per minute, and only covers the default branch and files smaller than 384 KB. Search is cheap, but it cannot be complete for any result set larger than 1,000 unless you split the query.

Where the commit walk eats the budget

Fetching the commit list is cheap. The per-commit detail calls are what cost you. The arithmetic below is my own and has not been benchmarked. It assumes the list endpoint returns 100 commits per page, which I have not verified.

Detail calls multiply the request count about 100 times over the list calls. A 100,000-commit repository takes roughly 20 hours of one user token over REST.

A clone is a single git operation. I have not measured its byte size, and nobody has published timings comparing clone and API at these sizes. Unauthenticated clones now have their own limits, and GitHub has not published the numbers. A secondhand per-IP figure circulates in community threads, but it is unverified, so do not plan around it. Authenticate your clones.

The archive's costs are bytes and query spend:

• One compressed hour (2021-03-01-15) came to 104,531,294 bytes.
• One practitioner's download of 2018 to 2021 totalled about 1.3 TB, rising from 175 GB for 2018 to 443 GB for 2021.
• The BigQuery cost of pulling one year of events for a set of repositories is not publicly established. Price a single month on your own project before you commit to a year.

The six scoring dimensions

Every request gets scored on two request-level gates and four route-level dimensions, each on a 0, 1 or 2 scale. A 0 on a gate stops the request. A 0 on a route dimension rules that route out. Of the routes that remain, the highest total wins.

The gates apply to the request as a whole, because purpose and personal data do not depend on the pipe. The four route dimensions are scored separately for the API, the archive and the clone.

Gate 1: Terms-of-use fit

• 2: The output is non-personal: repository counts, activity aggregates, dependency files.
• 1: The output names people. Nobody will be contacted at an address taken from commit metadata, and the use matches the purpose the person published the information for.
• 0: The plan does any of the following: sends unsolicited email to commit addresses; passes personal information to recruiters, headhunters or job boards; or calls a list of named people "research" without open-access publication.

What it proves: the purpose was tested against the platform's information-usage terms before anyone spent engineering time. When it lies: the purpose sentence is written to pass ("market research") while the deliverable is a contact list. Score the deliverable, not the label.

Gate 2: Personal-data exposure

• 2: No personal fields.
• 1: Public logins and fields the person published on their profile, and nothing more.
• 0: Commit author names or emails stay in the output, or identities are joined across sources so they can be contacted.

A 0 here does not automatically mean decline. Strike the field and re-score. Decline only if the requester cannot do the job without it. When it lies: the field list says "contributors" when it really means emails. Ask for the actual column list.

Field availability (per route)

• 2: The route returns every requested field directly, and you have confirmed it on a sample pull.
• 1: The route returns the field through a follow-up call, or through behaviour you have now tested yourself (for example, clone authorship on squash merges).
• 0: The route does not return the field. Examples: commit authors from the archive after the trim, or contactable emails from the hashed archive.

What it proves: the data exists on this route. When it lies: the route returns 200 OK with the field empty. Score from a sample pull, not from the documentation.

Rate-budget cost (per route)

• 2: Under one hour of the token's budget, paced below the secondary cap.
• 1: Several hours, but inside the job window.
• 0: Longer than the window, or relies on unauthenticated access at scale.

When it lies: the hourly total fits, but parallel workers trip the per-minute cap and leave gaps nobody sees.

Freshness need (per route)

• 2: The route's latency meets the freshness the requester stated.
• 1: The route meets it with a follow-up delta pass.
• 0: The route cannot meet it.

For reference: the API is live, the archive updates hourly, and a clone is a snapshot taken when you clone. Before the trim, events could take up to eight hours to appear in the Events API. When it lies: requesters often ask for "real-time" without thinking. Ask which decision would change if the data were a day old.

Provenance auditability (per route)

• 2: You can log the route, endpoint, API version, timestamp, token identity and query, and check the result set against an expected count.
• 1: You can log all of that, but cannot fully verify the result. Examples: search slices sitting near 1,000, or opaque archive identities.
• 0: The access is unauthenticated, or comes through a shared or third-party source whose query cannot be retrieved.

When it lies: a log exists, but it records a shared service token that no named person owns.

The decision rule

1. If either gate scores 0 after you strike fields, the verdict is decline.
2. Rule out any route with a 0 on any of the four route dimensions. If no route survives, the verdict is decline.
3. Pick the surviving route with the highest total across the four route dimensions.
4. On a tie, pick the route with the higher provenance score. If it is still tied, pick the one with the lower projected request count.

Most requests take the same shape: how deep the history goes against how fresh it needs to be.

Record every verdict in the same form, so that a second reviewer re-scores from the same evidence:

REPO DATA REQUEST - ROUTE VERDICT
Request ID:
Requester:
Purpose (one sentence, describing the deliverable):
Fields requested (column list):
Repositories and time window:

GATES (request-level, score 0 / 1 / 2)
Terms-of-use fit:        _   Reason:
Personal-data exposure:  _   Fields struck to reach this score:
AUP page read on (date):
Gate result: PASS / DECLINE

ROUTE SCORES (0 / 1 / 2; any 0 rules the route out)
                          API   Archive   Clone
Field availability         _       _        _
Rate-budget cost           _       _        _
Freshness need             _       _        _
Provenance auditability    _       _        _
Total                      _       _        _

Projected requests / hours:
Verdict: API / ARCHIVE / CLONE / DECLINE
Tie-break used (if any):
Rationale (three sentences maximum):
Reviewer 1:          Reviewer 2:          Same verdict: Y / N
Deletion date:

Where do requests fail the terms and personal-data gates?

Requests fail when the deliverable uses personal information from commits for something the person never authorised. The usual cases are unsolicited outreach to commit emails and lists handed to recruiters. The research permission does not rescue them, because it only covers public, non-personal information with open-access publications.

Section 7 of GitHub's Acceptable Use Policies, as reproduced word for word in a third-party reuse of the text, says the following:

• Researchers may use "public, non-personal information from the Service for research purposes, only if any publications resulting from that research are open access".
• Archivists may use public information for archival purposes.
• Use "for spamming purposes, including for the purposes of sending unsolicited emails to users" is prohibited.
• Selling personal information to recruiters, headhunters and job boards is prohibited.

Older wording limits personal information to "the purpose for which our user authorized it". A secondary source says the policies also require you to honour do-not-contact requests. I did not pull the current section 7 and 8 wording from GitHub's own page. Read it there before you adopt this gate, and put the date you read it in every verdict record.

History shows what this gate protects against. Before the archive began hashing emails on 2016-08-11, a third party had already extracted 5.8 million commit emails from it. Any request that amounts to rebuilding that list is a decline.

What replaces a declined request

A declined request usually still has a real need behind it: to find the people who maintain or contribute to a body of code. Most of the time, that population is smaller and easier to reach through profiles than through commits.

These counts come from headline-keyword matches in Refolk's index of professional profiles. Read them as indicative, not as a census. In a 25-profile sample, no single title appeared more than twice in either market, and Karlsruhe accounted for 4 of the 25 German profiles sampled. Both details come from a small sample.

A headline match proves that the person describes themselves that way. It does not prove they are a maintainer. It misleads in both directions: some people maintain projects without saying so, and some say so about a project they have left. Before outreach, check the claim against the repository.

This is where Refolk removes the friction. You ask in plain English for the maintainers you want, and get back profiles where people chose how to describe themselves and how to be reached. Commit metadata never comes into it, so the personal-data gate scores 1 instead of 0.

The cheapest route to a maintainer is usually their profile, not their commit history.

The route-verdict procedure

This procedure runs every repository data request through the same eight steps. Steps 1 to 5 decide the route, and steps 6 to 8 collect, check and trim the data. None of my sources documents this as a single sequence. It is assembled from the rate-limit, payload and provenance material above.

My sources do not settle whether the terms screen should come before field mapping. I put it first on purpose. A request that will be declined anyway should not cost anyone 30 minutes of payload research. My own desk estimate for steps 1 to 5 is a little over two hours in total: about 15 minutes for intake and about 30 for each of the rest. No source backs that figure, so time your first few runs.

If a request is declined at step 2, it is not a dead end. Re-scope it to a profile-level search, in Refolk or any other source that returns profiles people published themselves, and run it through the gates again. It will usually pass.

How does the verdict go wrong?

It usually goes wrong silently. The collection succeeds, returns 200 OK, and delivers a dataset with fields missing, results cut off or the wrong people in it. Each failure below comes with the check that catches it.

1. Silent field loss. You pull from the Events API or GH Archive for any period after October 7, 2025. PushEvents come back with no commits field, and the status is still 200 OK. This is the most expensive failure because nothing flags it. Check: assert a non-null count for every requested field before the dataset leaves the collection step.
2. Search truncation. Exactly 1,000 results with a full last page means the data was cut off, and no error was raised. Check: split the query by date or qualifier until every slice returns fewer than 1,000. Any request for "all contributors" via search cannot be audited until that is done.
3. Purpose laundering. A sourcing list gets labelled "research" to fit AUP section 7. The research permission covers non-personal information with open-access publication only. Check: does the output contain names or emails, and will it be published openly? If the answers are yes and no, it is not research.
4. Hashed-email false match. Archive identities appear as null or as a username hash followed by @. They look like addresses but cannot be resolved to one. Check: filter out any identity in that form before counting contactable people. Never report one as a contact.
5. Budget passes, secondary cap fails. The hourly maths fits, but a burst trips the 900-per-minute REST cap. The 403 and 429 responses leave gaps that look like quiet repositories. Check: compare the request log with the expected page count from the dry run, and re-fetch anything missing.
6. Unauthenticated clone at scale. Limits keyed to an IP address hit shared office networks and CI runners, and the job dies partway through. Because the numbers are unpublished, you cannot plan around them. Check: authenticate every clone and record the token identity.
7. Clone-route authorship assumptions. Nobody has verified whether a clone attributes squash-merged work to the right authors or applies .mailmap aliases the way you expect. Check: before relying on either, test both on a known repository where you already know who wrote what.

Failures 1 and 4 both make a dataset look richer than it is. Failures 2 and 5 make it look smaller than it is. A reviewer who only checks row counts will miss all four, which is why step 7 checks fields and pages, not rows.

What belongs in the provenance record?

The provenance record has to let someone who was not in the room re-run the collection or prove it wrong. That means it holds the purpose and verdict, the route and token identity, the exact query and configuration, a timestamp, how the result set was verified, and when the data will be deleted.

The research literature shows how often this record is missing. A systematic review of mining-software-repositories papers found that a third did not say how their data was retrieved, and 175 gave no collection timestamp.

Two published standards cover most of what you need:

• RDA data citation guidelines. These set out 14 recommendations, including timestamping (R2), a query store, query uniqueness, result-set verification, and storing the query itself.
• A repository-mining artifact standard. This asks for retrieval dates or timestamps, the data acquisition method, and references to the retrieval configuration.

Neither standard names token identity. I require it anyway. Since the 2025 changes, bulk collection runs on authenticated identities, so the token is part of how the data was gathered. A record without it cannot answer the first question an auditor will ask.

route: rest-api | graphql | archive | clone
endpoint_or_source:
api_version:
token_identity (named owner, not a shared account):
collected_at_utc:
query_exact:
search_slices (each with result count):
fields_requested:
fields_non_null_count:
expected_pages:
received_pages:
responses_403_or_429:
fields_dropped_after_collection:
delete_by:
verdict_record_id:

Keeping the verdict current

The verdict model holds up over time. The numbers inside it do not. Re-check the mechanisms on a schedule instead of trusting the values printed here. Every change described in this guide arrived as a platform announcement, and each one silently changed what at least one route returns.

What to re-check, and how:

• Event payloads. Pull one recent hour from the archive and one page from the Events API. Then confirm which fields a PushEvent and a PullRequestEvent carry. If the field list changed, update the field table before the next verdict.
• Rate limits. Re-read GitHub's REST, GraphQL and Search rate-limit documentation, and re-run the Dataset C arithmetic for your identity type.
• Terms. Re-read sections 7 and 8 of the Acceptable Use Policies on GitHub's own page, and update the date in your verdict template.
• Clone authorship. Re-run the squash-merge and .mailmap test on your known repository. Behaviour you have tested scores 1 on field availability. Behaviour you have only assumed scores 0.
• Reviewer agreement. Every so often, pick a few past verdicts and have a new reviewer score them cold. If they reach a different route, the rubric wording needs tightening.

To adopt this, start with the next request that lands on your desk. Fill in the verdict record by hand, have a colleague score it without seeing your scores, and compare. If you disagree on a score, the rubric wording that caused it is what to fix before the model becomes your standing gate.