# The Repo-Data Collection Clearance Standard for Lawful Use

*You can grade any repo-data collection job as cleared-to-run, needs-changes, or prohibited so two reviewers reach the same verdict.*

- Canonical URL: https://www.refolk.ai/guides/repo-data-collection-clearance-standard
- Pillar: Process, data, and compliance
- Format: Standard
- Published: 2026-09-15
- Last reviewed: 2026-09-15
- Reading time: 17 min

Deciding whether a repository-data collection job may run is a compliance judgement that happens before anyone writes a query. This standard is for recruiting operations, revenue operations, and anyone answerable for how the data was gathered. It gives you a single pass/fail rubric to grade any proposed repo-data job as cleared-to-run, needs-changes, or prohibited - by access method, fields taken, and downstream use - so two reviewers grading independently reach the same verdict.

Other guides in this library clear an already-collected batch for outreach or grade one mined email. This one sets the bar earlier: whether the collection method itself is permitted before it runs. It turns buried platform acceptable-use rules and hedgy vendor commentary into one rubric, covering the scrape-versus-API distinction, the two permitted purposes, and the recruiter and headhunter outbound-use prohibition.

## What this standard grades, and what "cleared" means

This standard grades a *proposed collection job* on three axes - access method, fields taken, and downstream use - and returns one of three verdicts: cleared-to-run, needs-changes, or prohibited. Cleared means every field maps to a permitted purpose under the source platform's terms, any personal data has a lawful basis and a retention limit, and a reproducible record exists.

The verdict is about permission to collect, not about whether the resulting list is worth sending. A job can produce technically excellent data and still be prohibited because its purpose is banned. The three verdicts are deliberately coarse so that reviewers agree:

- **Cleared-to-run.** The purpose is on the permitted list, personal-data obligations are planned, and the record is complete.
- **Needs-changes.** The purpose could be permitted but a field, a retention gap, or a missing source-notice must be fixed first. Re-grade after the fix.
- **Prohibited.** The purpose is banned outright - most often unsolicited recruiting outreach or selling personal information to recruiters. No change to method rescues it.

The load-bearing idea is that the access method never decides the verdict. GitHub wrote its recruiting ban to catch data "whether scraped, collected through our API, or obtained otherwise," so a reviewer who clears a job because it used the official API has misread the policy.

> Method is a red herring. Purpose is the gate, and the platform wrote it that way on purpose.

## The two access methods, defined the platform's way

Scraping and API collection are distinct methods with the same governing purpose ban, so you name the method to cite the right clause, not to decide whether the job may run. GitHub defines scraping as "extracting information from our Service via an automated process, such as a bot or webcrawler," and states plainly that scraping "does not refer to the collection of information through our API."

That distinction routes you to the correct clause. Scraping and general use are governed by the Acceptable Use Policies, Section 7, "Information Usage Restrictions." API access is governed additionally by the Terms of Service, Section H, the API terms. Both carry the same recruiting prohibition. The AUP says you may not use information from the Service "for spamming purposes, including for the purposes of sending unsolicited emails to users or selling personal information, such as to recruiters, headhunters, and job boards." The API terms repeat it: you may not use the API "to download data or Content from GitHub for spamming purposes, including for the purposes of selling GitHub users' personal information, such as to recruiters, headhunters, and job boards."

> **Rule:** The purpose ban is method-neutral
>
> The recruiting, headhunter, and job-board prohibition applies whether data was scraped, collected through the API, or obtained otherwise. Naming the method selects the clause to cite. It never converts a prohibited purpose into a cleared one.

Method still carries a practical constraint worth recording. Unauthenticated GitHub API access is limited to 60 requests per hour, and a search query returns a maximum of 1,000 results. Those limits shape feasibility, not permission.

## The permitted-purpose list, by platform

GitHub expressly permits exactly two purposes and expressly prohibits the recruiting purpose; GitLab bans obtaining information for unsolicited communication or for selling personal data. GitHub's AUP permits research and archival on narrow terms: "Researchers may use public, non-personal information from the Service for research purposes, only if any publications resulting from that research are open access. Archivists may use public information from the Service for archival purposes."

Read those carve-outs literally. Research requires *non-personal* data and *open-access* publication. A sourcing pipeline meets neither, so leaning on the research exemption converts a prohibited job into a fabricated "cleared." Archival is for preserving public information, not for building an outreach list.

| Platform | Expressly permitted | Expressly prohibited | Applies to scraping and API |
| --- | --- | --- | --- |
| GitHub | Open-access research on non-personal data; archival of public information | Unsolicited email to users; selling personal info to recruiters, headhunters, job boards | Yes |
| GitLab | None named; you must not misuse personal data | Obtaining info for unsolicited communication; selling users' personal data | AUP applies to all services |

GitLab's Acceptable Use Policy states you must not use its services "to obtain information for the purposes of sending unsolicited communication to users or selling users' personal data." That matters because GitLab exposes commit data the same way GitHub does: "for each commit, the names (account name and full name) and email address as stored in the configuration of Git are visible." The same field that makes a commit log useful for outreach is the field that makes the outreach prohibited.

## Public data is not the same as permitted data

A public repository being readable does not make collecting from it permitted; legality under anti-hacking law and permission under a platform's terms are separate questions with separate answers. The most-cited scraping case, hiQ Labs v. LinkedIn, is routinely misread as a green light. It is not.

The Ninth Circuit held that scraping public data was not a Computer Fraud and Abuse Act violation. But that ruling "was limited in scope to the CFAA and did not apply to potential claims against web scrapers under other theories, including trespass to chattels, copyright infringement, misappropriation, unjust enrichment, conversion, breach of contract or breach of privacy claims." hiQ then agreed to a stipulated judgment: "$500,000 judgment to be entered against hiQ for a breach of contract based on LinkedIn's user agreement."

The mechanism is simple. Platforms shifted enforcement from anti-hacking law to their user agreements, and those agreements bind regardless of whether the data is public. So "it is public" answers a question this standard does not ask. This standard asks whether the platform's terms permit your purpose, and, for personal data, whether you have a lawful basis.

**$500,000 - Judgment hiQ paid for breaching LinkedIn's user agreement**

The CFAA win did not shield hiQ from the contract claim, which binds public data too.

## Commit email: the field that carries the risk

Commit author email is personal data, and it is the one field that is simultaneously the most useful for outreach and unambiguously regulated, which is why it drives most needs-changes and prohibited verdicts. Repo names, star counts, and topics are non-personal and sit under the platform terms alone. Author email identifies a person, so it pulls in the platform ban plus GDPR obligations.

| Artifact taken | Personal data? | Governing obligation |
| --- | --- | --- |
| Repo name, stars, topics | No | Platform terms only |
| Commit author email | Yes | Platform ban plus GDPR Article 14 notice plus retention limit |
| Profile plus contact for outreach | Yes | Prohibited purpose under GitHub AUP |

Two traps hide inside this field. First, noreply addresses. GitHub noreply commit emails since July 18, 2017 use the ID-plus-username at users.noreply.github.com format. They look anonymised, but they still map to an account and a username, so treat them as personal data. Second, the mistaken belief that email is "just metadata." It is not. Under GDPR it needs a lawful basis and a retention limit, and where the data subject is in the EU it triggers an Article 14 source-notice.

> **Watch out:** Do not classify commit email as non-personal
>
> Author email - including ID-based noreply addresses - identifies a person. Listing it as metadata skips the lawful basis, the retention limit, and the one-month source-notice. That single misclassification turns a prohibited job into a false "cleared."

If your intent is outbound sourcing rather than raw collection, the friction the platform terms create is real: the useful field is the banned field. This is where describing the person you want in plain English, and letting the resolution happen against an index built for that purpose, removes the temptation to mine commit logs directly. [Refolk](/) is built to answer that kind of request without you touching a prohibited collection path.

I ran this search: `Data-protection officers and privacy compliance analysts at US software companies` - [see the full result list](https://www.refolk.ai/s/wzknb2r11n).

*Returns named people matched to data-protection titles, resolved from public professional profiles rather than mined from commit logs.*

## The lawful-basis and retention obligations for personal data

For any personal data, GDPR requires a lawful basis, a documented retention period tied to a purpose, and, when data comes from a source other than the individual, a notice within one month. GDPR sets no fixed retention period: "organizations must define them, document them, justify them against a lawful basis and a stated purpose, and actually enforce them through deletion or anonymization."

Three obligations attach the moment you keep a personal field:

1. **Source-notice.** "GDPR requires you to inform individuals when you've collected their data from sources other than themselves. This notification must occur within one month of collection and include your identity, processing purposes, and their rights." Read the disproportionate-effort exemption narrowly.
2. **Purpose limitation.** "You can only use and retain personal data for your original, stated purpose. If that purpose changes for whatever reason, you must obtain additional consent before any further processing." A batch collected for code assessment cannot quietly become an outreach list.
3. **Retention enforcement.** A per-activity schedule records "the category of personal data involved, the purpose and lawful basis, the retention period derived from that basis, and the deletion or anonymization process that fires when the period expires."

Under CCPA and CPRA the notice at collection must state "the categories of personal information you collect; the purposes for which the personal information is collected or used; whether the information is sold or shared; and the length of time you intend to retain each category of personal information." The retention-length requirement is the one most collection jobs forget.

#### What binds a repo-data job, outermost first

1. **Platform terms** - The AUP and API terms set permitted and prohibited purposes for all data
2. **Contract enforceability** - The user agreement binds you even for public data, as hiQ shows
3. **GDPR / CCPA obligations** - Personal data needs a lawful basis, retention limit, and source-notice
4. **Your record** - A standalone RoPA row that a second reviewer can reproduce

*Every layer applies at once; passing the outer layer does not exempt you from the inner ones.*

## The record that makes a verdict reproducible

The absence of a Record of Processing Activities row is itself a finding: a job with no source, method, purpose, basis, date, and retention line cannot be graded cleared. GDPR Article 30 requires a "structured record-keeping document" - the RoPA - and the Irish Data Protection Commission states that "controllers and processors must be in a position to provide such records to the Data Protection Commission on request."

Article 30 was adopted in April 2016 and has been enforceable since May 2018. The small-organisation derogation for fewer than 250 employees is narrow and rarely applies where processing is not occasional, so do not assume it exempts you. A DPIA is not a substitute, and neither is a general spreadsheet. The record is a standalone, on-demand document.

**RoPA row for a repo-data collection job**

```
Source platform: GitHub (public repositories)
Access method: API | Scrape (bot/crawler)
Governing clause: AUP Section 7 | ToS Section H
Fields taken: repo metadata; commit author name; commit author email
Personal data present: Yes | No
Downstream purpose (one sentence):
Purpose bucket: research (open-access) | archival | internal code assessment | outbound recruiting
Verdict: cleared | needs-changes | prohibited
Lawful basis (if personal data):
Retention period + deletion trigger:
Article 14 source-notice due by (collection date + 1 month):
Collection date:
Reviewer 1 / Reviewer 2:
```

*One row per job. Fill every field before the job runs; a blank retention or lawful-basis cell means the job is not yet cleared.*

## The clearance procedure

Grade a job by running these seven steps in order; the verdict is not final until a second reviewer reproduces it from the record alone. Each step names what "done" looks like so two reviewers stop at the same place.

#### How to grade a repo-data collection job before it runs

1. **Classify the access method** - Confirm whether the job scrapes web pages via a bot or crawler, or uses the API, using the platform's own definitions. Done when the method is named and the governing clause is cited: AUP Section 7 for use, ToS Section H for API.
2. **Enumerate the fields taken** - List every exact field, from repo metadata to commit author name and email to profile and contact data. Done when each field is mapped to personal or non-personal. Commit author email is personal data.
3. **Declare the downstream use** - The requester states the purpose in one sentence. Done when it maps to a named bucket: open-access research, archival, internal code assessment, or outbound recruiting.
4. **Apply the pass/fail rule** - Grade the purpose against the permitted list, not the method. Done when a verdict is set. Unsolicited recruiting email or selling personal data to recruiters is an automatic prohibited.
5. **Run the legal-basis and retention check** - For any personal data, assign a GDPR lawful basis, a retention period, and, for EU subjects, plan the Article 14 source-notice within one month. Done when the retention period and deletion trigger are documented.
6. **Record it in the RoPA** - Append a row with source, access method, fields, purpose, lawful basis, date, retention, deletion process. Done when it is a single reproducible row a second reviewer could read cold.
7. **Replicate with a second reviewer** - An independent reviewer re-grades from the record alone. Done when the two verdicts match. If they diverge, escalate and resolve the record before the job runs.

The purpose bucket decides the outcome, so spend the judgement there. Use this matrix when a requester's stated purpose is ambiguous.

#### Purpose against personal-data content

Horizontal axis runs from Purpose permitted (research/archival/internal) to Purpose prohibited (outbound recruiting/selling). Vertical axis runs from No personal data to Personal data present.

| Quadrant | What it means |
| --- | --- |
| Permitted, no personal data | Cleared - record the RoPA row and run |
| Prohibited, no personal data | Prohibited - purpose ban applies regardless of fields |
| Permitted, personal data present | Needs-changes - add lawful basis, retention, source-notice |
| Prohibited, personal data present | Prohibited - the worst case; do not collect |

*The verdict is driven by the vertical axis; personal data raises the obligations but does not rescue a banned purpose.*

## How this goes wrong: the false positives to catch

Most bad clearances come from a small set of predictable misreadings, and each has a one-line check that flips the verdict back. Give this section weight; catching these is the whole point of a second reviewer.

- **"We used the API, so we're fine."** The reviewer clears an API job for recruiting outbound. Check: the recruiting and spam ban is method-neutral and named in both the AUP and the API terms. Grade on purpose, not method.
- **"It's public data, hiQ says it's legal."** The reviewer treats the CFAA ruling as blanket permission. Check: hiQ still paid $500,000 on contract. Confirm the platform's purpose ban separately from any CFAA question.
- **"Commit emails are just metadata."** The field list omits author email as non-personal. Check: author email identifies a person and is GDPR personal data. It needs a lawful basis and a retention limit.
- **"The research exemption covers us."** The reviewer claims research cover for commercial sourcing. Check: the exemption is limited to non-personal data with open-access publication. Internal recruiting meets neither.
- **"We'll notify later."** The reviewer skips the Article 14 source-notice. Check: the notice is due within one month, and the disproportionate-effort exemption is read narrowly.
- **"We logged it somewhere."** A DPIA or a general spreadsheet is treated as the record. Check: the RoPA is a standalone document with source, method, purpose, basis, date, and retention, reproducible by a second reviewer.
- **"Noreply emails are safe to keep."** The reviewer retains ID-based noreply addresses as non-personal. Check: they still map to an account and username. Treat them as personal data.

The through-line is that every false positive substitutes an easy signal - method used, data public, field looks anonymised - for the actual gate, which is purpose plus personal-data status plus a producible record.

> **Note:** Where the sources genuinely diverge
>
> The sources agree on the grading steps. They diverge only on whether scraping public data is "legal" in the abstract: hiQ says it is not a CFAA violation, but contract and GDPR still bind. This standard resolves the ambiguity by grading permission and lawful basis, not abstract legality.

## The clearance checklist

Run this before you call a job cleared. Every item is a checkable statement, not a topic, and a single "no" sends the job to needs-changes or prohibited.

#### Before this collection job runs

- [ ] The access method is named and the correct governing clause is cited (AUP Section 7 or ToS Section H)
- [ ] Every collected field is mapped to personal or non-personal, with commit author email marked personal
- [ ] The downstream purpose is written in one sentence and mapped to a named bucket
- [ ] The purpose is on the permitted list, not outbound recruiting or selling personal data
- [ ] Any research claim rests on non-personal data with open-access publication, or it is dropped
- [ ] Every personal field has a lawful basis and a retention period with a deletion trigger
- [ ] For EU data subjects, the Article 14 source-notice is scheduled within one month of collection
- [ ] A standalone RoPA row exists with source, method, fields, purpose, basis, date, and retention
- [ ] A second reviewer reproduced the same verdict from the record alone

## Keeping the standard current

This standard depends on wording that platforms revise, so re-read the source clauses on a fixed cadence rather than trusting a cached memory of them. The GitHub Acceptable Use Policies Section 7, the Terms of Service Section H, and the GitLab Acceptable Use Policy are the three documents whose text drives every verdict here. Check them quarterly and whenever a large collection job is proposed, and diff the recruiting and permitted-purpose language against your copy.

Two mechanisms to watch specifically. First, enforcement posture: GitHub "retains full discretion to take action in response to a violation of these policies, including account suspension, account termination, or removal of content," and API abuse "may result in the temporary or permanent suspension of your Account's access to the API." That discretion means the cost of a wrong clearance is not hypothetical. Second, retention and notice rules under GDPR and CCPA/CPRA evolve through guidance, not just statute, so re-check the source-notice window and the notice-at-collection requirements when you refresh the standard.

Route the temptation that drives most violations away from raw collection entirely. In Refolk's index, there are 64 US privacy and compliance professionals with GDPR-related profiles matching data-protection titles, and the index is built to resolve requests for people without you mining a prohibited path. Where the market skew matters to your sourcing, Refolk's index shows the concentration plainly.

| Market | Technical recruiters with Boolean-search skill | Share of the two-market total |
| --- | --- | --- |
| United States | 11 | 92% |
| Germany | 1 | 8% |

When you can describe the person you want and get them back from an index designed for that purpose, the incentive to scrape commit logs for email disappears, and with it the field that carried most of your legal risk. That is the cleanest way to keep every future job on the cleared side of this standard.

## Frequently asked questions

### Is it legal to scrape GitHub emails for recruiting?

No, not under GitHub's own rules. GitHub's Acceptable Use Policies prohibit using information from the Service for spamming, including sending unsolicited email to users or selling personal information to recruiters, headhunters, and job boards. That prohibition applies whether the data was scraped, collected through the API, or obtained otherwise. Even where scraping public data avoids a CFAA claim, the platform's terms still bind you contractually, and commit email is personal data under GDPR.

### Does using the official API instead of scraping make a recruiting job compliant?

No. The recruiting and spam ban is method-neutral: GitHub names it in both the Acceptable Use Policies and the separate API terms, which prohibit downloading data to sell users' personal information to recruiters. Clearing a job because it used the API is a common false positive. Grade the job on its downstream purpose, not on how the data was accessed.

### Doesn't the hiQ v. LinkedIn case make scraping public data legal?

It is narrower than that. The Ninth Circuit held that scraping public data was not a CFAA violation, but that ruling covered only the CFAA. It did not touch contract, trespass, copyright, misappropriation, or privacy claims. hiQ ultimately agreed to a $500,000 judgment for breaching LinkedIn's user agreement. Confirm the platform's terms and any GDPR obligations separately.

### Are GitHub noreply commit emails safe to keep since they hide the real address?

No. Noreply addresses in the ID plus username at users.noreply.github.com format still map to a specific account and username, so they identify a person and count as personal data. Treat them the same as any other author email: assign a lawful basis, set a retention limit, and plan the source-notice. Do not classify them as non-personal metadata.

### What record do I need to keep to prove a collection job was cleared?

A standalone Record of Processing Activities row under GDPR Article 30. It should carry the source, access method, fields taken, purpose, lawful basis, collection date, retention period, and deletion process. A DPIA or a general spreadsheet is not a substitute; the Irish DPC expects a producible-on-demand record. The test is whether a second reviewer could reproduce the verdict from that row alone.

---

*From the Refolk guide library. I revise these guides rather than replacing them, so the current version is always at https://www.refolk.ai/guides/repo-data-collection-clearance-standard*