The Outside-Contributor PR Triage Read: Review, Ask for Proof, or Close

A pull request just landed from an account nobody on the team recognizes, and someone has about five minutes to decide what happens to it. This guide is for the maintainer, engineering manager, or devrel lead making that call. It gives one scored read per PR across five public dimensions, lands each PR on Review, Ask for Proof, or Close with a reason, and sets the bar for the repo's trusted-contributor bypass list.

What does the outside-PR triage read decide?

The read picks one of three outcomes for a pull request from an account you don't recognize. Review means a human reviewer is assigned now. Ask for Proof means the author must answer one specific question before review. Close means the PR is closed with a stated reason and a way back. A separate, slower decision sits on top: whether the author has earned a place on the bypass list that exempts them from the repo's open-PR cap.

The volume problem is real. In LF Research's survey of 234 contributors and maintainers, maintainers estimated that roughly 1 in 4 incoming pull requests is "AI slop" (low-quality AI-generated content). In large projects the share rises to 31%. The survey's exact cutoff for "large" is not confirmed. One secondary summary ties it to projects with more than 50 contributors. Coolify's maintainers were closing more than 120 low-quality AI-generated PRs by hand every month.

Policy has not caught up:

• Only 35% of projects have a policy permitting AI-assisted contributions.
• 32% have no policy and none planned.
• 53% of respondents disagree that the technology has improved trust in their communities.

This read is not an AI detector. It asks whether the author understands the change and stands behind it, and that question matters whether or not a model wrote the first draft. In a matched sample of 567 PRs, agent-authored PRs were accepted 83.8% of the time against 91.0% for human-written ones, with similar review times. Agent output is not junk by default. A read that closes everything that looks generated will close good work and drive off real first-time contributors.

Current public advice tends to be a policy template (disclose AI use, open an issue first) or a bot that prints an account-trust score. Neither tells the person doing the review how to weigh a 42-day-old account with two self-merged PRs against a clean, issue-linked diff. This read does, and the worked example below scores exactly that case.

The five dimensions: what each proves and how it lies

Five public dimensions make up the read: account history, cross-repo PR behavior, issue linkage, diff-to-claim fit, and responsiveness to a human question. Score each one 0, 1, or 2. The first four are static and take under five minutes together. The fifth is asynchronous and is only scored once you have asked.

Account history

Check the account's age, profile, and contribution calendar. Score it by age band:

• 0: under 30 days
• 1: 30 to 90 days
• 2: over 90 days

The 30-day line matches anti-slop's default. Its docs note that raising the gate to 60 or 90 days gives stronger protection but may affect legitimate new GitHub users. Age proves only that the account was not created for this PR. It is wrong in both directions. A genuine first-timer fails it by definition. The JiaT75 account behind the xz backdoor was created January 26, 2021, and its first xz patch arrived October 29, 2021, so it would have passed every age gate in use today. This is the weakest dimension, and it can never close a PR on its own.

Cross-repo PR behavior

Collect three things:

• Merge rate across GitHub: merged PRs as a share of merged plus closed-unmerged. Count only merges made by someone other than the author.
• Merger diversity: the number of distinct maintainers who have merged the author's PRs.
• Burst check: whether the author opened many PRs across unrelated repos in a short window.

Score 2 if the merge rate is at or above 30% and at least 2 maintainers who are independent of each other have merged the author's work. Score 1 if only one of those holds. Score 0 if neither holds, or if you see a burst. The 30% line matches anti-slop's default global merge ratio and the example config for contributor-report. The diversity of 2 matches contributor-report's example. No public tool I found publishes a threshold for histories made only of self-merges. Merger diversity is the nearest proxy, which is why self-merges must be left out.

GitHub's open-PR cap does not help here. It applies per repository, and GitHub itself says the cap does nothing when someone opens pull requests across hundreds of repositories at once. This check stays manual. It is also the slowest part of the read, because you have to work out whether the maintainers who merged someone's work are actually independent of them. Refolk brings back a person's public footprint across GitHub, LinkedIn, and the open web in one request, so you can see employer and org overlap without opening a dozen tabs.

Issue linkage

• 2: the PR links an issue that a maintainer has triaged
• 1: the PR links an issue that nobody has triaged, or one the author opened just before the PR
• 0: no linked issue for a non-trivial change

A triaged issue shows that someone on the team agreed the problem exists. Nobody has published a measured effect of issue-first rules on drive-by PR volume, so treat this as a signal, not a gate. Linkage is easy to fake by opening an issue minutes before the PR. Check the issue's timestamp and who acknowledged it.

Diff-to-claim fit

Read the description first, then the diff.

• 2: the diff does exactly what the description says, with no unrelated files
• 1: the core change matches, but there is drift: formatting churn, stray files, or claims slightly beyond what the code does
• 0: the description claims behavior the diff does not implement, or the diff touches substantial unrelated files

A good fit shows that someone produced a coherent change. It does not show intent. The xz credibility-building patches were clean and did what they said. That is why a perfect fit should always come with a look at who is pushing for the merge.

Responsiveness to a human question

• 2: the answer addresses the specific line or failing case you asked about, and it is correct
• 1: fluent but generic, restating the description or answering a nearby question
• 0: no reply within the stated window, or a wrong answer

This is the only dimension that tests understanding directly. It fails when the question is lazy. A broad question like "can you explain the change?" lets a generic answer score as a pass.

Published trust thresholds you can borrow

Four public tools or rules publish thresholds for some mix of account age, merge rate, merger diversity, and earlier merges in the same repo. None publishes a threshold for histories made only of self-merges or for repos created per day. Use their defaults to calibrate your own judgement, not as verdicts.

Some context on each:

• anti-slop ships 34 rules and runs them all at once.
• contributor-report defaults every threshold to 0, so it does nothing until you configure it.
• pr-slop-stopper scores from -100 to +100. It warns at -10 and closes at -40.
• pr-slop-gate labels at 0.5, comments at 0.65, and closes at 0.85. Auto-close is off by default.

Two of the four exempt anyone who already has a single merge in the repo. That is the most important line in the table. Trust built inside one repo is the weakest kind and the easiest to farm, and the bypass section below comes back to it.

How the read turns scores into an outcome

These cut lines are my proposal. They have not been measured. Add up the four static dimensions for a total from 0 to 8.

• Review: static total 6 to 8, and diff fit is 2.
• Ask for Proof: static total 3 to 5. Also use this when diff fit is 1, or when the static total is high but unfamiliar accounts are pushing for the merge.
• Close: static total 0 to 2, or diff fit of 0 alongside another 0. An account-history 0 counts toward a Close only if another dimension is also 0.

After an Ask, a specific answer (2) moves the PR to Review. A generic answer (1) gets one sharper follow-up, and the PR closes if the second answer is still generic. No reply within the window (0) means Close, with an invitation to reopen.

Worked example. A 42-day-old account with two self-merged PRs opens a clean diff linked to a triaged issue:

• Account history scores 1.
• Cross-repo scores 0, because self-merges count toward neither the merge rate nor diversity.
• Issue linkage scores 2.
• Diff fit scores 2.

The static total is 5, which means Ask for Proof, not Close. If the author answers the line-level question specifically, the PR goes to Review. This is the path a real first-time contributor should take.

PR: #[number]   Author: @[handle]   Read by: @[maintainer]
Disclosure: declared / undeclared / unknowable
Account history: [0-2]   (age band: <30 / 30-90 / >90 days)
Cross-repo: [0-2]   (merge rate excl. self-merges: [x]%   independent mergers: [n]   burst: yes/no)
Issue linkage: [0-2]   (issue #[n], triaged by @[maintainer] / untriaged / none)
Diff-to-claim fit: [0-2]   (unrelated files: yes/no)
Static total: [0-8]
Human question: asked yes/no   answer: [0-2] / pending / lapsed
Commenters pushing for merge checked: yes/no
Open drafts by author in this repo: [n]
Outcome: Review / Ask for Proof / Close
Reason: [one sentence the author will see]
Bypass candidate: no / watch / yes   (merges reviewed by: @[a], @[b])

How to run the read in five minutes

Run the cheap checks that ignore the content first, then rate the diff, then ask your question within the five-minute window. The time budgets below are proposals and have not been measured. Treat them as a pace guide, not a promise.

Sources disagree on the order. pr-slop-stopper skips scoring anyone with a prior merge in the repo, which skips steps 2 and 3. anti-slop runs its 34 checks at once. The Good Egg scorer ignores content entirely. I keep the checks that ignore content first because they are fast, but step 5 always runs. A strong history never excuses a diff that doesn't match its description.

Step 6 behaves differently depending on the band. For a PR in the Review band, the assigned reviewer's first comment does the job. For the Ask band, the question is the whole outcome. For the Close band, skip it and write the reason.

The question has to be one that only understanding can answer. Pick a specific line and an input, state, or failing case that the description does not mention. If the answer is in the PR text, an agent can find it.

Thanks for the PR. Before I assign a reviewer, one question about the change itself:

In [file]:[line], what happens when [specific input or state the description does not cover]? If you can, point to the test that covers it, or describe the failure you would expect without your change.

I'll check back in [N] days. If I haven't heard from you by then I'll close this to keep the queue clear, and you're welcome to reopen with the answer.

Who belongs on the pull request limit bypass list?

Only authors with repeated merges in this repo, reviewed by more than one maintainer, including some substantive diffs. Someone whose only credential is one accepted typo fix does not qualify. The bypass list exempts trusted contributors from GitHub's open-PR limit without giving them full contributor access.

How the limit works

• It is opt-in, and I found no documented default value.
• It applies only to users without write access.
• Only open, non-draft pull requests count.
• PRs opened by Copilot or another AI agent count toward the limit.
• Bypass exempts someone from the limit but does not grant write access.
• A separate setting can disable pull requests entirely or restrict them to collaborators.
• A third-party PR reports that the limit has no REST API. Check this before you script anything against it.
• GitHub describes two features as planned: a setting that counts drafts, and automatic bypass based on trust signals such as account age, number of merged PRs in the repository, and organization status.

Those planned automatic-bypass signals are the cheapest ones to fake. Account age can be waited out, and merges in one repo can be farmed. If automatic bypass ships, set it to require independent reviewers, not just a merge count.

Track record is a legitimate signal, which is exactly why it attracts gaming. In one study, agent PR acceptance was 37% when the repository had never seen the agent before, rising to 70% after 20 or more prior merges. Maintainers already lean on history. The job is to test whether that history is independent.

Before adding someone, look at where their merges came from. If you want to calibrate what a real multi-maintainer history looks like in a large project, pull a reference set of people who have one:

How the read goes wrong: failure modes and false positives

The read fails in seven known ways. Most of them either punish a genuine newcomer or let a patient attacker through. Each one below comes with the check that catches it.

1. Account age punishes genuine first-timers. A sparse new account can trip enough default checks to be closed with no AI involved. First-timers are not the problem: in one study of security-related AI PRs, first-time contributors had higher odds of being merged (odds ratio 4.27). That sample is narrow, so don't generalize the number, but don't assume the opposite either. Check: never close on age alone. Require a second negative signal.
2. Merger diversity gives a false negative. Good Egg scores Guillermo Rauch MEDIUM on Next.js, his own company's repo, because he has zero merged PRs there. Owners and staff often have thin PR histories in their own repos. Check: look at merges in other repos by maintainers who are independent of each other.
3. The bypass list becomes a backdoor. One merged typo turns off Actions approval permanently. The xz attacker spent about six months on harmless patches, from October 2021 until the first pressure email on April 22, 2022. Check: require merges reviewed by more than one maintainer and some substantive diffs before bypass.
4. "Independent" reviewers are sock puppets. Approvals or pressure comments come from new accounts with little footprint. In the xz case, the accounts named Jigar Kumar and Dennis Ens pushed for the maintainer to hand over control, and Jigar Kumar "is never seen again." Check: score the commenters, not just the author.
5. The human question gets an agent's answer. A fluent, generic reply is the false positive here. Check: ask about a specific line or a failing case that the description does not mention, and score a generic answer as 1, not 2.
6. Drafts slip past the cap. Drafts don't count toward the open-PR limit, so an author can keep work in flight without ever reaching it. Check: sort the author's open drafts and log the count.
7. "Review" turns into an unreviewed merge. In a hand inspection of merged agentic PRs, most showed no feedback loop at all. Check: Review must mean a named human actually commented before merge.

Failure modes 3 and 4 share a lesson. Both xz signals sat outside the diff. The patches fit their descriptions, and the account was old enough. What gave the attack away was who was pushing for the merge and how little history those accounts had.

Who sits in the triage chair?

Mostly developer relations staff and senior engineers, not people with open-source job titles. In Refolk's index of professional profiles, open-source titles barely exist as a hiring category. That makes the read's main risk inconsistency between reviewers, so write it down.

The US developer relations figure may be capped at 300, so the ratio is at least 4.9x and could be higher. The same pattern holds in another market:

Berlin accounts for 9 of the 25 profiles in the German sample. The claim that triage falls mostly to developer relations and staff engineers is my inference from these counts. It has not been measured directly. It matters because large projects see more slop (31% against 25% overall), and the people triaging that volume are rotating through it alongside other work. Three practical consequences follow:

• Write the read down. The log line and the Ask comment are the parts that survive a handoff.
• Calibrate across reviewers. Have two people score the same handful of PRs independently and compare where they land.
• Staff the rotation from developer relations titles. If you are hiring or borrowing reviewers, searching for open-source titles will turn up mostly founders. Refolk lets you ask for developer advocates who maintain or review community repos by name and place, which is the pool that actually does this work.

Keeping the read current

The read depends on three things that change without warning: GitHub's PR controls, tool defaults, and disclosure conventions. Re-check each one whenever you review the triage policy, and note the date you checked in the policy file.

• GitHub's limit and bypass settings. Planned features, including counting drafts and automatic bypass from trust signals, would change steps 3 and 7. Check the changelog and the repository's interaction-limits settings directly rather than relying on this guide's description.
• Tool defaults. anti-slop, contributor-report, pr-slop-stopper, and pr-slop-gate all publish their thresholds in their docs. If you borrow a cut line, re-read the source before you change your own.
• Disclosure conventions. 76% of projects with policies mandate disclosure, and 76% require contributors to personally test and review AI-assisted code. The kernel's template is being withdrawn. Keep disclosure as context so the score does not depend on a trailer.
• The bypass list itself. Review it on a schedule. Remove anyone whose recent merges were all reviewed by a single maintainer.