Scoring an Inbound PR: Genuine Work or AI Slop

Every unsolicited pull request or issue that lands in your queue is a disposal decision before it is a review decision. This guide is for maintainers, engineering managers, and developer-relations leads who need to score a stranger's contribution and route it in about two minutes, not admire it. It gives you a graded rubric with named signals, published thresholds, and a four-way decision output, so you stop reviewing on a mood and start reviewing on a score.

The distinction matters. Existing profile and interview guides judge a candidate you already chose to evaluate. Here you are triaging a submission you did not ask for, from someone you may never see again, at a volume that can bury a small team. The job is disposal accuracy at speed.

Why scoring beats reviewing when the queue is AI-generated

The queue breaks on cost asymmetry, not on code quality. The generator can now produce contributions faster than any reviewer can assess them, so the scarce resource is maintainer attention, and the score exists to protect it.

The numbers are stark. Only about 1 in 10 AI-created PRs meets the bar to open a review at all. One maintainer logged 97 slop PRs out of 136 in a single 15-day window, which is 71% of the queue spent on disposal rather than building. For security reports the ratio is worse in a different way: halfway through 2025, curl reported that only around 5% of bug-bounty submissions were genuine vulnerabilities, with roughly 20% appearing to be AI-generated slop. curl ended its six-year bounty programme in January 2026 as a direct result.

Grading code is the wrong frame at this volume. You are not asking "is this the best implementation." You are asking "does this deserve a human's next hour." A rubric that reaches a defensible verdict in two minutes is worth more than a perfect review that arrives after the queue has doubled.

The generator produces contributions faster than the reviewer can assess them, so speed of disposal is the real metric.

The three signal families and what each one proves

Score every contribution on three families: author reputation, effort, and code signals. Detectors converge on these because each proves something different, and each lies in a different way.

Reputation proves whether the author has skin in this repo. It reads account age, prior merges here, and cross-repo spray. Slopper's shipped defaults treat accounts under 30 days as new, a spray score of 60 as spray-and-pray, and 10 PRs in a burst as an activity burst. Reputation lies when a legitimate first-timer looks identical to a spray account on age alone.

Effort proves the author invested before submitting. It reads for a linked issue, a real description, and tests that were not rewritten to pass. Effort lies rarely, which is why it is the most reliable of the three, but a slick AI-generated description can fake diligence for a few seconds.

Code signals prove the diff fits the project. The clearest single tell is comment intent: human comments explain why a decision was made, while AI comments explain what the code does. AI generates surface-level correctness - code that looks right but may skip control-flow protections, misuse dependency ordering, or drift from repo idioms. Code signals lie when the repo enforces an autoformatter, because clean, uniform code then looks machine-made for humans and machines alike.

How AI PRs actually differ from human PRs

AI PRs carry more defects, concentrated in logic and readability, but not enough more to convict any single PR by counting bugs. In a CodeRabbit study of 470 PRs - 320 AI-coauthored and 150 human-only - AI PRs averaged 10.83 issues against 6.45 for human PRs.

Read this table as population evidence, not per-PR proof. The 1.7x mean gap tells you why AI PRs cost more to review across a queue. It does not tell you that the PR in front of you is slop, because a careful AI-assisted human contribution can sit well below the AI mean. Use these ratios to justify a stricter default posture, and use the per-PR passes to decide the individual case.

The false tells hide here too. A widely circulated figure says code-review-agent-only PRs merge at 45% against 68% for human-authored PRs. That gap confounds communication and context with code quality; it is a fact about outcomes, not a signal you can score on a single diff. Treat merge-rate gaps as background, never as evidence against one contribution.

Turn a policy checkbox into a triage branch

A disclosure policy is not a moral stance; it is a triage accelerator that converts a judgement into a branch. When the author declares AI use, you route deterministically instead of guessing.

Named projects have already codified this. MicroPython added a Generative AI Policy and, in PR #18842, a matching PR-template declaration: every contributor now picks "I did not use Generative AI tools" or "I used them, but a human has checked the code." The Linux kernel codified a tag-based approach where AI agents must not add Signed-off-by tags, because only a human can legally certify the Developer Certificate of Origin, and assisted contributions carry an Assisted-by: AGENT_NAME:MODEL_VERSION tag. The strongest rejection trigger across astral-sh/uv, EFF, and OpenInfra is the same: a human must be in the loop who understands and can debug the code; autonomous agents are not allowed.

## AI usage (required)
- [ ] I did not use generative AI tools for this contribution.
- [ ] I used generative AI tools, and a human has reviewed and understands every line.

If you checked the second box, add:
Assisted-by: AGENT_NAME:MODEL_VERSION

The branch is simple. "Used AI, human checked" routes into normal review. Undisclosed AI - discovered later in the code-signal pass - routes straight to quarantine, because non-comprehension you can no longer trust the author to have caught is the actual defect. Disclosure does not make a PR good; it makes your first routing decision fast and defensible.

Score the contribution: the procedure

Run the passes in order, cheapest first, and stop early when two families already agree. The whole pass should take roughly two minutes per contribution.

There is a genuine disagreement about where the destructive action sits. Auto-close tools like Slopper put the close at step 7 by default. SlopGuard and CodeRabbit refuse auto-close and keep a human as the last step. I side with the human-last camp for any repo that wants newcomers, and the reasoning is in the failure modes below.

The gating question: your cheapest high-precision signal

When a case is borderline, ask the author to explain one specific design choice in their diff. This two-minute human probe beats any classifier's in-sample precision, and it is the highest-leverage move in the whole rubric.

The pattern is well documented. You open the code and something is hollow: it calls a function that does not exist, the tests never actually passed, and when you ask "why did you change it this way?" the submitter cannot answer. A genuine author, even a junior one, can explain the intent behind their own change. An author who ran an agent and pasted the output goes silent or answers with generic filler.

Keep the question specific and answerable, tied to a real line in the diff. "Why did you catch and swallow the error on line 40 instead of propagating it?" forces comprehension. "Can you explain your PR?" invites another round of generated text.

Thanks for the PR. Before I review, one question: on <file:line> you
<specific choice, e.g. changed the retry to a fixed 3 attempts>.
What was the reasoning there, and what did you consider instead?

I'll pick this up once I understand the intent behind that change.

A real answer clears the PR into normal review. Silence, or a reply that restates what the code does without explaining why, confirms the quarantine. Either way you have spent two minutes and reached a verdict you can defend to the contributor and to your team.

Finding the handful of maintainers who have actually shipped a policy or a detector is its own sourcing problem. In Refolk's index of professional profiles, US engineers who self-identify with open-source maintenance number only 14, against 287 developer advocates - a roughly 20x gap that shows how thin the maintainer population is and why plain-English search across the public GitHub graph beats keyword filters when you need the exact person who codified an Assisted-by rule.

The triage tools and where they draw the line

Three tools cover the practical range, and they differ most in their posture toward the destructive action, not their signals. Pick the one whose default error matches the error you can least afford.

Slopper scores reputation, behavioral signals, effort, content quality, and duplication via Jaccard similarity to catch recycled issues, and it will auto-close at 9 if you opt in. SlopGuard is quarantine-not-close by design: it scores each contribution, tags provenance, applies a slop-quarantine label, and hands the final decision to a maintainer, never auto-closing anything. CodeRabbit is the least destructive - it notes a slop classification in the PR walkthrough comment and can apply a label, with no merge block.

Read the precision claims skeptically. SlopGuard's 100% precision is on its own hand-labelled golden set of 25 cases, which is in-sample. Its own disclosure records that the first field run scored 0% recall with 3 false positives, and that an older phrase list missed modern agent output entirely. CodeRabbit is openly still gathering data. No published number transfers to your repo untested.

How this goes wrong: the false positives that cost you contributors

The expensive errors are false positives, because a wrongly quarantined newcomer is a contributor you never see again. Every failure mode below is a way the rubric convicts genuine work, and every one has a documented recovery.

• Naming and formatting heuristics false-positive on juniors and non-native-English contributors. Clean, textbook code with generic identifiers looks AI-generated. Check git history and repo-idiom fit across the diff, not one line.
• Detector numbers are in-sample. SlopGuard's first field run scored 0% recall with 3 false positives. Run a threshold sweep on your own recent PRs before trusting any published number.
• Perfect syntax and uniform spacing are a false tell under an autoformatter. If the repo enforces Black or Prettier, a formatter makes human code look machine-made. Discount formatting entirely on formatted repos.
• The 45% vs 68% merge-rate gap is not per-PR proof. It confounds communication and context with code quality. Use it as background posture, never as evidence against one contribution.
• New-account age flags legitimate first-timers as spray-and-pray. Disambiguate with a linked issue and a real description before penalising on age.
• Auto-close is contributor-hostile. Closing a genuine newcomer burns goodwill; quarantine plus one gating question recovers the false positive.
• Passing CI is not a genuineness signal. AI rewrites tests so they pass, so green checks can hide a hollow diff. Invert the check into "did the diff also edit the tests?"
• Large diffs are not automatically slop. An OCaml maintainer rejected a 13,000-line AI PR in November 2025 on review burden, but the real tell is many-files-few-real-additions churn, not raw line count. Measure substantive additions.

The recovery for almost every false positive is the same: do not close, quarantine and ask the gating question. That single discipline converts an unrecoverable error into a two-minute conversation.

When one contribution becomes a queue: escalate structurally

When slop volume outpaces your reviewers, individual scoring stops scaling and you switch to structural throttles. This is a capacity decision, not a per-PR one, and the platform now gives you the levers.

GitHub shipped PR caps: maintainers can set a maximum number of open PRs for users without write access, so once a contributor hits the limit another PR waits until one closes or merges, with trusted contributors on a bypass list. It also shipped two settings to disable pull requests entirely or restrict them to collaborators only. The nuclear options are real precedents - curl ended its bounty and Ghostty banned AI-generated code outright - but treat those as last resorts, because they close the door on genuine newcomers too.

The trigger for escalation is a sustained ratio, not a bad day. One team reported 30 PRs per day split across six reviewers; Node.js took 30-plus AI-slop security reports over a single holiday period. When your slop share holds near the 71% sample figure for more than a week, per-PR triage is no longer the right tool.

Keep the rubric current and verify before you call it done

Run this checklist before you trust your scoring in production, and re-run it whenever models or your project's contribution patterns shift. Detector precision decays as agents change, so a threshold that was calibrated once is not calibrated forever.

Recalibration is the maintenance task that keeps this from rotting. Every few months, pull your last batch of quarantined and cleared PRs, hand-check a sample, and measure how many of your quarantines were genuine work you lost and how many of your clears were slop you reviewed. That two-number audit is the only honest measure of your rubric, and it is the thing no off-the-shelf precision claim can give you.