The Ecosystem-CI Admission Standard: Admit, Probation, or Drop
Grade any downstream repo against six written criteria, reach the same Admit, Probation, or Drop verdict another maintainer would, and re-grade the suite every quarter.
Key takeaways
- Opt-in ecosystem-CI suites hold about 20 entries, so one flaky slot costs far more trust than it does in Crater, which tests 44k projects and expects spurious noise.
- A downstream suite is only green against the current release if it ran more than zero tests, because example scripts have exited with code 0 while an example failed.
- Comparing a PR run with the latest scheduled run tells you nothing when the scheduled run is already red, so require a green scheduled baseline before blaming upstream.
- Google measured flakiness at 0.5% of small tests and 14% of large tests, and ecosystem suites are large integration tests, so plan for the high end.
- No ecosystem-CI project publishes a numeric flake threshold, so this standard grades by attribution: three consecutive downstream-attributed reds drop a suite, three consecutive green scheduled runs admit one.
- In Refolk's index only 3 of 849 US Release or Build Engineers list GitHub Actions as a skill (0.35%), so find suite owners by title and GitHub activity, not skill tags.
This standard is for maintainers of a widely used library who run downstream repositories' tests against unreleased builds and have to decide which repos earn a slot and which to evict. It gives six gradeable criteria, a verdict rule that returns Admit, Probation, or Drop, an eight-step admission procedure, and a quarterly re-grade checklist. Two maintainers grading the same repo with it should land on the same verdict.
What an ecosystem-CI slot is, and why admission needs a rule
An ecosystem-CI slot is one downstream repository whose own tests your runner clones, rebuilds against your unreleased version, and runs. Admission needs a written rule because an opt-in suite is small, so every flaky, rotted, or redundant slot takes a visible share of the team's trust in the whole suite.
The mechanics are well documented elsewhere, so I will only fix the vocabulary:
- Suite - one downstream entry. In Vite's ecosystem CI, each suite is a TypeScript file that calls
runInRepowith a repo, a tracked branch, package overrides, a build script, an optional beforeTest step, and a test script. - Override - the package-manager mechanism that forces your build into the partner's install. Vue's runner "will use pnpm override to force install the specific version of Vue in the downstream projects and then run their tests", and sets an
ECOSYSTEM_CIenvironment variable so the partner can tell it is running. - Upstream selector - how you choose which of your builds to test. Vite's runner takes
--tag,--branch, or--commit, and with--releasethe build is skipped and your package is fetched from the registry instead. - Scheduled run - a periodic run of every suite against your main branch, used as the comparison point for PR runs.
- Flake - a red result that does not reproduce when re-run against the same upstream commit.
Rust's Crater shows the other model. It has no opt-in: it runs cargo check, build, and test plus some tools across crates from the registry and GitHub, and compares the results with the latest stable. Crater grew from 21k tested projects to 44k, and it accepts spurious failures as a cost of that scale, handling them with triage and a blacklist.
Opt-in suites sit at about 20 entries. At that size you cannot absorb noise statistically. One suite that goes red every other week becomes the reason people stop reading the table, and a suite nobody reads protects nothing. That is the case for a rubric with an eviction rule.
The six admission criteria, and what each proves
A downstream repo earns a slot by passing six criteria: a runnable entry, a real green baseline, an override that holds, stable results, a responsive owner, and coverage no other suite already provides. Each one proves something specific, and each has a characteristic way of lying.
| Criterion | Passes when | What it proves | How it lies |
|---|---|---|---|
| Runnable entry | Tests run after a clone, under the override | The runner can exercise the repo unattended | Passes locally only with a browser or network the runner lacks |
| Real green baseline | Green on current release, test count above zero | A later red is news | Script exits 0 with zero tests run |
| Override holds | Job logs the resolved upstream version | You tested your build | A third tool pins its own copy |
| Stability | Every red attributed; finishes inside its timeout | A red means something | Compared against an already-red scheduled run |
| Responsive owner | Named person answers before the next scheduled run | Downstream breakage gets fixed | A team handle nobody watches |
| Distinct coverage | Names an upstream surface no admitted suite claims | The slot adds signal | Statement lists surfaces the tests never touch |
Runnable entry
The partner's test script must run after a plain clone with the package manager your runner supports. Vite's tooling says its utilities "focus on pnpm based projects" and asks partners to consider switching. Solid's partner PR is the model to copy: its script "needs no browser, no network and no example installs, so vite-ecosystem-ci can run it after it replaces Vite." If the script needs a browser, the entry must install it in beforeTest, as Vike's entry does with pnpm exec playwright install chromium.
Real green baseline
Run the suite against your current published release, not your main branch. If it fails there, any later red is uninterpretable, because you cannot tell whether your change or the partner's existing breakage caused it. The pass must include a test count above zero.
Override holds
The job must print the upstream version it actually resolved. If a third tool in the partner's stack brings its own copy of your package, you are testing the registry release and calling it your branch.
Stability: flake and runtime
Every red in the grading window is attributed to upstream, downstream, or the runner, and the suite finishes inside its own timeout-minutes. On GitHub-hosted runners each job can run for up to 6 hours, and a job with no timeout-minutes inherits the default ceiling of 360 minutes. Self-hosted runners allow 5 days. No project has published a per-suite duration it will tolerate, so set a timeout for each suite when you grade it, write it in the record, and count any timeout as a stability event.
Responsive owner
A named person on the partner side answers a failure ping before the next scheduled run. This is gradeable without a stopwatch: either the reply came before the next run or it did not.
Distinct coverage
The partner writes a coverage statement naming the upstream surfaces their tests exercise, and at least one of those surfaces is claimed by no admitted suite. Solid's statement is the model: its script "checks the built plugin against the Vite version installed in the repository: dev client and SSR transforms, HMR runtime injection, a client build, an SSR build." Nobody has published a method that uses import or call-site analysis against upstream exports to decide admission, so this criterion rests on declared coverage. Treat it as a claim you spot-check, not a measurement.
Verdict rules: Admit, Probation, or Drop
The verdict follows mechanically from the six criteria: Admit needs all six, Probation means the suite runs cleanly but has not yet earned trust, and Drop means a hard criterion failed or the suite proved itself noisy or redundant. Runnable entry, real green baseline, and override holds are hard criteria; the other three are measured over time.
Admit when all of these hold:
- The three hard criteria pass.
- The suite has three consecutive green scheduled runs.
- No red in the grading window is unattributed, and none hit the timeout.
- The owner answered the most recent ping before the next scheduled run.
- The coverage statement names at least one surface no other admitted suite claims.
Probation when the hard criteria pass but any of these is true:
- The suite is new and has fewer than three consecutive green scheduled runs.
- One measured criterion failed for the first time in this window.
- The coverage statement is missing or still under review.
On probation the suite runs on schedule, but its red does not block a release or open an upstream investigation until someone triages it.
Drop when any of these is true:
- A hard criterion fails and the owner has not fixed it by the next re-grade.
- Three consecutive scheduled reds are attributed to the downstream repo.
- The suite fails probation twice.
- Every surface in its coverage statement is already claimed by an admitted suite.
The three-in-a-row rule is borrowed, not invented: Google can report a flaky test as failed "only if it fails three times in a row". I apply the same count in both directions so admission and eviction are symmetric. A team may raise the count, but write the new number into the grading record so every grader uses it.
Reliability against distinct coverage
Where candidate repos come from
Candidates come from three public sources - the GitHub dependents list, deps.dev dependents, and code search - and each one misses consumers the others catch. Tag every candidate with its source and treat the combined list as a sample, never a census.
| Source | What it counts | What it misses |
|---|---|---|
| GitHub dependents | Public repos depending on your repo or packages | Private repos, repos with the graph off; counts approximate |
| deps.dev dependents | Packages on npm, Cargo, Maven, PyPI | Repos, private dependents; counts are per version |
| Code search | Literal imports in public code | Vendored and monorepo consumers, coverage unverified |
GitHub says plainly that dependents "are other public repositories that depend on the repository or on packages that it publishes" and that the information "is not reported for private repositories." It also warns that "the dependent counts are approximate and may not always match the dependents listed." The "Used by" sidebar only appears once more than 100 repositories depend on your package.
The list has also become opt-in on the dependent's side. Public repositories can now turn the dependency graph off, and a repo that does drops out of your count. One maintainer found a repo now "requires explicitly enabling the dependency graph" to show up, and Streamlit reported a 95% decline in its used-by growth after the change.
deps.dev counts packages, not repositories, and "only public dependents are counted." Its website shows "a precomputed sample of the dependents", and one tool found its own figure wrong by a factor of 680 because counts are per version. Query both the default version and the latest stable before you trust a number.
Who owns partner suites, and how to find them
Every slot needs two people: an owner on the partner side who keeps the test script working, and someone upstream who triages reds. Both are scarce, and searching by skill tag finds almost none of them.
Refolk's index gives a sense of the supply of people whose job title is the work itself:
| Slice (Refolk's index) | Profiles | Note |
|---|---|---|
| US, Release or Build Engineer title | 849 | 36.9x Germany |
| US, title plus GitHub Actions skill | 3 | 0.35% of the title pool |
| Germany, Release or Build Engineer title | 23 | Baseline for the ratio |
In a 25-person sample of the US pool, the top employers were Apple (9), Google (4), and IBM (3). The German pool includes people at NVIDIA (2), JetBrains, SUSE, and Parity Technologies.
The practical consequence: search by title and by what people have done on GitHub, not by self-reported skills. The people who already understand ecosystem CI are the ones who have opened PRs to an ecosystem-CI repo or maintain a test script written for one. Refolk takes that as a plain-English request and returns matching people from GitHub, LinkedIn, and the open web, which is faster than reading PR histories by hand.
A limit on these figures: they count job titles. Partner owners are often library maintainers with no release-engineering title at all. For a specific candidate repo, the author of the prerequisite PR on the partner side is usually your owner, and the person to name in the grading record.
Admitting a candidate, step by step
Admission takes eight steps. The first four take days and need the partner owner for step 2; probation in step 5 takes weeks of scheduled runs.
Ecosystem-CI admission procedure
- Build a source-tagged candidate listPull GitHub dependents, deps.dev direct dependents, and code search results, and tag each candidate with its source. Done when one list exists with a source on every row.
- Confirm partner prerequisites with the ownerThe owner confirms a test script that runs after clone, uses the supported package manager, accepts the override, and names a tracked branch. Done when the script passes locally under the override.
- Write the suite entryAdd a runInRepo-style entry with repo, branch, build, optional beforeTest, and test script, then regenerate the workflow. Done when the entry is merged.
- Prove a real green baselineRun against the current release fetched from the registry. Done when it passes with a test count above zero and the job prints the resolved upstream version.
- Run probation on scheduled runsSet and record the suite's timeout-minutes and compare each run with the latest scheduled run. Done at three consecutive greens or a downstream-attributed red.
- Check coverage overlapCompare the partner's coverage statement with every admitted suite's statement. Done when at least one surface is unclaimed or the candidate is marked redundant.
- Attribute every red before counting itClassify each red as upstream, downstream, or runner, record kill signals, and re-run timeouts locally. Done when no red in the window is unattributed.
- Grade and record the verdictApply the verdict rules and write Admit, Probation, or Drop in the grading record with date and evidence. Done when the record is committed and dropped suites are out of the workflow.
Notes on the steps:
- Step 2 is the partner's work, not yours. A partner PR to Vite's runner listed what was needed: a runInRepo file, running
pnpm workflow:generate, and a choice of branch to track. Vite's own guidance is to copy an existing test and add the suite to the workflow lists only "once you are confident the suite works." - Steps 3 and 4 have no fixed order in the sources. I run the baseline from a branch before merging so a broken entry never lands, but either order works if step 4 passes before probation starts.
- Step 5 can be sped up on demand. In Vite, a user with triage permission can comment
/ecosystem-ci runon a PR, or name a single suite, to trigger a run outside the schedule. On-demand runs help attribution but do not count toward the three consecutive scheduled greens. - Step 6 has no documented industry practice. Declared coverage is the best available instrument; spot-check one claimed surface by reading the partner's tests.
Use this message for step 2 and step 6 together, so the owner sees the whole ask at once:
Hi <owner name>, I maintain <library>. I would like to add <partner repo> to the ecosystem CI suite that runs your tests against unreleased builds of <library>, so I catch breakage before you do. To earn a slot, a suite needs: 1. A test script that runs after a plain clone, with no browser, network, or example installs unless the entry installs them first. 2. The supported package manager (<package manager>), so the version override applies. 3. A branch for the runner to track: <branch>. 4. A named owner who answers a failure ping before the next scheduled run. One more question: which parts of <library> do your tests exercise? A short list is enough, for example "dev transforms, SSR build, plugin hooks". I use it to make sure each suite covers something the others do not. New suites start on probation and are admitted after three consecutive green scheduled runs. The full standard is here: <link to your grading policy>. Thanks, <your name>
Replace the angle-bracket fields; keep the four prerequisites and the coverage question unchanged so every partner gets the same ask.
Measuring flake and runtime without a published threshold
No ecosystem-CI project has published a numeric flake-rate threshold or a tolerated per-suite runtime, so this standard grades stability by attribution and consecutive runs rather than a percentage. The published flake research sets expectations for how noisy these suites will be.
| Test size (Google) | Flaky share |
|---|---|
| Small | 0.5% |
| Medium | 1.6% |
| Large | 14% |
The wider picture points the same way. Google reported that about 1.5% of all its test runs returned a flaky result, almost 16% of its tests had some flakiness, and 84% of pass-to-fail transitions involved a flaky test. A study of GitHub projects found one in eleven commits (9%) had at least one red build caused by a flaky test. Crater's triage guide says test failures in its runs are "mostly spurious."
How to attribute a red
Vite's bot posts a per-suite table with the columns suite, result, and latest scheduled, comparing each PR run with the last scheduled run. That comparison is the core attribution tool, with one condition covered in the failure modes: it only works when the scheduled run was green. Crater's guidance adds two habits worth adopting: re-run timed-out tests locally, and when one root dependency breaks, triage it once and ignore its fallout in everything downstream of it.
Work through a red from the outside in. Only the innermost layer is evidence about your change.
Layers of a red suite result
- Install and overridePostinstall fails or the wrong upstream version resolves
- Partner buildThe downstream build breaks before any test starts
- Runner killTimeout or out-of-memory signal, recorded rather than guessed
- Test countZero tests ran, so pass or fail is meaningless
- Test assertionsThe only layer where a red is evidence about upstream
If your team wants a percentage ceiling anyway, choose it explicitly, write it into the grading record, and say in the policy that it is a house rule. I do not supply one, because no published figure supports any particular value.
How this goes wrong: failure modes and false positives
Most grading errors come from trusting a signal that looks like evidence but measures something else: a green that ran nothing, a comparison against a red baseline, or a dependents count treated as a census. Each failure mode below comes with what it looks like and the check that catches it.
A green that ran zero tests
Looks like: the suite passes on the current release. Actually: the script exited 0 without exercising anything. Solid's old example script "exited with code 0 when an example failed." Check: assert in the job that the test count is above zero, and fail the run if it is not.
Attribution against a red baseline
Looks like: the PR column and the latest-scheduled column both fail, so the PR is cleared. Actually: when main is already red, the comparison cannot separate upstream breakage from downstream breakage. One Vite PR showed both columns failing across 8 suites at once. Check: require a green scheduled baseline before blaming or clearing upstream. If main is red, fix main first.
Out-of-memory and timeout masking
Looks like: a crash filed as spurious and ignored. Actually: the kill hid a real regression. Crater auto-files timeouts and out-of-memory kills as spurious, and in one case picked up only 2 of 21 genuine regressions because the kill took the rest. In another run, 19 crates "always die in crater due to what looks like OOM." Check: record the kill signal on every red. A suite that is killed on most scheduled runs is a Drop candidate, because it can never show you a regression.
Override leakage through a third tool
Looks like: an install failure blamed on your build. Actually: a tool in the partner's stack carries its own copy of your package. In Vite's vinext suite, "installation fails in postinstall before tests can run" when vinext uses Vite+. Check: assert the resolved upstream version inside the job, before tests start.
Near-green results counted without attribution
Looks like: a suite that is basically fine. With the override applied by hand, vinext showed 574 passed, 3 skipped, and 1 failed. Actually: one unattributed failure is still unattributed. Check: grade it like any other red: classify it, then apply the rules.
Overcounted failure lists
Looks like: a regression report naming every affected project. Actually: the list includes bystanders. In one Rust beta Crater run, "Not all of the 156 crates listed are affected by this. Only 64 of them had the 'No such file or directory' error." Check: confirm the specific error in each suite's log before charging it to the regression.
The dependents list as a census
Looks like: the full set of candidates. Actually: private consumers, repos that turned the dependency graph off, and vendored copies never appear, and GitHub calls its own counts approximate. Check: combine three sources and tag each candidate, as in the discovery section.
deps.dev counts read once
Looks like: a precise dependents number. Actually: counts are per version, and one tool found its figure wrong by a factor of 680. Check: query the default version and the latest stable, and use the larger list.
Skill-tag sourcing for owners
Looks like: a filter for "GitHub Actions" finds your CI people. Actually: in Refolk's index only 0.35% of US Release or Build Engineers carry that tag. Check: search by title and by GitHub activity instead.
A suite nobody reads protects nothing, so evicting a noisy slot is maintenance, not a loss of coverage.
Keeping the suite current: the quarterly re-grade
Re-grade every admitted and probation suite each quarter against the same six criteria. A suite keeps its slot by passing again, not by having passed once.
Suites rot quietly. Partners change package managers, switch tracked branches, add tools that bring their own copy of your package, or lose the person who wrote the test script. Crater handles this with a blacklist and a bot that asks, "If you notice any spurious failure please add them to the blacklist!" At about 20 slots you can afford to look at each one by hand, so do it on a calendar rather than waiting for the table to go red. No project has published how long it takes between a suite's first failure and its removal, which is one more reason to set your own cadence and write it down.
When an owner goes quiet, the suite does not have to die with them. Check who has been committing to the partner repo recently and ask them to take over; Refolk can return the recent active committers on a named repository with their profiles, which turns a dead ping into a new named owner. If nobody takes it on before the next re-grade, the owner criterion fails and the verdict rules decide the rest.
For a peer example of written add and remove rules, the vite-plus-ecosystem-ci catalog keeps an ecosystem.json list and a TESTING.md describing when suites are added and removed. Compare its rules with yours when you revise this policy.
Keep one record per suite in the repo next to the suite file, so the grade and its evidence travel together:
Suite: <owner/repo>
Tracked branch: <branch>
Graded on: <date> Grader: <name>
Source tags: <github-dependents | deps.dev | code-search>
HARD CRITERIA (any fail = cannot be Admitted)
[ ] Runnable entry: tests run after clone under the override
[ ] Real green baseline: passes on current release, test count = <n> (must be > 0)
[ ] Override holds: resolved upstream version logged = <version>
MEASURED CRITERIA
[ ] Stability: consecutive green scheduled runs = <n> (Admit needs 3)
Unattributed reds in window = <n> (must be 0)
timeout-minutes = <value> Timeouts hit in window = <n>
Downstream-attributed reds in a row = <n> (3 = Drop)
[ ] Responsive owner: <name>, last ping answered before next run? <yes/no>
[ ] Distinct coverage: surfaces claimed = <list>
Surfaces no other admitted suite claims = <list> (must not be empty)
Probation count to date: <n> (second failed probation = Drop)
VERDICT: <Admit | Probation | Drop>
Evidence links: <run URLs>One file per suite, committed beside the suite entry; fill every field at each grade so the verdict can be re-derived from the record alone.
Quarterly re-grade checklist
- Every suite in the workflow has a grading record dated this quarter.
- The latest scheduled run of main is green before any suite is graded on attribution.
- Each suite's baseline run on the current release shows a test count above zero.
- Each job log prints the resolved upstream version, and it matches the build under test.
- Every red since the last re-grade is classified as upstream, downstream, or runner, with kill signals recorded.
- No suite has three consecutive downstream-attributed scheduled reds still sitting in the workflow.
- Every suite has a named owner who answered their last ping before the next scheduled run.
- Every admitted suite still claims at least one upstream surface no other admitted suite claims.
- Every suite's timeout-minutes is set explicitly and recorded, not left at the 360-minute default.
- Suites that failed probation twice have been removed from the workflow, and their owners told why.
- The candidate list has been refreshed from all three discovery sources and new candidates have been sent the partner request.
Questions practitioners ask
How do I set up ecosystem CI for my library?
Write one entry per downstream repo that clones it, forces your unreleased build in through a package-manager override, and runs the partner's own tests. Vite's runner does this with a runInRepo call naming the repo, tracked branch, build script, optional beforeTest step, and test script. Vue's runner uses a pnpm override and sets an ECOSYSTEM_CI environment variable so partners can detect the run. Wiring it up is the easy part. Deciding who gets a slot is what this standard covers.
Which dependents should I run against a release candidate?
Run the ones that pass all six criteria in this standard: a runnable entry, a real green baseline with more than zero tests, an override that holds, stable results, a responsive owner, and coverage of an API surface no other suite claims. Popularity is not a criterion. A heavily used dependent that exercises the same surfaces as a suite you already run adds runtime and flake risk without adding signal.
How do I remove a flaky project from ecosystem CI without a fight?
Agree on the eviction rule before anyone's suite fails. Under this standard a suite is dropped after three consecutive scheduled reds that triage attributes to the downstream repo, or if it fails probation twice. Because the rule is written down and graded from the record, the removal is a lookup, not a judgement call. Tell the owner what evidence triggered it and what would earn re-admission.
What flake rate is acceptable for a downstream suite?
No ecosystem-CI project has published a numeric flake threshold, so any percentage you pick is your own policy, not an industry figure. The published reference points are Google's: 0.5% of small tests, 1.6% of medium, and 14% of large tests showed flakiness. This standard avoids a percentage and grades by attribution and consecutive runs instead. If your team wants a number, write it into the grading record so every grader uses the same one.
Is the GitHub dependents list enough to find candidates?
No. GitHub reports dependents for public repositories only, says its counts are approximate, and drops any public repo that has turned its dependency graph off. One project reported a 95% decline in its used-by growth after that option appeared. Combine it with deps.dev dependents and code search, tag each candidate with its source, and treat the result as a sample, never a census.
Try it on the search you came here for
Stop building boolean strings. Just describe the person.
Type one sentence. I plan the search, read GitHub, public LinkedIn and Crunchbase records, and the open web as it is right now, and hand back a ranked list with the reason next to every name.
01Describe them
One plain sentence. Role, city, stack, stage, whatever matters to you.
02I read the web live
GitHub, public LinkedIn and Crunchbase records, the open web. Not a database that went stale last quarter.
03You read the shortlist
Ranked, with the reasoning under every name. Open a profile, ask a follow-up, narrow it down.
- Staff backend engineers in NYC who shipped Rust in production
- Series A fintechs in SF under 50 people, growing headcount this year
- Maintainers of fast-growing Rust web frameworks on GitHub
- No boolean, no filters, no seat to buy. One box.
- Read at search time, so a profile updated yesterday counts today.
- Every step visible as it runs, every name with its reason.
500 free credits on sign-up. No card, no demo call. See real searches.