The Bellwether-Repo Score: Track It, Corroborate It, or Drop It

You are mapping a technology category and need one open-source project you can trust as a proxy for the whole category's trajectory. This guide gives strategy, research, and talent-intelligence teams a weighted rubric for that upstream decision: score any candidate repository on defined dimensions, then classify it as a primary market indicator, corroboration only, or a drop. It is built on the 2024-2026 fake-star measurement work rather than the star-count shortcut that most repo guides still lean on.

The judgement is repeatable and it is reviewable. By the end you will have a table you can defend in a leadership review, where every cell traces to a signal that proves something specific and fails in a way you have already named.

Why a star count cannot anchor a category read

Stars are the most manipulated and least usage-linked signal a repository exposes, so no market map should rest on them. In a study by researchers at Socket, Carnegie Mellon, and NC State, the StarScout tool over GHArchive flagged roughly 4.5 million suspected inauthentic stars from 1.32 million accounts across 22,915 repositories. Filtering to a defensible core - repositories with a single-month anomalous spike where fakes exceeded 10% of stars - left 3.1 million fake stars given by 278,000 accounts to 15,835 repositories. The ICSE 2026 version of the work reports six million suspected fake stars.

The concentration matters more than the raw total. In July 2024, about 15.8% of repositories with more than 50 stars were caught in these campaigns, with a peak of 3,216 repositories and 30,779 participating users in that single month. If you pick a proxy on star rank alone during a window like that, roughly one in six star-heavy candidates is a manipulated read.

There is a deeper reason to distrust stars beyond fraud. The Linux Foundation critical-projects methodology warns that stars are a biased signal because not all projects are hosted on GitHub, so a star-ranked category read silently drops anything living on other forges. Published work favors dependency and download signals instead: forks correlate with usage and offer a proxy for adoption, and there is a positive correlation between a Python package's centrality in dependency networks and its download statistics. Lead with usage. Treat stars as the last column you check, and only after the screen below.

The dimensions that decide a bellwether

A trustworthy category proxy has to clear five dimensions, each measuring a different failure. The score is not a single number you average blindly; it is a set of gates, where two of them can veto a repo no matter how well it does elsewhere.

The five dimensions, and what each one proves:

Star authenticity and governance neutrality are gates. A repo that fails the fake-star screen is demoted before it is scored on anything else, and a repo whose trajectory is one company's roadmap is capped at corroboration regardless of its usage numbers. The other three dimensions are weighted contributors: usage carries the most weight because it is the signal published work trusts most, category share establishes that the repo is representative rather than a fringe player, and labor demand corroborates timing.

Running the fake-star screen as a solo analyst

You can approximate the published detectors without a data-engineering team by copying their two core signatures and one solo-analyst adaptation. StarScout uses a low-activity signature, which catches stars from accounts that go stale after starring one or a few repositories, and a lockstep signature, which catches clusters of accounts repeatedly starring the same cluster of repositories in short time windows.

A follow-on study operationalized a version a single analyst can run. It uses three signals and flags any starring event that trips two or more of them as anomalous, a threshold chosen to balance sensitivity against false positives:

• Account age: the account was seven days old or younger when it starred.
• Empty profile: zero of five profile fields were filled.
• Temporal burst: 50 or more stars went to the same repo in the same clock hour.

There is a second, almost free check. The baseline repository deletion ratio for non-anomalous accounts is about 5.03%. In the flagged set, roughly 91% of repositories and 62% of suspicious accounts had been deleted by October 2024. If a repo's stargazer cohort deletes far above 5%, that is the cheapest single tell you have that the attention was inorganic.

For each stargazer in a suspicious spike month, mark it if 2+ of:
  [ ] account age <= 7 days at time of starring
  [ ] 0 of 5 profile fields filled
  [ ] part of a 50+ stars-to-this-repo-in-one-hour burst
Spike test: does any single month's stars exceed 10% of total,
  and is >10% of that month's stars marked above?
Decay test: did the star curve fall back within ~2 months of the spike?
Deletion test: is the stargazer deletion rate well above the 5.03% baseline?
Verdict: two or more tests positive -> DEMOTE the star signal for this repo.

Do not over-fire. Below 50 stars the detector is noisy, and a legitimate niche repo can trip a single signal. The published cutoff demotes only on the >10% single-month spike, so treat one lonely signal as noise, not evidence.

Reading usage the market actually pays for

Off-platform usage is the dimension that separates a proxy from a popularity contest, because it counts installs and dependents rather than attention. Package registry requests are the fundamental measure of how lively an ecosystem's usage is: NPM served 4.5 trillion download requests in 2024, up 70% year over year, and PyPI served 530 billion, up 87%. Those volumes make per-repo download trends a usable signal once you control for automation.

Four downstream signals, drawn from an adoption benchmark, give a usage read independent of stars:

• Registry monthly downloads, for example PyPI or npm.
• GitHub dependent-repository counts.
• Dependent-package counts from a dependency graph.
• Code-search hits in dependency files such as requirements.txt and pyproject.toml.

Concentration in these signals is real and worth measuring. One adoption study found the top five agent frameworks account for over 93% of category downloads, with LangChain alone at 233 million downloads per month. When a category is that top-heavy, the ranking question is nearly settled by usage before you touch governance.

Refolk lets you corroborate that usage picture from the practitioner side without scraping profiles by hand. Where downloads tell you machines are pulling a package, the human bench tells you who has actually shipped with it.

Two guards keep usage honest. High registry numbers can be automated CI or mirror pulls rather than humans, so compare the download trend against dependent-repo growth: divergence signals bot traffic. And fork count rises from course assignments and template copies, so check the ratio of forks carrying independent commits against empty forks before you read forks as adoption.

Governance concentration: proxy or vendor telemetry

A repository whose commits come overwhelmingly from one employer is a vendor telemetry feed, not a category proxy, because its trajectory follows one firm's roadmap. The standard measure is the bus factor: the smallest number of people who make 50% of contributions. Pair it with the single largest employer's commit share, and the picture is unambiguous.

All three are heavily single-company on commit share, but bus factor tells you how brittle that is. Transformers, at a bus factor of 5, concentrates half its commits in five people inside one firm - so its trajectory tracks that firm, and it caps at corroboration only no matter how strong its stars look. PyTorch and TensorFlow spread the same 80% across far more contributors, which is healthier but still single-vendor.

Governance structure is the mitigant. Single-vendor risk is a company-led approach where one organisation retains primary control over governance, roadmap, and licensing. Establishing a Technical Steering Committee with multiple organisations represented and documented voting prevents single-vendor dominance even where a company started the project. So the governance flag has two inputs: the numbers above, and whether a neutral steering body actually exists.

The OpenSSF Criticality Score offers reusable componentry here without asking you to invent a formula. It outputs a 0-1 importance score from a project's age, its number of individual contributors and organisations, user involvement in new issues and updates, and a rough dependency estimate. You do not have to adopt its exact weights, but its inputs are a sound starting basket for the share and governance columns.

Computing category share without biasing the denominator

Category share tells you whether a repo carries a meaningful slice of the category, and the only defensible way to compute it is against a fixed basket using one consistent source per column. There is no single published formula for share of category, so this is method by analogy: for your defined basket of category repos, express each repo's contributors, commits, and opened-issue counts as a percentage of the basket total.

The Linux Foundation methodology is explicit about the trap: dependency information and issue activity are not consistently measured across projects. If you count GitHub issues for one repo and registry dependents for another, your denominator is meaningless. Pick one source per column and apply it to every repo in the basket.

Category concentration is normal and should not alarm you when it appears. A single dominant repo taking a large share is common in fast-moving categories, as the top-five agent frameworks accounting for over 93% of downloads shows. What you are guarding against is not concentration itself but a share computed on mismatched sources that flatters or buries a repo by accident.

Weight the human bench, not just the code

Practitioner depth is a corroborating dimension that often contradicts the tidy narrative a star chart tells, because talent concentrates far harder than category rhetoric implies. Consider two frameworks that get framed as a two-horse race:

In Refolk's index of professional profiles, the United States holds 18,409 LangChain professionals against 900 for LlamaIndex - a 20.5x gap. Weighted by practitioner base, the "two-horse category" framing collapses, and the smaller repo is corroboration at best in a US market map. A star or download chart that shows them closer together is measuring attention, not who can actually build.

Geography multiplies the risk, so representativeness has to be scored per region:

That 7.2x US-to-Germany footprint means a proxy validated on US signals can badly misstate a European market map. A repo can be a clean primary indicator in one region and only corroboration in another, and your table should say so per region rather than assign one global verdict.

Seniority skew changes what the proxy measures. Only 25.5% of US LangChain professionals sit in the senior band, or 4,699 of 18,409 in Refolk's index. A headcount-based read is therefore dominated by early-career adopters, which is useful for gauging hype but weaker as a signal of enterprise commitment. If your market map is about production spend, weight the senior bench harder than the raw count.

The scoring procedure, end to end

Run these eight steps per repo in the basket, in this order. Steps three and four come before any star-based ranking on purpose: the usage and fraud checks decide what the star curve is even worth.

The verdict logic is simple once the gates are respected. A repo that fails the fake-star screen or has a low bus factor with high single-vendor commits cannot be a primary indicator, whatever else it scores. Among the survivors, strong off-platform usage plus a meaningful category share makes a primary indicator; weak usage or a thin practitioner bench makes it corroboration only; and a repo that is niche on every dimension is a drop.

Labor demand is the corroboration that confirms timing. Skill demand is measured as the ratio of job postings mentioning a skill to total postings, and demand for labor with a skill proxies for use of the technology. But job posts are a proxy, not proof: a mention usually but not always means adoption. Because companies reuse job descriptions, total mention volume reflects past decisions, while first mentions capture the moment of adoption. Count first mentions to read timing, not total mentions.

How this goes wrong

The failure modes below are where a proxy read quietly breaks, and each one has a specific check. Give this section real weight, because a scoring model that overclaims is worse than none.

• Star spike misread as momentum. A campaign spike looks like virality. Check whether more than 10% of a single month's stars come from low-activity or lockstep accounts and whether the star count later decayed within about two months, the burden pattern.
• Downloads inflated by CI or mirrors. High registry numbers can be automated pulls, not humans. Compare the download trend against dependent-repo growth; divergence signals bot traffic.
• Forks that are not usage. Fork count rises from course assignments and template copies. Check the ratio of forks with independent commits against empty forks.
• Contributor growth mistaken for adoption. More issues and pull requests can mean instability, not more users; Scarf's guidance is explicit that a rise in contributors, PRs, or issues does not mean the user base is growing and could indicate the opposite. Pair contribution counts with downloads before concluding growth.
• Single-vendor repo treated as a neutral proxy. A repo with a bus factor of 5 and 80% single-company commits mirrors one vendor's roadmap. Compute bus factor and largest-employer commit share, and require a neutral steering body before promoting to primary.
• Job mentions counted as live demand. Reused descriptions inflate mention volume for decisions made years ago. Use first-mention counts.
• Category share across inconsistent sources. Mixing GitHub issues with registry dependents biases the denominator. Use one consistent source per column across the whole basket.
• 50-star floor false positives. Below 50 stars the detector is noisy and a legitimate niche repo can be flagged. Demote only on the >10% spike criterion, not on any single signal.

Keeping the read current

A bellwether score is a snapshot of moving signals, so schedule a re-score rather than treating the verdict as permanent. Star campaigns, download trends, and hiring velocity all shift, and the two-month decay pattern means a proxy can look strong today and hollow next quarter.

When you re-run, keep the tooling stable so changes reflect the world, not your method. The open resources hold up well for this: GHArchive for events, the OpenSSF Criticality Score dataset for share componentry, ecosyste.ms for dependency and usage data, and StarScout for anomaly ground truth. Pair those with a practitioner-depth read - Refolk answers the "who actually ships with this" question in plain English, which is the half of adoption the code graph never shows. Hold the sources constant, move the dates, and your verdict table stays defensible from one leadership review to the next.