The Library Bake-Off: Scoring Two Dependencies to Pick One

You have two open-source libraries in front of you, both do the job, and you have to commit this week. This guide is for the engineer or lead who has to make that comparative call and defend it later, not to re-run a generic "seven factors" checklist that never forces a choice. It gives you a head-to-head scorecard, evidence-backed dimensions, an explicit tie-break when scores are close, and a fallback plan so the library you rejected is still available with its migration cost written down.

Most guidance online scores one library in isolation or points you at a popularity chart. That does not help when both candidates already clear the basic bar. The whole difficulty of "library A vs library B" is discriminating between two things that both look fine. That is what this method is built for.

Why a comparative scorecard beats a single-library checklist

A single-library checklist tells you whether a dependency is acceptable. A comparative scorecard tells you which of two acceptable dependencies to build on, which is the decision you actually have. The two questions need different machinery.

The failure of the popularity chart is structural. Stars have only moderate correlation with actual usage or importance, and as of July 2024, 16.66% of repositories with more than 50 stars had engaged in fake-star activity. Download counts are worse: they are naive by design, and attackers have inflated a package past 50,000 downloads in three days using automated systems. So when two libraries tie and you reach for "the more popular one," you reintroduce the weakest signal you have at the exact moment precision matters most.

The defensible move is to score across dimensions where the evidence is real, then break ties on a structural property rather than a gamed one. The dimensions that carry weight:

Note that the two dimensions most guides skip, migration cost and talent-pool depth, are the two that decide the hard cases. Migration cost feeds the tie-break directly. Talent-pool depth quantifies what happens if the maintainer disappears and you have to take the code on yourself.

The scoring dimensions and what each one proves

Score each dimension the same way for both candidates so the comparison is fair, then apply weights that sum to 100. There are no publicly prescribed weights, so the sum-to-100 discipline is what keeps the exercise honest rather than any authoritative ratio.

Maintenance health. The one hard operational threshold comes from OpenSSF Scorecard's Maintained check: one activity per week over a 90-day window yields a perfect score. Scorecard runs 18 checks in total, each returning 0 to 10, aggregated as a risk-weighted average. Record the Maintained score for both repositories. Treat it as a floor, not a verdict, because it is retrospective and offers no insight into the future.

Bus factor. The truck factor measures how many developers would need to leave for the project to be critically affected. Avelino's algorithm identifies file experts through the Degree of Authorship model, iteratively removes the developer who is expert for the largest number of files, and stops when more than half the project's files are abandoned. A truck factor of 1 or 2 is the red flag. But concentration is the norm, so this dimension only separates candidates at the extremes.

API stability. Quantify this as the library's own rate of breaking changes shipped in non-major releases. Do not trust the version scheme. About one third of all releases introduce at least one breaking change, and the figure is the same for minor and major releases, so version numbers do not reliably signal interface stability. In npm, 44% of manifesting breaking changes were introduced in minor and patch releases that should be backward compatible.

Security. Query OSV per package and version and count advisories. This is a lagging signal, so read it alongside maintenance health rather than as a standalone score.

Migration cost. Estimate the switching cost each way and label each candidate a one-way or two-way door. This is the dimension the tie-break weights heavily, so spend real time here.

Talent-pool depth. How many people could maintain or replace this if you had to. This is where the obvious winner can invert.

Where "market share" misleads: talent-pool depth

Ecosystem "market share" narratives can mislead a hiring-adjacent decision, and the correction is to count actual professionals rather than trust reputation. The library everyone calls dominant is not always the one with the deepest replacement pool.

In Refolk's index of professional profiles, the supposedly dominant UI library is not the deepest one:

That 1.23x edge to Vue.js does not by itself decide a UI-library bake-off, but it does mean the "everyone knows React, so hiring is easier" assumption is worth checking against real counts before you let it swing a decision.

Geography compounds this. If you adopt a library maintained by a thin team and it stalls, your replacement pool is a real, quantifiable component of migration cost, and it varies enormously by market:

A German team adopting PyTorch faces a replacement pool nearly six times thinner than a US team would. That does not make the choice wrong, but it belongs in the migration-cost column as a number, not a shrug.

Counting the people who could actually take a dependency on if its maintainer walked is tedious by hand across GitHub, LinkedIn, and the open web. Refolk does it from a plain-English query, which turns "is there a replacement pool here" from a guess into a number you can put in the scorecard.

How to run the bake-off, step by step

Run the eight steps in order. The whole exercise fits inside a week and most steps take an hour. The engineering steps can run in parallel once the scorecard skeleton is agreed.

There is one legitimate ordering disagreement. Decision-matrix practice puts weighting before scoring, which is how the steps above run. But you can apply a Bezos-style reversibility read first as a triage gate: if adopting either candidate is a cheap two-way door, the whole exercise deserves less depth than if you are walking through a one-way door. Use reversibility to decide how hard to work, then run the matrix.

The tie-break: reversibility, not popularity

When the weighted totals land within a few points of each other, break the tie by picking the candidate that is cheaper to abandon. Two options within a few points usually means either both are genuinely defensible, in which case you pick on speed or reversibility, or that a criterion is missing from your columns.

That last clause is not optional. Before you invoke the reversibility rule, confirm you did not omit a dimension. A near-tie is often a signal that the scorecard is incomplete, not that the two libraries are equal. Add the missing column, re-score, and see if the tie survives.

If it does, reversibility decides it. The reasoning comes from the two-way-door test: a one-way door is irreversible, where reversal is possible only at tremendous cost, versus a two-way door where you can leave and look elsewhere. For a library, reversibility is migration cost. So the tie-break is precise: pick the candidate you could rip out later at lower switching cost.

This is why popularity cannot be the tie-break. Popularity and switching cost point in opposite directions, and popularity is the signal most vulnerable to gaming. Anchoring on reversibility keeps the decision structural.

How this goes wrong: failure modes and false positives

Every dimension in this scorecard has a way of lying, and most of them lie in the direction that flatters a bad candidate. The signals that are easiest to read are also the easiest to fake, so treat each automated number as a claim to verify, not a fact.

Maintained-score false positive. A repository can hit a perfect Maintained score with trivial weekly commits - typo fixes, bot version bumps - because the check counts activity, not substance. The 90-day window is retrospective and blind to imminent abandonment. Check: read the actual commits and issue responses, not just the score. The XZ Utils and Log4j cases are the standing reminders that "looked maintained" and "was safe" are different claims.

Download pumping. A package can look actively maintained and legitimate through a dense release history and 50,000 downloads that are pure bot traffic. Attackers publish hundreds of versions, each drawing 100 to 150 downloads from automated systems. Check: compare CI-like to human-like downloads using the installer user-agent and version fields. A package that is 98% Linux-in-container and 2% human is used only in automated pipelines, a different risk profile than the raw count suggests.

Fake stars. A high star count with a burst that later vanishes is a classic tell; 16.66% of repositories with more than 50 stars were implicated as of July 2024, and 90.42% of repositories flagged by the StarScout detector have since been deleted. Check: look for star-velocity anomalies and lockstep accounts, and discount stars as a decision signal regardless.

SemVer trust. Assuming minor and patch upgrades are safe is the most common stability error. About one third of releases break something and 44% of npm breaks land in minor and patch releases. Check: measure the library's own non-major break rate from its changelog; do not trust the number scheme.

Bus factor from commit count. Many commits by one author still yields truck factor 1. Raw contributor headcount masks concentration. Check: run the Degree-of-Authorship truck factor, not committer headcount. Two studies agree the risk is widespread:

Because concentration is the norm, a low bus factor cannot by itself disqualify a candidate. It discriminates only at the extremes, so weight it modestly rather than treating it as a gate.

deps.dev maintenance blind spot. The maintained-or-deprecated field is not set for any package version outside the PyPI system, so a non-PyPI package can look fine only because the field is unset, not because it is healthy. Check: fall back to Scorecard plus repository activity rather than reading an empty field as a pass.

Recording the pick and the fallback

Do not throw the runner-up away. Record it as a documented fallback with its migration cost, so if the winner stalls, deprecates, or ships a breaking change you cannot absorb, you already know the exit and its price. The output of this method is two things: a defensible pick and a fallback you can reach for without re-running the whole exercise.

The fallback record is what turns a one-time decision into a maintainable one. Six months from now, when the winner's bus factor drops to 1 or an OSV advisory lands, the person on call should not have to rediscover why you chose what you chose or what the alternative was.

Decision: adopt <winning library> for <use case>
Date and owner: <date> / <name>

Scorecard (weights sum to 100):
| Dimension | Weight | <A> score | <B> score |
| Maintenance health | ?? | ?? | ?? |
| Bus factor | ?? | ?? | ?? |
| API stability | ?? | ?? | ?? |
| Security | ?? | ?? | ?? |
| Migration cost | ?? | ?? | ?? |
| Talent-pool depth | ?? | ?? | ?? |
| Weighted total | 100 | ?? | ?? |

Margin: <gap between totals>
Tie-break applied: <yes/no>. If yes: picked <winner> because it is the more reversible / lower switching-cost option.
Disqualifiers logged at the gate: <any candidate ruled out and why>

Fallback: <runner-up library>
Migration cost from winner to fallback: <estimate + one-way or two-way door>
Re-check trigger: <e.g. Maintained score drops below target, bus factor hits 1, or a high-severity OSV advisory lands>

Before you call the decision done, run the checklist. It catches the cases where the score is right but the read underneath it was not.

Keeping the decision current

A library bake-off is a snapshot, and the Maintained metric that anchors it is inherently retrospective, providing only a short-term view of the past 90 days. So the decision has a shelf life, and the fallback record is what lets you refresh it cheaply.

Set the re-check triggers in the decision record to concrete events, not calendar reminders. The three that matter most: the winner's Maintained score dropping below the one-activity-per-week target, its bus factor falling to 1, and a high-severity OSV advisory landing on the version you depend on. Any of these should reopen the record and the fallback in it.

deps.dev v3 comes with a guarantee never to make incompatible changes, so the tooling underneath your monitoring is stable even where the libraries you monitor are not. Wire the Scorecard run and the OSV query into whatever you already run in CI, and the re-check becomes an alert rather than a project. The point of writing the fallback down was never documentation for its own sake. It was so the next decision, when the trigger fires, takes an afternoon instead of a week.