Scoring a Vendor's Engineering Reliability From Public Signals

Before a third-party API or platform becomes a hard dependency in your product, someone has to decide whether the vendor's engineering is healthy enough to build on. This guide is for engineering managers, technical founders, and technical sourcers who need to make that call from public evidence alone, before signing anything or opening a sales cycle. It gives you a fixed set of weighted dimensions, a way to score each from artifacts anyone can read, and a rule for turning the total into build, sandbox, or avoid.

Existing procurement checklists assume you already have sandbox access and a security questionnaire in flight. This one runs earlier and colder. Every signal here comes from something the vendor publishes: a status page, a changelog, a deprecation policy, an SDK repo, a trust center. The point is to filter candidates on a Tuesday afternoon, before anyone talks to sales, and to arrive at a number you can defend to your team.

Why score from public signals at all

The short answer: the evidence that predicts whether a vendor will still be a good dependency in two years is almost entirely public, and you can read it before you commit a single engineer-hour. Deprecation policies, incident histories, changelog cadence, and SDK responsiveness are all published, and they say more about long-term engineering health than a sales demo ever will.

Making a vendor a hard dependency means your product breaks when theirs does and your roadmap bends when theirs does. That is a bet on the vendor's engineering discipline over the life of your integration, not on the feature set you see today. A feature set is easy to demo. Discipline shows up in how long they promise to keep an API alive, whether they honor that promise, how fast they answer issues, and whether they publish incidents honestly.

The trap is treating "well-run vendor" as a vibe. It is not a vibe; it is a number, and the range is enormous. Published deprecation floors alone span from 60 days to 24 months, which means two vendors both described as "reliable" can differ by an order of magnitude on the one axis that determines how much migration pain you inherit. A scoring rubric forces the number into the open.

A feature set is easy to demo. Engineering discipline shows up in what a vendor promises and whether it keeps the promise.

The six dimensions and how to weight them

Score six dimensions. Each maps to a public artifact, each proves something specific, and each has a characteristic way of lying to you. The weights below are a defensible default; adjust them for your context, but keep the total at 100 so scores stay comparable across vendors.

Deprecation policy carries the most weight because it is the dimension that most directly determines your future pain. Trust artifacts and engineering footprint carry the least because each proves something narrower than an engineer usually assumes: SOC 2 answers a security question, not a reliability one, and headcount is one signal among several.

Scoring deprecation policy: the highest-weight dimension

Read the vendor's published notice window and record it in months. The market genuinely spans 60 days to 24 months, so this single number tells you more about your future migration burden than any other signal, and a vendor with no published policy scores zero here.

Named floors are public and vary widely. The table below collects the ones the dossier establishes, each from the vendor's own policy page.

Six months is the common REST-API baseline. Bucket the vendor accordingly: 18 months or more is excellent, 6 to 12 is workable, under 90 days means you should assume you will be migrating on the vendor's schedule, not yours.

The signal lies when the policy is published but not honored. Anthropic commits to at least 60 days notice for publicly released models, yet a third party computed, from Anthropic's own dates, a median announce-to-shutdown gap of 63 days across 19 models, with consumer-app removals running far shorter than the API floor. The lesson is not that Anthropic is unreliable; the floor and the measured median actually track closely. The lesson is that you verify the promise against dated retirements on the deprecation page, because a policy page and a retirement record are two different documents and only one of them is binding.

Scoring incident history without being fooled by silence

Pull the status page and count incidents over the trailing 12 months, then note how far back the records go. The counterintuitive rule: an empty history is a red flag, not a green one, because a page that has never shown an incident almost always means the vendor does not post, not that nothing ever broke.

Public status pages are the primary source, but retention is not standardized. Statuspage exposes incidents through a public JSON API and distinguishes historical incidents, backfilled from before the account existed, from realtime incidents created during live events. No major status-page vendor publishes a fixed retention floor, so how far back history goes is a per-vendor choice, and third-party aggregators do not reliably archive it either. That means "not established publicly" is the honest answer for any universal retention number; you read what this vendor chose to keep.

Score three things: whether real incidents appear at all, whether components report actual uptime numbers, and whether the postmortem detail suggests engineers who understand their own failures. A vendor with a dozen well-written incident writeups over a year is more trustworthy than one with a pristine, silent page. Honesty about failure is itself the signal.

Scoring SDK responsiveness and release cadence

Judge whether a human actually staffs the integration you depend on by measuring time to first response on the official SDK repo and the gap since the last substantive maintainer commit. Fast, human, resolved responses on a repo that ships real work is the pattern you want; slow or bot-only responses on a stale repo is the pattern that abandons you mid-integration.

Time to first response is the recognized metric. CHAOSS defines it as how long a community member takes to respond to an issue or pull request, and treats it as a signal that the community is active and engaging. Prefer the median over the mean, because the median is closer to how people perceive the time it takes to move something forward. A precise numeric cut-off between "maintained" and "abandoned" is not established publicly; the closest documented convention is isitmaintained.com, which ignores issues older than six months when computing median time to close. Borrow that six-month window as a sensible default for your own sampling.

For cadence, count releases or changelog entries over the trailing 6 to 12 months and read the notes. No universal cadence threshold exists, but named feeds give reference points: GitHub ships dated API versions and announces each via its changelog, and Stripe documents a monthly-minor, twice-yearly-major rhythm. Academic work on maintenance cessation uses the maintainer inactivity interval, the time since the last maintainer activity, as a governance-continuity feature. Combine the two: substantive releases on a rhythm, plus a short inactivity interval on real pull requests, is a healthy repo.

Both signals lie in the same direction, toward false confidence. A fast first response may be an auto-reply or a triage label rather than a human, so confirm the responder is a maintainer other than the issue author and that the thread reaches resolution. A high release count may be dependency bumps rather than feature or fix work, so read the release notes, not just the count.

Reading the engineering footprint behind the marketing

The team behind a vendor is a public signal, and the useful read is a ratio, not a raw count. Industry-wide, reliability engineering headcount vastly outnumbers developer-relations headcount, so a vendor that is loud on developer marketing but thin on reliability engineering is inverting the normal ratio, and that inversion is scoreable.

In Refolk's index of professional profiles, the US market shows 6,568 Site Reliability Engineers against 318 people in Developer Advocate, Developer Relations, or Developer Experience roles: a 20.7x ratio. When you look at a candidate vendor and find a large, visible DevRel presence with almost no reliability-engineering bench behind it, you are looking at a company optimized to be adopted, not to stay up. That is worth a mark against.

Geography compresses the DevRel signal, so normalize before you judge. US DevRel presence is about 6.8x the UK's (318 versus 47) in Refolk's index. A "small DevRel team" for a European vendor is often market-normal rather than neglect, and holding it to a US-sized expectation will make you discard good vendors. When you size the senior bench behind a smaller vendor, ask in plain terms rather than guessing from a homepage.

Because that read is a judgement call across two variables, the DevRel-versus-reliability balance and the geographic baseline, it plots cleanly on a matrix. Use it to decide what the footprint actually tells you.

What trust artifacts prove, and what they do not

Confirm three trust artifacts exist and note their scope: SOC 2 Type II, a published SLA percentage, and a trust center. The critical discipline is knowing exactly what each proves, because the most common scoring error is treating a SOC 2 badge as evidence of reliability, which it is not.

SOC 2 Type II tests whether security controls were actually followed throughout an observation window, not just designed on audit day, so it proves operational effectiveness of controls over 3 to 12 months. It is an independent CPA opinion, and it is explicitly not a certification or a badge. It does not prove uptime, API stability, or deprecation discipline. It answers a different question than the one an engineer building on the platform is asking. It still matters commercially: 29% of organizations reported losing potential new business due to the absence of a required compliance certification like SOC 2 Type II, and a recent report often satisfies enterprise security questionnaires, compressing due diligence from weeks to days. But under this rubric it earns only its 10 points, and only when it is Type II and current.

Two distinctions keep you honest. First, a published SLA is a target, not a record; pair it with the incident history to see whether the vendor actually hits it. Second, a trust center often lets you request the SOC 2 report without a sales rep, but the report itself is usually gated behind an NDA. So "verifiable without a sales call" means you can confirm existence and scope, not read the full findings. Score "confirmed exists and in scope" as present; do not inflate it to "read the findings."

How to score a vendor step by step

Run these eight steps in order. The whole pass takes about two hours for a single vendor, and the last two steps are what make the score defensible rather than arbitrary.

Map the total to a decision with a fixed band: 75 and above is build, 50 to 74 is sandbox (integrate behind an abstraction layer and keep an exit path warm), and below 50 is avoid unless nothing else exists. The bands are less important than applying the same ones to every vendor in the comparison.

Vendor <name>: <total>/100 (dep <n>, incidents <n>, SDK <n>, cadence <n>, trust <n>, footprint <n>) -> <BUILD | SANDBOX | AVOID>. Weakest dimension: <name>. Re-check: <date>.

How this scoring goes wrong

Every dimension has a characteristic false positive, and the failure modes below are where careful engineers still get it wrong. Read this section as the most valuable part of the framework, because a rubric that produces confident wrong scores is worse than no rubric.

• Empty status page read as reliability. A page with zero incidents usually means the vendor rarely posts, not that nothing broke. Check whether any historical incidents exist and whether components report real uptime numbers. A page that has never shown an incident is a red flag.
• Deprecation policy exists but is not honored. A published floor can coexist with much shorter real removals, as Anthropic's consumer-app case shows against its 60-day API floor. Verify the policy against the actual dated retirements, not just the promise.
• Changelog cadence inflated by trivial commits. Frequent releases can be dependency bumps rather than feature or fix work. Read the release notes, not the count, and check the maintainer inactivity interval on substantive pull requests.
• Median response time gamed by bot replies. A fast first response may be an auto-reply or a triage label, not a human. Confirm the responder is a maintainer other than the author, per the CHAOSS definition, and that the thread reaches resolution.
• SOC 2 badge mistaken for reliability proof. SOC 2 attests to security-control operation, not uptime or API longevity. Confirm the report is Type II rather than Type I, that it is current, and read the exact observation-period dates. A two-year-old report is stale.
• Small-vendor headcount misread. A tiny DevRel or SRE team can still ship reliably; absence of a large team is not proof of fragility. Weight headcount as one signal among several and always score a peer vendor for relative context.
• Trust center presence read as report access. A trust center can list certifications you still cannot read without an NDA. Distinguish "confirmed exists and in scope" from "read the findings."

There is a cheaper, positive signal worth catching too. Documentation discipline itself predicts engineering health. One maintainer reported that adding a DEPRECATED.md table cut deprecation-related issues by roughly 30% the next cycle. Dated changelogs and explicit deprecation tables are free to read and correlate with vendors that take churn seriously, so treat their presence as a mild positive across the cadence and deprecation dimensions.

Before you call the score final

Run this checklist before you record a decision. It exists to catch the failure modes above, and none of its items take more than a minute once you have your artifacts open.

Keeping the score current

A reliability score is a snapshot, and the signals that produced it drift. Set a re-check date on every recorded score and re-run the cheap dimensions on a cadence, because the difference between a healthy dependency and a liability often shows up months before it hits your production traffic.

Re-check deprecation and incident history quarterly; both change on the vendor's schedule and both are fast to sample. Watch the SDK's maintainer inactivity interval continuously if you can, since a widening gap on substantive pull requests is the earliest public sign of maintenance cessation. And when a vendor's team composition shifts, the public record moves before the status page does: engineers leaving for competitors, a shrinking reliability bench, or a sudden DevRel expansion unmatched by hiring on the reliability side. Those are exactly the attrition and investment signals worth surfacing early, and asking for the right people in plain language beats reconstructing an org chart by hand. Refolk turns "site reliability engineers at this vendor who joined in the last two years" into a straight read on whether reliability headcount is growing or shrinking, which is the leading indicator this whole rubric is trying to approximate.