The Vendor Reliability Score: Grading an API From Its Incident History

You are about to make a third-party API or SaaS a hard dependency, and you need to know whether its real reliability is good enough to build a core feature on. This guide is for the engineering managers, technical founders, and platform leads who own that call. It turns a candidate vendor's public status page and published SLA into a defensible 0-100 reliability score across named dimensions, and lands you on one of three verdicts: build on it directly, build on it with a fallback, or reject it.

The premise is simple and uncomfortable. The advertised SLA is a billing document, not a reliability measurement. It tells you what the vendor will refund when they miss, not how often they miss. The incident log tells you how often they miss. Most vetting guides take the SLA percentage at face value; this one separates the headline number from what the incident history actually shows, and treats the gap as the score.

Why the advertised SLA is not the reliability number

The SLA percentage is a contractual promise governed by an exclusion clause, and the exclusion clause, not the percentage, sets real reliability. A 99.9% commitment sounds like near-perfection until you read what is subtracted from the denominator before the division happens.

Downtime allowance per tier is fixed math. Each added nine cuts the allowance by a factor of ten. Here is the canonical reference every practitioner keeps within reach:

Now read the fine print. Standard SaaS SLAs carve out a consistent set of exclusions: scheduled maintenance with at least 48 hours notice up to a monthly cap, emergency maintenance, force majeure, the customer's own acts or equipment, third-party services, and beta or free features. Scheduled maintenance is the largest exclusion by impact. Vendor-preferred language commonly grants 4 to 8 hours per month excluded from measurement. Combine that with the 43.8 minutes permitted under 99.9% and the effective customer experience can include up to 9 hours of monthly unavailability without an SLA breach.

The mechanism is the SLA formula itself: it subtracts excluded minutes from total minutes before dividing. A broad exclusion scope can convert a 99.9% stated SLA into 99.5% effective availability. When people ask how to compare SLA vs actual uptime, this is the whole answer. Read the exclusions first, compute the effective number, and only then decide whether the headline was ever worth anything.

The seven dimensions of a reliability score

A reliability score has seven measurable dimensions, each computed directly from public status-page records and each answering a different question about how a vendor fails. The point of naming all seven is that a single blended number hides the specific way a vendor is dangerous.

The metrics vocabulary is standard across the reliability field: reliability (SLO attainment and error budget remaining), occurrence (unique incident count and rate), and recovery (median, p90, mean, maximum, completed count, and open ages). The seven dimensions map onto that vocabulary and onto the fields a status API actually returns.

• Observed uptime. Sum the component-outage minutes and divide by the window. This is what actually happened, independent of the SLA math. It lies when the window is short and a bad quarter has already scrolled off the 50-incident cap.
• Incident frequency. Count incidents and divide by months in the window. It proves how often something goes wrong. It lies when a vendor grows: raw count rises with surface area, so normalise per component and read the trend, not the absolute.
• MTTR, with p90. Subtract created_at from resolved_at, average it, and always report p90 alongside. The mean proves typical recovery; p90 and the single worst incident prove the tail you actually carry. The mean lies by hiding long tails.
• Severity distribution. Count incidents by the impact field, which buckets as None, Minor, Major, or Critical. It proves how bad the bad days are. It lies when a vendor overuses "Degraded Performance" for what is really an outage.
• Trend. Compare frequency and MTTR quarter over quarter. It proves whether the vendor is getting better or worse, which matters more than any single snapshot for a dependency you will carry for years.
• Disclosure latency. Subtract outage onset from the first update timestamp. It proves how honest and fast the vendor is about admitting failure. Long gaps usually mean missing automation or gaps in the escalation process.
• Blast-radius scope. Count how many components go non-operational per incident. It proves whether failures are contained or systemic. A vendor whose every incident takes down half the platform is a different risk from one with isolated component failures.

What the public record hides and how to catch it

Public status-page data is capped, mutable, and self-serving, so read it as a document the vendor controls rather than as ground truth. Three structural problems distort what you see, and each has a specific detection method.

The first is the window cap. The dominant hosted status-page API returns only the 50 most recent incidents, including all unresolved incidents plus those in Resolved and Postmortem state. This is confirmed across Atlassian, Cloudflare, and Red Hat deployments. The cap systematically flatters unreliable vendors: the worse the vendor, the faster their bad quarter scrolls past the 50-incident window, so the visible record self-cleans exactly when it matters most. Detection is mechanical. Record the created_at of the oldest retrievable incident. If that window is under six months, your score is provisional and you must say so.

The second is retroactive editing. Backfilled degradations are a first-party product feature: a vendor can retroactively document disruptions that were not previously announced, with each update assignable to its original timestamp so the incident appears in uptime history. Retrospective incidents let a vendor declare a status-page incident after the fact. This means the public record is mutable, and two identical-looking histories can have different integrity. Detection: look for incidents where created_at long precedes display_at, and postmortems dated after the event.

The third is understated severity. Overusing "Degraded Performance" is a known failure. If customers cannot complete core workflows, that is an outage, not degradation. A green "Operational" board during a tenfold latency spike is the same lie by omission. Detection: cross-reference third-party monitors and public outage aggregators against the vendor's own labels during known slowdowns.

Independent verification is where diligence gets expensive, and the talent to do it is scarce and lopsided. In Refolk's index, 9,915 people in the United States hold a Site Reliability or Reliability Engineer title, against 1,776 in the United Kingdom and 748 in Germany.

The US has 13.3 times Germany's reliability-engineer population and roughly 5.6 times the UK's. Most buyers outside a few hubs cannot staff deep due diligence and must lean on the public record, which is exactly the record that self-cleans and can be edited. That is the argument for a repeatable scoring method: it lets a small team extract a defensible verdict from what is visible, instead of pretending they have an SRE team they do not have.

One finding from Refolk's index is worth pausing on, because it changes who you talk to. A US search for the reliability-engineer title restricted to Director, VP, or CXO seniority returned zero. The title pool concentrates almost entirely at the individual-contributor level. The people who actually own vendor reliability decisions are titled Head of Infrastructure or VP Engineering, not SRE. If you are sourcing the person who signs off on a dependency, search by function, not by the letters "SRE".

The scoring procedure

Run the eight steps below in order, from scoping the dependency to writing the verdict. The whole pass takes a working day for one engineer and one manager, and it produces an archived record you can defend and re-run later.

One note on order. Some checklists run the independent cross-check first and treat the status page as secondary, because a third-party monitor is data the vendor does not control. If you have access to a reliable outage aggregator for this vendor, running step five before step four is defensible. The steps as written assume the status page is your primary source, which is the common case for buyers without an SRE team.

Here is a rubric skeleton you can copy. Weights are a starting point, not a standard; tune them to how the dependency will be used.

Observed uptime (effective, post-exclusion)   weight 25   score /100 = ____
Incident frequency (per month, trend-adjusted) weight 15   score /100 = ____
MTTR p90 and worst single incident             weight 20   score /100 = ____
Severity distribution (Major + Critical share) weight 15   score /100 = ____
Trend (quarter over quarter direction)         weight 10   score /100 = ____
Disclosure latency (onset to first update)     weight 10   score /100 = ____
Blast-radius scope (components down/incident)   weight 5    score /100 = ____
Penalties: exclusion breadth (-0 to -15), under-reporting evidence (-0 to -20)
Composite = sum(weight x dimension score) - penalties
Window note: oldest retrievable incident = ______ ; if < 6 months, mark score PROVISIONAL

Turning the score into a verdict

The verdict has three outcomes: build on it directly, build on it with a fallback, or reject. No engineering guide grades observed reliability on a public 0-100 scale, so treat the specific thresholds below as my opinion, not an established standard, and calibrate them against your own error budget.

The load-bearing math is not opinion, though. Dependencies compound. End-to-end availability multiplies across every required dependency; it does not average. Two independent components each delivering 99.95% produce roughly 99.9% together before your application contributes any failures. Promise 99.9% on top of two dependencies that each promise 99.9%, and the math is already against you. Four independent 99.9% dependencies in one path yield about 99.6% end-to-end, roughly 2.9 hours of monthly downtime. This is why a "good" vendor can still be a reject: not because its own score is bad, but because it is the fourth good vendor in a path that has already spent its budget.

As a starting calibration for a synchronous hard dependency: a composite at or above 85 with a non-provisional window and a flat or improving trend earns build-directly. A composite from 65 to 84, or any score sitting on a provisional window under six months, earns build-with-fallback: you proceed, but you design and test a degraded mode, a cache, or a secondary provider. Below 65, or any month observed below 95% in the record, is a reject for the hot path. That 95% floor mirrors the standard chronic-failure exit trigger written into procurement contracts: a breach in three of six months, or any single month below 95%.

Whatever thresholds you set, apply the compounding budget before you commit. Write down the target end-to-end availability for the feature, multiply the candidate against every other vendor already in the path, and only then read the verdict. A 99.95% approval is meaningless if three other 99.9% vendors share the request path and drag the end-to-end figure below 99.7%.

How this scoring goes wrong

The score fails in predictable ways, and every failure mode is a false positive that makes a bad vendor look safe. Learn the seven below and check for each before you sign off; this section is the most valuable part of the standard because it is where defensible verdicts are won or lost.

• Trusting the headline SLA number. A clean 99.99% claim whose exclusions hide 8 hours of monthly maintenance reads as safe and is not. Check: compute effective availability after subtracting the exclusion cap from the promise.
• Scoring MTTR on the mean. A "12-minute MTTR" that averages fifty 5-minute blips with one 6-hour outage looks excellent. Check: report p90 and the single worst incident, and score the tail.
• Reading only the 50-incident window. A vendor looks stable because their noisy quarter scrolled off the API cap. Check: record the oldest retrievable incident date; if the window is under six months, mark the score provisional and say so in writing.
• Believing "Operational" during a slowdown. A green board while latency runs tenfold is a lie by label. Check: cross-reference third-party monitors and downgrade-labeled updates against known slow periods.
• Missing backfilled or retro edits. A smooth history that was quietly reconstructed after the fact scores clean. Check: look for incidents where created_at long precedes display_at, and postmortems dated after the event.
• Scoring one vendor in isolation. A 99.95% approval while three other 99.9% vendors sit in the same path yields sub-99.7% end-to-end. Check: multiply availabilities across the whole request path.
• Counting frequency without exposure. Incident count rises only because the vendor grew and added components. Check: normalise per component or per month, and read the trend, not the absolute.

Keeping the score current

A reliability score is a snapshot of a moving target, so schedule a re-run rather than treating the verdict as permanent. Trend is one of the seven dimensions for exactly this reason: a vendor that scored 88 last year and is degrading quarter over quarter is a different risk from one holding steady.

Before you call any single pass done, run this check.

Re-run the full pass on a fixed cadence, quarterly for a synchronous hard dependency, and immediately after any incident you personally felt. Two changes should force a re-score outside the cadence: a shift in the vendor's SLA exclusion language, which can silently move the effective number, and a change to your own architecture that adds a dependency to the request path, which changes the compounding budget for every vendor already in it. Keep the archived JSON from each pass. The archive is what lets you measure trend honestly, because it is the one copy of the vendor's history the vendor cannot backfill.

Finally, know who owns this decision when you need to escalate or hire for it. Vendor-reliability ownership hides under non-obvious titles. In Refolk's index, a leadership-seniority search for reliability engineers returned zero, so the people who make the build-directly, fallback, or reject call are titled Head of Infrastructure or VP Engineering. When you need that person, whether to ratify a verdict or to build the team that runs this process, search by the function they own, not by the acronym on the job ladder. Refolk resolves those searches from public records so you can name the role in plain English and get the right people back.