# The Vendor Reliability Score: Grading an API From Its Incident History

*You will convert a candidate vendor's public status page and SLA into a 0-100 reliability score and a build, fallback, or reject verdict.*

- Canonical URL: https://www.refolk.ai/guides/vendor-reliability-score
- Pillar: Engineering and open source
- Format: Framework
- Published: 2026-09-24
- Last reviewed: 2026-09-24
- Reading time: 17 min

You are about to make a third-party API or SaaS a hard dependency, and you need to know whether its real reliability is good enough to build a core feature on. This guide is for the engineering managers, technical founders, and platform leads who own that call. It turns a candidate vendor's public status page and published SLA into a defensible 0-100 reliability score across named dimensions, and lands you on one of three verdicts: build on it directly, build on it with a fallback, or reject it.

The premise is simple and uncomfortable. The advertised SLA is a billing document, not a reliability measurement. It tells you what the vendor will refund when they miss, not how often they miss. The incident log tells you how often they miss. Most vetting guides take the SLA percentage at face value; this one separates the headline number from what the incident history actually shows, and treats the gap as the score.

## Why the advertised SLA is not the reliability number

The SLA percentage is a contractual promise governed by an exclusion clause, and the exclusion clause, not the percentage, sets real reliability. A 99.9% commitment sounds like near-perfection until you read what is subtracted from the denominator before the division happens.

Downtime allowance per tier is fixed math. Each added nine cuts the allowance by a factor of ten. Here is the canonical reference every practitioner keeps within reach:

| Tier | Per month | Per year |
|---|---|---|
| 99.9% | 43.8 min | 8.76 hr |
| 99.95% | 21.9 min | 4.38 hr |
| 99.99% | 4.4 min | 52.6 min |
| 99.999% | 26 sec | 5.26 min |

Now read the fine print. Standard SaaS SLAs carve out a consistent set of exclusions: scheduled maintenance with at least 48 hours notice up to a monthly cap, emergency maintenance, force majeure, the customer's own acts or equipment, third-party services, and beta or free features. Scheduled maintenance is the largest exclusion by impact. Vendor-preferred language commonly grants 4 to 8 hours per month excluded from measurement. Combine that with the 43.8 minutes permitted under 99.9% and the effective customer experience can include up to 9 hours of monthly unavailability without an SLA breach.

| Scenario | Counted downtime/mo | Effective downtime/mo |
|---|---|---|
| 99.9% headline | 43.8 min | 43.8 min |
| 99.9% + 4 hr maintenance excluded | 43.8 min | up to ~4.7 hr |
| 99.9% + 8 hr maintenance excluded | 43.8 min | up to ~8.7 hr |

The mechanism is the SLA formula itself: it subtracts excluded minutes from total minutes before dividing. A broad exclusion scope can convert a 99.9% stated SLA into 99.5% effective availability. When people ask how to compare SLA vs actual uptime, this is the whole answer. Read the exclusions first, compute the effective number, and only then decide whether the headline was ever worth anything.

> **Rule:** Score the effective number, never the headline
>
> State the vendor's availability as the promise minus the exclusion cap. If a 99.99% SLA excludes 8 hours of monthly maintenance, your working number is closer to 99.0%, and that is the figure the score is built on.

## The seven dimensions of a reliability score

A reliability score has seven measurable dimensions, each computed directly from public status-page records and each answering a different question about how a vendor fails. The point of naming all seven is that a single blended number hides the specific way a vendor is dangerous.

The metrics vocabulary is standard across the reliability field: reliability (SLO attainment and error budget remaining), occurrence (unique incident count and rate), and recovery (median, p90, mean, maximum, completed count, and open ages). The seven dimensions map onto that vocabulary and onto the fields a status API actually returns.

- **Observed uptime.** Sum the component-outage minutes and divide by the window. This is what actually happened, independent of the SLA math. It lies when the window is short and a bad quarter has already scrolled off the 50-incident cap.
- **Incident frequency.** Count incidents and divide by months in the window. It proves how often something goes wrong. It lies when a vendor grows: raw count rises with surface area, so normalise per component and read the trend, not the absolute.
- **MTTR, with p90.** Subtract created_at from resolved_at, average it, and always report p90 alongside. The mean proves typical recovery; p90 and the single worst incident prove the tail you actually carry. The mean lies by hiding long tails.
- **Severity distribution.** Count incidents by the impact field, which buckets as None, Minor, Major, or Critical. It proves how bad the bad days are. It lies when a vendor overuses "Degraded Performance" for what is really an outage.
- **Trend.** Compare frequency and MTTR quarter over quarter. It proves whether the vendor is getting better or worse, which matters more than any single snapshot for a dependency you will carry for years.
- **Disclosure latency.** Subtract outage onset from the first update timestamp. It proves how honest and fast the vendor is about admitting failure. Long gaps usually mean missing automation or gaps in the escalation process.
- **Blast-radius scope.** Count how many components go non-operational per incident. It proves whether failures are contained or systemic. A vendor whose every incident takes down half the platform is a different risk from one with isolated component failures.

#### What each layer of a reliability score is built on

1. **Verdict** - Build directly, build with fallback, or reject
2. **Composite 0-100 score** - Weighted dimensions, penalised for exclusions and under-reporting
3. **Seven dimensions** - Observed uptime, frequency, MTTR, severity, trend, disclosure latency, blast radius
4. **Public record** - Archived status-page JSON, bounded by the 50-incident cap
5. **SLA definitions** - Effective availability after exclusions

*The composite score sits on top of raw metrics, which sit on top of the public record, which sits on top of the SLA's definitions.*

## What the public record hides and how to catch it

Public status-page data is capped, mutable, and self-serving, so read it as a document the vendor controls rather than as ground truth. Three structural problems distort what you see, and each has a specific detection method.

The first is the window cap. The dominant hosted status-page API returns only the 50 most recent incidents, including all unresolved incidents plus those in Resolved and Postmortem state. This is confirmed across Atlassian, Cloudflare, and Red Hat deployments. The cap systematically flatters unreliable vendors: the worse the vendor, the faster their bad quarter scrolls past the 50-incident window, so the visible record self-cleans exactly when it matters most. Detection is mechanical. Record the created_at of the oldest retrievable incident. If that window is under six months, your score is provisional and you must say so.

**50 - Incidents the dominant status-page API returns, maximum**

A noisy vendor may expose only a few months of history, and the noisier they are, the shorter that window gets.

The second is retroactive editing. Backfilled degradations are a first-party product feature: a vendor can retroactively document disruptions that were not previously announced, with each update assignable to its original timestamp so the incident appears in uptime history. Retrospective incidents let a vendor declare a status-page incident after the fact. This means the public record is mutable, and two identical-looking histories can have different integrity. Detection: look for incidents where created_at long precedes display_at, and postmortems dated after the event.

The third is understated severity. Overusing "Degraded Performance" is a known failure. If customers cannot complete core workflows, that is an outage, not degradation. A green "Operational" board during a tenfold latency spike is the same lie by omission. Detection: cross-reference third-party monitors and public outage aggregators against the vendor's own labels during known slowdowns.

> **Watch out:** A green board is a claim, not a measurement
>
> Vendors mark themselves operational during real degradations, and they can backfill or downgrade incidents after the fact. Treat the status page as the vendor's argument about its reliability, then verify it against data the vendor does not control.

Independent verification is where diligence gets expensive, and the talent to do it is scarce and lopsided. In Refolk's index, 9,915 people in the United States hold a Site Reliability or Reliability Engineer title, against 1,776 in the United Kingdom and 748 in Germany.

| Market | Title holders | Ratio vs Germany |
|---|---|---|
| United States | 9,915 | 13.3x |
| United Kingdom | 1,776 | 2.4x |
| Germany | 748 | 1.0x |

The US has 13.3 times Germany's reliability-engineer population and roughly 5.6 times the UK's. Most buyers outside a few hubs cannot staff deep due diligence and must lean on the public record, which is exactly the record that self-cleans and can be edited. That is the argument for a repeatable scoring method: it lets a small team extract a defensible verdict from what is visible, instead of pretending they have an SRE team they do not have.

I ran this search: `Heads of Infrastructure and VP Engineering at Series B payments or identity API startups in the UK` - [see the full result list](https://www.refolk.ai/s/khavjcb621).

*Returns the leadership who actually own vendor-reliability decisions, since a title search for reliability engineers at that seniority returns almost nobody.*

One finding from Refolk's index is worth pausing on, because it changes who you talk to. A US search for the reliability-engineer title restricted to Director, VP, or CXO seniority returned zero. The title pool concentrates almost entirely at the individual-contributor level. The people who actually own vendor reliability decisions are titled Head of Infrastructure or VP Engineering, not SRE. If you are sourcing the person who signs off on a dependency, search by function, not by the letters "SRE".

## The scoring procedure

Run the eight steps below in order, from scoping the dependency to writing the verdict. The whole pass takes a working day for one engineer and one manager, and it produces an archived record you can defend and re-run later.

#### From candidate vendor to a written verdict

1. **Scope the dependency** - Decide whether the vendor sits in the synchronous request path or is async and deferrable. Done when you know the maximum blast radius if it fails.
2. **Pull the public record** - Hit the vendor's status API and archive the JSON, noting you get at most the 50 most recent incidents. Record the date of the oldest one so you know your true window. Done when the raw set is saved with its window bound written down.
3. **Read the SLA fine print** - Extract the uptime commitment, the measurement formula, and the full exclusion list plus the scheduled-maintenance monthly cap. Done when you can state the effective post-exclusion availability, not the headline.
4. **Compute the seven dimensions** - Derive observed uptime, incident frequency, MTTR with p90, severity mix, trend, disclosure latency, and blast-radius scope from the archived timestamps and component fields. Done when you have seven raw numbers.
5. **Cross-check independent data** - Compare status-page timestamps to a third-party monitor or public outage aggregator, flagging operational labels during known degradations and any backfilled incidents. Done when you have an under-reporting adjustment factor.
6. **Normalise to a 0-100 score** - Weight the seven dimensions, then penalise for exclusion breadth and under-reporting evidence. Done when you have one number per dimension and a composite.
7. **Apply the compounding budget** - Multiply the candidate's observed availability against every other vendor already in that request path. Done when you have a realistic end-to-end availability figure.
8. **Land the verdict** - Decide build-directly, build-with-fallback, or reject. Done when you have a written decision with the score and, if needed, the fallback design.

One note on order. Some checklists run the independent cross-check first and treat the status page as secondary, because a third-party monitor is data the vendor does not control. If you have access to a reliable outage aggregator for this vendor, running step five before step four is defensible. The steps as written assume the status page is your primary source, which is the common case for buyers without an SRE team.

Here is a rubric skeleton you can copy. Weights are a starting point, not a standard; tune them to how the dependency will be used.

**Reliability score rubric (0-100 composite)**

```
Observed uptime (effective, post-exclusion)   weight 25   score /100 = ____
Incident frequency (per month, trend-adjusted) weight 15   score /100 = ____
MTTR p90 and worst single incident             weight 20   score /100 = ____
Severity distribution (Major + Critical share) weight 15   score /100 = ____
Trend (quarter over quarter direction)         weight 10   score /100 = ____
Disclosure latency (onset to first update)     weight 10   score /100 = ____
Blast-radius scope (components down/incident)   weight 5    score /100 = ____
Penalties: exclusion breadth (-0 to -15), under-reporting evidence (-0 to -20)
Composite = sum(weight x dimension score) - penalties
Window note: oldest retrievable incident = ______ ; if < 6 months, mark score PROVISIONAL
```

*Adjust weights to your use case. A synchronous payment path should weight MTTR and blast radius higher than a nightly batch job would.*

## Turning the score into a verdict

The verdict has three outcomes: build on it directly, build on it with a fallback, or reject. No engineering guide grades observed reliability on a public 0-100 scale, so treat the specific thresholds below as my opinion, not an established standard, and calibrate them against your own error budget.

The load-bearing math is not opinion, though. Dependencies compound. End-to-end availability multiplies across every required dependency; it does not average. Two independent components each delivering 99.95% produce roughly 99.9% together before your application contributes any failures. Promise 99.9% on top of two dependencies that each promise 99.9%, and the math is already against you. Four independent 99.9% dependencies in one path yield about 99.6% end-to-end, roughly 2.9 hours of monthly downtime. This is why a "good" vendor can still be a reject: not because its own score is bad, but because it is the fourth good vendor in a path that has already spent its budget.

#### Score against request-path position

Horizontal axis runs from Score low (weak record) to Score high (strong record). Vertical axis runs from Async or deferrable to Synchronous request path.

| Quadrant | What it means |
| --- | --- |
| Async and weak | Build with a queue and retry; the async buffer absorbs the weakness |
| Async and strong | Build directly; low risk, keep monitoring the trend |
| Synchronous and weak | Reject, or redesign to remove it from the hot path before reconsidering |
| Synchronous and strong | Build directly only if the compounding budget still has room |

*The same score means different things depending on whether the vendor is in the synchronous path.*

As a starting calibration for a synchronous hard dependency: a composite at or above 85 with a non-provisional window and a flat or improving trend earns build-directly. A composite from 65 to 84, or any score sitting on a provisional window under six months, earns build-with-fallback: you proceed, but you design and test a degraded mode, a cache, or a secondary provider. Below 65, or any month observed below 95% in the record, is a reject for the hot path. That 95% floor mirrors the standard chronic-failure exit trigger written into procurement contracts: a breach in three of six months, or any single month below 95%.

> A good vendor can still be a reject, not for its own record but because it is the fourth good vendor in a path that already spent its budget.

Whatever thresholds you set, apply the compounding budget before you commit. Write down the target end-to-end availability for the feature, multiply the candidate against every other vendor already in the path, and only then read the verdict. A 99.95% approval is meaningless if three other 99.9% vendors share the request path and drag the end-to-end figure below 99.7%.

## How this scoring goes wrong

The score fails in predictable ways, and every failure mode is a false positive that makes a bad vendor look safe. Learn the seven below and check for each before you sign off; this section is the most valuable part of the standard because it is where defensible verdicts are won or lost.

- **Trusting the headline SLA number.** A clean 99.99% claim whose exclusions hide 8 hours of monthly maintenance reads as safe and is not. Check: compute effective availability after subtracting the exclusion cap from the promise.
- **Scoring MTTR on the mean.** A "12-minute MTTR" that averages fifty 5-minute blips with one 6-hour outage looks excellent. Check: report p90 and the single worst incident, and score the tail.
- **Reading only the 50-incident window.** A vendor looks stable because their noisy quarter scrolled off the API cap. Check: record the oldest retrievable incident date; if the window is under six months, mark the score provisional and say so in writing.
- **Believing "Operational" during a slowdown.** A green board while latency runs tenfold is a lie by label. Check: cross-reference third-party monitors and downgrade-labeled updates against known slow periods.
- **Missing backfilled or retro edits.** A smooth history that was quietly reconstructed after the fact scores clean. Check: look for incidents where created_at long precedes display_at, and postmortems dated after the event.
- **Scoring one vendor in isolation.** A 99.95% approval while three other 99.9% vendors sit in the same path yields sub-99.7% end-to-end. Check: multiply availabilities across the whole request path.
- **Counting frequency without exposure.** Incident count rises only because the vendor grew and added components. Check: normalise per component or per month, and read the trend, not the absolute.

> **Tip:** The two-timestamp tell for edited history
>
> The fastest integrity check is created_at against display_at on each incident. When created_at sits far before display_at, the incident was declared or surfaced after the fact. One or two are normal; a pattern of them means the visible history was reconstructed.

## Keeping the score current

A reliability score is a snapshot of a moving target, so schedule a re-run rather than treating the verdict as permanent. Trend is one of the seven dimensions for exactly this reason: a vendor that scored 88 last year and is degrading quarter over quarter is a different risk from one holding steady.

Before you call any single pass done, run this check.

#### Before you record the verdict

- [ ] The score is built on effective availability, with the SLA exclusion cap subtracted from the headline.
- [ ] The oldest retrievable incident date is recorded, and any window under six months is flagged provisional.
- [ ] MTTR is reported as p90 and the single worst incident, not the mean alone.
- [ ] Status-page labels were cross-checked against at least one independent monitor or aggregator.
- [ ] The incident set was scanned for created_at preceding display_at and post-dated postmortems.
- [ ] The candidate's availability was multiplied against every other vendor in the same request path.
- [ ] The verdict is written down with the composite score and, if fallback, the degraded-mode design.

Re-run the full pass on a fixed cadence, quarterly for a synchronous hard dependency, and immediately after any incident you personally felt. Two changes should force a re-score outside the cadence: a shift in the vendor's SLA exclusion language, which can silently move the effective number, and a change to your own architecture that adds a dependency to the request path, which changes the compounding budget for every vendor already in it. Keep the archived JSON from each pass. The archive is what lets you measure trend honestly, because it is the one copy of the vendor's history the vendor cannot backfill.

Finally, know who owns this decision when you need to escalate or hire for it. Vendor-reliability ownership hides under non-obvious titles. In Refolk's index, a leadership-seniority search for reliability engineers returned zero, so the people who make the build-directly, fallback, or reject call are titled Head of Infrastructure or VP Engineering. When you need that person, whether to ratify a verdict or to build the team that runs this process, search by the function they own, not by the acronym on the job ladder. [Refolk](/) resolves those searches from public records so you can name the role in plain English and get the right people back.

## Frequently asked questions

### How do I evaluate vendor reliability if I can't trust the SLA number?

Grade the observed record instead of the promise. Pull the vendor's status-page incident history, compute effective availability after subtracting the SLA's exclusion cap, and score seven dimensions: observed uptime, incident frequency, MTTR with p90, severity mix, trend, disclosure latency, and blast radius. The SLA tells you what the vendor will pay for a miss; the incident log tells you how often they miss.

### What is the difference between SLA and actual uptime?

The SLA is a contractual promise with an exclusion list; actual uptime is what your users experience. A 99.9% SLA permits 43.8 minutes of monthly downtime, but scheduled-maintenance carve-outs of 4 to 8 hours per month are excluded from the measurement, so the same headline can deliver closer to 99.5% experienced availability. Always compute effective availability by subtracting the exclusion cap from the promise.

### How much incident history can I actually see on a public status page?

Usually not much. The dominant hosted status-page API returns only the 50 most recent incidents, including unresolved, resolved, and postmortem states. A noisy vendor may expose only a few months. Record the date of the oldest retrievable incident; if your window is under six months, treat the score as provisional and say so in the writeup.

### Should I use the mean or p90 for MTTR when scoring a vendor?

Use p90 and report the single worst incident alongside the mean. The mean hides long tails: a twelve-minute average MTTR can be fifty five-minute blips plus one six-hour outage. For a hard dependency, the tail is the risk you actually carry, so score the recovery distribution, not its center.

### How does depending on multiple vendors change the reliability math?

Availabilities multiply across every required dependency, they do not average. Two independent 99.95% components combine to roughly 99.9% before your own application adds any failures, and four 99.9% dependencies in one path yield about 99.6% end-to-end. Budget your target across every vendor in the request path before you approve any single one.

### Can a vendor edit its public incident history?

Yes. Backfilled and retrospective incidents are a shipped feature on major status-page products, letting a vendor add an incident after the fact and assign it its original timestamp. That means two identical-looking histories can have different integrity. Look for incidents where created_at long precedes display_at and postmortems dated after the event.

---

*From the Refolk guide library. I revise these guides rather than replacing them, so the current version is always at https://www.refolk.ai/guides/vendor-reliability-score*