# The Open-Weight Model Selection Playbook: Self-Host, Route, or Stay on API

*You will go from a written task contract to a license-and-hardware-gated shortlist, an eval on your own data, and a defensible self-host, route, or stay-on-API verdict.*

- Canonical URL: https://www.refolk.ai/guides/open-weight-model-selection-playbook
- Pillar: Engineering and open source
- Format: Playbook
- Published: 2026-09-27
- Last reviewed: 2026-09-27
- Reading time: 17 min
- Keywords: how to evaluate open-weight LLM before production, self-host LLM model selection, build an LLM eval set for a workload, open-weight model license commercial use, cost per successful task LLM

## Key takeaways

- License and hardware, not benchmark rank, kill most candidates first: MiniMax's commercial-agreement requirement applies from dollar one, not at scale, so gate on the license text before you eval.
- The right cost comparison is the same model on a hosted endpoint, not a frontier API. Raw GPU cost of about $0.10 per million tokens loses to hosted endpoints serving the identical model from $0.17 to $0.60 per million.
- A curated eval of 50 to 200 real cases, oversampling hard and adversarial ones, out-diagnoses a thousand synthetic rows, because coverage drives diagnostic power, not row count.
- Reproducibility and correctness are different problems: one Qwen model produced 80 unique completions in 1,000 temperature-zero attempts, collapsing to 1 with batch-invariant kernels.
- Talent supply can gate the stack choice. In Refolk's index there are 893 US professionals listing vLLM against 66 with TensorRT-LLM, a roughly 13.5x depth gap that makes vLLM the safer default for staffability.
- Utilization dominates break-even more than GPU sticker price: at 10% utilization every self-hosted number is 10x worse and all-in cost runs 3 to 5x the raw GPU bill.

You need to pick one open-weight model to self-host for a specific production task and hand your team and procurement a verdict they can attack and you can defend. This playbook is for engineering managers, technical founders, and infra leads running that decision once, properly. It orders the work the way the decision actually collapses: license and hardware gate the shortlist, then your own data ranks it, then serving and cost decide whether you self-host at all.

Most guides lead with a leaderboard. That is the expensive mistake. A model's rank is irrelevant if its license forbids your use or its weights do not fit your GPU, and you will have spent your eval budget benchmarking things you can never ship. This document reverses the order on purpose.

## Why order the work license-first, not leaderboard-first

License and hardware, not quality, eliminate most candidates before a single benchmark runs. Gate on them first and you spend your eval budget only on models you could actually deploy.

The mechanism is simple. A usage threshold or a bespoke commercial clause makes a model unusable regardless of how well it scores. MiniMax's commercial-agreement requirement, for example, applies from dollar one, not at scale, so read before you build. Ranking a model you can never legally ship is pure waste. The same is true of hardware: a model that will not fit your target VRAM is out no matter what a leaderboard says.

#### The selection pipeline, in the order it actually decides

1. **Task contract** - Write the workload spec, scoring unit, and pass thresholds
2. **License and hardware gate** - Drop anything unshippable or that will not fit VRAM
3. **Eval on your data** - Score the survivors on frozen, real cases
4. **Serving under load** - Load-test the top one or two on your engine
5. **Cost and verdict** - Compute cost-per-successful-task, then self-host, route, or stay

*License and hardware gate the shortlist before any benchmark runs, which is the reverse of ranking-first blog posts.*

There is a second reason to front-load the license read. The words "open source" and "open weight" are not the same thing. Under OSAID 1.0, an open-source AI system must provide model parameters, complete training and inference code, and sufficiently detailed information about training data under OSI-approved terms. An open-weight model releases trained weights but may withhold some of those components, so many models marketed as open source are more accurately open weight. That distinction is not pedantry - it is the difference between a license you can rely on and one you have to read line by line.

## What a defensible task contract contains

A task contract is a written spec that names the one workload, defines what a correct output is, and sets the pass thresholds - all before you touch a model. It is the artifact everything downstream depends on.

Write it first because it forces the hard decision that every eval quietly depends on: the scoring unit. State exactly what a correct output is and how one is judged - exact match, schema-valid, numeric tolerance, or a rubric - and resolve ambiguity before you write a single metric. If two reviewers would score the same output differently, the contract is not finished.

Then define the criteria. Quality is not one number; it is 3 to 5 orthogonal criteria that move independently. Score with a mix of rule-based checks where the answer is deterministic, LLM-as-judge for the fuzzy dimensions, and sampled human review to keep the judge honest. Deterministic checks are preferred wherever the task allows them, because they never drift.

> **Rule:** The scoring unit is decided before you see any model
>
> Name what a correct output is - exact match, schema-valid, numeric tolerance, or a rubric - in the contract, not after you have seen a model's output. Deciding it afterwards is how a model gets graded to pass.

## The license and hardware gate

Check the declared license on the exact checkpoint, confirm it fits your commercial scale and redistribution needs, and confirm the weights fit your target VRAM. Anything that fails either gate is dropped before it reaches the eval set.

Read the license at the checkpoint level, not the family level. Different checkpoints in one family can carry different terms. Hugging Face model cards declare a machine-readable license field in YAML metadata that you can filter on, but treat that field as a pointer, not a conclusion. An identifier of "other" with a custom license name - as Qwen, Kimi, and MiniMax's M2 use - signals a bespoke text you must read in full before deployment.

The tiers you will meet, and how each one bites:

| License tier | Examples and mechanism | What to check |
|---|---|---|
| Permissive | Apache 2.0, MIT allow commercial use, often with attribution | Apache and OpenMDW grant patents; MIT is silent on patents |
| Copyleft | GPL-3.0 requires source sharing for derivative works | Whether your deployment triggers redistribution |
| Gated / threshold | Llama uses MAU tests; some Qwen checkpoints require an ongoing license request | Whether you will cross the threshold in the deployment's life |
| Bespoke agreement | Some Kimi and MiniMax releases impose attribution or separate-agreement duties | MiniMax applies its agreement from dollar one, not at scale |

Two traps hide here. First, the patent question: MIT says nothing about patents, while Apache 2.0 and OpenMDW grant them explicitly, which is a real reason a cautious business prefers Apache. Second, heritage: an Apache 2.0 model fine-tuned on non-commercial base weights may not be legally safe for profit, so trace the lineage rather than trusting the top-level card.

On the hardware side, quantization is your lever and your trap. GGUF is a container, not a quantization - it can hold unquantized F16 weights just as well as Q4_K_M - so read the model card, not the filename. Quantization pays off: an 8B model at about 16GB in F16 lands around 4.5GB after Q4_K_M, roughly 72% smaller. And QLoRA reaches further than most teams expect, quantizing a model to 4 bits then training with LoRA, which allows fine-tuning a 65B model on a single 48GB GPU. Confirm the serving format too: safetensors is the standard storage format that engines like vLLM expect by default.

## Building and freezing the eval set

Build a small, curated set of real cases from your production traces, oversample the hard and adversarial ones, and freeze it. Coverage, not row count, drives diagnostic power.

The size guidance from practitioners who publish their methods clusters tightly:

| Source | Recommended count | Note |
|---|---|---|
| Respan | 50 to 200 | Focused eval |
| AgentsCamp | 20 to 50 | Oversample hard cases |
| AI/TLDR | 50 to 100, grow to a few hundred | CI eval maturing |
| LangSmith writeup | 50 to 100 | Curated input-output pairs |

The best dataset comes from reality: convert production traces directly into eval examples, turning failures into permanent test cases. Small wins because interpretation gets easier - with 50 examples a 2% swing is one case - and because each failure category needs only about five representatives to become diagnostic. What you want is a map of tasks, segments, and failure modes, each represented well enough to score, not a pile of easy in-distribution rows.

**50-200 - Curated examples for a focused workload eval**

Quality of coverage matters more than size; a first pass can start at 20 to 50 real, hard cases.

For any high-risk slice, size up. A critical slice needs enough examples to move its score out of statistical noise, often a few hundred. And guard the set: the moment an eval set leaks it becomes training data for the next model and stops measuring anything. Keep it private and versioned.

> **Watch out:** Padding hides the failures you built the set to find
>
> Adding easy in-distribution rows inflates the average and buries the hard cases. If your set's mean looks great but a known failure category is under-represented, you are measuring the wrong thing well.

## Pinning the run so results mean something

Pin everything that changes an output - model revision, tokenizer, chat template, quantization, engine commit, drivers, sampling params, dataset version - so a re-run reproduces your baseline within a stated noise band. Reproducibility and correctness are separate problems, and pinning solves only the first.

The full list to pin or record: model and tokenizer revisions, chat template, quantization artifacts, inference-engine commit, CUDA and driver versions, kernel libraries, GPU topology, precision, and scheduler configuration. Control or at least record batch size, concurrency, request order, speculative decoding, caching, tensor parallelism, and prefill behavior. Seed and temperature help but do not finish the job: no combination of temperature and seed gave deterministic answers for all models, though temperature 0 with a fixed seed reduced variability for most.

The scale of the drift is easy to underestimate. One report describes a Qwen model generating 80 unique completions across 1,000 temperature-zero attempts under default settings; with batch-invariant kernels enabled, all 1,000 matched. LLM-as-judge is no safer: even at temperature 0 with top_k 1, a couple of borderline items stayed non-reproducible across hundreds of calls.

> A pinned but production-mismatched eval measures the wrong system perfectly consistently.

Here is the decision the sources genuinely disagree on. Pinning enforce_eager and max_num_seqs=1 gives a clean measurement of the model, but it means your eval environment no longer matches production serving conditions - you are measuring the model, not the production system. Decide by your goal: measure the model clean when you are ranking candidates, then re-measure at production concurrency before you trust the numbers for a go-live decision.

At this point you have survivors that are legal, that fit, and that you can score reproducibly. The remaining work - ranking on your data, then load and cost - is where a good search saves you weeks, because the scarce input is people who have done this before.

I ran this search: `Find ML engineers in the US who have shipped vLLM in production and worked on continuous batching or KV-cache tuning.` - [see the full result list](https://www.refolk.ai/s/kg1e3jng2x).

*Returns named engineers with hands-on serving experience, so you can staff the load-test and serving stage rather than learning it live.*

## The procedure, start to finish

Run these eight steps in order. Each has an owner and a rough duration; a small team can complete the whole pass in roughly two working weeks.

#### Open-weight model selection, end to end

1. **Write the task contract** - Define the one workload, the scoring unit, and pass thresholds before touching a model. State what a correct output is - exact match, schema-valid, numeric tolerance, or rubric - and resolve ambiguity first. Owner: EM or tech lead, about half a day.
2. **Gate the shortlist on license and hardware** - Check the declared license on each candidate checkpoint and confirm commercial-scale and redistribution fit; drop dollar-one agreements and thresholds you will cross. Confirm VRAM fit. Owner: EM plus legal, about a day. Done: 3 to 6 usable candidates.
3. **Build and freeze the eval set** - Collect real inputs from production traces, oversample hard and adversarial cases, and freeze a versioned held-out set with expected behaviour per case. Keep it private. Owner: PM or domain expert plus eng, 2 to 3 days.
4. **Pin the run environment** - Pin model revision, tokenizer, chat template, quantization, engine commit, drivers and CUDA, sampling params, and dataset version. Decide clean-model versus production-concurrency by your goal. Owner: ML or infra eng, about a day.
5. **Score candidates on your data** - Run each candidate with rule-based checks, LLM-as-judge, and sampled human review; run each prompt N times, typically 3 to 10, and report mean plus standard deviation. Owner: ML eng, 2 to 3 days. Done: a ranked table across 3 to 5 criteria.
6. **Measure serving under load** - Stand up the top 1 to 2 on vLLM or equivalent, enable continuous batching and paged attention, and load-test p99 latency, throughput, and KV-cache pressure. Owner: infra eng, about 2 days. Done: p99 at target concurrency with no sustained preemption.
7. **Compute cost-per-successful-task and break-even** - Combine measured throughput with GPU lease price and the 1.3x to 2.0x ops multiplier; compare against the same model on a hosted endpoint at equal successful-task volume. Owner: EM, about a day.
8. **Write the verdict** - Assemble license, hardware-fit, quality, cost, and serving evidence into one defensible recommendation for procurement, citing your own run artifacts. Owner: EM, about half a day. Done: a one-page verdict.

When you score in step five, report distributions, not points. Run each prompt N times, typically 3 to 10, and report mean plus standard deviation, not a single score. A candidate that wins on the mean but has a fat tail on a critical criterion is not a winner.

## Serving, cost, and the honest comparison

Continuous batching plus PagedAttention are the throughput mechanisms that separate a model that runs from one that serves. Together they yield two to four times the throughput of naive serving, and paged attention cuts KV memory waste by up to 96%.

A production-grade engine gives you state-of-the-art serving throughput, PagedAttention KV memory management, continuous batching, chunked prefill, prefix caching, and flexible CUDA graph execution. During the load test, watch two signals. Track the end-to-end request latency histogram including queue time at p99 against your budget. And treat sustained gpu_cache_usage_perc above 90% as the point where the server is approaching its KV cache limit and will begin preempting sequences - a preempting server is one that will fail under real traffic.

Now the cost decision, and the trap inside it. Break-even estimates disagree by roughly 50x depending on what you compare against:

| Compared against | Break-even | Note |
|---|---|---|
| Premium API | 5 to 10M tokens/month | Easiest bar to clear |
| Budget API | 50 to 100M tokens/month | Much higher bar |
| Frontier list price, steady load | 50 to 100M tokens/day | Clearly ahead past 100M/day if quality holds |
| Same model, hosted endpoint | Near-saturation only | Margin is a few times, not 100x |

The load-bearing row is the last one. At high-concurrency batching, a single H100 serving gpt-oss-120b gets raw GPU cost down to roughly $0.10 per million tokens - but hosted endpoints serve the identical model for $0.60 per million output tokens and as little as $0.17. That is a margin of a few times, not the hundredfold the hourly-rate arithmetic suggests. Compute cost-per-successful-task, not cost-per-token, and multiply the raw GPU cost by 1.3x to 2.0x to get the true total cost of self-hosting.

> **Tip:** Compare against the same model, then check utilization
>
> Before you believe a self-host saving, price the identical model on a real hosted endpoint. Then re-run the math at your actual utilization: at 10% utilization every self-hosted number is 10x worse.

One factor most cost models omit entirely is staffing. Serving needs scarce skills, and the stack you pick narrows your hiring pool. In Refolk's index of professional profiles there are 893 US professionals listing vLLM as a skill against just 66 with TensorRT-LLM - vLLM talent is roughly 13.5x deeper.

| Skill / market | Count in Refolk's index | Derived |
|---|---|---|
| vLLM, US | 893 | Baseline |
| vLLM, Germany | 90 | US is about 9.9x deeper |
| TensorRT-LLM, US | 66 | vLLM about 13.5x deeper in US |
| LLM Evaluation, US | 933 | Slightly larger pool than vLLM US |

Read that as a default: vLLM is the safer choice for staffability. If your verdict depends on a rarer stack, factor the hiring difficulty into the cost, and use [Refolk](/) to check whether the people who can run it are reachable before you commit.

## How this goes wrong

The failure modes below are where defensible-looking selections quietly break. Each has a false positive and a check.

- **Leaderboard-first shortlisting.** Picking the top-ranked model skips the gates. False positive: an Apache 2.0 model card on a checkpoint whose base weights are non-commercial. Check heritage, because a permissive fine-tune of non-commercial weights may not be legally safe for profit.
- **Trusting the family, not the checkpoint.** Different checkpoints in one family carry different licenses. Check the declared YAML license field per checkpoint, and read the full text whenever the identifier is "other".
- **Padding the eval set.** Easy in-distribution rows inflate the average and hide failures. Check for a map of tasks, segments, and failure modes, each represented well enough to score.
- **Assuming temperature 0 means reproducible.** Single-run verdicts present noise as signal. Run N rollouts and report variance; a harness that reports a single-run verdict can present noise as a safety property.
- **Eval environment not matching production.** Measuring at batch size 1 misses the batch effects users hit. Pinning max_num_seqs=1 means you are measuring the model, not the production system, so re-measure at production concurrency before go-live.
- **Same-model hosted API undercuts your math.** False positive: 100x savings from hourly-rate arithmetic. Check against a real hosted price for the identical model; a DeepSeek-class 671B MoE run yourself loses to hosted pricing at every utilization level.
- **Ignoring idle GPU time.** Break-even assumes high utilization. At 10% utilization every self-hosted number is 10x worse and all-in cost runs 3 to 5x the raw GPU bill.
- **Reading the filename, not the card.** A "GGUF" or "MLX" label tells you nothing about precision or provenance. Read the model card, not the filename.

#### The self-host, route, or stay decision

Horizontal axis runs from Low steady volume to High steady volume. Vertical axis runs from Open model misses quality bar to Open model meets quality bar.

| Quadrant | What it means |
| --- | --- |
| Misses bar, low volume | Stay on API; revisit when a better open checkpoint ships |
| Misses bar, high volume | Route: hosted open-weight endpoint or a stronger API for the hard slice |
| Meets bar, low volume | Route to a hosted endpoint of the same model; self-hosting sits idle |
| Meets bar, high volume | Self-host if utilization stays high and you can staff the serving stack |

*Two axes decide the verdict: whether an open model clears your quality bar, and whether your steady volume justifies the fixed serving bill.*

## The pre-verdict checklist

Run this before you write the one-page recommendation. Every item should be answerable from your own artifacts, not from a public benchmark.

#### Before you call the verdict

- [ ] Every candidate's license was read on the exact checkpoint, and none carries a threshold you will cross or a dollar-one agreement you have not signed
- [ ] Each candidate fits target VRAM at the precision you will actually serve, confirmed from the model card, not the filename
- [ ] The eval set is real, frozen, versioned, private, and covers each failure category with enough examples to be diagnostic
- [ ] Scores report mean plus standard deviation across N rollouts, not single-run numbers
- [ ] The top candidate was load-tested at production concurrency with p99 latency inside budget and no sustained KV-cache preemption
- [ ] Cost-per-successful-task uses the 1.3x to 2.0x ops multiplier and is compared against the same model on a real hosted endpoint at your actual utilization
- [ ] The verdict cites your own run artifacts and states its limits, including where evidence is thin

## Keeping the verdict current

A model verdict is a snapshot, not a standard, because checkpoints, prices, and hosted endpoints all move. Write the verdict so it can be re-run rather than re-argued.

Three things decay fastest. License terms change between checkpoints, so re-read the card on any version bump. Hosted-endpoint prices for the same model fall, which shifts your break-even, so re-price against a live endpoint each quarter rather than trusting a number in the doc. And your own traffic mix drifts, so fold new production failures back into the frozen eval set as permanent cases - the set that catches next quarter's regression is the one you keep feeding.

State the limits plainly in the verdict itself. The tie between latency percentiles, GPU memory, and cost-per-successful-task into a single scorecard is practitioner convention, not a published standard, so present it as your team's method and show the inputs. A verdict that names what it does not know is the one procurement can trust.

## Frequently asked questions

### How many examples do I need to evaluate an open-weight LLM before production?

Aim for a curated 50 to 200 real cases for a focused eval, with a first-pass floor around 20 to 50 that oversamples the hard and adversarial inputs. Quality of coverage matters more than row count. With 50 examples a 2% swing is a single case, which is easy to interpret. Grow to a few hundred once the suite matures, and reserve a few hundred specifically for any critical, high-risk slice so its score clears statistical noise.

### At what volume does self-hosting an LLM beat an API?

It depends entirely on the comparison. Against premium APIs, break-even typically lands at 5 to 10 million tokens per month; against budget APIs, 50 to 100 million per month. Against frontier list prices at steady load, one analysis puts it at 50 to 100 million tokens per day. But the honest comparison is the same model on a hosted endpoint, where the margin is a few times at best, and utilization below full saturation erases it fast.

### Does setting temperature to 0 make my eval reproducible?

No. Temperature 0 with a fixed seed reduces variability but does not guarantee identical output. One Qwen model produced 80 unique completions across 1,000 temperature-zero attempts under default settings; only batch-invariant kernels collapsed all 1,000 to a single answer. Report mean plus standard deviation across N rollouts rather than a single-run verdict, or you risk presenting noise as a safety property.

### How do I check whether an open-weight model is safe for commercial use?

Read the declared license field on the exact checkpoint, not the family, and read the full text if the identifier is 'other'. Permissive licenses like Apache 2.0 and MIT allow commercial use; Apache and OpenMDW also grant patents explicitly while MIT is silent. Watch for usage-scale thresholds and dollar-one agreements, and trace heritage, since a permissively licensed fine-tune of non-commercial base weights may not be safe to ship.

### What serving features actually matter for production self-hosting?

Continuous batching plus PagedAttention are the core throughput mechanisms, together yielding two to four times the throughput of naive serving, with paged attention cutting KV memory waste by up to 96%. Add chunked prefill, prefix caching, and CUDA graph execution. In production, watch the p99 end-to-end latency histogram including queue time, and treat sustained GPU cache usage above 90% as the point where the server starts preempting sequences.

---

*From the Refolk guide library. I revise these guides rather than replacing them, so the current version is always at https://www.refolk.ai/guides/open-weight-model-selection-playbook*