Measurement methodology

The public method behind every measured claim

Version 2, updated Sep 20, 2026. How we define, date and limit every published number.

We update this method when our measurement rules change. Measurement dates change only when the underlying evidence changes.

What we measure

Every result names the metric it measures. All definitions.

Tokens per second expresses how quickly a serving system produces tokens over an interval.

Time to first token measures elapsed time from request arrival until the first generated token becomes available.

The fiftieth and ninety-ninth percentiles mark different positions in an ordered latency distribution.

Quantization quality recovery is the measured retention of task performance after a model moves to lower precision.

Baseline and candidate runs use the same GPU, request protocol, and task definition.
Engine versions are shown where the source records them. The historical /benchmarks records lack engine and runtime versions and are marked incomplete.
Benchmark records preserve their reported date and available conditions. Missing fields are disclosed; versions from unrelated packages are not substituted.
Quality uses the same protocol and scoring method as its recorded baseline.
Derived cost stays labeled as derived from the recorded rate and measured throughput.
Ratios compare the published baseline and optimized values.
Full definition: Comparison ratios are computed at render time from both absolutes and are never stored.
Cache hit rate describes qualifying cached input, not a typical rate or guarantee.
Full definition: Cache hit rate is the highest qualifying account's cached-input share on paid settled traffic: agent sessions of five or more turns, at least 10 million billed input tokens per account, excluding the SDK canary. Provider-reported cached counts are preferred, billing-derived otherwise; rates are floored to one decimal, published only at 70% or above, and revalidated hourly over 24 hours (7 days only where labeled). A labeled flagship rate can be borrowed only when a model's own gates decline to measure; the published figure is not a typical rate or a guarantee. After a failed refresh, the last cached rate can remain visible.

We publish measured rows, their conditions, dates, source status, and known absences.
An unmeasured combination renders an absence. We do not project it or fill it from a vendor table.
Vendor specifications appear only as labeled citations with a source and retrieval date.
A cited specification never becomes our measured data.
We do not present vendor-copied numbers as our measurements.

Every measured claim carries an as-of or verified date from its owning evidence.
Update cadence follows measurement changes, not deploy timing.
A deploy alone does not advance a measurement date.
The visible method version changes when this public contract changes.

Reproduction requires the engine/version, driver and container, token lengths, concurrency, request rate, warmup, repeat count, and workload dataset. A result missing these details is incomplete.
Package evidence is separate from the historical /benchmarks records. A package version does not establish which runtime produced an older result.
The historical automated load-test records describe serving tests on one GPU, but do not identify the harness or dataset. The A100 record reports batch size 24; the T4 batch size is not recorded.
Comparison rows preserve their source conditions verbatim, including workload, hardware, precision, concurrency, and statistic identity.
The engine-comparison harness and raw data are not publicly available. Those self-reported results cannot currently be independently reproduced from this publication.

Use a workspace API key and pay for input, cached input, and output tokens.

TLS in transit, AES-256 at rest

Workspace isolation

No training on your data

SOC 2 Type II

Every result names the metric it measures. All definitions.

Tokens per second expresses how quickly a serving system produces tokens over an interval.

Time to first token measures elapsed time from request arrival until the first generated token becomes available.

The fiftieth and ninety-ninth percentiles mark different positions in an ordered latency distribution.

Quantization quality recovery is the measured retention of task performance after a model moves to lower precision.

Baseline and candidate runs use the same GPU, request protocol, and task definition.
Engine versions are shown where the source records them. The historical /benchmarks records lack engine and runtime versions and are marked incomplete.
Benchmark records preserve their reported date and available conditions. Missing fields are disclosed; versions from unrelated packages are not substituted.
Quality uses the same protocol and scoring method as its recorded baseline.
Derived cost stays labeled as derived from the recorded rate and measured throughput.
Ratios compare the published baseline and optimized values.
Full definition: Comparison ratios are computed at render time from both absolutes and are never stored.
Cache hit rate describes qualifying cached input, not a typical rate or guarantee.
Full definition: Cache hit rate is the highest qualifying account's cached-input share on paid settled traffic: agent sessions of five or more turns, at least 10 million billed input tokens per account, excluding the SDK canary. Provider-reported cached counts are preferred, billing-derived otherwise; rates are floored to one decimal, published only at 70% or above, and revalidated hourly over 24 hours (7 days only where labeled). A labeled flagship rate can be borrowed only when a model's own gates decline to measure; the published figure is not a typical rate or a guarantee. After a failed refresh, the last cached rate can remain visible.

We publish measured rows, their conditions, dates, source status, and known absences.
An unmeasured combination renders an absence. We do not project it or fill it from a vendor table.
Vendor specifications appear only as labeled citations with a source and retrieval date.
A cited specification never becomes our measured data.
We do not present vendor-copied numbers as our measurements.

Reproduction requires the engine/version, driver and container, token lengths, concurrency, request rate, warmup, repeat count, and workload dataset. A result missing these details is incomplete.
Package evidence is separate from the historical /benchmarks records. A package version does not establish which runtime produced an older result.
The historical automated load-test records describe serving tests on one GPU, but do not identify the harness or dataset. The A100 record reports batch size 24; the T4 batch size is not recorded.
Comparison rows preserve their source conditions verbatim, including workload, hardware, precision, concurrency, and statistic identity.
The engine-comparison harness and raw data are not publicly available. Those self-reported results cannot currently be independently reproduced from this publication.