Serving performance, with records attached.

2 self-reported records, provenance incomplete. How we measure

Last revised Sep 20, 2026

About these records

Historical self-reported measurements with incomplete runtime and workload provenance. These records cannot be independently reproduced from the published details. Cost figures are GPU compute estimates, not all-in bills.

Explore Model APIs

Explore the reported results

About the charts

Switch one view in place. Labels, statistic identity, source, and date stay attached to every chart.

Dataset revision 2026-09-20

Request throughput, Llama 3.1 8B Instruct

NVIDIA A100 (40 GB). Requests per second. Higher is better.

Baseline FP162.24
Batched serving3.061

Source:

RunInfra historical record llama31-8b-a100-2026-07-12, incomplete provenance, measured Jul 12, 2026

What this metric means

Request throughput is the sustained number of requests the endpoint completed under the disclosed workload. Higher is better. The bars preserve the historical baseline and optimized values. Missing runtime and workload details prevent independent reproduction.

Request throughput, Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB). Requests per second. Higher is better.

Baseline FP164.2
Optimized serving8.479

Source:

RunInfra historical record qwen25-05b-t4-2026-07-11, incomplete provenance, measured Jul 11, 2026

What this metric means

Show both records at a glance

Historical records at a glance

Reported values retain their original dates and statistics. The records below disclose missing provenance, method uncertainty, quality, and cost basis.

Llama 3.1 8B Instruct

NVIDIA A100 (40 GB), recorded Jul 12, 2026

Time to first token: 71.12 msp50, batched serving (FP16, batch 24)
Output throughput: 391.76 tok/smeasured, batched serving (FP16, batch 24)
Per 1M output tokens: $1.489derived from recorded GPU rate and measured saturation, batched serving config
Quality vs FP16 baseline: 0.973reported candidate, precision unverified

Historical record llama31-8b-a100-2026-07-12, incomplete provenance

Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB), recorded Jul 11, 2026

Time to first token: 48.82 msp50, reported batched serving, precision unverified
Output throughput: 1,085 tok/smeasured, reported batched serving, precision unverified
Per 1M output tokens: $0.151derived from recorded GPU rate and measured saturation, reported batched serving, precision unverified
Quality vs FP16 baseline: 0.982measured, reported batched serving, precision unverified

Historical record qwen25-05b-t4-2026-07-11, incomplete provenance

Current measured results

Read the historical records

The available public record behind the charts, including missing run details and what the reported results do not establish.

Llama 3.1 8B Instruct

NVIDIA A100 (40 GB), recorded Jul 12, 2026

Method: Reported continuous-batching serving tuning and a separate candidate configuration. Quantization method and runtime kernel are unverified; the previous FP8 dynamic attribution is withdrawn.
Workload: RunInfra's automated load test on a single A100-40GB, identical workload on both sides. Serving statistics were measured at batch size 24. Historical self-reported result, incomplete provenance. Serving engine and version, CUDA/driver, container digest, input/output sequence lengths, concurrency, request rate, warmup, repeats, and load-test dataset are not recorded in this public dataset.
Cost basis: A100-40GB GPU compute at the rate recorded with the run ($2.10/hr). Excludes idle time, CPU, memory, networking, and platform fees. The $1.489 USD per 1M output tokens here, batched serving (FP16, batch 24), measured Jul 12, 2026, and the $0.206 in the H100 BF16 engine dataset (vLLM, concurrency 256, Jun 20, 2026) use different hardware and precision. The A100 engine, input/output lengths, and concurrency are unrecorded, so this is not a like-for-like cost comparison.

Baseline, FP16 serving

Request latency: 1,665.58 msp50, measured
Throughput: 2.24 req/smeasured
VRAM: 35.16 GBmeasured at load

RunInfra optimized, measured per configuration

Time to first token (batched serving): 71.12 msp50, measured
Time to first token (batched serving): 3,472.35 msp99, measured
Inter-token latency (batched serving): 13.2 msp50, measured
Output throughput (batched serving): 391.76 tok/smeasured
Peak VRAM (batched serving): 37 GBmeasured under batch load
Capacity (candidate, precision unverified): 226 req/minmeasured
Cost per request (candidate, precision unverified): $0.155derived from recorded GPU rate and measured saturation

Quality against the recorded baseline: 0.9733, first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline. 1.0 = no detected degradation on this composite under the disclosed workload.

Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.
Serving-tuned (reported FP16, batch 24) and the separate candidate are different configurations. Candidate precision and runtime kernel are unverified; neither this A100 record nor the T4 record establishes native FP8 performance.
Time to first token p99 under saturated batch load is 3472.35 ms; the p50 to p99 spread is the cost of batching for throughput.
Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees.

Historical record llama31-8b-a100-2026-07-12, incomplete provenance, dataset 2026-09-20

Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB), recorded Jul 11, 2026

Method: Reported continuous-batching serving. Quantization method and runtime kernel are unverified; the previous FP8 dynamic attribution is withdrawn.
Workload: RunInfra's automated load test on a single T4, identical workload on both sides. Historical self-reported result, incomplete provenance. Serving engine and version, CUDA/driver, container digest, input/output sequence lengths, concurrency, request rate, warmup, repeats, and load-test dataset are not recorded in this public dataset.
Cost basis: T4 GPU compute at the serverless rate recorded with the run ($0.59/hr). Excludes idle time, CPU, memory, networking, and platform fees.

Baseline, FP16 serving

Request latency: 793.65 msp50, measured
Request latency: 797.55 msp99, measured
Throughput: 4.2 req/smeasured
VRAM: 11.99 GBmeasured at load

RunInfra optimized, reported batched serving, precision unverified

Time to first token: 48.82 msp50, measured
Time to first token: 57.66 msp99, measured
Inter-token latency: 6.95 msp50, measured
Output throughput: 1,085 tok/smeasured
Capacity: 509 req/minmeasured
Peak VRAM: 14.38 GBmeasured under batch load
Candidate request latency (precision unverified): 150.23 msrequest latency, measured; percentile not recorded

Quality against the recorded baseline: 0.9817, first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline. 1.0 = no detected degradation on this composite under the disclosed workload.

Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.
Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees.
T4 does not provide native FP8 compute. This record does not establish which quantization or kernel ran, and must not be cited as an FP8 performance result.

Historical record qwen25-05b-t4-2026-07-11, incomplete provenance, dataset 2026-09-20

Price context, with citations

Public list prices stay separate from historical RunInfra results. Every entry keeps its original source and capture date.

Need a concrete starting point? Open the cost calculator. Choose a model, hardware, and traffic shape. The calculator keeps measured, cited, modeled, and absent states separate.

Llama 3.1 8B is no longer listed on Together's pricing page.

Serverless per-token list prices

Groq, Llama 3.1 8B Instant
Historical rates recorded July 12, 2026, not reverified. Groq now lists Contact Sales for this model (source read September 21, 2026).
$0.05 in / $0.08 out per 1M tokens
console.groq.com, as of Jul 12, 2026
DeepInfra, Llama 3.1 8B Instruct Turbo
$0.02 in / $0.03 out per 1M tokens
deepinfra.com, as of Jul 12, 2026
Groq, Llama 3.3 70B Versatile
Historical rates recorded July 12, 2026, not reverified. Groq now lists Contact Sales for this model (source read September 21, 2026).
$0.59 in / $0.79 out per 1M tokens
console.groq.com, as of Jul 12, 2026
DeepInfra, Llama 3.3 70B Instruct Turbo
$0.10 in / $0.32 out per 1M tokens
deepinfra.com, as of Jul 12, 2026
Together AI, Llama 3.3 70B
$1.04 in / $1.04 out per 1M tokens
www.together.ai, as of Jul 12, 2026

Dedicated GPU and whole-instance list prices

Together AI, H100 80GB, dedicated
$5.49 per GPU-hour
www.together.ai, as of Jul 12, 2026
Fireworks AI, H100 80GB, on-demand
$7.00 per GPU-hour
fireworks.ai, as of Jul 12, 2026
DeepInfra, H100 80GB
$2.20 per GPU-hour
deepinfra.com, as of Jul 12, 2026
Replicate, H100
$5.49 per GPU-hour
replicate.com, as of Jul 12, 2026
AWS EC2, p5.48xlarge (8x H100), us-east-1
Whole-instance SKU (about $6.88 per H100-hour as context; single H100s are not sold separately).
$55.04 per instance-hour
aws.amazon.com, as of Jul 12, 2026
Google Cloud, a3-highgpu-8g (8x H100), us-central1
Whole-instance SKU (about $11.06 per H100-hour as context).
$88.49 per instance-hour
cloud.google.com, as of Jul 12, 2026

What this page does not claim

The gaps stay visible so public pricing context is never mistaken for measured competitor performance.

We have not measured competitor latency or throughput. Provider numbers on this page are their public list prices, nothing else.
Rented per-token prices and our derived compute floor per token are different economic units (pooled utilization vs a GPU you saturate), so we never chart them on one scale.
Per-token prices across vendors are not directly comparable: tokenizers differ, and Anthropic's own pricing documentation notes its newer models' tokenizer produces roughly 30% more tokens for the same text (platform.claude.com pricing docs, as of 2026-07-12).
These historical records have incomplete runtime and workload provenance. They are not complete reproducible receipts and do not establish a verified quantization speedup.

Dataset changelog

Product updates

Sep 20, 2026Marked both historical records as incomplete and withdrew unsupported FP8 dynamic method attribution. Reported values and original measurement dates are unchanged.
Jul 12, 2026Added Llama 3.1 8B Instruct on NVIDIA A100-40GB with its 2026-07-12 measurement date and receipt record.
Jul 12, 2026First public receipt: Qwen2.5 0.5B Instruct on NVIDIA T4, measured 2026-07-11. Provider list prices captured 2026-07-12.

Start building on open models

One API key for the OpenAI and Anthropic SDKs.

Get an API key View pricing

TLS in transit, AES-256 at rest

Workspace isolation

No training on your data

SOC 2 Type II

2 self-reported records, provenance incomplete. How we measure

Serving performance, with records attached.

2 self-reported records, provenance incomplete. How we measure

Last revised Sep 20, 2026

About these records

Explore Model APIs

Dataset changelog

Sep 20, 2026Marked both historical records as incomplete and withdrew unsupported FP8 dynamic method attribution. Reported values and original measurement dates are unchanged.

Jul 12, 2026Added Llama 3.1 8B Instruct on NVIDIA A100-40GB with its 2026-07-12 measurement date and receipt record.

Jul 12, 2026First public receipt: Qwen2.5 0.5B Instruct on NVIDIA T4, measured 2026-07-11. Provider list prices captured 2026-07-12.