Loading compare
H100 BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026: vLLM 5,333 versus TensorRT-LLM 4,813 output tokens/s. Three timed repeats per concurrency without published confidence intervals do not establish a statistically resolved ordering or a universal ranking.
H100 BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026: vLLM 5,333 versus TensorRT-LLM 4,813 output tokens/s. TensorRT-LLM recorded lower TTFT at four of six concurrencies (1, 8, 32, and 64), vLLM at 128 and 256. Same condition: vLLM 2,221 versus TensorRT-LLM 2,848 ms TTFT p50. Derived cost, same condition: vLLM $0.206 versus TensorRT-LLM $0.228 per million output tokens. From three repeats, confidence intervals unpublished. No universal winner is claimed.
Throughput and TTFT were measured on NVIDIA H100 80GB at BF16. Derived cost compared on H100 BF16 only. The comparison uses Llama 3.1 8B Instruct, vLLM 0.23.0 and TensorRT-LLM 1.2.1, as of Jun 20, 2026. RunInfra's later pages record vLLM 0.25.1 in published model packages (read Sep 21, 2026). Those newer versions were not compared in this June sweep. Sources: published model packages.
Our measured latency and throughput leads change with concurrency, while backend scope limits every comparison.
We compare measured throughput and TTFT p50 at every published concurrency, then show derived cost cells at each published configuration. Every ratio sits beside both source values.
Llama 3.1 8B Instruct, NVIDIA H100 80GB, BF16, Jun 20, 2026. Three timed repeats; confidence intervals unpublished. Throughput: mean of repeats. TTFT: p50. Ratios are arithmetic comparisons, not statistical significance.
Concurrency (log scale). vLLM: solid; TensorRT-LLM: dashed. Zero-based vertical scale.
Concurrency (log scale). vLLM: solid; TensorRT-LLM: dashed. Zero-based vertical scale.
Measured throughput and TTFT use ratios from the two absolute values. Cost is derived from the recorded GPU rate; no cross-engine cost ratio is defined.
| Metric and condition | vLLM | TensorRT-LLM | Conditional verdict |
|---|---|---|---|
| Highest measured throughput on NVIDIA H100 80GB at BF16Shared sweep condition belowoutput tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off | vLLM 5,333 output tokens/s | TensorRT-LLM 4,813 output tokens/s | vLLM 9.8% higher; significance unknown Evidence and calculationvLLM recorded 5,333 output tokens/s versus TensorRT-LLM at 4,813 output tokens/s, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The recorded difference is 520 output tokens/s (9.8% of the larger value), from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.11x, an arithmetic comparison only.derived at render time from the two measured throughput values at concurrency 256, output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off |
| TTFT p50 at concurrency 1Shared sweep condition belowp50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | vLLM 35 ms | TensorRT-LLM 32 ms | TensorRT-LLM 8.6% lower; significance unknown Evidence and calculationvLLM recorded 35 ms versus TensorRT-LLM at 32 ms, under NVIDIA H100 80GB, BF16, concurrency 1, Llama 3.1 8B Instruct, as of Jun 20, 2026. The recorded difference is 3 ms (8.6% of the larger value), from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. |
| TTFT p50 at concurrency 8Shared sweep condition belowp50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | vLLM 177 ms | TensorRT-LLM 56 ms | TensorRT-LLM 68.4% lower; significance unknown Evidence and calculationvLLM recorded 177 ms versus TensorRT-LLM at 56 ms, under NVIDIA H100 80GB, BF16, concurrency 8, Llama 3.1 8B Instruct, as of Jun 20, 2026. The recorded difference is 121 ms (68.4% of the larger value), from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 3.16x, an arithmetic comparison only.derived at render time from the two measured latency values at concurrency 8, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 32Shared sweep condition belowp50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | vLLM 514 ms | TensorRT-LLM 235 ms | TensorRT-LLM 54.3% lower; significance unknown Evidence and calculationvLLM recorded 514 ms versus TensorRT-LLM at 235 ms, under NVIDIA H100 80GB, BF16, concurrency 32, Llama 3.1 8B Instruct, as of Jun 20, 2026. The recorded difference is 279 ms (54.3% of the larger value), from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 2.19x, an arithmetic comparison only.derived at render time from the two measured latency values at concurrency 32, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 64Shared sweep condition belowp50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | vLLM 790 ms | TensorRT-LLM 504 ms | TensorRT-LLM 36.2% lower; significance unknown Evidence and calculationvLLM recorded 790 ms versus TensorRT-LLM at 504 ms, under NVIDIA H100 80GB, BF16, concurrency 64, Llama 3.1 8B Instruct, as of Jun 20, 2026. The recorded difference is 286 ms (36.2% of the larger value), from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.57x, an arithmetic comparison only.derived at render time from the two measured latency values at concurrency 64, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 128Shared sweep condition belowp50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | vLLM 1,658 ms | TensorRT-LLM 1,691 ms | vLLM 2.0% lower; significance unknown Evidence and calculationvLLM recorded 1,658 ms versus TensorRT-LLM at 1,691 ms, under NVIDIA H100 80GB, BF16, concurrency 128, Llama 3.1 8B Instruct, as of Jun 20, 2026. The recorded difference is 33 ms (2.0% of the larger value), from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. |
| TTFT p50 at concurrency 256Shared sweep condition belowp50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | vLLM 2,221 ms | TensorRT-LLM 2,848 ms | vLLM 22.0% lower; significance unknown Evidence and calculationvLLM recorded 2,221 ms versus TensorRT-LLM at 2,848 ms, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The recorded difference is 627 ms (22.0% of the larger value), from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.28x, an arithmetic comparison only.derived at render time from the two measured latency values at concurrency 256, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| USD per 1M output tokens, derived from the recorded GPU rateNVIDIA H100 80GB, BF16, recorded cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026Derived from the recorded GPU rate and measured throughput at the recorded cost concurrency, not measured directly. | vLLM $0.206 BF16, derived | TensorRT-LLM $0.228 BF16, derived | 9.6% rounded cost gap Cost basisPublished derived costs: vLLM $0.206 versus TensorRT-LLM $0.228, under NVIDIA H100 80GB, BF16, recorded cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The gap between these rounded derived costs is 9.6% of the larger cost. The published throughput inputs differ by 9.8% of the larger mean. The throughput method uses three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. |
| USD per 1M output tokens, derived from the recorded GPU rateNVIDIA H100 80GB, FP8, recorded cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026Derived from the recorded GPU rate and measured throughput at the recorded cost concurrency, not measured directly. | vLLM $0.158 FP8, derived | TensorRT-LLM TensorRT-LLM at fp8 was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100. | Comparison unavailable Cost basisWe do not calculate a ratio because one published cost cell is absent. |
| USD per 1M output tokens, derived from the recorded GPU rateNVIDIA L40S 48GB, BF16, recorded cost concurrency 128, Llama 3.1 8B Instruct, as of Jun 20, 2026Derived from the recorded GPU rate and measured throughput at the recorded cost concurrency, not measured directly. | vLLM $0.425 BF16, derived | TensorRT-LLM TensorRT-LLM on the L40S was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100. | Comparison unavailable Cost basisWe do not calculate a ratio because one published cost cell is absent. |
We keep measured throughput on its own scale. Derived cost never shares this chart.
We separate one shared prefix from many distinct prefixes because the source scopes those workloads differently.
We did not measure TensorRT-LLM's prefix cache. Both prefix experiments ran vLLM and SGLang only.
TensorRT-LLM 1.2.1 ran the PyTorch backend, its only execution backend. NVIDIA's TensorRT-LLM 1.2 release notes document removal of the TensorRT backend and engine-build CLI.
We measured throughput and TTFT only for Llama 3.1 8B Instruct on NVIDIA H100 80GB at BF16. Derived cost is compared on H100 BF16 only. Single-engine cost rows do not establish a comparison. We did not test engine versions newer than vLLM 0.23.0 and TensorRT-LLM 1.2.1 on the Jun 20, 2026 as-of date.
The higher recorded throughput mean at the highest measured point was for vLLM. vLLM recorded 5,333 output tokens/s; TensorRT-LLM recorded 4,813 output tokens/s. The difference is 9.8% of the larger mean. Three timed repeats, confidence intervals unpublished; statistical significance is unknown. Condition: NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026.
The lower recorded TTFT p50 mean was for vLLM. vLLM recorded 2,221 ms; TensorRT-LLM recorded 2,848 ms. Three timed repeats, confidence intervals unpublished; statistical significance is unknown. Condition: NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026.
Published derived costs: vLLM $0.206; TensorRT-LLM $0.228 USD per 1M output tokens. Condition: NVIDIA H100 80GB, BF16, recorded cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. Both values are derived from the recorded GPU rate and measured throughput at the recorded cost concurrency. The gap between these rounded derived costs is 9.6% of the larger cost. The published throughput inputs differ by 9.8% of the larger mean. The throughput method uses three timed repeats; confidence intervals are unpublished, so statistical significance is unknown.
Use a workspace API key and pay for input, cached input, and output tokens.
View Model APIs