Serving engine

TensorRT-LLM: what RunInfra measured

TensorRT-LLM 1.2.1 on NVIDIA H100 80GB BF16: 4,813 output tokens/s and 2,848 ms TTFT p50 at concurrency 256 (Jun 20, 2026). Llama 3.1 8B Instruct, three timed repeats, confidence intervals unpublished. Throughput is the mean; TTFT is p50. Sweep, cost, prefix-cache results and published packages follow.

Measurement scope

RunInfra measured TensorRT-LLM 1.2.1 on Llama 3.1 8B Instruct with NVIDIA H100 80GB at BF16 across 6 published concurrency points, as of Jun 20, 2026. The page reports measured output throughput and TTFT, keeps derived cost on its separate source basis, and keeps published conditions attached to every row. No published catalog package currently serves on this engine, so comparison evidence is not presented as package availability.

TensorRT-LLM's PyTorch-backend run makes backend scope part of every throughput and latency reading.

Source article and method

Head-to-head comparisons

The comparison evidence stays inside one measured scope

Measured throughput and TTFT remain separate from derived cost. Every comparison ratio shows both source values and its full condition.

Scroll horizontally for all columns.

Output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off. Llama 3.1 8B Instruct, NVIDIA H100 80GB, BF16, Jun 20, 2026. Three timed repeats; confidence intervals unpublished. p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column.
Concurrency	Measured throughput	Measured TTFT p50	Full condition
1	152 output tokens/s	32 ms	Concurrency 1, shared sweep condition above
8	1,027 output tokens/s	56 ms	Concurrency 8, shared sweep condition above
32	2,978 output tokens/s	235 ms	Concurrency 32, shared sweep condition above
64	4,029 output tokens/s	504 ms	Concurrency 64, shared sweep condition above
128	4,774 output tokens/s	1,691 ms	Concurrency 128, shared sweep condition above
256	4,813 output tokens/s	2,848 ms	Concurrency 256, shared sweep condition above

Pairwise readings at the highest measured concurrency

output throughput: TensorRT-LLM 4,813, vLLM 5,333 output tokens/s. 9.8% gap; significance unknown.
Evidence and calculation
Recorded means for output throughput: vLLM recorded 5,333 output tokens/s versus TensorRT-LLM at 4,813 output tokens/s, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 9.8% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.11x.derived at render time from the two measured throughput values at concurrency 256, output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off
TTFT p50: TensorRT-LLM 2,848, vLLM 2,221 ms. 22.0% gap; significance unknown.
Evidence and calculation
Recorded means for TTFT p50: vLLM recorded 2,221 ms versus TensorRT-LLM at 2,848 ms, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 22.0% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.28x.derived at render time from the two measured latency values at concurrency 256, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column
output throughput: TensorRT-LLM 4,813, SGLang 5,235 output tokens/s. 8.1% gap; significance unknown.
Evidence and calculation
Recorded means for output throughput: SGLang recorded 5,235 output tokens/s versus TensorRT-LLM at 4,813 output tokens/s, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 8.1% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.09x.derived at render time from the two measured throughput values at concurrency 256, output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off
TTFT p50: TensorRT-LLM 2,848, SGLang 2,543 ms. 10.7% gap; significance unknown.
Evidence and calculation
Recorded means for TTFT p50: SGLang recorded 2,543 ms versus TensorRT-LLM at 2,848 ms, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 10.7% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.12x.derived at render time from the two measured latency values at concurrency 256, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column

Derived cost stays on its own basis

Scroll horizontally for all columns.

derived from the GPU hourly price and the measured saturation throughput for that configuration, not measured. Formula: cost per 1M tokens equals the GPU hourly price divided by sustained tokens per hour. These are reported derived costs. Underlying FP8 and L40S throughput rows are not published; rounded costs do not establish measured throughput or a speedup.
Configuration	Published value or absence	Full condition
NVIDIA H100 80GB BF16	$0.228 USD per 1M output tokens, derived	NVIDIA H100 80GB, BF16, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026
NVIDIA H100 80GB FP8	TensorRT-LLM at fp8 was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100.	NVIDIA H100 80GB, FP8, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026
NVIDIA L40S 48GB BF16	TensorRT-LLM on the L40S was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100.	NVIDIA L40S 48GB, BF16, source-selected cost concurrency 128, Llama 3.1 8B Instruct, as of Jun 20, 2026

Prefix-cache evidence depends on the workload

We did not measure TensorRT-LLM's prefix cache. Both prefix experiments ran vLLM and SGLang only.

One client drives all three engines with identical request streams, so the timing definitions are the same everywhere. Time to first token is the time to the first streamed token. Throughput is output tokens per second. Warm up, then three timed repeats per operating point, charted as the mean. Raw repeats and confidence intervals are not publicly available. Prefix caching is off for the unique-prompt sweeps. Concurrency swept from 1 to 256 at 1,024 input and 256 output tokens, unique prompts, prefix caching off. Same weights and same request stream across engines, single GPU, tensor-parallel size 1. 0 published package rows retain their recorded engine version, concurrency, and verified date.

Where package measurements are listed: We report package measurements here, and each package remains subject to its listed license.

Citation: RunInfra (2026). Source article and method, Jun 20, 2026..

Published packages keep their own measured conditions

These rows come from the published catalog, not the comparison sweep. Each package retains its own model, engine version, concurrency, and verified date.

No published package currently serves on this engine.

The limitations bound every result

TensorRT-LLM 1.2 removed the TensorRT backend; PyTorch is the only execution backend in the tested 1.2.1 release. Our single-prefix cache test is not the many-distinct-prefix case SGLang's RadixAttention is designed for, so we ran that separately too. These results apply only to the tested configurations.
One H100 80GB and one L40S 48GB, single GPU, tensor-parallel size 1.
Same Llama-3.1-8B-Instruct weights and same request stream across engines. Three timed repeats. Raw repeats and confidence intervals are not publicly available.
Latency here is TTFT and per-request percentiles, not goodput at a fixed SLO. Whether first-token latency is acceptable depends on your use case; an SLO-goodput sweep is future work.
Numbers are self-reported on our harness and are not cross-calibrated against an external suite like MLPerf. The harness and raw data are not publicly available, so this publication does not support independent reproduction.
TensorRT-LLM 1.2.1 ran the PyTorch backend, its only execution backend. NVIDIA's TensorRT-LLM 1.2 release notes document removal of the TensorRT backend and engine-build CLI.
The 4,813 tokens/s result is the highest throughput reported for TensorRT-LLM 1.2.1 in this sweep, not a ceiling for other workloads or configurations.
The L40S sweep stopped at concurrency 128 (its saturation), the H100 at 256.
Engines move weekly. These numbers are vLLM 0.23.0, SGLang 0.5.13, TensorRT-LLM 1.2.1, as of Jun 20, 2026.

What is not measured

No other model is measured in this comparison dataset. It covers Llama 3.1 8B Instruct.
No GPU outside NVIDIA H100 80GB and NVIDIA L40S 48GB is represented in the dataset.
No engine version newer than TensorRT-LLM 1.2.1 is measured for this page's comparison evidence.
No published fp8 throughput or latency rows. The source publishes fp8 only as derived GPU cost per 1M output tokens at saturation for vLLM and SGLang; its underlying throughput is not published. These cost cells do not establish an fp8 speed ranking.
No published L40S throughput or latency rows. L40S appears only in derived GPU cost cells for vLLM and SGLang; their underlying throughput is not published. Every published unique-prompt sweep row here is H100 bf16.
No comparison with older TensorRT-LLM backends. The tested version 1.2.1 supports only PyTorch; the TensorRT backend was removed in version 1.2.
No prefix-cache latency values. The source states one engine had lower first-token latency on the many-prefix workload but publishes no latency numbers for either prefix experiment, so that comparison stays qualitative.
No hardware or precision label for the single-shared-prefix sweep. Its caption names only the concurrency, so those rows must not be presented under a GPU or a precision the source does not state.
No goodput at a fixed service-level objective, and no accuracy or output-quality comparison between engines. This dataset is throughput, first-token latency, and derived cost only.
No cross-engine or cross-precision ratio is stored. Pages derive every multiplier and percentage from the absolute pair at render time and print both absolute values beside it.
Not cross-calibrated against an external benchmark suite. These are self-reported numbers; the harness and raw data are not publicly available for independent reproduction.
No claim that these results still hold. The dataset is pinned to one as-of date and one engine version triple, and engines change on a weekly cadence.

The same evidence answers common engine questions

How did TensorRT-LLM compare with vLLM on throughput at the highest measured concurrency?

Recorded means for output throughput. TensorRT-LLM recorded 4,813 output tokens/s; vLLM recorded 5,333 output tokens/s. The difference is 9.8% of the larger mean. Three timed repeats, confidence intervals unpublished; statistical significance is unknown. Condition: NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026.

How did TensorRT-LLM compare with SGLang on TTFT p50 at the highest measured concurrency?

Recorded means for TTFT p50. TensorRT-LLM recorded 2,848 ms; SGLang recorded 2,543 ms. The difference is 10.7% of the larger mean. Three timed repeats, confidence intervals unpublished; statistical significance is unknown. Condition: NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026.

How did TensorRT-LLM compare with vLLM on derived cost in the shared published configuration?

We report these derived costs. TensorRT-LLM was $0.228; vLLM was $0.206 USD per 1M output tokens. Condition: NVIDIA H100 80GB, BF16, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. Both values are derived from the recorded GPU rate and measured saturation throughput. Rounded cost differences do not resolve a statistically significant ranking.

Serving engine

TensorRT-LLM: what RunInfra measured

Measurement scope

TensorRT-LLM's PyTorch-backend run makes backend scope part of every throughput and latency reading.

Source article and method

Head-to-head comparisons

The comparison evidence stays inside one measured scope

Measured throughput and TTFT remain separate from derived cost. Every comparison ratio shows both source values and its full condition.

Scroll horizontally for all columns.

Output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off. Llama 3.1 8B Instruct, NVIDIA H100 80GB, BF16, Jun 20, 2026. Three timed repeats; confidence intervals unpublished. p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column.
Concurrency	Measured throughput	Measured TTFT p50	Full condition
1	152 output tokens/s	32 ms	Concurrency 1, shared sweep condition above
8	1,027 output tokens/s	56 ms	Concurrency 8, shared sweep condition above
32	2,978 output tokens/s	235 ms	Concurrency 32, shared sweep condition above
64	4,029 output tokens/s	504 ms	Concurrency 64, shared sweep condition above
128	4,774 output tokens/s	1,691 ms	Concurrency 128, shared sweep condition above
256	4,813 output tokens/s	2,848 ms	Concurrency 256, shared sweep condition above

Pairwise readings at the highest measured concurrency

output throughput: TensorRT-LLM 4,813, vLLM 5,333 output tokens/s. 9.8% gap; significance unknown.
Evidence and calculation
Recorded means for output throughput: vLLM recorded 5,333 output tokens/s versus TensorRT-LLM at 4,813 output tokens/s, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 9.8% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.11x.derived at render time from the two measured throughput values at concurrency 256, output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off
TTFT p50: TensorRT-LLM 2,848, vLLM 2,221 ms. 22.0% gap; significance unknown.
Evidence and calculation
Recorded means for TTFT p50: vLLM recorded 2,221 ms versus TensorRT-LLM at 2,848 ms, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 22.0% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.28x.derived at render time from the two measured latency values at concurrency 256, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column
output throughput: TensorRT-LLM 4,813, SGLang 5,235 output tokens/s. 8.1% gap; significance unknown.
Evidence and calculation
Recorded means for output throughput: SGLang recorded 5,235 output tokens/s versus TensorRT-LLM at 4,813 output tokens/s, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 8.1% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.09x.derived at render time from the two measured throughput values at concurrency 256, output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off
TTFT p50: TensorRT-LLM 2,848, SGLang 2,543 ms. 10.7% gap; significance unknown.
Evidence and calculation
Recorded means for TTFT p50: SGLang recorded 2,543 ms versus TensorRT-LLM at 2,848 ms, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026. The difference is 10.7% of the larger mean, from three timed repeats; confidence intervals are unpublished, so statistical significance is unknown. The ratio is 1.12x.derived at render time from the two measured latency values at concurrency 256, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column

Derived cost stays on its own basis

Scroll horizontally for all columns.

derived from the GPU hourly price and the measured saturation throughput for that configuration, not measured. Formula: cost per 1M tokens equals the GPU hourly price divided by sustained tokens per hour. These are reported derived costs. Underlying FP8 and L40S throughput rows are not published; rounded costs do not establish measured throughput or a speedup.
Configuration	Published value or absence	Full condition
NVIDIA H100 80GB BF16	$0.228 USD per 1M output tokens, derived	NVIDIA H100 80GB, BF16, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026
NVIDIA H100 80GB FP8	TensorRT-LLM at fp8 was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100.	NVIDIA H100 80GB, FP8, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of Jun 20, 2026
NVIDIA L40S 48GB BF16	TensorRT-LLM on the L40S was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100.	NVIDIA L40S 48GB, BF16, source-selected cost concurrency 128, Llama 3.1 8B Instruct, as of Jun 20, 2026

Prefix-cache evidence depends on the workload

We did not measure TensorRT-LLM's prefix cache. Both prefix experiments ran vLLM and SGLang only.

Where package measurements are listed: We report package measurements here, and each package remains subject to its listed license.

Citation: RunInfra (2026). Source article and method, Jun 20, 2026..

Published packages keep their own measured conditions

These rows come from the published catalog, not the comparison sweep. Each package retains its own model, engine version, concurrency, and verified date.

No published package currently serves on this engine.

The limitations bound every result

TensorRT-LLM 1.2 removed the TensorRT backend; PyTorch is the only execution backend in the tested 1.2.1 release. Our single-prefix cache test is not the many-distinct-prefix case SGLang's RadixAttention is designed for, so we ran that separately too. These results apply only to the tested configurations.
One H100 80GB and one L40S 48GB, single GPU, tensor-parallel size 1.
Same Llama-3.1-8B-Instruct weights and same request stream across engines. Three timed repeats. Raw repeats and confidence intervals are not publicly available.
Latency here is TTFT and per-request percentiles, not goodput at a fixed SLO. Whether first-token latency is acceptable depends on your use case; an SLO-goodput sweep is future work.
Numbers are self-reported on our harness and are not cross-calibrated against an external suite like MLPerf. The harness and raw data are not publicly available, so this publication does not support independent reproduction.
TensorRT-LLM 1.2.1 ran the PyTorch backend, its only execution backend. NVIDIA's TensorRT-LLM 1.2 release notes document removal of the TensorRT backend and engine-build CLI.
The 4,813 tokens/s result is the highest throughput reported for TensorRT-LLM 1.2.1 in this sweep, not a ceiling for other workloads or configurations.
The L40S sweep stopped at concurrency 128 (its saturation), the H100 at 256.
Engines move weekly. These numbers are vLLM 0.23.0, SGLang 0.5.13, TensorRT-LLM 1.2.1, as of Jun 20, 2026.

What is not measured

No other model is measured in this comparison dataset. It covers Llama 3.1 8B Instruct.
No GPU outside NVIDIA H100 80GB and NVIDIA L40S 48GB is represented in the dataset.
No engine version newer than TensorRT-LLM 1.2.1 is measured for this page's comparison evidence.
No published fp8 throughput or latency rows. The source publishes fp8 only as derived GPU cost per 1M output tokens at saturation for vLLM and SGLang; its underlying throughput is not published. These cost cells do not establish an fp8 speed ranking.
No published L40S throughput or latency rows. L40S appears only in derived GPU cost cells for vLLM and SGLang; their underlying throughput is not published. Every published unique-prompt sweep row here is H100 bf16.
No comparison with older TensorRT-LLM backends. The tested version 1.2.1 supports only PyTorch; the TensorRT backend was removed in version 1.2.
No prefix-cache latency values. The source states one engine had lower first-token latency on the many-prefix workload but publishes no latency numbers for either prefix experiment, so that comparison stays qualitative.
No hardware or precision label for the single-shared-prefix sweep. Its caption names only the concurrency, so those rows must not be presented under a GPU or a precision the source does not state.
No goodput at a fixed service-level objective, and no accuracy or output-quality comparison between engines. This dataset is throughput, first-token latency, and derived cost only.
No cross-engine or cross-precision ratio is stored. Pages derive every multiplier and percentage from the absolute pair at render time and print both absolute values beside it.
Not cross-calibrated against an external benchmark suite. These are self-reported numbers; the harness and raw data are not publicly available for independent reproduction.
No claim that these results still hold. The dataset is pinned to one as-of date and one engine version triple, and engines change on a weekly cadence.

The same evidence answers common engine questions

How did TensorRT-LLM compare with vLLM on throughput at the highest measured concurrency?

How did TensorRT-LLM compare with SGLang on TTFT p50 at the highest measured concurrency?

How did TensorRT-LLM compare with vLLM on derived cost in the shared published configuration?

Head-to-head comparisons

The comparison evidence stays inside one measured scope

Pairwise readings at the highest measured concurrency

Derived cost stays on its own basis

Prefix-cache evidence depends on the workload

Published packages keep their own measured conditions

The limitations bound every result

What is not measured

The same evidence answers common engine questions

Head-to-head comparisons

The comparison evidence stays inside one measured scope

Pairwise readings at the highest measured concurrency

Derived cost stays on its own basis

Prefix-cache evidence depends on the workload

Published packages keep their own measured conditions

The limitations bound every result

What is not measured

The same evidence answers common engine questions

Call Nemotron 3.5 Lightning 30B with Model APIs