Skip to content

RunInfraby RightNow

Model APIsNew
Pricing
Research
Contact

Dashboard Sign in Get started

Measured serving engines for open model inference

Loading engines

Engines

Measured serving engines for open model inference

The published evidence currently covers 3 engines and 4 of 4 published packages.

SGLang
5,235 output tokens/s, 2,543 ms TTFT p50 at concurrency 256
0 published packages
Comparison scope
Llama 3.1 8B Instruct on NVIDIA H100 80GB, BF16, measured Jun 20, 2026.
Measured versions
0.5.13
0.5.13
Evidence sources
Comparison rows and published package records stay separated on the detail page.
TensorRT-LLM
4,813 output tokens/s, 2,848 ms TTFT p50 at concurrency 256
0 published packages
Comparison scope
Llama 3.1 8B Instruct on NVIDIA H100 80GB, BF16, measured Jun 20, 2026.
Measured versions
1.2.1
1.2.1
Evidence sources
Comparison rows and published package records stay separated on the detail page.
vLLM
5,333 output tokens/s, 2,221 ms TTFT p50 at concurrency 256
4 published packages
Comparison scope
Llama 3.1 8B Instruct on NVIDIA H100 80GB, BF16, measured Jun 20, 2026.
Measured versions
0.23.0, 0.23.1, and 0.25.1
0.23.0, 0.23.1 (unverified, no PyPI release found September 21, 2026), and 0.25.1
Evidence sources
Comparison rows and published package records stay separated on the detail page.

Call Nemotron 3.5 Lightning 30B with Model APIs

Use a workspace API key and pay for input, cached input, and output tokens.

View Model APIs

TLS in transit, AES-256 at rest

Workspace isolation

No training on your data

SOC 2 Type II

RunInfraby RightNow

© 2026 RunInfra. All rights reserved.

Join the community System status

Coding plans Model APIs Pricing Startups Benchmarks Docs Research News Brand assets Contact

Backed by

Y Combinator

Ask AI about RunInfra

Part of RightNow

Security DPA AUP Cookies Terms Privacy