LLM GPU Token Speed Calculator
This LLM GPU token speed calculator follows a prompt sent to a large language model (LLM) on a graphics card (GPU) until its answer is complete, in six steps, and checks that the model and its conversations fit in the card's memory (VRAM). Model and Card takes the model's and the card's figures, Memory fits the users' conversations beside the weights and the serving software's working memory, Prompt Processing gives the time to the first token of the answer, Speed for One User the tokens per second that follow, Speed for Several Users the same when the card writes several answers at once, and From Prompt to Answer shows each user's answer appearing as the serving software schedules them.
A token is a word or a part of a word. A language model reads a conversation and writes its answer one token at a time, so the tokens per second set how fast the answer appears.
The calculator covers Qwen3.8-27B with its weights in 4 or 8 bits, Arete-27B in 4 bits and Bielik-11B-v3.0 in 8 or 16 bits, on a GeForce RTX 3080, 3090, 4080, 5070, 5080 or 5090, an A100 80GB PCIe, an H100 NVL or H200 NVL, or a card of your own.
Its estimates come from the cards' data sheets, the models' files and the serving software's code, and its practical estimates from what vLLM's own steps reached on two cards, a GeForce RTX 3090 and an RTX 5090, and its 8-bit kernels on an H200 NVL. Checked Against Measurements sets the results beside vLLM's on those three cards, with the error at each point. Every other card and model is estimated without a check, and the page says so beside its results.
The results can be downloaded as a PNG or copied as text at the end of the page.
Worked Example
The calculator opens with Qwen3.8-27B, its weights stored in 4 bits each (NVFP4), on one GeForce RTX 5090, for one user with a conversation of 4,096 tokens and its cache in 8 bits. The weights take 21.07 GB of the 33.85 GB that programs can use on the card. The serving software, vLLM, may use 92% of it, 31.14 GB, and needs 2.547 GB of that for its own work, which leaves 7.521 GB for conversations: 217,952 tokens for one user, as vLLM counts them, less than the longest conversation the model accepts, 262,144 tokens. In practice, with what vLLM took in its own start on the cards measured, 6.281 to 7.150 GB is left, 180,320 to 206,976 tokens.
Each new token runs 821 kernels, reads 17.61 GB of weights and reads and writes 0.4615 GB for the user: the fixed state, the activations and the cache. At 92% of the card's 1,792 GB/s the weights take 10.68 ms and the user's bytes 0.2799 ms, and the kernels take 0.7292 ms to start and reach memory, so one user gets 85.55 tokens per second. The bandwidth alone would allow 99.17. In practice, with the figures measured in vLLM's own steps, one user gets 66.74 to 74.21.
Before the first token of the answer, the card processes the prompt of 3,584 tokens in 3 chunks of up to 1,568 tokens. Their matrix multiplications take 206.8 ms at the tensor cores' rates, full attention 14.19 ms, the linear attention 8.487 ms and the small kernels' reading and writing 63.07 ms; the engine adds 2.850 ms between chunks. The first token comes after 0.2954 s, and the answer of 512 tokens is complete after 6.266 s. In practice the first token comes after 0.4476 to 0.4519 s, and the answer is complete after 7.331 to 8.105 s.
With four users sending such prompts at the same moment, vLLM processes the prompts one after another, so the first tokens come after 0.4199 s to 1.182 s. While all four write, each gets 79.81 tokens per second, 319.3 together, and the last answer is complete after 7.571 s. In practice each gets 61.13 to 68.49.
Open the calculator
The calculator, with its equations and assumptions, opens from a link we send to your email address. We use the address to send the link, and write to you about our services only if you tick the newsletter box. Details in the privacy policy.
Memory
This step checks that the weights, the serving software's working memory and every user's conversation memory fit in the share of the card's memory the serving software may use.
A user's conversation memory is the fixed state plus the cache of every token so far; vLLM keeps the fixed state twice for prefix caching, once while the conversation is within its first block. The serving software, here vLLM, a widely used open-source program that runs language models for many users at once, keeps 8% of the card's memory free by default, so it may use 92%, and it needs part of that for its own work, its working memory, counted here from its code.
Beside the estimate the step gives two more counts of the tokens per user. The theoretical maximum takes the share of the data sheet's memory less the weights, with no driver's reserve and no working memory: no serving software can leave more. The practical estimate puts what vLLM took in its own start on the cards measured in place of the design's figures, as a range.
Card memory, to scale
What each part does and costs
- Reserved memory. Why: the card's driver keeps part of the memory for itself. Performance: 488 MiB, as an RTX 5090's nvidia-smi reported it; the card memory is what is left of the data sheet's.
- Libraries. Why: cuBLAS, NVIDIA's library of matrix multiplications, keeps a workspace for its methods. Performance: 32 MiB on cards from the Hopper architecture on, such as the RTX 5090, and 4 MiB on older ones, such as the RTX 3090; inside .
- Attention workspace. Why: with an 8-bit cache vLLM runs attention with FlashInfer, which keeps a workspace for the partial results of its kernels. Performance: 394 MiB with an 8-bit cache and none with a 16-bit one, inside .
- Working memory for one step. Why: vLLM sets aside the most memory its largest step needs, found at start-up by running each kind of step once: one image of the largest size through the image encoder, a step of 2,048 prompt tokens (8,192 on the H100 NVL and H200 NVL) and the choice of the next token for 256 users at once. Performance: the image encoder 1.852 GB, the prompt step 0.5956 GB and the choice 0.6357 GB; the largest, 1.852 GB, is inside .
- CUDA graphs. Why: vLLM records the card's work for one step, for steps of 1 to 512 tokens, so it can replay it without the processor starting each step, which makes every token faster. Performance: 248.5 MB by vLLM's code, inside . In the runs, with vLLM's limit of 16 users at once, its own estimate was 0.54 GiB on the RTX 5090 and 0.19 GiB on the RTX 3090; without the graphs vLLM left 0.21 GiB more for conversations on the RTX 3090. Their effect on speed was not measured: with them switched off, vLLM also skips preparing its kernels in advance, and its first tokens took about 10 s each.
- Blocks. Why: vLLM gives out conversation memory in blocks, so many conversations can grow and end without moving memory about. Performance: a block holds 1,568 tokens of cache, 51.38 MB here, and vLLM keeps one block spare; the tokens that fit are rounded down to whole blocks.
- Prefix caching. Why: vLLM keeps the processed text of earlier requests, so that a new request starting with the same text, such as the same instructions or document, need not process it again; for the fixed state it keeps the state at the end of each whole block, and two states per user are in memory at a time once the conversation is longer than one block. It is vLLM's default. Performance: one more fixed state per user, 153.9 MB here, in two blocks for each group of 16 linear-attention layers.
Equations
where is the memory one user's conversation holds, the copies vLLM keeps of the fixed state , 1 while the conversation is within its first block of tokens and 2 after, the conversation length in tokens and the cache per token; is the weights on the card, the serving software's working memory, the users at once, the share of memory the serving software may use and the card memory; is the number of tokens that fit per user, before vLLM's whole blocks.
Conversation memory 2 × 153.9 MB + 4,096 × 32.77 kB = 442.1 MB per user; needed 21.07 GB + 2.547 GB + 1 × 0.4421 GB = 24.06 GB ≤ 92% × 33.85 GB = 31.14 GB: it fits; tokens per user ((31.14 GB − 21.07 GB − 2.547 GB) ÷ 1 − 2 × 153.9 MB) ÷ 32.77 kB = 220,131, 217,952 in vLLM's whole blocks.
is the memory one conversation holds: its fixed state , twice once the conversation is longer than one block, and the cache of the full-attention layers, for each of its tokens. The model fits when its weights , the serving software's working memory and the users' conversations together take no more than the share of the card memory . Solving the same sum for gives the longest conversation each user can have, .
vLLM counts in blocks of tokens: the smallest multiple of 16 whose cache in one layer, , with the cache per token in one layer, holds one layer's fixed state . A block spans the full-attention layers; blocks fit in the memory left. One is kept spare, and each user's fixed state takes two blocks in each group of linear-attention layers, so each user can have tokens, the result shown. A conversation of at most tokens takes one block in each group, so more users of short conversations fit than the formula gives.
is the card's memory in its data sheet, , less the driver's reserve, = 488 MiB. No NVIDIA document gives the reserve; 488 MiB is what an RTX 5090's nvidia-smi reported with driver 570, and the calculator takes it for every card.
is the memory vLLM takes besides the weights before it sizes the memory for conversations, counted from its code at its default settings: at most 256 users at once and 2,048 tokens in a step. is cuBLAS's workspace, and FlashInfer's, with an 8-bit cache only.
The largest step is the largest of , the image encoder on an image of the largest size, 0 for a model without one; , a step of 2,048 prompt tokens, with the image's output held; and , the choice of the next token for 256 users, whose scores over the vocabulary are held in 16 and 32 bits. is the CUDA graphs: for each of vLLM's two kinds, the activations of its largest step and 1 MiB for each further graph, the least vLLM's own estimate takes.
Card memory 32 GiB − 488 MiB = 33.85 GB; working memory 33.55 MB + 413.1 MB + max(1.852 GB, 0.5956 GB, 0.6357 GB) + 248.5 MB = 2.547 GB.
is the theoretical maximum of the memory for conversations, the share of the data sheet's memory less the weights as the files hold them; the tokens per user, , follow from it as does, before vLLM's whole blocks.
is the practical estimate, with three terms measured in vLLM's own start with Qwen3.8-27B NVFP4: the memory programs could use beside , = −170 MiB on the RTX 5090 and +38 MiB on the RTX 3090, by nvidia-smi; the weights vLLM loaded beyond the files', = 1.05% and 1.40% of ; and vLLM's working memory beyond , = 0.781 and 0.188 GB, mostly its CUDA graphs. The range takes every term at the value that leaves less memory for conversations, then at the one that leaves more, for every card and model; its tokens per user are counted in vLLM's whole blocks.
Theoretical maximum 92% × 34.36 GB − 21.07 GB = 10.54 GB, tokens per user (10.54 GB ÷ 1 − 2 × 153.9 MB) ÷ 32.77 kB = 312,226. Practical, slow end 92% × (33.85 GB − 0.1783 GB) − 21.07 GB × 1.014 − (2.547 GB + 0.7809 GB) = 6.281 GB, 180,320 tokens in vLLM's whole blocks; fast end 7.150 GB, 206,976 tokens.
Assumptions
- Every user's conversation has the same length.
- The working memory is counted from vLLM's code at its default settings. Other settings and other versions use other amounts, and the code of the kernels, loaded as each is first used, is not counted: no source gives its size.
- The driver's reserve is the same for every card and driver, 488 MiB.
- The cache is kept for every token of a conversation; none is moved out of the card.
- The practical estimate's terms were measured for Qwen3.8-27B NVFP4 on two cards. With Bielik-11B-v3.0, which took no part, vLLM's memory for conversations lies inside the range on the RTX 5090; on the RTX 3090 vLLM left more than the range's fast end, by 0.2% with its 8-bit weights and by 52% with its 16-bit ones, 0.41 against 0.27 GB: it loaded Bielik's weights as the files hold them, and Bielik's CUDA graphs took 0.11 GiB or less.
Prompt Processing
This step gives the time from sending a prompt to the first token of the answer, from the arithmetic of processing the prompt in chunks.
Before it writes the answer, the model processes the whole prompt: every prompt token passes through every layer, as a new token does, but many tokens at a time, so each weight read from memory serves a whole chunk of them. Processing a prompt is therefore limited by the card's arithmetic rather than by its bandwidth. The matrix multiplications take two operations per weight and token, and full attention compares each prompt token with every token before it, which grows with the square of the prompt's length.
vLLM puts at most 2,048 tokens through the card in one step, 8,192 on the H100 NVL and H200 NVL, and processes a prompt in chunks of that size; for a model with linear attention it cuts each back to whole blocks of = 1,568 tokens, keeping the fixed state at the end of each. The arithmetic runs at the tensor-core rates of the card's data sheet, at the share the sources measured: 95.5% for the matrix multiplications and 85.0% for attention.
While it processes a prompt, the processor hands the card each of a chunk's kernels itself, about 14 µs each; a chunk takes the card's work or the processor's, whichever is longer, and about 1 ms more before the next step.
Beside the estimate the step gives two more times. The theoretical minimum does the arithmetic at the card's highest rates in its data sheet and reads the weights once at the full bandwidth, with nothing else: no card and no serving software can be faster. The practical estimate puts what vLLM's own prompt steps reached on the two cards measured in place of the sources' figures, as a range from both cards' slower figures to their faster ones; on the H100 NVL and H200 NVL, for 8-bit attention and 8-bit matrix multiplications, what those kernels reached on an H200 NVL.
Time to first token, to scale
What each part does and costs
- Chunks. Why: vLLM puts at most 2,048 tokens through the card in one step, 8,192 on the H100 NVL and H200 NVL, and processes a long prompt in pieces of whole blocks within that, keeping the fixed state at the end of each, so that a later prompt starting with the same text can reuse it. Performance: each chunk reads the weights once, and the engine takes 0.95 ms between steps.
- Matrix multiplications. Why: as for a new token, each prompt token is multiplied by every weight it uses, two operations per weight. Performance: = 48.70 billion operations a token; at the RTX 5090's tensor-core rates for the model's formats, 884.2 TFLOPS, × 95.5%, 57.7 µs a token, the largest part of the time up to 32,768 tokens.
- Full attention. Why: each prompt token is compared with every token before it in the 16 full-attention layers, which is how the model finds what it needs anywhere in a long document. Performance: it grows with the square of the prompt's length: 14.19 ms at 3,584 tokens, 5% of the time; about half the time at 65,536 tokens, and 65% at 131,072.
- Linear attention. Why: the linear-attention layers compute their fixed state chunk by chunk, 64 tokens at a time, instead of token by token. Performance: = 0.42 billion operations a token, 8.49 ms at 3,584 tokens.
- The small kernels. Why: the norms, additions, gates and number conversions between the matrix multiplications, each reading and writing the tokens' numbers. Performance: = 29.0 MB a token, 63.07 ms at 3,584 tokens, 21% of the time.
- The processor. Why: vLLM runs a prompt's chunk without a CUDA graph, so the processor issues each of its kernels. Performance: 1,157 kernels × 14.3 µs = 16.5 ms a chunk, shorter than the card's work on a chunk of 1,568 tokens, so the card does not wait for it.
Equations
where is the time to the first token; the sum runs over the prompt's chunks of tokens; is the card's work on a chunk; is the matrix multiplications' operations per token, the card's tensor-core rate for the model's number formats and the share they reach; is the weights' reading, as in Speed for One User; is full attention's operations in the chunk and the linear attention's per token, at the 16-bit rate × ; is the small kernels' bytes per token; is a chunk's kernels and the processor's time per kernel; is the engine's time per step; is the prompt tokens per second.
Chunks of 1,568, 1,568 and 448 tokens; matrix multiplications 3,584 × 48.70 GFLOP ÷ 844.1 TFLOPS = 206.8 ms; full attention 14.19 ms; linear attention 8.487 ms; the small kernels' reading and writing 63.07 ms; the card's work 292.5 ms; the processor issues 1,157 kernels a chunk at 14.27 µs each, 16.51 ms; each chunk takes the longer, then 0.9500 ms for the engine: time to first token 0.2954 s; prompt tokens per second 3,584 ÷ 0.2954 s = 12,133.
counts two operations, a multiplication and an addition, for each weight the matrix multiplications use per token: matrices of rows and columns of each kind, in every layer and the output layer. For Qwen3.8-27B it is 48.70 billion. is the card's tensor-core rate for those operations, each matrix at the data sheet's rate of its number format: 884.2 TFLOPS on an RTX 5090 for the 4-bit model, and 71.2 TFLOPS on an RTX 3090, whose Marlin kernels multiply in 16 bits.
is the operations of full attention for a chunk of tokens that starts after tokens: each token's query is multiplied with the key of every token before it, of them on average, and the results with their values, two operations each, for its query heads of numbers in each of the full-attention layers.
The chunks are blocks of tokens, the last one shorter. A chunk's matrix multiplications take their arithmetic or the reading of the weights, whichever is longer: a short chunk is limited by the bandwidth like a new token. The linear attention computes its fixed state in pieces of 64 tokens, = 0.42 billion operations a token for Qwen3.8-27B, counted from vLLM's code. The small kernels between the matrix multiplications read and write = 29.0 MB a token on an RTX 5090, counted from the model's configuration.
A prompt's chunk runs a new token's kernels, with the linear attention's two in each layer replaced by nine for the chunked computation: = 1,157 for the 4-bit model on an RTX 5090. vLLM runs them without CUDA graphs, so the processor issues them one by one while the card works on the earlier ones, and the chunk takes the card's work or the processor's time, whichever is longer. The engine then takes before the next step.
= 95.5%: a 16-bit matrix multiplication reached about 200 of the RTX 5090's 209.5 TFLOPS (Zhang et al., 2025). = 85.0%: FlashAttention reached 147 to 209 TFLOPS on the RTX 5090, 178 in the middle of that range (the same paper). = 14.3 µs: 743 kernels took 14 ms without a CUDA graph and 3.4 ms with one (Ghosh et al., 2025). = 0.95 ms: vLLM's scheduling time per step (Li et al., 2025). None of the four was measured on the cards here or set from their runs.
On the H100 NVL and H200 NVL, vLLM multiplies 4-bit weights with its Marlin kernels, which use the tensor-core instructions of the cards before Hopper; on a Hopper card those instructions reach on average 62.9% of its peak rate (Luo et al., 2024). Their 4-bit rate is therefore 62.9% of the 16-bit rate, 525.5 TFLOPS, while the theoretical minimum keeps the 16-bit peak, 835.5 TFLOPS. The A100 80GB PCIe, RTX 3080 and RTX 3090 multiply 8-bit and 4-bit weights at their 16-bit rate, and the RTX 4080, which multiplies 8-bit numbers itself, its 4-bit weights: on the cards before Hopper those instructions reach about their peak rate (Luo et al., 2024).
is the theoretical minimum. and are the card's highest rates in its data sheet for each number format, with 16-bit accumulation where it gives one: 838 TFLOPS for 8-bit numbers on an RTX 5090, and 142.3 for the 16-bit numbers of the RTX 3090's Marlin kernels; bfloat16, in which attention runs, has only its 32-bit-accumulate rate; on the H100 NVL and H200 NVL with an 8-bit cache, attention runs in 8 bits, at the 8-bit peak. The estimate uses the 32-bit-accumulate rates throughout. The small kernels, the processor and the engine add nothing.
is the practical estimate. Its figures come from profiles of vLLM's prompt steps, Qwen3.8-27B NVFP4 on the two cards measured, each a part's time at the estimate's rates divided by its measured time: the matrix multiplications = 106.0% (RTX 5090) and 99.9% (RTX 3090), full attention = 68.4% (RTX 5090, at 65,536 tokens), the linear attention = 20.7% and 24.4%, and the small kernels = 220% and 129%: they moved fewer bytes than counts. The weights are read at Speed for One User's .
On the H100 NVL and H200 NVL with an 8-bit cache, vLLM runs attention in 8 bits: there = 23.3% of the 8-bit peak, and the 8-bit matrix multiplications reach 65.3% to 71.7% of it, the lower figure when the card held them for 15 s at its power limit, timed with vLLM's own kernels on an H200 NVL at Qwen3.8-27B's sizes.
On the RTX 5090 a chunk waits for the processor, which took = 121 µs for each kernel counted, by vLLM's own timers; on the RTX 3090 the card's longer work hides it. Between chunks the card stood idle for = 8.9 and 10.4 ms. The range takes every figure at the slower of its values, then every one at the faster, for every card and model; a card without a figure takes the other's. A chunk's practical work is never shorter than its theoretical minimum.
Theoretical minimum: matrix multiplications 135.7 ms, full attention 12.05 ms, linear attention 7.211 ms, at the peak rates: 0.1550 s. Practical, slow end: the first chunk's work 124.7 ms at the measured figures, against 1,157 kernels × 121.3 µs = 140.3 ms for the processor; each chunk takes the longer, then 10.35 ms: 0.4519 s. Fast end, every figure at the faster card's value: 0.4476 s.
Assumptions
- One prompt at a time, of new text: nothing of it is found in vLLM's prefix cache. A prompt that starts with text processed before, such as the same instructions, is faster.
- , , and are the same for every card and processor, from the sources above.
- Full attention runs at the share FlashAttention reached at head dimension 128. This model's heads are 256 wide, and vLLM runs FlashInfer with an 8-bit cache; on the cards measured it reached less, so long prompts are estimated fast.
- The processor issues an operation in . On an RTX 5090 vLLM's operations took longer to issue, and the card waited for the processor within short chunks, so short prompts are estimated fast there (Checked Against Measurements).
- Nothing outside the card is counted: the network, turning the text into tokens, and the server's queue.
- Up to the longest prompt without preemption. Above it, vLLM runs out of blocks while it processes the prompt, sets the prompt aside and processes part of it again, which takes longer than shown, by an amount not modelled.
- The practical estimate's figures were measured for Qwen3.8-27B NVFP4 on two cards. Bielik-11B-v3.0, which took no part in them, lies inside the range at 7 of 18 prompts: its long prompts were up to 6.2% faster and its short ones up to 8.6% slower; the causes were not profiled.
Speed for One User
This step gives the tokens per second one user gets, from the bytes each new token reads and writes, the share of the card's bandwidth the reading reaches and the kernels each token runs.
Making a new token is limited by the card's memory bandwidth, the bytes per second it can read from its memory: each token reads the weights it uses once and reads and writes the user's conversation memory, and the arithmetic on them takes less time than the reading. A memory throughput test reached 92% of the maker's bandwidth on a GeForce RTX 4090, and the calculator takes that share for every card.
The card does a token's work as a sequence of kernels, the small programs it runs one after another, about 800 of them, counted from vLLM's code. Each adds under a microsecond, together about 0.7 ms a token.
Beside the estimate the step gives two more figures. The theoretical maximum reads everything at the full bandwidth with no time per kernel: no card and no serving software can do better. The practical estimate puts, in place of the sources' figures, what vLLM's own decode steps reached on the two cards measured. It is a range, from both cards' slower figures to their faster ones, because a card or model nobody has measured has no figures of its own.
Time per token, to scale
Equations
where is the estimated number of tokens per second; is the number of kernels each token runs; is a kernel's time to start and its time until its first read from memory arrives; is the weights read for every token and the user's bytes: the fixed state read and written, the activations and logits , and the cache read, in bytes; is the share of the bandwidth reached; is the card's memory bandwidth, in bytes per second; is the bandwidth's limit, reading alone.
Kernels 821 × (0.5532 µs + 0.3350 µs) = 0.7292 ms; weights 17.61 GB ÷ (92% × 1,792 GB/s) = 10.68 ms; the user's bytes 0.4615 GB ÷ (92% × 1,792 GB/s) = 0.2799 ms; estimate 1 ÷ (0.7292 ms + 10.68 ms + 0.2799 ms) = 85.55 tokens/s; bandwidth's limit 1,792 GB/s ÷ 18.07 GB = 99.17 tokens/s.
For each new token the card runs kernels, reads of weights and reads and writes the user's bytes. Each kernel takes to start and until its first read arrives, and the bytes move at the share of the bandwidth ; together they give one user's . Reading everything at the full bandwidth, with no time per kernel, would give : no card and no serving software can do better.
The linear-attention layers update each user's fixed state in place, so every token reads it and writes it back, . Each kernel writes its results for the next one to read, the activations, and the last matrix multiplication writes a score for each of the 248,320 entries of the vocabulary, the logits, from which the token is picked. Together these are , counted from the model's configuration: 19.40 MB a token on an RTX 5090 and 20.39 MB on an RTX 3090, whose matrix multiplications take their inputs in 16 bits. The full-attention layers read the cache of every token so far, .
The kernels are counted from vLLM's code. vLLM compiles the model with torch.compile, which joins the norms, additions and gates between two matrix multiplications into one kernel; the matrix multiplications, attention and the linear-attention updates are kernels of their own. That gives 11 kernels in each full-attention layer, 12 in each linear-attention layer and 4 once a token.
A card that multiplies 4-bit numbers itself, as the RTX 5090 does, runs one more kernel in each layer and one more once, to convert the inputs of the 4-bit matrices: 821 for the 4-bit model on an RTX 5090, 756 on an RTX 3090 and for the 8-bit model. vLLM records a token's kernels once and replays them as one CUDA graph, so the processor does not start each one.
= 92% is the share of the maker's bandwidth reached in a memory throughput test on a GeForce RTX 4090 (Luo et al., 2024). = 0.553 µs: 1,025 empty kernels in one CUDA graph took 567 µs on a GeForce RTX 3060 (Hoffman and Oh, 2024). = 0.335 µs: a read from the card's memory took 876.7 clock cycles on a GeForce RTX 5080 (Jarmusch et al., 2025), at its boost clock of 2,617 MHz. None of the three was measured on the cards here or set from their runs.
is the practical estimate. It takes its figures from profiles of vLLM's decode steps: Qwen3.8-27B NVFP4, one user, on the RTX 5090 and RTX 3090 measured, with the profiler's own time taken off. The matrix multiplications read the weights at = 78.2% and 83.6% of the bandwidth. The small kernels, the gaps between kernels and the choice of the token took = 1.98 and 1.38 µs for each kernel the calculator counts, and attention a fixed = 18.2 and 12.9 µs in each full-attention layer.
The fixed state was read and written at = 47.9% and 57.9% of the bandwidth and the cache read at = 95.7% and 74.8%, and the card stood idle for = 0.015 and 0.051 ms between two steps. The activations and logits are part of the small kernels' time. The range takes every figure at the slower of its two values, then every one at the faster, for every card and model.
Practical, slow end: weights 17.61 GB ÷ (78.23% × 1,792 GB/s) = 12.56 ms; kernels 821 × 1.976 µs = 1.622 ms; attention's fixed cost 16 × 18.24 µs = 0.2918 ms; fixed state 2 × 153.9 MB ÷ (47.94% × 1,792 GB/s) = 0.3584 ms; cache 4,096 × 32.77 kB ÷ (74.77% × 1,792 GB/s) = 0.1002 ms; idle 0.05116 ms; 1 ÷ 14.98 ms = 66.74 tokens/s. Fast end, every figure at the faster card's value: 74.21 tokens/s.
Assumptions
- One user at a time, on one card; several users are in Speed for Several Users.
- , and are the same for every card, from the sources above.
- The weights are read at , the share a plain read reaches. vLLM's matrix multiplications also unpack the 4-bit and 8-bit weights as they read them, and no source gives the share they reach at one user's sizes; on the cards measured they reached 78% of the bandwidth on the RTX 5090 and 84% on the RTX 3090, so the estimate is fast.
- Each kernel is counted once, as vLLM's code gives them, with one read from memory. On the cards measured vLLM ran 1,163 kernels a token on the RTX 5090 and 926 on the RTX 3090, because torch.compile joined fewer operations than the count assumes, and its kernels read memory more than once; both make the estimate fast.
- Other card is taken as one that multiplies 4-bit numbers itself, as the RTX 5090 does.
- The practical estimate's figures were measured for one model, Qwen3.8-27B NVFP4, on two cards. Bielik-11B-v3.0, which took no part in them, ran faster than the practical range at 12 of its 18 measured points: by 0.2% to 3.0% with its 8-bit weights and with its 16-bit weights on the RTX 3090, and by 6.3% to 11.0% with its 16-bit weights on the RTX 5090. Bielik was not profiled, so the cause is not established.
Speed for Several Users
This step gives each user's tokens per second when the card writes several answers at once, and all users' tokens per second together.
The card makes one new token for every user in one step: it reads the weights once for all of them, and each user's conversation memory for that user. A further user therefore costs far less than a further card: on an RTX 5090, 16 users each got 58.7 tokens per second where one got 68.9, 940 tokens per second together.
Each further user adds only their own bytes: their fixed state, read and written, their activations and logits, and their cache, as in Speed for One User. The kernels and the reading of the weights are shared by all users in the step. The users and the conversation length are those of Memory.
One step for all users, to scale
Equations
where is the time of one step, which makes one token for each user; is the users at once; , , , , and are as in Speed for One User, and is one user's bytes at the conversation length ; and are each user's tokens per second and all users' together.
One user at once: step 0.7292 ms + (17.61 GB + 0.4615 GB) ÷ (92% × 1,792 GB/s) = 11.69 ms; each user 1 ÷ 11.69 ms = 85.55 tokens/s; all users 1 × 85.55 = 85.55 tokens/s.
The users at once, , are the most users, at most and vLLM's limit of 256 at once, whose conversations of tokens fit for their whole length: is Memory's tokens per user for users. vLLM itself starts a request when its blocks fit at that moment, and preempts it later if the conversations grow past what fits: on an RTX 3090, three users of 2,304 tokens all started, and two of them were preempted.
Each step runs the kernels and reads the weights once for all users, and each user's bytes for that user. A step of users therefore takes longer than one user's only by : on an RTX 5090, with conversations of 4,096 tokens, 0.28 ms for each further user against a step of 11.69 ms.
reads the weights once and every user's bytes at the full bandwidth, with no time per kernel. is Speed for One User's practical step for users: the weights, the kernels, attention's fixed cost and the idle time once a step, and each user's fixed state and cache, with the same figures , , , , and . They were measured at one user and are kept for several; Checked Against Measurements gives how far that holds.
Theoretical maximum 1,792 GB/s ÷ (17.61 GB + 1 × 0.4615 GB) = 99.17 tokens/s each; practical, slow end 1 ÷ 14.98 ms = 66.74 tokens/s each, fast end 74.21 tokens/s each.
Assumptions
- Every user's conversation has the same length , and every user is writing an answer; the prompts processed at the same time are in From Prompt to Answer.
- The step is Speed for One User's for users, with its causes of error: the estimate is fast, by more at 2 to 4 users on an RTX 5090, where NVIDIA's matrix library, cuBLAS, picks a slower method for one small matrix (Checked Against Measurements).
- The matrix multiplications' arithmetic grows with the users and is not counted. At the data sheets' rates it would take longer than reading the weights from 29 users on an RTX 3090 and 186 on an RTX 5090, for the 4-bit model; the estimate is then fast.
- Compared with vLLM up to 16 users at once.
- The practical estimate keeps the figures measured at one user. Against vLLM at 2 to 16 users at once, vLLM's speed lies inside the range at 17 of 35 steps: on the RTX 3090 at all 11; on the RTX 5090 below it by 0.6% to 6.4% at 2 to 4 users of Qwen3.8, where cuBLAS picks its slower method, and above it by up to 7.5% from 12 users and with Bielik-11B-v3.0.
From Prompt to Answer
This step follows every user's prompt from the moment it is sent to the last token of the answer, as vLLM schedules them, and shows the answers as they appear.
With several users the steps above run together. While one user's prompt is processed, the users already answering wait for it: vLLM makes their next tokens in the same step, and a step with a chunk of a prompt takes as long as the chunk.
vLLM puts at most 2,048 tokens through the card in one step, 8,192 on the H100 NVL and H200 NVL: first one token for each answer being written and the next chunk of each prompt it has started, then the prompts waiting, in the order they came, as long as each chunk fits whole. While users are answering, a step of 2,048 tokens therefore has room for one chunk of 1,568 tokens. The simulation follows these rules, read from vLLM's code, with each step's time from the steps above; all users send a prompt of tokens at the same moment.
Beside the estimate the step gives two more times for the answers. The theoretical minimum runs the same steps at Prompt Processing's theoretical minimum and Speed for Several Users' theoretical maximum; the practical estimate runs them at their practical steps, as a range.
How the simulation schedules the steps
- Sending: all users send their prompts at the same moment; they reach vLLM one after another, so its first step holds only the first.
- Each step: the users answering get one token each and each prompt in progress its next chunk, if it fits; then waiting prompts start, in turn, up to the users that fit in memory, until a prompt's chunk does not fit.
- A step's time: with prompt chunks, their time from Prompt Processing and the bytes of the users answering in it, as in Speed for Several Users; without, a step from Speed for Several Users.
- Tokens: a user's first token comes when the prompt's last chunk is done, and then one token a step until the answer is complete. A user who does not fit in memory waits until an answer ends.
Simulated streamed output
The answers appear in real time as the simulation gives them; the text is a sample answer cut into Qwen3.8-27B's own tokens, each a word or part of one, repeated when the answer is longer.
Each user, from the prompt to the answer, to scale
Equations
One user: the prompt in 3 chunks, first token after 0.2954 s; the other 511 tokens one a step, the answer complete after 6.266 s; mean time per token (6.266 s − 0.2954 s) ÷ 511 = 11.68 ms.
A step that processes prompt chunks takes the card's work on them, , as in Prompt Processing with the weights read once for all of them, or the processor's time , whichever is longer, then the engine's , plus for the users answering in it: their bytes, each user's at the share of the bandwidth, as in Speed for Several Users.
On an RTX 5090, profiled with one user answering, the step's matrix multiplications took 13.9 ms longer than the prompt's alone, because NVIDIA's matrix libraries pick slower methods for 1,025 rows than for 1,024; the calculator does not count it. On an RTX 3090 the step took as long as the prompt's alone, 767 against 771 ms.
The mean time per token, , is each answer's time from its first token to its last, , over its other tokens, averaged over the users. It counts the steps in which an answer waits for other users' prompts, unlike the time between tokens of Speed for Several Users.
The theoretical minimum and the practical estimate run the same simulation with other steps. A step with prompt chunks takes Prompt Processing's minimum or practical time, the users answering in it adding their bytes at the full bandwidth, or their fixed state at and their cache at ; a step of answers only takes Speed for Several Users' theoretical or practical step.
Theoretical minimum: first token 0.1550 s, whole answer 5.305 s, 10.08 ms a token. Practical: first token 0.4476 to 0.4519 s, whole answer 7.331 to 8.105 s, 13.47 to 14.98 ms a token.
Assumptions
- All users send their prompts at the same moment, all of tokens, and all answers are tokens long; nothing is found in vLLM's prefix cache.
- The users at once are those whose conversations of tokens fit for their whole length. vLLM may start more and preempt them; that is not simulated.
- The simulation's errors against vLLM, and their causes as far as they are established, are under Checked Against Measurements.
- The practical range takes one model's figures, measured at one user. Set beside vLLM, the whole answer of one prompt and 256 tokens lies inside it at all 11 of Qwen3.8's prompts and 4 of Bielik-11B-v3.0's 18, the others up to 9.1% shorter; with several users, the last first token at 14 of 35 steps and the mean time per token at 7, mostly shorter (Checked Against Measurements).
Checked Against Measurements
This section sets the calculator's results beside vLLM's own on the three cards measured.
In the tables below, “Modeled vs vLLM” is how far the modeled estimate lies above (+) or below (−) vLLM's figure, as a percentage of vLLM's figure. “Practical vs vLLM” reads “inside” where vLLM's figure lies within the practical range; otherwise it is how far the range's nearer end lies from vLLM's figure, measured the same way.
Memory and speed for one user
The memory left for conversations is 18% above vLLM's on the RTX 5090 and 47% above on the RTX 3090; one user's estimated speed is 23 to 28% above vLLM's on the RTX 5090 and 13 to 14% above on the RTX 3090, and vLLM's lies inside the practical range at 6 of 7 lengths. vLLM's memory for conversations and its tokens lie inside the practical memory range on both cards.
vLLM served Qwen3.8-27B NVFP4 on a GeForce RTX 5090 and a GeForce RTX 3090, each card on its own, at its default settings but a limit of 16 users at once, with an 8-bit cache. The memory figures are vLLM's own, from its log; the speeds come from the median time between two new tokens, one request at a time, three requests of 256 new tokens each per conversation length. The calculator's memory, counted with the same limit, and one user's speed were written down from vLLM's code, the data sheets and the sources before they were set beside these runs; nothing in them was fitted to them.
| Card, share | Result | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|
| RTX 5090, 92% | Memory for conversations | 7.747 GB | 6.507 to 7.376 GB | 6.582 GB | +17.7% | inside |
| RTX 5090, 92% | Tokens for one user | 224,224 | 186,592 to 213,248 | 189,728 | +18.2% | inside |
| RTX 3090, 98% | Memory for conversations | 1.389 GB | 0.1383 to 1.020 GB | 0.9449 GB | +47.0% | inside |
| RTX 3090, 98% | Tokens for one user | 31,360 | 0 to 18,816 | 17,248 | +81.8% | inside |
| RTX 3090, 92% | Fits | No | No | Does not start |
| Card | Conversation | Modeled | Practical | Bandwidth's limit | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|
| RTX 5090 | 1,024 tokens | 85.98 | 67.06 to 74.52 | 99.71 | 67.34 | +27.7% | inside |
| RTX 5090 | 16,384 tokens | 83.78 | 65.42 to 72.92 | 96.99 | 66.37 | +26.2% | inside |
| RTX 5090 | 65,536 tokens | 77.44 | 60.65 to 68.25 | 89.21 | 62.12 | +24.7% | inside |
| RTX 5090 | 131,072 tokens | 70.35 | 55.27 to 62.88 | 80.60 | 57.36 | +22.6% | inside |
| RTX 3090 | 1,024 tokens | 46.42 | 37.56 to 41.04 | 52.08 | 41.09 | +13.0% | −0.1% |
| RTX 3090 | 8,192 tokens | 45.84 | 37.09 to 40.60 | 51.40 | 40.30 | +13.7% | inside |
| RTX 3090 | 16,000 tokens | 45.22 | 36.60 to 40.14 | 50.69 | 39.83 | +13.5% | inside |
The calculator leaves 1.17 GB more for conversations than vLLM on the RTX 5090 and 0.45 GB more on the RTX 3090. On the RTX 3090 little is left after the weights, so a smaller error is 47% of the memory for conversations and 82% of the tokens. vLLM's working memory was the same at 92% and 98% on the RTX 3090, within 0.01 GB, as the calculator's is.
vLLM's own figures split the difference into three causes. The usable memory: the RTX 5090 had 0.17 GB less than its data sheet gives after the driver's reserve; on the RTX 3090 the driver reserved less, which gave vLLM 0.04 GB more. The weights: vLLM loaded 1.0% and 1.4% more than the model's files hold, 0.22 and 0.30 GB. The working memory: vLLM's was 0.79 and 0.19 GB larger than counted, mostly its CUDA graphs, 0.54 and 0.19 GiB by its own estimate against 0.02 GB counted. Together they account for the difference within 0.01 GB; the RTX 5090's other 0.2 GB of working memory has no established cause.
The practical memory range takes its terms from these two runs, so here it is checked in-sample: vLLM's memory for conversations and its tokens lie inside it on both cards. With Bielik-11B-v3.0, which took no part, they lie inside it on the RTX 5090 and above it on the RTX 3090, as Memory's assumptions give.
One user's estimate is fast for reasons found in profiles of vLLM's steps, used here for the causes only. At 1,024 tokens the estimated step is 3.22 ms shorter than vLLM's on the RTX 5090 and 2.80 ms on the RTX 3090. vLLM's matrix multiplications read the weights at 78% and 84% of the bandwidth, not 92%: 1.87 and 2.03 ms. vLLM ran 1,163 and 926 kernels a token, not 821 and 756, and its kernels read memory more than once: 1.38 and 0.55 ms. On the RTX 3090 attention's fixed cost adds about 0.21 ms. Together these account for the difference within 0.03 ms on both cards. The 8-bit model was not measured.
The practical range takes its figures from profiles of this model on these cards, so here it is checked in-sample: vLLM's speed lies inside it at 6 of the 7 lengths, and 0.1% above it at the seventh. Bielik-11B-v3.0, measured separately and not used for it, lies inside it at 6 of 18 points; at the others vLLM was faster, by up to 3.0%, and by 6.3% to 11.0% with its 16-bit weights on the RTX 5090.
Time to first token
On the RTX 3090 the estimate is 1% to 2% slow; on the RTX 5090 it is 5% to 14% fast from 16,384 tokens and up to 46% fast for short prompts. Above vLLM's preemption limit the time is not modelled, and on the RTX 3090 the calculator's limit is about twice vLLM's. vLLM's time lies inside the practical range at 10 of the 12 prompts.
The same cards also processed prompts of 1,024 to 180,000 tokens and served up to 24 users, one card at a time, with an 8-bit cache and vLLM's limit of users at once set to 16. Each prompt was text of its own, so that nothing of it was found in vLLM's prefix cache, as its counters confirmed. Each prompt length was measured with three prompts, one after another; several users with twice as many requests of 256 tokens as users, as many at a time as users, as vllm bench serve sends them.
| Card | Prompt, tokens | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|
| RTX 5090 | 65,536 | 9.874 | 10.783 to 11.424 | 10.875 | −9.2% | inside |
| RTX 5090 | 1,024 | 0.082 | 0.149 to 0.151 | 0.150 | −45.4% | inside |
| RTX 5090 | 4,096 | 0.339 | 0.448 to 0.452 | 0.433 | −21.6% | +3.3% |
| RTX 5090 | 16,384 | 1.579 | 1.722 to 1.831 | 1.656 | −4.6% | +4.0% |
| RTX 5090 | 32,768 | 3.751 | 3.962 to 4.252 | 3.979 | −5.7% | inside |
| RTX 5090 | 131,072 | 29.235 | 33.264 to 34.608 | 33.710 | −13.3% | inside |
| RTX 5090 | 180,000 | 49.876 | 57.728 to 59.596 | 58.128 | −14.2% | inside |
| RTX 3090 | 1,024 | 0.781 | 0.714 to 0.770 | 0.765 | +2.1% | inside |
| RTX 3090 | 4,096 | 3.163 | 2.897 to 3.122 | 3.096 | +2.2% | inside |
| RTX 3090 | 8,192 | 6.436 | 5.929 to 6.379 | 6.342 | +1.5% | inside |
| RTX 3090 | 12,000 | 9.575 | 8.862 to 9.520 | 9.444 | +1.4% | inside |
| RTX 3090 | 12,500 | 9.994 | 9.253 to 9.938 | 9.868 | +1.3% | inside |
The estimates were written down from the data sheets and the sources before they were set beside these runs, and nothing was fitted to them. On the RTX 3090, where a chunk's work takes about 0.7 s, they are 1% to 2% slow at every length measured.
Short prompts on the RTX 5090 took up to 1.8 times as long as estimated. vLLM issues a prompt's operations from the processor more slowly than gives, and when a chunk's work on the card is short the card waits for them: in a profile of one 4,096-token prompt the card was idle for 0.208 s of 0.514 s, in 5,124 gaps, none longer than 1 ms, and vLLM's own timers put the wait inside its prompt processing, not before it. Long prompts on the RTX 5090 are 9% to 14% fast because full attention reached about 68% of the 16-bit rate there, not 85%.
At 16,000 tokens, above vLLM's limit of 12,544, the RTX 3090 took 24.09 s, as vLLM preempted each prompt 5 times; the estimate does not model that. The calculator puts the limit at 26,656 tokens with the runs' settings, as its memory for conversations is larger than vLLM's (Memory and speed for one user), so it shows no preemption there.
The practical range takes its figures from profiles of this model at 1,024 and 65,536 tokens on the RTX 5090 and 1,024 on the RTX 3090, and from vLLM's timers at 1,024 and 4,096 tokens on the RTX 5090. vLLM's time lies inside it at 10 of the 12 prompts, and is 3.2% and 3.9% shorter at 4,096 and 16,384 tokens on the RTX 5090. With Bielik-11B-v3.0, which took no part, it lies inside at 7 of 18 prompts: the long ones up to 6.2% shorter, the short ones up to 8.6% longer.
Several users
Each user's speed is 13 to 38% above vLLM's on the RTX 5090 and 18 to 19% above on the RTX 3090, at 21 measured steps up to 16 users at once; vLLM's lies inside the practical range at all 5 RTX 3090 steps and 4 of the 16 RTX 5090 steps.
| Card | Conversation, tokens | Users at once | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|
| RTX 5090 | 1,024 | 2 | 84.4 | 65.4 to 72.8 | 61.2 | +37.9% | +6.9% |
| RTX 5090 | 1,024 | 4 | 81.3 | 62.2 to 69.6 | 61.1 | +33.2% | +1.9% |
| RTX 5090 | 1,024 | 8 | 75.9 | 56.8 to 63.9 | 62.3 | +21.8% | inside |
| RTX 5090 | 1,024 | 16 | 66.9 | 48.3 to 54.9 | 58.7 | +13.9% | −6.5% |
| RTX 5090 | 16,384 | 2 | 80.2 | 62.3 to 69.8 | 59.3 | +35.3% | +5.1% |
| RTX 5090 | 16,384 | 4 | 74.0 | 56.9 to 64.3 | 56.5 | +30.9% | +0.6% |
| RTX 5090 | 4,096 | 3 | 81.6 | 62.9 to 70.3 | 61.0 | +33.8% | +3.0% |
| RTX 5090 | 4,096 | 6 | 76.3 | 57.8 to 65.1 | 62.7 | +21.8% | inside |
| RTX 5090 | 4,096 | 12 | 67.6 | 49.8 to 56.7 | 58.3 | +15.9% | −2.7% |
| RTX 5090 | 32,768 | 2 | 76.3 | 59.3 to 66.9 | 57.2 | +33.3% | +3.7% |
| RTX 5090 | 32,768 | 3 | 71.6 | 55.5 to 63.0 | 55.1 | +30.0% | +0.8% |
| RTX 5090 | 1,024 | 2 | 84.4 | 65.4 to 72.8 | 61.2 | +37.8% | +6.8% |
| RTX 3090 | 1,024 | 2 | 45.5 | 36.5 to 40.0 | 38.3 | +18.7% | inside |
| RTX 3090 | 1,024 | 4 | 43.8 | 34.7 to 38.2 | 37.0 | +18.3% | inside |
| RTX 3090 | 1,024 | 4 | 43.8 | 34.7 to 38.2 | 37.2 | +17.9% | inside |
| RTX 5090 | 1,024 | 3 | 82.8 | 63.8 to 71.1 | 61.2 | +35.3% | +4.2% |
| RTX 5090 | 1,024 | 5 | 79.9 | 60.8 to 68.0 | 66.0 | +21.0% | inside |
| RTX 5090 | 1,024 | 16 | 66.9 | 48.3 to 54.9 | 59.1 | +13.2% | −7.0% |
| RTX 5090 | 16,384 | 6 | 68.6 | 52.4 to 59.6 | 57.7 | +18.9% | inside |
| RTX 3090 | 1,024 | 3 | 44.7 | 35.6 to 39.1 | 37.6 | +18.7% | inside |
| RTX 3090 | 2,048 | 2 | 45.4 | 36.4 to 39.9 | 38.1 | +19.0% | inside |
Each user's speed is fast for the causes of one user's, given under Memory and speed for one user: vLLM reads the weights below 92% of the bandwidth and runs more kernels than counted. From 5 users on the RTX 5090 the estimate is 13% to 22% fast, on the RTX 3090 18% to 19%. At 2 to 4 users on the RTX 5090 it is 30% to 38% fast, for a further cause the calculator does not count, a slower method of NVIDIA's matrix library.
The slower method's cause was found in profiles of both cards: one small matrix of the linear-attention layers, multiplied 48 times a step, which NVIDIA's matrix library runs with a quick method for 1 user and from 5 users, and with a slow one at 2 to 4, about 1.2 ms more a step on the RTX 5090 and 1.4 ms on the RTX 3090.
The practical range keeps the figures measured at one user. vLLM's speed lies inside it at all 5 steps on the RTX 3090 and at 4 of the 16 on the RTX 5090. On the RTX 5090 it is below the range at 2 to 4 users, by 0.6% to 6.4%, for the slower method of cuBLAS, which one user's profiles did not contain. From 12 users it is above it, by 2.8% to 7.5%: the range charges each further user's fixed state and cache at one user's measured shares.
With Bielik-11B-v3.0, which took no part in the figures, vLLM's speed lies inside the range at all 5 RTX 3090 steps and 2 of the 8 RTX 5090 steps, and above it at the other 6, by up to 4.3%.
From prompt to answer
The simulated mean time per token is up to 26% short, as the steps are; the last user's first token is within 14% where the card stays busy, up to 56% early with short prompts on the RTX 5090, and up to 65% early where the calculator runs more users at once than vLLM did. The practical range holds the whole answer of one prompt at all 11 of Qwen3.8's prompts.
| Card | Prompt, tokens | Users (at once) | Time per token, modeled | Time per token, vLLM | Modeled vs vLLM | Last first token, modeled | Last first token, vLLM | Modeled vs vLLM |
|---|---|---|---|---|---|---|---|---|
| RTX 5090 | 1,024 | 2 | 12.1 | 16.3 | −26.1% | 0.16 | 0.30 | −45.3% |
| RTX 5090 | 1,024 | 4 | 12.9 | 16.9 | −23.8% | 0.33 | 0.58 | −43.4% |
| RTX 5090 | 1,024 | 8 | 14.6 | 17.8 | −18.0% | 0.66 | 1.14 | −42.0% |
| RTX 5090 | 1,024 | 16 | 18.0 | 22.1 | −18.5% | 1.33 | 2.54 | −47.5% |
| RTX 5090 | 16,384 | 2 | 16.7 | 20.5 | −18.4% | 3.16 | 3.51 | −9.8% |
| RTX 5090 | 16,384 | 4 | 26.3 | 30.8 | −14.7% | 6.35 | 7.27 | −12.6% |
| RTX 5090 | 4,096 | 3 | 14.0 | 17.6 | −20.1% | 1.02 | 1.32 | −22.4% |
| RTX 5090 | 4,096 | 6 | 17.6 | 21.1 | −16.7% | 2.05 | 2.78 | −26.4% |
| RTX 5090 | 4,096 | 12 | 24.6 | 28.9 | −15.0% | 4.13 | 5.45 | −24.3% |
| RTX 5090 | 32,768 | 2 | 23.4 | 26.2 | −10.9% | 7.52 | 7.86 | −4.4% |
| RTX 5090 | 32,768 | 3 | 34.5 | 36.5 | −5.7% | 11.31 | 12.21 | −7.4% |
| RTX 5090 | 1,024 | 2 | 12.1 | 16.3 | −26.0% | 0.16 | 0.33 | −50.4% |
| RTX 3090 | 1,024 | 2 | 24.2 | 26.8 | −9.7% | 1.56 | 1.55 | +1.0% |
| RTX 3090 | 1,024 | 4 | 29.5 | 31.0 | −4.7% | 3.13 | 3.09 | +1.2% |
| RTX 3090 | 1,024 | 5 (4) | 32.2 | 32.1 | +0.2% | 3.91 | 11.15 | −64.9% |
| RTX 3090 | 2,048 | 3* | 30.3 | 31.2 | −3.1% | 4.71 | 11.92 | −60.5% |
| RTX 5090 | 1,024 | 3 | 12.5 | 16.7 | −25.1% | 0.25 | 0.55 | −55.5% |
| RTX 5090 | 1,024 | 5 | 13.3 | 16.1 | −16.9% | 0.41 | 0.69 | −40.5% |
| RTX 5090 | 1,024 | 24 (16) | 18.4 | 22.5 | −18.4% | 5.60 | 7.60 | −26.3% |
| RTX 5090 | 16,384 | 6 | 35.8 | 36.7 | −2.7% | 9.56 | 9.86 | −3.0% |
| RTX 3090 | 1,024 | 3 | 26.9 | 28.7 | −6.3% | 2.34 | 2.31 | +1.3% |
| RTX 3090 | 2,048 | 2 | 23.9 | 26.6 | −10.1% | 3.14 | 3.06 | +2.4% |
* vLLM started all 3 users and preempted 2 of them; the calculator, with its larger memory for conversations, runs all 3 without preemption. In brackets, the users vLLM ran at once where it ran fewer than were sent; the calculator runs all 5 on the RTX 3090 and 16 on the RTX 5090.
The mean time per token counts the steps in which an answer waits for other users' prompts; it was up to twice the time between tokens of the same run. The simulation gives it up to 26% short: its steps are Speed for Several Users', fast for the causes given there, and on the RTX 5090 a user answering in a step that processes a prompt made vLLM's matrix multiplications 13.9 ms longer in its profiles, as NVIDIA's matrix libraries pick slower methods for 1,025 rows than for 1,024, where the calculator counts only that user's bytes.
The last user's first token is within 14% wherever the card is busy throughout its chunks and the calculator runs as many users at once as vLLM did: on the RTX 3090, and on the RTX 5090 from 16,384 tokens. With prompts of 1,024 to 4,096 tokens on the RTX 5090 it comes 22% to 56% early, for the cause of the short prompts' error above: a step that processes a prompt while users answer starts 1,974 to 2,118 small programs, against 1,610 for the prompt alone, and starting them sets its time.
On the RTX 3090, with 5 users of 1,024 tokens and 3 of 2,048, it comes 65% and 61% early. vLLM's memory held fewer users than the calculator's: the fifth user waited for an answer to end, and two of three users were preempted and their prompts processed again. The calculator starts them all at once.
| Card | Prompt, tokens | Users (at once) | Time per token, practical | Time per token, vLLM | Practical vs vLLM | Last first token, practical | Last first token, vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|---|
| RTX 5090 | 1,024 | 2 | 14.1 to 15.7 | 16.3 | −3.7% | 0.30 to 0.30 | 0.30 | +0.001% |
| RTX 5090 | 1,024 | 4 | 15.6 to 17.3 | 16.9 | inside | 0.60 to 0.60 | 0.58 | +3.3% |
| RTX 5090 | 1,024 | 8 | 18.4 to 20.4 | 17.8 | +3.5% | 1.20 to 1.22 | 1.14 | +5.9% |
| RTX 5090 | 1,024 | 16 | 24.1 to 26.6 | 22.1 | +9.3% | 2.43 to 2.46 | 2.54 | −3.3% |
| RTX 5090 | 16,384 | 2 | 18.9 to 20.9 | 20.5 | inside | 3.45 to 3.67 | 3.51 | inside |
| RTX 5090 | 16,384 | 4 | 29.4 to 32.2 | 30.8 | inside | 6.93 to 7.37 | 7.27 | inside |
| RTX 5090 | 4,096 | 3 | 16.6 to 18.3 | 17.6 | inside | 1.35 to 1.36 | 1.32 | +2.3% |
| RTX 5090 | 4,096 | 6 | 21.3 to 23.3 | 21.1 | +1.2% | 2.70 to 2.73 | 2.78 | −1.9% |
| RTX 5090 | 4,096 | 12 | 30.7 to 33.1 | 28.9 | +6.2% | 5.45 to 5.51 | 5.45 | inside |
| RTX 5090 | 32,768 | 2 | 25.7 to 28.4 | 26.2 | inside | 7.94 to 8.53 | 7.86 | +1.0% |
| RTX 5090 | 32,768 | 3 | 37.4 to 41.0 | 36.5 | +2.3% | 11.94 to 12.83 | 12.21 | inside |
| RTX 5090 | 1,024 | 2 | 14.1 to 15.7 | 16.3 | −3.7% | 0.30 to 0.30 | 0.33 | −8.4% |
| RTX 3090 | 1,024 | 2 | 27.0 to 29.5 | 26.8 | +0.7% | 1.43 to 1.54 | 1.55 | −0.3% |
| RTX 3090 | 1,024 | 4 | 32.3 to 35.4 | 31.0 | +4.2% | 2.86 to 3.09 | 3.09 | −0.1% |
| RTX 3090 | 1,024 | 5 (4) | 34.9 to 38.3 | 32.1 | +8.8% | 3.57 to 3.86 | 11.15 | −65.4% |
| RTX 3090 | 2,048 | 3* | 32.7 to 35.8 | 31.2 | +4.8% | 4.30 to 4.64 | 11.92 | −61.1% |
| RTX 5090 | 1,024 | 3 | 14.9 to 16.5 | 16.7 | −1.1% | 0.45 to 0.45 | 0.55 | −17.9% |
| RTX 5090 | 1,024 | 5 | 16.3 to 18.0 | 16.1 | +1.4% | 0.75 to 0.76 | 0.69 | +8.6% |
| RTX 5090 | 1,024 | 24 (16) | 24.8 to 27.3 | 22.5 | +10.3% | 8.03 to 8.68 | 7.60 | +5.7% |
| RTX 5090 | 16,384 | 6 | 39.8 to 43.4 | 36.7 | +8.2% | 10.44 to 11.11 | 9.86 | +5.8% |
| RTX 3090 | 1,024 | 3 | 29.6 to 32.5 | 28.7 | +3.4% | 2.14 to 2.31 | 2.31 | −0.02% |
| RTX 3090 | 2,048 | 2 | 26.7 to 29.3 | 26.6 | +0.6% | 2.86 to 3.09 | 3.06 | inside |
The practical range runs the same simulation with the practical steps. In the table, vLLM's mean time per token lies inside it at 5 of the 22 steps and its last first token at 5; with Bielik-11B-v3.0, at 2 and 9 of 13. The mean time per token is mostly shorter than the range, by up to 9.3%, as the practical steps of several users are (Several users).
The last first token is within 8% of the range at 30 of the 35 steps. It came later at 2 and 3 users of 1,024 tokens on the RTX 5090 (+9.2% and +21.7%), at 2 users of the 16-bit Bielik on the RTX 3090 (+19.8%), and far later where vLLM ran fewer users at once than the calculator: +157% and +189% on the RTX 3090. The whole answer of one prompt and 256 tokens lies inside the range at all 11 of Qwen3.8's prompts and 4 of Bielik's 18, the others up to 9.1% shorter.
One of the simulation's errors has no established cause. The first user's first token came 0.10 to 0.71 s later than simulated on both cards, also where one prompt alone took about as long as estimated: 0.37 s later at 1,024 tokens on the RTX 3090. The profiles show its prompt processed alone in the first step, as simulated. Finding it needs vLLM's own timing of each request with several users, which was not measured.
Bielik-11B-v3.0
With Bielik-11B-v3.0 the memory for conversations is 2% to 7% above vLLM's, 60% with its 16-bit weights on the RTX 3090; one user's speed is 4% to 20% above vLLM's; the time to first token is within 9%, but 21% short for 1,024 tokens with the 8-bit weights on the RTX 5090; the users at once are right at all 13 steps.
vLLM served Bielik-11B-v3.0-Instruct, with its weights in 16 bits (BF16) and in 8 bits (FP8-Dynamic), on the same two cards, at its default settings, a 16-bit cache, 92% of the memory and a limit of 16 users at once. The calculator's figures for it were written down from the model's files, its configuration and vLLM's code before the runs, and none of its figures, the practical estimates' included, was taken from them. The 16-bit weights fit on the RTX 3090 only with conversations of up to 1,968 tokens, so they were measured at 512 to 1,536 tokens there.
| Card | Weights | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM | Tokens, modeled | Tokens, practical | Tokens, vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|---|---|---|---|
| RTX 3090 | 8-bit | 11.554 GB | 10.449 to 11.283 GB | 11.307 GB | +2.2% | −0.2% | 56,400 | 50,992 to 55,072 | 55,200 | +2.2% | −0.2% |
| RTX 3090 | 16-bit | 0.6534 GB | 0.0000 to 0.2685 GB | 0.4080 GB | +60.1% | −34.2% | 3,168 | 0 to 1,280 | 1,983 | +59.8% | −35.5% |
| RTX 5090 | 8-bit | 19.428 GB | 18.322 to 19.157 GB | 18.909 GB | +2.7% | inside | 94,832 | 89,440 to 93,520 | 92,288 | +2.8% | inside |
| RTX 5090 | 16-bit | 8.527 GB | 7.269 to 8.142 GB | 7.978 GB | +6.9% | inside | 41,616 | 35,472 to 39,728 | 38,960 | +6.8% | inside |
| Card | Weights | Conversation, tokens | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|
| RTX 3090 | 8-bit | 1,024 | 72.95 | 56.96 to 62.93 | 64.50 | +13.1% | −2.4% |
| RTX 3090 | 8-bit | 4,096 | 69.26 | 54.19 to 60.27 | 60.12 | +15.2% | inside |
| RTX 3090 | 8-bit | 8,192 | 64.88 | 50.88 to 57.05 | 56.09 | +15.7% | inside |
| RTX 3090 | 8-bit | 16,384 | 57.60 | 45.35 to 51.54 | 49.55 | +16.2% | inside |
| RTX 3090 | 8-bit | 24,000 | 52.16 | 41.19 to 47.30 | 45.37 | +15.0% | inside |
| RTX 3090 | 16-bit | 512 | 38.10 | 30.97 to 33.67 | 34.49 | +10.5% | −2.4% |
| RTX 3090 | 16-bit | 1,024 | 37.93 | 30.82 to 33.54 | 34.36 | +10.4% | −2.4% |
| RTX 3090 | 16-bit | 1,536 | 37.75 | 30.68 to 33.41 | 33.92 | +11.3% | −1.5% |
| RTX 5090 | 8-bit | 1,024 | 134.81 | 98.00 to 110.92 | 114.11 | +18.1% | −2.8% |
| RTX 5090 | 8-bit | 4,096 | 128.21 | 93.69 to 106.58 | 109.74 | +16.8% | −2.9% |
| RTX 5090 | 8-bit | 8,192 | 120.36 | 88.50 to 101.30 | 99.95 | +20.4% | inside |
| RTX 5090 | 8-bit | 16,384 | 107.23 | 79.67 to 92.17 | 90.18 | +18.9% | inside |
| RTX 5090 | 8-bit | 24,000 | 97.35 | 72.91 to 85.04 | 85.23 | +14.2% | −0.2% |
| RTX 5090 | 16-bit | 1,024 | 71.50 | 55.93 to 61.65 | 68.41 | +4.5% | −9.9% |
| RTX 5090 | 16-bit | 4,096 | 69.60 | 54.50 to 60.29 | 66.63 | +4.5% | −9.5% |
| RTX 5090 | 16-bit | 8,192 | 67.22 | 52.70 to 58.56 | 63.10 | +6.5% | −7.2% |
| RTX 5090 | 16-bit | 16,384 | 62.92 | 49.44 to 55.39 | 58.90 | +6.8% | −6.0% |
| RTX 5090 | 16-bit | 24,000 | 59.38 | 46.75 to 52.73 | 56.86 | +4.4% | −7.3% |
| Card | Weights | Prompt, tokens | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|
| RTX 3090 | 8-bit | 1,024 | 0.347 | 0.318 to 0.341 | 0.335 | +3.8% | inside |
| RTX 3090 | 8-bit | 4,096 | 1.473 | 1.360 to 1.448 | 1.403 | +5.0% | inside |
| RTX 3090 | 8-bit | 8,192 | 3.173 | 3.003 to 3.178 | 3.012 | +5.3% | inside |
| RTX 3090 | 8-bit | 16,384 | 7.254 | 7.135 to 7.486 | 6.914 | +4.9% | +3.2% |
| RTX 3090 | 8-bit | 24,000 | 11.864 | 11.991 to 12.506 | 11.358 | +4.5% | +5.6% |
| RTX 3090 | 16-bit | 512 | 0.172 | 0.167 to 0.173 | 0.188 | −8.5% | −8.0% |
| RTX 3090 | 16-bit | 1,024 | 0.347 | 0.329 to 0.341 | 0.359 | −3.1% | −5.0% |
| RTX 3090 | 16-bit | 1,536 | 0.526 | 0.493 to 0.513 | 0.523 | +0.5% | −2.1% |
| RTX 5090 | 8-bit | 1,024 | 0.064 | 0.076 to 0.078 | 0.081 | −20.7% | −4.6% |
| RTX 5090 | 8-bit | 4,096 | 0.285 | 0.276 to 0.297 | 0.295 | −3.4% | inside |
| RTX 5090 | 8-bit | 8,192 | 0.647 | 0.647 to 0.690 | 0.661 | −2.0% | inside |
| RTX 5090 | 8-bit | 16,384 | 1.603 | 1.678 to 1.764 | 1.606 | −0.2% | +4.5% |
| RTX 5090 | 8-bit | 24,000 | 2.769 | 2.983 to 3.109 | 2.798 | −1.0% | +6.6% |
| RTX 5090 | 16-bit | 1,024 | 0.121 | 0.118 to 0.124 | 0.129 | −6.4% | −3.8% |
| RTX 5090 | 16-bit | 4,096 | 0.510 | 0.481 to 0.511 | 0.492 | +3.7% | inside |
| RTX 5090 | 16-bit | 8,192 | 1.096 | 1.054 to 1.119 | 1.066 | +2.8% | inside |
| RTX 5090 | 16-bit | 16,384 | 2.502 | 2.489 to 2.621 | 2.445 | +2.3% | +1.8% |
| RTX 5090 | 16-bit | 24,000 | 4.085 | 4.169 to 4.365 | 4.038 | +1.2% | +3.2% |
| Card | Weights | Conversation, tokens | Users at once, vLLM (modeled) | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|---|
| RTX 3090 | 8-bit | 1,024 | 4 (4) | 68.7 | 53.9 to 59.9 | 59.4 | +15.7% | inside |
| RTX 3090 | 8-bit | 1,024 | 8 (8) | 63.7 | 50.2 to 56.4 | 54.5 | +17.0% | inside |
| RTX 3090 | 8-bit | 1,024 | 16 (16) | 55.6 | 44.2 to 50.4 | 46.9 | +18.6% | inside |
| RTX 3090 | 8-bit | 4,096 | 8 (8) | 46.4 | 36.9 to 42.8 | 40.9 | +13.5% | inside |
| RTX 3090 | 16-bit | 512 | 2 (2) | 37.9 | 30.8 to 33.5 | 33.2 | +14.0% | inside |
| RTX 5090 | 8-bit | 1,024 | 4 (4) | 127.2 | 93.2 to 106.1 | 109.4 | +16.3% | −3.0% |
| RTX 5090 | 8-bit | 1,024 | 8 (8) | 118.3 | 87.4 to 100.2 | 102.3 | +15.6% | −2.0% |
| RTX 5090 | 8-bit | 1,024 | 16 (16) | 103.7 | 77.9 to 90.3 | 91.3 | +13.6% | −1.1% |
| RTX 5090 | 8-bit | 4,096 | 8 (8) | 86.9 | 65.8 to 77.4 | 77.8 | +11.7% | −0.4% |
| RTX 5090 | 16-bit | 1,024 | 4 (4) | 69.3 | 54.3 to 60.1 | 62.7 | +10.6% | −4.1% |
| RTX 5090 | 16-bit | 1,024 | 8 (8) | 66.6 | 52.3 to 58.2 | 58.2 | +14.3% | −0.1% |
| RTX 5090 | 16-bit | 1,024 | 16 (16) | 61.7 | 48.7 to 54.7 | 53.9 | +14.5% | inside |
| RTX 5090 | 16-bit | 4,096 | 8 (8) | 55.3 | 43.7 to 49.7 | 49.5 | +11.9% | inside |
The memory for conversations is 2% to 7% above vLLM's, and 60% with the 16-bit weights on the RTX 3090, where 0.24 GB is large beside the 0.41 GB left after 22.34 GB of weights: vLLM's working memory was 0.13 GiB above the calculator's, its largest step larger than counted, and the memory it saw about 0.10 GiB below the data sheet's less the reserve. vLLM loaded the weights as the files hold them. Whether a conversation's length fits was right in all four runs.
One user's speed is fast for the causes found with Qwen3.8: the matrix multiplications read the weights below 92% of the bandwidth, and vLLM runs more kernels than counted. The 16-bit weights on the RTX 5090, read by plain 16-bit kernels, are closest, 4% to 7%; the 8-bit ones on the RTX 5090, whose inputs a kernel of their own quantises, furthest, 14% to 20%. Bielik was not profiled, so the split by cause is not measured.
The time to first token is within 9% everywhere but the RTX 5090's 1,024-token prompt with the 8-bit weights, 21% short: as with Qwen3.8, a short chunk on the RTX 5090 waits for the processor, which the practical estimate includes. The calculator's time to the complete answer of 256 tokens is 1% to 16% short, as its speed is fast. With several users the users at once are right at all 13 steps, and each user's speed is 11% to 19% fast, for one user's causes.
vLLM's figures lie inside the practical ranges at 6 of the 18 lengths for one user's speed, 7 of the 18 prompts, 7 of the 13 several-users steps and 2 of the 4 runs' memory. Elsewhere vLLM was mostly faster than the ranges, or left more memory: their figures come from Qwen3.8's profiles and runs.
Arete-27B
On the RTX 3090 one user's estimated speed is 14% to 16% above vLLM's at six lengths, and vLLM's lies inside the practical range at all six. The time to first token is within 6%, and vLLM's is at most 2.2% slower than the practical range. With two users, each user's estimated speed is 21% above vLLM's, inside the practical range.
With images allowed, the memory for conversations is 7% and 22% above vLLM's in two starts, and the practical range holds one and is 2.4% short of the other. With text only it is 55% above, because vLLM's working memory was 2.0 GB larger than the calculator counts, for a reason not yet known. The practical range for text only therefore takes the working memory measured in that start.
Arete-27B, in a 4-bit build (W4A16), ran on a GeForce RTX 3090 with the same vLLM. Its estimates were written down before they were set beside those runs, apart from the practical memory range with text only, which takes one of them.
| Card | Setup | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM | Tokens, modeled | Tokens, practical | Tokens, vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|---|---|---|---|
| RTX 3090 | 95%, 16-bit cache | 3.525 GB | 2.314 to 3.181 GB | 2.899 GB | +21.6% | inside | 47,824 | 29,792 to 42,336 | 33,981 | +40.7% | inside |
| RTX 3090 | 98.9%, 8-bit cache, text only | 6.443 GB | 3.940 to 4.263 GB | 4.155 GB | +55.0% | inside | 185,024 | 108,192 to 117,600 | 108,669 | +70.3% | inside |
| RTX 3090 | 98.9%, 8-bit cache | 4.097 GB | 2.879 to 3.754 GB | 3.844 GB | +6.6% | −2.3% | 112,896 | 76,832 to 103,488 | 100,519 | +12.3% | inside |
| Card | Setup | Conversation, tokens | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|
| RTX 3090 | 95%, 16-bit cache | 1,024 | 53.42 | 42.91 to 47.05 | 46.82 | +14.1% | inside |
| RTX 3090 | 95%, 16-bit cache | 8,192 | 51.90 | 41.71 to 45.92 | 44.91 | +15.6% | inside |
| RTX 3090 | 98.9%, 8-bit cache, text only | 1,024 | 53.54 | 43.01 to 47.14 | 46.85 | +14.3% | inside |
| RTX 3090 | 98.9%, 8-bit cache, text only | 8,192 | 52.77 | 42.40 to 46.57 | 45.90 | +15.0% | inside |
| RTX 3090 | 98.9%, 8-bit cache, text only | 32,768 | 50.29 | 40.42 to 44.70 | 43.90 | +14.5% | inside |
| RTX 3090 | 98.9%, 8-bit cache, text only | 65,536 | 47.32 | 38.06 to 42.42 | 41.52 | +14.0% | inside |
| Card | Setup | Prompt, tokens | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|
| RTX 3090 | 98.9%, 8-bit cache, text only | 1,024 | 0.781 | 0.714 to 0.770 | 0.787 | −0.8% | −2.2% |
| RTX 3090 | 98.9%, 8-bit cache, text only | 8,192 | 6.436 | 5.929 to 6.379 | 6.400 | +0.6% | −0.3% |
| RTX 3090 | 98.9%, 8-bit cache, text only | 32,768 | 28.358 | 26.939 to 28.735 | 28.954 | −2.1% | −0.8% |
| RTX 3090 | 98.9%, 8-bit cache, text only | 65,536 | 63.695 | 62.547 to 66.139 | 67.188 | −5.2% | −1.6% |
| Card | Setup | Conversation, tokens | Users at once, vLLM (modeled) | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|---|
| RTX 3090 | 98.9%, 8-bit cache, text only | 16,384 | 2 (2) | 49.3 | 39.3 to 43.6 | 40.8 | +20.8% | inside |
Qwen3.8-27B on the H200 NVL
On an H200 NVL the estimated time to first token is 44% to 52% short of vLLM's, as vLLM runs attention there in 8 bits, slower than the 16 bits the estimate takes. vLLM's time lies inside the practical range or 1.2% above it for prompts of about 4,000 tokens, and 9% to 12% above it from 129,189 tokens, mostly for the layer that drafts tokens ahead, which the calculator does not model.
Qwen3.8-27B's 8-bit checkpoint ran on an H200 NVL with the same vLLM, with 75% of the card, an 8-bit cache, text only, and its extra prediction layer drafting three tokens ahead (MTP). Each time is the first question on a new document, with no other request on the card.
With an 8-bit cache on the H100 NVL and H200 NVL, vLLM runs attention in 8 bits. Timed alone at these prompts' sizes, that kernel reached 389 TFLOPS, 23% of the card's 8-bit peak and less than the same kernel in 16 bits, 650 to 683 TFLOPS; the 8-bit matrix multiplications reached 65% to 72% of the 8-bit peak, the lower figure when the card held them at its power limit. The practical range takes these figures. The estimate keeps its sources' 16-bit attention: no source gives a figure for this kernel in 8 bits.
The layer that drafts tokens ahead processes every prompt too, one more full-attention layer with its weights: about 2.2 s at 254,332 tokens. With it, and the kernels that convert the inputs of the 8-bit matrix multiplications, which run on their own on this card, about 1.4 s more, the practical range comes within 2% to 9% of vLLM's times.
| Card | Setup | Prompt, tokens | Modeled | Practical | vLLM | Modeled vs vLLM | Practical vs vLLM |
|---|---|---|---|---|---|---|---|
| H200 NVL | 75%, 8-bit cache, text only, MTP | 3,845 | 0.150 | 0.298 to 0.320 | 0.303 | −50.3% | inside |
| H200 NVL | 75%, 8-bit cache, text only, MTP | 129,189 | 9.494 | 14.497 to 15.320 | 16.965 | −44.0% | −9.7% |
| H200 NVL | 75%, 8-bit cache, text only, MTP | 253,763 | 27.403 | 44.406 to 46.028 | 50.804 | −46.1% | −9.4% |
| H200 NVL | 75%, 8-bit cache, text only, MTP | 3,954 | 0.155 | 0.298 to 0.320 | 0.324 | −52.2% | −1.2% |
| H200 NVL | 75%, 8-bit cache, text only, MTP | 129,231 | 9.499 | 14.497 to 15.320 | 17.302 | −45.1% | −11.5% |
| H200 NVL | 75%, 8-bit cache, text only, MTP | 254,332 | 27.505 | 44.615 to 46.239 | 50.866 | −45.9% | −9.1% |
Cite as: Jacek Garbuliński, LLM GPU Token Speed Calculator, Blixild, 2026, https://blixild.com/llm-gpu-token-speed-calculator.html
Questions and Answers
When and why should I use this calculator?
Before buying or renting a card, or choosing a model's precision, to check that the model fits with the users and conversation lengths needed, how long each user waits for the first token and what speed each user then gets. It shows how memory and speed depend on each other: a longer conversation or more users take memory and slow each token, and several users' prompts delay each other's answers. The results rest on the model's files, the cards' data sheets, vLLM's code and published measurements; a test on the card decides.
Why does the speed depend on memory bandwidth rather than on the card's computing power?
For one user, each new token needs every weight it uses once, and each weight takes only a couple of arithmetic operations. A modern card does those operations faster than it can read the weights from its memory, so the reading sets the pace. That is why the bandwidth's limit, , bounds the speed, and why fewer bits per weight make the model faster as well as smaller.
What is the serving software's working memory, and why does it differ between cards?
It is the memory vLLM uses besides the weights and the conversations: the memory its largest step needs, the recorded CUDA graphs and its libraries' workspaces. vLLM finds it at start-up and keeps the rest for conversations. The calculator counts it from vLLM's code: for the 4-bit model at vLLM's default settings, 2.547 GB on an RTX 5090 and 2.518 GB on an RTX 3090, most of it the image encoder's step. In the runs vLLM took 0.79 and 0.19 GB more than counted, mostly for its CUDA graphs, so it left less for conversations than the calculator.
Why does a longer conversation slow down each token?
Because every new token reads the whole cache of the full-attention layers, which grows by with every token. On the RTX 5090 one user got 67.34 tokens per second at 1,024 tokens and 57.36 at 131,072; the calculator's estimate falls from 85.98 to 70.35, fast at both for the reasons under Checked Against Measurements. The linear-attention layers keep a fixed state instead, which is why the slowdown is moderate: only 16 of the 64 layers keep a cache.
Why does the 4-bit model run on an RTX 3090, which has no 4-bit arithmetic?
vLLM keeps the weights in 4 bits in memory and widens them to 16 bits as it uses them. The card still reads 4 bits per weight, so the reading, which sets the speed, is as for a native 4-bit card. It measured 41.09 tokens per second, 79% of the bandwidth's limit. That run used a memory share of 98%: at vLLM's default, 92%, the weights and vLLM's working memory leave no room for a conversation on this card, and vLLM does not start.
Does the 8-bit model fit on these cards?
By the calculator, not at vLLM’s default settings: its weights take 30.39 GB, more than the 23.24 GB vLLM may use on an RTX 3090, and on an RTX 5090 the weights and the working memory leave 1.796 GB too little; the RTX 3080, 4080, 5070 and 5080 hold 10 to 16 GB, less than the weights. On the A100 80GB PCIe, H100 NVL and H200 NVL it fits, with room for one user’s conversation of the model’s full 262,144 tokens. It was not measured.
How many users can share one card?
As many as the memory holds for their conversations, which Memory shows, up to vLLM's limit of 256 at once. Each further user costs far less than the first, because the card reads the weights once for all users in each step: on an RTX 5090 one user got 68.9 tokens per second, and 16 users 58.7 each, 940 together. Speed for Several Users estimates them, 13% to 38% fast at the steps measured, and From Prompt to Answer shows how long each user waits while the other users' prompts are processed.
Why does the first token take longer when several users send prompts at the same moment?
Because vLLM processes the prompts one after another, about one chunk of 1,568 tokens per step while answers are being written, so the last user's first token comes after roughly all the prompts' time. On an RTX 5090, 16 users with prompts of 1,024 tokens got their first tokens between 0.36 and 2.54 s. The answers already started slow down meanwhile, because each step with a prompt chunk takes as long as the chunk; From Prompt to Answer shows both.
Limitations
The calculator estimates memory and speed from data sheets, the model's files and the serving software's code, and a real setup can differ from it in the ways below.
- Bandwidth reached. Every byte is taken to move at 92% of the maker's bandwidth, the share a memory test reached. Matrix multiplications that unpack 4-bit or 8-bit weights as they read them usually reach less, by an amount that depends on the kernel, the number format and the card, and the speed is then lower than estimated.
- Kernels. The kernels a token runs are counted from the serving software's code, with the operations a compiler is expected to join counted as one. If it joins fewer, or a kernel reads memory more than once, each token takes longer, by about a microsecond for each further kernel.
- Serving software. The figures follow vLLM 0.31.1rc1 at its default settings. Another version or serving program, running without CUDA graphs, speculative decoding, or other limits on users and tokens per step change the kernels, the working memory and the order in which requests are processed.
- Processor. The processor starts a prompt's kernels one by one while the card works. A slower or busy processor makes the card wait, and the first token comes later than estimated.
- Clocks, power and other programs. The data sheet's bandwidth and arithmetic rates are peak figures at the boost clock. A card held to a lower power limit, running hot, or shared with other programs or a display reaches less, and has less memory free.
- Memory. The memory programs can use depends on the driver and the operating system, and the serving software's working memory on its version and settings; both change the tokens that fit.
- Arithmetic. A new token's time is taken from the bytes it reads and writes. With many users at once or very long conversations, the arithmetic grows with them and can take longer than the reading, and the speed is then lower than estimated.
- Requests. The time a request spends outside the card, in the network, the tokeniser and the server's queue, is not counted, nor the work of sampling settings, structured output or returned probabilities; each adds to the time the user waits.
- Models. Only the models listed are covered, from their files and configuration. A model with another layout, such as one with full attention only or with experts, or in another number format, has other bytes per token, another cache and other kernels.
- One card. A model split over several cards adds the time of their communication, which is not modelled.
- Checks. The estimates were set beside measurements on two consumer cards with GDDR memory, the RTX 3090 and RTX 5090, for Qwen3.8-27B at 4 bits and Bielik-11B-v3.0 at 8 and 16 bits, and on the RTX 3090 for Arete-27B, also with text only; and on an H200 NVL for the time to first token of Qwen3.8-27B's 8-bit checkpoint. The RTX 3080, 4080, 5070 and 5080, the A100 80GB PCIe and H100 NVL, and the H200 NVL's other results are estimated without a check.
Sources
Ghosh, A., Nayak, A., Panwar, A. and Basu, A. (2025). PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch. arXiv:2503.19779v3, §2. https://arxiv.org/
Hoffman, H. and Oh, F. (2024). Constant Time Launch for Straight-Line CUDA Graphs and Other Performance Enhancements. NVIDIA Technical Blog, 11 September 2024. Table 1. https://developer.nvidia.com/
Jarmusch, A., Graddon, N. and Chandrasekaran, S. (2025). Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks. arXiv:2507.10789v2, §IV.D. https://arxiv.org/
Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J. and Stoica, I. (2025). Continnum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live. arXiv:2511.02230v7, Table 4. https://arxiv.org/
Luo, W., Fan, R., Li, Z., Du, D., Wang, Q. and Chu, X. (2024). Benchmarking and Dissecting the Nvidia Hopper GPU Architecture. IPDPS 2024. arXiv:2402.13499v1, §IV.B and §IV.C. https://arxiv.org/
NVIDIA (2025). NVIDIA RTX Blackwell GPU Architecture, V1.1. Table 1, GeForce RTX 5090 vs GeForce RTX 4090 vs GeForce RTX 3090; Table 4, GeForce RTX 5080 vs GeForce RTX 4080 vs GeForce RTX 3080; Table 6, GeForce RTX 5070 vs GeForce RTX 4070 vs GeForce RTX 3070; GDDR7 Memory Subsystem; Appendix A. https://images.nvidia.com/
NVIDIA (2026). NVIDIA A100, H100 GPU and H200 GPU, product pages, Specifications: A100 80GB PCIe, H100 NVL, H200 NVL. https://www.nvidia.com/
NVIDIA (2026). Qwen3.8-27B-NVFP4, config.json, revision 482ca0f. Hugging Face. https://huggingface.co/
Qwen (2026). Qwen3.8-27B-FP8, config.json, revision 017b9c7. Hugging Face. https://huggingface.co/
NVIDIA (2026). cuBLAS Library, v13.4, §2.4.8. https://docs.nvidia.com/
Zhang, J., Wei, J., Wang, H., Zhang, P., Xu, X., Huang, H., Jiang, K., Chen, J. and Zhu, J. (2025). SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training. arXiv:2505.11594v3, A.4 and Figure 4. https://arxiv.org/
vLLM (2026). vLLM 0.31.1rc1.dev273, commit 2eb491e18: vllm/
vgerber (2025). IsaacSim 5.x crashes when extension with ogn is disabled while action graph with extension nodes is present. GitHub, isaac-sim/IsaacSim, issue #370, 19 December 2025: nvidia-smi -q of a GeForce RTX 5090, driver 570.195.03. https://github.com/
Send an enquiry
To run a language model on your own card, or to rent compute for one, send an enquiry.
To measure the speed of the models we already serve rather than estimate it, try them through our API at ai.blixild.com, paid per token.
Describe your requirements briefly. We respond within one business day.
contact@blixild.com