LLM GPU Token Speed Calculator

This LLM GPU token speed calculator follows a prompt sent to a large language model (LLM) on a graphics card (GPU) until its answer is complete, in six steps, and checks that the model and its conversations fit in the card's memory (VRAM). Model and Card takes the model's and the card's figures, Memory fits the users' conversations beside the weights and the serving software's working memory, Prompt Processing gives the time to the first token of the answer, Speed for One User the tokens per second that follow, Speed for Several Users the same when the card writes several answers at once, and From Prompt to Answer shows each user's answer appearing as the serving software schedules them.

LLM GPU token speed calculator: from inputs to resultsThe six steps with their inputs and results, in the order a prompt meets them: the model and the card, the memory, the prompt's processing, the speed for one user and for several, and the answers as they appear.InputsStepsResultsModel and weights precision, card,cache precision, imagesModel and CardWeights W, cache per token K, fixedstate S, card memory M and bandwidth BW, K, S, M, BUsers at once, conversation length(tokens), memory share u (%)MemoryFits or not; memory for conversationsand tokens per user: theoretical, modeledand practicalC, U, TPrompt length (tokens)PromptProcessingTime to first token: theoretical, modeledand practical; longest prompt withoutpreemptiontime to first tokenSpeed for OneUserThe theoretical maximum, the modeledestimate and the practical estimate oftokens per secondone user's stepSpeed for SeveralUsersUsers at once, and each user's and allusers' tokens per second: the theoreticalmaximum, the modeled estimate and thepractical estimateall users' stepAnswer length (tokens)From Prompt toAnswerEach user's first token and whole answer:theoretical, modeled and practical; theanswers as they appear

A token is a word or a part of a word. A language model reads a conversation and writes its answer one token at a time, so the tokens per second set how fast the answer appears.

The calculator covers Qwen3.8-27B with its weights in 4 or 8 bits, Arete-27B in 4 bits and Bielik-11B-v3.0 in 8 or 16 bits, on a GeForce RTX 3080, 3090, 4080, 5070, 5080 or 5090, an A100 80GB PCIe, an H100 NVL or H200 NVL, or a card of your own.

Its estimates come from the cards' data sheets, the models' files and the serving software's code, and its practical estimates from what vLLM's own steps reached on two cards, a GeForce RTX 3090 and an RTX 5090, and its 8-bit kernels on an H200 NVL. Checked Against Measurements sets the results beside vLLM's on those three cards, with the error at each point. Every other card and model is estimated without a check, and the page says so beside its results.

The results can be downloaded as a PNG or copied as text at the end of the page.

Worked Example

The calculator opens with Qwen3.8-27B, its weights stored in 4 bits each (NVFP4), on one GeForce RTX 5090, for one user with a conversation of 4,096 tokens and its cache in 8 bits. The weights take 21.07 GB of the 33.85 GB that programs can use on the card. The serving software, vLLM, may use 92% of it, 31.14 GB, and needs 2.547 GB of that for its own work, which leaves 7.521 GB for conversations: 217,952 tokens for one user, as vLLM counts them, less than the longest conversation the model accepts, 262,144 tokens. In practice, with what vLLM took in its own start on the cards measured, 6.281 to 7.150 GB is left, 180,320 to 206,976 tokens.

Each new token runs 821 kernels, reads 17.61 GB of weights and reads and writes 0.4615 GB for the user: the fixed state, the activations and the cache. At 92% of the card's 1,792 GB/s the weights take 10.68 ms and the user's bytes 0.2799 ms, and the kernels take 0.7292 ms to start and reach memory, so one user gets 85.55 tokens per second. The bandwidth alone would allow 99.17. In practice, with the figures measured in vLLM's own steps, one user gets 66.74 to 74.21.

Before the first token of the answer, the card processes the prompt of 3,584 tokens in 3 chunks of up to 1,568 tokens. Their matrix multiplications take 206.8 ms at the tensor cores' rates, full attention 14.19 ms, the linear attention 8.487 ms and the small kernels' reading and writing 63.07 ms; the engine adds 2.850 ms between chunks. The first token comes after 0.2954 s, and the answer of 512 tokens is complete after 6.266 s. In practice the first token comes after 0.4476 to 0.4519 s, and the answer is complete after 7.331 to 8.105 s.

With four users sending such prompts at the same moment, vLLM processes the prompts one after another, so the first tokens come after 0.4199 s to 1.182 s. While all four write, each gets 79.81 tokens per second, 319.3 together, and the last answer is complete after 7.571 s. In practice each gets 61.13 to 68.49.

Open the calculator

The calculator, with its equations and assumptions, opens from a link we send to your email address. We use the address to send the link, and write to you about our services only if you tick the newsletter box. Details in the privacy policy.

Model and Card

This step takes the model's weights and conversation memory, and the card's memory and bandwidth.

Each new token is computed from the model's weights, the numbers it learned in training, and from the user's conversation memory, the numbers it keeps about the conversation so far. Both are read from the card's memory for every token, so their size sets both what fits and how fast the answer comes. The weights are measured part by part from the model's files; what each part is for and what it costs are under the diagrams below.

The precision is the number of bits each weight is stored in. Fewer bits take less memory and fewer bytes to read for each token, so the model fits on a smaller card and answers faster. NVFP4 (4-bit) and FP8 (8-bit) are NVIDIA's number formats; W4A16 stores the weights as 4-bit whole numbers and does the arithmetic in 16 bits. 27B means 27 billion weights. Arete-27B is a fine-tune of Qwen3.8-27B with the same layers.

The graphics card. Its memory sets what fits, and its memory bandwidth, the bytes per second it can read from that memory, sets the speed.

Conversation cache precision

The cache keeps, for every token of the conversation, the keys and values the full-attention layers computed for it, so they are not computed again for each new token. Fewer bits per number fit more tokens in memory and read faster: 8-bit holds twice as many tokens as 16-bit.

Images

Qwen3.8-27B and Arete-27B read images as well as text. With text only, vLLM does not load their image encoder, which frees about 0.9 GB of weights, but its own memory test then sets aside more working memory: on an RTX 3090, conversations gained only about 0.3 GB. The practical estimate takes the working memory measured there; the modeled estimate does not, because the cause is not yet known. Bielik-11B-v3.0 reads text only.

Weights on the card WW21.07 GBread per token 17.61 GB
Cache per token KK32.77 kBin 16 full-attention layers
Fixed state per user SS153.9 MBin 48 linear-attention layers
Card memory MM33.85 GBbandwidth 1,792 GB/s

How Qwen3.8-27B NVFP4 makes one new token

Solid outlines run for every new token; dashed ones only when used. The sizes are measured from the model's files.

What each part does and costs

  • Embeddings. Why: the layers work on numbers, so each token is first replaced by its row in this table, one row of numbers for each of the 248,320 tokens the model knows. Performance: 2.543 GB of memory, but only the token's row, 10.24 kB, is read for each token, so the table costs memory and almost no speed.
  • Image encoder. Why: it lets the model take images, by turning an image into tokens the layers can read. Performance: 0.9215 GB of memory; it runs only when an image is sent, so it does not slow down text.
  • Layers. Why: they do the model's work; every token passes through all 64 of them in turn, and each layer refines the token's numbers with its weights and the earlier tokens. Performance: their weights, 16.89 GB, are read for every token, 93.48% of the bytes each token reads and writes. Their parts are in the next diagram.
  • Hidden state. The token's list of 5,120 numbers as it leaves the last layer. It is worked out on the card as the token passes through and is not kept per user, so it costs neither memory nor speed worth counting.
  • Output layer. Why: it turns the hidden state into a score for each of the tokens the model knows, and the next token is picked from those scores. Performance: 0.7152 GB, read for every token, 3.958% of the bytes each token reads and writes.
  • Extra prediction layer. Why: it guesses the token after next, for speculative decoding: the serving software checks the guess while it makes the next token, so one read of the weights can give two tokens when the guess is right. Performance: 0.8494 GB more memory when it is loaded; vLLM loads it only when speculative decoding is switched on, so the calculator leaves it out.

Inside one layer, and one user's fixed state

What each part does and costs

  • Full attention. Why: it lets the token look back at every earlier token of the conversation exactly, through the key and value kept for each of them, the cache. Performance: the cache grows by 2,048 B a token in each of the 16 full-attention layers, 32.77 kB a token in all, and every new token reads all of it: at 4,096 tokens that is 134.2 MB, 0.7428% of the bytes each token reads and writes. The longer the conversation, the more memory and the slower each token.
  • Linear attention. Why: it keeps one summary of a fixed size, the fixed state, instead of a cache, so its memory and its reads do not grow with the conversation; a summary of a fixed size cannot keep every detail of a long conversation, which the full-attention layers keep exactly. Performance: 3.207 MB per layer and user at any length; the model has 48 of these layers to 16 full-attention layers.
  • Heads. Why: attention is split into heads, parallel parts that each follow the conversation in their own way. Performance: the cache and the fixed state grow with the number of heads and with their size, the numbers in each key or value.
  • Short convolution. Why: before the linear attention, a small filter mixes each token's numbers with those of the 3 tokens before it, so nearby words are taken together. Performance: it keeps those tokens, 61,440 B per layer and user, small next to the fixed state.
  • MLP (multilayer perceptron). Why: it works on each token's numbers on their own, widening its 5,120 numbers to 17,408 and narrowing them back; it holds most of the model's weights. Performance: 0.1504 GB a layer, 9.626 GB in all, read for every token, 53.27% of the bytes each token reads and writes; it keeps nothing per user.

How the fixed state works

A linear-attention layer keeps no cache of past tokens. Each of its 48 heads keeps one table of 128 × 128 numbers, its fixed state, which sums up the conversation so far. For every new token, a head turns the token into three lists of 128 numbers, a key (what the token is about), a value (what it adds) and a query (what it looks for), and works on its table in three steps:

  1. Fade: every number in the table is multiplied by the same factor, just under 1, which the layer works out from the token, so older content counts a little less.
  2. Write: what the table holds under the key is moved towards the value, by a share the layer also works out from the token; the value replaces part of what was there rather than being added on top.
  3. Read: the query is multiplied by the table, which gives the head's output; the stored values whose keys are most like the query count most.

The table is the same size however long the conversation is: 48 × 128 × 128 numbers of 4 bytes (stored in 32 bits) = 3.146 MB a layer for each user. With the state of the layer's short convolution, the last 3 tokens of its 10,240 channels, 2 bytes each, a layer holds 3.207 MB per user, and the 48 linear-attention layers hold SS = 153.9 MB. A full-attention layer instead keeps the key and value of every token, so its cache grows by KK with every token.

Equations

K=2LfullnkvdbK = 2\, L_{\mathrm{full}}\, n_{\mathrm{kv}}\, d\, b S=Llin(nvdlin2bs+c(k−1)bf)S = L_{\mathrm{lin}}\, (n_v\, d_{\mathrm{lin}}^2\, b_s + c\, (k - 1)\, b_f)

where KK is the cache per token, in bytes; 2 counts a key and a value; LfullL_{\mathrm{full}} is the number of full-attention layers, nkvn_{\mathrm{kv}} their heads that keep keys and values and dd the numbers in each key or value; bb is the bytes per cached value, 1 or 2; SS is the fixed state per user, in bytes; LlinL_{\mathrm{lin}} is the number of linear-attention layers, nvn_v their heads and dlind_{\mathrm{lin}} the size of a head; bsb_s, 4 bytes, is the bytes per number of the state; cc is the numbers per token the short convolution keeps, kk the tokens it spans and bfb_f, 2 bytes, the bytes per number of its state.

Cache per token 2 × 16 × 4 × 256 × 1 B = 32,768 B = 32.77 kB; fixed state 48 × (48 × 128² × 4 B + 10,240 × 3 × 2 B) = 153.9 MB per user.

Wread=W−Wemb−WimgW_{\mathrm{read}} = W - W_{\mathrm{emb}} - W_{\mathrm{img}} c=(2nk+nv)dlinc = (2\, n_k + n_v)\, d_{\mathrm{lin}}

WW is everything the serving software loads: every part of the model's files but the extra prediction layer, which vLLM does not load unless speculative decoding is switched on. WreadW_{\mathrm{read}} is the part read for every new token: WW without the embeddings WembW_{\mathrm{emb}}, of which only the token's row is read, and the image encoder WimgW_{\mathrm{img}}, which runs only for an image.

cc is the number of channels of the linear-attention layers' short convolution: a key and a query for each of the nkn_k key heads and a value for each of the nvn_v value heads, each dlind_{\mathrm{lin}} numbers long. The convolution keeps the last k−1k - 1 tokens of each channel.

Assumptions

  • The weights are those of the model's files at one fixed version; vLLM held 1.0% (RTX 5090) and 1.4% (RTX 3090) more, which the calculator does not count.
  • The card memory MM is the data sheet's less the driver's reserve, 488 MiB, as Memory explains.
  • The A100 80GB PCIe, H100 NVL and H200 NVL take their memory, bandwidth and tensor-core rates from NVIDIA’s product pages, each dense rate half the pages’ rate with sparsity; none of them was measured (Limitations).

Memory

This step checks that the weights, the serving software's working memory and every user's conversation memory fit in the share of the card's memory the serving software may use.

A user's conversation memory is the fixed state plus the cache of every token so far; vLLM keeps the fixed state twice for prefix caching, once while the conversation is within its first block. The serving software, here vLLM, a widely used open-source program that runs language models for many users at once, keeps 8% of the card's memory free by default, so it may use 92%, and it needs part of that for its own work, its working memory, counted here from its code.

Beside the estimate the step gives two more counts of the tokens per user. The theoretical maximum takes the share uu of the data sheet's memory less the weights, with no driver's reserve and no working memory: no serving software can leave more. The practical estimate puts what vLLM took in its own start on the cards measured in place of the design's figures, as a range.

The conversations the card serves at the same time.

The tokens of one conversation so far: the prompt, the documents in it and the answers. It is also called the context length.

vLLM's default is 92%. On an RTX 3090 the 4-bit model needs 98%: at 92% vLLM does not start.

FitsYes24.06 GB of 31.14 GB
Theoretical maximum, tokens per user Tfit,maxT_{\mathrm{fit},\mathrm{max}}262,14410.54 GB for conversations; the model's maximum, 312,226 would fit
Modeled estimate, tokens per user TfitT_{\mathrm{fit}}217,9527.521 GB for conversations; 220,131 before whole blocks
Practical estimate, tokens per user Tfit,pracT_{\mathrm{fit},\mathrm{prac}}180,320 to 206,9766.281 to 7.150 GB for conversations

Card memory, to scale

What each part does and costs

  • Reserved memory. Why: the card's driver keeps part of the memory for itself. Performance: 488 MiB, as an RTX 5090's nvidia-smi reported it; the card memory MM is what is left of the data sheet's.
  • Libraries. Why: cuBLAS, NVIDIA's library of matrix multiplications, keeps a workspace for its methods. Performance: 32 MiB on cards from the Hopper architecture on, such as the RTX 5090, and 4 MiB on older ones, such as the RTX 3090; inside RR.
  • Attention workspace. Why: with an 8-bit cache vLLM runs attention with FlashInfer, which keeps a workspace for the partial results of its kernels. Performance: 394 MiB with an 8-bit cache and none with a 16-bit one, inside RR.
  • Working memory for one step. Why: vLLM sets aside the most memory its largest step needs, found at start-up by running each kind of step once: one image of the largest size through the image encoder, a step of 2,048 prompt tokens (8,192 on the H100 NVL and H200 NVL) and the choice of the next token for 256 users at once. Performance: the image encoder 1.852 GB, the prompt step 0.5956 GB and the choice 0.6357 GB; the largest, 1.852 GB, is inside RR.
  • CUDA graphs. Why: vLLM records the card's work for one step, for steps of 1 to 512 tokens, so it can replay it without the processor starting each step, which makes every token faster. Performance: 248.5 MB by vLLM's code, inside RR. In the runs, with vLLM's limit of 16 users at once, its own estimate was 0.54 GiB on the RTX 5090 and 0.19 GiB on the RTX 3090; without the graphs vLLM left 0.21 GiB more for conversations on the RTX 3090. Their effect on speed was not measured: with them switched off, vLLM also skips preparing its kernels in advance, and its first tokens took about 10 s each.
  • Blocks. Why: vLLM gives out conversation memory in blocks, so many conversations can grow and end without moving memory about. Performance: a block holds 1,568 tokens of cache, 51.38 MB here, and vLLM keeps one block spare; the tokens that fit are rounded down to whole blocks.
  • Prefix caching. Why: vLLM keeps the processed text of earlier requests, so that a new request starting with the same text, such as the same instructions or document, need not process it again; for the fixed state it keeps the state at the end of each whole block, and two states per user are in memory at a time once the conversation is longer than one block. It is vLLM's default. Performance: one more fixed state per user, 153.9 MB here, in two blocks for each group of 16 linear-attention layers.

Equations

Cm=nsS+TKC_m = n_s\, S + T\, K W+R+UCm≤uMW + R + U\, C_m \le u\, M Tfit=(uM−W−R)/U−2SKT_{\mathrm{fit}} = \frac{(u\, M - W - R)/U - 2\,S}{K}

where CmC_m is the memory one user's conversation holds, nsn_s the copies vLLM keeps of the fixed state SS, 1 while the conversation is within its first block of btb_t tokens and 2 after, TT the conversation length in tokens and KK the cache per token; WW is the weights on the card, RR the serving software's working memory, UU the users at once, uu the share of memory the serving software may use and MM the card memory; TfitT_{\mathrm{fit}} is the number of tokens that fit per user, before vLLM's whole blocks.

Conversation memory 2 × 153.9 MB + 4,096 × 32.77 kB = 442.1 MB per user; needed 21.07 GB + 2.547 GB + 1 × 0.4421 GB = 24.06 GB ≤ 92% × 33.85 GB = 31.14 GB: it fits; tokens per user ((31.14 GB − 21.07 GB − 2.547 GB) ÷ 1 − 2 × 153.9 MB) ÷ 32.77 kB = 220,131, 217,952 in vLLM's whole blocks.

bt=16⌈Sl/(16Kl)⌉b_t = 16\, \lceil S_l/(16\, K_l) \rceil n=⌊(uM−W−R)/(LfullbtKl)⌋n = \lfloor (u\, M - W - R)/(L_{\mathrm{full}}\, b_t\, K_l) \rfloor TvLLM=bt(⌊(n−1)/U⌋−2Llin/Lfull)T_{\mathrm{vLLM}} = b_t\, (\lfloor (n - 1)/U \rfloor - 2\, L_{\mathrm{lin}}/L_{\mathrm{full}})

CmC_m is the memory one conversation holds: its fixed state SS, twice once the conversation is longer than one block, and the cache of the full-attention layers, KK for each of its TT tokens. The model fits when its weights WW, the serving software's working memory RR and the UU users' conversations together take no more than the share uu of the card memory MM. Solving the same sum for TT gives the longest conversation each user can have, TfitT_{\mathrm{fit}}.

vLLM counts in blocks of btb_t tokens: the smallest multiple of 16 whose cache in one layer, btKlb_t K_l, with KlK_l the cache per token in one layer, holds one layer's fixed state SlS_l. A block spans the LfullL_{\mathrm{full}} full-attention layers; nn blocks fit in the memory left. One is kept spare, and each user's fixed state takes two blocks in each group of LfullL_{\mathrm{full}} linear-attention layers, so each user can have TvLLMT_{\mathrm{vLLM}} tokens, the result shown. A conversation of at most btb_t tokens takes one block in each group, so more users of short conversations fit than the formula gives.

M=Msheet−MresM = M_{\mathrm{sheet}} - M_{\mathrm{res}} R=Rlib+Rws+max(Renc,Rfwd,Rsmp)+RgrR = R_{\mathrm{lib}} + R_{\mathrm{ws}} + \max(R_{\mathrm{enc}},\ R_{\mathrm{fwd}},\ R_{\mathrm{smp}}) + R_{\mathrm{gr}}

MM is the card's memory in its data sheet, MsheetM_{\mathrm{sheet}}, less the driver's reserve, MresM_{\mathrm{res}} = 488 MiB. No NVIDIA document gives the reserve; 488 MiB is what an RTX 5090's nvidia-smi reported with driver 570, and the calculator takes it for every card.

RR is the memory vLLM takes besides the weights before it sizes the memory for conversations, counted from its code at its default settings: at most 256 users at once and 2,048 tokens in a step. RlibR_{\mathrm{lib}} is cuBLAS's workspace, and RwsR_{\mathrm{ws}} FlashInfer's, with an 8-bit cache only.

The largest step is the largest of RencR_{\mathrm{enc}}, the image encoder on an image of the largest size, 0 for a model without one; RfwdR_{\mathrm{fwd}}, a step of 2,048 prompt tokens, with the image's output held; and RsmpR_{\mathrm{smp}}, the choice of the next token for 256 users, whose scores over the vocabulary are held in 16 and 32 bits. RgrR_{\mathrm{gr}} is the CUDA graphs: for each of vLLM's two kinds, the activations of its largest step and 1 MiB for each further graph, the least vLLM's own estimate takes.

Card memory 32 GiB − 488 MiB = 33.85 GB; working memory 33.55 MB + 413.1 MB + max(1.852 GB, 0.5956 GB, 0.6357 GB) + 248.5 MB = 2.547 GB.

Mc,max=uMsheet−WM_{c,\mathrm{max}} = u\, M_{\mathrm{sheet}} - W Mc,prac=u(M+ΔM)−W(1+w)−(R+ΔR)M_{c,\mathrm{prac}} = u\,(M + \Delta M) - W\,(1 + w) - (R + \Delta R)

Mc,maxM_{c,\mathrm{max}} is the theoretical maximum of the memory for conversations, the share uu of the data sheet's memory less the weights as the files hold them; the tokens per user, Tfit,maxT_{\mathrm{fit},\mathrm{max}}, follow from it as TfitT_{\mathrm{fit}} does, before vLLM's whole blocks.

Mc,pracM_{c,\mathrm{prac}} is the practical estimate, with three terms measured in vLLM's own start with Qwen3.8-27B NVFP4: the memory programs could use beside MM, ΔM\Delta M = −170 MiB on the RTX 5090 and +38 MiB on the RTX 3090, by nvidia-smi; the weights vLLM loaded beyond the files', ww = 1.05% and 1.40% of WW; and vLLM's working memory beyond RR, ΔR\Delta R = 0.781 and 0.188 GB, mostly its CUDA graphs. The range takes every term at the value that leaves less memory for conversations, then at the one that leaves more, for every card and model; its tokens per user are counted in vLLM's whole blocks.

Theoretical maximum 92% × 34.36 GB − 21.07 GB = 10.54 GB, tokens per user (10.54 GB ÷ 1 − 2 × 153.9 MB) ÷ 32.77 kB = 312,226. Practical, slow end 92% × (33.85 GB − 0.1783 GB) − 21.07 GB × 1.014 − (2.547 GB + 0.7809 GB) = 6.281 GB, 180,320 tokens in vLLM's whole blocks; fast end 7.150 GB, 206,976 tokens.

Assumptions

  • Every user's conversation has the same length.
  • The working memory RR is counted from vLLM's code at its default settings. Other settings and other versions use other amounts, and the code of the kernels, loaded as each is first used, is not counted: no source gives its size.
  • The driver's reserve is the same for every card and driver, 488 MiB.
  • The cache is kept for every token of a conversation; none is moved out of the card.
  • The practical estimate's terms were measured for Qwen3.8-27B NVFP4 on two cards. With Bielik-11B-v3.0, which took no part, vLLM's memory for conversations lies inside the range on the RTX 5090; on the RTX 3090 vLLM left more than the range's fast end, by 0.2% with its 8-bit weights and by 52% with its 16-bit ones, 0.41 against 0.27 GB: it loaded Bielik's weights as the files hold them, and Bielik's CUDA graphs took 0.11 GiB or less.

Prompt Processing

This step gives the time from sending a prompt to the first token of the answer, from the arithmetic of processing the prompt in chunks.

Before it writes the answer, the model processes the whole prompt: every prompt token passes through every layer, as a new token does, but many tokens at a time, so each weight read from memory serves a whole chunk of them. Processing a prompt is therefore limited by the card's arithmetic rather than by its bandwidth. The matrix multiplications take two operations per weight and token, and full attention compares each prompt token with every token before it, which grows with the square of the prompt's length.

vLLM puts at most 2,048 tokens through the card in one step, 8,192 on the H100 NVL and H200 NVL, and processes a prompt in chunks of that size; for a model with linear attention it cuts each back to whole blocks of btb_t = 1,568 tokens, keeping the fixed state at the end of each. The arithmetic runs at the tensor-core rates of the card's data sheet, at the share the sources measured: 95.5% for the matrix multiplications and 85.0% for attention.

While it processes a prompt, the processor hands the card each of a chunk's kernels itself, about 14 µs each; a chunk takes the card's work or the processor's, whichever is longer, and about 1 ms more before the next step.

Beside the estimate the step gives two more times. The theoretical minimum does the arithmetic at the card's highest rates in its data sheet and reads the weights once at the full bandwidth, with nothing else: no card and no serving software can be faster. The practical estimate puts what vLLM's own prompt steps reached on the two cards measured in place of the sources' figures, as a range from both cards' slower figures to their faster ones; on the H100 NVL and H200 NVL, for 8-bit attention and 8-bit matrix multiplications, what those kernels reached on an H200 NVL.

The tokens the model processes before it answers: the instructions, the question and any documents sent with them.

Theoretical minimum tfirst,mint_{\mathrm{first},\mathrm{min}}0.1550 s23,127 prompt tokens per second
Modeled estimate tfirstt_{\mathrm{first}}0.2954 s3 chunks of up to 1,568 tokens; 12,133 prompt tokens per second
Practical estimate tfirst,pract_{\mathrm{first},\mathrm{prac}}0.4476 to 0.4519 s7,930 to 8,008 prompt tokens per second
Longest prompt without preemption213,248 tokenswith this memory for conversations

Time to first token, to scale

What each part does and costs

  • Chunks. Why: vLLM puts at most 2,048 tokens through the card in one step, 8,192 on the H100 NVL and H200 NVL, and processes a long prompt in pieces of whole blocks within that, keeping the fixed state at the end of each, so that a later prompt starting with the same text can reuse it. Performance: each chunk reads the weights once, and the engine takes 0.95 ms between steps.
  • Matrix multiplications. Why: as for a new token, each prompt token is multiplied by every weight it uses, two operations per weight. Performance: FF = 48.70 billion operations a token; at the RTX 5090's tensor-core rates for the model's formats, 884.2 TFLOPS, × 95.5%, 57.7 µs a token, the largest part of the time up to 32,768 tokens.
  • Full attention. Why: each prompt token is compared with every token before it in the 16 full-attention layers, which is how the model finds what it needs anywhere in a long document. Performance: it grows with the square of the prompt's length: 14.19 ms at 3,584 tokens, 5% of the time; about half the time at 65,536 tokens, and 65% at 131,072.
  • Linear attention. Why: the linear-attention layers compute their fixed state chunk by chunk, 64 tokens at a time, instead of token by token. Performance: FlinF_{\mathrm{lin}} = 0.42 billion operations a token, 8.49 ms at 3,584 tokens.
  • The small kernels. Why: the norms, additions, gates and number conversions between the matrix multiplications, each reading and writing the tokens' numbers. Performance: EE = 29.0 MB a token, 63.07 ms at 3,584 tokens, 21% of the time.
  • The processor. Why: vLLM runs a prompt's chunk without a CUDA graph, so the processor issues each of its kernels. Performance: 1,157 kernels × 14.3 µs = 16.5 ms a chunk, shorter than the card's work on a chunk of 1,568 tokens, so the card does not wait for it.

Equations

ci=max(niFεgRg,WreadηB)+Ai+niFlinεaR16+niEηBc_i = \max\Big( \frac{n_i\, F}{\varepsilon_g R_g},\ \frac{W_{\mathrm{read}}}{\eta\, B} \Big) + \frac{A_i + n_i F_{\mathrm{lin}}}{\varepsilon_a R_{16}} + \frac{n_i\, E}{\eta\, B} tfirst=∑i[max(ci,Nptop)+tstep]t_{\mathrm{first}} = \sum\nolimits_i \Big[ \max\big( c_i,\ N_p\, t_{\mathrm{op}} \big) + t_{\mathrm{step}} \Big] TPSprompt=Ptfirst\mathrm{TPS}_{\mathrm{prompt}} = \frac{P}{t_{\mathrm{first}}}

where tfirstt_{\mathrm{first}} is the time to the first token; the sum runs over the prompt's chunks ii of nin_i tokens; cic_i is the card's work on a chunk; FF is the matrix multiplications' operations per token, RgR_g the card's tensor-core rate for the model's number formats and εg\varepsilon_g the share they reach; Wread/(ηB)W_{\mathrm{read}}/(\eta\, B) is the weights' reading, as in Speed for One User; AiA_i is full attention's operations in the chunk and FlinF_{\mathrm{lin}} the linear attention's per token, at the 16-bit rate R16R_{16} × εa\varepsilon_a; EE is the small kernels' bytes per token; NpN_p is a chunk's kernels and topt_{\mathrm{op}} the processor's time per kernel; tstept_{\mathrm{step}} is the engine's time per step; TPSprompt\mathrm{TPS}_{\mathrm{prompt}} is the prompt tokens per second.

Chunks of 1,568, 1,568 and 448 tokens; matrix multiplications 3,584 × 48.70 GFLOP ÷ 844.1 TFLOPS = 206.8 ms; full attention 14.19 ms; linear attention 8.487 ms; the small kernels' reading and writing 63.07 ms; the card's work 292.5 ms; the processor issues 1,157 kernels a chunk at 14.27 µs each, 16.51 ms; each chunk takes the longer, then 0.9500 ms for the engine: time to first token 0.2954 s; prompt tokens per second 3,584 ÷ 0.2954 s = 12,133.

F=2∑jajbjkjF = 2 \sum\nolimits_j a_j\, b_j\, k_j Ai=4Lfullnqdni(si+ni2)A_i = 4\, L_{\mathrm{full}}\, n_q\, d\, n_i \Big(s_i + \frac{n_i}{2}\Big) ni=min(bt,P−si)n_i = \min(b_t,\ P - s_i)

FF counts two operations, a multiplication and an addition, for each weight the matrix multiplications use per token: kjk_j matrices of aja_j rows and bjb_j columns of each kind, in every layer and the output layer. For Qwen3.8-27B it is 48.70 billion. RgR_g is the card's tensor-core rate for those operations, each matrix at the data sheet's rate of its number format: 884.2 TFLOPS on an RTX 5090 for the 4-bit model, and 71.2 TFLOPS on an RTX 3090, whose Marlin kernels multiply in 16 bits.

AiA_i is the operations of full attention for a chunk of nin_i tokens that starts after sis_i tokens: each token's query is multiplied with the key of every token before it, si+ni/2s_i + n_i/2 of them on average, and the results with their values, two operations each, for its nqn_q query heads of dd numbers in each of the LfullL_{\mathrm{full}} full-attention layers.

The chunks are blocks of btb_t tokens, the last one shorter. A chunk's matrix multiplications take their arithmetic or the reading of the weights, whichever is longer: a short chunk is limited by the bandwidth like a new token. The linear attention computes its fixed state in pieces of 64 tokens, FlinF_{\mathrm{lin}} = 0.42 billion operations a token for Qwen3.8-27B, counted from vLLM's code. The small kernels between the matrix multiplications read and write EE = 29.0 MB a token on an RTX 5090, counted from the model's configuration.

A prompt's chunk runs a new token's kernels, with the linear attention's two in each layer replaced by nine for the chunked computation: NpN_p = 1,157 for the 4-bit model on an RTX 5090. vLLM runs them without CUDA graphs, so the processor issues them one by one while the card works on the earlier ones, and the chunk takes the card's work or the processor's time, whichever is longer. The engine then takes tstept_{\mathrm{step}} before the next step.

εg\varepsilon_g = 95.5%: a 16-bit matrix multiplication reached about 200 of the RTX 5090's 209.5 TFLOPS (Zhang et al., 2025). εa\varepsilon_a = 85.0%: FlashAttention reached 147 to 209 TFLOPS on the RTX 5090, 178 in the middle of that range (the same paper). topt_{\mathrm{op}} = 14.3 µs: 743 kernels took 14 ms without a CUDA graph and 3.4 ms with one (Ghosh et al., 2025). tstept_{\mathrm{step}} = 0.95 ms: vLLM's scheduling time per step (Li et al., 2025). None of the four was measured on the cards here or set from their runs.

On the H100 NVL and H200 NVL, vLLM multiplies 4-bit weights with its Marlin kernels, which use the tensor-core instructions of the cards before Hopper; on a Hopper card those instructions reach on average 62.9% of its peak rate (Luo et al., 2024). Their 4-bit rate is therefore 62.9% of the 16-bit rate, 525.5 TFLOPS, while the theoretical minimum keeps the 16-bit peak, 835.5 TFLOPS. The A100 80GB PCIe, RTX 3080 and RTX 3090 multiply 8-bit and 4-bit weights at their 16-bit rate, and the RTX 4080, which multiplies 8-bit numbers itself, its 4-bit weights: on the cards before Hopper those instructions reach about their peak rate (Luo et al., 2024).

tfirst,min=∑i[max(niFRgpeak,WreadB)+Ai+niFlinR16peak]t_{\mathrm{first},\mathrm{min}} = \sum\nolimits_i \Big[ \max\Big( \frac{n_i\, F}{R_g^{\mathrm{peak}}},\ \frac{W_{\mathrm{read}}}{B} \Big) + \frac{A_i + n_i F_{\mathrm{lin}}}{R_{16}^{\mathrm{peak}}} \Big] tfirst,prac=∑i[max(ciprac,Nptp)+tg]t_{\mathrm{first},\mathrm{prac}} = \sum\nolimits_i \big[ \max\big( c_i^{\mathrm{prac}},\ N_p\, t_p \big) + t_g \big] ciprac=max(niFgRg,WreadeWB)+AiaR16+niFlinlR16+niEsBc_i^{\mathrm{prac}} = \max\Big( \frac{n_i\, F}{g\, R_g},\ \frac{W_{\mathrm{read}}}{e_W\, B} \Big) + \frac{A_i}{a\, R_{16}} + \frac{n_i\, F_{\mathrm{lin}}}{l\, R_{16}} + \frac{n_i\, E}{s\, B}

tfirst,mint_{\mathrm{first},\mathrm{min}} is the theoretical minimum. RgpeakR_g^{\mathrm{peak}} and R16peakR_{16}^{\mathrm{peak}} are the card's highest rates in its data sheet for each number format, with 16-bit accumulation where it gives one: 838 TFLOPS for 8-bit numbers on an RTX 5090, and 142.3 for the 16-bit numbers of the RTX 3090's Marlin kernels; bfloat16, in which attention runs, has only its 32-bit-accumulate rate; on the H100 NVL and H200 NVL with an 8-bit cache, attention runs in 8 bits, at the 8-bit peak. The estimate uses the 32-bit-accumulate rates throughout. The small kernels, the processor and the engine add nothing.

tfirst,pract_{\mathrm{first},\mathrm{prac}} is the practical estimate. Its figures come from profiles of vLLM's prompt steps, Qwen3.8-27B NVFP4 on the two cards measured, each a part's time at the estimate's rates divided by its measured time: the matrix multiplications gg = 106.0% (RTX 5090) and 99.9% (RTX 3090), full attention aa = 68.4% (RTX 5090, at 65,536 tokens), the linear attention ll = 20.7% and 24.4%, and the small kernels ss = 220% and 129%: they moved fewer bytes than EE counts. The weights are read at Speed for One User's eWe_W.

On the H100 NVL and H200 NVL with an 8-bit cache, vLLM runs attention in 8 bits: there aa = 23.3% of the 8-bit peak, and the 8-bit matrix multiplications reach 65.3% to 71.7% of it, the lower figure when the card held them for 15 s at its power limit, timed with vLLM's own kernels on an H200 NVL at Qwen3.8-27B's sizes.

On the RTX 5090 a chunk waits for the processor, which took tpt_p = 121 µs for each kernel counted, by vLLM's own timers; on the RTX 3090 the card's longer work hides it. Between chunks the card stood idle for tgt_g = 8.9 and 10.4 ms. The range takes every figure at the slower of its values, then every one at the faster, for every card and model; a card without a figure takes the other's. A chunk's practical work is never shorter than its theoretical minimum.

Theoretical minimum: matrix multiplications 135.7 ms, full attention 12.05 ms, linear attention 7.211 ms, at the peak rates: 0.1550 s. Practical, slow end: the first chunk's work 124.7 ms at the measured figures, against 1,157 kernels × 121.3 µs = 140.3 ms for the processor; each chunk takes the longer, then 10.35 ms: 0.4519 s. Fast end, every figure at the faster card's value: 0.4476 s.

Assumptions

  • One prompt at a time, of new text: nothing of it is found in vLLM's prefix cache. A prompt that starts with text processed before, such as the same instructions, is faster.
  • εg\varepsilon_g, εa\varepsilon_a, topt_{\mathrm{op}} and tstept_{\mathrm{step}} are the same for every card and processor, from the sources above.
  • Full attention runs at the share FlashAttention reached at head dimension 128. This model's heads are 256 wide, and vLLM runs FlashInfer with an 8-bit cache; on the cards measured it reached less, so long prompts are estimated fast.
  • The processor issues an operation in topt_{\mathrm{op}}. On an RTX 5090 vLLM's operations took longer to issue, and the card waited for the processor within short chunks, so short prompts are estimated fast there (Checked Against Measurements).
  • Nothing outside the card is counted: the network, turning the text into tokens, and the server's queue.
  • Up to the longest prompt without preemption. Above it, vLLM runs out of blocks while it processes the prompt, sets the prompt aside and processes part of it again, which takes longer than shown, by an amount not modelled.
  • The practical estimate's figures were measured for Qwen3.8-27B NVFP4 on two cards. Bielik-11B-v3.0, which took no part in them, lies inside the range at 7 of 18 prompts: its long prompts were up to 6.2% faster and its short ones up to 8.6% slower; the causes were not profiled.

Speed for One User

This step gives the tokens per second one user gets, from the bytes each new token reads and writes, the share of the card's bandwidth the reading reaches and the kernels each token runs.

Making a new token is limited by the card's memory bandwidth, the bytes per second it can read from its memory: each token reads the weights it uses once and reads and writes the user's conversation memory, and the arithmetic on them takes less time than the reading. A memory throughput test reached 92% of the maker's bandwidth on a GeForce RTX 4090, and the calculator takes that share for every card.

The card does a token's work as a sequence of kernels, the small programs it runs one after another, about 800 of them, counted from vLLM's code. Each adds under a microsecond, together about 0.7 ms a token.

Beside the estimate the step gives two more figures. The theoretical maximum reads everything at the full bandwidth with no time per kernel: no card and no serving software can do better. The practical estimate puts, in place of the sources' figures, what vLLM's own decode steps reached on the two cards measured. It is a range, from both cards' slower figures to their faster ones, because a card or model nobody has measured has no figures of its own.

Theoretical maximum TPSmax\mathrm{TPS}_{\mathrm{max}}99.17 tokens/sreading alone at the full bandwidth
Modeled estimate TPS\mathrm{TPS}85.55 tokens/sreading at 92% of the bandwidth
Practical estimate TPSprac\mathrm{TPS}_{\mathrm{prac}}66.74 to 74.21 tokens/swith the figures measured in vLLM's own steps

Time per token, to scale

Equations

TPS=1Nk(λ+τ)+(Wread+C)/(ηB)\mathrm{TPS} = \frac{1}{N_k\,(\lambda + \tau) + (W_{\mathrm{read}} + C)/(\eta\, B)} TPSmax=BWread+C\mathrm{TPS}_{\mathrm{max}} = \frac{B}{W_{\mathrm{read}} + C}

where TPS\mathrm{TPS} is the estimated number of tokens per second; NkN_k is the number of kernels each token runs; λ\lambda is a kernel's time to start and τ\tau its time until its first read from memory arrives; WreadW_{\mathrm{read}} is the weights read for every token and C=2S+a+TKC = 2S + a + T K the user's bytes: the fixed state read and written, the activations and logits aa, and the cache read, in bytes; η\eta is the share of the bandwidth reached; BB is the card's memory bandwidth, in bytes per second; TPSmax\mathrm{TPS}_{\mathrm{max}} is the bandwidth's limit, reading alone.

Kernels 821 × (0.5532 µs + 0.3350 µs) = 0.7292 ms; weights 17.61 GB ÷ (92% × 1,792 GB/s) = 10.68 ms; the user's bytes 0.4615 GB ÷ (92% × 1,792 GB/s) = 0.2799 ms; estimate 1 ÷ (0.7292 ms + 10.68 ms + 0.2799 ms) = 85.55 tokens/s; bandwidth's limit 1,792 GB/s ÷ 18.07 GB = 99.17 tokens/s.

For each new token the card runs NkN_k kernels, reads WreadW_{\mathrm{read}} of weights and reads and writes the user's CC bytes. Each kernel takes λ\lambda to start and τ\tau until its first read arrives, and the bytes move at the share η\eta of the bandwidth BB; together they give one user's TPS\mathrm{TPS}. Reading everything at the full bandwidth, with no time per kernel, would give TPSmax\mathrm{TPS}_{\mathrm{max}}: no card and no serving software can do better.

The linear-attention layers update each user's fixed state SS in place, so every token reads it and writes it back, 2S2S. Each kernel writes its results for the next one to read, the activations, and the last matrix multiplication writes a score for each of the 248,320 entries of the vocabulary, the logits, from which the token is picked. Together these are aa, counted from the model's configuration: 19.40 MB a token on an RTX 5090 and 20.39 MB on an RTX 3090, whose matrix multiplications take their inputs in 16 bits. The full-attention layers read the cache of every token so far, TKT K.

The kernels are counted from vLLM's code. vLLM compiles the model with torch.compile, which joins the norms, additions and gates between two matrix multiplications into one kernel; the matrix multiplications, attention and the linear-attention updates are kernels of their own. That gives 11 kernels in each full-attention layer, 12 in each linear-attention layer and 4 once a token.

A card that multiplies 4-bit numbers itself, as the RTX 5090 does, runs one more kernel in each layer and one more once, to convert the inputs of the 4-bit matrices: 821 for the 4-bit model on an RTX 5090, 756 on an RTX 3090 and for the 8-bit model. vLLM records a token's kernels once and replays them as one CUDA graph, so the processor does not start each one.

η\eta = 92% is the share of the maker's bandwidth reached in a memory throughput test on a GeForce RTX 4090 (Luo et al., 2024). λ\lambda = 0.553 µs: 1,025 empty kernels in one CUDA graph took 567 µs on a GeForce RTX 3060 (Hoffman and Oh, 2024). τ\tau = 0.335 µs: a read from the card's memory took 876.7 clock cycles on a GeForce RTX 5080 (Jarmusch et al., 2025), at its boost clock of 2,617 MHz. None of the three was measured on the cards here or set from their runs.

TPSprac=1Wread/(eWB)+Nkk+Lfulla0+2S/(eSB)+TK/(eCB)+ti\mathrm{TPS}_{\mathrm{prac}} = \frac{1}{W_{\mathrm{read}}/(e_W\, B) + N_k\, k + L_{\mathrm{full}}\, a_0 + 2S/(e_S\, B) + T K/(e_C\, B) + t_i}

TPSprac\mathrm{TPS}_{\mathrm{prac}} is the practical estimate. It takes its figures from profiles of vLLM's decode steps: Qwen3.8-27B NVFP4, one user, on the RTX 5090 and RTX 3090 measured, with the profiler's own time taken off. The matrix multiplications read the weights at eWe_W = 78.2% and 83.6% of the bandwidth. The small kernels, the gaps between kernels and the choice of the token took kk = 1.98 and 1.38 µs for each kernel the calculator counts, and attention a fixed a0a_0 = 18.2 and 12.9 µs in each full-attention layer.

The fixed state was read and written at eSe_S = 47.9% and 57.9% of the bandwidth and the cache read at eCe_C = 95.7% and 74.8%, and the card stood idle for tit_i = 0.015 and 0.051 ms between two steps. The activations and logits are part of the small kernels' time. The range takes every figure at the slower of its two values, then every one at the faster, for every card and model.

Practical, slow end: weights 17.61 GB ÷ (78.23% × 1,792 GB/s) = 12.56 ms; kernels 821 × 1.976 µs = 1.622 ms; attention's fixed cost 16 × 18.24 µs = 0.2918 ms; fixed state 2 × 153.9 MB ÷ (47.94% × 1,792 GB/s) = 0.3584 ms; cache 4,096 × 32.77 kB ÷ (74.77% × 1,792 GB/s) = 0.1002 ms; idle 0.05116 ms; 1 ÷ 14.98 ms = 66.74 tokens/s. Fast end, every figure at the faster card's value: 74.21 tokens/s.

Assumptions

  • One user at a time, on one card; several users are in Speed for Several Users.
  • η\eta, λ\lambda and τ\tau are the same for every card, from the sources above.
  • The weights are read at η\eta, the share a plain read reaches. vLLM's matrix multiplications also unpack the 4-bit and 8-bit weights as they read them, and no source gives the share they reach at one user's sizes; on the cards measured they reached 78% of the bandwidth on the RTX 5090 and 84% on the RTX 3090, so the estimate is fast.
  • Each kernel is counted once, as vLLM's code gives them, with one read from memory. On the cards measured vLLM ran 1,163 kernels a token on the RTX 5090 and 926 on the RTX 3090, because torch.compile joined fewer operations than the count assumes, and its kernels read memory more than once; both make the estimate fast.
  • Other card is taken as one that multiplies 4-bit numbers itself, as the RTX 5090 does.
  • The practical estimate's figures were measured for one model, Qwen3.8-27B NVFP4, on two cards. Bielik-11B-v3.0, which took no part in them, ran faster than the practical range at 12 of its 18 measured points: by 0.2% to 3.0% with its 8-bit weights and with its 16-bit weights on the RTX 3090, and by 6.3% to 11.0% with its 16-bit weights on the RTX 5090. Bielik was not profiled, so the cause is not established.

Speed for Several Users

This step gives each user's tokens per second when the card writes several answers at once, and all users' tokens per second together.

The card makes one new token for every user in one step: it reads the weights once for all of them, and each user's conversation memory for that user. A further user therefore costs far less than a further card: on an RTX 5090, 16 users each got 58.7 tokens per second where one got 68.9, 940 tokens per second together.

Each further user adds only their own bytes: their fixed state, read and written, their activations and logits, and their cache, as in Speed for One User. The kernels and the reading of the weights are shared by all users in the step. The users UU and the conversation length TT are those of Memory.

Users at once UrU_r1the one asked
Theoretical maximum, each user TPSeach,max\mathrm{TPS}_{\mathrm{each},\mathrm{max}}99.17 tokens/sall users together 99.17 tokens/s
Modeled estimate, each user TPSeach\mathrm{TPS}_{\mathrm{each}}85.55 tokens/sone step 11.69 ms; all users together 85.55 tokens/s
Practical estimate, each user TPSeach,prac\mathrm{TPS}_{\mathrm{each},\mathrm{prac}}66.74 to 74.21 tokens/sall users together 66.74 to 74.21 tokens/s

One step for all users, to scale

Equations

tstep=Nk(λ+τ)+Wread+UrCηBt_{\mathrm{step}} = N_k\,(\lambda + \tau) + \frac{W_{\mathrm{read}} + U_r\, C}{\eta\, B} TPSeach=1tstep\mathrm{TPS}_{\mathrm{each}} = \frac{1}{t_{\mathrm{step}}} TPSall=Urtstep\mathrm{TPS}_{\mathrm{all}} = \frac{U_r}{t_{\mathrm{step}}}

where tstept_{\mathrm{step}} is the time of one step, which makes one token for each user; UrU_r is the users at once; NkN_k, λ\lambda, τ\tau, WreadW_{\mathrm{read}}, η\eta and BB are as in Speed for One User, and CC is one user's bytes at the conversation length TT; TPSeach\mathrm{TPS}_{\mathrm{each}} and TPSall\mathrm{TPS}_{\mathrm{all}} are each user's tokens per second and all users' together.

One user at once: step 0.7292 ms + (17.61 GB + 0.4615 GB) ÷ (92% × 1,792 GB/s) = 11.69 ms; each user 1 ÷ 11.69 ms = 85.55 tokens/s; all users 1 × 85.55 = 85.55 tokens/s.

Ur=max{k≤min(U,256):TvLLM(k)≥T}U_r = \max\{ k \le \min(U, 256) : T_{\mathrm{vLLM}}(k) \ge T \}

The users at once, UrU_r, are the most users, at most UU and vLLM's limit of 256 at once, whose conversations of TT tokens fit for their whole length: TvLLM(k)T_{\mathrm{vLLM}}(k) is Memory's tokens per user for kk users. vLLM itself starts a request when its blocks fit at that moment, and preempts it later if the conversations grow past what fits: on an RTX 3090, three users of 2,304 tokens all started, and two of them were preempted.

Each step runs the kernels and reads the weights once for all users, and each user's bytes CC for that user. A step of UrU_r users therefore takes longer than one user's only by (Ur−1)C/(ηB)(U_r - 1)\, C/(\eta B): on an RTX 5090, with conversations of 4,096 tokens, 0.28 ms for each further user against a step of 11.69 ms.

TPSeach,max=BWread+UrC\mathrm{TPS}_{\mathrm{each},\mathrm{max}} = \frac{B}{W_{\mathrm{read}} + U_r\, C} TPSeach,prac=1Wread/(eWB)+Nkk+Lfulla0+Ur(2S/(eSB)+TK/(eCB))+ti\mathrm{TPS}_{\mathrm{each},\mathrm{prac}} = \frac{1}{W_{\mathrm{read}}/(e_W\, B) + N_k\, k + L_{\mathrm{full}}\, a_0 + U_r\,(2S/(e_S\, B) + T K/(e_C\, B)) + t_i}

TPSeach,max\mathrm{TPS}_{\mathrm{each},\mathrm{max}} reads the weights once and every user's bytes at the full bandwidth, with no time per kernel. TPSeach,prac\mathrm{TPS}_{\mathrm{each},\mathrm{prac}} is Speed for One User's practical step for UrU_r users: the weights, the kernels, attention's fixed cost and the idle time once a step, and each user's fixed state and cache, with the same figures eWe_W, kk, a0a_0, eSe_S, eCe_C and tit_i. They were measured at one user and are kept for several; Checked Against Measurements gives how far that holds.

Theoretical maximum 1,792 GB/s ÷ (17.61 GB + 1 × 0.4615 GB) = 99.17 tokens/s each; practical, slow end 1 ÷ 14.98 ms = 66.74 tokens/s each, fast end 74.21 tokens/s each.

Assumptions

  • Every user's conversation has the same length TT, and every user is writing an answer; the prompts processed at the same time are in From Prompt to Answer.
  • The step is Speed for One User's for UrU_r users, with its causes of error: the estimate is fast, by more at 2 to 4 users on an RTX 5090, where NVIDIA's matrix library, cuBLAS, picks a slower method for one small matrix (Checked Against Measurements).
  • The matrix multiplications' arithmetic grows with the users and is not counted. At the data sheets' rates it would take longer than reading the weights from 29 users on an RTX 3090 and 186 on an RTX 5090, for the 4-bit model; the estimate is then fast.
  • Compared with vLLM up to 16 users at once.
  • The practical estimate keeps the figures measured at one user. Against vLLM at 2 to 16 users at once, vLLM's speed lies inside the range at 17 of 35 steps: on the RTX 3090 at all 11; on the RTX 5090 below it by 0.6% to 6.4% at 2 to 4 users of Qwen3.8, where cuBLAS picks its slower method, and above it by up to 7.5% from 12 users and with Bielik-11B-v3.0.

From Prompt to Answer

This step follows every user's prompt from the moment it is sent to the last token of the answer, as vLLM schedules them, and shows the answers as they appear.

With several users the steps above run together. While one user's prompt is processed, the users already answering wait for it: vLLM makes their next tokens in the same step, and a step with a chunk of a prompt takes as long as the chunk.

vLLM puts at most 2,048 tokens through the card in one step, 8,192 on the H100 NVL and H200 NVL: first one token for each answer being written and the next chunk of each prompt it has started, then the prompts waiting, in the order they came, as long as each chunk fits whole. While users are answering, a step of 2,048 tokens therefore has room for one chunk of 1,568 tokens. The simulation follows these rules, read from vLLM's code, with each step's time from the steps above; all UU users send a prompt of PP tokens at the same moment.

Beside the estimate the step gives two more times for the answers. The theoretical minimum runs the same steps at Prompt Processing's theoretical minimum and Speed for Several Users' theoretical maximum; the practical estimate runs them at their practical steps, as a range.

The tokens of each answer. With the prompt it makes the conversation, P+AP + A tokens, which must fit in memory for each user.

Theoretical minimum, whole answer5.305 sfirst token 0.1550 s; 10.08 ms a token
Modeled estimate, whole answer6.266 sfirst token 0.2954 s; 11.68 ms a token
Practical estimate, whole answer7.331 to 8.105 sfirst token 0.4476 to 0.4519 s; 13.47 to 14.98 ms a token

How the simulation schedules the steps

  1. Sending: all users send their prompts at the same moment; they reach vLLM one after another, so its first step holds only the first.
  2. Each step: the users answering get one token each and each prompt in progress its next chunk, if it fits; then waiting prompts start, in turn, up to the users that fit in memory, until a prompt's chunk does not fit.
  3. A step's time: with prompt chunks, their time from Prompt Processing and the bytes of the users answering in it, as in Speed for Several Users; without, a step from Speed for Several Users.
  4. Tokens: a user's first token comes when the prompt's last chunk is done, and then one token a step until the answer is complete. A user who does not fit in memory waits until an answer ends.

Simulated streamed output

0.0 s

The answers appear in real time as the simulation gives them; the text is a sample answer cut into Qwen3.8-27B's own tokens, each a word or part of one, repeated when the answer is longer.

Each user, from the prompt to the answer, to scale

Equations

One user: the prompt in 3 chunks, first token after 0.2954 s; the other 511 tokens one a step, the answer complete after 6.266 s; mean time per token (6.266 s − 0.2954 s) ÷ 511 = 11.68 ms.

tmixed=max(c,Nptop)+tstep+twt_{\mathrm{mixed}} = \max\big( c,\ N_p\, t_{\mathrm{op}} \big) + t_{\mathrm{step}} + t_w tmean=1U∑ktend,k−tfirst,kA−1t_{\mathrm{mean}} = \frac{1}{U} \sum\nolimits_k \frac{t_{\mathrm{end},k} - t_{\mathrm{first},k}}{A - 1}

A step that processes prompt chunks takes the card's work on them, cc, as in Prompt Processing with the weights read once for all of them, or the processor's time NptopN_p\, t_{\mathrm{op}}, whichever is longer, then the engine's tstept_{\mathrm{step}}, plus twt_w for the users answering in it: their bytes, each user's CC at the share η\eta of the bandwidth, as in Speed for Several Users.

On an RTX 5090, profiled with one user answering, the step's matrix multiplications took 13.9 ms longer than the prompt's alone, because NVIDIA's matrix libraries pick slower methods for 1,025 rows than for 1,024; the calculator does not count it. On an RTX 3090 the step took as long as the prompt's alone, 767 against 771 ms.

The mean time per token, tmeant_{\mathrm{mean}}, is each answer's time from its first token to its last, tend,k−tfirst,kt_{\mathrm{end},k} - t_{\mathrm{first},k}, over its other A−1A - 1 tokens, averaged over the UU users. It counts the steps in which an answer waits for other users' prompts, unlike the time between tokens of Speed for Several Users.

The theoretical minimum and the practical estimate run the same simulation with other steps. A step with prompt chunks takes Prompt Processing's minimum or practical time, the users answering in it adding their bytes at the full bandwidth, or their fixed state at eSe_S and their cache at eCe_C; a step of answers only takes Speed for Several Users' theoretical or practical step.

Theoretical minimum: first token 0.1550 s, whole answer 5.305 s, 10.08 ms a token. Practical: first token 0.4476 to 0.4519 s, whole answer 7.331 to 8.105 s, 13.47 to 14.98 ms a token.

Assumptions

  • All users send their prompts at the same moment, all of PP tokens, and all answers are AA tokens long; nothing is found in vLLM's prefix cache.
  • The users at once are those whose conversations of P+AP + A tokens fit for their whole length. vLLM may start more and preempt them; that is not simulated.
  • The simulation's errors against vLLM, and their causes as far as they are established, are under Checked Against Measurements.
  • The practical range takes one model's figures, measured at one user. Set beside vLLM, the whole answer of one prompt and 256 tokens lies inside it at all 11 of Qwen3.8's prompts and 4 of Bielik-11B-v3.0's 18, the others up to 9.1% shorter; with several users, the last first token at 14 of 35 steps and the mean time per token at 7, mostly shorter (Checked Against Measurements).

Checked Against Measurements

This section sets the calculator's results beside vLLM's own on the three cards measured.

In the tables below, “Modeled vs vLLM” is how far the modeled estimate lies above (+) or below (−) vLLM's figure, as a percentage of vLLM's figure. “Practical vs vLLM” reads “inside” where vLLM's figure lies within the practical range; otherwise it is how far the range's nearer end lies from vLLM's figure, measured the same way.

Memory and speed for one user

The memory left for conversations is 18% above vLLM's on the RTX 5090 and 47% above on the RTX 3090; one user's estimated speed is 23 to 28% above vLLM's on the RTX 5090 and 13 to 14% above on the RTX 3090, and vLLM's lies inside the practical range at 6 of 7 lengths. vLLM's memory for conversations and its tokens lie inside the practical memory range on both cards.

vLLM served Qwen3.8-27B NVFP4 on a GeForce RTX 5090 and a GeForce RTX 3090, each card on its own, at its default settings but a limit of 16 users at once, with an 8-bit cache. The memory figures are vLLM's own, from its log; the speeds come from the median time between two new tokens, one request at a time, three requests of 256 new tokens each per conversation length. The calculator's memory, counted with the same limit, and one user's speed were written down from vLLM's code, the data sheets and the sources before they were set beside these runs; nothing in them was fitted to them.

Memory: the calculator beside vLLM
Card, share uuResultModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 5090, 92%Memory for conversations7.747 GB6.507 to 7.376 GB6.582 GB+17.7%inside
RTX 5090, 92%Tokens for one user224,224186,592 to 213,248189,728+18.2%inside
RTX 3090, 98%Memory for conversations1.389 GB0.1383 to 1.020 GB0.9449 GB+47.0%inside
RTX 3090, 98%Tokens for one user31,3600 to 18,81617,248+81.8%inside
RTX 3090, 92%FitsNoNoDoes not start
Speed for one user, in tokens per second
CardConversationModeledPracticalBandwidth's limitvLLMModeled vs vLLMPractical vs vLLM
RTX 50901,024 tokens85.9867.06 to 74.5299.7167.34+27.7%inside
RTX 509016,384 tokens83.7865.42 to 72.9296.9966.37+26.2%inside
RTX 509065,536 tokens77.4460.65 to 68.2589.2162.12+24.7%inside
RTX 5090131,072 tokens70.3555.27 to 62.8880.6057.36+22.6%inside
RTX 30901,024 tokens46.4237.56 to 41.0452.0841.09+13.0%−0.1%
RTX 30908,192 tokens45.8437.09 to 40.6051.4040.30+13.7%inside
RTX 309016,000 tokens45.2236.60 to 40.1450.6939.83+13.5%inside

The calculator leaves 1.17 GB more for conversations than vLLM on the RTX 5090 and 0.45 GB more on the RTX 3090. On the RTX 3090 little is left after the weights, so a smaller error is 47% of the memory for conversations and 82% of the tokens. vLLM's working memory was the same at 92% and 98% on the RTX 3090, within 0.01 GB, as the calculator's is.

vLLM's own figures split the difference into three causes. The usable memory: the RTX 5090 had 0.17 GB less than its data sheet gives after the driver's reserve; on the RTX 3090 the driver reserved less, which gave vLLM 0.04 GB more. The weights: vLLM loaded 1.0% and 1.4% more than the model's files hold, 0.22 and 0.30 GB. The working memory: vLLM's was 0.79 and 0.19 GB larger than counted, mostly its CUDA graphs, 0.54 and 0.19 GiB by its own estimate against 0.02 GB counted. Together they account for the difference within 0.01 GB; the RTX 5090's other 0.2 GB of working memory has no established cause.

The practical memory range takes its terms from these two runs, so here it is checked in-sample: vLLM's memory for conversations and its tokens lie inside it on both cards. With Bielik-11B-v3.0, which took no part, they lie inside it on the RTX 5090 and above it on the RTX 3090, as Memory's assumptions give.

One user's estimate is fast for reasons found in profiles of vLLM's steps, used here for the causes only. At 1,024 tokens the estimated step is 3.22 ms shorter than vLLM's on the RTX 5090 and 2.80 ms on the RTX 3090. vLLM's matrix multiplications read the weights at 78% and 84% of the bandwidth, not 92%: 1.87 and 2.03 ms. vLLM ran 1,163 and 926 kernels a token, not 821 and 756, and its kernels read memory more than once: 1.38 and 0.55 ms. On the RTX 3090 attention's fixed cost adds about 0.21 ms. Together these account for the difference within 0.03 ms on both cards. The 8-bit model was not measured.

The practical range takes its figures from profiles of this model on these cards, so here it is checked in-sample: vLLM's speed lies inside it at 6 of the 7 lengths, and 0.1% above it at the seventh. Bielik-11B-v3.0, measured separately and not used for it, lies inside it at 6 of 18 points; at the others vLLM was faster, by up to 3.0%, and by 6.3% to 11.0% with its 16-bit weights on the RTX 5090.

Time to first token

On the RTX 3090 the estimate is 1% to 2% slow; on the RTX 5090 it is 5% to 14% fast from 16,384 tokens and up to 46% fast for short prompts. Above vLLM's preemption limit the time is not modelled, and on the RTX 3090 the calculator's limit is about twice vLLM's. vLLM's time lies inside the practical range at 10 of the 12 prompts.

The same cards also processed prompts of 1,024 to 180,000 tokens and served up to 24 users, one card at a time, with an 8-bit cache and vLLM's limit of users at once set to 16. Each prompt was text of its own, so that nothing of it was found in vLLM's prefix cache, as its counters confirmed. Each prompt length was measured with three prompts, one after another; several users with twice as many requests of 256 tokens as users, as many at a time as users, as vllm bench serve sends them.

Time to first token for one prompt, in seconds
CardPrompt, tokensModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 509065,5369.87410.783 to 11.42410.875−9.2%inside
RTX 50901,0240.0820.149 to 0.1510.150−45.4%inside
RTX 50904,0960.3390.448 to 0.4520.433−21.6%+3.3%
RTX 509016,3841.5791.722 to 1.8311.656−4.6%+4.0%
RTX 509032,7683.7513.962 to 4.2523.979−5.7%inside
RTX 5090131,07229.23533.264 to 34.60833.710−13.3%inside
RTX 5090180,00049.87657.728 to 59.59658.128−14.2%inside
RTX 30901,0240.7810.714 to 0.7700.765+2.1%inside
RTX 30904,0963.1632.897 to 3.1223.096+2.2%inside
RTX 30908,1926.4365.929 to 6.3796.342+1.5%inside
RTX 309012,0009.5758.862 to 9.5209.444+1.4%inside
RTX 309012,5009.9949.253 to 9.9389.868+1.3%inside

The estimates were written down from the data sheets and the sources before they were set beside these runs, and nothing was fitted to them. On the RTX 3090, where a chunk's work takes about 0.7 s, they are 1% to 2% slow at every length measured.

Short prompts on the RTX 5090 took up to 1.8 times as long as estimated. vLLM issues a prompt's operations from the processor more slowly than topt_{\mathrm{op}} gives, and when a chunk's work on the card is short the card waits for them: in a profile of one 4,096-token prompt the card was idle for 0.208 s of 0.514 s, in 5,124 gaps, none longer than 1 ms, and vLLM's own timers put the wait inside its prompt processing, not before it. Long prompts on the RTX 5090 are 9% to 14% fast because full attention reached about 68% of the 16-bit rate there, not 85%.

At 16,000 tokens, above vLLM's limit of 12,544, the RTX 3090 took 24.09 s, as vLLM preempted each prompt 5 times; the estimate does not model that. The calculator puts the limit at 26,656 tokens with the runs' settings, as its memory for conversations is larger than vLLM's (Memory and speed for one user), so it shows no preemption there.

The practical range takes its figures from profiles of this model at 1,024 and 65,536 tokens on the RTX 5090 and 1,024 on the RTX 3090, and from vLLM's timers at 1,024 and 4,096 tokens on the RTX 5090. vLLM's time lies inside it at 10 of the 12 prompts, and is 3.2% and 3.9% shorter at 4,096 and 16,384 tokens on the RTX 5090. With Bielik-11B-v3.0, which took no part, it lies inside at 7 of 18 prompts: the long ones up to 6.2% shorter, the short ones up to 8.6% longer.

Several users

Each user's speed is 13 to 38% above vLLM's on the RTX 5090 and 18 to 19% above on the RTX 3090, at 21 measured steps up to 16 users at once; vLLM's lies inside the practical range at all 5 RTX 3090 steps and 4 of the 16 RTX 5090 steps.
Each user's speed with several users, in tokens per second
CardConversation, tokensUsers at onceModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 50901,024284.465.4 to 72.861.2+37.9%+6.9%
RTX 50901,024481.362.2 to 69.661.1+33.2%+1.9%
RTX 50901,024875.956.8 to 63.962.3+21.8%inside
RTX 50901,0241666.948.3 to 54.958.7+13.9%−6.5%
RTX 509016,384280.262.3 to 69.859.3+35.3%+5.1%
RTX 509016,384474.056.9 to 64.356.5+30.9%+0.6%
RTX 50904,096381.662.9 to 70.361.0+33.8%+3.0%
RTX 50904,096676.357.8 to 65.162.7+21.8%inside
RTX 50904,0961267.649.8 to 56.758.3+15.9%−2.7%
RTX 509032,768276.359.3 to 66.957.2+33.3%+3.7%
RTX 509032,768371.655.5 to 63.055.1+30.0%+0.8%
RTX 50901,024284.465.4 to 72.861.2+37.8%+6.8%
RTX 30901,024245.536.5 to 40.038.3+18.7%inside
RTX 30901,024443.834.7 to 38.237.0+18.3%inside
RTX 30901,024443.834.7 to 38.237.2+17.9%inside
RTX 50901,024382.863.8 to 71.161.2+35.3%+4.2%
RTX 50901,024579.960.8 to 68.066.0+21.0%inside
RTX 50901,0241666.948.3 to 54.959.1+13.2%−7.0%
RTX 509016,384668.652.4 to 59.657.7+18.9%inside
RTX 30901,024344.735.6 to 39.137.6+18.7%inside
RTX 30902,048245.436.4 to 39.938.1+19.0%inside

Each user's speed is fast for the causes of one user's, given under Memory and speed for one user: vLLM reads the weights below 92% of the bandwidth and runs more kernels than counted. From 5 users on the RTX 5090 the estimate is 13% to 22% fast, on the RTX 3090 18% to 19%. At 2 to 4 users on the RTX 5090 it is 30% to 38% fast, for a further cause the calculator does not count, a slower method of NVIDIA's matrix library.

The slower method's cause was found in profiles of both cards: one small matrix of the linear-attention layers, multiplied 48 times a step, which NVIDIA's matrix library runs with a quick method for 1 user and from 5 users, and with a slow one at 2 to 4, about 1.2 ms more a step on the RTX 5090 and 1.4 ms on the RTX 3090.

The practical range keeps the figures measured at one user. vLLM's speed lies inside it at all 5 steps on the RTX 3090 and at 4 of the 16 on the RTX 5090. On the RTX 5090 it is below the range at 2 to 4 users, by 0.6% to 6.4%, for the slower method of cuBLAS, which one user's profiles did not contain. From 12 users it is above it, by 2.8% to 7.5%: the range charges each further user's fixed state and cache at one user's measured shares.

With Bielik-11B-v3.0, which took no part in the figures, vLLM's speed lies inside the range at all 5 RTX 3090 steps and 2 of the 8 RTX 5090 steps, and above it at the other 6, by up to 4.3%.

From prompt to answer

The simulated mean time per token is up to 26% short, as the steps are; the last user's first token is within 14% where the card stays busy, up to 56% early with short prompts on the RTX 5090, and up to 65% early where the calculator runs more users at once than vLLM did. The practical range holds the whole answer of one prompt at all 11 of Qwen3.8's prompts.
From prompt to answer: the mean time per token in ms, and the last first token of the users who sent their prompts together, in seconds
CardPrompt, tokensUsers (at once)Time per token, modeledTime per token, vLLMModeled vs vLLMLast first token, modeledLast first token, vLLMModeled vs vLLM
RTX 50901,024212.116.3−26.1%0.160.30−45.3%
RTX 50901,024412.916.9−23.8%0.330.58−43.4%
RTX 50901,024814.617.8−18.0%0.661.14−42.0%
RTX 50901,0241618.022.1−18.5%1.332.54−47.5%
RTX 509016,384216.720.5−18.4%3.163.51−9.8%
RTX 509016,384426.330.8−14.7%6.357.27−12.6%
RTX 50904,096314.017.6−20.1%1.021.32−22.4%
RTX 50904,096617.621.1−16.7%2.052.78−26.4%
RTX 50904,0961224.628.9−15.0%4.135.45−24.3%
RTX 509032,768223.426.2−10.9%7.527.86−4.4%
RTX 509032,768334.536.5−5.7%11.3112.21−7.4%
RTX 50901,024212.116.3−26.0%0.160.33−50.4%
RTX 30901,024224.226.8−9.7%1.561.55+1.0%
RTX 30901,024429.531.0−4.7%3.133.09+1.2%
RTX 30901,0245 (4)32.232.1+0.2%3.9111.15−64.9%
RTX 30902,0483*30.331.2−3.1%4.7111.92−60.5%
RTX 50901,024312.516.7−25.1%0.250.55−55.5%
RTX 50901,024513.316.1−16.9%0.410.69−40.5%
RTX 50901,02424 (16)18.422.5−18.4%5.607.60−26.3%
RTX 509016,384635.836.7−2.7%9.569.86−3.0%
RTX 30901,024326.928.7−6.3%2.342.31+1.3%
RTX 30902,048223.926.6−10.1%3.143.06+2.4%

* vLLM started all 3 users and preempted 2 of them; the calculator, with its larger memory for conversations, runs all 3 without preemption. In brackets, the users vLLM ran at once where it ran fewer than were sent; the calculator runs all 5 on the RTX 3090 and 16 on the RTX 5090.

The mean time per token counts the steps in which an answer waits for other users' prompts; it was up to twice the time between tokens of the same run. The simulation gives it up to 26% short: its steps are Speed for Several Users', fast for the causes given there, and on the RTX 5090 a user answering in a step that processes a prompt made vLLM's matrix multiplications 13.9 ms longer in its profiles, as NVIDIA's matrix libraries pick slower methods for 1,025 rows than for 1,024, where the calculator counts only that user's bytes.

The last user's first token is within 14% wherever the card is busy throughout its chunks and the calculator runs as many users at once as vLLM did: on the RTX 3090, and on the RTX 5090 from 16,384 tokens. With prompts of 1,024 to 4,096 tokens on the RTX 5090 it comes 22% to 56% early, for the cause of the short prompts' error above: a step that processes a prompt while users answer starts 1,974 to 2,118 small programs, against 1,610 for the prompt alone, and starting them sets its time.

On the RTX 3090, with 5 users of 1,024 tokens and 3 of 2,048, it comes 65% and 61% early. vLLM's memory held fewer users than the calculator's: the fifth user waited for an answer to end, and two of three users were preempted and their prompts processed again. The calculator starts them all at once.

From prompt to answer, the practical range: the mean time per token in ms, and the last first token of the users who sent their prompts together, in seconds
CardPrompt, tokensUsers (at once)Time per token, practicalTime per token, vLLMPractical vs vLLMLast first token, practicalLast first token, vLLMPractical vs vLLM
RTX 50901,024214.1 to 15.716.3−3.7%0.30 to 0.300.30+0.001%
RTX 50901,024415.6 to 17.316.9inside0.60 to 0.600.58+3.3%
RTX 50901,024818.4 to 20.417.8+3.5%1.20 to 1.221.14+5.9%
RTX 50901,0241624.1 to 26.622.1+9.3%2.43 to 2.462.54−3.3%
RTX 509016,384218.9 to 20.920.5inside3.45 to 3.673.51inside
RTX 509016,384429.4 to 32.230.8inside6.93 to 7.377.27inside
RTX 50904,096316.6 to 18.317.6inside1.35 to 1.361.32+2.3%
RTX 50904,096621.3 to 23.321.1+1.2%2.70 to 2.732.78−1.9%
RTX 50904,0961230.7 to 33.128.9+6.2%5.45 to 5.515.45inside
RTX 509032,768225.7 to 28.426.2inside7.94 to 8.537.86+1.0%
RTX 509032,768337.4 to 41.036.5+2.3%11.94 to 12.8312.21inside
RTX 50901,024214.1 to 15.716.3−3.7%0.30 to 0.300.33−8.4%
RTX 30901,024227.0 to 29.526.8+0.7%1.43 to 1.541.55−0.3%
RTX 30901,024432.3 to 35.431.0+4.2%2.86 to 3.093.09−0.1%
RTX 30901,0245 (4)34.9 to 38.332.1+8.8%3.57 to 3.8611.15−65.4%
RTX 30902,0483*32.7 to 35.831.2+4.8%4.30 to 4.6411.92−61.1%
RTX 50901,024314.9 to 16.516.7−1.1%0.45 to 0.450.55−17.9%
RTX 50901,024516.3 to 18.016.1+1.4%0.75 to 0.760.69+8.6%
RTX 50901,02424 (16)24.8 to 27.322.5+10.3%8.03 to 8.687.60+5.7%
RTX 509016,384639.8 to 43.436.7+8.2%10.44 to 11.119.86+5.8%
RTX 30901,024329.6 to 32.528.7+3.4%2.14 to 2.312.31−0.02%
RTX 30902,048226.7 to 29.326.6+0.6%2.86 to 3.093.06inside

The practical range runs the same simulation with the practical steps. In the table, vLLM's mean time per token lies inside it at 5 of the 22 steps and its last first token at 5; with Bielik-11B-v3.0, at 2 and 9 of 13. The mean time per token is mostly shorter than the range, by up to 9.3%, as the practical steps of several users are (Several users).

The last first token is within 8% of the range at 30 of the 35 steps. It came later at 2 and 3 users of 1,024 tokens on the RTX 5090 (+9.2% and +21.7%), at 2 users of the 16-bit Bielik on the RTX 3090 (+19.8%), and far later where vLLM ran fewer users at once than the calculator: +157% and +189% on the RTX 3090. The whole answer of one prompt and 256 tokens lies inside the range at all 11 of Qwen3.8's prompts and 4 of Bielik's 18, the others up to 9.1% shorter.

One of the simulation's errors has no established cause. The first user's first token came 0.10 to 0.71 s later than simulated on both cards, also where one prompt alone took about as long as estimated: 0.37 s later at 1,024 tokens on the RTX 3090. The profiles show its prompt processed alone in the first step, as simulated. Finding it needs vLLM's own timing of each request with several users, which was not measured.

Bielik-11B-v3.0

With Bielik-11B-v3.0 the memory for conversations is 2% to 7% above vLLM's, 60% with its 16-bit weights on the RTX 3090; one user's speed is 4% to 20% above vLLM's; the time to first token is within 9%, but 21% short for 1,024 tokens with the 8-bit weights on the RTX 5090; the users at once are right at all 13 steps.

vLLM served Bielik-11B-v3.0-Instruct, with its weights in 16 bits (BF16) and in 8 bits (FP8-Dynamic), on the same two cards, at its default settings, a 16-bit cache, 92% of the memory and a limit of 16 users at once. The calculator's figures for it were written down from the model's files, its configuration and vLLM's code before the runs, and none of its figures, the practical estimates' included, was taken from them. The 16-bit weights fit on the RTX 3090 only with conversations of up to 1,968 tokens, so they were measured at 512 to 1,536 tokens there.

Bielik-11B-v3.0: memory for conversations, and the tokens for one user
CardWeightsModeledPracticalvLLMModeled vs vLLMPractical vs vLLMTokens, modeledTokens, practicalTokens, vLLMModeled vs vLLMPractical vs vLLM
RTX 30908-bit11.554 GB10.449 to 11.283 GB11.307 GB+2.2%−0.2%56,40050,992 to 55,07255,200+2.2%−0.2%
RTX 309016-bit0.6534 GB0.0000 to 0.2685 GB0.4080 GB+60.1%−34.2%3,1680 to 1,2801,983+59.8%−35.5%
RTX 50908-bit19.428 GB18.322 to 19.157 GB18.909 GB+2.7%inside94,83289,440 to 93,52092,288+2.8%inside
RTX 509016-bit8.527 GB7.269 to 8.142 GB7.978 GB+6.9%inside41,61635,472 to 39,72838,960+6.8%inside
Bielik-11B-v3.0: speed for one user, in tokens per second
CardWeightsConversation, tokensModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 30908-bit1,02472.9556.96 to 62.9364.50+13.1%−2.4%
RTX 30908-bit4,09669.2654.19 to 60.2760.12+15.2%inside
RTX 30908-bit8,19264.8850.88 to 57.0556.09+15.7%inside
RTX 30908-bit16,38457.6045.35 to 51.5449.55+16.2%inside
RTX 30908-bit24,00052.1641.19 to 47.3045.37+15.0%inside
RTX 309016-bit51238.1030.97 to 33.6734.49+10.5%−2.4%
RTX 309016-bit1,02437.9330.82 to 33.5434.36+10.4%−2.4%
RTX 309016-bit1,53637.7530.68 to 33.4133.92+11.3%−1.5%
RTX 50908-bit1,024134.8198.00 to 110.92114.11+18.1%−2.8%
RTX 50908-bit4,096128.2193.69 to 106.58109.74+16.8%−2.9%
RTX 50908-bit8,192120.3688.50 to 101.3099.95+20.4%inside
RTX 50908-bit16,384107.2379.67 to 92.1790.18+18.9%inside
RTX 50908-bit24,00097.3572.91 to 85.0485.23+14.2%−0.2%
RTX 509016-bit1,02471.5055.93 to 61.6568.41+4.5%−9.9%
RTX 509016-bit4,09669.6054.50 to 60.2966.63+4.5%−9.5%
RTX 509016-bit8,19267.2252.70 to 58.5663.10+6.5%−7.2%
RTX 509016-bit16,38462.9249.44 to 55.3958.90+6.8%−6.0%
RTX 509016-bit24,00059.3846.75 to 52.7356.86+4.4%−7.3%
Bielik-11B-v3.0: time to first token for one prompt, in seconds
CardWeightsPrompt, tokensModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 30908-bit1,0240.3470.318 to 0.3410.335+3.8%inside
RTX 30908-bit4,0961.4731.360 to 1.4481.403+5.0%inside
RTX 30908-bit8,1923.1733.003 to 3.1783.012+5.3%inside
RTX 30908-bit16,3847.2547.135 to 7.4866.914+4.9%+3.2%
RTX 30908-bit24,00011.86411.991 to 12.50611.358+4.5%+5.6%
RTX 309016-bit5120.1720.167 to 0.1730.188−8.5%−8.0%
RTX 309016-bit1,0240.3470.329 to 0.3410.359−3.1%−5.0%
RTX 309016-bit1,5360.5260.493 to 0.5130.523+0.5%−2.1%
RTX 50908-bit1,0240.0640.076 to 0.0780.081−20.7%−4.6%
RTX 50908-bit4,0960.2850.276 to 0.2970.295−3.4%inside
RTX 50908-bit8,1920.6470.647 to 0.6900.661−2.0%inside
RTX 50908-bit16,3841.6031.678 to 1.7641.606−0.2%+4.5%
RTX 50908-bit24,0002.7692.983 to 3.1092.798−1.0%+6.6%
RTX 509016-bit1,0240.1210.118 to 0.1240.129−6.4%−3.8%
RTX 509016-bit4,0960.5100.481 to 0.5110.492+3.7%inside
RTX 509016-bit8,1921.0961.054 to 1.1191.066+2.8%inside
RTX 509016-bit16,3842.5022.489 to 2.6212.445+2.3%+1.8%
RTX 509016-bit24,0004.0854.169 to 4.3654.038+1.2%+3.2%
Bielik-11B-v3.0: each user's speed with several users, in tokens per second
CardWeightsConversation, tokensUsers at once, vLLM (modeled)ModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 30908-bit1,0244 (4)68.753.9 to 59.959.4+15.7%inside
RTX 30908-bit1,0248 (8)63.750.2 to 56.454.5+17.0%inside
RTX 30908-bit1,02416 (16)55.644.2 to 50.446.9+18.6%inside
RTX 30908-bit4,0968 (8)46.436.9 to 42.840.9+13.5%inside
RTX 309016-bit5122 (2)37.930.8 to 33.533.2+14.0%inside
RTX 50908-bit1,0244 (4)127.293.2 to 106.1109.4+16.3%−3.0%
RTX 50908-bit1,0248 (8)118.387.4 to 100.2102.3+15.6%−2.0%
RTX 50908-bit1,02416 (16)103.777.9 to 90.391.3+13.6%−1.1%
RTX 50908-bit4,0968 (8)86.965.8 to 77.477.8+11.7%−0.4%
RTX 509016-bit1,0244 (4)69.354.3 to 60.162.7+10.6%−4.1%
RTX 509016-bit1,0248 (8)66.652.3 to 58.258.2+14.3%−0.1%
RTX 509016-bit1,02416 (16)61.748.7 to 54.753.9+14.5%inside
RTX 509016-bit4,0968 (8)55.343.7 to 49.749.5+11.9%inside

The memory for conversations is 2% to 7% above vLLM's, and 60% with the 16-bit weights on the RTX 3090, where 0.24 GB is large beside the 0.41 GB left after 22.34 GB of weights: vLLM's working memory was 0.13 GiB above the calculator's, its largest step larger than counted, and the memory it saw about 0.10 GiB below the data sheet's less the reserve. vLLM loaded the weights as the files hold them. Whether a conversation's length fits was right in all four runs.

One user's speed is fast for the causes found with Qwen3.8: the matrix multiplications read the weights below 92% of the bandwidth, and vLLM runs more kernels than counted. The 16-bit weights on the RTX 5090, read by plain 16-bit kernels, are closest, 4% to 7%; the 8-bit ones on the RTX 5090, whose inputs a kernel of their own quantises, furthest, 14% to 20%. Bielik was not profiled, so the split by cause is not measured.

The time to first token is within 9% everywhere but the RTX 5090's 1,024-token prompt with the 8-bit weights, 21% short: as with Qwen3.8, a short chunk on the RTX 5090 waits for the processor, which the practical estimate includes. The calculator's time to the complete answer of 256 tokens is 1% to 16% short, as its speed is fast. With several users the users at once are right at all 13 steps, and each user's speed is 11% to 19% fast, for one user's causes.

vLLM's figures lie inside the practical ranges at 6 of the 18 lengths for one user's speed, 7 of the 18 prompts, 7 of the 13 several-users steps and 2 of the 4 runs' memory. Elsewhere vLLM was mostly faster than the ranges, or left more memory: their figures come from Qwen3.8's profiles and runs.

Arete-27B

On the RTX 3090 one user's estimated speed is 14% to 16% above vLLM's at six lengths, and vLLM's lies inside the practical range at all six. The time to first token is within 6%, and vLLM's is at most 2.2% slower than the practical range. With two users, each user's estimated speed is 21% above vLLM's, inside the practical range.

With images allowed, the memory for conversations is 7% and 22% above vLLM's in two starts, and the practical range holds one and is 2.4% short of the other. With text only it is 55% above, because vLLM's working memory was 2.0 GB larger than the calculator counts, for a reason not yet known. The practical range for text only therefore takes the working memory measured in that start.

Arete-27B, in a 4-bit build (W4A16), ran on a GeForce RTX 3090 with the same vLLM. Its estimates were written down before they were set beside those runs, apart from the practical memory range with text only, which takes one of them.

Arete-27B: memory for conversations, and the tokens for one user
CardSetupModeledPracticalvLLMModeled vs vLLMPractical vs vLLMTokens, modeledTokens, practicalTokens, vLLMModeled vs vLLMPractical vs vLLM
RTX 309095%, 16-bit cache3.525 GB2.314 to 3.181 GB2.899 GB+21.6%inside47,82429,792 to 42,33633,981+40.7%inside
RTX 309098.9%, 8-bit cache, text only6.443 GB3.940 to 4.263 GB4.155 GB+55.0%inside185,024108,192 to 117,600108,669+70.3%inside
RTX 309098.9%, 8-bit cache4.097 GB2.879 to 3.754 GB3.844 GB+6.6%−2.3%112,89676,832 to 103,488100,519+12.3%inside
Arete-27B: speed for one user, in tokens per second
CardSetupConversation, tokensModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 309095%, 16-bit cache1,02453.4242.91 to 47.0546.82+14.1%inside
RTX 309095%, 16-bit cache8,19251.9041.71 to 45.9244.91+15.6%inside
RTX 309098.9%, 8-bit cache, text only1,02453.5443.01 to 47.1446.85+14.3%inside
RTX 309098.9%, 8-bit cache, text only8,19252.7742.40 to 46.5745.90+15.0%inside
RTX 309098.9%, 8-bit cache, text only32,76850.2940.42 to 44.7043.90+14.5%inside
RTX 309098.9%, 8-bit cache, text only65,53647.3238.06 to 42.4241.52+14.0%inside
Arete-27B: time to first token for one prompt, in seconds
CardSetupPrompt, tokensModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 309098.9%, 8-bit cache, text only1,0240.7810.714 to 0.7700.787−0.8%−2.2%
RTX 309098.9%, 8-bit cache, text only8,1926.4365.929 to 6.3796.400+0.6%−0.3%
RTX 309098.9%, 8-bit cache, text only32,76828.35826.939 to 28.73528.954−2.1%−0.8%
RTX 309098.9%, 8-bit cache, text only65,53663.69562.547 to 66.13967.188−5.2%−1.6%
Arete-27B: each user's speed with several users, in tokens per second
CardSetupConversation, tokensUsers at once, vLLM (modeled)ModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
RTX 309098.9%, 8-bit cache, text only16,3842 (2)49.339.3 to 43.640.8+20.8%inside

Qwen3.8-27B on the H200 NVL

On an H200 NVL the estimated time to first token is 44% to 52% short of vLLM's, as vLLM runs attention there in 8 bits, slower than the 16 bits the estimate takes. vLLM's time lies inside the practical range or 1.2% above it for prompts of about 4,000 tokens, and 9% to 12% above it from 129,189 tokens, mostly for the layer that drafts tokens ahead, which the calculator does not model.

Qwen3.8-27B's 8-bit checkpoint ran on an H200 NVL with the same vLLM, with 75% of the card, an 8-bit cache, text only, and its extra prediction layer drafting three tokens ahead (MTP). Each time is the first question on a new document, with no other request on the card.

With an 8-bit cache on the H100 NVL and H200 NVL, vLLM runs attention in 8 bits. Timed alone at these prompts' sizes, that kernel reached 389 TFLOPS, 23% of the card's 8-bit peak and less than the same kernel in 16 bits, 650 to 683 TFLOPS; the 8-bit matrix multiplications reached 65% to 72% of the 8-bit peak, the lower figure when the card held them at its power limit. The practical range takes these figures. The estimate keeps its sources' 16-bit attention: no source gives a figure for this kernel in 8 bits.

The layer that drafts tokens ahead processes every prompt too, one more full-attention layer with its weights: about 2.2 s at 254,332 tokens. With it, and the kernels that convert the inputs of the 8-bit matrix multiplications, which run on their own on this card, about 1.4 s more, the practical range comes within 2% to 9% of vLLM's times.

Qwen3.8-27B 8-bit on the H200 NVL: time to first token for one prompt, in seconds
CardSetupPrompt, tokensModeledPracticalvLLMModeled vs vLLMPractical vs vLLM
H200 NVL75%, 8-bit cache, text only, MTP3,8450.1500.298 to 0.3200.303−50.3%inside
H200 NVL75%, 8-bit cache, text only, MTP129,1899.49414.497 to 15.32016.965−44.0%−9.7%
H200 NVL75%, 8-bit cache, text only, MTP253,76327.40344.406 to 46.02850.804−46.1%−9.4%
H200 NVL75%, 8-bit cache, text only, MTP3,9540.1550.298 to 0.3200.324−52.2%−1.2%
H200 NVL75%, 8-bit cache, text only, MTP129,2319.49914.497 to 15.32017.302−45.1%−11.5%
H200 NVL75%, 8-bit cache, text only, MTP254,33227.50544.615 to 46.23950.866−45.9%−9.1%

Cite as: Jacek Garbuliński, LLM GPU Token Speed Calculator, Blixild, 2026, https://blixild.com/llm-gpu-token-speed-calculator.html

Questions and Answers

When and why should I use this calculator?

Before buying or renting a card, or choosing a model's precision, to check that the model fits with the users and conversation lengths needed, how long each user waits for the first token and what speed each user then gets. It shows how memory and speed depend on each other: a longer conversation or more users take memory and slow each token, and several users' prompts delay each other's answers. The results rest on the model's files, the cards' data sheets, vLLM's code and published measurements; a test on the card decides.

Why does the speed depend on memory bandwidth rather than on the card's computing power?

For one user, each new token needs every weight it uses once, and each weight takes only a couple of arithmetic operations. A modern card does those operations faster than it can read the weights from its memory, so the reading sets the pace. That is why the bandwidth's limit, B/(Wread+C)B/(W_{\mathrm{read}} + C), bounds the speed, and why fewer bits per weight make the model faster as well as smaller.

What is the serving software's working memory, and why does it differ between cards?

It is the memory vLLM uses besides the weights and the conversations: the memory its largest step needs, the recorded CUDA graphs and its libraries' workspaces. vLLM finds it at start-up and keeps the rest for conversations. The calculator counts it from vLLM's code: for the 4-bit model at vLLM's default settings, 2.547 GB on an RTX 5090 and 2.518 GB on an RTX 3090, most of it the image encoder's step. In the runs vLLM took 0.79 and 0.19 GB more than counted, mostly for its CUDA graphs, so it left less for conversations than the calculator.

Why does a longer conversation slow down each token?

Because every new token reads the whole cache of the full-attention layers, which grows by KK with every token. On the RTX 5090 one user got 67.34 tokens per second at 1,024 tokens and 57.36 at 131,072; the calculator's estimate falls from 85.98 to 70.35, fast at both for the reasons under Checked Against Measurements. The linear-attention layers keep a fixed state instead, which is why the slowdown is moderate: only 16 of the 64 layers keep a cache.

Why does the 4-bit model run on an RTX 3090, which has no 4-bit arithmetic?

vLLM keeps the weights in 4 bits in memory and widens them to 16 bits as it uses them. The card still reads 4 bits per weight, so the reading, which sets the speed, is as for a native 4-bit card. It measured 41.09 tokens per second, 79% of the bandwidth's limit. That run used a memory share of 98%: at vLLM's default, 92%, the weights and vLLM's working memory leave no room for a conversation on this card, and vLLM does not start.

Does the 8-bit model fit on these cards?

By the calculator, not at vLLM’s default settings: its weights take 30.39 GB, more than the 23.24 GB vLLM may use on an RTX 3090, and on an RTX 5090 the weights and the working memory leave 1.796 GB too little; the RTX 3080, 4080, 5070 and 5080 hold 10 to 16 GB, less than the weights. On the A100 80GB PCIe, H100 NVL and H200 NVL it fits, with room for one user’s conversation of the model’s full 262,144 tokens. It was not measured.

How many users can share one card?

As many as the memory holds for their conversations, which Memory shows, up to vLLM's limit of 256 at once. Each further user costs far less than the first, because the card reads the weights once for all users in each step: on an RTX 5090 one user got 68.9 tokens per second, and 16 users 58.7 each, 940 together. Speed for Several Users estimates them, 13% to 38% fast at the steps measured, and From Prompt to Answer shows how long each user waits while the other users' prompts are processed.

Why does the first token take longer when several users send prompts at the same moment?

Because vLLM processes the prompts one after another, about one chunk of 1,568 tokens per step while answers are being written, so the last user's first token comes after roughly all the prompts' time. On an RTX 5090, 16 users with prompts of 1,024 tokens got their first tokens between 0.36 and 2.54 s. The answers already started slow down meanwhile, because each step with a prompt chunk takes as long as the chunk; From Prompt to Answer shows both.

Limitations

The calculator estimates memory and speed from data sheets, the model's files and the serving software's code, and a real setup can differ from it in the ways below.
  • Bandwidth reached. Every byte is taken to move at 92% of the maker's bandwidth, the share a memory test reached. Matrix multiplications that unpack 4-bit or 8-bit weights as they read them usually reach less, by an amount that depends on the kernel, the number format and the card, and the speed is then lower than estimated.
  • Kernels. The kernels a token runs are counted from the serving software's code, with the operations a compiler is expected to join counted as one. If it joins fewer, or a kernel reads memory more than once, each token takes longer, by about a microsecond for each further kernel.
  • Serving software. The figures follow vLLM 0.31.1rc1 at its default settings. Another version or serving program, running without CUDA graphs, speculative decoding, or other limits on users and tokens per step change the kernels, the working memory and the order in which requests are processed.
  • Processor. The processor starts a prompt's kernels one by one while the card works. A slower or busy processor makes the card wait, and the first token comes later than estimated.
  • Clocks, power and other programs. The data sheet's bandwidth and arithmetic rates are peak figures at the boost clock. A card held to a lower power limit, running hot, or shared with other programs or a display reaches less, and has less memory free.
  • Memory. The memory programs can use depends on the driver and the operating system, and the serving software's working memory on its version and settings; both change the tokens that fit.
  • Arithmetic. A new token's time is taken from the bytes it reads and writes. With many users at once or very long conversations, the arithmetic grows with them and can take longer than the reading, and the speed is then lower than estimated.
  • Requests. The time a request spends outside the card, in the network, the tokeniser and the server's queue, is not counted, nor the work of sampling settings, structured output or returned probabilities; each adds to the time the user waits.
  • Models. Only the models listed are covered, from their files and configuration. A model with another layout, such as one with full attention only or with experts, or in another number format, has other bytes per token, another cache and other kernels.
  • One card. A model split over several cards adds the time of their communication, which is not modelled.
  • Checks. The estimates were set beside measurements on two consumer cards with GDDR memory, the RTX 3090 and RTX 5090, for Qwen3.8-27B at 4 bits and Bielik-11B-v3.0 at 8 and 16 bits, and on the RTX 3090 for Arete-27B, also with text only; and on an H200 NVL for the time to first token of Qwen3.8-27B's 8-bit checkpoint. The RTX 3080, 4080, 5070 and 5080, the A100 80GB PCIe and H100 NVL, and the H200 NVL's other results are estimated without a check.

Sources

Ghosh, A., Nayak, A., Panwar, A. and Basu, A. (2025). PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch. arXiv:2503.19779v3, §2. https://arxiv.org/abs/2503.19779v3

Hoffman, H. and Oh, F. (2024). Constant Time Launch for Straight-Line CUDA Graphs and Other Performance Enhancements. NVIDIA Technical Blog, 11 September 2024. Table 1. https://developer.nvidia.com/blog/constant-time-launch-for-straight-line-cuda-graphs-and-other-performance-enhancements/

Jarmusch, A., Graddon, N. and Chandrasekaran, S. (2025). Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks. arXiv:2507.10789v2, §IV.D. https://arxiv.org/abs/2507.10789v2

Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J. and Stoica, I. (2025). Continnum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live. arXiv:2511.02230v7, Table 4. https://arxiv.org/abs/2511.02230v7

Luo, W., Fan, R., Li, Z., Du, D., Wang, Q. and Chu, X. (2024). Benchmarking and Dissecting the Nvidia Hopper GPU Architecture. IPDPS 2024. arXiv:2402.13499v1, §IV.B and §IV.C. https://arxiv.org/abs/2402.13499v1

NVIDIA (2025). NVIDIA RTX Blackwell GPU Architecture, V1.1. Table 1, GeForce RTX 5090 vs GeForce RTX 4090 vs GeForce RTX 3090; Table 4, GeForce RTX 5080 vs GeForce RTX 4080 vs GeForce RTX 3080; Table 6, GeForce RTX 5070 vs GeForce RTX 4070 vs GeForce RTX 3070; GDDR7 Memory Subsystem; Appendix A. https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf

NVIDIA (2026). NVIDIA A100, H100 GPU and H200 GPU, product pages, Specifications: A100 80GB PCIe, H100 NVL, H200 NVL. https://www.nvidia.com/en-us/data-center/a100/, https://www.nvidia.com/en-us/data-center/h100/, https://www.nvidia.com/en-us/data-center/h200/

NVIDIA (2026). Qwen3.8-27B-NVFP4, config.json, revision 482ca0f. Hugging Face. https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4/blob/482ca0f3832238542f8f5295dde86b5f22711d80/config.json

Qwen (2026). Qwen3.8-27B-FP8, config.json, revision 017b9c7. Hugging Face. https://huggingface.co/Qwen/Qwen3.8-27B-FP8/blob/017b9c7af6b5689d5dd426a76e0bc077eb5ca20a/config.json

NVIDIA (2026). cuBLAS Library, v13.4, §2.4.8. https://docs.nvidia.com/cuda/cublas/index.html

Zhang, J., Wei, J., Wang, H., Zhang, P., Xu, X., Huang, H., Jiang, K., Chen, J. and Zhu, J. (2025). SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training. arXiv:2505.11594v3, A.4 and Figure 4. https://arxiv.org/abs/2505.11594v3

vLLM (2026). vLLM 0.31.1rc1.dev273, commit 2eb491e18: vllm/config/cache.py, vllm/v1/worker/gpu_worker.py and gpu_model_runner.py, vllm/v1/core/kv_cache_utils.py, vllm/v1/kv_cache_interface.py, vllm/v1/core/block_pool.py, vllm/platforms/interface.py, vllm/model_executor/models/qwen3_5.py, vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py, vllm/third_party/flash_linear_attention/ops/fused_recurrent.py and fused_sigmoid_gating.py, vllm/v1/core/sched/scheduler.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/core/single_type_kv_cache_manager.py, vllm/config/vllm.py, vllm/envs.py, vllm/model_executor/models/qwen3_next.py, vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py, vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py, vllm/third_party/flash_linear_attention/ops/utils.py and chunk.py, vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py, vllm/model_executor/kernels/linear/__init__.py and nvfp4/cutlass.py, flashinfer.py and marlin.py, csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_entry.cu and marlin/marlin_mma.h. https://github.com/vllm-project/vllm/tree/2eb491e18

vgerber (2025). IsaacSim 5.x crashes when extension with ogn is disabled while action graph with extension nodes is present. GitHub, isaac-sim/IsaacSim, issue #370, 19 December 2025: nvidia-smi -q of a GeForce RTX 5090, driver 570.195.03. https://github.com/isaac-sim/IsaacSim/issues/370

Send an enquiry

To run a language model on your own card, or to rent compute for one, send an enquiry.

To measure the speed of the models we already serve rather than estimate it, try them through our API at ai.blixild.com, paid per token.

Describe your requirements briefly. We respond within one business day.

contact@blixild.com

By submitting this form, you agree to be contacted regarding your enquiry.