One person in front of the screen
—
questions and answers per minute
Here the human is the limit, not the card: somebody has to type the question and read the answer.
Conversation
What does it feel like to talk to this card?
A document gets processed and that is that. A chat is lived: the person types, waits and reads while the model answers. Here you will see the conversation at the real speed of the card you pick, how many questions and answers it gets through in a minute, and how many people it can serve at once.
Loading the data…
Live
The conversation, at its real pace
Nothing here is a recorded video: the person types at human speed, the model reads the whole conversation so far (that is the time to the first word) and then writes at the real tokens per second of the card you picked.
Scenario
Thinking mode
Speed
«Thinking mode». When the model reasons before answering it writes five to ten times more tokens than you see: a hidden draft. It does not change how fast it writes, but it delays the first word, and that is exactly what people experience as slowness. Turn it on above and watch the clock.
The measure
The chat minute
The honest question is not «how many tokens per second», it is how many questions and answers fit in a minute. And there are two very different answers: one person on their own (who also has to read and type) and the card serving without stopping.
The card serving without stopping
—
answers a minute it can generate
Requests in parallel
Why parallel adds up. While the model writes one answer, the card is not at its limit: it has compute to spare between one token and the next. Serving several conversations at once makes each one a little slower but raises the total quite a lot. The limit is the card's memory, which has to hold the context of every open conversation.
Capacity
How many people does it serve at once?
A person in a conversation does not type non-stop: they think, they read, they get distracted. On average they send a message every three quarters of a minute. That is why a single card serves many more people than you would think.
—
—
people talking at once without waiting
—
—
That is the gap covered by a customer support team of that size during the busiest hour, with this card alone answering the first line.
—
An office of that size using the assistant every day (around 30 messages per person per day), leaving plenty of headroom for the busier hours.
Where the number comes from. Answers per minute from the card × (seconds between messages ÷ 60), then 30 % taken off as headroom so nobody sits watching «typing…». It is a capacity estimate, not a promise: a spike of simultaneous messages always makes a queue.
Head to head
The same conversation on other cards
Same model and same quantisation, same conversation, different card.
How it feels
With the model and quantisation picked above, this is how each card feels while writing. The line that matters is 5 tokens per second: below it, you wait for the computer; above 30, the computer waits for you.
| Speed | Feels like | What you notice | Cards here |
|---|
Where these speeds come from
The conversation above is simulated: the person types at human speed and the model reads and writes at the speed of the card you picked. That speed is measured on the RTX 3090 Ti and estimated for the others from memory bandwidth and compute.
How every figure is worked out: Method.