How fast can a 27B model run at home?

Running language models on your own hardware has quietly become normal. A few years ago it meant renting cloud GPUs, or settling for a model small enough to disappoint you. Today a 27-billion-parameter model compresses to about 17 GB and fits on a single consumer graphics card.

Qwen released a new model recently, and someone ported it to Apple’s MLX within days. I have four fairly different machines at home, so I spent two nights running Qwen3.8-27B on all of them. The numbers ended up less interesting than the reason behind them.

TL; DR

Qwen3.8-27B at 4-bit runs between 12 and 75 tokens per second depending on the machine, all predicted almost exactly by memory bandwidth. That lets you sanity-check any benchmark on a napkin, which is how I caught an impossible measurement before posting it. The model itself is surprisingly good at reasoning, useful for translation, and hopeless at counting letters.

The four machines

• hope: RTX 5090, NVIDIA’s current top consumer card.
• white: RTX 3090, the 2020 flagship, still great thanks to its 24 GB.
• macmini: M4 Pro, 24 GB unified memory, CPU and GPU sharing one pool.
• spark: NVIDIA DGX Spark, GB10 chip, 128 GB unified memory.

All four ran at 4-bit quantization: each weight stored with 4 bits instead of the original 16, four times smaller, slightly less accurate. Everyone running locally makes this trade; full precision wouldn’t fit on any of these machines. The NVIDIA side used ollama with a GGUF file, the Mac used Apple’s MLX.

The setup: three runs per machine, 300 tokens each, averaged, temperature=0, numbers from the servers’ own counters so network latency isn’t mixed in. One thing cost me an hour: models are slow when they first wake up. white showed 23.8 tok/s cold and 42 warm; everything below is from a warmed-up model.

Two numbers matter here. Generation is how fast the model writes its answer, what you feel while watching text appear. Prompt processing is how fast it reads your input, the pause before the first word.

The results

File sizes differ by runtime: the ollama GGUF is 17 GB (17.7 on hope), the MLX build 15 GB.

Why memory bandwidth explains everything

To produce a single token, the model has to read every one of its weights out of memory, then do it all again for the next token, until the answer ends. Picture a chef who has to walk the whole pantry and touch every shelf before cooking any dish: chopping speed doesn’t matter, only walking speed.

Memory bandwidth is how fast a machine moves data out of memory. Since each token means reading the whole model, the ceiling fits on a napkin:

ceiling (tokens/sec) = memory bandwidth ÷ model size

• M4 Pro: 273 GB/s ÷ 15 GB ≈ 18 tok/s ceiling, measured 15.1 (84% of ceiling)
• RTX 3090: 936 GB/s ÷ 17 GB ≈ 55 tok/s ceiling, measured 42.6 (78%)
• RTX 5090: 1792 GB/s ÷ 17.7 GB ≈ 101 tok/s ceiling, measured 75.5 (75%)

Every machine lands between 75 and 84% of its own ceiling. These are different manufacturers, architectures, memory technologies and frameworks; if any of that mattered much, the percentages would be scattered. All four run about as fast as physics allows. The missing 15 to 25% is mostly dequantization: the math units can’t multiply 4-bit numbers directly, so every weight gets unpacked into a larger format on its way in.

Two jobs, two bottlenecks

There’s something odd in the bottom two rows: the Spark generates tokens more slowly than a Mac mini (12.6 against 15.1) but processes prompts five times faster (259 against 50). The two jobs lean on different parts of the machine. Generation is limited by memory bandwidth, and the GB10’s memory is roughly as fast as the M4 Pro’s, so they land in the same place (the Spark trails because its GGUF file is simply bigger). Prompt processing is limited by compute, because the whole prompt goes through at once, and the Spark’s Blackwell tensor cores are built exactly for that.

So the Spark isn’t a bad machine, single-user speed is just the wrong axis to judge it on: its real advantages are 128 GB of memory and throughput when many requests arrive at once.

The measurement that couldn’t be real

While setting up the 5090 I ran qwen3.6:35b as a warm-up and got 169 tok/s. I was genuinely excited until I did the napkin math: that model is 23 GB, its ceiling should be around 75 tok/s, and I had measured more than double the physical maximum. Instead of posting it, I looked at the metadata:

expert_count: 256
expert_used_count: 8

That explains it. qwen3.6:35b is a mixture-of-experts (MoE) model: instead of one big network, it has 256 smaller sub-networks called experts, and a router picks 8 of them per token while the rest sit idle. Like a hospital where reception sends you to the 3 specialists you need instead of making all 256 doctors examine you.

So it has 35 billion parameters on disk but reads only a fraction per token. The formula still holds, I was plugging in the wrong model size. Qwen3.8-27B is dense, every weight read every time, so comparing the two would be meaningless. With the right model, the 5090 gave 75.5 tok/s, at 75% of its ceiling like everything else.

Is the model any good?

Published benchmarks don’t help much here: if a problem is on the internet, the model has probably memorized the answer. So I wrote 24 questions with a single rule: take a famous problem and change it so the correct answer changes. A model that memorized the original gives the old answer; one that reasons has to notice the change.

It got 23 out of 24. My favourite was the “boy born on Tuesday” probability puzzle, famous for its counterintuitive 13/27 answer. I replaced “born on a Tuesday” with “born on a weekend”, which changes the answer to 6/13, and the model built the 196-outcome sample space from scratch and got it right. I also gave it a binary search with a subtle bug and asked for an input that would hang it; its suggestion, find([1], 2), was more minimal than mine, and it really does hang.

To be fair, 24 questions I wrote myself is not an evaluation: I chose the problems, so I chose the difficulty. The one failure is more interesting.

I asked for a sentence whose words are 1, 2, 3, 4, 5 and 6 letters long, in that order. It thought for 479 seconds and returned nothing. It can count fine, the problem is structural: language models don’t see letters, they see tokens, chunks that are usually a whole word. “person” arrives as one indivisible symbol whose six letters are invisible from the inside. Don’t put any model on anagrams and acrostics.

One question I initially scored as wrong turned out to be my bug. The model answered {"city": "İstanbul"} and my check did "istanbul" in c.lower(), which fails: in Python, "İstanbul".lower() yields an i plus a separate combining dot, identical on screen, different as bytes. Years of Turkish software and this still caught me. Normalize your strings before evaluating on non-English text.

What I’d actually use it for

Translating datasets. It held up on the three hardest things to automate: idioms came through as idioms, terminology stayed consistent across a long document, and formatting survived without a “Here is the translation:” preamble sneaking into the output. For long free text I’d still add a back-translation check.

Long context on a single card. Of the 64 layers, 48 use linear attention and only 16 use classic full attention. Standard attention lets every token look at every other, but its memory cost grows with the square of the context length; linear attention replaces it with a fixed-size running summary. The hybrid brings the KV cache, the model’s working memory, down to about 64 KB per token, making 128k context practical on one card. There’s presumably a quality cost somewhere; I didn’t test long context, so I won’t pretend to know where.

A few people asked about NVFP4, a 4-bit float format that Blackwell tensor cores process directly, skipping dequantization. I couldn’t test it (it needs vLLM, and vLLM on ARM64 Blackwell is its own project), but the 5090 was already at 75% of its ceiling, so the best case is roughly 90 tok/s. The impressive numbers going around probably come from batch serving.

Summary

Generation speed at home comes down to memory bandwidth divided by model size: all four machines landed at 75 to 84% of that ceiling regardless of vendor or software. The same formula doubles as a sanity check, which is how it caught the mixture-of-experts number. On the quality side, 23 of 24 altered problems, the one failure coming from tokenization rather than reasoning.

If you have a 24 GB card at home, this model fits on it and runs at conversational speed. I’m pretty happy with where things have landed.

Sources

• Qwen models on Hugging Face
• ollama
• MLX, Apple’s array framework
• llama.cpp, where GGUF comes from
• Boy or girl paradox, the original “Tuesday boy” problem
• Twitter: @cobanov