Running Qwen3.8-27B locally with llama.cpp

Qwen released Qwen3.8-27B on August 14. It is a dense 27B model, with no mixture-of-experts routing: every parameter works on every token. It has thinking mode on by default, a 262,144 token context window, and an Apache 2.0 license.

Release-week posts on X called these the most important 20 gigabytes in human history. The same was said about DeepSeek R1. With less than 18 GB of weights now it performs frontier-tier coding benchmarks, on a license that lets you run them anywhere, forever.

On Arena’s WebDev leaderboard it ranks 18th of 128 models, a 27B open-weights model scoring within a point of GLM-5.2-max and ahead of Gemini 3.7 Flash, DeepSeek V4 Pro, and Claude Opus 4.8.

At 4-bit that is a home-server model. It serves me around 13 tokens per second on my AMD Ryzen 9 8945HS with the Radeon 780M iGPU with the tuned flags below (8 at defaults) (I paid 829,99 eur for this new 64 GB RAM mini PC in 2024) — on coding workloads; chat-style prompts run at around 6, the reason is the acceptance rate of speculative decoding, explained in the last section.

This is my setup: the official llama.cpp Docker image with Vulkan, serving an OpenAI-compatible API.

GGUF

Unsloth’s Dynamic quants are the usual reference. The file that balances quality and size is for me Qwen3.8-27B-UD-Q4_K_XL.gguf at 17.6 GB. Download it to the models/ folder:

wget https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -P ./models

The model is a vision-language model, but the GGUF splits the vision encoder into a separate projector file. If you want image input, download mmproj-F16.gguf too and pass it to the server with --mmproj. For text, the main file is enough.

compose.yml

services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-vulkan
    restart: unless-stopped
    command: >-
      -m /root/models/Qwen3.8-27B-UD-Q4_K_XL.gguf
      --alias local
      --host 0.0.0.0 --port 11434
      --ctx-size 131072
      --flash-attn on
      --cache-type-k q8_0
      --cache-type-v q8_0
      --jinja
      --spec-type draft-mtp
      --spec-draft-n-max 4
      --parallel 1
      --ubatch-size 2048
      --cache-ram 4096
    devices:
      - /dev/dri
    ports:
      - "11434:11434"
    volumes:
      - ./models:/root/models

What each choice does:

• server-vulkan + /dev/dri: the Vulkan build runs on AMD and Intel GPUs, including iGPUs, with just the DRI device nodes passed through: no CUDA toolkit, no ROCm, no nvidia-container-toolkit. llama.cpp offloads layers to the GPU by default (--n-gpu-layers auto), so no offload flag is needed.
• --ctx-size 131072: half of the native 262,144. The context window costs RAM, and with 17.6 GB of weights the KV cache decides how much is left.
• --cache-type-k q8_0 / --cache-type-v q8_0: the KV cache at 8-bit instead of 16-bit, roughly halving its size with negligible quality loss. This is what keeps a large window affordable next to the weights.
• --jinja: serve with the model’s own chat template instead of a built-in fallback. For Qwen3.8 that is what makes thinking mode and tool calling behave as designed.
• --alias local + port 11434: the model answers to the name local, and the port is Ollama’s default, so any client already pointed at http://localhost:11434/v1 keeps working.
• --spec-type draft-mtp: speculative decoding with the model’s built-in multi-token prediction head, the free speed-up explained below. --spec-draft-n-max 4 drafts up to four tokens per pass, and --parallel 1 keeps a single stream, which is the regime the gain assumes: with several concurrent streams the advantage disappears.
• --ubatch-size 2048: the physical prompt-processing batch. The default 512 leaves the GPU starved during prefill; 2048 processes 2k tokens per pass. Long prompts still land at 78–83 tokens per second, GPU-compute bound.
• --cache-ram 4096: llama.cpp keeps recent prompts in RAM so returning conversations skip reprefill. An uncapped cache evicts constantly when one slot’s KV is bigger than the cap’s share — at 131072 context with q8_0, one slot is ~4.5 GB, so the default 8192 MiB cap thrashed my logs with “making room, removing oldest” every minute. 4096 MiB keeps exactly the most recent prompt cached.

Bring it up:

docker compose up -d

And test it at http://localhost:11434/.

The speed-up: MTP

Qwen trained multi-token prediction (MTP) heads into Qwen3.8, and unsloth kept them inside the GGUFs. llama.cpp can draft tokens with that built-in head and verify them with the main model: speculative decoding with no extra model file and no custom build. The --spec- lines in the compose turn it on; without them the MTP head loads and sits unused.

The gains hold across vendors, and they are hardware-dependent: on my Ryzen 9 8945HS the flag doubles the decode speed, from 4 to 8 tokens per second.

I also tried Ternary Bonsai 2 27B, PrismML’s ternary rebuild of the same model in 7.2 GB. It ran much slower, about 2 tokens per second against the 8 of the Q4 quant with MTP, so I stayed with the unsloth GGUF.

More Tuning

Three weeks after the original settings, I measured and rebuilt the command line:

• Draft depth 2 → 4: 8.2 to 12.8 tokens per second (+55%), still accepting ~93% of drafted tokens — measured on agent-style prompts; on chat-style prose the order flips and draft 2 is faster (see the table below).
• --ubatch-size 2048 (default 512): prefill only moved 77.6 to 82.7 tokens per second on a 15k-token prompt — prefill on this iGPU is compute-bound and the 780M itself is the limit, not the batch size. It belongs in the setup anyway as the saner default.
• --cache-ram 4096: not a speed change but an operation fix. One 131072 slot with q8_0 KV is ~4.5 GB, above the default 8192 MiB cache cap’s per-slot share, so the server was evicting 1–1.8 GB prompt-cache entries every minute (“making room for prompt cache entry” in the logs). With the cap at 4096 the evictions are gone and a re-sent prompt prefills from cache: 15k tokens in 0.5 s instead of ~3 minutes of full re-prefill.

While an agent-style long prompt is being prefilled, decode drops to ~5 tokens per second; once the prompt is cached, a re-run of the same conversation starts answering instantly with no prefill at all. The cache cap matters more than any other flag for that pattern.

Measuring the parameters

After the tuning session I measured the whole flag set as a fixed test: five 256-token generations of a story prompt at temperature 0, plus one cold 14k-token prompt processing pass, restarting the container for each configuration. Generation is the median of the five runs. The MTP accepted column is how many of the drafted tokens the main model confirmed in those runs. It is not a setting — it is a property of the text being generated, and it is the number that decides whether speculative decoding speeds anything up at all.

Configuration	Generation tok/s	Prompt processing tok/s	MTP accepted
Bare defaults	4.15	69.5 ¹	—
Bare defaults + MTP n=4	6.01	68.1 ¹	34.6%
My current config	5.66	63.1	33.8%
+ threads 8 (physical cores)	5.85	69.6	33.8%
+ threads 16 (SMT)	6.14	72.1	33.8%
MTP off	4.40	70.1	—
Flash-attn off + MTP	6.4 ²	—	—
Flash-attn off + MTP off	4.45	72.0	—
+ threads-batch 16	6.07	71.5	33.8%
Old params (draft 2, ubatch 512)	6.66	72.4	50.2%

• The acceptance rate of speculative decoding explains the speeds: the MTP head accepted 34% of drafts on the story prompt, 72–87% on a code continuation prompt, ~93% in my real coding sessions. Same flags: 13 tok/s for code, around 6 for chat. MTP never adds tokens itself — it only multiplies whatever the text lets survive verification.
• Draft depth follows the text: draft 2 is faster on prose (6.66 vs 5.66), draft 4 on code (9.9 vs 8.9 on my code prompt). Keep 4 for coding.
• Flash-attn stays on: with FA off (f16 KV) decode is identical (4.45 vs 4.40) and the long prefill crashes at 131k context, so the quantized KV cache needs it.
• Prompt processing sits at 63–72 tok/s in every configuration — the 780M is compute-bound there. --threads-batch 16 is the one cheap measured tweak: prefill 63.1 → 71.5 tok/s and a small decode nudge to 6.07.