This month my feeds filled up with posts about Qwen3.8-27B running at 50 to 75 tokens per second on an RTX 3090 with llama.cpp. Great numbers. My spare Linux laptop has an RTX 3060 Laptop GPU with 6 GB of VRAM, which is not even half of what the Q4 weights need. I wanted to know if the model could run on it at all.

It can. As I write this, a coding agent on my LAN is talking to Qwen3.8-27B served from that laptop. Decode speed is about 3 tokens per second. That is slow, and it is real, and getting there taught me more about where the actual bottleneck lives than any of the viral posts did.

This is the full story: the hardware, what broke, the technique that worked, exact commands, and measured numbers. It is written for people with a similar class of laptop (4 to 8 GB of VRAM, 32 GB or more of system RAM). It is not a recipe for 75 tok/s.

The short answer

Qwen3.8-27B Q4_K_M (a 15.32 GiB GGUF, 27.32 B parameters) runs on a 6 GB GPU if you have around 32 GiB of system RAM, a current llama.cpp CUDA build, and you stop trying to put all the weights in VRAM. llama.cpp can auto-fit about 5.2 GiB of layers onto the GPU and keep the rest in system RAM. The GPU-resident layers fly; the RAM-resident layers are the bottleneck.

Measured on the same machine:

On a small function-writing eval (temperature 0, hidden tests), the 7B passed 2 of 3 in about 4 seconds. The 27B passed 3 of 3 in about 4 minutes. So the daily driver stays the 7B, and the 27B is what I call when quality is worth the wait.

The machine

An Ubuntu 24.04 laptop that lives on my LAN as a light model box:

The hybrid AMD + NVIDIA setup matters more than it looks. Two of the worst failures in this project were not model problems, they were “keep this laptop alive” problems. More on that below.

Why 6 GB fails in the first place

The 3090 posts all assume one thing: the whole model already lives in VRAM. Once that is true, the famous tricks (Q4 weights, MTP speculative decoding, CUDA graphs, quantized KV cache) stack nicely. On 6 GB that assumption is dead on arrival, so it helps to know what is actually inside the file.

Qwen3.8-27B is a 64-layer hybrid, three linear-attention blocks for every full-attention block:

[GDN][GDN][GDN][Attn]  x 16

The KV cache is a non-issue at my context sizes: about 0.25 GiB at 4k tokens. The problem is that roughly 63% of the parameters are feed-forward weights, and every decode token has to read all 64 FFN blocks. Whatever does not fit in VRAM gets read from system RAM through the CPU, over and over. That memory traffic is the 3 tok/s.

This also explains why MTP, the speculative multi-token head everyone cites for the big speedups, is the wrong first move here. Speculation multiplies the number of forward passes you verify. When each pass is bottlenecked on reading FFN weights from RAM, you are multiplying the slow part.

The technique: RAM copy + GPU fit

The configuration that actually works, in the order the pieces matter:

1. Q4_K_M quantization (Unsloth Dynamic). 15.32 GiB on disk, 4-bit weights.
2. llama.cpp with CUDA, not Ollama, for the 27B. Ollama is fine for the 7B that fits entirely on the GPU.
3. --fit on -fitt 400. This packs as many layers as fit in 6 GB and leaves 400 MiB free for compute buffers and KV. One critical detail: do not pass -ngl 99 alongside it. If ngl is set, the fit logic aborts and llama.cpp tries a single ~14.6 GiB cudaMalloc, which fails instantly.
4. -lm none. This loads the GGUF into anonymous process memory instead of relying only on mmap page cache. The llama-server process resident size lands between 11 and 16 GiB, which is exactly the point: the model is really in RAM, not in a cache the kernel can evict.
5. 16 CPU threads (-t 16) for the layers that stay in RAM.
6. Flash attention on, KV cache quantized to q8_0 (-ctk q8_0 -ctv q8_0), to keep VRAM for weights.
7. Context 4096. Not 90k. On this card that fight is not winnable.
8. reasoning_effort=low in the chat template. Qwen3.8 is a thinking model and will produce reasoning traces regardless; none is not a valid value, only xhigh, medium and low are.

Two more findings that cost me time. CUDA graphs are already active in current llama.cpp builds (the logs show graphs reused=62 on a completion), so there is nothing extra to enable there. And on this hybrid laptop the discrete GPU must be kept powered at all times; letting it enter runtime D3 idle coincided with hard freezes of the whole machine.

What I measured and discarded

The hand-pinning result is worth a sentence. Because of the 3:1 hybrid layout, I expected that choosing exactly which FFNs go to CPU would beat llama.cpp’s sequential auto-fit. It did not. When the bottleneck is total bytes read from RAM per token, it barely matters which layers those bytes belong to.

Keeping the laptop alive

None of the inference numbers matter if the machine freezes. Three pins made it stable, and I would do them first on any similar hybrid laptop.

Pin the kernel. This chassis lost ethernet, wifi and the mouse on kernel 6.17.0-35 and on the HWE 7.0 series. I hold 6.17.0-29-generic, block the newer packages with an apt preferences file, and set Prompt=never for release upgrades.

Pin the NVIDIA userspace to the loaded kernel module. My working combination is 580.159.03 on both sides. Software Updater happily pulled a 580.173 userspace on top of that, and nvidia-smi refused to talk to the mismatched module. The fix is to downgrade the userspace back to the module’s version and hold it. Small trap inside the trap: apt pin files must end in .pref, and a filename with extra dots is silently ignored.

Never let the dGPU power off. Runtime D3 on this AMD + NVIDIA combo produced silent hard freezes (the journal just stops). Two files fix it:

# /etc/modprobe.d/zz-nvidia-keep-powered.conf
options nvidia NVreg_DynamicPowerManagement=0x00

# /etc/udev/rules.d/99-nvidia-no-autosuspend.rules
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{power/control}="on"

One more practical note: the box has no CUDA toolkit and no nvcc. I used Unsloth’s portable linux-x64 CUDA 13 build of llama.cpp (build 10472, SM 86 included) and pointed LD_LIBRARY_PATH at the CUDA runtime libraries from pip (nvidia-cuda-runtime, nvidia-cublas). Do not mix in Ollama’s bundled libcudart; that combination segfaulted libggml-cuda.so for me. You also specifically need a llama.cpp recent enough to know the qwen35 architecture.

Serving it on the LAN

The server wrapper, trimmed to the flags that matter:

#!/usr/bin/env bash
set -euo pipefail
export LD_LIBRARY_PATH="$HOME/.local/lib/python3.12/site-packages/nvidia/cu13/lib:$HOME/llama.cpp${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

exec "$HOME/llama.cpp/llama-server" \
  --model "$HOME/models/Qwen3.8-27B-UD-Q4_K_M.gguf" \
  --alias Qwen3.8-27B \
  --host 0.0.0.0 --port 8080 \
  --fit on -fitt 400 -fitc 2048 \
  -c 4096 \
  -lm none \
  -t 16 \
  -fa on -ctk q8_0 -ctv q8_0 \
  --jinja \
  --chat-template-kwargs '{"reasoning_effort":"low"}' \
  -np 1 --no-webui

A systemd unit runs it (Restart=on-failure, TimeoutStartSec=300, since the first load copies 15 GiB into RAM and takes a minute or two). Then from any machine on the LAN:

curl -s http://192.168.1.50:8080/health
# {"status":"ok"}

curl -s http://192.168.1.50:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-27B",
    "temperature": 0.2,
    "max_tokens": 32,
    "messages": [{"role":"user","content":"Reply with only: pong"}]
  }'

Measured over the LAN: about 3.2 tok/s decode. If another machine cannot reach port 8080, check for a local firewall on the serving laptop; I keep mine off on this box.

For OpenCode this is just another OpenAI-compatible provider:

"lanbox": {
  "npm": "@ai-sdk/openai-compatible",
  "name": "27B on the LAN",
  "options": { "baseURL": "http://192.168.1.50:8080/v1", "apiKey": "local" },
  "models": {
    "Qwen3.8-27B": {
      "name": "Qwen3.8-27B (RAM+GPU fit)",
      "limit": { "context": 4096, "output": 2048 }
    }
  }
}

It is slow. It works. Expect visible reasoning traces even at reasoning_effort=low.

One gotcha when checking RAM

If you watch htop while this runs, the green “used” bar can stay small and make you think the model is not really loaded. mmap-loaded weights hide in the yellow cache portion, which the kernel can evict under pressure. With -lm none the honest signal is the llama-server process RES column: 11 to 16 GiB while serving. The technique is using that RAM; the green bar just does not show it.

Does this transfer to your laptop?

Yes, if it is in the same class. You need a 4 to 8 GB discrete NVIDIA GPU (3050, 3060, 4050 laptop tier), at least 32 GiB of RAM, and a llama.cpp CUDA build that understands the qwen35 architecture. With 16 GiB of RAM this Q4 does not fit; use a 7B, or a 2-bit quant of the 27B. With 12 GB or more of VRAM you can skip all of this, pass -ngl 99, and the 3090-style advice starts to apply to you.

The transferable recipe:

1. Stabilize the machine first: NVIDIA userspace matching the kernel module, and no runtime D3 on hybrid laptops.
2. Q4 GGUF on disk, llama.cpp CUDA build.
3. llama-server --fit on -fitt 400 -lm none -t <nproc> -c 4096, and make sure -ngl is unset.
4. Serve the OpenAI-compatible API on the LAN and point your tools at it.
5. Keep a 7B loaded nowhere near it (one model on the GPU at a time) for the interactive 50 tok/s work.

What does not transfer from the viral posts: the 75 tok/s, the 90k context, and MTP as a first move. Those all assume the model is GPU-resident. On 6 GB you are running a different experiment, one about how much intelligence you can stream through a memory bus, and 3 tokens per second of a 27B that solves problems the 7B cannot is a perfectly usable answer for an overnight agent.

The numbers, for citation

Engine: Unsloth llama.cpp portable build 10472, CUDA 13, SM 86. Benchmarks are llama-bench with 64-token runs, 3 repetitions, on the machine described above.