Qwen3.8:27B on vLLM — The Alternate Option to llama.cpp

llama.cpp is a single-user runtime — one person, one model, on your machine. vLLM does the opposite job: one model serving many users at once, with continuous batching so nobody waits in line. Below we turn Qwen3.8-27B (the same weights you already run locally) into a shared OpenAI-compatible API server.

In this guide we will:

• Pick your quantization: AWQ INT4 vs FP8 vs NVFP4
• Install vLLM in an isolated environment with uv
• Download Qwen3.8-27B AWQ INT4 weights (~21GB)
• Split the model across both GPUs with tensor parallelism 2
• Test the OpenAI-compatible API — chat, fast mode, and vision
• Expose it to other devices on your network (optional)
• Tune parameters & fix OOM (if it happens)
• Make it auto-start on boot — desktop icon or service
• Swap in uncensored weights (optional)

Step 1: Pick Your Quantization

A 27B dense model in BF16 needs ~54GB; our two cards hold 32GB total. Quantization is not optional, and AWQ INT4 fits with room left for context. (Driver + nvidia-smi are already set up from the llama.cpp guide.)

Same 27B weights at three precision levels, split across both GPUs (--tensor-parallel-size 2, ~14.4GB usable per card):

Format	Repo (HuggingFace)	Weight size	Weights/card	Left for KV + overhead	Sweet spot
AWQ INT4 ⭐	cyankiwi/Qwen3.8-27B-AWQ-INT4	21.0 GB	~10.5 GB	~3.9 GB/card → largest pool	Long context, multiple concurrent agents
FP8 (official)	Qwen/Qwen3.8-27B-FP8	~29–30 GB	~14.5 GB	tight (~0–1.5 GB/card)	Maximum fidelity, shorter max context
NVFP4	unsloth/Qwen3.8-27B-NVFP4	24.6 GiB	~12.3 GB	~2 GB/card	Native sm_120 kernels — fastest of the three

Note: No separate CUDA toolkit needed — the PyPI wheel bundles its own CUDA runtime. Your only job is that nvidia-smi works.

Step 2: Install vLLM in an Isolated Environment

Ubuntu 26.04 protects system Python from direct pip, and a dedicated env keeps ~6GB of torch away from your LM Studio tools. uv creates it — downloading Python 3.12 if you don’t have one:

curl -LsSf https://astral.sh/uv/install.sh | sh
[ -f "$HOME/.local/bin/env" ] && source "$HOME/.local/bin/env"
which uv    # must print a path ending in /uv — empty = open a new terminal

# --seed puts a real pip inside the venv (without it, bare 'pip install' hits system Python)
uv venv --python 3.12 --seed ~/vllm-env
source ~/vllm-env/bin/activate

uv pip install vllm    # ~6–8GB — vLLM + CUDA-bundled PyTorch

Note: --seed is deliberate: a plain uv venv ships no pip, so bare pip install falls through to Ubuntu’s system Python and dies with PEP 668. With it, both uv pip and plain pip work — we use uv pip because it’s faster.

Verify the version and that PyTorch sees your GPUs:

vllm --version    # expect >= 0.17

python -c "import torch; \
print('CUDA avail:', torch.cuda.is_available()); \
print('GPU 0:', torch.cuda.get_device_name(0)); \
print('Capability:', torch.cuda.get_device_capability(0))"

You want capability (12, 0) — sm_120 consumer Blackwell; an older number like (8,9) means the driver is too old and PyTorch fell back.

Note: If torch.cuda.is_available() is False but nvidia-smi works: driver/wheel mismatch — recreate the venv from scratch, don’t patch it.

One Small System Package vLLM Will Need Later

vLLM’s sampling path uses FlashInfer, which JIT-compiles CUDA kernels on first use — its build tool is ninja, and a fresh Ubuntu doesn’t ship it. Without it the first launch dies in warmup with a traceback ending in FileNotFoundError: ... 'ninja'. Install now:

sudo apt update && sudo apt install -y ninja-build
command -v ninja    # → /usr/bin/ninja = ready

Step 3: Download the Model Weights

Qwen3.8-27B: dense 27B, hybrid attention (long context is cheap), native image input, built-in draft head for faster decoding, 262K token window by default.

Download AWQ INT4 — highest community download count:

uv pip install -U huggingface_hub    # make sure the hf CLI is current
hf download cyankiwi/Qwen3.8-27B-AWQ-INT4 --local-dir "~/LOCAL MODEL/cyankiwi/Qwen3.8-27B-AWQ-INT4"

The model lands in ~/LOCAL MODEL/cyankiwi/Qwen3.8-27B-AWQ-INT4 (~21GB); keep ≥30GB free on that disk. Every future download uses the same <author>/<model> pattern, so your folder tree mirrors HuggingFace’s layout.

The two alternatives:

hf download Qwen/Qwen3.8-27B-FP8 --local-dir "~/LOCAL MODEL/Qwen/Qwen3.8-27B-FP8"
hf download unsloth/Qwen3.8-27B-NVFP4 --local-dir "~/LOCAL MODEL/unsloth/Qwen3.8-27B-NVFP4"

Step 4: Launch vLLM Across Both GPUs

Make sure both GPUs are idle first (close llama.cpp / ComfyUI if they hold VRAM). Launch with tensor parallelism 2 — each GPU holds half the weights plus its own share of the KV pool:

#!/usr/bin/env bash
source "$HOME/vllm-env/bin/activate"
exec vllm serve \
  "$HOME/LOCAL MODEL/cyankiwi/Qwen3.8-27B-AWQ-INT4" \
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --port 8000

Used the default HF cache in Step 3 (no --local-dir)? Just pass the repo ID instead: vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4.

Running FP8 or NVFP4? Same flags — point the model path at that folder, and for FP8 lower --max-model-len to 32768. Both load out of the box on consumer sm_120.

Key Parameter Breakdown

Flag	Purpose
--served-model-name qwen3.8-27b	Name clients use in API calls. Dashes, not Ollama-style colons (qwen3.8:27b) — some SDKs break on colons
--tensor-parallel-size 2	Splits weights across both GPUs. The whole reason FP8 is even feasible on 2×16GB
--gpu-memory-utilization 0.90	vLLM reserves this fraction of each GPU’s VRAM for weights + KV; the KV pool gets what remains
--max-model-len 65536	Cap on prompt+completion tokens per request (native window is 262K — raising it grows worst-case KV reservation)
--kv-cache-dtype fp8	Halves the KV footprint vs BF16 at negligible quality cost — doubles how many concurrent long sessions fit
--reasoning-parser qwen3	Moves the thinking chain into a separate reasoning_content field instead of gluing it to content. Not optional in practice for this generation
--enable-auto-tool-choice --tool-call-parser qwen3_coder	Structured function calling for agent use. Drop both if you only want plain chat
--enable-prefix-caching	Reuses KV blocks across requests sharing a prompt prefix (system prompts, few-shot context)

Run It With a Log You Can Come Back To

Launching bare in a terminal is fine for trying things — but close the window and the error history goes with it. For any run you want to diagnose later, append > ~/vllm.log 2>&1 & to Step 4’s launch command:

tail -f ~/vllm.log    # live view of that same run — Ctrl+C stops following, not the server

Once you have Step 8’s launch script it collapses to one line. This is exactly how this video’s vllm.log was produced:

bash "$HOME/Shell Scripts/vllm-start.sh" > ~/vllm.log 2>&1 &
tail -f ~/vllm.log

Whatever dies, dies into that file — full tracebacks included. Stop the backgrounded server with pkill -f "vllm serve".

Reading the Startup Log

Healthy cold start on this rig — real numbers from this video’s first launch:

Loading safetensors checkpoint shards: 100% Completed | 5/5 [~28 s]
Model loading took 9.72 GiB memory and ~32 seconds
torch.compile took 42.78 s in total        ← first launch only — cached afterwards
GPU KV cache size: ~200K–280K tokens (AWQ, this rig)
Uvicorn running on http://0.0.0.0:8000     ← server is live, Step 5 can start

Budget ~5 minutes for the very first cold start; every launch after is faster — vLLM caches compiled artifacts in ~/.cache/vllm, FlashInfer’s JIT kernels in ~/.cache/flashinfer. The “No available shared memory broadcast block” INFO during compilation is normal — the engine waiting on those one-time builds.

Two log lines decide what you can tune later:

... GPU KV cache size: 245,312 tokens ...
... Maximum concurrency for 65,536 tokens per request: xx.x% ...

• KV cache size (tokens) = total pool across both cards at your --max-model-len — ~200K–280K with AWQ on this rig, far less with FP8.
• Maximum concurrency = how many simultaneous max-length requests fit; a casual agent stack wants >10% headroom here.

If the server prints those lines and serves on port 8000, this step is done — if it dies with an OOM at startup, go to Step 7 instead of guessing.

Step 5: Test the OpenAI-Compatible API

Test with plain curl — no SDK needed, and it looks good on camera. Keep the server in Terminal 1; open a second terminal for everything below:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Explain what hybrid attention is in one sentence."}],
    "max_tokens": 150,
    "temperature": 0.7
  }' | python -m json.tool

Thinking is on by default; --reasoning-parser keeps the chain in a separate reasoning_content field so content stays clean. To skip the thinking phase per request:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Ping — reply with just PONG."}],
    "max_tokens": 50,
    "extra_body": {"chat_template_kwargs": {"enable_thinking": false}}
  }' | python -m json.tool

Vision test (it’s a native VLM)

The same endpoint takes images — no second service needed:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Describe this image in one line."},
      {"type": "image_url", "image_url": {
        "url": "https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png"
      }}
    ]}],
    "max_tokens": 100,
    "extra_body": {"chat_template_kwargs": {"enable_thinking": false}}
  }' | python -m json.tool

Verify It Works

curl http://localhost:8000/v1/models | python -m json.tool    # model is registered
nvidia-smi    # both GPUs should show VRAM activity + tensor-parallel processes

Note: Ports never collide: LM Studio stays on 1111, vLLM takes 8000. Any OpenAI SDK works unchanged — switch backends by changing one line (base_url):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")  # "EMPTY" on localhost; your --api-key if Step 6 is on

resp = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Summarize this file structure in 3 bullets."}],
)
print(resp.choices[0].message.content)

Step 6: Expose It Beyond This Machine (Optional)

By default vLLM binds to 127.0.0.1 — only this machine can reach it. When another device needs access, add two flags:

#!/usr/bin/env bash
source "$HOME/vllm-env/bin/activate"
exec vllm serve \
  "$HOME/LOCAL MODEL/cyankiwi/Qwen3.8-27B-AWQ-INT4" \
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --host 0.0.0.0 \
  --api-key papaya \
  --port 8000

• --host 0.0.0.0 — listen on every interface; other devices hit http://<your-LAN-IP>:8000/v1
• --api-key — your key, exactly as typed (papaya). Pick one fixed string and reuse it forever: clients save it once, restarts change nothing. Optional by default, mandatory in practice once you expose.

Verify from another device on your network — no key returns 401 Unauthorized, with the key you get 200 + the model list:

curl -i http://192.168.x.x:8000/v1/models
curl -s http://192.168.x.x:8000/v1/models \
  -H "Authorization: Bearer papaya"

Note: A localhost-only server without a key is fine if only you use this machine — the risk starts at --host 0.0.0.0: an open port with no password is free compute for anyone on the network.

Step 7: Tune Parameters & Troubleshooting

Each lever mapped to its symptom — the numbers come from Step 4’s log lines:

Lever	Flag	When to pull it
Context cap	--max-model-len	Raise toward 131072/262144 if KV pool headroom is huge; lower if concurrency % is tiny
Memory budget	--gpu-memory-utilization 0.90–0.95 ceiling	OOM at load → lower. Raising past 0.92 only when desktop session & other programs are off that GPU (check nvidia-smi baseline first)
Drop vision tower	--language-model-only	Text-only workloads — hands back ~2–2.5GB to the KV pool
Graph capture memory	--enforce-eager	Startup dies during CUDA graph capture even though load succeeded
Speculative decode (MTP)	--speculative-config '{"method":"mtp","num_speculative_tokens":3}'	Single-user latency focus — the checkpoint ships its own draft head, verified in all three precisions above

Thirty-Second Throughput Check

Quick enough to prove “it runs fast” on camera — no benchmark tool needed:

time curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Count from 1 to 50."}],"max_tokens":400,"extra_body":{"chat_template_kwargs":{"enable_thinking":false}}}' \
  > /dev/null

Watch the throughput lines vLLM prints in the server log after each batch. Real numbers: vllm bench serve --help.

Step 8: Auto-Start on Boot

vLLM is a CLI server — no app icon like LM Studio. Two ways to get one-click launch after reboot: Option A for daily use, Option B if you want it headless and always-on. Both point at the same script, so swapping models later means editing one file only — change the model path on the serve line and nothing else.

The Launch Script (shared by both options)

Create ~/Shell Scripts/vllm-start.sh (same folder as your other scripts):

nano "$HOME/Shell Scripts/vllm-start.sh"

Paste this:

#!/usr/bin/env bash
source "$HOME/vllm-env/bin/activate"
exec vllm serve \
  "$HOME/LOCAL MODEL/cyankiwi/Qwen3.8-27B-AWQ-INT4" \
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --port 8000

Make it executable — skip this and the first launch dies with Permission denied:

chmod +x "$HOME/Shell Scripts/vllm-start.sh"

exec replaces the shell with vLLM itself, so closing the window or stopping the service lands directly on the server. Used Step 6’s flags? Add --host 0.0.0.0 --api-key papaya before --port 8000.

Note: Line 2 activates the venv from inside the script, so this same file works from a terminal, a desktop icon, or systemd — no pre-activation needed anywhere.

Option A: Desktop Icon (your LM Studio habit)

A terminal window runs the server inside it; close the window = server stops:

nano ~/.local/share/applications/vllm.desktop

Paste this:

[Desktop Entry]
Type=Application
Name=vLLM Qwen3.8-27B
Comment=Qwen3.8:27B on vLLM — close this window to stop the server
Exec=bash -c '"$HOME/Shell Scripts/vllm-start.sh"; echo "Server stopped."; exec bash'
Terminal=true
Icon=utilities-terminal
Categories=Utility;Development;

Open your app menu and search vLLM (new icons sometimes need a logout/login). First launch asks you to trust the launcher — click Trust and Launch. The trailing exec bash keeps the window open after exit, so crashes stay visible.

Option B: systemd User Service (headless, always-on)

Boots with the machine, restarts on crash, no window needed:

nano ~/.config/systemd/user/vllm.service

Paste this:

[Unit]
Description=vLLM server — Qwen3.8-27B (AWQ INT4)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=%h/LOCAL\ MODEL    # systemd needs the backslash — a raw space would split this into two tokens
Environment="PATH=%h/vllm-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
ExecStart="%h/Shell Scripts/vllm-start.sh"
Restart=on-failure
RestartSec=10

[Install]
WantedBy=default.target

Load it and start it now:

systemctl --user daemon-reload
systemctl --user enable --now vllm     # start now + at every login
journalctl --user -u vllm -f           # live logs (Ctrl+C to leave)

Day-to-day: systemctl --user status vllm, systemctl --user stop vllm. Up right after power-on with no login? One command: loginctl enable-linger $USER.

Note: Both options bind port 8000 — run one or the other, not both. The second dies instantly with “address already in use.”

Bonus: Uncensored Weights (Optional)

Standard checkpoints keep the base model’s guardrails. If your private endpoint needs looser behavior, these are drop-in swaps — same layout, only line one changes:

Repo	Format	Size	What you get
twolven/Qwen3.8-27B-abliterated-AWQ-MTP ⭐	AWQ W4A16	18.2 GiB	Same budget class as the mainline AWQ; vision tower + MTP draft head included, so Step 7’s speculative lever still works
philbert440/Qwen3.8-27B-Uncensored-Aggressive-W4A16-AWQ	AWQ W4A16	18.2 GiB	Aggressive abliteration recipe — the loosest of the three; vision + MTP included
hwkranger/Qwen3.8-27B-heretic-ara-NVFP4	NVFP4 (modelopt)	19.2 GiB	Heretic “ARA” recipe at native sm_120 precision — fastest of the three; MTP included

hf download twolven/Qwen3.8-27B-abliterated-AWQ-MTP --local-dir "~/LOCAL MODEL/twolven/Qwen3.8-27B-abliterated-AWQ-MTP"
hf download philbert440/Qwen3.8-27B-Uncensored-Aggressive-W4A16-AWQ --local-dir "~/LOCAL MODEL/philbert440/Qwen3.8-27B-Uncensored-Aggressive-W4A16-AWQ"
hf download hwkranger/Qwen3.8-27B-heretic-ara-NVFP4 --local-dir "~/LOCAL MODEL/hwkranger/Qwen3.8-27B-heretic-ara-NVFP4"

Relaunch Step 4’s command with the model path pointed at the new folder — every other flag stays:

source "$HOME/vllm-env/bin/activate"
exec vllm serve \
  "$HOME/LOCAL MODEL/twolven/Qwen3.8-27B-abliterated-AWQ-MTP" \
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --port 8000

Note: The most-downloaded uncensored of this family is orcarouter/Qwen3.8-27B-Uncensored-FP8 (~29 GiB): auto-gated — accept its license on the repo page once, then download with your HF token (hf auth login). DIY instead? The p-e-w/heretic pipeline runs abliteration over any base checkpoint.

---

You now have a 27B VLM serving OpenAI-compatible chat and vision on both GPUs. AWQ INT4 by default for the largest KV pool; official FP8 when quality matters most; NVFP4 to show what sm_120 kernels do. Same weights, three formats, one endpoint. 🚀

References

• Official vLLM recipe for Qwen3.8-27B — official launch commands and KV numbers per precision.
• p-e-w/heretic — the “censorship removal” pipeline behind most abliterated checkpoints in this family.
• orcarouter/Qwen3.8-27B-Uncensored-FP8 — most-downloaded uncensored of this family (~29 GiB block-FP8, auto-gated).