Linux Native AI Setup Guide

Running your AI tools natively on bare-metal Linux completely removes the virtualization overhead of a Windows VM. This allows your LLMs to communicate directly with your dual RTX 5060 Ti GPUs, resulting in significantly faster token generation and lower latency.

In this guide, we will:

1. Compile llama.cpp from source with CUDA support for maximum GPU performance
2. Install & configure llama.cpp & LM Studio as the GUI model manager
3. Install OpenClaw and route it to your local LLM server
4. Install Hermes Agent and connect it to the same instance
5. Set up systemd services so everything auto-starts on boot (after login)

---

Step 1: Compile llama.cpp from Source (CUDA + Dual GPU)

llama.cpp is the core inference engine that powers both LM Studio and local OpenAI-compatible APIs. Compiling it yourself ensures full CUDA optimization for your dual RTX 5060 Ti GPUs.

Prerequisites

# Install build dependencies
sudo apt update
sudo apt install -y build-essential cmake git curl wget

# Install NVIDIA CUDA toolkit (if not already installed)
sudo apt install -y nvidia-cuda-toolkit

Verify NVIDIA driver and CUDA are working:

nvidia-smi
nvcc --version

Clone & Build llama.cpp

# Clone the repository
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# Pull latest release tag (b9878 as of July 2026)
git checkout b9878

# Configure with CUDA enabled
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_MAX_DEVICES=2


# Build with all available cores
cmake --build build --config Release -j$(nproc)

Note :

If you happen to face the same error like me, you may use the following trick to fix it.

1. Open Terminal and input the following command :

sudo nano /usr/local/cuda-13/targets/x86_64-linux/include/crt/math_functions.h

2. In your text editor, modify the exact lines you found by adding noexcept(true) right before the closing semicolon, change them to look like this:

For rsqrt (around line 629):
extern  DEVICE_FUNCTION_DECL device_builtin float                   rsqrtf(float x) noexcept(true);

Second Edit :

For rsqrtf (around line 653):
extern  DEVICE_FUNCTION_DECL device_builtin float                   rsqrtf(float x) noexcept(true);

Note: Make sure device_builtin has the “t” in it, as it might just be a small typo in your terminal message.

3. Save the file (Ctrl + O, then Enter), exit (Ctrl + X), and run your build:

rm -rf build
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

Install System-Wide (Optional but Recommended)

sudo cmake --install build

This installs llama-cli, llama-server, and libraries to /usr/local/bin/.

Test the Build

Download a small test model and run inference:

# Pull a small model for testing
./build/bin/llama-cli -m <your-model.gguf> -ngl 999 -p "Hello, how are you?" -n 32

---

Step 2: Configure llama-server with Tested Parameters (Primary Usage)

Use the parameters you’ve already tested on Windows — adapted for Linux paths. This is your primary way to run models at full speed.

Key Parameter Breakdown

Category	Parameter	Purpose
Model	-m “path”	Main model file (Qwen3.6 27B Q6_K)
	—mmproj “path”	Vision projection for image understanding
	—image-min-tokens 1024	Minimum tokens for image analysis
Memory	—no-mmap	Load entirely into RAM/VRAM — avoids disk I/O
	-ngl 99	Offload all layers to GPU(s)
	—split-mode tensor	Tensor parallelism across GPUs
	—tensor-split 1,1	Equal split across dual RTX 5060 Ti (32GB pooled)
Performance	—flash-attn on	Flash attention — faster context processing
	—batch-size 4096	Parallel token batch size
	—ubatch-size 1024	Unbatched (single-request) size
	—threads 8 / —threads-batch 8	CPU thread count for preprocessing/postprocessing
Context	—ctx-size 65536	64K context window
	—cache-type-k q4_0 / —cache-type-v q4_0	Quantized KV-cache — saves VRAM vs BF16
Sampling	—temp 0.6 / —top-p 0.95 / —top-k 20 / —min-p 0.00	Temperature & top-p/k sampling for output quality
	—repeat-penalty 1.1	Reduces repetitive outputs
Speculative	—spec-type draft-mtp	MTP speculative decoding — faster token generation
	—spec-draft-n-max 3 / —spec-draft-n-min 2	Speculative draft range (2-3 tokens)
Chat	—jinja	Enable Jinja chat templating
	—chat-template-kwargs …	Preserve thinking/reasoning tags in output
Server	—host 0.0.0.0 / —port 8080	Listen on all interfaces, port 8080
Monitoring	—props / —metrics / —perf	Expose performance metrics & monitoring endpoints

Shorcut for Shell Script

To turn your parameters into a clickable shortcut on your Ubuntu desktop, you need to create a shell script .sh that calls the freshly compiled ./build/bin/llama-server binary, and pair it with a .desktop launcher.

Step 1: Create the Executable Shell Script

1. Open a terminal and create the new shell script file:

nano ~/Desktop/qwen27b.sh

2. Paste the following configuration, which points to your local compiled directory and targets your exact flags:

#!/bin/bash
Navigate to your freshly compiled llama.cpp repository
cd "/home/tbn/llama.cpp"

./build/bin/llama-server \
  -m "/home/tbn/.cache/lm-studio/models/Qwen3.6-27B/your-model.gguf" \
  --mmproj "/home/tbn/.cache/lm-studio/models/Qwen3.6-27B/your-mmproj.gguf" \
  --image-min-tokens 1024 \
  --no-mmap \
  --ctx-size 65536 \
  --flash-attn on \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --fit off \
  --split-mode tensor \
  --tensor-split 1,1 \
  --parallel 1 \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 999 \
  --threads 8 \
  --threads-batch 8 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.00 \
  --presence-penalty 0.00 \
  --repeat-penalty 1.1 \
  --jinja \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --reasoning-format none \
  --reasoning-budget 16000 \
  --props \
  --metrics \
  --perf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 2

echo ""
read -p "Server stopped. Press Enter to exit..." temp

3. Save the file by hitting Ctrl + O, click Enter, then exit using Ctrl + X.

4. Grant system execution permissions to the script:

chmod +x ~/Desktop/qwen27b.sh

Step 2: Create the Double-Click Desktop Launcher Ubuntu requires a .desktop map to launch scripts directly into an active terminal window upon a double-click event.

1. Generate the shortcut profile file on your desktop:

nano ~/Desktop/Qwen27b.desktop

2. Paste the launcher schema layout:

# text
[Desktop Entry]
Version=1.0
Type=Application
Name=Qwen3.6:27b
Comment=Launch Local Qwen LLM Server
Exec=/home/tbn/Desktop/qwen27b.sh
Icon=utilities-terminal
Terminal=true
Categories=Development;

3. Save and close (Ctrl + O, Enter, Ctrl + X).

Step 3: Authorize Execution on the Desktop

1. Minimize your windows and locate the new QwenServer.desktop file on your actual Ubuntu desktop screen.

2. Right-click on the icon.

3. Click and select “Allow Launching” from the context dropdown menu.

The file icon will automatically transform into a system terminal grid logo. You can now double-click it anytime to host your model’s OpenAI-compatible API on port 8080 instantly.

Verify It Works

In another terminal:

Check model is loaded
curl http://127.0.0.1:8080/v1/models | python3 -m json.tool

Quick chat test

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-27B","messages":[{"role":"user","content":"Hello"}]}' | python3 -m json.tool

Check GPU memory usage (both GPUs should show VRAM activity)

nvidia-smi

---

Step 3: Install LM Studio as Backup/Alternative

LM Studio is a convenient GUI fallback — use it for quick model downloads and as a backup server if llama.cpp needs maintenance.

Download & Install

cd ~/Downloads

# Download latest AppImage from lmstudio.ai/download
wget https://github.com/AIDot-OpenLLM/lm-studio/releases/latest/download/LM-Studio-x86_64.AppImage -O LM-Studio.AppImage

chmod +x LM-Studio.AppImage

First Launch & Model Setup

1. Double-click LM-Studio.AppImage to launch
2. Go to the 🔍 Search tab → search for Qwen3.6:27b or your preferred model
3. Click Download and wait for it to finish (stores in ~/.cache/lm-studio/models/)

Configure Server Settings

Navigate to the 🔄 Local Server tab and configure:

Setting	Value	Notes
Context Length	65536 (Max)	65K context window
GPU Offload	MAX	All layers on GPU — essential for speed
K/V Cache Quantization	Q4_0	Balances memory vs quality
Host	127.0.0.1	Local only (change to 0.0.0.0 for network access)
Port	1234	Default port

Start the Server

Click “Start Server” — it will listen on:

http://127.0.0.1:1234/v1

Test the API endpoint:

curl http://127.0.0.1:1234/v1/models | python3 -m json.tool

You should see your loaded model listed.

---

Step 4: Install & Configure OpenClaw

OpenClaw is an AI agent framework that connects to local LLM backends. Since LM Studio runs locally, routing is straightforward.

Install OpenClaw

# Install via npm (recommended)
npm install -g openclaw

# Verify installation
openclaw --version

Configure OpenClaw to Use Local llama.cpp Server

openclaw configure

When prompted:

1. Model Provider: Select OpenAI Compatible
2. API Base URL: Enter http://127.0.0.1:8080/v1 (your llama-server port)
3. API Key: Leave empty or type llama-cpp (not required for local hosting)
4. Model Name: Your loaded model name

Verify Connection

openclaw dashboard

OpenClaw should connect to your llama.cpp server and display the active model.

---

Step 5: Install & Configure Hermes Agent

Hermes Agent is a full-featured AI assistant that can connect to local LLM servers via OpenAI-compatible endpoints.

Install Hermes Agent

# Run the official installer
curl -sSL https://hermes-agent.com | bash

# Verify installation
hermes --version

Configure Hermes to Use Local llama.cpp Server

hermes setup

During setup:

1. **Provider:** Choose `OpenAI Compatible`
2. **Base URL:** Enter
`http://127.0.0.1:8080/v1` (same as OpenClaw)
3. **API Key:** Type `llama-cpp` (no real key needed for local hosting)
4. **Model:** Your loaded model name

Test Hermes Agent

hermes chat "Hello, can you hear me?"

Hermes should respond using your locally hosted model.

---

Step 6: Set Up Auto-Start on Boot (systemd Services)

Make all services start automatically when Ubuntu boots — they’ll launch as soon as you log in.

Create a systemd Service for llama-server

Create the service file:

sudo nano /etc/systemd/system/llama-cpp.service

Add this content (adjust paths to match your setup):

[Unit]
Description=llama.cpp Server (CUDA)
After=network.target nvidia-driver.service
Wants=nvidia-driver.service

[Service]
Type=simple
User=tbn
WorkingDirectory=/home/tbn
ExecStart=/usr/local/bin/llama-server \
  -m /home/tbn/.cache/lm-studio/models/Qwen3.6-27B/your-model.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999 \
  --tensor-split 50:50 \
  -c 65536 \
  -tb 4096 \
  --mlock \
  -ctk q4_0
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Note: Replace the --model path with your actual model file location. You can find it in LM Studio’s model browser.

Create a systemd Service for OpenClaw

sudo nano /etc/systemd/system/openclaw.service

[Unit]
Description=OpenClaw AI Agent
After=llama-cpp.service
Requires=llama-cpp.service

[Service]
Type=simple
User=tbn
WorkingDirectory=/home/tbn
ExecStart=openclaw serve --host 127.0.0.1 --port 8080
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Create a systemd Service for Hermes Agent

sudo nano /etc/systemd/system/hermes-agent.service

[Unit]
Description=Hermes AI Agent
After=openclaw.service llama-cpp.service
Requires=llama-cpp.service

[Service]
Type=simple
User=tbn
WorkingDirectory=/home/tbn
ExecStart=hermes serve --host 127.0.0.1 --port 9000
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Enable & Start All Services

# Reload systemd to pick up new services
sudo systemctl daemon-reload

# Enable all services (auto-start on boot)
sudo systemctl enable llama-cpp.service
sudo systemctl enable openclaw.service
sudo systemctl enable hermes-agent.service

# Start them now (without rebooting)
sudo systemctl start llama-cpp.service
sudo systemctl start openclaw.service
sudo systemctl start hermes-agent.service

Verify Services Are Running

systemctl status llama-cpp.service
systemctl status openclaw.service
systemctl status hermes-agent.service

All three should show active (running).

Check Logs (If Something Goes Wrong)

journalctl -u lm-studio-server.service -f
journalctl -u openclaw.service -f
journalctl -u hermes-agent.service -f

---

Quick Reference: Service Management Commands

Action	Command
Start all	sudo systemctl start llama-cpp openclaw hermes-agent
Stop all	sudo systemctl stop llama-cpp openclaw hermes-agent
Restart one	sudo systemctl restart ‘service-name’
Check status	systemctl status ‘service-name’
View logs	journalctl -u ‘service-name’ -f
Disable auto-start	sudo systemctl disable ‘service-name’

---

Troubleshooting

llama.cpp doesn’t detect GPU

# Check NVIDIA driver is loaded
nvidia-smi

# Check CUDA visibility
env | grep -i cuda

# Rebuild with verbose output to see compiler flags
cmake --build build --config Release -j$(nproc) 2>&1 | grep -i cuda

Services fail to start on boot

Make sure: User= field matches your username (replace “tbn” if different) Model path in llama-cpp.service is correct and the file exists Test manually first before relying on systemd:

sudo -u tbn /usr/local/bin/llama-server \
  -m /path/to/your/model.gguf \
  --host 127.0.0.1 --port 8080 -ngl 999 -c 4096

Port conflicts

If ports 1234, 8080, or 9000 are already in use, change them in the service files and configuration wizards.

---

Summary

You now have a fully native Linux AI stack:

✅ llama.cpp compiled from source with CUDA — maximum GPU performance

✅ **Dual RTX 5060 Ti GPUs** pooled via tensor splitting (32GB VRAM)
✅ **llama-server** running as OpenAI-compatible API on port 8080
✅ **LM Studio** used only for model downloads (not inference)
✅ **OpenClaw** connected to llama.cpp server
✅ **Hermes Agent** also using the same llama.cpp instance
✅ **All services auto-start on boot** via systemd

Your dual GPUs are now directly powering your AI agents with zero virtualization overhead. Enjoy the speed! 🚀
---