Three AMD Strix Halo boxes as a local model farm: what actually works

By Farough Darakhshan · 10/20/2026

I bought three AMD Strix Halo mini PCs to run language models locally for the back office of a water-damage restoration business. The idea was simple. A lot of the reading in this work is bulk reading: photos, PDFs, carrier estimates, long email threads. Cloud models are fast, but every page costs money. A box on a shelf costs electricity.

This is what I learned in the first month, with the numbers I measured. Some of it works well. Some of it bit me hard.

The hardware and the operating system

Each box runs Rex OS, which is Debian 13 with AMD's packages on top. Each has 125 GB of RAM, a 1.6 TB NVMe drive, and an integrated GPU that maps system memory instead of owning its own.

That last point matters more than anything else on these machines. The GPU shows 512 MB of dedicated VRAM. Everything a model needs comes from GTT, the shared memory the GPU is allowed to map. On the first box that is about 94 GB.

There is a setting in Rex OS called "video shared RAM %". It writes a kernel option and takes effect after a reboot. It is a ceiling, not a reservation. Memory the GPU does not use stays free for the operating system. I run the first box at 75% and the other two at 90%, which is what the largest model needs.

The first trap: rocm-smi and amd-smi lie on this chip. They report 512 MB of VRAM and zero load while the GPU is pegged. The real number is in /sys/class/drm/card1/device/gpu_busy_percent. Every dashboard I built reads that file.

Lemonade as the model server

All three boxes run Lemonade 11.5.0 as the model server. One server gives me several API shapes on the same port:

• OpenAI-style chat completions, for most scripts
• the Responses API, which Codex uses
• an Anthropic Messages endpoint, meant for Claude Code
• embeddings
• an MCP endpoint

Every route needs a bearer token, and the firewall only lets in the handful of machines that are supposed to talk to it.

Rex OS pins Lemonade to 11.5.0. A tool-schema bug that breaks Claude Code against Lemonade was fixed upstream in 11.5.1, but that fix does not reach these boxes until Rex ships it. So Claude Code against the local models is on hold. I check the package policy before I assume any upstream fix is installed.

Two more Lemonade rules I keep:

• Never install the llamacpp:rocm backend through Lemonade. The llama.cpp build Rex ships as the system build is the one that actually runs on this GPU.
• --parallel 1 is set globally to avoid another known bug, and raised only per model where I have tested it.

Model placement

The boxes are not interchangeable. I gave each one a job.

The first box is the fast one. It keeps Qwen3-Coder-30B-A3B for code, Qwen3.6-35B-A3B for vision, Qwen3-Embedding-4B for embeddings and bge-reranker-v2-m3 for reranking.

The second box is the heavy one. It keeps Qwen3.5-122B-A10B loaded as a checker for work the smaller models produce.

The third box came online on 9/28 with the same memory setting as the second, so it can take the large model later.

One compatibility fact cost me an afternoon: Codex cannot use the Qwen3.5 or Qwen3.6 models through Lemonade. Lemonade itself warns about it. Codex jobs go to the coder model.

Context slots split your context

This one is easy to miss. When you give llama.cpp --parallel 4, it does not give each request the full context window. It divides it.

The 35B vision model at a 32,768-token context with four slots gave each request 8,000 tokens. That was too small for the 60,000-character document chunks I was sending. I raised it to 65,536 with four slots, which is 16,000 tokens per request, and I keep each input under about 12,000 tokens.

The out-of-memory trap

Lemonade will load any model a request names. That is convenient on a laptop. On a shared box it is dangerous.

On 9/27 a test loaded a third model next to the two already resident on the heavy box. The kernel ran out of memory and killed the user systemd manager. When systemd restarted it, every enabled user service came back up, including one I had stopped two hours earlier on purpose.

On 10/19 it happened again from a different direction. One test call named the 122B model on a box that did not have it loaded. Lemonade loaded it beside everything else, the box ran out of memory, and every rootless container on it died at once, including the claims app I am building.

The rules I run on now:

• The heavy box holds the 122B plus one other model, never more.
• Nothing routes to a model on that box unless it is already loaded.
• A service that must stay down gets disabled, not only stopped.
• After any model work, I list running services and compare them with what I started.

Headless and remote

All three boxes boot without anyone logging in. After a cold boot, the first two accepted a remote connection in about 17 to 18 seconds. Wake-on-LAN shipped turned off, which is why my first remote power-on failed. It is on now.

Measured speed

Vision is the bulk of the work: reading job-site photos and moisture meter displays. I measured it on 10/01.

• One photo costs 2.2 to 2.5 seconds of image encoding, and that part is serial.
• A box does 0.25 to 0.35 photos a second no matter how many requests run at once. More concurrency only raises the wait.
• A PDF page takes 15 to 17 seconds alone, and 37 to 70 seconds under load.
• 10,295 photos take 8 to 11 hours per box.

That is slow next to a cloud model and free next to a cloud bill. Work that can run overnight goes to the boxes. Work someone is waiting for goes to the cloud.

Watch out for llama.cpp's prompt cache when you benchmark. Rerunning the same photos was eight times faster because the prompt was cached. Speed numbers need fresh inputs.

Do they match a cloud model?

On some jobs, yes. On a graded test drafting scope lines, the local 35B scored 59.3 against Claude Haiku's 59.2. On a harder job, finding gaps in a claim's evidence, the local model scored an F1 of 19.4 against Haiku's 22.2. Plain code with no model scored 21.5. The data, not the model, was the weak part there.

So I test each job type before I move it. A box earns a job by matching the cloud on a graded set.

Never idle, never busywork

An idle box is money sitting on a shelf. Each box runs an idle worker. When its GPU stays under about 20% for two minutes, it takes the next item from a queue that fits the models it already has loaded.

The queue holds only real work: open claims, unpaid invoices, disputes, jobs active in the last 30 days. Labeling photos of a claim that was paid two years ago is not work, it is heat. When the real queue runs out, the dashboard says "backlog empty" instead of inventing something.

Every job runs through one logged wrapper, so whether the boxes earn their keep is decided from the log, not from my impression.

What I would tell someone starting

1. Read GPU load from sysfs, not from the vendor tools.
2. Treat the shared-memory setting as a ceiling and size models to it.
3. Pin your models, and never let a stray request load a new one on a full box.
4. Divide context by slots before you size your inputs.
5. Measure throughput with fresh inputs.
6. Grade each job against a cloud model before you move it.

I'm Farough (Frank) Darakhshan. I write these notes as I build the systems that run a restoration back office. The rest of the projects are listed at faroughdarakhshan.com/projects, and the public code is on faroughdarakhshan.org.