Tiered Inference at Home: Three Local Models, One Consumer GPU, and Hermes Routing the Traffic

TL;DR: Shockwave now runs a two-tier AI stack. A cloud frontier model handles the reasoning; three small models running locally on a 16 GB AMD GPU handle the high-volume, low-complexity work at zero marginal cost. Hermes decides which tier each task lands on. All three local models stay resident on the GPU at once, serving between 55 and 85 tokens per second.

It has been a few months since the last Shockwave update, and the interesting part of this one is not new hardware; it is what the existing hardware is finally being asked to do. Getting the server to behave like a real studio assistant meant being more deliberate about which model does which job. The result is a hybrid stack: Claude Opus over the API for anything that requires actual judgment, and a local, GPU-accelerated Ollama instance for the endless small stuff.

Why Run a Local Model Tier Alongside a Cloud Model?

Most setups reach for a frontier API for everything, then watch the bill climb as the low-complexity calls take over: tag this, classify that, summarize this feed item. Those tasks do not need a frontier model. They need a fast, cheap, always-available one.

The efficient answer is tiered inference:

• Frontier cloud model (Claude, Gemini, DeepSeek): complex reasoning, planning, code, judgment.
• Local small models: the reflexes; routing, extraction, labeling, first-pass summarization.

Running the small tier on a GPU I already own converts a recurring per-token cost into a fixed one I have already paid, removes rate limits and external dependencies, and keeps that data in the house. The setup is modest, but it does what I need.

What Hardware Is This Running On?

I covered this in the first couple of entries in the series, but it is worth restating in context.

Worth emphasizing: this is a consumer card, not a datacenter part. ROCm support for RDNA4 is what makes local model serving viable on off-the-shelf hardware.

What Is in the Local AI Stack?

Ollama as the Inference Server

Ollama runs as a systemd service (ollama.service) with auto-restart on failure. The API is exposed on 0.0.0.0:11434, reachable by Hermes, n8n, other machines on the network, and Docker containers alike. It is fully GPU-accelerated through ROCm, verified at 100% GPU placement rather than CPU fallback. That fallback is a common silent failure when the service user is not in the GPU access groups.

The Model Lineup

This one took me some time to settle on. I had never really worked with Qwen before, so no time like the present. The lineup skews Qwen, Gemma, and Phi, a standing preference to avoid certain vendors. All of them are small, and all of them stay resident on the GPU:

*Measured generation throughput; GPU, single-stream, 2026-08-18.

Swapping any of them out is a configuration change, not a rebuild, which was the point of setting it up this way.

Tuning Ollama for Concurrency

The daemon is tuned for simultaneous multi-model, multi-consumer use:

OLLAMA_MAX_LOADED_MODELS=3   # keep all 3 small models hot, no reload thrash
OLLAMA_NUM_PARALLEL=2        # concurrent requests per model
OLLAMA_KEEP_ALIVE=30m        # stay warm 30 min between jobs

All three sit on the GPU at once, using roughly 7.5 GB of the available 16 GB, with no cold-start penalty on repeat calls. Ollama’s FAQ documents these variables and their defaults.

How Does Hermes Route Work Between the Two Tiers?

Hermes is the connective tissue: one agent core across CLI, messaging platforms, and scheduled jobs. It is wired to the local Ollama as a routable secondary provider (http://127.0.0.1:11434/v1, OpenAI-compatible) while keeping the cloud frontier model as the default for reasoning.

Switching tiers mid-session is a one-liner (/model qwen2.5:3b, then /model opus to come back), and delegated subtasks can be pushed to the local tier automatically without me having to do a thing.

Hermes is also connected to a Docker MCP gateway , which gives the agent sandboxed web-fetch capability through the Model Context Protocol; a tidy way to extend an agent through containerized, isolated tools. I talk to the server over Discord and Signal, so I can give direction and start workflows from anywhere, and rclone connects it to shared drives and cloud volumes well beyond my own network.

What Does Tiered Inference Actually Save?

• Recurring cost becomes fixed cost. High-volume classification and extraction that would otherwise meter per token now runs on hardware I already bought.
• No rate limits and no external dependency on the cheap tier. Automations can hammer the local endpoint freely.
• The data stays home for anything routed locally.
• Consumer hardware is enough. A card under $500 serves three models concurrently at 55 to 85 tokens per second. The barrier to entry for local AI is lower than most people assume.
• The routing layer is where the value is. Tiered inference only pays off if something decides which tier each task belongs to. That is the agent’s job.

What Did I Learn Setting This Up?

1. Verify GPU placement explicitly. Ollama falls back to CPU quietly if the service user is not in the GPU access groups. Confirm 100% GPU with a live call; do not assume the setup is correct.
2. Right-size the model to the task. Reasoning models that emit long internal thinking chains are the wrong tool for cheap, high-volume work; they burn tokens on deliberation you do not need. Small instruct models win here.
3. Container networking matters. localhost inside a Docker container is not the host. Reaching a host service from a container needs the Docker gateway IP or an explicit host mapping. If you are new to local AI setups, learn containerization early; Docker and Podman are both good places to start.
4. Concurrency needs explicit tuning. Out of the box, Ollama unloads models quickly and serves one at a time. A few environment variables turn it into a proper multi-model, multi-consumer server.
5. Consumer AMD and ROCm are real now. RDNA4 inference works well enough for most of what I ask of it. AMD is a genuine alternative to NVIDIA for this use case.

What Is Next for Shockwave?

n8n workflow integration is the immediate one: routing automation steps, like the daily tech brief’s summarization, through the local models for zero-cost processing. Beyond that, expanding the local model roster as new small models ship, and a lightweight routing policy so Hermes selects the tier by task type instead of waiting for me to switch it manually.

More to come. Happy hacking.