§1The problem
The strongest models don't fit on an 8 GB card, and paid APIs charge per use and send the text elsewhere. I wanted privacy, controlled cost and speed, without leaving a rented machine running and forgotten.
§2What I built
- Local serverllama-server with the industry-standard chat API: starts light, loads the model on the first request, sleeps after 15 min idle and unloads per call. Used by other apps of mine (photo analysis, coding agent).
- LLM on RunPodA shortcut that creates the pod, downloads the model, opens the coding agent connected to it and deletes the pod when the window closes. A light version (4 GPUs, ~US$ 2.36/h) and a full version (756 GB) that only starts after I approve the cost in writing.
- 3D on RunPodComfyUI with photo-to-3D and video-to-3D-body models, behind a token lock.
- Music with vocals on RunPodA music model with its own screen; finished songs download to my computer automatically.
- All of themA fresh pod on every use, no persistent disk, deleted automatically when idle or when the terminal closes.
§3The hard parts
6 entries- 01
Compiling CUDA without sudo
The prebuilt llama.cpp releases for Linux don't ship CUDA. I assembled the toolkit from NVIDIA's standalone packages inside my user folder, solved hidden dependencies (libnvvm, a lib64 symlink, rpath) and copy the libraries next to the binary.
- 02
Tuning a 35B MoE model on an 8 GB card
All experts on the CPU, quantised context cache, speculative decoding and vision on the GPU. Measured: ~370 tok/s reading the prompt and ~44 tok/s generating, using 6.7 GB of VRAM (the Vulkan build generated at 35 tok/s).
- 03
Resilient GPU booking
The start script walks a list of GPUs in price order when RunPod says “no capacity”, filters by CUDA version read from the API schema and respects a price ceiling per hour.
- 04
Cost control
A watchdog inside the pod deletes it when idle or if the model never becomes ready; a warning before shutting down; a time limit chosen per session; a minimum-balance lock on the expensive model.
- 05
Several windows on the same pod
2 agents with separate contexts on a shared cache; the pod is only deleted when the last window closes. Tested with 2 conversations of 85k and 150k tokens.
- 06
Access lock
ComfyUI and the music generator have no password, so a Python layer (aiohttp/FastAPI) requires a token per pod.
§4Result
- tok/s generating, 35B MoE on 8 GB
- ~44
- tok/s reading the prompt
- ~370
- VRAM used
- 6.7 GB
The local server is in use: it replaced Ollama and serves my apps. The light LLM version came up in ~3 min with a 262,144-token context and 2 simultaneous conversations in the October 2026 test; the pod deleted itself after 10 min idle. The full LLM, 3D and music shortcuts have all run on real pods.
§5Stack
- llama.cpp (CUDA, self-compiled)
- vLLM
- ComfyUI
- FastAPI / aiohttp
- Bash
- Python
- RunPod REST and GraphQL APIs
- pm2
- GGUF