HostYourAI, NL company offering full EU based open model hosting, both shared and single tenant/bespoke. might be a good way for my company and personally.

Comprehensive Guide: Local Agentic Coding Workflow & Model Selection

1. Core Workflow Architecture & Mechanics

• Hardware & VRAM Dynamics:

  • Local LLMs run on GPU VRAM (or Unified Memory on Apple Silicon).
  • Exceeding GPU memory overflows data into system RAM, dropping speeds significantly (e.g., from ~120 tokens/sec down to ~20 tokens/sec).
  • Parameter size and context window scale directly with VRAM consumption.

• Quantization & Optimization:

  • Quantization (such as Q4 4-bit) compresses model weights to reduce VRAM requirements by 50-75% with minimal accuracy loss.
  • Mixture of Experts (MoE) architectures load active parameter layers into VRAM while offloading inactive/less critical layers to CPU/RAM.

• Multi-Tiered Tool Infrastructure:

  • LM Studio (Inference Engine): Local runner exposing an OpenAI-compatible API endpoint (/v1) with fine-grained GPU offloading controls.
  • Continue Extension (VS Code): Handles rapid inline autocompletion using small, low-latency models (~1.5B parameters) with response times under 250ms.
  • GitHub Copilot / VS Code Insiders: Integrates local models via Custom OpenAI Endpoints for full agentic codebase modification.
  • Pi CLI / Qwen Code (Terminal Agent): Terminal-based open-source harness connecting to local endpoints for repository analysis, multi-file edits, and bug fixing.

2. Recommended Software & Models

• Recommended Software Stack:

  • Host / Engine: LM Studio (for GPU offloading control, quantization loading, and OpenAI API local serving).
  • Code Editor & Autocomplete: VS Code + Continue extension (autocomplete) and VS Code Insiders / GitHub Copilot (custom local agent setup).
  • Terminal Agent Harness: Pi (pi via pi.dev) or Qwen Code CLI for terminal-native, multi-file project execution.

• Recommended Models:

  • For Autocomplete: Qwen 2.5 Coder 1.5B (Q4/Q8). Requires ~1 GB VRAM, ensuring sub-250ms completion latency.
  • For Agentic Coding & Reasoning: Qwen 3.6 35B-A3B (MoE with 3B active parameters) or Qwen 2.5 Coder 32B / 14B (Q4 quantization). Chosen for native tool use (function calling), image/vision capability, and multi-step reasoning.

3. Local Workflow vs. Claude Code Comparison

• Cost & Execution Limits:

  • Local Setup: 100% Free & Unlimited. No per-token costs, API rate limits, or monthly tier restrictions after initial hardware acquisition.
  • Claude Code: Subscription & API-based. Subject to usage caps, monthly plan costs, or per-token API charges for Anthropic models.

• Privacy & Security:

  • Local Setup: Fully air-gapped and 100% private. Source code and context never leave your machine.
  • Claude Code: Cloud-dependent. Prompts, code context, and project files are sent to cloud servers for processing.

• Speed & Performance:

  • Local Setup: Hardware dependent. Fast on small models; larger 30B+ reasoning models run slower on consumer GPUs compared to cloud infrastructure.
  • Claude Code: High speed and throughput powered by managed cloud infrastructure.

• Ecosystem & Provider Flexibility:

  • Local Setup: Vendor Agnostic. Swap open-source models (Qwen, Llama, DeepSeek) seamlessly inside terminal harnesses or VS Code.
  • Claude Code: Locked Ecosystem. Exclusively tied to Anthropic Claude models and API platform.

• Architectural Reasoning:

  • Local Setup: Handles small-to-medium tasks and feature additions well, but smaller local models fall slightly short on massive multi-file refactoring compared to top-tier cloud models.
  • Claude Code: High-level architectural reasoning and multi-file refactoring capabilities out-of-the-box.

article explaining for three AI tasks, image labeling, image captioning and speech transcription, how to do them locally in browser (w a local webserver). The speech transcription used whisper as local model, I prefer Nvidia Parakeet for its multilingual capabilities. But the setup is interesting. It realistically describes on-device speeds (on M2 a 2 to 5x transcription vs real time. But you can deploy these as webworkers nicely it seems

via [[Stephen Downes p]]

Video on running an ollama coding assistant on Hetzner. Does not mention the fact it runs on a GEX44 at Hetzner, which starts at 200 Euro/month

mentions different ways of installing a local set-up for AI. Also mentions much cheeper options than Anthropic, specifically OpenRouter.ai. Does not mention running your own local install on a cloud VPS.

Comparison video of Claude Code using Anthropics cloud models vs local models on a M4 128GB. Still a heavy lift, fans spinning, memory usage almost at full capacity. But it works. Means that for my M1 16GB a smaller model is all that works, and you need to leave room for context loading too. For one-offs like code generation and for interactive in moving contexts there's different needs.

openclaw can detect presence of ollama if it is visible at this specific localhost address. basically if you have ollama running it will be detected. Meaning I could run openclaw fully locally.

the local models ollama recommends for use in claudecode - qwen3-code - glm-4.7 - gpt-oss:20b - gpt-oss:120b

ollama can also be used with claude code through their endpoint. This allows to use the open Chinese coding models like qwen3

one can run claude code using local models through the LM Studio endpoint. so that you don't use Claude in the cloud.

you run openclaw yourself. I think I saw [[Martijn Aslander p]] use it on a VPS yday.

vlam: Veilige lokale AI-modellen, pilot toepassing AI in NL overheid

vlam.ai pilots bij rijksoverheid, genoemd in [[Artificiële intelligentie vooral een bestuurlijke uitdaging - Digitale Overheid]]

this. Need to understand better conceptually the diff set-ups I have, and how I might switch between them.

Cloud models got much better stil than local models. Coding agents made a huge difference, with it Claude Code becomes very useful

Local models improved a lot in 2025. Mentions Llama 3.3 70B, Mistral Small 3, and the Chinese 20-30B parameter models.

LM Studio can run LLMs locally (I have llama and phi installed). It also has an API over a localhost webserver. I use that API to make llama available in Obsidian using the Copilot plugin.

This is the API documentation. #openvraag other scripts / [[Persoonlijke tools 20200619203600]] I can use this in?

https://web.archive.org/web/20241112124556/https://www.heise.de/news/Lokale-KI-Nemotron-70B-auf-vier-Mac-minis-M4-Pro-mit-Thunderbolt-5-Interconnect-10014248.html

Exolabs.net experiment running large LLMs locally on 4 combined Mac Mini's. Links to preview and github shared code. For 6600-9360 you can run a cluster of 4 Minis locally. Affordable for SME outfits.