Choosing local LLMs for a two-model pipeline

2.8 GB at Q4_K_S. Hybrid Mamba-Transformer architecture - 42 layers total: 21 Mamba-2, 4 Attention, 17 MLP ^[1]. Compressed from Nemotron-Nano-9B-v2 using NVIDIA's Elastic framework - router-guided structured pruning across Mamba heads, hidden dimensions, FFN channels, and depth, followed by two-stage distillation on 63B+ tokens ^[1]. Not trained from scratch at 4B.

State-of-the-art instruction following for its size class on IFBench and IFEval. Strong tool-use performance. Competitive hallucination avoidance. NVIDIA's own GGUF release is Q4_K_M at 2.84 GB, benchmarked at 18 tokens/second on a Jetson Orin Nano 8GB ^[2]. On a desktop GPU with headroom, it's considerably faster.

This model exists specifically for the job we're giving it. NVIDIA built it for edge deployment and agentic routing - their published use cases include local voice assistants and AI game NPCs ^[2]. We're using it for the routing half: classifying prompts, extracting structured fields, and checking the 30B's output.

31.6B total parameters, but only 3.2B active per token - 128 experts, 6 consulted per request, the rest stay quiet. At Q4_K_S it's ~22 GB. That sounds like 10 GB of headroom on a 32 GB card - until you account for the KV caches. Both models' attention state lives in VRAM alongside the weights.

The MoE architecture is why this works. A dense 32B model like QwQ-32B activates all 32 billion parameters on every token. This model activates 3.2 billion. The generation speed is closer to a small model than a large one, but the reasoning quality comes from having 30B+ parameters available for specialization. Trained on 25 trillion tokens with multi-environment reinforcement learning via synchronous GRPO.

NVIDIA reports up to 3.3x higher throughput than Qwen3-30B-A3B and 2.2x higher than GPT-OSS-20B. It outperforms both on standard benchmarks. Supports 1M token context if you need it - though you shouldn't, for reasons covered below.

Same MoE trick - 30B total, ~3B active - but at 17.5 GB (Q4_K_S), leaving more VRAM headroom. Strong on code generation and structured output. When the pipeline is code-heavy, this is the better pick for the workhorse slot.

QwQ-32B (19.9 GB, dense) and Gemma-3-27B (16.4 GB, dense) both produce good output. Both are noticeably slower at generation because every parameter is active on every token. For interactive inference where the user is watching tokens stream in, that speed difference is the difference between "responsive" and "waiting." MoE models with comparable quality and 3x the throughput make dense models hard to justify for this use case.

For classification, give closed-form tasks with explicit valid outputs. "Classify into exactly one: CODE, REASONING, SIMPLE, CONTEXT_LOOKUP. Respond with only the category name." The model can't hallucinate a fifth category if the grammar constraint won't let it.

GBNF grammar constraints in LM Studio enforce this at the token level - regex-strength enforcement over the output space. The model physically cannot produce invalid output.

JSON schema with grammar constraints eliminates hallucination in extraction tasks. The model produces a fixed structure and nothing else. It cannot invent fields or omit required fields.

{
                    "intent": "...",
                    "key_entities": [...],
                    "complexity": "low|medium|high"
                    }

When the 4B decides an inference result should become a persistent context entry - a piece of knowledge the system remembers - validate it. Does the generated title relate to the source content? Is the classification consistent with the extracted entities? Agent-generated entries start at moderate confidence (0.50). The system's retrieval scoring naturally deprioritizes entries that don't corroborate with anything else.

Temperature 0.0 for all classification and routing calls. Max tokens inside the budgets above: 128-256 for classification, 512-1024 for structured extraction. Reasoning mode off. Each model stays in its lane. The 4B classifies, extracts, and routes. The 30B generates. The 4B validates the 30B's output.

The router runs at Q8_0. The workhorse runs at Q4_K_S. These aren't the same model at different sizes - they're different models with different jobs - but even if they were, you'd still want different quantization per role.

The router's job is fragile - like a switchboard operator. It doesn't need to be eloquent, but it absolutely cannot connect you to the wrong line. Picking from an enum of four categories, extracting structured JSON, deciding whether to short-circuit - these tasks live on the margin between correct and wrong. A single flipped token in a classification response sends garbage downstream. Higher precision (Q8_0, ~4 GB for a 4B model) costs almost nothing in VRAM and eliminates quantization-induced errors on the component where errors are most expensive.

The workhorse's job is tolerant. Generating a multi-paragraph response with 30B+ parameters backing it has natural redundancy - the model can route around minor quantization artifacts through sheer capacity. Running it at Q4_K_S (~22 GB) leaves room for the router's Q8_0 (~4 GB) plus both models' KV caches within 32 GB. That allocation spends precision where the task is brittle and compression where the task is robust.

The alternative - switching quantization per request - doesn't exist. GGUF weights are loaded at server startup. Changing quantization means unloading the model, loading a different weight file, and reinitializing. On a 22 GB model, that's seconds of downtime in a pipeline designed to add 100-300ms. Each instance is tuned - context length, max tokens, quantization, temperature - for exactly one job.

LM Studio supports multiple loaded models simultaneously - you can have the 4B router and the 30B workhorse loaded in the same instance and route API calls to each by model name. That's enough for a single-machine pipeline.

LM Link takes it further. It connects multiple machines running LM Studio into a mesh VPN - models loaded on remote devices appear as if they're local, accessible through the same localhost:1234 endpoint. Any tool that already connects to your local LM Studio server works with remote models transparently. In our case, the desktop's RTX 5090 runs the 30B workhorse while a secondary machine with an RTX 4090 Mobile handles the router - dedicated silicon for each role, no VRAM contention, end-to-end encrypted over Tailscale.

For a pipeline that's meant to be invisible, this is the right architecture. The router runs on its own GPU with its own tuned context window and quantization. The workhorse gets the full 32 GB of the 5090. Neither model competes for VRAM, and the API surface is identical to a single-machine setup.