Speculative Decoding
That's a thing. So, I am using still no GPU being short of 800 bugs for a appropriate NVIDIA card and that means that even with 64GB RAM I won't be able to run certain model architectures by data transfer bottle necks.
(DeepSeek AI)
You're right to focus on this — MoE (Mixture of Experts) models are a completely different beast on a CPU-only machine, and understanding why will save you a lot of frustration.
The core issue is that **MoE models are fundamentally memory-bandwidth-bound, and a CPU's memory bandwidth is its weakest link** for this workload.
### 🧠The MoE Paradox: Sparse Compute, Dense Memory Traffic
The appeal of MoE is that it activates only a small fraction of its total parameters for each token. For example, a 30B MoE model might only use 3B active parameters per token. On paper, this sounds perfect for a CPU: less compute per token.
However, the **memory access pattern is the opposite of sparse**. Every token must still be routed through the model's full architecture, and the router must load and evaluate the expert weights to decide which ones to activate. The key problem is that the expert weights for *all* experts must be resident in memory — they can't be loaded from disk on demand without catastrophic latency.
This is why MoE models are so memory-hungry. A 30B MoE model with 3B active parameters still needs ~18–22 GB of RAM for its weights in 4-bit quantization. That's because the *total* parameter count determines memory capacity, while the *active* parameter count determines compute.
### 🚧 The CPU Memory Bandwidth Bottleneck
Here's the critical constraint: **CPU RAM bandwidth is roughly 10–20x lower than GPU VRAM bandwidth**. When you run an MoE model entirely on CPU, every token generation requires streaming the active expert weights from system RAM through the CPU's memory bus.
For a 3B-active MoE model, you're moving ~3 GB of weights per token. If your CPU's memory bandwidth is, say, 50 GB/s, that's a theoretical maximum of ~16 tokens/second — before accounting for attention, routing overhead, and other memory traffic. In practice, you'll see far less.
Research on this exact problem confirms the severity: one study found that MoE models exhibit **higher inference latency and lower throughput than dense baselines on both GPU and CPU**, precisely because the computational savings from skipping inactive experts are **offset by the overhead introduced by conditional routing**.
### 🔄 The CPU-GPU Hybrid Trap (And Why Pure CPU Is Even Worse)
Most MoE deployments use a **hybrid approach**: the GPU handles the dense attention and shared layers, while the expert weights live in CPU RAM and are streamed to the GPU on demand. This works because the GPU's high bandwidth compensates for the PCIe transfer overhead.
But on a **pure CPU machine**, there is no high-bandwidth staging area. The CPU must both compute *and* fetch weights from the same memory pool. This is why pure CPU MoE inference is notoriously slow: one benchmark showed a Phi-3.5 MoE model dropping from **13.8 tokens/second (GPU) to 4.5 tokens/second (CPU-only)** — a 3.1x slowdown.
### 📋 What This Means for You (64 GB CPU-Only)
With 64 GB of RAM, you *can* fit many MoE models in memory. But the token generation speed will be **poorly interactive**. Your 32B dense model with speculative decoding is actually a smarter choice for CPU inference because:
- **Dense models have predictable, sequential memory access patterns** that CPUs handle well.
- **Speculative decoding (your 1.5B draft model) works brilliantly on CPU** because the draft model is small enough to run in cache, and the verification pass is batched.
- **No routing overhead** — every token uses the same weights, so the CPU's prefetcher can work effectively.
### ✅ What to Use Instead
Given your 64 GB CPU-only setup, here's what works well:
#### 1. **Dense Models (Your Current Approach)**
Your Qwen2.5-Coder-32B in Q5_K_M quantization is essentially the sweet spot. It's large enough to be highly capable, but dense architecture means predictable memory access. With speculative decoding, you're getting the best possible CPU performance.
#### 2. **CPU-Optimized MoE Models (If You Want to Try)**
If you want to experiment with MoE, look for models specifically designed for CPU inference. The standout is **Maple Preview 20B-A1B** (ternary 2-bit quantization). It's a 20B MoE with only 1B active parameters, and it runs at **28–34 tokens/second on a 4-core laptop CPU**. That's because the ternary quantization is so aggressive that the entire model fits in a 5.5 GB file, and the memory traffic per token is minimal.
#### 3. **LiquidAI LFM2.5 Family**
The **LFM2.5-8B-A1B** is an MoE model with 8.3B total and 1.5B active parameters, designed for efficient on-device inference. It's a good middle ground if you want MoE architecture but need CPU-friendly performance.
#### 4. **Small Dense Models (For Speed)**
If raw speed matters more than quality, small dense models like **Gemma 3 (1B)**, **DeepSeek R1 (1.5B)**, or **Llama 3.2 (1B/3B)** run comfortably on CPU. But you'll lose the coding capability of your 32B model.
### 💡 The Bottom Line
| Model Type | CPU Suitability | Why |
|---|---|---|
Dense 32B + Spec. Dec. ✅ Excellent
The core issue is that **MoE models are fundamentally memory-bandwidth-bound, and a CPU's memory bandwidth is its weakest link** for this workload.
### 🧠The MoE Paradox: Sparse Compute, Dense Memory Traffic
The appeal of MoE is that it activates only a small fraction of its total parameters for each token. For example, a 30B MoE model might only use 3B active parameters per token. On paper, this sounds perfect for a CPU: less compute per token.
However, the **memory access pattern is the opposite of sparse**. Every token must still be routed through the model's full architecture, and the router must load and evaluate the expert weights to decide which ones to activate. The key problem is that the expert weights for *all* experts must be resident in memory — they can't be loaded from disk on demand without catastrophic latency.
This is why MoE models are so memory-hungry. A 30B MoE model with 3B active parameters still needs ~18–22 GB of RAM for its weights in 4-bit quantization. That's because the *total* parameter count determines memory capacity, while the *active* parameter count determines compute.
### 🚧 The CPU Memory Bandwidth Bottleneck
Here's the critical constraint: **CPU RAM bandwidth is roughly 10–20x lower than GPU VRAM bandwidth**. When you run an MoE model entirely on CPU, every token generation requires streaming the active expert weights from system RAM through the CPU's memory bus.
For a 3B-active MoE model, you're moving ~3 GB of weights per token. If your CPU's memory bandwidth is, say, 50 GB/s, that's a theoretical maximum of ~16 tokens/second — before accounting for attention, routing overhead, and other memory traffic. In practice, you'll see far less.
Research on this exact problem confirms the severity: one study found that MoE models exhibit **higher inference latency and lower throughput than dense baselines on both GPU and CPU**, precisely because the computational savings from skipping inactive experts are **offset by the overhead introduced by conditional routing**.
### 🔄 The CPU-GPU Hybrid Trap (And Why Pure CPU Is Even Worse)
Most MoE deployments use a **hybrid approach**: the GPU handles the dense attention and shared layers, while the expert weights live in CPU RAM and are streamed to the GPU on demand. This works because the GPU's high bandwidth compensates for the PCIe transfer overhead.
But on a **pure CPU machine**, there is no high-bandwidth staging area. The CPU must both compute *and* fetch weights from the same memory pool. This is why pure CPU MoE inference is notoriously slow: one benchmark showed a Phi-3.5 MoE model dropping from **13.8 tokens/second (GPU) to 4.5 tokens/second (CPU-only)** — a 3.1x slowdown.
### 📋 What This Means for You (64 GB CPU-Only)
With 64 GB of RAM, you *can* fit many MoE models in memory. But the token generation speed will be **poorly interactive**. Your 32B dense model with speculative decoding is actually a smarter choice for CPU inference because:
- **Dense models have predictable, sequential memory access patterns** that CPUs handle well.
- **Speculative decoding (your 1.5B draft model) works brilliantly on CPU** because the draft model is small enough to run in cache, and the verification pass is batched.
- **No routing overhead** — every token uses the same weights, so the CPU's prefetcher can work effectively.
### ✅ What to Use Instead
Given your 64 GB CPU-only setup, here's what works well:
#### 1. **Dense Models (Your Current Approach)**
Your Qwen2.5-Coder-32B in Q5_K_M quantization is essentially the sweet spot. It's large enough to be highly capable, but dense architecture means predictable memory access. With speculative decoding, you're getting the best possible CPU performance.
#### 2. **CPU-Optimized MoE Models (If You Want to Try)**
If you want to experiment with MoE, look for models specifically designed for CPU inference. The standout is **Maple Preview 20B-A1B** (ternary 2-bit quantization). It's a 20B MoE with only 1B active parameters, and it runs at **28–34 tokens/second on a 4-core laptop CPU**. That's because the ternary quantization is so aggressive that the entire model fits in a 5.5 GB file, and the memory traffic per token is minimal.
#### 3. **LiquidAI LFM2.5 Family**
The **LFM2.5-8B-A1B** is an MoE model with 8.3B total and 1.5B active parameters, designed for efficient on-device inference. It's a good middle ground if you want MoE architecture but need CPU-friendly performance.
#### 4. **Small Dense Models (For Speed)**
If raw speed matters more than quality, small dense models like **Gemma 3 (1B)**, **DeepSeek R1 (1.5B)**, or **Llama 3.2 (1B/3B)** run comfortably on CPU. But you'll lose the coding capability of your 32B model.
### 💡 The Bottom Line
| Model Type | CPU Suitability | Why |
|---|---|---|
Dense 32B + Spec. Dec. ✅ Excellent
Predictable memory access, batch verification
CPU-optimized MoE (Maple, LFM2.5) ✅ Good
CPU-optimized MoE (Maple, LFM2.5) ✅ Good
Aggressive quantization, small active params
Standard MoE (Qwen3-30B, DeepSeek) ❌ Poor
Standard MoE (Qwen3-30B, DeepSeek) ❌ Poor
Routing overhead, high memory traffic per token
Small dense (1B-3B)** ✅ Fast
Small dense (1B-3B)** ✅ Fast
Low memory footprint, but lower quality
Your current setup — a dense 32B model with a 1.5B speculative decoding draft — is actually the **optimal configuration for a CPU-only machine**. You get the quality of a large dense model with the speed boost of speculative decoding, without the memory bandwidth penalties that plague MoE architectures.
If you want to experiment with MoE later, start with **Maple Preview 20B-A1B** — it's the only MoE I've seen that genuinely runs well on pure CPU, and it's small enough to test without disrupting your current setup.
Your current setup — a dense 32B model with a 1.5B speculative decoding draft — is actually the **optimal configuration for a CPU-only machine**. You get the quality of a large dense model with the speed boost of speculative decoding, without the memory bandwidth penalties that plague MoE architectures.
If you want to experiment with MoE later, start with **Maple Preview 20B-A1B** — it's the only MoE I've seen that genuinely runs well on pure CPU, and it's small enough to test without disrupting your current setup.
--
With Speculative Decoding I add on top of being capable of using a large model quite some speed...
(DeepSeek)
Here's a summary of what speculative decoding gives you, compared to the standard Ollama + Llama 3 setup you were using before.
### ⚙️ The Core Mechanism: How It Actually Works
The fundamental bottleneck with local LLMs is **memory bandwidth**. Generating a single token requires a full forward pass through every model weight, and successive tokens cannot be parallelized — each one depends on the last.
Speculative decoding breaks that dependency chain by pairing a **small draft model** with your **large target model**:
1. The draft model quickly guesses the next several tokens (your 1.5B model).
2. The large target model (your 32B) verifies *all* those guesses in a **single parallel forward pass**, because every position's inputs are already known.
3. The target model accepts the prefix of guesses it agrees with and replaces the first rejection with its own prediction. The process repeats.
The critical guarantee: the accept/reject rule is mathematically constructed so the final output distribution is **identical** to what the large model would have produced alone. You get no quality loss.
### 📊 What You Gain (and When)
The speedup depends entirely on the **acceptance rate** — how often the draft model's guesses match what the 32B model would have chosen. Your logs earlier showed **100% acceptance** on a simple prompt (`draft_n: 10, draft_n_accepted: 10`), which is the best-case scenario. In practice:
* **On CPU (your setup):** Benchmarks show **2–3× speedups** are typical for large targets, because the draft model is genuinely tiny relative to the 32B per-token cost.
* **Real-world example:** A CPU-only test with a 3B target went from 12.9 tok/s to 22.1 tok/s — a **1.72× overall speedup**, peaking at 2.03× on math tasks.
* **When it backfires:** If the draft model is poorly aligned with the target (low acceptance rate), or the target model is already small and fast, the overhead of running two models can make things **slower**, not faster. One benchmark saw a 7B target actually **regress** on a fast GPU.
In your specific case — a 32B model on CPU with a 1.5B draft — the conditions are ideal. The draft is small enough to be nearly free, and the 32B target is slow enough that batching verification across 5 drafted tokens saves substantial memory-bandwidth round-trips.
### 🆚 Speculative Decoding vs. Ollama + Llama 3
Here's the key difference: **Ollama does not expose speculative decoding the way llama.cpp does.**
| Aspect
### ⚙️ The Core Mechanism: How It Actually Works
The fundamental bottleneck with local LLMs is **memory bandwidth**. Generating a single token requires a full forward pass through every model weight, and successive tokens cannot be parallelized — each one depends on the last.
Speculative decoding breaks that dependency chain by pairing a **small draft model** with your **large target model**:
1. The draft model quickly guesses the next several tokens (your 1.5B model).
2. The large target model (your 32B) verifies *all* those guesses in a **single parallel forward pass**, because every position's inputs are already known.
3. The target model accepts the prefix of guesses it agrees with and replaces the first rejection with its own prediction. The process repeats.
The critical guarantee: the accept/reject rule is mathematically constructed so the final output distribution is **identical** to what the large model would have produced alone. You get no quality loss.
### 📊 What You Gain (and When)
The speedup depends entirely on the **acceptance rate** — how often the draft model's guesses match what the 32B model would have chosen. Your logs earlier showed **100% acceptance** on a simple prompt (`draft_n: 10, draft_n_accepted: 10`), which is the best-case scenario. In practice:
* **On CPU (your setup):** Benchmarks show **2–3× speedups** are typical for large targets, because the draft model is genuinely tiny relative to the 32B per-token cost.
* **Real-world example:** A CPU-only test with a 3B target went from 12.9 tok/s to 22.1 tok/s — a **1.72× overall speedup**, peaking at 2.03× on math tasks.
* **When it backfires:** If the draft model is poorly aligned with the target (low acceptance rate), or the target model is already small and fast, the overhead of running two models can make things **slower**, not faster. One benchmark saw a 7B target actually **regress** on a fast GPU.
In your specific case — a 32B model on CPU with a 1.5B draft — the conditions are ideal. The draft is small enough to be nearly free, and the 32B target is slow enough that batching verification across 5 drafted tokens saves substantial memory-bandwidth round-trips.
### 🆚 Speculative Decoding vs. Ollama + Llama 3
Here's the key difference: **Ollama does not expose speculative decoding the way llama.cpp does.**
| Aspect
| Ollama + Llama 3 | Your llama.cpp + Spec. Decoding
| :--- | :--- | :--- |
Model quality
| :--- | :--- | :--- |
Model quality
Llama 3 (8B, 70B, etc.) | Qwen2.5-Coder 32B
Draft model
None — single model | Qwen2.5-Coder 1.5B
Speedup mechanism
Speedup mechanism
Standard autoregressive | Draft-then-verify, ~2–3× on CPU |
Speed control
Speed control
`num_thread`, GPU layers | `-md`, `--draft-max`, `--draft-min`
Output quality
Output quality
Fixed to model | **Identical** to 32B alone
Ollama has only recently begun adding speculative decoding, and only for specific model families like Gemma 4 via MTP (multi-token prediction). For Qwen models on CPU, Ollama gives you **no draft-model acceleration** — it runs the 32B model straight through, token by token.
Your llama.cpp setup, by contrast, is doing exactly the thing Ollama doesn't: it's using a **smaller companion model to draft tokens**, then verifying them in parallel. That's why your 32B model is usable at interactive speeds on CPU at all.
### 💡 The Practical Takeaway
* **Ollama + Llama 3** is a single-model system. Simple, but no draft-model trick — your 32B model would run at baseline CPU speed (roughly 2.5 tok/s in your earlier logs without spec. dec.).
* **Your llama.cpp + 1.5B draft** is a two-model system. The 1.5B drafts tokens cheaply; the 32B verifies them in batches. On a good acceptance rate, you're effectively getting several tokens per 32B forward pass instead of one.
The cost is **memory**: you're holding both models in RAM simultaneously (the 32B at ~22 GB, the 1.5B at ~1.2 GB). But on a machine with 64 GB, that's a worthwhile trade for a 2–3× speedup.
If you ever test Ollama with a Llama 3 8B model on the same hardware, you'll see it's faster *per token* simply because the model is smaller — but the quality is lower. Speculative decoding lets you keep the 32B's quality while clawing back some of that speed.
Ollama has only recently begun adding speculative decoding, and only for specific model families like Gemma 4 via MTP (multi-token prediction). For Qwen models on CPU, Ollama gives you **no draft-model acceleration** — it runs the 32B model straight through, token by token.
Your llama.cpp setup, by contrast, is doing exactly the thing Ollama doesn't: it's using a **smaller companion model to draft tokens**, then verifying them in parallel. That's why your 32B model is usable at interactive speeds on CPU at all.
### 💡 The Practical Takeaway
* **Ollama + Llama 3** is a single-model system. Simple, but no draft-model trick — your 32B model would run at baseline CPU speed (roughly 2.5 tok/s in your earlier logs without spec. dec.).
* **Your llama.cpp + 1.5B draft** is a two-model system. The 1.5B drafts tokens cheaply; the 32B verifies them in batches. On a good acceptance rate, you're effectively getting several tokens per 32B forward pass instead of one.
The cost is **memory**: you're holding both models in RAM simultaneously (the 32B at ~22 GB, the 1.5B at ~1.2 GB). But on a machine with 64 GB, that's a worthwhile trade for a 2–3× speedup.
If you ever test Ollama with a Llama 3 8B model on the same hardware, you'll see it's faster *per token* simply because the model is smaller — but the quality is lower. Speculative decoding lets you keep the 32B's quality while clawing back some of that speed.
--
That means my Cortex version has a brain now. ...here.
The Cortex Version has a large model that will do batch process analytics of the several small LLM databases and data vaults. It will run most of the time as a background process and save its result in .txt or .md files.
The stand alone version is ready to be build.
The Nexus Version is coming alive at the same time as the Cortex Version. It uses the Cluster hosting several models over its nodes, having no main large model. It's load balancer worked in a first test run and triggered some Architecture thoughts.
The Cyberdeck AI Knowledge Management System. here...
#opensource #cyberpunkcoltoure
