fix(context): align guidance with 64k minimum

This commit is contained in:
helix4u
2026-05-24 23:23:12 -07:00
committed by Teknium
parent 1d5deac346
commit 3b839f4369
6 changed files with 41 additions and 35 deletions
+2 -2
View File
@@ -110,9 +110,9 @@ The `--cache-type-k q4_0 --cache-type-v q4_0` flags are the most important optim
| q8_0 | ~8 GB |
| **q4_0** | **~4 GB** |
On an 8 GB Mac, use `q4_0` KV cache and reduce context to `-c 32768` (32K). On 16 GB, you can comfortably do 128K context. On 32 GB+, you can run larger models or multiple parallel slots.
On an 8 GB Mac, use `q4_0` KV cache and choose a smaller model that can still fit Hermes' 64K minimum context. On 16 GB, you can comfortably do 128K context. On 32 GB+, you can run larger models or multiple parallel slots.
If you're still running out of memory, reduce context size first (`-c`), then try a smaller quantization (Q3_K_M instead of Q4_K_M).
If you're still running out of memory, reduce context only while staying at or above Hermes' 64K minimum; otherwise switch to a smaller model or smaller quantization (Q3_K_M instead of Q4_K_M).
### Test it
+5 -5
View File
@@ -156,19 +156,19 @@ Switch models on the fly inside a session:
### Increase Ollama's Context Window
By default, Ollama uses a 2048-token context. For agentic work (tool calls, long conversations), you need more:
By default, Ollama uses a 2048-token context. Hermes requires at least 64,000 tokens for agentic work with tools:
```bash
# Create a Modelfile that extends context
cat > /tmp/Modelfile << 'EOF'
FROM gemma4:31b
PARAMETER num_ctx 16384
PARAMETER num_ctx 64000
EOF
ollama create gemma4-16k -f /tmp/Modelfile
ollama create gemma4-64k -f /tmp/Modelfile
```
Then update your Hermes config to use `gemma4-16k` as the model name.
Then update your Hermes config to use `gemma4-64k` as the model name.
### Keep the Model Loaded
@@ -311,7 +311,7 @@ Your only cost is electricity — roughly $0.010.05 per session depending on
## What's Better with Cloud Models
- **Very complex multi-step reasoning** — 70B+ or cloud models like Claude Opus are noticeably better
- **Long context windows** — cloud models offer 100K1M tokens; local models are typically 8K32K
- **Long context windows** — cloud models offer 100K1M tokens; local runtimes often default below Hermes' 64K minimum unless you configure them
- **Speed on large responses** — cloud inference is faster than CPU-only local for long generations
The sweet spot: use local for everyday tasks, set up a cloud fallback for the hard stuff.