fix(context): align guidance with 64k minimum
This commit is contained in:
@@ -110,9 +110,9 @@ The `--cache-type-k q4_0 --cache-type-v q4_0` flags are the most important optim
|
||||
| q8_0 | ~8 GB |
|
||||
| **q4_0** | **~4 GB** |
|
||||
|
||||
On an 8 GB Mac, use `q4_0` KV cache and reduce context to `-c 32768` (32K). On 16 GB, you can comfortably do 128K context. On 32 GB+, you can run larger models or multiple parallel slots.
|
||||
On an 8 GB Mac, use `q4_0` KV cache and choose a smaller model that can still fit Hermes' 64K minimum context. On 16 GB, you can comfortably do 128K context. On 32 GB+, you can run larger models or multiple parallel slots.
|
||||
|
||||
If you're still running out of memory, reduce context size first (`-c`), then try a smaller quantization (Q3_K_M instead of Q4_K_M).
|
||||
If you're still running out of memory, reduce context only while staying at or above Hermes' 64K minimum; otherwise switch to a smaller model or smaller quantization (Q3_K_M instead of Q4_K_M).
|
||||
|
||||
### Test it
|
||||
|
||||
|
||||
@@ -156,19 +156,19 @@ Switch models on the fly inside a session:
|
||||
|
||||
### Increase Ollama's Context Window
|
||||
|
||||
By default, Ollama uses a 2048-token context. For agentic work (tool calls, long conversations), you need more:
|
||||
By default, Ollama uses a 2048-token context. Hermes requires at least 64,000 tokens for agentic work with tools:
|
||||
|
||||
```bash
|
||||
# Create a Modelfile that extends context
|
||||
cat > /tmp/Modelfile << 'EOF'
|
||||
FROM gemma4:31b
|
||||
PARAMETER num_ctx 16384
|
||||
PARAMETER num_ctx 64000
|
||||
EOF
|
||||
|
||||
ollama create gemma4-16k -f /tmp/Modelfile
|
||||
ollama create gemma4-64k -f /tmp/Modelfile
|
||||
```
|
||||
|
||||
Then update your Hermes config to use `gemma4-16k` as the model name.
|
||||
Then update your Hermes config to use `gemma4-64k` as the model name.
|
||||
|
||||
### Keep the Model Loaded
|
||||
|
||||
@@ -311,7 +311,7 @@ Your only cost is electricity — roughly $0.01–0.05 per session depending on
|
||||
## What's Better with Cloud Models
|
||||
|
||||
- **Very complex multi-step reasoning** — 70B+ or cloud models like Claude Opus are noticeably better
|
||||
- **Long context windows** — cloud models offer 100K–1M tokens; local models are typically 8K–32K
|
||||
- **Long context windows** — cloud models offer 100K–1M tokens; local runtimes often default below Hermes' 64K minimum unless you configure them
|
||||
- **Speed on large responses** — cloud inference is faster than CPU-only local for long generations
|
||||
|
||||
The sweet spot: use local for everyday tasks, set up a cloud fallback for the hard stuff.
|
||||
|
||||
Reference in New Issue
Block a user