Merge main into bb/gui.
Resolve merge conflicts while preserving bb/gui dashboard paths and STT provider support.
This commit is contained in:
@@ -24,12 +24,15 @@ Key implementation files:
|
||||
```text
|
||||
hermes acp / hermes-acp / python -m acp_adapter
|
||||
-> acp_adapter.entry.main()
|
||||
-> parse --version / --check / --setup before server startup
|
||||
-> load ~/.hermes/.env
|
||||
-> configure stderr logging
|
||||
-> construct HermesACPAgent
|
||||
-> acp.run_agent(agent, use_unstable_protocol=True)
|
||||
```
|
||||
|
||||
The Zed ACP Registry path launches the same adapter through `uvx --from 'hermes-agent[acp]==<version>' hermes-acp`, pointed at the `hermes-agent` PyPI release.
|
||||
|
||||
Stdout is reserved for ACP JSON-RPC transport. Human-readable logs go to stderr.
|
||||
|
||||
## Major components
|
||||
@@ -146,7 +149,7 @@ Instead it reuses Hermes' runtime resolver:
|
||||
- `acp_adapter/auth.py`
|
||||
- `hermes_cli/runtime_provider.py`
|
||||
|
||||
So ACP advertises and uses the currently configured Hermes provider/credentials.
|
||||
So ACP advertises and uses the currently configured Hermes provider/credentials. It also always advertises a terminal setup auth method (`hermes-setup`, args `--setup`) so first-run registry clients can open Hermes' interactive model/provider configuration before starting a normal ACP session.
|
||||
|
||||
## Working directory binding
|
||||
|
||||
|
||||
@@ -182,6 +182,7 @@ When you call `ctx.register_platform()`, the following integration points are ha
|
||||
| Connected platform validation | Registry `validate_config()` called |
|
||||
| User authorization | `allowed_users_env` / `allow_all_env` checked |
|
||||
| Env-only auto-enable | `env_enablement_fn` seeds `PlatformConfig.extra` + `home_channel` |
|
||||
| YAML config bridge | `apply_yaml_config_fn` translates `config.yaml` keys into env vars / extras |
|
||||
| Cron delivery | `cron_deliver_env_var` makes `deliver=<name>` work |
|
||||
| `hermes config` UI entries | `requires_env` / `optional_env` in `plugin.yaml` auto-populate |
|
||||
| send_message tool | Routes through live gateway adapter |
|
||||
@@ -239,6 +240,46 @@ def register(ctx):
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## YAML→env Config Bridge
|
||||
|
||||
Some users prefer setting `config.yaml` keys (`my_platform.require_mention`, `my_platform.allowed_channels`, etc.) over env vars. The `apply_yaml_config_fn` hook lets your plugin own this translation instead of forcing core `gateway/config.py` to know your platform's YAML schema.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
def _apply_yaml_config(yaml_cfg: dict, platform_cfg: dict) -> dict | None:
|
||||
"""Translate config.yaml `my_platform:` keys into env vars / extras.
|
||||
|
||||
yaml_cfg — the full top-level parsed config.yaml dict
|
||||
platform_cfg — the platform's own sub-dict (yaml_cfg.get("my_platform", {}))
|
||||
|
||||
May mutate os.environ directly (use `not os.getenv(...)` guards to
|
||||
preserve env > YAML precedence) and/or return a dict to merge into
|
||||
PlatformConfig.extra. Return None or {} for no extras.
|
||||
"""
|
||||
if "require_mention" in platform_cfg and not os.getenv("MY_PLATFORM_REQUIRE_MENTION"):
|
||||
os.environ["MY_PLATFORM_REQUIRE_MENTION"] = str(platform_cfg["require_mention"]).lower()
|
||||
allowed = platform_cfg.get("allowed_channels")
|
||||
if allowed is not None and not os.getenv("MY_PLATFORM_ALLOWED_CHANNELS"):
|
||||
if isinstance(allowed, list):
|
||||
allowed = ",".join(str(v) for v in allowed)
|
||||
os.environ["MY_PLATFORM_ALLOWED_CHANNELS"] = str(allowed)
|
||||
return None # nothing extra to merge into PlatformConfig.extra
|
||||
|
||||
def register(ctx):
|
||||
ctx.register_platform(
|
||||
name="my_platform",
|
||||
...,
|
||||
apply_yaml_config_fn=_apply_yaml_config,
|
||||
)
|
||||
```
|
||||
|
||||
The hook is invoked during `load_gateway_config()` after the generic shared-key loop (which handles common keys like `unauthorized_dm_behavior`, `notice_delivery`, `reply_prefix`, `require_mention`, etc.) and before `_apply_env_overrides()`, so your plugin only needs to bridge **platform-specific** keys.
|
||||
|
||||
Exceptions raised by the hook are swallowed and logged at debug level — a misbehaving plugin never aborts gateway config load.
|
||||
|
||||
|
||||
## Cron Delivery
|
||||
|
||||
To let `deliver=my_platform` cron jobs route to a configured home channel, set `cron_deliver_env_var` to the env var name that holds the default chat/room/channel ID:
|
||||
|
||||
@@ -127,7 +127,6 @@ hermes-agent/
|
||||
├── cron/ # Scheduler (jobs.py, scheduler.py)
|
||||
├── plugins/memory/ # Memory provider plugins
|
||||
├── plugins/context_engine/ # Context engine plugins
|
||||
├── environments/ # RL training environments (Atropos)
|
||||
├── skills/ # Bundled skills (always available)
|
||||
├── optional-skills/ # Official optional skills (install explicitly)
|
||||
├── website/ # Docusaurus documentation site
|
||||
@@ -185,7 +184,6 @@ If you are new to the codebase:
|
||||
8. **[Gateway Internals](./gateway-internals.md)** — messaging platform gateway
|
||||
9. **[Context Compression & Prompt Caching](./context-compression-and-caching.md)** — compression and caching
|
||||
10. **[ACP Internals](./acp-internals.md)** — IDE integration
|
||||
11. **[Environments, Benchmarks & Data Generation](./environments.md)** — RL training
|
||||
|
||||
## Major Subsystems
|
||||
|
||||
@@ -247,11 +245,11 @@ Exposes Hermes as an editor-native agent over stdio/JSON-RPC for VS Code, Zed, a
|
||||
|
||||
→ [ACP Internals](./acp-internals.md)
|
||||
|
||||
### RL / Environments / Trajectories
|
||||
### Trajectories
|
||||
|
||||
Full environment framework for evaluation and RL training. Integrates with Atropos, supports multiple tool-call parsers, and generates ShareGPT-format trajectories.
|
||||
Generates ShareGPT-format trajectories from agent sessions for training data generation.
|
||||
|
||||
→ [Environments, Benchmarks & Data Generation](./environments.md), [Trajectories & Training Format](./trajectory-format.md)
|
||||
→ [Trajectories & Training Format](./trajectory-format.md)
|
||||
|
||||
## Design Principles
|
||||
|
||||
|
||||
@@ -50,9 +50,6 @@ export VIRTUAL_ENV="$(pwd)/venv"
|
||||
|
||||
# Install with all extras (messaging, cron, CLI menus, dev tools)
|
||||
uv pip install -e ".[all,dev]"
|
||||
# tinker-atropos is a git submodule — needs `git submodule update --init` first
|
||||
# if you didn't clone with `--recurse-submodules`
|
||||
uv pip install -e "./tinker-atropos"
|
||||
|
||||
# Optional: browser tools
|
||||
npm install
|
||||
|
||||
@@ -360,7 +360,7 @@ All hub-installed skills go through a security scanner that checks for:
|
||||
Trust levels:
|
||||
- `builtin` — ships with Hermes (always trusted)
|
||||
- `official` — from `optional-skills/` in the repo (builtin trust, no third-party warning)
|
||||
- `trusted` — from openai/skills, anthropics/skills
|
||||
- `trusted` — from openai/skills, anthropics/skills, huggingface/skills
|
||||
- `community` — non-dangerous findings can be overridden with `--force`; `dangerous` verdicts remain blocked
|
||||
|
||||
Hermes can now consume third-party skills from multiple external discovery models:
|
||||
|
||||
@@ -1,520 +0,0 @@
|
||||
---
|
||||
sidebar_position: 5
|
||||
title: "Environments, Benchmarks & Data Generation"
|
||||
description: "Building RL training environments, running evaluation benchmarks, and generating SFT data with the Hermes-Agent Atropos integration"
|
||||
---
|
||||
|
||||
# Environments, Benchmarks & Data Generation
|
||||
|
||||
Hermes Agent includes a full environment framework that connects its tool-calling capabilities to the [Atropos](https://github.com/NousResearch/atropos) RL training framework. This enables three workflows:
|
||||
|
||||
1. **RL Training** — Train language models on multi-turn agentic tasks with GRPO
|
||||
2. **Benchmarks** — Evaluate models on standardised agentic benchmarks
|
||||
3. **Data Generation** — Generate SFT training data from agent rollouts
|
||||
|
||||
All three share the same core: an **environment** class that defines tasks, runs an agent loop, and scores the output.
|
||||
|
||||
:::info Repo environments vs RL training tools
|
||||
The Python environment framework documented here lives under the repo's `environments/` directory and is the implementation-level API for Hermes/Atropos integration. This is separate from the user-facing `rl_*` tools, which operate as an orchestration surface for remote RL training workflows.
|
||||
:::
|
||||
|
||||
:::tip Quick Links
|
||||
- **Want to run benchmarks?** Jump to [Available Benchmarks](#available-benchmarks)
|
||||
- **Want to train with RL?** See [RL Training Tools](/user-guide/features/rl-training) for the agent-driven interface, or [Running Environments](#running-environments) for manual execution
|
||||
- **Want to create a new environment?** See [Creating Environments](#creating-environments)
|
||||
:::
|
||||
|
||||
## Architecture
|
||||
|
||||
The environment system is built on a three-layer inheritance chain:
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class BaseEnv {
|
||||
Server management
|
||||
Worker scheduling
|
||||
Wandb logging
|
||||
CLI: serve / process / evaluate
|
||||
}
|
||||
|
||||
class HermesAgentBaseEnv {
|
||||
Terminal backend configuration
|
||||
Tool resolution
|
||||
Agent loop engine
|
||||
ToolContext access
|
||||
}
|
||||
|
||||
class TerminalTestEnv {
|
||||
Stack testing
|
||||
}
|
||||
|
||||
class HermesSweEnv {
|
||||
SWE training
|
||||
}
|
||||
|
||||
class TerminalBench2EvalEnv {
|
||||
Benchmark evaluation
|
||||
}
|
||||
|
||||
class TBLiteEvalEnv {
|
||||
Fast benchmark
|
||||
}
|
||||
|
||||
class YCBenchEvalEnv {
|
||||
Long-horizon benchmark
|
||||
}
|
||||
|
||||
BaseEnv <|-- HermesAgentBaseEnv
|
||||
HermesAgentBaseEnv <|-- TerminalTestEnv
|
||||
HermesAgentBaseEnv <|-- HermesSweEnv
|
||||
HermesAgentBaseEnv <|-- TerminalBench2EvalEnv
|
||||
TerminalBench2EvalEnv <|-- TBLiteEvalEnv
|
||||
TerminalBench2EvalEnv <|-- YCBenchEvalEnv
|
||||
```
|
||||
|
||||
### BaseEnv (Atropos)
|
||||
|
||||
The foundation from `atroposlib`. Provides:
|
||||
- **Server management** — connects to OpenAI-compatible APIs (VLLM, SGLang, OpenRouter)
|
||||
- **Worker scheduling** — parallel rollout coordination
|
||||
- **Wandb integration** — metrics logging and rollout visualisation
|
||||
- **CLI interface** — three subcommands: `serve`, `process`, `evaluate`
|
||||
- **Eval logging** — `evaluate_log()` saves results to JSON + JSONL
|
||||
|
||||
### HermesAgentBaseEnv
|
||||
|
||||
The hermes-agent layer (`environments/hermes_base_env.py`). Adds:
|
||||
- **Terminal backend configuration** — sets `TERMINAL_ENV` for sandboxed execution (local, Docker, Modal, Daytona, SSH, Singularity)
|
||||
- **Tool resolution** — `_resolve_tools_for_group()` calls hermes-agent's `get_tool_definitions()` to get the right tool schemas based on enabled/disabled toolsets
|
||||
- **Agent loop integration** — `collect_trajectory()` runs `HermesAgentLoop` and scores the result
|
||||
- **Two-phase operation** — Phase 1 (OpenAI server) for eval/SFT, Phase 2 (VLLM ManagedServer) for full RL with logprobs
|
||||
- **Async safety patches** — monkey-patches Modal backend to work inside Atropos's event loop
|
||||
|
||||
### Concrete Environments
|
||||
|
||||
Your environment inherits from `HermesAgentBaseEnv` and implements five methods:
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `setup()` | Load dataset, initialise state |
|
||||
| `get_next_item()` | Return the next item for rollout |
|
||||
| `format_prompt(item)` | Convert an item into the user message |
|
||||
| `compute_reward(item, result, ctx)` | Score the rollout (0.0–1.0) |
|
||||
| `evaluate()` | Periodic evaluation logic |
|
||||
|
||||
## Core Components
|
||||
|
||||
### Agent Loop
|
||||
|
||||
`HermesAgentLoop` (`environments/agent_loop.py`) is the reusable multi-turn agent engine. It runs the same tool-calling pattern as hermes-agent's main loop:
|
||||
|
||||
1. Send messages + tool schemas to the API via `server.chat_completion()`
|
||||
2. If the response contains `tool_calls`, dispatch each via `handle_function_call()`
|
||||
3. Append tool results to the conversation, go back to step 1
|
||||
4. If no `tool_calls`, the agent is done
|
||||
|
||||
Tool calls execute in a thread pool (`ThreadPoolExecutor(128)`) so that async backends (Modal, Docker) don't deadlock inside Atropos's event loop.
|
||||
|
||||
Returns an `AgentResult`:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class AgentResult:
|
||||
messages: List[Dict[str, Any]] # Full conversation history
|
||||
turns_used: int # Number of LLM calls made
|
||||
finished_naturally: bool # True if model stopped on its own
|
||||
reasoning_per_turn: List[Optional[str]] # Extracted reasoning content
|
||||
tool_errors: List[ToolError] # Errors encountered during tool dispatch
|
||||
managed_state: Optional[Dict] # VLLM ManagedServer state (Phase 2)
|
||||
```
|
||||
|
||||
### Tool Context
|
||||
|
||||
`ToolContext` (`environments/tool_context.py`) gives reward functions direct access to the **same sandbox** the model used during its rollout. The `task_id` scoping means all state (files, processes, browser tabs) is preserved.
|
||||
|
||||
```python
|
||||
async def compute_reward(self, item, result, ctx: ToolContext):
|
||||
# Run tests in the model's terminal sandbox
|
||||
test = ctx.terminal("pytest -v")
|
||||
if test["exit_code"] == 0:
|
||||
return 1.0
|
||||
|
||||
# Check if a file was created
|
||||
content = ctx.read_file("/workspace/solution.py")
|
||||
if content.get("content"):
|
||||
return 0.5
|
||||
|
||||
# Download files for local verification
|
||||
ctx.download_file("/remote/output.bin", "/local/output.bin")
|
||||
return 0.0
|
||||
```
|
||||
|
||||
Available methods:
|
||||
|
||||
| Category | Methods |
|
||||
|----------|---------|
|
||||
| **Terminal** | `terminal(command, timeout)` |
|
||||
| **Files** | `read_file(path)`, `write_file(path, content)`, `search(query, path)` |
|
||||
| **Transfers** | `upload_file()`, `upload_dir()`, `download_file()`, `download_dir()` |
|
||||
| **Web** | `web_search(query)`, `web_extract(urls)` |
|
||||
| **Browser** | `browser_navigate(url)`, `browser_snapshot()` |
|
||||
| **Generic** | `call_tool(name, args)` — escape hatch for any hermes-agent tool |
|
||||
| **Cleanup** | `cleanup()` — release all resources |
|
||||
|
||||
### Tool Call Parsers
|
||||
|
||||
For **Phase 2** (VLLM ManagedServer), the server returns raw text without structured tool calls. Client-side parsers in `environments/tool_call_parsers/` extract `tool_calls` from raw output:
|
||||
|
||||
```python
|
||||
from environments.tool_call_parsers import get_parser
|
||||
|
||||
parser = get_parser("hermes") # or "mistral", "llama3_json", "qwen", "deepseek_v3", etc.
|
||||
content, tool_calls = parser.parse(raw_model_output)
|
||||
```
|
||||
|
||||
Available parsers: `hermes`, `mistral`, `llama3_json`, `llama4_json`, `qwen`, `qwen3_coder`, `deepseek_v3`, `deepseek_v3_1` (alias `deepseek_v31`), `kimi_k2`, `longcat`, `glm45`, `glm47`.
|
||||
|
||||
In Phase 1 (OpenAI server type), parsers are not needed — the server handles tool call parsing natively.
|
||||
|
||||
## Available Benchmarks
|
||||
|
||||
### TerminalBench2
|
||||
|
||||
**89 challenging terminal tasks** with per-task Docker sandbox environments.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **What it tests** | Single-task coding/sysadmin ability |
|
||||
| **Scoring** | Binary pass/fail (test suite verification) |
|
||||
| **Sandbox** | Modal cloud sandboxes (per-task Docker images) |
|
||||
| **Tools** | `terminal` + `file` |
|
||||
| **Tasks** | 89 tasks across multiple categories |
|
||||
| **Cost** | ~$50–200 for full eval (parallel execution) |
|
||||
| **Time** | ~2–4 hours |
|
||||
|
||||
```bash
|
||||
python environments/benchmarks/terminalbench_2/terminalbench2_env.py evaluate \
|
||||
--config environments/benchmarks/terminalbench_2/default.yaml
|
||||
|
||||
# Run specific tasks
|
||||
python environments/benchmarks/terminalbench_2/terminalbench2_env.py evaluate \
|
||||
--config environments/benchmarks/terminalbench_2/default.yaml \
|
||||
--env.task_filter fix-git,git-multibranch
|
||||
```
|
||||
|
||||
Dataset: [NousResearch/terminal-bench-2](https://huggingface.co/datasets/NousResearch/terminal-bench-2) on HuggingFace.
|
||||
|
||||
### TBLite (OpenThoughts Terminal Bench Lite)
|
||||
|
||||
**100 difficulty-calibrated tasks** — a faster proxy for TerminalBench2.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **What it tests** | Same as TB2 (coding/sysadmin), calibrated difficulty tiers |
|
||||
| **Scoring** | Binary pass/fail |
|
||||
| **Sandbox** | Modal cloud sandboxes |
|
||||
| **Tools** | `terminal` + `file` |
|
||||
| **Tasks** | 100 tasks: Easy (40), Medium (26), Hard (26), Extreme (8) |
|
||||
| **Correlation** | r=0.911 with full TB2 |
|
||||
| **Speed** | 2.6–8× faster than TB2 |
|
||||
|
||||
```bash
|
||||
python environments/benchmarks/tblite/tblite_env.py evaluate \
|
||||
--config environments/benchmarks/tblite/default.yaml
|
||||
```
|
||||
|
||||
TBLite is a thin subclass of TerminalBench2 — only the dataset and timeouts differ. Created by the OpenThoughts Agent team (Snorkel AI + Bespoke Labs). Dataset: [NousResearch/openthoughts-tblite](https://huggingface.co/datasets/NousResearch/openthoughts-tblite).
|
||||
|
||||
### YC-Bench
|
||||
|
||||
**Long-horizon strategic benchmark** — the agent plays CEO of an AI startup.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **What it tests** | Multi-turn strategic coherence over hundreds of turns |
|
||||
| **Scoring** | Composite: `0.5 × survival + 0.5 × normalised_funds` |
|
||||
| **Sandbox** | Local terminal (no Modal needed) |
|
||||
| **Tools** | `terminal` only |
|
||||
| **Runs** | 9 default (3 presets × 3 seeds), sequential |
|
||||
| **Cost** | ~$50–200 for full eval |
|
||||
| **Time** | ~3–6 hours |
|
||||
|
||||
```bash
|
||||
# Install yc-bench (optional dependency)
|
||||
pip install "hermes-agent[yc-bench]"
|
||||
|
||||
# Run evaluation
|
||||
bash environments/benchmarks/yc_bench/run_eval.sh
|
||||
|
||||
# Or directly
|
||||
python environments/benchmarks/yc_bench/yc_bench_env.py evaluate \
|
||||
--config environments/benchmarks/yc_bench/default.yaml
|
||||
|
||||
# Quick single-preset test
|
||||
python environments/benchmarks/yc_bench/yc_bench_env.py evaluate \
|
||||
--config environments/benchmarks/yc_bench/default.yaml \
|
||||
--env.presets '["fast_test"]' --env.seeds '[1]'
|
||||
```
|
||||
|
||||
YC-Bench uses [collinear-ai/yc-bench](https://github.com/collinear-ai/yc-bench) — a deterministic simulation with 4 skill domains (research, inference, data_environment, training), prestige system, employee management, and financial pressure. Unlike TB2's per-task binary scoring, YC-Bench measures whether an agent can maintain coherent strategy over hundreds of compounding decisions.
|
||||
|
||||
## Training Environments
|
||||
|
||||
### TerminalTestEnv
|
||||
|
||||
A minimal self-contained environment with inline tasks (no external dataset). Used for **validating the full stack** end-to-end. Each task asks the model to create a file at a known path; the verifier checks the content.
|
||||
|
||||
```bash
|
||||
# Process mode (saves rollouts to JSONL, no training server needed)
|
||||
python environments/terminal_test_env/terminal_test_env.py process \
|
||||
--env.data_path_to_save_groups terminal_test_output.jsonl
|
||||
|
||||
# Serve mode (connects to Atropos API for RL training)
|
||||
python environments/terminal_test_env/terminal_test_env.py serve
|
||||
```
|
||||
|
||||
### HermesSweEnv
|
||||
|
||||
SWE-bench style training environment. The model gets a coding task, uses terminal + file + web tools to solve it, and the reward function runs tests in the same Modal sandbox.
|
||||
|
||||
```bash
|
||||
python environments/hermes_swe_env/hermes_swe_env.py serve \
|
||||
--openai.model_name YourModel \
|
||||
--env.dataset_name bigcode/humanevalpack \
|
||||
--env.terminal_backend modal
|
||||
```
|
||||
|
||||
## Running Environments
|
||||
|
||||
Every environment is a standalone Python script with three CLI subcommands:
|
||||
|
||||
### `evaluate` — Run a benchmark
|
||||
|
||||
For eval-only environments (benchmarks). Runs all items, computes metrics, logs to wandb.
|
||||
|
||||
```bash
|
||||
python environments/benchmarks/tblite/tblite_env.py evaluate \
|
||||
--config environments/benchmarks/tblite/default.yaml \
|
||||
--openai.model_name anthropic/claude-sonnet-4.6
|
||||
```
|
||||
|
||||
No training server or `run-api` needed. The environment handles everything.
|
||||
|
||||
### `process` — Generate SFT data
|
||||
|
||||
Runs rollouts and saves scored trajectories to JSONL. Useful for generating training data without a full RL loop.
|
||||
|
||||
```bash
|
||||
python environments/terminal_test_env/terminal_test_env.py process \
|
||||
--env.data_path_to_save_groups output.jsonl \
|
||||
--openai.model_name anthropic/claude-sonnet-4.6
|
||||
```
|
||||
|
||||
Output format: each line is a scored trajectory with the full conversation history, reward, and metadata.
|
||||
|
||||
### `serve` — Connect to Atropos for RL training
|
||||
|
||||
Connects the environment to a running Atropos API server (`run-api`). Used during live RL training.
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start the Atropos API
|
||||
run-api
|
||||
|
||||
# Terminal 2: Start the environment
|
||||
python environments/hermes_swe_env/hermes_swe_env.py serve \
|
||||
--openai.model_name YourModel
|
||||
```
|
||||
|
||||
The environment receives items from Atropos, runs agent rollouts, computes rewards, and sends scored trajectories back for training.
|
||||
|
||||
## Two-Phase Operation
|
||||
|
||||
### Phase 1: OpenAI Server (Eval / SFT)
|
||||
|
||||
Uses `server.chat_completion()` with `tools=` parameter. The server (VLLM, SGLang, OpenRouter, OpenAI) handles tool call parsing natively. Returns `ChatCompletion` objects with structured `tool_calls`.
|
||||
|
||||
- **Use for**: evaluation, SFT data generation, benchmarks, testing
|
||||
- **Placeholder tokens** are created for the Atropos pipeline (since real token IDs aren't available from the OpenAI API)
|
||||
|
||||
### Phase 2: VLLM ManagedServer (Full RL)
|
||||
|
||||
Uses ManagedServer for exact token IDs + logprobs via `/generate`. A client-side [tool call parser](#tool-call-parsers) reconstructs structured `tool_calls` from raw output.
|
||||
|
||||
- **Use for**: full RL training with GRPO/PPO
|
||||
- **Real tokens**, masks, and logprobs flow through the pipeline
|
||||
- Set `tool_call_parser` in config to match your model's format (e.g., `"hermes"`, `"qwen"`, `"mistral"`)
|
||||
|
||||
## Creating Environments
|
||||
|
||||
### Training Environment
|
||||
|
||||
```python
|
||||
from environments.hermes_base_env import HermesAgentBaseEnv, HermesAgentEnvConfig
|
||||
from atroposlib.envs.server_handling.server_manager import APIServerConfig
|
||||
|
||||
class MyEnvConfig(HermesAgentEnvConfig):
|
||||
my_custom_field: str = "default_value"
|
||||
|
||||
class MyEnv(HermesAgentBaseEnv):
|
||||
name = "my-env"
|
||||
env_config_cls = MyEnvConfig
|
||||
|
||||
@classmethod
|
||||
def config_init(cls):
|
||||
env_config = MyEnvConfig(
|
||||
enabled_toolsets=["terminal", "file"],
|
||||
terminal_backend="modal",
|
||||
max_agent_turns=30,
|
||||
)
|
||||
server_configs = [APIServerConfig(
|
||||
base_url="https://openrouter.ai/api/v1",
|
||||
model_name="anthropic/claude-sonnet-4.6",
|
||||
server_type="openai",
|
||||
)]
|
||||
return env_config, server_configs
|
||||
|
||||
async def setup(self):
|
||||
from datasets import load_dataset
|
||||
self.dataset = list(load_dataset("my-dataset", split="train"))
|
||||
self.iter = 0
|
||||
|
||||
async def get_next_item(self):
|
||||
item = self.dataset[self.iter % len(self.dataset)]
|
||||
self.iter += 1
|
||||
return item
|
||||
|
||||
def format_prompt(self, item):
|
||||
return item["instruction"]
|
||||
|
||||
async def compute_reward(self, item, result, ctx):
|
||||
# ctx gives full tool access to the rollout's sandbox
|
||||
test = ctx.terminal("pytest -v")
|
||||
return 1.0 if test["exit_code"] == 0 else 0.0
|
||||
|
||||
async def evaluate(self, *args, **kwargs):
|
||||
# Periodic evaluation during training
|
||||
pass
|
||||
|
||||
if __name__ == "__main__":
|
||||
MyEnv.cli()
|
||||
```
|
||||
|
||||
### Eval-Only Benchmark
|
||||
|
||||
For benchmarks, follow the pattern used by TerminalBench2, TBLite, and YC-Bench:
|
||||
|
||||
1. **Create under** `environments/benchmarks/your-benchmark/`
|
||||
2. **Set eval-only config**: `eval_handling=STOP_TRAIN`, `steps_per_eval=1`, `total_steps=1`
|
||||
3. **Stub training methods**: `collect_trajectories()` returns `(None, [])`, `score()` returns `None`
|
||||
4. **Implement** `rollout_and_score_eval(eval_item)` — the per-item agent loop + scoring
|
||||
5. **Implement** `evaluate()` — orchestrates all runs, computes aggregate metrics
|
||||
6. **Add streaming JSONL** for crash-safe result persistence
|
||||
7. **Add cleanup**: `KeyboardInterrupt` handling, `cleanup_all_environments()`, `_tool_executor.shutdown()`
|
||||
8. **Run with** `evaluate` subcommand
|
||||
|
||||
See `environments/benchmarks/yc_bench/yc_bench_env.py` for a clean, well-documented reference implementation.
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### HermesAgentEnvConfig Fields
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `enabled_toolsets` | `List[str]` | `None` (all) | Which hermes toolsets to enable |
|
||||
| `disabled_toolsets` | `List[str]` | `None` | Toolsets to filter out |
|
||||
| `distribution` | `str` | `None` | Probabilistic toolset distribution name |
|
||||
| `max_agent_turns` | `int` | `30` | Max LLM calls per rollout |
|
||||
| `agent_temperature` | `float` | `1.0` | Sampling temperature |
|
||||
| `system_prompt` | `str` | `None` | System message for the agent |
|
||||
| `terminal_backend` | `str` | `"local"` | `local`, `docker`, `modal`, `daytona`, `ssh`, `singularity` |
|
||||
| `terminal_timeout` | `int` | `120` | Seconds per terminal command |
|
||||
| `terminal_lifetime` | `int` | `3600` | Max sandbox lifetime |
|
||||
| `dataset_name` | `str` | `None` | HuggingFace dataset identifier |
|
||||
| `tool_pool_size` | `int` | `128` | Thread pool size for tool execution |
|
||||
| `tool_call_parser` | `str` | `"hermes"` | Parser for Phase 2 raw output |
|
||||
| `extra_body` | `Dict` | `None` | Extra params for OpenAI API (e.g., OpenRouter provider prefs) |
|
||||
| `eval_handling` | `Enum` | `STOP_TRAIN` | `STOP_TRAIN`, `LIMIT_TRAIN`, `NONE` |
|
||||
|
||||
### YAML Configuration
|
||||
|
||||
Environments can be configured via YAML files passed with `--config`:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
enabled_toolsets: ["terminal", "file"]
|
||||
max_agent_turns: 60
|
||||
max_token_length: 32000
|
||||
agent_temperature: 0.8
|
||||
terminal_backend: "modal"
|
||||
terminal_timeout: 300
|
||||
dataset_name: "NousResearch/terminal-bench-2"
|
||||
tokenizer_name: "NousResearch/Hermes-3-Llama-3.1-8B"
|
||||
use_wandb: true
|
||||
wandb_name: "my-benchmark"
|
||||
|
||||
openai:
|
||||
base_url: "https://openrouter.ai/api/v1"
|
||||
model_name: "anthropic/claude-sonnet-4.6"
|
||||
server_type: "openai"
|
||||
health_check: false
|
||||
```
|
||||
|
||||
YAML values override `config_init()` defaults. CLI arguments override YAML values:
|
||||
|
||||
```bash
|
||||
python my_env.py evaluate \
|
||||
--config my_config.yaml \
|
||||
--openai.model_name anthropic/claude-opus-4.6 # overrides YAML
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### For all environments
|
||||
|
||||
- Python >= 3.11
|
||||
- `atroposlib`: `pip install git+https://github.com/NousResearch/atropos.git`
|
||||
- An LLM API key (OpenRouter, OpenAI, or self-hosted VLLM/SGLang)
|
||||
|
||||
### For Modal-sandboxed benchmarks (TB2, TBLite)
|
||||
|
||||
- [Modal](https://modal.com) account and CLI: `pip install "hermes-agent[modal]"`
|
||||
- `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` environment variables
|
||||
|
||||
### For YC-Bench
|
||||
|
||||
- `pip install "hermes-agent[yc-bench]"` (installs the yc-bench CLI + SQLAlchemy)
|
||||
- No Modal needed — runs with local terminal backend
|
||||
|
||||
### For RL training
|
||||
|
||||
- `TINKER_API_KEY` — API key for the [Tinker](https://tinker.computer) training service
|
||||
- `WANDB_API_KEY` — for Weights & Biases metrics tracking
|
||||
- The `tinker-atropos` submodule (at `tinker-atropos/` in the repo)
|
||||
|
||||
See [RL Training](/user-guide/features/rl-training) for the agent-driven RL workflow.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
environments/
|
||||
├── hermes_base_env.py # Abstract base class (HermesAgentBaseEnv)
|
||||
├── agent_loop.py # Multi-turn agent engine (HermesAgentLoop)
|
||||
├── tool_context.py # Per-rollout tool access for reward functions
|
||||
├── patches.py # Async-safety patches for Modal backend
|
||||
│
|
||||
├── tool_call_parsers/ # Phase 2 client-side parsers
|
||||
│ ├── hermes_parser.py # Hermes/ChatML <tool_call> format
|
||||
│ ├── mistral_parser.py # Mistral [TOOL_CALLS] format
|
||||
│ ├── llama_parser.py # Llama 3 JSON tool calling
|
||||
│ ├── qwen_parser.py # Qwen format
|
||||
│ ├── deepseek_v3_parser.py # DeepSeek V3 format
|
||||
│ └── ... # + kimi_k2, longcat, glm45/47, etc.
|
||||
│
|
||||
├── terminal_test_env/ # Stack validation (inline tasks)
|
||||
├── hermes_swe_env/ # SWE-bench training environment
|
||||
│
|
||||
└── benchmarks/ # Evaluation benchmarks
|
||||
├── terminalbench_2/ # 89 terminal tasks, Modal sandboxes
|
||||
├── tblite/ # 100 calibrated tasks (fast TB2 proxy)
|
||||
└── yc_bench/ # Long-horizon strategic benchmark
|
||||
```
|
||||
@@ -0,0 +1,231 @@
|
||||
---
|
||||
sidebar_position: 12
|
||||
title: "Video Generation Provider Plugins"
|
||||
description: "How to build a video-generation backend plugin for Hermes Agent"
|
||||
---
|
||||
|
||||
# Building a Video Generation Provider Plugin
|
||||
|
||||
Video-gen provider plugins register a backend that services every `video_generate` tool call. Built-in providers (xAI, FAL) ship as plugins. Add a new one, or override a bundled one, by dropping a directory into `plugins/video_gen/<name>/`.
|
||||
|
||||
:::tip
|
||||
Video-gen mirrors [Image Generation Provider Plugins](/docs/developer-guide/image-gen-provider-plugin) almost line-for-line — if you've built an image-gen backend, you already know the shape. The main differences: a `capabilities()` method advertising modalities/aspect-ratios/durations, and a routing convention (pass `image_url` to use image-to-video, omit it to use text-to-video — the provider picks the right endpoint internally).
|
||||
:::
|
||||
|
||||
## The unified surface (one tool, two modalities)
|
||||
|
||||
The `video_generate` tool exposes two modalities through one parameter:
|
||||
|
||||
- **Text-to-video** — call with `prompt` only. The provider routes to its text-to-video endpoint.
|
||||
- **Image-to-video** — call with `prompt` + `image_url`. The provider routes to its image-to-video endpoint.
|
||||
|
||||
Edit and extend are intentionally out of scope. Most backends don't support them and the inconsistency would force per-backend prose into the agent's tool description.
|
||||
|
||||
## How discovery works
|
||||
|
||||
Hermes scans for video-gen backends in three places:
|
||||
|
||||
1. **Bundled** — `<repo>/plugins/video_gen/<name>/` (auto-loaded with `kind: backend`)
|
||||
2. **User** — `~/.hermes/plugins/video_gen/<name>/` (opt-in via `plugins.enabled`)
|
||||
3. **Pip** — packages declaring a `hermes_agent.plugins` entry point
|
||||
|
||||
Each plugin's `register(ctx)` function calls `ctx.register_video_gen_provider(...)`. The active provider is picked by `video_gen.provider` in `config.yaml`; `hermes tools` → Video Generation walks users through selection. Unlike `image_generate`, there is no in-tree legacy backend — every provider is a plugin.
|
||||
|
||||
## Directory structure
|
||||
|
||||
```
|
||||
plugins/video_gen/my-backend/
|
||||
├── __init__.py # VideoGenProvider subclass + register()
|
||||
└── plugin.yaml # Manifest with kind: backend
|
||||
```
|
||||
|
||||
## The VideoGenProvider ABC
|
||||
|
||||
Subclass `agent.video_gen_provider.VideoGenProvider`. Required: `name` property and `generate()` method.
|
||||
|
||||
```python
|
||||
# plugins/video_gen/my-backend/__init__.py
|
||||
from typing import Any, Dict, List, Optional
|
||||
import os
|
||||
|
||||
from agent.video_gen_provider import (
|
||||
VideoGenProvider,
|
||||
error_response,
|
||||
success_response,
|
||||
)
|
||||
|
||||
|
||||
class MyVideoGenProvider(VideoGenProvider):
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "my-backend"
|
||||
|
||||
@property
|
||||
def display_name(self) -> str:
|
||||
return "My Backend"
|
||||
|
||||
def is_available(self) -> bool:
|
||||
return bool(os.environ.get("MY_API_KEY"))
|
||||
|
||||
def list_models(self) -> List[Dict[str, Any]]:
|
||||
# Each entry is a model FAMILY — a name the user picks once.
|
||||
# Your provider's generate() routes within the family based on
|
||||
# whether image_url was passed.
|
||||
return [
|
||||
{
|
||||
"id": "fast",
|
||||
"display": "Fast",
|
||||
"speed": "~30s",
|
||||
"strengths": "Cheapest tier",
|
||||
"price": "$0.05/s",
|
||||
"modalities": ["text", "image"], # advisory
|
||||
},
|
||||
]
|
||||
|
||||
def default_model(self) -> Optional[str]:
|
||||
return "fast"
|
||||
|
||||
def capabilities(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"modalities": ["text", "image"],
|
||||
"aspect_ratios": ["16:9", "9:16"],
|
||||
"resolutions": ["720p", "1080p"],
|
||||
"min_duration": 1,
|
||||
"max_duration": 10,
|
||||
"supports_audio": False,
|
||||
"supports_negative_prompt": True,
|
||||
"max_reference_images": 0,
|
||||
}
|
||||
|
||||
def get_setup_schema(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"name": "My Backend",
|
||||
"badge": "paid",
|
||||
"tag": "Short description shown in `hermes tools`",
|
||||
"env_vars": [
|
||||
{
|
||||
"key": "MY_API_KEY",
|
||||
"prompt": "My Backend API key",
|
||||
"url": "https://mybackend.example.com/keys",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
def generate(
|
||||
self,
|
||||
prompt: str,
|
||||
*,
|
||||
model: Optional[str] = None,
|
||||
image_url: Optional[str] = None,
|
||||
reference_image_urls: Optional[List[str]] = None,
|
||||
duration: Optional[int] = None,
|
||||
aspect_ratio: str = "16:9",
|
||||
resolution: str = "720p",
|
||||
negative_prompt: Optional[str] = None,
|
||||
audio: Optional[bool] = None,
|
||||
seed: Optional[int] = None,
|
||||
**kwargs: Any, # always ignore unknown kwargs for forward-compat
|
||||
) -> Dict[str, Any]:
|
||||
# ROUTE: image_url presence picks the endpoint.
|
||||
if image_url:
|
||||
endpoint = "my-backend/image-to-video"
|
||||
modality_used = "image"
|
||||
else:
|
||||
endpoint = "my-backend/text-to-video"
|
||||
modality_used = "text"
|
||||
|
||||
# ... call your API ...
|
||||
|
||||
return success_response(
|
||||
video="https://your-cdn/output.mp4",
|
||||
model=model or "fast",
|
||||
prompt=prompt,
|
||||
modality=modality_used,
|
||||
aspect_ratio=aspect_ratio,
|
||||
duration=duration or 5,
|
||||
provider=self.name,
|
||||
)
|
||||
|
||||
|
||||
def register(ctx) -> None:
|
||||
ctx.register_video_gen_provider(MyVideoGenProvider())
|
||||
```
|
||||
|
||||
## The plugin manifest
|
||||
|
||||
```yaml
|
||||
# plugins/video_gen/my-backend/plugin.yaml
|
||||
name: my-backend
|
||||
version: 1.0.0
|
||||
description: "My video generation backend"
|
||||
author: Your Name
|
||||
kind: backend
|
||||
requires_env:
|
||||
- MY_API_KEY
|
||||
```
|
||||
|
||||
## The `video_generate` schema
|
||||
|
||||
The tool exposes one schema across every backend. Providers ignore parameters they don't support.
|
||||
|
||||
| Parameter | What it does |
|
||||
|---|---|
|
||||
| `prompt` | Text instruction (required) |
|
||||
| `image_url` | When set → image-to-video; when omitted → text-to-video |
|
||||
| `reference_image_urls` | Style/character refs (provider-dependent) |
|
||||
| `duration` | Seconds — provider clamps |
|
||||
| `aspect_ratio` | `"16:9"`, `"9:16"`, `"1:1"`, ... — provider clamps |
|
||||
| `resolution` | `"480p"` / `"540p"` / `"720p"` / `"1080p"` — provider clamps |
|
||||
| `negative_prompt` | Content to avoid (Pixverse/Kling only) |
|
||||
| `audio` | Native audio (Veo3 / Pixverse pricing tier) |
|
||||
| `seed` | Reproducibility |
|
||||
| `model` | Override the active model/family |
|
||||
|
||||
The provider's `capabilities()` advertises which of these are honored. The agent sees the active backend's capabilities in the tool description, dynamically rebuilt when the user changes backend via `hermes tools`.
|
||||
|
||||
## Model families and endpoint routing (the FAL pattern)
|
||||
|
||||
When your backend has multiple endpoints per "model" — like FAL, where every family (Veo 3.1, Pixverse v6, Kling O3) has both a `/text-to-video` and an `/image-to-video` URL — represent each **family** as one catalog entry. Your `generate()` picks the right endpoint based on whether `image_url` was passed:
|
||||
|
||||
```python
|
||||
FAMILIES = {
|
||||
"veo3.1": {
|
||||
"text_endpoint": "fal-ai/veo3.1",
|
||||
"image_endpoint": "fal-ai/veo3.1/image-to-video",
|
||||
# ... family-specific capability flags ...
|
||||
},
|
||||
}
|
||||
|
||||
def generate(self, prompt, *, image_url=None, model=None, **kwargs):
|
||||
family_id, family = _resolve_family(model)
|
||||
endpoint = family["image_endpoint"] if image_url else family["text_endpoint"]
|
||||
# ... build payload from family's declared capability flags, call endpoint ...
|
||||
```
|
||||
|
||||
The user picks `veo3.1` once in `hermes tools`. The agent never thinks about endpoints — it just passes (or doesn't pass) `image_url`.
|
||||
|
||||
## Selection precedence
|
||||
|
||||
For per-instance model knobs (see `plugins/video_gen/fal/__init__.py`):
|
||||
|
||||
1. `model=` keyword from the tool call
|
||||
2. `<PROVIDER>_VIDEO_MODEL` env var
|
||||
3. `video_gen.<provider>.model` in `config.yaml`
|
||||
4. `video_gen.model` in `config.yaml` (when it's one of your IDs)
|
||||
5. Provider's `default_model()`
|
||||
|
||||
## Response shape
|
||||
|
||||
`success_response()` and `error_response()` produce the dict shape every backend returns. Use them — don't hand-roll the dict.
|
||||
|
||||
Success keys: `success`, `video` (URL or absolute path), `model`, `prompt`, `modality` (`"text"` or `"image"`), `aspect_ratio`, `duration`, `provider`, plus `extra`.
|
||||
|
||||
Error keys: `success`, `video` (None), `error`, `error_type`, `model`, `prompt`, `aspect_ratio`, `provider`.
|
||||
|
||||
## Where to save artifacts
|
||||
|
||||
If your backend returns base64, use `save_b64_video()` to write under `$HERMES_HOME/cache/videos/`. For raw bytes from a follow-up HTTP fetch, use `save_bytes_video()`. Otherwise return the upstream URL directly — the gateway resolves remote URLs on delivery.
|
||||
|
||||
## Testing
|
||||
|
||||
Drop a smoke test under `tests/plugins/video_gen/test_<name>_plugin.py`. The xAI and FAL tests show the pattern — register, verify catalog, exercise routing both with and without `image_url`, assert clean error responses on missing auth.
|
||||
@@ -128,6 +128,43 @@ If you want to clone the repo and install from source — for contributing, runn
|
||||
|
||||
---
|
||||
|
||||
## Non-Sudo / System Service User Installs
|
||||
|
||||
Running Hermes as a dedicated unprivileged user (e.g. a `hermes` systemd service account, or any user without `sudo` access) is supported. The only thing on the install path that genuinely needs root is Playwright's `--with-deps` step, which `apt`-installs shared libraries (`libnss3`, `libxkbcommon`, etc.) used by Chromium. The installer detects whether sudo is available and gracefully degrades when it isn't — it will install the Chromium binary into the service user's own Playwright cache and print the exact command an administrator needs to run separately.
|
||||
|
||||
**Recommended split (Debian/Ubuntu):**
|
||||
|
||||
1. **One time, as an admin user with sudo**, install the system libraries Chromium needs:
|
||||
```bash
|
||||
sudo npx playwright install-deps chromium
|
||||
```
|
||||
(You can run this from anywhere — `npx` will fetch Playwright on the fly.)
|
||||
|
||||
2. **As the unprivileged service user**, run the regular installer. It will detect the missing sudo, skip `--with-deps`, and install Chromium into the user's local Playwright cache:
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
If you want to skip the Playwright step entirely — for example because you're running headless and don't need browser automation — pass `--skip-browser`:
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash -s -- --skip-browser
|
||||
```
|
||||
|
||||
3. **Make `hermes` available to the service user's shells.** The installer writes the launcher to `~/.local/bin/hermes`. System service accounts often have a minimal PATH that doesn't include `~/.local/bin`. Either add it to the user's environment, or symlink the launcher into a system location:
|
||||
```bash
|
||||
# Option A — add to the service user's profile
|
||||
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
|
||||
|
||||
# Option B — symlink system-wide (run as an admin)
|
||||
sudo ln -s /home/hermes/.hermes/hermes-agent/venv/bin/hermes /usr/local/bin/hermes
|
||||
```
|
||||
|
||||
4. **Verify:** `hermes doctor` should now run cleanly. If you get `ModuleNotFoundError: No module named 'dotenv'`, you're invoking the repo source `hermes` file (`~/.hermes/hermes-agent/hermes`) with system Python instead of the venv launcher (`~/.hermes/hermes-agent/venv/bin/hermes`) — fix step 3.
|
||||
|
||||
The same pattern works on Arch (the installer uses pacman with the same sudo-detection logic), Fedora/RHEL, and openSUSE — those distros don't support `--with-deps` at all, so an administrator always installs the system libraries separately. The relevant `dnf`/`zypper` commands are printed by the installer.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Solution |
|
||||
|
||||
@@ -123,13 +123,11 @@ If you installed manually (not via the quick installer):
|
||||
cd /path/to/hermes-agent
|
||||
export VIRTUAL_ENV="$(pwd)/venv"
|
||||
|
||||
# Pull latest code and submodules
|
||||
# Pull latest code
|
||||
git pull origin main
|
||||
git submodule update --init --recursive
|
||||
|
||||
# Reinstall (picks up new dependencies)
|
||||
uv pip install -e ".[all]"
|
||||
uv pip install -e "./tinker-atropos"
|
||||
|
||||
# Check for new config options
|
||||
hermes config check
|
||||
|
||||
@@ -20,6 +20,7 @@ Hermes has several distinct pluggable interfaces — some use Python `register_*
|
||||
| A **memory backend** (Honcho/Mem0/Supermemory/etc.) | [Memory Provider Plugins](/docs/developer-guide/memory-provider-plugin) |
|
||||
| A **context-compression engine** | [Context Engine Plugins](/docs/developer-guide/context-engine-plugin) |
|
||||
| An **image-generation backend** | [Image Generation Provider Plugins](/docs/developer-guide/image-gen-provider-plugin) |
|
||||
| A **video-generation backend** | [Video Generation Provider Plugins](/docs/developer-guide/video-gen-provider-plugin) |
|
||||
| A **TTS backend** (any CLI — Piper, VoxCPM, Kokoro, voice cloning, …) | [TTS custom command providers](/docs/user-guide/features/tts#custom-command-providers) — config-driven, no Python needed |
|
||||
| An **STT backend** (custom whisper / ASR CLI) | [Voice Message Transcription](/docs/user-guide/features/tts#voice-message-transcription-stt) — set `HERMES_LOCAL_STT_COMMAND` to a shell template |
|
||||
| **External tools via MCP** (filesystem, GitHub, Linear, any MCP server) | [MCP](/docs/user-guide/features/mcp) — declare `mcp_servers.<name>` in `config.yaml` |
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
---
|
||||
sidebar_position: 16
|
||||
title: "xAI Grok OAuth (SuperGrok Subscription)"
|
||||
description: "Sign in with your SuperGrok subscription to use Grok models in Hermes Agent — no API key required"
|
||||
---
|
||||
|
||||
# xAI Grok OAuth (SuperGrok Subscription)
|
||||
|
||||
Hermes Agent supports xAI Grok through a browser-based OAuth login flow against [accounts.x.ai](https://accounts.x.ai), using your existing **SuperGrok subscription**. No `XAI_API_KEY` is required — log in once and Hermes automatically refreshes your session in the background.
|
||||
|
||||
The transport reuses the `codex_responses` adapter (xAI exposes a Responses-style endpoint), so reasoning, tool-calling, streaming, and prompt caching work without any adapter changes.
|
||||
|
||||
The same OAuth bearer token is also reused by every direct-to-xAI surface in Hermes — TTS, image generation, video generation, and transcription — so a single login covers all four.
|
||||
|
||||
## Overview
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Provider ID | `xai-oauth` |
|
||||
| Display name | xAI Grok OAuth (SuperGrok Subscription) |
|
||||
| Auth type | Browser OAuth 2.0 PKCE (loopback callback) |
|
||||
| Transport | xAI Responses API (`codex_responses`) |
|
||||
| Default model | `grok-4.3` |
|
||||
| Endpoint | `https://api.x.ai/v1` |
|
||||
| Auth server | `https://accounts.x.ai` |
|
||||
| Requires env var | No (`XAI_API_KEY` is **not** used for this provider) |
|
||||
| Subscription | [SuperGrok](https://x.ai/grok) (any active tier) |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.9+
|
||||
- Hermes Agent installed
|
||||
- An active SuperGrok subscription on your xAI account
|
||||
- A browser available on the local machine (or use `--no-browser` for remote sessions)
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Launch the provider and model picker
|
||||
hermes model
|
||||
# → Select "xAI Grok OAuth (SuperGrok Subscription)" from the provider list
|
||||
# → Hermes opens your browser to accounts.x.ai
|
||||
# → Approve access in the browser
|
||||
# → Pick a model (grok-4.3 is at the top)
|
||||
# → Start chatting
|
||||
|
||||
hermes
|
||||
```
|
||||
|
||||
After the first login, credentials are stored under `~/.hermes/auth.json` and refreshed automatically before they expire.
|
||||
|
||||
## Logging In Manually
|
||||
|
||||
You can trigger a login without going through the model picker:
|
||||
|
||||
```bash
|
||||
hermes auth add xai-oauth
|
||||
```
|
||||
|
||||
### Remote / headless sessions
|
||||
|
||||
On servers, containers, or SSH sessions where no browser is available, Hermes detects the remote environment and prints the authorization URL instead of opening a browser. Open the URL on any device with a browser, complete the consent flow, and Hermes finishes the loopback exchange when the redirect comes back.
|
||||
|
||||
If you need to force this behaviour explicitly:
|
||||
|
||||
```bash
|
||||
hermes auth add xai-oauth --no-browser
|
||||
```
|
||||
|
||||
## How the Login Works
|
||||
|
||||
1. Hermes opens your browser to `accounts.x.ai`.
|
||||
2. You sign in (or confirm your existing session) and approve access.
|
||||
3. xAI redirects back to Hermes and the tokens are saved to `~/.hermes/auth.json`.
|
||||
4. From then on, Hermes refreshes the access token in the background — you stay signed in until you `hermes auth remove xai-oauth` or revoke access from your xAI account settings.
|
||||
|
||||
## Checking Login Status
|
||||
|
||||
```bash
|
||||
hermes doctor
|
||||
```
|
||||
|
||||
The `◆ Auth Providers` section will show the current state of every provider, including `xai-oauth`.
|
||||
|
||||
## Switching Models
|
||||
|
||||
```bash
|
||||
hermes model
|
||||
# → Select "xAI Grok OAuth (SuperGrok Subscription)"
|
||||
# → Pick from the model list (grok-4.3 is pinned to the top)
|
||||
```
|
||||
|
||||
Or set the model directly:
|
||||
|
||||
```bash
|
||||
hermes config set model.default grok-4.3
|
||||
hermes config set model.provider xai-oauth
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
After login, `~/.hermes/config.yaml` will contain:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
default: grok-4.3
|
||||
provider: xai-oauth
|
||||
base_url: https://api.x.ai/v1
|
||||
```
|
||||
|
||||
### Provider aliases
|
||||
|
||||
All of the following resolve to `xai-oauth`:
|
||||
|
||||
```bash
|
||||
hermes --provider xai-oauth # canonical
|
||||
hermes --provider grok-oauth # alias
|
||||
hermes --provider x-ai-oauth # alias
|
||||
hermes --provider xai-grok-oauth # alias
|
||||
```
|
||||
|
||||
## Direct-to-xAI Tools (TTS / Image / Video / Transcription)
|
||||
|
||||
Once you're logged in via OAuth, every direct-to-xAI tool reuses the same bearer token automatically — there is **no separate setup** unless you'd rather use an API key.
|
||||
|
||||
To pick a backend for each tool:
|
||||
|
||||
```bash
|
||||
hermes tools
|
||||
# → Text-to-Speech → "xAI TTS"
|
||||
# → Image Generation → "xAI Grok Imagine (image)"
|
||||
# → Video Generation → "xAI Grok Imagine"
|
||||
```
|
||||
|
||||
If OAuth tokens are already stored, the picker confirms it and skips the credential prompt. If neither OAuth nor `XAI_API_KEY` is set, the picker offers a 3-choice menu: OAuth login, paste API key, or skip.
|
||||
|
||||
:::note Video generation is off by default
|
||||
The `video_gen` toolset is disabled by default. Enable it in `hermes tools` → `🎬 Video Generation` (press space) before the agent can call `video_generate`. Otherwise the agent may fall back to the bundled ComfyUI skill, which is also tagged for video generation.
|
||||
:::
|
||||
|
||||
### Models
|
||||
|
||||
| Tool | Model | Notes |
|
||||
|------|-------|-------|
|
||||
| Chat | `grok-4.3` | Default; auto-selected when you log in via OAuth |
|
||||
| Chat | `grok-4.20-0309-reasoning` | Reasoning variant |
|
||||
| Chat | `grok-4.20-0309-non-reasoning` | Non-reasoning variant |
|
||||
| Chat | `grok-4.20-multi-agent-0309` | Multi-agent variant |
|
||||
| Image | `grok-imagine-image` | Default; ~5–10 s |
|
||||
| Image | `grok-imagine-image-quality` | Higher fidelity; ~10–20 s |
|
||||
| Video | `grok-imagine-video` | Text-to-video and image-to-video; up to 7 reference images |
|
||||
| TTS | (default voice) | xAI `/v1/tts` endpoint |
|
||||
|
||||
The chat catalog is derived live from the on-disk `models.dev` cache; new xAI releases appear automatically once that cache refreshes. `grok-4.3` is always pinned to the top of the list.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Effect |
|
||||
|----------|--------|
|
||||
| `XAI_BASE_URL` | Override the default `https://api.x.ai/v1` endpoint (rarely needed). |
|
||||
| `HERMES_INFERENCE_PROVIDER` | Force the active provider at runtime, e.g. `HERMES_INFERENCE_PROVIDER=xai-oauth hermes`. |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Token expired — not re-logging in automatically
|
||||
|
||||
Hermes refreshes the token before each session and again reactively on a 401. If refresh fails with `invalid_grant` (the refresh token was revoked, or the account was rotated), Hermes surfaces a typed re-auth message instead of crashing.
|
||||
|
||||
**Fix:** run `hermes auth add xai-oauth` again to start a fresh login.
|
||||
|
||||
### Authorization timed out
|
||||
|
||||
The loopback listener has a finite expiry window (default 180 s). If you don't approve the login in time, Hermes raises a timeout error.
|
||||
|
||||
**Fix:** re-run `hermes auth add xai-oauth` (or `hermes model`). The flow starts fresh.
|
||||
|
||||
### State mismatch (possible CSRF)
|
||||
|
||||
Hermes detected that the `state` value returned by the authorization server doesn't match what it sent.
|
||||
|
||||
**Fix:** re-run the login. If it persists, check for a proxy or redirect that is modifying the OAuth response.
|
||||
|
||||
### Logging in from a remote server
|
||||
|
||||
On SSH or container sessions Hermes prints the authorization URL instead of opening a browser. Open the URL on any device with a browser and complete the consent there — the loopback callback comes back to your remote host.
|
||||
|
||||
You can also force this behaviour:
|
||||
|
||||
```bash
|
||||
hermes auth add xai-oauth --no-browser
|
||||
```
|
||||
|
||||
### "No xAI credentials found" error at runtime
|
||||
|
||||
The auth store has no `xai-oauth` entry and no `XAI_API_KEY` is set. You haven't logged in yet, or the credential file was deleted.
|
||||
|
||||
**Fix:** run `hermes model` and pick the xAI Grok OAuth provider, or run `hermes auth add xai-oauth`.
|
||||
|
||||
## Logging Out
|
||||
|
||||
To remove all stored xAI Grok OAuth credentials:
|
||||
|
||||
```bash
|
||||
hermes auth logout xai-oauth
|
||||
```
|
||||
|
||||
This clears both the singleton OAuth entry in `auth.json` and any credential-pool rows for `xai-oauth`. Use `hermes auth remove xai-oauth <index|id|label>` if you only want to drop a single pool entry (run `hermes auth list xai-oauth` to see them).
|
||||
|
||||
## See Also
|
||||
|
||||
- [AI Providers reference](../integrations/providers.md)
|
||||
- [Environment Variables](../reference/environment-variables.md)
|
||||
- [Configuration](../user-guide/configuration.md)
|
||||
- [Voice & TTS](../user-guide/features/tts.md)
|
||||
@@ -97,5 +97,4 @@ See the [Messaging Gateway overview](/docs/user-guide/messaging) for the platfor
|
||||
|
||||
## Training & Evaluation
|
||||
|
||||
- **[RL Training](/docs/user-guide/features/rl-training)** — Generate trajectory data from agent sessions for reinforcement learning and model fine-tuning. Supports Atropos environments with customizable reward functions.
|
||||
- **[Batch Processing](/docs/user-guide/features/batch-processing)** — Run the agent across hundreds of prompts in parallel, generating structured ShareGPT-format trajectory data for training data generation or evaluation.
|
||||
|
||||
@@ -20,6 +20,7 @@ You need at least one way to connect to an LLM. Use `hermes model` to switch pro
|
||||
| **GitHub Copilot ACP** | `hermes model` (spawns local `copilot --acp --stdio`) |
|
||||
| **Anthropic** | `hermes model` (Claude Max + extra usage credits via OAuth; also supports Anthropic API key or manual setup-token — see note below) |
|
||||
| **OpenRouter** | `OPENROUTER_API_KEY` in `~/.hermes/.env` |
|
||||
| **NovitaAI** | `NOVITA_API_KEY` in `~/.hermes/.env` (provider: `novita`, 200+ models, Model API, Agent Sandbox, GPU Cloud) |
|
||||
| **AI Gateway** | `AI_GATEWAY_API_KEY` in `~/.hermes/.env` (provider: `ai-gateway`) |
|
||||
| **z.ai / GLM** | `GLM_API_KEY` in `~/.hermes/.env` (provider: `zai`) |
|
||||
| **Kimi / Moonshot** | `KIMI_API_KEY` in `~/.hermes/.env` (provider: `kimi-coding`) |
|
||||
@@ -267,6 +268,10 @@ model:
|
||||
These providers have built-in support with dedicated provider IDs. Set the API key and use `--provider` to select:
|
||||
|
||||
```bash
|
||||
# NovitaAI Model API
|
||||
hermes chat --provider novita --model moonshotai/kimi-k2.5
|
||||
# Requires: NOVITA_API_KEY in ~/.hermes/.env
|
||||
|
||||
# z.ai / ZhipuAI GLM
|
||||
hermes chat --provider zai --model glm-5
|
||||
# Requires: GLM_API_KEY in ~/.hermes/.env
|
||||
@@ -316,7 +321,7 @@ model:
|
||||
default: "zai-org/GLM-5.1-FP8"
|
||||
```
|
||||
|
||||
Base URLs can be overridden with `GLM_BASE_URL`, `KIMI_BASE_URL`, `MINIMAX_BASE_URL`, `MINIMAX_CN_BASE_URL`, `DASHSCOPE_BASE_URL`, `XIAOMI_BASE_URL`, `GMI_BASE_URL`, or `TOKENHUB_BASE_URL` environment variables.
|
||||
Base URLs can be overridden with `NOVITA_BASE_URL`, `GLM_BASE_URL`, `KIMI_BASE_URL`, `MINIMAX_BASE_URL`, `MINIMAX_CN_BASE_URL`, `DASHSCOPE_BASE_URL`, `XIAOMI_BASE_URL`, `GMI_BASE_URL`, or `TOKENHUB_BASE_URL` environment variables.
|
||||
|
||||
:::note Z.AI Endpoint Auto-Detection
|
||||
When using the Z.AI / GLM provider, Hermes automatically probes multiple endpoints (global, China, coding variants) to find one that accepts your API key. You don't need to set `GLM_BASE_URL` manually — the working endpoint is detected and cached automatically.
|
||||
@@ -326,12 +331,37 @@ When using the Z.AI / GLM provider, Hermes automatically probes multiple endpoin
|
||||
|
||||
xAI is wired through the Responses API (`codex_responses` transport) for automatic reasoning support on Grok 4 models — no `reasoning_effort` parameter needed, the server reasons by default. Set `XAI_API_KEY` in `~/.hermes/.env` and pick xAI in `hermes model`, or drop `grok` as a shortcut into `/model grok-4-1-fast-reasoning`.
|
||||
|
||||
SuperGrok subscribers can sign in with browser OAuth instead of using an API key — pick **xAI Grok OAuth (SuperGrok Subscription)** in `hermes model`, or run `hermes auth add xai-oauth`. The same OAuth bearer token is automatically reused by direct-to-xAI tools (TTS, image gen, video gen, transcription). See the [xAI Grok OAuth guide](../guides/xai-grok-oauth.md) for the full flow.
|
||||
|
||||
When using xAI as a provider (any base URL containing `x.ai`), Hermes automatically enables prompt caching by sending the `x-grok-conv-id` header with every API request. This routes requests to the same server within a conversation session, allowing xAI's infrastructure to reuse cached system prompts and conversation history.
|
||||
|
||||
No configuration is needed — caching activates automatically when an xAI endpoint is detected and a session ID is available. This reduces latency and cost for multi-turn conversations.
|
||||
|
||||
xAI also ships a dedicated TTS endpoint (`/v1/tts`). Select **xAI TTS** in `hermes tools` → Voice & TTS, or see the [Voice & TTS](../user-guide/features/tts.md#text-to-speech) page for config.
|
||||
|
||||
### NovitaAI
|
||||
|
||||
[NovitaAI](https://novita.ai) is the AI-native cloud for builders and agents. Its three product lines are Model API for 200+ models, Agent Sandbox for building and running AI agents, and GPU Cloud for scalable compute, all available from one platform.
|
||||
|
||||
```bash
|
||||
# Use any available model
|
||||
hermes chat --provider novita --model moonshotai/kimi-k2.5
|
||||
# Requires: NOVITA_API_KEY in ~/.hermes/.env
|
||||
|
||||
# Short alias
|
||||
hermes chat --provider novita-ai --model deepseek/deepseek-v3-0324
|
||||
```
|
||||
|
||||
Or set it permanently in `config.yaml`:
|
||||
```yaml
|
||||
model:
|
||||
provider: "novita"
|
||||
default: "moonshotai/kimi-k2.5"
|
||||
base_url: "https://api.novita.ai/openai/v1"
|
||||
```
|
||||
|
||||
Get your API key at [novita.ai/settings/key-management](https://novita.ai/settings/key-management). The base URL can be overridden with `NOVITA_BASE_URL`.
|
||||
|
||||
### Ollama Cloud — Managed Ollama Models, OAuth + API Key
|
||||
|
||||
[Ollama Cloud](https://ollama.com/cloud) hosts the same open-weight catalog as local Ollama but without the GPU requirement. Pick it in `hermes model` as **Ollama Cloud**, paste your API key from [ollama.com/settings/keys](https://ollama.com/settings/keys), and Hermes auto-discovers the available models.
|
||||
@@ -1327,7 +1357,6 @@ You can switch between providers at any time with `hermes model` — no restart
|
||||
| Premium TTS voices | [ElevenLabs](https://elevenlabs.io/) | `ELEVENLABS_API_KEY` |
|
||||
| OpenAI TTS + voice transcription | [OpenAI](https://platform.openai.com/api-keys) | `VOICE_TOOLS_OPENAI_KEY` |
|
||||
| Mistral TTS + voice transcription | [Mistral](https://console.mistral.ai/) | `MISTRAL_API_KEY` |
|
||||
| RL Training | [Tinker](https://tinker-console.thinkingmachines.ai/) + [WandB](https://wandb.ai/) | `TINKER_API_KEY`, `WANDB_API_KEY` |
|
||||
| Cross-session user modeling | [Honcho](https://honcho.dev/) | `HONCHO_API_KEY` |
|
||||
| Semantic long-term memory | [Supermemory](https://supermemory.ai) | `SUPERMEMORY_API_KEY` |
|
||||
|
||||
@@ -1417,7 +1446,7 @@ fallback_model:
|
||||
|
||||
When activated, the fallback swaps the model and provider mid-session without losing your conversation. The chain is tried entry-by-entry; activation is one-shot per session.
|
||||
|
||||
Supported providers: `openrouter`, `nous`, `openai-codex`, `copilot`, `copilot-acp`, `anthropic`, `gemini`, `google-gemini-cli`, `qwen-oauth`, `huggingface`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth`, `deepseek`, `nvidia`, `xai`, `ollama-cloud`, `bedrock`, `ai-gateway`, `azure-foundry`, `opencode-zen`, `opencode-go`, `kilocode`, `xiaomi`, `arcee`, `gmi`, `stepfun`, `lmstudio`, `alibaba`, `alibaba-coding-plan`, `tencent-tokenhub`, `custom`.
|
||||
Supported providers: `openrouter`, `nous`, `openai-codex`, `copilot`, `copilot-acp`, `anthropic`, `gemini`, `google-gemini-cli`, `qwen-oauth`, `huggingface`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth`, `deepseek`, `nvidia`, `xai`, `xai-oauth`, `ollama-cloud`, `bedrock`, `ai-gateway`, `azure-foundry`, `opencode-zen`, `opencode-go`, `kilocode`, `xiaomi`, `arcee`, `gmi`, `stepfun`, `lmstudio`, `alibaba`, `alibaba-coding-plan`, `tencent-tokenhub`, `custom`.
|
||||
|
||||
:::tip
|
||||
Fallback is configured exclusively through `config.yaml` — or interactively via `hermes fallback`. For full details on when it triggers, how the chain advances, and how it interacts with auxiliary tasks and delegation, see [Fallback Providers](/docs/user-guide/features/fallback-providers).
|
||||
|
||||
@@ -40,6 +40,7 @@ hermes [global-options] <command> [subcommand/options]
|
||||
| `hermes model` | Interactively choose the default provider and model. |
|
||||
| `hermes fallback` | Manage fallback providers tried when the primary model errors. |
|
||||
| `hermes gateway` | Run or manage the messaging gateway service. |
|
||||
| `hermes proxy` | Local OpenAI-compatible proxy that attaches OAuth provider credentials. See [Subscription Proxy](../user-guide/features/subscription-proxy.md). |
|
||||
| `hermes lsp` | Manage Language Server Protocol integration (semantic diagnostics for write_file/patch). |
|
||||
| `hermes setup` | Interactive setup wizard for all or part of the configuration. |
|
||||
| `hermes whatsapp` | Configure and pair the WhatsApp bridge. |
|
||||
@@ -91,7 +92,7 @@ Common options:
|
||||
| `-q`, `--query "..."` | One-shot, non-interactive prompt. |
|
||||
| `-m`, `--model <model>` | Override the model for this run. |
|
||||
| `-t`, `--toolsets <csv>` | Enable a comma-separated set of toolsets. |
|
||||
| `--provider <provider>` | Force a provider: `auto`, `openrouter`, `nous`, `openai-codex`, `copilot-acp`, `copilot`, `anthropic`, `gemini`, `google-gemini-cli`, `huggingface`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth`, `kilocode`, `xiaomi`, `arcee`, `gmi`, `alibaba`, `alibaba-coding-plan` (alias `alibaba_coding`), `deepseek`, `nvidia`, `ollama-cloud`, `xai` (alias `grok`), `qwen-oauth`, `bedrock`, `opencode-zen`, `opencode-go`, `ai-gateway`, `azure-foundry`, `lmstudio`, `stepfun`, `tencent-tokenhub` (alias `tencent`, `tokenhub`). |
|
||||
| `--provider <provider>` | Force a provider: `auto`, `openrouter`, `nous`, `openai-codex`, `copilot-acp`, `copilot`, `anthropic`, `gemini`, `google-gemini-cli`, `huggingface`, `novita`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth`, `kilocode`, `xiaomi`, `arcee`, `gmi`, `alibaba`, `alibaba-coding-plan` (alias `alibaba_coding`), `deepseek`, `nvidia`, `ollama-cloud`, `xai` (alias `grok`), `xai-oauth` (alias `grok-oauth`), `qwen-oauth`, `bedrock`, `opencode-zen`, `opencode-go`, `ai-gateway`, `azure-foundry`, `lmstudio`, `stepfun`, `tencent-tokenhub` (alias `tencent`, `tokenhub`). |
|
||||
| `-s`, `--skills <name>` | Preload one or more skills for the session (can be repeated or comma-separated). |
|
||||
| `-v`, `--verbose` | Verbose output. |
|
||||
| `-Q`, `--quiet` | Programmatic mode: suppress banner/spinner/tool previews. |
|
||||
|
||||
@@ -67,6 +67,8 @@ All variables go in `~/.hermes/.env`. You can also set them with `hermes config
|
||||
| `DASHSCOPE_BASE_URL` | Custom DashScope base URL (default: `https://dashscope-intl.aliyuncs.com/compatible-mode/v1`; use `https://dashscope.aliyuncs.com/compatible-mode/v1` for mainland-China region) |
|
||||
| `DEEPSEEK_API_KEY` | DeepSeek API key for direct DeepSeek access ([platform.deepseek.com](https://platform.deepseek.com/api_keys)) |
|
||||
| `DEEPSEEK_BASE_URL` | Custom DeepSeek API base URL |
|
||||
| `NOVITA_API_KEY` | NovitaAI API key — AI-native cloud for Model API, Agent Sandbox, and GPU Cloud ([novita.ai/settings/key-management](https://novita.ai/settings/key-management)) |
|
||||
| `NOVITA_BASE_URL` | Override NovitaAI base URL (default: `https://api.novita.ai/openai/v1`) |
|
||||
| `NVIDIA_API_KEY` | NVIDIA NIM API key — Nemotron and open models ([build.nvidia.com](https://build.nvidia.com)) |
|
||||
| `NVIDIA_BASE_URL` | Override NVIDIA base URL (default: `https://integrate.api.nvidia.com/v1`; set to `http://localhost:8000/v1` for a local NIM endpoint) |
|
||||
| `STEPFUN_API_KEY` | StepFun API key — Step-series models ([platform.stepfun.com](https://platform.stepfun.com)) |
|
||||
@@ -103,7 +105,7 @@ For native Anthropic auth, Hermes prefers Claude Code's own credential files whe
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `HERMES_INFERENCE_PROVIDER` | Override provider selection: `auto`, `custom`, `openrouter`, `nous`, `openai-codex`, `copilot`, `copilot-acp`, `anthropic`, `huggingface`, `gemini`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth` (browser OAuth login — no API key required; see [MiniMax OAuth guide](../guides/minimax-oauth.md)), `kilocode`, `xiaomi`, `arcee`, `gmi`, `stepfun`, `alibaba`, `alibaba-coding-plan` (alias `alibaba_coding`), `deepseek`, `nvidia`, `ollama-cloud`, `xai` (alias `grok`), `google-gemini-cli`, `qwen-oauth`, `bedrock`, `opencode-zen`, `opencode-go`, `ai-gateway`, `tencent-tokenhub` (default: `auto`) |
|
||||
| `HERMES_INFERENCE_PROVIDER` | Override provider selection: `auto`, `custom`, `openrouter`, `nous`, `openai-codex`, `copilot`, `copilot-acp`, `anthropic`, `huggingface`, `novita`, `gemini`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth` (browser OAuth login — no API key required; see [MiniMax OAuth guide](../guides/minimax-oauth.md)), `kilocode`, `xiaomi`, `arcee`, `gmi`, `stepfun`, `alibaba`, `alibaba-coding-plan` (alias `alibaba_coding`), `deepseek`, `nvidia`, `ollama-cloud`, `xai` (alias `grok`), `xai-oauth` (browser OAuth login for SuperGrok subscribers — no API key required; see [xAI Grok OAuth guide](../guides/xai-grok-oauth.md)), `google-gemini-cli`, `qwen-oauth`, `bedrock`, `opencode-zen`, `opencode-go`, `ai-gateway`, `tencent-tokenhub` (default: `auto`) |
|
||||
| `HERMES_PORTAL_BASE_URL` | Override Nous Portal URL (for development/testing) |
|
||||
| `NOUS_INFERENCE_BASE_URL` | Override Nous inference API URL |
|
||||
| `HERMES_NOUS_MIN_KEY_TTL_SECONDS` | Min agent key TTL before re-mint (default: 1800 = 30min) |
|
||||
@@ -133,6 +135,7 @@ For native Anthropic auth, Hermes prefers Claude Code's own credential files whe
|
||||
| `CAMOFOX_SESSION_KEY` | Optional Camofox session key used when creating tabs for `CAMOFOX_USER_ID` |
|
||||
| `CAMOFOX_ADOPT_EXISTING_TAB` | Set to `true` to reuse an existing Camofox tab before creating a new one |
|
||||
| `BROWSER_INACTIVITY_TIMEOUT` | Browser session inactivity timeout in seconds |
|
||||
| `AGENT_BROWSER_ARGS` | Extra Chromium launch flags (comma- or newline-separated). Hermes auto-injects `--no-sandbox,--disable-dev-shm-usage` when running as root or on AppArmor-restricted unprivileged user namespaces (Ubuntu 23.10+, DGX Spark, many container images); set this manually only to override or add other flags. |
|
||||
| `FAL_KEY` | Image generation ([fal.ai](https://fal.ai/)) |
|
||||
| `GROQ_API_KEY` | Groq Whisper STT API key ([groq.com](https://groq.com/)) |
|
||||
| `ELEVENLABS_API_KEY` | ElevenLabs premium TTS voices ([elevenlabs.io](https://elevenlabs.io/)) |
|
||||
@@ -145,8 +148,6 @@ For native Anthropic auth, Hermes prefers Claude Code's own credential files whe
|
||||
| `HONCHO_BASE_URL` | Base URL for self-hosted Honcho instances (default: Honcho cloud). No API key required for local instances |
|
||||
| `HINDSIGHT_TIMEOUT` | Timeout in seconds for Hindsight memory-provider API calls (default: `60`). Bump this if your Hindsight instance is slow to respond during `/sync` or `on_session_switch` and you're seeing timeouts in `errors.log`. |
|
||||
| `SUPERMEMORY_API_KEY` | Semantic long-term memory with profile recall and session ingest ([supermemory.ai](https://supermemory.ai)) |
|
||||
| `TINKER_API_KEY` | RL training ([tinker-console.thinkingmachines.ai](https://tinker-console.thinkingmachines.ai/)) |
|
||||
| `WANDB_API_KEY` | RL training metrics ([wandb.ai](https://wandb.ai/)) |
|
||||
| `DAYTONA_API_KEY` | Daytona cloud sandboxes ([daytona.io](https://daytona.io/)) |
|
||||
| `VERCEL_TOKEN` | Vercel Sandbox access token ([vercel.com](https://vercel.com/)) |
|
||||
| `VERCEL_PROJECT_ID` | Vercel project ID (required with `VERCEL_TOKEN`) |
|
||||
@@ -515,6 +516,8 @@ Advanced per-platform knobs for throttling the outbound message batcher. Most us
|
||||
| `HERMES_HUMAN_DELAY_MIN_MS` | Custom delay range minimum (ms) |
|
||||
| `HERMES_HUMAN_DELAY_MAX_MS` | Custom delay range maximum (ms) |
|
||||
| `HERMES_QUIET` | Suppress non-essential output (`true`/`false`) |
|
||||
| `CODEX_HOME` | When [Codex app-server runtime](../user-guide/features/codex-app-server-runtime) is enabled, override the directory Codex CLI reads its config + auth from (default: `~/.codex`). Hermes' migration writes the managed block to `<CODEX_HOME>/config.toml`. |
|
||||
| `HERMES_KANBAN_TASK` | Set by the kanban dispatcher when spawning a worker (task UUID). Workers and the spawned `hermes-tools` MCP subprocess inherit it so kanban tools gate correctly. Don't set manually. |
|
||||
| `HERMES_API_TIMEOUT` | LLM API call timeout in seconds (default: `1800`) |
|
||||
| `HERMES_API_CALL_STALE_TIMEOUT` | Non-streaming stale-call timeout in seconds (default: `300`). Auto-disabled for local providers when left unset. Also configurable via `providers.<id>.stale_timeout_seconds` or `providers.<id>.models.<model>.stale_timeout_seconds` in `config.yaml`. |
|
||||
| `HERMES_STREAM_READ_TIMEOUT` | Streaming socket read timeout in seconds (default: `120`). Auto-increased to `HERMES_API_TIMEOUT` for local providers. Increase if local LLMs time out during long code generation. |
|
||||
|
||||
@@ -38,7 +38,7 @@ hermes skills uninstall <skill-name>
|
||||
|
||||
| Skill | Description |
|
||||
|-------|-------------|
|
||||
| [**base**](/docs/user-guide/skills/optional/blockchain/blockchain-base) | Query Base (Ethereum L2) blockchain data with USD pricing — wallet balances, token info, transaction details, gas analysis, contract inspection, whale detection, and live network stats. Uses Base RPC + CoinGecko. No API key required. |
|
||||
| [**evm**](/docs/user-guide/skills/optional/blockchain/blockchain-evm) | Read-only EVM client: wallets, tokens, gas across 8 chains. |
|
||||
| [**solana**](/docs/user-guide/skills/optional/blockchain/blockchain-solana) | Query Solana blockchain data with USD pricing — wallet balances, token portfolios with values, transaction details, NFTs, whale detection, and live network stats. Uses Solana RPC + CoinGecko. No API key required. |
|
||||
|
||||
## communication
|
||||
@@ -120,7 +120,6 @@ hermes skills uninstall <skill-name>
|
||||
| [**faiss**](/docs/user-guide/skills/optional/mlops/mlops-faiss) | Facebook's library for efficient similarity search and clustering of dense vectors. Supports billions of vectors, GPU acceleration, and various index types (Flat, IVF, HNSW). Use for fast k-NN search, large-scale vector retrieval, or whe... |
|
||||
| [**optimizing-attention-flash**](/docs/user-guide/skills/optional/mlops/mlops-flash-attention) | Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training/running transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster in... |
|
||||
| [**guidance**](/docs/user-guide/skills/optional/mlops/mlops-guidance) | Control LLM output with regex and grammars, guarantee valid JSON/XML/code generation, enforce structured formats, and build multi-step workflows with Guidance - Microsoft Research's constrained generation framework |
|
||||
| [**hermes-atropos-environments**](/docs/user-guide/skills/optional/mlops/mlops-hermes-atropos-environments) | Build, test, and debug Hermes Agent RL environments for Atropos training. Covers the HermesAgentBaseEnv interface, reward functions, agent loop integration, evaluation with tools, wandb logging, and the three CLI modes (serve/process/eva... |
|
||||
| [**huggingface-tokenizers**](/docs/user-guide/skills/optional/mlops/mlops-huggingface-tokenizers) | Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integ... |
|
||||
| [**instructor**](/docs/user-guide/skills/optional/mlops/mlops-instructor) | Extract structured data from LLM responses with Pydantic validation, retry failed extractions automatically, parse complex JSON with type safety, and stream partial results with Instructor - battle-tested structured output library |
|
||||
| [**lambda-labs-gpu-cloud**](/docs/user-guide/skills/optional/mlops/mlops-lambda-labs) | Reserved and on-demand GPU cloud instances for ML training and inference. Use when you need dedicated GPU instances with simple SSH access, persistent filesystems, or high-performance multi-node clusters for large-scale training. |
|
||||
|
||||
@@ -50,6 +50,7 @@ Type `/` in the CLI to open the autocomplete menu. Built-in commands are case-in
|
||||
|---------|-------------|
|
||||
| `/config` | Show current configuration |
|
||||
| `/model [model-name]` | Show or change the current model. Supports: `/model claude-sonnet-4`, `/model provider:model` (switch providers), `/model custom:model` (custom endpoint), `/model custom:name:model` (named custom provider), `/model custom` (auto-detect from endpoint), and user-defined aliases (`/model fav`, `/model grok` — see [Custom model aliases](#custom-model-aliases)). Use `--global` to persist the change to config.yaml. **Note:** `/model` can only switch between already-configured providers. To add a new provider, exit the session and run `hermes model` from your terminal. |
|
||||
| `/codex-runtime [auto\|codex_app_server\|on\|off]` | Toggle the optional [Codex app-server runtime](../user-guide/features/codex-app-server-runtime) for OpenAI/Codex models. `auto` (default) uses Hermes' standard chat completions; `codex_app_server` hands turns to a `codex app-server` subprocess for native shell, apply_patch, ChatGPT subscription auth, and migrated Codex plugins. Effective on next session. |
|
||||
| `/personality` | Set a predefined personality |
|
||||
| `/verbose` | Cycle tool progress display: off → new → all → verbose. Can be [enabled for messaging](#notes) via config. |
|
||||
| `/fast [normal\|fast\|status]` | Toggle fast mode — OpenAI Priority Processing / Anthropic Fast Mode. Options: `normal`, `fast`, `status`. |
|
||||
@@ -180,6 +181,7 @@ The messaging gateway supports the following built-in commands inside Telegram,
|
||||
| `/status` | Show session info. |
|
||||
| `/stop` | Kill all running background processes and interrupt the running agent. |
|
||||
| `/model [provider:model]` | Show or change the model. Supports provider switches (`/model zai:glm-5`), custom endpoints (`/model custom:model`), named custom providers (`/model custom:local:qwen`), auto-detect (`/model custom`), and user-defined aliases (`/model fav`, `/model grok` — see [Custom model aliases](#custom-model-aliases)). Use `--global` to persist the change to config.yaml. **Note:** `/model` can only switch between already-configured providers. To add a new provider or set up API keys, use `hermes model` from your terminal (outside the chat session). |
|
||||
| `/codex-runtime [auto\|codex_app_server\|on\|off]` | Toggle the optional [Codex app-server runtime](../user-guide/features/codex-app-server-runtime). Persists to `model.openai_runtime` in config.yaml and evicts the cached agent so the next message picks up the new runtime. Effective on next session. |
|
||||
| `/personality [name]` | Set a personality overlay for the session. |
|
||||
| `/fast [normal\|fast\|status]` | Toggle fast mode — OpenAI Priority Processing / Anthropic Fast Mode. |
|
||||
| `/retry` | Retry the last message. |
|
||||
|
||||
@@ -148,21 +148,6 @@ Registered only when the agent is spawned by the kanban dispatcher (`HERMES_KANB
|
||||
|------|-------------|----------------------|
|
||||
| `mixture_of_agents` | Route a hard problem through multiple frontier LLMs collaboratively. Makes 5 API calls (4 reference models + 1 aggregator) with maximum reasoning effort — use sparingly for genuinely difficult problems. Best for: complex math, advanced alg… | OPENROUTER_API_KEY |
|
||||
|
||||
## `rl` toolset
|
||||
|
||||
| Tool | Description | Requires environment |
|
||||
|------|-------------|----------------------|
|
||||
| `rl_check_status` | Get status and metrics for a training run. RATE LIMITED: enforces 30-minute minimum between checks for the same run. Returns WandB metrics: step, state, reward_mean, loss, percent_correct. | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_edit_config` | Update a configuration field. Use rl_get_current_config() first to see all available fields for the selected environment. Each environment has different configurable options. Infrastructure settings (tokenizer, URLs, lora_rank, learning_ra… | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_get_current_config` | Get the current environment configuration. Returns only fields that can be modified: group_size, max_token_length, total_steps, steps_per_eval, use_wandb, wandb_name, max_num_workers. | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_get_results` | Get final results and metrics for a completed training run. Returns final metrics and path to trained weights. | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_list_environments` | List all available RL environments. Returns environment names, paths, and descriptions. TIP: Read the file_path with file tools to understand how each environment works (verifiers, data loading, rewards). | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_list_runs` | List all training runs (active and completed) with their status. | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_select_environment` | Select an RL environment for training. Loads the environment's default configuration. After selecting, use rl_get_current_config() to see settings and rl_edit_config() to modify them. | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_start_training` | Start a new RL training run with the current environment and config. Most training parameters (lora_rank, learning_rate, etc.) are fixed. Use rl_edit_config() to set group_size, batch_size, wandb_project before starting. WARNING: Training… | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_stop_training` | Stop a running training job. Use if metrics look bad, training is stagnant, or you want to try different settings. | TINKER_API_KEY, WANDB_API_KEY |
|
||||
| `rl_test_inference` | Quick inference test for any environment. Runs a few steps of inference + scoring using OpenRouter. Default: 3 steps x 16 completions = 48 rollouts per model, testing 3 models = 144 total. Tests environment loading, prompt construction, in… | TINKER_API_KEY, WANDB_API_KEY |
|
||||
|
||||
## `session_search` toolset
|
||||
|
||||
| Tool | Description | Requires environment |
|
||||
|
||||
@@ -45,7 +45,7 @@ Or in-session:
|
||||
```
|
||||
/tools list
|
||||
/tools disable browser
|
||||
/tools enable rl
|
||||
/tools enable homeassistant
|
||||
```
|
||||
|
||||
## Core Toolsets
|
||||
@@ -66,11 +66,11 @@ Or in-session:
|
||||
| `homeassistant` | `ha_call_service`, `ha_get_state`, `ha_list_entities`, `ha_list_services` | Smart home control via Home Assistant. Only available when `HASS_TOKEN` is set. |
|
||||
| `computer_use` | `computer_use` | Background macOS desktop control via cua-driver — does not steal cursor/focus. Works with any tool-capable model. macOS only; requires `cua-driver` on `$PATH`. |
|
||||
| `image_gen` | `image_generate` | Text-to-image generation via FAL.ai (with opt-in OpenAI / xAI backends). |
|
||||
| `video_gen` | `video_generate` | Text-to-video and image-to-video via plugin-registered backends (xAI Grok-Imagine, FAL.ai Veo 3.1 / Pixverse v6 / Kling O3). Pass `image_url` to animate an image; omit it for text-to-video. |
|
||||
| `kanban` | `kanban_block`, `kanban_comment`, `kanban_complete`, `kanban_create`, `kanban_heartbeat`, `kanban_link`, `kanban_show` | Multi-agent coordination tools — only registered when the agent is spawned by the kanban dispatcher (`HERMES_KANBAN_TASK` env set). Lets workers mark tasks done with structured handoffs, block for human input, heartbeat during long ops, comment on threads, and (for orchestrators) fan out into child tasks. |
|
||||
| `memory` | `memory` | Persistent cross-session memory management. |
|
||||
| `messaging` | `send_message` | Send messages to other platforms (Telegram, Discord, etc.) from within a session. |
|
||||
| `moa` | `mixture_of_agents` | Multi-model consensus via Mixture of Agents. |
|
||||
| `rl` | `rl_check_status`, `rl_edit_config`, `rl_get_current_config`, `rl_get_results`, `rl_list_environments`, `rl_list_runs`, `rl_select_environment`, `rl_start_training`, `rl_stop_training`, `rl_test_inference` | RL training environment management (Atropos). |
|
||||
| `safe` | `image_generate`, `vision_analyze`, `web_extract`, `web_search` (via `includes`) | Read-only research + media generation. No file writes, no terminal, no code execution. |
|
||||
| `search` | `web_search` | Web search only (without extract). |
|
||||
| `session_search` | `session_search` | Search past conversation sessions. |
|
||||
|
||||
@@ -813,12 +813,16 @@ Every model slot in Hermes — auxiliary tasks, compression, fallback — uses t
|
||||
|
||||
When `base_url` is set, Hermes ignores the provider and calls that endpoint directly (using `api_key` or `OPENAI_API_KEY` for auth). When only `provider` is set, Hermes uses that provider's built-in auth and base URL.
|
||||
|
||||
Available providers for auxiliary tasks: `auto`, `main`, plus any provider in the [provider registry](/docs/reference/environment-variables) — `openrouter`, `nous`, `openai-codex`, `copilot`, `copilot-acp`, `anthropic`, `gemini`, `google-gemini-cli`, `qwen-oauth`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth`, `deepseek`, `nvidia`, `xai`, `ollama-cloud`, `alibaba`, `bedrock`, `huggingface`, `arcee`, `xiaomi`, `kilocode`, `opencode-zen`, `opencode-go`, `ai-gateway`, `azure-foundry` — or any named custom provider from your `custom_providers` list (e.g. `provider: "beans"`).
|
||||
Available providers for auxiliary tasks: `auto`, `main`, plus any provider in the [provider registry](/docs/reference/environment-variables) — `openrouter`, `nous`, `openai-codex`, `copilot`, `copilot-acp`, `anthropic`, `gemini`, `google-gemini-cli`, `qwen-oauth`, `zai`, `kimi-coding`, `kimi-coding-cn`, `minimax`, `minimax-cn`, `minimax-oauth`, `deepseek`, `nvidia`, `xai`, `xai-oauth`, `ollama-cloud`, `alibaba`, `bedrock`, `huggingface`, `arcee`, `xiaomi`, `kilocode`, `opencode-zen`, `opencode-go`, `ai-gateway`, `azure-foundry` — or any named custom provider from your `custom_providers` list (e.g. `provider: "beans"`).
|
||||
|
||||
:::tip MiniMax OAuth
|
||||
`minimax-oauth` logs in via browser OAuth (no API key needed). Run `hermes model` and select **MiniMax (OAuth)** to authenticate. Auxiliary tasks use `MiniMax-M2.7-highspeed` automatically. See the [MiniMax OAuth guide](../guides/minimax-oauth.md).
|
||||
:::
|
||||
|
||||
:::tip xAI Grok OAuth
|
||||
`xai-oauth` logs in via browser OAuth for SuperGrok subscribers (no API key needed). Run `hermes model` and select **xAI Grok OAuth (SuperGrok Subscription)** to authenticate. The same OAuth token is reused for every direct-to-xAI surface (chat, auxiliary tasks, TTS, image gen, video gen, transcription). See the [xAI Grok OAuth guide](../guides/xai-grok-oauth.md).
|
||||
:::
|
||||
|
||||
:::warning `"main"` is for auxiliary tasks only
|
||||
The `"main"` provider option means "use whatever provider my main agent uses" — it's only valid inside `auxiliary:`, `compression:`, and `fallback_model:` configs. It is **not** a valid value for your top-level `model.provider` setting. If you use a custom OpenAI-compatible endpoint, set `provider: custom` in your `model:` section. See [AI Providers](/docs/integrations/providers) for all main model provider options.
|
||||
:::
|
||||
@@ -980,6 +984,7 @@ These options apply to **auxiliary task configs** (`auxiliary:`, `compression:`,
|
||||
| `"nous"` | Force Nous Portal | `hermes auth` |
|
||||
| `"codex"` | Force Codex OAuth (ChatGPT account). Supports vision (gpt-5.3-codex). | `hermes model` → Codex |
|
||||
| `"minimax-oauth"` | Force MiniMax OAuth (browser login, no API key). Uses MiniMax-M2.7-highspeed for auxiliary tasks. | `hermes model` → MiniMax (OAuth) |
|
||||
| `"xai-oauth"` | Force xAI Grok OAuth (browser login for SuperGrok subscribers, no API key). Same OAuth token covers chat, TTS, image, video, and transcription. | `hermes model` → xAI Grok OAuth (SuperGrok Subscription) |
|
||||
| `"main"` | Use your active custom/main endpoint. This can come from `OPENAI_BASE_URL` + `OPENAI_API_KEY` or from a custom endpoint saved via `hermes model` / `config.yaml`. Works with OpenAI, local models, or any OpenAI-compatible API. **Auxiliary tasks only — not valid for `model.provider`.** | Custom endpoint credentials + base URL |
|
||||
|
||||
Direct API-key providers from the main provider catalog also work here when you want side tasks to bypass your default router. `gmi` is valid once `GMI_API_KEY` is configured:
|
||||
@@ -1588,7 +1593,7 @@ security:
|
||||
```
|
||||
|
||||
- `redact_secrets` — when `true`, automatically detects and redacts patterns that look like API keys, tokens, and passwords in tool output before it enters the conversation context and logs. **Off by default** — enable if you commonly work with real credentials in tool output and want a safety net. Set to `true` explicitly to turn on.
|
||||
- `tirith_enabled` — when `true`, terminal commands are scanned by [Tirith](https://github.com/StackGuardian/tirith) before execution to detect potentially dangerous operations.
|
||||
- `tirith_enabled` — when `true`, terminal commands are scanned by [Tirith](https://github.com/sheeki03/tirith) before execution to detect potentially dangerous operations.
|
||||
- `tirith_path` — path to the tirith binary. Set this if tirith is installed in a non-standard location.
|
||||
- `tirith_timeout` — maximum seconds to wait for a tirith scan. Commands proceed if the scan times out.
|
||||
- `tirith_fail_open` — when `true` (default), commands are allowed to execute if tirith is unavailable or fails. Set to `false` to block commands when tirith cannot verify them.
|
||||
|
||||
@@ -45,6 +45,14 @@ This installs the `agent-client-protocol` dependency and enables:
|
||||
- `hermes-acp`
|
||||
- `python -m acp_adapter`
|
||||
|
||||
For Zed registry installs, Zed launches Hermes through the official ACP Registry entry. That entry uses a `uvx` distribution that runs:
|
||||
|
||||
```bash
|
||||
uvx --from 'hermes-agent[acp]==<version>' hermes-acp
|
||||
```
|
||||
|
||||
Make sure `uv` is available on `PATH` before using the registry install path.
|
||||
|
||||
## Launching the ACP server
|
||||
|
||||
Any of the following starts Hermes in ACP mode:
|
||||
@@ -63,6 +71,34 @@ python -m acp_adapter
|
||||
|
||||
Hermes logs to stderr so stdout remains reserved for ACP JSON-RPC traffic.
|
||||
|
||||
For non-interactive checks:
|
||||
|
||||
```bash
|
||||
hermes acp --version
|
||||
hermes acp --check
|
||||
```
|
||||
|
||||
### Browser tools (optional)
|
||||
|
||||
Browser tools (`browser_navigate`, `browser_click`, etc.) depend on the
|
||||
`agent-browser` npm package and Chromium, which aren't part of the Python
|
||||
wheel. Install them with:
|
||||
|
||||
```bash
|
||||
hermes acp --setup-browser # interactive (prompts before ~400 MB download)
|
||||
hermes acp --setup-browser --yes # accept the download non-interactively
|
||||
```
|
||||
|
||||
This is the standalone command. The Zed registry's terminal-auth flow (`hermes acp --setup`) also offers the browser bootstrap as a follow-up question after model selection, so most users never need to run `--setup-browser` directly.
|
||||
|
||||
What it does:
|
||||
|
||||
- Installs Node.js 22 LTS into `~/.hermes/node/` if missing
|
||||
- `npm install -g agent-browser @askjo/camofox-browser` into that prefix (no sudo needed — `npm`'s `--prefix` points at the user-writable Hermes-managed Node)
|
||||
- Installs Playwright Chromium, or uses a detected system Chrome/Chromium when available
|
||||
|
||||
The bootstrap is idempotent — re-running it is fast and skips work that's already done.
|
||||
|
||||
## Editor setup
|
||||
|
||||
### VS Code
|
||||
@@ -90,7 +126,19 @@ If you want to define Hermes manually, add it through VS Code settings under `ac
|
||||
|
||||
### Zed
|
||||
|
||||
Example settings snippet:
|
||||
Zed v0.221.x and newer installs external agents through the official ACP Registry.
|
||||
|
||||
1. Open the Agent Panel.
|
||||
2. Click **Add Agent**, or run the `zed: acp registry` command.
|
||||
3. Search for **Hermes Agent**.
|
||||
4. Install it and start a new Hermes external-agent thread.
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- Configure Hermes provider credentials first with `hermes model`, or set them in `~/.hermes/.env` / `~/.hermes/config.yaml`.
|
||||
- Install `uv` so the registry launcher can run `uvx --from 'hermes-agent[acp]==<version>' hermes-acp`.
|
||||
|
||||
For local development before the registry entry is available, use a custom agent server in Zed settings:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -98,9 +146,9 @@ Example settings snippet:
|
||||
"hermes-agent": {
|
||||
"type": "custom",
|
||||
"command": "hermes",
|
||||
"args": ["acp"],
|
||||
},
|
||||
},
|
||||
"args": ["acp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -114,18 +162,23 @@ Use an ACP-compatible plugin and point it at:
|
||||
|
||||
## Registry manifest
|
||||
|
||||
The ACP registry manifest lives at:
|
||||
The source copy of Hermes' official ACP Registry metadata lives at:
|
||||
|
||||
```text
|
||||
acp_registry/agent.json
|
||||
acp_registry/icon.svg
|
||||
```
|
||||
|
||||
It advertises a command-based agent whose launch command is:
|
||||
The upstream registry PR copies those files into the top-level `hermes-agent/` directory in `agentclientprotocol/registry`.
|
||||
|
||||
The registry entry uses a `uvx` distribution that points directly at the `hermes-agent` PyPI release:
|
||||
|
||||
```text
|
||||
hermes acp
|
||||
uvx --from 'hermes-agent[acp]==<version>' hermes-acp
|
||||
```
|
||||
|
||||
The registry CI verifies that the pinned version exists on PyPI, so the manifest's `version` and uvx `package` pin must always match `pyproject.toml`. `scripts/release.py` keeps them in lockstep automatically.
|
||||
|
||||
## Configuration and credentials
|
||||
|
||||
ACP mode uses the same Hermes configuration as the CLI:
|
||||
@@ -135,7 +188,7 @@ ACP mode uses the same Hermes configuration as the CLI:
|
||||
- `~/.hermes/skills/`
|
||||
- `~/.hermes/state.db`
|
||||
|
||||
Provider resolution uses Hermes' normal runtime resolver, so ACP inherits the currently configured provider and credentials.
|
||||
Provider resolution uses Hermes' normal runtime resolver, so ACP inherits the currently configured provider and credentials. Hermes also advertises a terminal auth method (`--setup`) for first-run registry clients; this opens Hermes' interactive model/provider setup.
|
||||
|
||||
## Session behavior
|
||||
|
||||
@@ -171,29 +224,36 @@ On timeout or error, the approval bridge denies the request.
|
||||
|
||||
Check:
|
||||
|
||||
- the editor is pointed at the correct `acp_registry/` path
|
||||
- Hermes is installed and on your PATH
|
||||
- the ACP extra is installed (`pip install -e '.[acp]'`)
|
||||
- In Zed, open the ACP Registry with `zed: acp registry` and search for **Hermes Agent**.
|
||||
- For manual/local development, verify the custom `agent_servers` command points to `hermes acp`.
|
||||
- Hermes is installed and on your PATH.
|
||||
- The ACP extra is installed (`pip install -e '.[acp]'`).
|
||||
- `uv` is installed if launching from the official Zed registry entry.
|
||||
|
||||
### ACP starts but immediately errors
|
||||
|
||||
Try these checks:
|
||||
|
||||
```bash
|
||||
hermes acp --version
|
||||
hermes acp --check
|
||||
hermes doctor
|
||||
hermes status
|
||||
hermes acp
|
||||
```
|
||||
|
||||
### Missing credentials
|
||||
|
||||
ACP mode does not have its own login flow. It uses Hermes' existing provider setup. Configure credentials with:
|
||||
ACP mode uses Hermes' existing provider setup. Configure credentials with:
|
||||
|
||||
```bash
|
||||
hermes model
|
||||
```
|
||||
|
||||
or by editing `~/.hermes/.env`.
|
||||
or by editing `~/.hermes/.env`. Registry clients can also trigger Hermes' terminal auth flow, which runs the same interactive provider/model setup.
|
||||
|
||||
### Zed registry launcher cannot find uv
|
||||
|
||||
Install `uv` from the official uv installation docs, then retry the Hermes Agent thread from Zed.
|
||||
|
||||
## See also
|
||||
|
||||
|
||||
@@ -368,6 +368,13 @@ BROWSERBASE_SESSION_TIMEOUT=600000
|
||||
|
||||
# Inactivity timeout before auto-cleanup in seconds (default: 120)
|
||||
BROWSER_INACTIVITY_TIMEOUT=120
|
||||
|
||||
# Extra Chromium launch flags (comma- or newline-separated). Hermes auto-injects
|
||||
# `--no-sandbox,--disable-dev-shm-usage` when it detects root or AppArmor-restricted
|
||||
# unprivileged user namespaces (Ubuntu 23.10+, DGX Spark, many container images),
|
||||
# so most users don't need to set this. Set it manually only if you need a flag
|
||||
# Hermes doesn't add automatically; setting it disables the auto-injection.
|
||||
AGENT_BROWSER_ARGS=--no-sandbox
|
||||
```
|
||||
|
||||
### Install agent-browser CLI
|
||||
|
||||
@@ -0,0 +1,444 @@
|
||||
---
|
||||
title: Codex App-Server Runtime (optional)
|
||||
sidebar_label: Codex App-Server Runtime
|
||||
---
|
||||
|
||||
# Codex App-Server Runtime
|
||||
|
||||
Hermes can optionally hand `openai/*` and `openai-codex/*` turns to the [Codex CLI app-server](https://github.com/openai/codex) instead of running its own tool loop. When enabled, terminal commands, file edits, sandboxing, and MCP tool calls all execute inside Codex's runtime — Hermes becomes the shell around it (sessions DB, slash commands, gateway, memory and skill review).
|
||||
|
||||
This is **opt-in only**. Default Hermes behavior is unchanged unless you flip the flag. Hermes never auto-routes you onto this runtime.
|
||||
|
||||
## Why
|
||||
|
||||
- Run OpenAI agent turns against your **ChatGPT subscription** (no API key required) using the same auth flow Codex CLI uses.
|
||||
- Use **Codex's own toolset and sandbox** — `shell` for terminal/read/write/search, `apply_patch` for structured edits, `update_plan` for planning, all running inside seatbelt/landlock sandboxing.
|
||||
- **Native Codex plugins** — Linear, GitHub, Gmail, Calendar, Canva, etc. — installed via `codex plugin` are auto-migrated and active in your Hermes session.
|
||||
- **Hermes' richer tools come along** — web_search, web_extract, browser automation, vision, image generation, skills, and TTS work via an MCP callback. Codex calls back into Hermes for tools it doesn't have built in.
|
||||
- **Memory and skill nudges keep working** — Codex's events are projected into Hermes' message shape so the self-improvement loop sees a normal-looking transcript.
|
||||
|
||||
## What tools the model actually has
|
||||
|
||||
This is the part most users want to know up front. When this runtime is on, the model running your turn has three independent sources of tools:
|
||||
|
||||
### 1. Codex's built-in toolset (always on)
|
||||
|
||||
These ship with `codex app-server` itself — no Hermes involvement, no MCP, no plugins. All five are available the moment the runtime starts:
|
||||
|
||||
- **`shell`** — runs arbitrary shell commands inside the sandbox. This is how the model reads files (`cat`, `head`, `tail`), writes them (`echo > foo`, heredocs), searches them (`find`, `rg`, `grep`), navigates directories (`ls`, `cd`), runs builds, manages processes, and anything else you'd do in bash.
|
||||
- **`apply_patch`** — applies a structured multi-file diff in Codex's patch format. The model uses this for non-trivial code edits (adding a function, refactoring across files); shell heredocs are still available for one-off writes.
|
||||
- **`update_plan`** — codex's internal todo / plan tracker. Equivalent of Hermes' `todo` tool, but managed entirely inside codex's runtime.
|
||||
- **`view_image`** — load a local image file into the conversation so the model can see it.
|
||||
- **`web_search`** — codex has its own built-in web search when configured. Hermes also exposes `web_search` (Firecrawl-backed) via the callback below; the model picks whichever it prefers.
|
||||
|
||||
So **anything you'd do via terminal — read/write/search/find/run — codex does natively**. The sandbox profile (`:workspace` by default when you enable the runtime) controls what's writable.
|
||||
|
||||
### 2. Native Codex plugins (auto-migrated from your `codex plugin` install)
|
||||
|
||||
When you enable the runtime, Hermes queries codex's `plugin/list` RPC and writes a `[plugins."<name>@openai-curated"]` entry for every plugin you have installed. The plugins themselves are managed by codex and authorized once via codex's own UI.
|
||||
|
||||
Examples (the ones the OpenClaw thread highlighted as "YouTube-video-worthy"):
|
||||
|
||||
- **Linear** — find/update issues
|
||||
- **GitHub** — search code, view PRs, comment
|
||||
- **Gmail** — read/send mail
|
||||
- **Google Calendar** — create/find events
|
||||
- **Outlook calendar/email** — same shape via the Microsoft connector
|
||||
- **Canva** — design generation
|
||||
- ...whatever else you've installed via `codex plugin marketplace add openai-curated` + `codex plugin install ...`
|
||||
|
||||
What's NOT migrated:
|
||||
- Plugins you haven't installed yet — install them in Codex first.
|
||||
- ChatGPT app marketplace entries (`app/list`) — these are already enabled inside codex by virtue of your account auth.
|
||||
|
||||
### 3. Hermes tool callback (MCP server, registered in `~/.codex/config.toml`)
|
||||
|
||||
Hermes registers itself as an MCP server so codex can call back for tools codex doesn't ship with. Available via the callback:
|
||||
|
||||
- **`web_search`** / **`web_extract`** — Firecrawl-backed; tends to be cleaner than scraping for structured content.
|
||||
- **`browser_navigate` / `browser_click` / `browser_type` / `browser_press` / `browser_snapshot` / `browser_scroll` / `browser_back` / `browser_get_images` / `browser_console` / `browser_vision`** — full browser automation via Camofox or Browserbase.
|
||||
- **`vision_analyze`** — call a separate vision model to inspect an image (different from codex's `view_image` which loads it into the conversation).
|
||||
- **`image_generate`** — image generation through Hermes' image_gen plugin chain.
|
||||
- **`skill_view` / `skills_list`** — read from Hermes' skill library.
|
||||
- **`text_to_speech`** — TTS through Hermes' configured provider.
|
||||
|
||||
When the model wants one of these, codex spawns the `hermes_tools_mcp_server` subprocess via stdio MCP, the call is dispatched through `model_tools.handle_function_call()` (same code path as Hermes' default runtime), and the result is returned to codex like any other MCP response.
|
||||
|
||||
### What's NOT available on this runtime
|
||||
|
||||
These four Hermes tools require the running AIAgent context (mid-loop state) to dispatch, and a stateless MCP callback can't drive them. Switch back to the default runtime (`/codex-runtime auto`) when you need any of them:
|
||||
|
||||
- **`delegate_task`** — spawn subagents
|
||||
- **`memory`** — Hermes' persistent memory store
|
||||
- **`session_search`** — cross-session search
|
||||
- **`todo`** — Hermes' todo store (codex's `update_plan` is the in-runtime equivalent)
|
||||
|
||||
## Workflow features (`/goal`, kanban, cron)
|
||||
|
||||
### `/goal` (the Ralph loop)
|
||||
|
||||
**Works on this runtime.** Goals persist in `state_meta` keyed by session id, the continuation prompt feeds back as a normal user message through `run_conversation()`, and codex executes the next turn natively. The goal judge runs via the auxiliary client (configured via `auxiliary.goal_judge` in config.yaml), independent of which runtime is active. The judge's "blocked, needs user input" verdict is a clean escape if codex stalls on approvals.
|
||||
|
||||
**One thing to be aware of:** each continuation prompt is a fresh codex turn, which means codex re-evaluates command approval policy from scratch. If you're doing a long-running goal with lots of writes, expect more approval prompts than you'd see on a single in-session task. Set `default_permissions = ":workspace"` (which Hermes does automatically when you enable the runtime) so simple workspace writes don't require prompting.
|
||||
|
||||
### Kanban (multi-agent worktree dispatch)
|
||||
|
||||
**Works on this runtime, with one subtle dependency.** The kanban dispatcher spawns each worker as a separate `hermes chat -q` subprocess that reads the user's config — which means if `model.openai_runtime: codex_app_server` is set globally, workers also come up on the codex runtime.
|
||||
|
||||
What works inside a codex-runtime worker:
|
||||
- Codex's full toolset (shell, apply_patch, update_plan, view_image, web_search) — the worker does its actual task work natively
|
||||
- The migrated codex plugins — Linear, GitHub, etc.
|
||||
- The Hermes tool callback for browser_*, vision, image_gen, skills, TTS
|
||||
|
||||
What also works because the MCP callback exposes them:
|
||||
- **`kanban_complete` / `kanban_block` / `kanban_comment` / `kanban_heartbeat`** — the worker handoff tools. These read `HERMES_KANBAN_TASK` from env (set by the dispatcher), gate access correctly, and write to `~/.hermes/kanban.db`. Without these in the callback, a worker on this runtime could do its task but couldn't report back, hanging until the dispatcher's timeout.
|
||||
- **`kanban_show` / `kanban_list`** — read-only board queries for the worker to check its own context.
|
||||
- **`kanban_create` / `kanban_unblock` / `kanban_link`** — orchestrator-only operations. Available for orchestrator agents running on the codex runtime that need to dispatch new tasks.
|
||||
|
||||
The kanban tools are gated by `HERMES_KANBAN_TASK` env var the dispatcher sets — that var is propagated to the codex subprocess (codex inherits env) and from there to the spawned `hermes-tools` MCP server subprocess. So the tools see the right task id and gate correctly.
|
||||
|
||||
### Cron jobs
|
||||
|
||||
**Not specifically tested.** Cron jobs run via `cronjob` → `AIAgent.run_conversation`, the same code path as the CLI. If the cron job's config has `openai_runtime: codex_app_server` it'll run on codex. The same tool-availability rules apply — codex built-ins + plugins + MCP callback work, agent-loop tools (delegate_task, memory, session_search, todo) don't. If your cron job relies on those, scope the cron to a profile that uses the default runtime.
|
||||
|
||||
## Trade-offs
|
||||
|
||||
| | Hermes default runtime | Codex app-server (opt-in) |
|
||||
|---|---|---|
|
||||
| `delegate_task` subagents | yes | not available — needs agent loop context |
|
||||
| `memory`, `session_search`, `todo` | yes | not available — needs agent loop context |
|
||||
| `web_search`, `web_extract` | yes | yes (via MCP callback) |
|
||||
| Browser automation (Camofox/Browserbase) | yes | yes (via MCP callback) |
|
||||
| `vision_analyze`, `image_generate` | yes | yes (via MCP callback) |
|
||||
| `skill_view`, `skills_list` | yes | yes (via MCP callback) |
|
||||
| `text_to_speech` | yes | yes (via MCP callback) |
|
||||
| Codex `shell` (terminal/read/write/search/find/run) | — | yes (Codex built-in) |
|
||||
| Codex `apply_patch` (structured multi-file edits) | — | yes (Codex built-in) |
|
||||
| Codex `update_plan` (in-runtime todo) | — | yes (Codex built-in) |
|
||||
| Codex `view_image` (load image into conversation) | — | yes (Codex built-in) |
|
||||
| Codex sandbox (seatbelt/landlock, profiles) | — | yes (Codex built-in) |
|
||||
| ChatGPT subscription auth | — | yes (via `openai-codex` provider) |
|
||||
| Native Codex plugins (Linear, GitHub, etc.) | — | yes (auto-migrated) |
|
||||
| User MCP servers | yes | yes (auto-migrated to codex) |
|
||||
| Memory + skill review (background) | yes | yes (via item projection) |
|
||||
| Multi-turn conversations | yes | yes |
|
||||
| `/goal` (Ralph loop) | yes | yes |
|
||||
| Kanban worker dispatch | yes | yes (via callback) |
|
||||
| Kanban orchestrator tools | yes | yes (via callback) |
|
||||
| All gateway platforms | yes | yes |
|
||||
| Non-OpenAI providers | yes | n/a — OpenAI/Codex-scoped |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Codex CLI installed:**
|
||||
```bash
|
||||
npm i -g @openai/codex
|
||||
codex --version # 0.130.0 or newer
|
||||
```
|
||||
2. **Codex OAuth login.** The codex subprocess reads `~/.codex/auth.json`. Two ways to populate it:
|
||||
```bash
|
||||
codex login # writes tokens to ~/.codex/auth.json
|
||||
```
|
||||
Hermes' own `hermes auth login codex` writes to `~/.hermes/auth.json` — that's a separate session. **Run `codex login` separately** if you haven't.
|
||||
|
||||
3. **(Optional) Install the Codex plugins you want.** When you enable the runtime, Hermes auto-migrates whichever curated plugins you've already installed via Codex CLI:
|
||||
```bash
|
||||
codex plugin marketplace add openai-curated
|
||||
# then via codex's TUI, install Linear / GitHub / Gmail / etc.
|
||||
```
|
||||
Hermes will discover them and write `[plugins."<name>@openai-curated"]` entries to `~/.codex/config.toml` automatically.
|
||||
|
||||
## Enabling
|
||||
|
||||
In a Hermes session:
|
||||
|
||||
```
|
||||
/codex-runtime codex_app_server
|
||||
```
|
||||
|
||||
That command:
|
||||
- Verifies the `codex` CLI is installed (blocks with an install hint if not).
|
||||
- Persists `model.openai_runtime: codex_app_server` to your config.yaml.
|
||||
- Migrates user MCP servers from `~/.hermes/config.yaml` to `~/.codex/config.toml`.
|
||||
- **Discovers and migrates installed native Codex plugins** (Linear, GitHub, Gmail, Calendar, Canva, etc.) by querying Codex's `plugin/list` RPC.
|
||||
- **Registers Hermes' own tools as an MCP server** so the codex subprocess can call back for tools codex doesn't ship with.
|
||||
- **Writes `default_permissions = ":workspace"`** so the sandbox allows writes within the workspace without prompting for every operation.
|
||||
- Tells you what was migrated. Takes effect on the **next** session — the current cached agent keeps the prior runtime so prompt caches stay valid.
|
||||
|
||||
Synonyms: `/codex-runtime on`, `/codex-runtime off`, `/codex-runtime auto`.
|
||||
|
||||
To check current state without changing anything:
|
||||
```
|
||||
/codex-runtime
|
||||
```
|
||||
|
||||
You can also set it manually in `~/.hermes/config.yaml`:
|
||||
```yaml
|
||||
model:
|
||||
openai_runtime: codex_app_server # default is "auto" (= Hermes runtime)
|
||||
```
|
||||
|
||||
## Self-improvement loop (memory + skill nudges)
|
||||
|
||||
Hermes' background self-improvement fires on counter thresholds:
|
||||
|
||||
- Every 10 user prompts → a forked review agent looks at the conversation and decides whether anything should be saved to memory.
|
||||
- Every 10 tool iterations within a single turn → same idea but for skills (`skill_manage` writes).
|
||||
|
||||
**Both keep working on the codex runtime.** The codex path projects each completed `commandExecution` / `fileChange` / `mcpToolCall` / `dynamicToolCall` item into a synthetic `assistant tool_call` + `tool` result message, so by the time the review runs it sees the same shape it sees on the default Hermes runtime.
|
||||
|
||||
How the wiring stays equivalent:
|
||||
|
||||
| | Default runtime | Codex runtime |
|
||||
|---|---|---|
|
||||
| `_turns_since_memory` increments | per user prompt, in run_conversation pre-loop | same code path, before the early-return |
|
||||
| `_iters_since_skill` increments | per tool iteration in the chat-completions loop | by `turn.tool_iterations` after the codex turn returns |
|
||||
| Memory trigger (`_turns_since_memory >= _memory_nudge_interval`) | computed in pre-loop, fires after response | computed in pre-loop, passed through to codex helper |
|
||||
| Skill trigger (`_iters_since_skill >= _skill_nudge_interval`) | computed after the loop | computed after the codex turn |
|
||||
| `_spawn_background_review(messages_snapshot=..., review_memory=..., review_skills=...)` | called when either trigger fires | called identically when either trigger fires |
|
||||
|
||||
One detail: the review fork itself needs to call Hermes' agent-loop tools (`memory`, `skill_manage`), which require Hermes' own dispatch. So when the parent agent is on `codex_app_server`, the review fork is **downgraded to `codex_responses`** — same OAuth credentials, same `openai-codex` provider, but talks to OpenAI's Responses API directly so Hermes owns the loop and the agent-loop tools work. This is invisible to the user.
|
||||
|
||||
Net effect: enable the codex runtime and your memory + skill nudges keep firing exactly as they would otherwise.
|
||||
|
||||
## How approvals work
|
||||
|
||||
Codex requests approval before executing commands or applying patches. These get translated into Hermes' standard "Dangerous Command" prompt:
|
||||
|
||||
```
|
||||
╭───────────────────────────────────────╮
|
||||
│ Dangerous Command │
|
||||
│ │
|
||||
│ /bin/bash -lc 'echo hello > foo.txt' │
|
||||
│ │
|
||||
│ ❯ 1. Allow once │
|
||||
│ 2. Allow for this session │
|
||||
│ 3. Deny │
|
||||
│ │
|
||||
│ Codex requests exec in /your/cwd │
|
||||
╰───────────────────────────────────────╯
|
||||
```
|
||||
|
||||
- **Allow once** → approve this single command.
|
||||
- **Allow for this session** → Codex won't re-prompt for similar commands.
|
||||
- **Deny** → command is rejected; Codex continues in read-only mode.
|
||||
|
||||
For `apply_patch` (file edit) approvals, Hermes shows a summary of what changed (`1 add, 1 update: /tmp/new.py, /tmp/old.py`) when codex provides the data via the corresponding `fileChange` item.
|
||||
|
||||
## Permission profiles
|
||||
|
||||
Codex has three built-in permission profiles:
|
||||
- `:read-only` — no writes; every shell command requires approval
|
||||
- `:workspace` — writes within the current workspace allowed without prompts (Hermes' default when you enable the runtime)
|
||||
- `:danger-no-sandbox` — no sandbox at all (don't use this unless you understand it)
|
||||
|
||||
You can override the default in `~/.codex/config.toml` outside Hermes' managed block:
|
||||
|
||||
```toml
|
||||
default_permissions = ":read-only"
|
||||
```
|
||||
|
||||
(Hermes will preserve your override on re-migration as long as it lives outside the `# managed by hermes-agent` markers.)
|
||||
|
||||
## Auxiliary tasks and ChatGPT subscription token cost
|
||||
|
||||
When this runtime is on with the `openai-codex` provider, **auxiliary tasks (title generation, context compression, vision auto-detect, session search summarization, the background self-improvement review fork) also flow through your ChatGPT subscription by default**, because Hermes' auxiliary client uses the main provider/model when no per-task override is set.
|
||||
|
||||
This isn't specific to `codex_app_server` — it's true for the existing `codex_responses` path too — but it's more visible here because you're explicitly opting in for the subscription billing.
|
||||
|
||||
To route specific aux tasks to a cheaper / different model, set explicit overrides in `~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: openrouter
|
||||
model: google/gemini-3-flash-preview
|
||||
context_compression:
|
||||
provider: openrouter
|
||||
model: google/gemini-3-flash-preview
|
||||
vision_detect:
|
||||
provider: openrouter
|
||||
model: google/gemini-3-flash-preview
|
||||
session_search:
|
||||
provider: openrouter
|
||||
model: google/gemini-3-flash-preview
|
||||
goal_judge:
|
||||
provider: openrouter
|
||||
model: google/gemini-3-flash-preview
|
||||
```
|
||||
|
||||
The self-improvement review fork inherits the main runtime via `_current_main_runtime()` and Hermes downgrades it from `codex_app_server` to `codex_responses` automatically (so the fork can actually call `memory` and `skill_manage` — Hermes' own agent-loop tools). That fork still uses your subscription auth unless you've routed aux tasks elsewhere.
|
||||
|
||||
## Editing `~/.codex/config.toml` safely
|
||||
|
||||
Hermes wraps everything it manages between two marker comments:
|
||||
|
||||
```toml
|
||||
# managed by hermes-agent — `hermes codex-runtime migrate` regenerates this section
|
||||
default_permissions = ":workspace"
|
||||
[mcp_servers.filesystem]
|
||||
...
|
||||
[plugins."github@openai-curated"]
|
||||
...
|
||||
# end hermes-agent managed section
|
||||
```
|
||||
|
||||
Anything **outside** that block is yours. Re-running migration (via `/codex-runtime codex_app_server` or whenever you toggle the runtime on) replaces the managed block in place but preserves user content above and below it verbatim. This means you can:
|
||||
|
||||
- Add your own MCP servers Hermes doesn't know about
|
||||
- Override `default_permissions` to `:read-only` if you prefer to be prompted
|
||||
- Configure codex-only options (model, providers, otel, etc.)
|
||||
- Add user-defined permission profiles in `[permissions.<name>]` tables
|
||||
|
||||
Anything you add **inside** the managed block will get clobbered on the next migration. If you need a tweak that requires editing the managed block, file an issue and we'll add the knob.
|
||||
|
||||
## Multi-profile / multi-tenant setups
|
||||
|
||||
By default, Hermes points the codex subprocess at `~/.codex/` regardless of which Hermes profile is active. This means `hermes -p work` and `hermes -p personal` share the same Codex auth, plugins, and config. For most users this is the right behavior — it matches what running `codex` CLI directly would do.
|
||||
|
||||
If you want per-profile Codex isolation (separate auth, separate installed plugins, separate config), set `CODEX_HOME` explicitly per profile. The cleanest way is to point at a directory under your `HERMES_HOME`:
|
||||
|
||||
```bash
|
||||
# Inside the work profile, you might wrap hermes:
|
||||
CODEX_HOME=~/.hermes/profiles/work/codex hermes chat
|
||||
```
|
||||
|
||||
You'll need to re-run `codex login` once with that `CODEX_HOME` set so the OAuth tokens land in the profile-scoped location. After that, `hermes -p work` will operate on isolated Codex state.
|
||||
|
||||
We don't auto-scope this because moving an existing user's `~/.codex/` would silently invalidate their Codex CLI auth — anyone who already ran `codex login` would have to re-authenticate. Opt-in feels safer than surprising users.
|
||||
|
||||
## HOME environment variable passthrough
|
||||
|
||||
Hermes does NOT rewrite `HOME` when spawning the codex app-server subprocess (we use `os.environ.copy()` and only overlay `CODEX_HOME` and `RUST_LOG`). This means:
|
||||
|
||||
- Commands codex runs via its `shell` tool see the real user `HOME` and find `~/.gitconfig`, `~/.gh/`, `~/.aws/`, `~/.npmrc`, etc. correctly.
|
||||
- Codex's internal state stays isolated through `CODEX_HOME` (which points at `~/.codex/` by default).
|
||||
|
||||
This matches the boundary OpenClaw arrived at after some early experimentation: isolate Codex's state, leave the user's home alone. (Cf. openclaw/openclaw#81562.)
|
||||
|
||||
## MCP server migration
|
||||
|
||||
Hermes' `mcp_servers` config is auto-translated to the TOML format Codex expects. The migration runs every time you enable the runtime and is idempotent — re-runs replace the managed section but preserve any user-edited Codex config.
|
||||
|
||||
What translates:
|
||||
|
||||
| Hermes (`config.yaml`) | Codex (`config.toml`) |
|
||||
|---|---|
|
||||
| `command` + `args` + `env` | stdio transport |
|
||||
| `url` + `headers` | streamable_http transport |
|
||||
| `timeout` | `tool_timeout_sec` |
|
||||
| `connect_timeout` | `startup_timeout_sec` |
|
||||
| `enabled: false` | `enabled = false` |
|
||||
|
||||
What's not migrated:
|
||||
- Hermes-specific keys like `sampling` (Codex's MCP client has no equivalent — these are dropped with a per-server warning).
|
||||
|
||||
## Native Codex plugin migration
|
||||
|
||||
Plugins installed via `codex plugin` (Linear, GitHub, Gmail, Calendar, Canva, etc.) are discovered through Codex's `plugin/list` RPC. For each plugin where `installed: true`, Hermes writes a `[plugins."<name>@openai-curated"]` block enabling it in your Hermes session.
|
||||
|
||||
This means: when your friend says "I have Calendar and GitHub set up in my Codex CLI" and they enable Hermes' codex runtime, Hermes activates those automatically. No re-configuration needed.
|
||||
|
||||
What's NOT migrated:
|
||||
- Plugins you haven't installed yet — install them in Codex first.
|
||||
- Plugins where codex reports `availability != AVAILABLE` (broken install, expired OAuth, removed from marketplace, etc.). These are skipped to avoid writing config that would fail at activation time.
|
||||
- ChatGPT app marketplace entries (the per-account `app/list` results — these are already enabled inside codex by virtue of your account auth).
|
||||
- Plugin OAuth — you authorize each plugin once in Codex itself; Hermes doesn't touch credentials.
|
||||
|
||||
## Hermes tool callback (the new MCP server)
|
||||
|
||||
Codex's built-in toolset covers shell/file ops/patches but doesn't have web search, browser automation, vision, image generation, etc. To keep those usable in a codex turn, Hermes registers itself as an MCP server in `~/.codex/config.toml`:
|
||||
|
||||
```toml
|
||||
[mcp_servers.hermes-tools]
|
||||
command = "/path/to/python"
|
||||
args = ["-m", "agent.transports.hermes_tools_mcp_server"]
|
||||
env = { HERMES_HOME = "/your/.hermes", PYTHONPATH = "...", HERMES_QUIET = "1" }
|
||||
startup_timeout_sec = 30.0
|
||||
tool_timeout_sec = 600.0
|
||||
```
|
||||
|
||||
When the model calls `web_search` (or another exposed Hermes tool), codex spawns the `hermes_tools_mcp_server` subprocess via stdio, the request is dispatched through `model_tools.handle_function_call()`, and the result is projected back to codex like any other MCP response.
|
||||
|
||||
**Tools available via the callback:** `web_search`, `web_extract`, `browser_navigate`, `browser_click`, `browser_type`, `browser_press`, `browser_snapshot`, `browser_scroll`, `browser_back`, `browser_get_images`, `browser_console`, `browser_vision`, `vision_analyze`, `image_generate`, `skill_view`, `skills_list`, `text_to_speech`.
|
||||
|
||||
**Tools NOT available:** `delegate_task`, `memory`, `session_search`, `todo`. These need the running AIAgent context to dispatch (mid-loop state) and a stateless MCP callback can't drive them. Use the default Hermes runtime (`/codex-runtime auto`) when you need these.
|
||||
|
||||
## Disabling
|
||||
|
||||
Switch back at any time:
|
||||
|
||||
```
|
||||
/codex-runtime auto
|
||||
```
|
||||
|
||||
Effective on the next session. The Codex managed block stays in `~/.codex/config.toml` so you can re-enable later without losing config — or remove it manually if you prefer.
|
||||
|
||||
## Limitations
|
||||
|
||||
This runtime is **opt-in beta**. Working as of Hermes Agent 2026.5 + Codex CLI 0.130.0:
|
||||
|
||||
- Multi-turn conversations
|
||||
- `commandExecution` and `fileChange` (apply_patch) approvals via Hermes UI
|
||||
- MCP tool calls (verified against `@modelcontextprotocol/server-filesystem` and the new `hermes-tools` callback)
|
||||
- Native Codex plugin migration (verified against Linear / GitHub / Calendar inventory)
|
||||
- Deny/cancel paths
|
||||
- Toggle on/off cycle
|
||||
- Memory and skill nudge counters (verified live via integration tests)
|
||||
- Hermes web_search through codex (verified live: "OpenAI Codex CLI – Getting Started" returned end-to-end)
|
||||
|
||||
Known limitations:
|
||||
|
||||
- **Hermes auth and codex auth are separate sessions.** You need both `codex login` AND `hermes auth login codex` for the cleanest UX (the runtime uses codex's session for the LLM call). This is a deliberate design choice in Hermes' `_import_codex_cli_tokens` — Hermes won't share OAuth state with codex CLI to avoid clobbering each other on token refresh.
|
||||
- **`delegate_task`, `memory`, `session_search`, `todo` are unavailable on this runtime.** They need the running AIAgent context which a stateless MCP callback can't provide. Use `/codex-runtime auto` when you need these.
|
||||
- **No inline patch preview in approval prompts when codex doesn't track the changeset.** Codex's `fileChange` approval params don't always carry the changeset. Hermes caches the data from the corresponding `item/started` notification when possible, but if approval arrives before the item has streamed, the prompt falls back to whatever `reason` codex provides.
|
||||
- **Sub-second cancellation isn't guaranteed.** Mid-stream interrupts (Ctrl+C while codex is responding) are sent via `turn/interrupt`, but if codex has already flushed the final message, you get the response anyway.
|
||||
|
||||
If you find a bug, [open an issue](https://github.com/NousResearch/hermes-agent/issues) with the output of `hermes logs --since 5m`. Mention `codex-runtime` in the title so it's easy to triage.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─── Hermes shell (CLI / TUI / gateway) ───┐
|
||||
│ sessions DB · slash commands · memory │
|
||||
│ & skill review · cron · session pickers │
|
||||
└──┬──────────────────────────────────────┬┘
|
||||
│ user_message final │
|
||||
▼ text + │
|
||||
┌──────────────────────────────────┐ projected │
|
||||
│ AIAgent.run_conversation() │ messages │
|
||||
│ if api_mode == codex_app_server │ │
|
||||
│ → CodexAppServerSession │ │
|
||||
│ else: chat_completions / codex_responses (default)
|
||||
└────┬─────────────────────────────┘ │
|
||||
│ JSON-RPC over stdio │
|
||||
▼ │
|
||||
┌──────────────────────────────────┐ │
|
||||
│ codex app-server (subprocess) │──────────────┘
|
||||
│ thread/start, turn/start │
|
||||
│ item/* notifications │
|
||||
│ shell + apply_patch + update_plan│
|
||||
│ view_image + sandbox │
|
||||
│ ┌─────────────────────────┐ │
|
||||
│ │ MCP client │ │
|
||||
│ │ ├─ user MCP servers │ │
|
||||
│ │ ├─ native plugins │ │
|
||||
│ │ │ (linear, github, │ │
|
||||
│ │ │ gmail, calendar, │ │
|
||||
│ │ │ canva, ...) │ │
|
||||
│ │ └─ hermes-tools ───────┼─────────────────┐
|
||||
│ │ (callback to │ │ │
|
||||
│ │ Hermes' richer │ │ │
|
||||
│ │ tools) │ │ │
|
||||
│ └─────────────────────────┘ │ │
|
||||
└──────────────────────────────────┘ │
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ hermes_tools_mcp_server.py (subprocess on demand) │
|
||||
│ web_search, web_extract, browser_*, vision_analyze, │
|
||||
│ image_generate, skill_view, skills_list, text_to_speech│
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
For implementation details, see [PR #24182](https://github.com/NousResearch/hermes-agent/pull/24182) and the [Codex app-server protocol README](https://github.com/openai/codex/blob/main/codex-rs/app-server/README.md).
|
||||
@@ -522,6 +522,86 @@ print(json.dumps({"wakeAgent": True, "context": {"new_issues": latest - prev}}))
|
||||
|
||||
When `wakeAgent` is omitted, the default is `true` (wake the agent as usual).
|
||||
|
||||
#### Recipes: cheap pre-run gates
|
||||
|
||||
The `wakeAgent` gate gives you a $0 way to decide whether a scheduled job should spend any LLM tokens at all. Three patterns cover most use cases.
|
||||
|
||||
**File-change gate** — only run when a watched file has new content since the last successful tick. The scheduler records each job's `last_run_at`; compare it against the file's mtime.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# ~/.hermes/scripts/feed-changed.sh
|
||||
FEED="$HOME/data/feed.json"
|
||||
STATE="$HOME/.hermes/scripts/.feed-changed.last"
|
||||
test -f "$FEED" || { echo '{"wakeAgent": false}'; exit 0; }
|
||||
mtime=$(stat -c %Y "$FEED")
|
||||
last=$(cat "$STATE" 2>/dev/null || echo 0)
|
||||
if [ "$mtime" -le "$last" ]; then
|
||||
echo '{"wakeAgent": false}'
|
||||
else
|
||||
echo "$mtime" > "$STATE"
|
||||
echo '{"wakeAgent": true}'
|
||||
fi
|
||||
```
|
||||
|
||||
```text
|
||||
cronjob(action="create", name="process-feed",
|
||||
schedule="every 30m",
|
||||
script="feed-changed.sh",
|
||||
prompt="A new ~/data/feed.json has landed. Summarize what changed.")
|
||||
```
|
||||
|
||||
**External-flag gate** — only run when some other process has signalled readiness (e.g. a deploy hook drops a file, a CI job sets a value in your state store).
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# ~/.hermes/scripts/flag-ready.sh
|
||||
if test -f /tmp/new-data-ready; then
|
||||
rm -f /tmp/new-data-ready
|
||||
echo '{"wakeAgent": true}'
|
||||
else
|
||||
echo '{"wakeAgent": false}'
|
||||
fi
|
||||
```
|
||||
|
||||
```text
|
||||
cronjob(action="create", name="nightly-analysis",
|
||||
schedule="0 9 * * *",
|
||||
script="flag-ready.sh",
|
||||
prompt="Run the nightly analysis over today's batch.")
|
||||
```
|
||||
|
||||
**SQL-count gate** — only run when there are new rows to process in your own database. The script can also pass the count through to the agent via `context`, so the agent knows how much it's looking at without re-querying.
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python
|
||||
# ~/.hermes/scripts/new-rows.py
|
||||
import json, sqlite3
|
||||
conn = sqlite3.connect("/home/me/data/app.db")
|
||||
n = conn.execute(
|
||||
"SELECT COUNT(*) FROM messages WHERE ts > strftime('%s','now','-2 hours')"
|
||||
).fetchone()[0]
|
||||
if n < 1:
|
||||
print(json.dumps({"wakeAgent": False}))
|
||||
else:
|
||||
print(json.dumps({"wakeAgent": True, "context": {"new_rows": n}}))
|
||||
```
|
||||
|
||||
```text
|
||||
cronjob(action="create", name="summarize-new-msgs",
|
||||
schedule="every 2h",
|
||||
script="new-rows.py",
|
||||
prompt="Summarize the new messages from the last 2 hours.")
|
||||
```
|
||||
|
||||
The same pattern works for any data source you can query from a script — Postgres, an HTTP API, your own state store — without baking a SQL evaluator into the cron subsystem.
|
||||
|
||||
:::tip
|
||||
Hermes's own `~/.hermes/state.db` is an internal schema that changes between releases. Don't query it from a pre-run gate — point at your own database or feed instead.
|
||||
:::
|
||||
|
||||
Credit: this recipe set was prompted by @iankar8's exploration in [#2654](https://github.com/NousResearch/hermes-agent/pull/2654), which proposed adding sql/file/command triggers as a parallel mechanism. The `script` + `wakeAgent` gate already covers all three cases at $0, so the work landed as documentation instead.
|
||||
|
||||
### Chaining jobs: `context_from`
|
||||
|
||||
A cron job can consume the most recent successful output of one or more other jobs by listing their names (or IDs) in `context_from`:
|
||||
|
||||
@@ -66,6 +66,7 @@ Both `provider` and `model` are **required**. If either is missing, the fallback
|
||||
| Google Gemini (OAuth) | `google-gemini-cli` | `hermes model` (Google OAuth; optional: `HERMES_GEMINI_PROJECT_ID`) |
|
||||
| Google AI Studio | `gemini` | `GOOGLE_API_KEY` (alias: `GEMINI_API_KEY`) |
|
||||
| xAI (Grok) | `xai` (alias `grok`) | `XAI_API_KEY` (optional: `XAI_BASE_URL`) |
|
||||
| xAI Grok OAuth (SuperGrok) | `xai-oauth` (alias `grok-oauth`) | `hermes model` → xAI Grok OAuth (browser login; SuperGrok subscription) |
|
||||
| AWS Bedrock | `bedrock` | Standard boto3 auth (`AWS_REGION` + `AWS_PROFILE` or `AWS_ACCESS_KEY_ID`) |
|
||||
| Qwen Portal (OAuth) | `qwen-oauth` | `hermes model` (Qwen Portal OAuth; optional: `HERMES_QWEN_BASE_URL`) |
|
||||
| MiniMax (OAuth) | `minimax-oauth` | `hermes model` (MiniMax portal OAuth) |
|
||||
|
||||
@@ -21,7 +21,7 @@ install, no separate daemon to manage.
|
||||
## When LSP runs
|
||||
|
||||
LSP is gated on **git workspace detection**. When the agent's working
|
||||
directory (or the file being edited) is inside a git worktree, LSP
|
||||
directory (or the file being edited) is inside a git repository, LSP
|
||||
runs against that workspace. When neither is in a git repo, LSP
|
||||
stays dormant — useful for messaging gateways where the cwd is the
|
||||
user's home directory and there's no project to diagnose.
|
||||
@@ -249,5 +249,6 @@ the next edit re-spawns.
|
||||
|
||||
**Editing a file outside any git repo**
|
||||
|
||||
By design, LSP only runs inside git worktrees. Run `git init` in the
|
||||
project, or accept the in-process syntax-only fallback.
|
||||
By design, LSP only runs inside a git repository. If the project isn't
|
||||
yet initialized, run `git init` to enable LSP diagnostics. Otherwise the
|
||||
in-process syntax-only fallback applies.
|
||||
|
||||
@@ -109,6 +109,7 @@ Every `ctx.*` API below is available inside a plugin's `register(ctx)` function.
|
||||
| Distribute via pip | `[project.entry-points."hermes_agent.plugins"]` |
|
||||
| Register a gateway platform (Discord, Telegram, IRC, …) | `ctx.register_platform(name, label, adapter_factory, check_fn, ...)` — see [Adding Platform Adapters](/docs/developer-guide/adding-platform-adapters) |
|
||||
| Register an image-generation backend | `ctx.register_image_gen_provider(provider)` — see [Image Generation Provider Plugins](/docs/developer-guide/image-gen-provider-plugin) |
|
||||
| Register a video-generation backend | `ctx.register_video_gen_provider(provider)` — see [Video Generation Provider Plugins](/docs/developer-guide/video-gen-provider-plugin) |
|
||||
| Register a context-compression engine | `ctx.register_context_engine(engine)` — see [Context Engine Plugins](/docs/developer-guide/context-engine-plugin) |
|
||||
| Register a memory backend | Subclass `MemoryProvider` in `plugins/memory/<name>/__init__.py` — see [Memory Provider Plugins](/docs/developer-guide/memory-provider-plugin) (uses a separate discovery system) |
|
||||
| Run a host-owned LLM call | `ctx.llm.complete(...)` / `ctx.llm.complete_structured(...)` — borrow the user's active model + auth for a one-shot completion with optional JSON schema validation. See [Plugin LLM Access](/docs/developer-guide/plugin-llm-access) |
|
||||
@@ -230,6 +231,7 @@ The table above shows the four plugin categories, but within "General plugins" t
|
||||
| A **memory backend** (Honcho, Mem0, Supermemory, …) | Memory plugin — subclass `MemoryProvider` in `plugins/memory/<name>/` | [Memory Provider Plugins](/docs/developer-guide/memory-provider-plugin) |
|
||||
| A **context-compression strategy** | Context-engine plugin — `ctx.register_context_engine()` | [Context Engine Plugins](/docs/developer-guide/context-engine-plugin) |
|
||||
| An **image-generation backend** (DALL·E, SDXL, …) | Backend plugin — `ctx.register_image_gen_provider()` | [Image Generation Provider Plugins](/docs/developer-guide/image-gen-provider-plugin) |
|
||||
| A **video-generation backend** (Veo, Kling, Pixverse, Grok-Imagine, Runway, …) | Backend plugin — `ctx.register_video_gen_provider()` | [Video Generation Provider Plugins](/docs/developer-guide/video-gen-provider-plugin) |
|
||||
| A **TTS backend** (any CLI — Piper, VoxCPM, Kokoro, xtts, voice-cloning scripts, …) | Config-driven — declare under `tts.providers.<name>` with `type: command` in `config.yaml` | [TTS setup](/docs/user-guide/features/tts#custom-command-providers) |
|
||||
| An **STT backend** (custom whisper binary, local ASR CLI) | Config-driven — set `HERMES_LOCAL_STT_COMMAND` env var to a shell template | [Voice Message Transcription (STT)](/docs/user-guide/features/tts#voice-message-transcription-stt) |
|
||||
| **External tools via MCP** (filesystem, GitHub, Linear, Notion, any MCP server) | Config-driven — declare `mcp_servers.<name>` with `command:` / `url:` in `config.yaml`. Hermes auto-discovers the server's tools and registers them alongside built-ins. | [MCP](/docs/user-guide/features/mcp) |
|
||||
|
||||
@@ -1,234 +0,0 @@
|
||||
---
|
||||
sidebar_position: 13
|
||||
title: "RL Training"
|
||||
description: "Reinforcement learning on agent behaviors with Tinker-Atropos — environment discovery, training, and evaluation"
|
||||
---
|
||||
|
||||
# RL Training
|
||||
|
||||
Hermes Agent includes an integrated RL (Reinforcement Learning) training pipeline built on **Tinker-Atropos**. This enables training language models on environment-specific tasks using GRPO (Group Relative Policy Optimization) with LoRA adapters, orchestrated entirely through the agent's tool interface.
|
||||
|
||||
## Overview
|
||||
|
||||
The RL training system consists of three components:
|
||||
|
||||
1. **[Atropos](https://github.com/NousResearch/atropos)** — A trajectory API server that coordinates environment interactions, manages rollout groups, and computes advantages
|
||||
2. **[Tinker](https://thinkingmachines.ai/tinker/)** — A training service that handles model weights, LoRA training, sampling/inference, and optimizer steps
|
||||
3. **Environments** — Python classes that define tasks, scoring, and reward functions (e.g., GSM8K math problems)
|
||||
|
||||
The agent can discover environments, configure training parameters, launch training runs, and monitor metrics — all through a set of `rl_*` tools.
|
||||
|
||||
## Requirements
|
||||
|
||||
RL training requires:
|
||||
|
||||
- **Python >= 3.11** (Tinker package requirement)
|
||||
- **TINKER_API_KEY** — API key for the Tinker training service
|
||||
- **WANDB_API_KEY** — API key for [Weights & Biases](https://wandb.ai/) metrics tracking
|
||||
- The `tinker-atropos` submodule (at `tinker-atropos/` relative to the Hermes root)
|
||||
|
||||
```bash
|
||||
# Set up API keys
|
||||
hermes config set TINKER_API_KEY your-tinker-key
|
||||
hermes config set WANDB_API_KEY your-wandb-key
|
||||
```
|
||||
|
||||
When both keys are present and Python >= 3.11 is available, the `rl` toolset is automatically enabled.
|
||||
|
||||
## Available Tools
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `rl_list_environments` | Discover available RL environments |
|
||||
| `rl_select_environment` | Select an environment and load its config |
|
||||
| `rl_get_current_config` | View configurable and locked fields |
|
||||
| `rl_edit_config` | Modify configurable training parameters |
|
||||
| `rl_start_training` | Launch a training run (spawns 3 processes) |
|
||||
| `rl_check_status` | Monitor training progress and WandB metrics |
|
||||
| `rl_stop_training` | Stop a running training job |
|
||||
| `rl_get_results` | Get final metrics and model weights path |
|
||||
| `rl_list_runs` | List all active and completed runs |
|
||||
| `rl_test_inference` | Quick inference test using OpenRouter |
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Discover Environments
|
||||
|
||||
```
|
||||
List the available RL environments
|
||||
```
|
||||
|
||||
The agent calls `rl_list_environments()` which scans `tinker-atropos/tinker_atropos/environments/` using AST parsing to find Python classes inheriting from `BaseEnv`. Each environment defines:
|
||||
|
||||
- **Dataset loading** — where training data comes from (e.g., HuggingFace datasets)
|
||||
- **Prompt construction** — how to format items for the model
|
||||
- **Scoring/verification** — how to evaluate model outputs and assign rewards
|
||||
|
||||
### 2. Select and Configure
|
||||
|
||||
```
|
||||
Select the GSM8K environment and show me the configuration
|
||||
```
|
||||
|
||||
The agent calls `rl_select_environment("gsm8k_tinker")`, then `rl_get_current_config()` to see all parameters.
|
||||
|
||||
Configuration fields are divided into two categories:
|
||||
|
||||
**Configurable fields** (can be modified):
|
||||
- `group_size` — Number of completions per item (default: 16)
|
||||
- `batch_size` — Training batch size (default: 128)
|
||||
- `wandb_name` — WandB run name (auto-set to `{env}-{timestamp}`)
|
||||
- Other environment-specific parameters
|
||||
|
||||
**Locked fields** (infrastructure settings, cannot be changed):
|
||||
- `tokenizer_name` — Model tokenizer (e.g., `Qwen/Qwen3-8B`)
|
||||
- `rollout_server_url` — Atropos API URL (`http://localhost:8000`)
|
||||
- `max_token_length` — Maximum token length (8192)
|
||||
- `max_num_workers` — Maximum parallel workers (2048)
|
||||
- `total_steps` — Total training steps (2500)
|
||||
- `lora_rank` — LoRA adapter rank (32)
|
||||
- `learning_rate` — Learning rate (4e-5)
|
||||
- `max_token_trainer_length` — Max tokens for trainer (9000)
|
||||
|
||||
### 3. Start Training
|
||||
|
||||
```
|
||||
Start the training run
|
||||
```
|
||||
|
||||
The agent calls `rl_start_training()` which:
|
||||
|
||||
1. Generates a YAML config file merging locked settings with configurable overrides
|
||||
2. Creates a unique run ID
|
||||
3. Spawns three processes:
|
||||
- **Atropos API server** (`run-api`) — trajectory coordination
|
||||
- **Tinker trainer** (`launch_training.py`) — LoRA training + FastAPI inference server on port 8001
|
||||
- **Environment** (`environment.py serve`) — the selected environment connecting to Atropos
|
||||
|
||||
The processes start with staggered delays (5s for API, 30s for trainer, 90s more for environment) to ensure proper initialization order.
|
||||
|
||||
### 4. Monitor Progress
|
||||
|
||||
```
|
||||
Check the status of training run abc12345
|
||||
```
|
||||
|
||||
The agent calls `rl_check_status(run_id)` which reports:
|
||||
|
||||
- Process status (running/exited for each of the 3 processes)
|
||||
- Running time
|
||||
- WandB metrics (step, reward mean, percent correct, eval accuracy)
|
||||
- Log file locations for debugging
|
||||
|
||||
:::note Rate Limiting
|
||||
Status checks are rate-limited to once every **30 minutes** per run ID. This prevents excessive polling during long-running training jobs that take hours.
|
||||
:::
|
||||
|
||||
### 5. Stop or Get Results
|
||||
|
||||
```
|
||||
Stop the training run
|
||||
# or
|
||||
Get the final results for run abc12345
|
||||
```
|
||||
|
||||
`rl_stop_training()` terminates all three processes in reverse order (environment → trainer → API). `rl_get_results()` retrieves final WandB metrics and training history.
|
||||
|
||||
## Inference Testing
|
||||
|
||||
Before committing to a full training run, you can test if an environment works correctly using `rl_test_inference`. This runs a few steps of inference and scoring using OpenRouter — no Tinker API needed, just an `OPENROUTER_API_KEY`.
|
||||
|
||||
```
|
||||
Test the selected environment with inference
|
||||
```
|
||||
|
||||
Default configuration:
|
||||
- **3 steps × 16 completions = 48 rollouts per model**
|
||||
- Tests 3 models at different scales for robustness:
|
||||
- `qwen/qwen3-8b` (small)
|
||||
- `z-ai/glm-4.7-flash` (medium)
|
||||
- `minimax/minimax-m2.7` (large)
|
||||
- Total: ~144 rollouts
|
||||
|
||||
This validates:
|
||||
- Environment loads correctly
|
||||
- Prompt construction works
|
||||
- Inference response parsing is robust across model scales
|
||||
- Verifier/scoring logic produces valid rewards
|
||||
|
||||
## Tinker API Integration
|
||||
|
||||
The trainer uses the [Tinker](https://tinker.computer) API for model training operations:
|
||||
|
||||
- **ServiceClient** — Creates training and sampling clients
|
||||
- **Training client** — Handles forward-backward passes with importance sampling loss, optimizer steps (Adam), and weight checkpointing
|
||||
- **Sampling client** — Provides inference using the latest trained weights
|
||||
|
||||
The training loop:
|
||||
1. Fetches a batch of rollouts from Atropos (prompt + completions + scores)
|
||||
2. Converts to Tinker Datum objects with padded logprobs and advantages
|
||||
3. Runs forward-backward pass with importance sampling loss
|
||||
4. Takes an optimizer step (Adam: lr=4e-5, β1=0.9, β2=0.95)
|
||||
5. Saves weights and creates a new sampling client for next-step inference
|
||||
6. Logs metrics to WandB
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
api["Atropos API<br/>run-api<br/>port 8000"]
|
||||
env["Environment<br/>BaseEnv implementation"]
|
||||
infer["OpenAI / sglang<br/>inference API<br/>port 8001"]
|
||||
trainer["Tinker Trainer<br/>LoRA training + FastAPI"]
|
||||
|
||||
env <--> api
|
||||
env --> infer
|
||||
api -->|"batches: tokens, scores, logprobs"| trainer
|
||||
trainer -->|"serves inference"| infer
|
||||
```
|
||||
|
||||
## Creating Custom Environments
|
||||
|
||||
To create a new RL environment:
|
||||
|
||||
1. Create a Python file in `tinker-atropos/tinker_atropos/environments/`
|
||||
2. Define a class that inherits from `BaseEnv`
|
||||
3. Implement the required methods:
|
||||
- `load_dataset()` — Load your training data
|
||||
- `get_next_item()` — Provide the next item to the model
|
||||
- `score_answer()` — Score model outputs and assign rewards
|
||||
- `collect_trajectories()` — Collect and return trajectories
|
||||
4. Optionally define a custom config class inheriting from `BaseEnvConfig`
|
||||
|
||||
Study the existing `gsm8k_tinker.py` as a template. The agent can help you create new environments — it can read existing environment files, inspect HuggingFace datasets, and write new environment code.
|
||||
|
||||
## WandB Metrics
|
||||
|
||||
Training runs log to Weights & Biases with these key metrics:
|
||||
|
||||
| Metric | Description |
|
||||
|--------|-------------|
|
||||
| `train/loss` | Training loss (importance sampling) |
|
||||
| `train/learning_rate` | Current learning rate |
|
||||
| `reward/mean` | Mean reward across groups |
|
||||
| `logprobs/mean` | Mean reference logprobs |
|
||||
| `logprobs/mean_training` | Mean training logprobs |
|
||||
| `logprobs/diff` | Logprob drift (reference - training) |
|
||||
| `advantages/mean` | Mean advantage values |
|
||||
| `advantages/std` | Advantage standard deviation |
|
||||
|
||||
## Log Files
|
||||
|
||||
Each training run generates log files in `~/.hermes/logs/rl_training/`:
|
||||
|
||||
```
|
||||
logs/
|
||||
├── api_{run_id}.log # Atropos API server logs
|
||||
├── trainer_{run_id}.log # Tinker trainer logs
|
||||
├── env_{run_id}.log # Environment process logs
|
||||
└── inference_tests/ # Inference test results
|
||||
├── test_{env}_{model}.jsonl
|
||||
└── test_{env}_{model}.log
|
||||
```
|
||||
|
||||
These are invaluable for debugging when training fails or produces unexpected results.
|
||||
@@ -351,6 +351,7 @@ Hermes can install directly from GitHub repositories and GitHub-based taps. This
|
||||
Default taps (browsable without any setup):
|
||||
- [openai/skills](https://github.com/openai/skills)
|
||||
- [anthropics/skills](https://github.com/anthropics/skills)
|
||||
- [huggingface/skills](https://github.com/huggingface/skills)
|
||||
- [VoltAgent/awesome-agent-skills](https://github.com/VoltAgent/awesome-agent-skills)
|
||||
- [garrytan/gstack](https://github.com/garrytan/gstack)
|
||||
|
||||
@@ -445,7 +446,7 @@ Important behavior:
|
||||
|-------|--------|--------|
|
||||
| `builtin` | Ships with Hermes | Always trusted |
|
||||
| `official` | `optional-skills/` in the repo | Builtin trust, no third-party warning |
|
||||
| `trusted` | Trusted registries/repos such as `openai/skills`, `anthropics/skills` | More permissive policy than community sources |
|
||||
| `trusted` | Trusted registries/repos such as `openai/skills`, `anthropics/skills`, `huggingface/skills` | More permissive policy than community sources |
|
||||
| `community` | Everything else (`skills.sh`, well-known endpoints, custom GitHub repos, most marketplaces) | Non-dangerous findings can be overridden with `--force`; `dangerous` verdicts stay blocked |
|
||||
|
||||
### Update lifecycle
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
---
|
||||
sidebar_position: 15
|
||||
title: "Subscription Proxy"
|
||||
description: "Use your Nous Portal subscription (or other OAuth provider) as an OpenAI-compatible endpoint for external apps"
|
||||
---
|
||||
|
||||
# Subscription Proxy
|
||||
|
||||
The subscription proxy is a local HTTP server that lets external apps —
|
||||
OpenViking, Karakeep, Open WebUI, anything that speaks OpenAI-compatible
|
||||
chat completions — use your Hermes-managed provider subscription as their
|
||||
LLM endpoint. The proxy attaches the right credentials (refreshing them
|
||||
automatically) so the app never needs a static API key.
|
||||
|
||||
This is different from the [API server](./api-server.md):
|
||||
|
||||
| | API server | Subscription proxy |
|
||||
|---|---|---|
|
||||
| What it serves | Your agent (full toolset, memory, skills) | Raw model inference |
|
||||
| Use case | "Use Hermes as a chat backend" | "Use my Portal sub from another app" |
|
||||
| Auth | Your `API_SERVER_KEY` | Any bearer (proxy attaches the real one) |
|
||||
| Tool calls | Yes — the agent runs tools | No — passthrough only |
|
||||
|
||||
Use the API server when you want the **agent** as a backend. Use the
|
||||
proxy when you just want **the model** through your subscription.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Log into your provider (one-time)
|
||||
|
||||
```bash
|
||||
hermes login nous
|
||||
```
|
||||
|
||||
This opens your browser for the Nous Portal OAuth flow. Hermes stores
|
||||
the refresh token in `~/.hermes/auth.json` — the same place all Hermes
|
||||
provider logins live.
|
||||
|
||||
### 2. Start the proxy
|
||||
|
||||
```bash
|
||||
hermes proxy start
|
||||
```
|
||||
|
||||
```
|
||||
Starting Hermes proxy for Nous Portal
|
||||
Listening on: http://127.0.0.1:8645/v1
|
||||
Forwarding to: (resolved per-request from your subscription)
|
||||
Use any bearer token in the client — the proxy attaches your real credential.
|
||||
```
|
||||
|
||||
Leave this running in the foreground. Use `tmux`, `nohup`, or a systemd
|
||||
unit if you want it to survive logout.
|
||||
|
||||
### 3. Point your app at it
|
||||
|
||||
Any OpenAI-compatible app config takes the same triple:
|
||||
|
||||
```
|
||||
Base URL: http://127.0.0.1:8645/v1
|
||||
API key: anything (e.g. "sk-unused")
|
||||
Model: Hermes-4-70B # or Hermes-4.3-36B, Hermes-4-405B
|
||||
```
|
||||
|
||||
The proxy ignores the `Authorization` header from your app and attaches
|
||||
your real Portal credential to the upstream request. Refreshes happen
|
||||
automatically when the bearer approaches expiry.
|
||||
|
||||
## Available providers
|
||||
|
||||
```bash
|
||||
hermes proxy providers
|
||||
```
|
||||
|
||||
Currently shipped: `nous` (Nous Portal). More OAuth providers can be
|
||||
added by implementing the `UpstreamAdapter` interface in
|
||||
`hermes_cli/proxy/adapters/`.
|
||||
|
||||
## Check status
|
||||
|
||||
```bash
|
||||
hermes proxy status
|
||||
```
|
||||
|
||||
```
|
||||
Hermes proxy upstream adapters
|
||||
|
||||
[nous ] Nous Portal — ready (bearer expires 2026-05-15T06:43:21Z)
|
||||
```
|
||||
|
||||
If you see `not logged in`, run `hermes login nous`. If you see
|
||||
`credentials need attention`, your refresh token was revoked (rare —
|
||||
happens if you signed out from the Portal web UI) — just re-run
|
||||
`hermes login nous`.
|
||||
|
||||
## Allowed paths
|
||||
|
||||
The proxy only forwards paths the upstream actually serves. For Nous
|
||||
Portal:
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `/v1/chat/completions` | Chat completions (streaming + non-streaming) |
|
||||
| `/v1/completions` | Legacy text completions |
|
||||
| `/v1/embeddings` | Embeddings |
|
||||
| `/v1/models` | Model list |
|
||||
|
||||
Other paths (`/v1/images/generations`, `/v1/audio/speech`, etc.) return
|
||||
404 with a clear error pointing at the allowed paths. This keeps stray
|
||||
clients from leaking weird requests to the upstream.
|
||||
|
||||
## Configuring OpenViking to use Portal
|
||||
|
||||
[OpenViking](https://github.com/volcengine/OpenViking) is a context
|
||||
database that needs an LLM provider for its VLM (vision/language model
|
||||
used to extract memories) and embedding model. With the proxy, you can
|
||||
point its `vlm.api_base` at your local proxy:
|
||||
|
||||
Edit `~/.openviking/ov.conf`:
|
||||
|
||||
```json
|
||||
{
|
||||
"vlm": {
|
||||
"provider": "openai",
|
||||
"model": "Hermes-4-70B",
|
||||
"api_base": "http://127.0.0.1:8645/v1",
|
||||
"api_key": "unused-proxy-attaches-real-creds"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then start your proxy in a terminal alongside `openviking-server`:
|
||||
|
||||
```bash
|
||||
# Terminal 1
|
||||
hermes proxy start
|
||||
|
||||
# Terminal 2
|
||||
openviking-server
|
||||
```
|
||||
|
||||
OpenViking's VLM calls now flow through your Portal subscription. The
|
||||
embedding model side still needs its own provider — Portal does serve
|
||||
`/v1/embeddings` but the model selection depends on what your tier
|
||||
supports; check `portal.nousresearch.com/models`.
|
||||
|
||||
## Configuring Karakeep (or any bookmark/summarizer app)
|
||||
|
||||
[Karakeep](https://karakeep.app/) takes an OpenAI-compatible API for
|
||||
bookmark summarization. In its config:
|
||||
|
||||
```bash
|
||||
# Karakeep .env
|
||||
OPENAI_API_BASE_URL=http://127.0.0.1:8645/v1
|
||||
OPENAI_API_KEY=any-non-empty-string
|
||||
INFERENCE_TEXT_MODEL=Hermes-4-70B
|
||||
```
|
||||
|
||||
Same pattern works for Open WebUI, LobeChat, NextChat, or any other
|
||||
OpenAI-compatible client.
|
||||
|
||||
## Exposing on LAN
|
||||
|
||||
By default the proxy binds `127.0.0.1` (localhost only). To let other
|
||||
machines on your network use it:
|
||||
|
||||
```bash
|
||||
hermes proxy start --host 0.0.0.0 --port 8645
|
||||
```
|
||||
|
||||
⚠ **Be aware:** anyone on your network can now use your Portal
|
||||
subscription. The proxy has no auth of its own — it accepts any bearer.
|
||||
Use a firewall, VPN, or reverse proxy with proper auth if you expose
|
||||
this beyond your trusted network.
|
||||
|
||||
## Rate limits
|
||||
|
||||
Your Portal tier's RPM/TPM limits apply across the whole proxy. The
|
||||
proxy doesn't fan out or pool — it's a single bearer with your full
|
||||
subscription quota. Monitor usage at
|
||||
[portal.nousresearch.com](https://portal.nousresearch.com).
|
||||
|
||||
## Architecture
|
||||
|
||||
The proxy is intentionally minimal. Per request:
|
||||
|
||||
1. Receive `POST /v1/chat/completions` from your app
|
||||
2. Look up the adapter's current credential (refresh if expiring)
|
||||
3. Forward the request body verbatim, with `Authorization: Bearer <minted-key>`
|
||||
4. Stream the response back unchanged (SSE preserved)
|
||||
|
||||
No transformation. No logging of request bodies. No agent loop. The
|
||||
proxy is a credential-attaching pass-through.
|
||||
|
||||
## Future: more OAuth providers
|
||||
|
||||
The adapter system is pluggable. Adding a new provider (e.g.
|
||||
HuggingFace, GitHub Copilot's chat endpoint, Anthropic via OAuth)
|
||||
requires implementing `UpstreamAdapter` in
|
||||
`hermes_cli/proxy/adapters/<provider>.py` and registering it in
|
||||
`adapters/__init__.py`. Providers that aren't OpenAI-compatible at the
|
||||
protocol level (Anthropic Messages API, for example) would need a
|
||||
transformation layer, which is out of scope for the current shape.
|
||||
@@ -277,6 +277,7 @@ Discord behavior is controlled through two files: **`~/.hermes/.env`** for crede
|
||||
| `DISCORD_HOME_CHANNEL_NAME` | No | `"Home"` | Display name for the home channel in logs and status output. |
|
||||
| `DISCORD_COMMAND_SYNC_POLICY` | No | `"safe"` | Controls native slash-command startup sync. `"safe"` diffs existing global commands and only updates what changed, recreating commands when Discord metadata changes cannot be applied via patch. `"bulk"` preserves the old `tree.sync()` behavior. `"off"` skips startup sync entirely. |
|
||||
| `DISCORD_REQUIRE_MENTION` | No | `true` | When `true`, the bot only responds in server channels when `@mentioned`. Set to `false` to respond to all messages in every channel. |
|
||||
| `DISCORD_THREAD_REQUIRE_MENTION` | No | `false` | When `true`, the in-thread mention shortcut is disabled — threads are gated the same as channels, requiring `@mention` even after the bot has already participated. Use this when multiple bots share a thread and you want each to fire only on explicit `@mention`. |
|
||||
| `DISCORD_FREE_RESPONSE_CHANNELS` | No | — | Comma-separated channel IDs where the bot responds without requiring an `@mention`, even when `DISCORD_REQUIRE_MENTION` is `true`. |
|
||||
| `DISCORD_IGNORE_NO_MENTION` | No | `true` | When `true`, the bot stays silent if a message `@mentions` other users but does **not** mention the bot. Prevents the bot from jumping into conversations directed at other people. Only applies in server channels, not DMs. |
|
||||
| `DISCORD_AUTO_THREAD` | No | `true` | When `true`, automatically creates a new thread for every `@mention` in a text channel, so each conversation is isolated (similar to Slack behavior). Messages already inside threads or DMs are unaffected. |
|
||||
@@ -285,6 +286,8 @@ Discord behavior is controlled through two files: **`~/.hermes/.env`** for crede
|
||||
| `DISCORD_IGNORED_CHANNELS` | No | — | Comma-separated channel IDs where the bot **never** responds, even when `@mentioned`. Takes priority over all other channel settings. |
|
||||
| `DISCORD_ALLOWED_CHANNELS` | No | — | Comma-separated channel IDs. When set, the bot **only** responds in these channels (plus DMs if allowed). Overrides `config.yaml` `discord.allowed_channels`. Combine with `DISCORD_IGNORED_CHANNELS` to express allow/deny rules. |
|
||||
| `DISCORD_NO_THREAD_CHANNELS` | No | — | Comma-separated channel IDs where the bot responds directly in the channel instead of creating a thread. Only relevant when `DISCORD_AUTO_THREAD` is `true`. |
|
||||
| `DISCORD_HISTORY_BACKFILL` | No | `true` | When `true`, prepend recent channel scrollback (since the bot's last response) to the user message when the bot is mentioned. Recovers context the bot would otherwise miss with `require_mention`. Skipped in DMs and free-response channels. Set to `false` to disable. |
|
||||
| `DISCORD_HISTORY_BACKFILL_LIMIT` | No | `50` | Maximum number of messages to scan backwards when assembling the backfill block. In practice the scan usually stops earlier — at the bot's own last message in the channel. |
|
||||
| `DISCORD_REPLY_TO_MODE` | No | `"first"` | Controls reply-reference behavior: `"off"` — never reply to the original message, `"first"` — reply-reference on the first message chunk only (default), `"all"` — reply-reference on every chunk. |
|
||||
| `DISCORD_ALLOW_MENTION_EVERYONE` | No | `false` | When `false` (default), the bot cannot ping `@everyone` or `@here` even if its response contains those tokens. Set to `true` to opt back in. See [Mention Control](#mention-control) below. |
|
||||
| `DISCORD_ALLOW_MENTION_ROLES` | No | `false` | When `false` (default), the bot cannot ping `@role` mentions. Set to `true` to allow. |
|
||||
@@ -302,11 +305,14 @@ The `discord` section in `~/.hermes/config.yaml` mirrors the env vars above. Con
|
||||
# Discord-specific settings
|
||||
discord:
|
||||
require_mention: true # Require @mention in server channels
|
||||
thread_require_mention: false # If true, require @mention in threads too (multi-bot threads)
|
||||
free_response_channels: "" # Comma-separated channel IDs (or YAML list)
|
||||
auto_thread: true # Auto-create threads on @mention
|
||||
reactions: true # Add emoji reactions during processing
|
||||
ignored_channels: [] # Channel IDs where bot never responds
|
||||
no_thread_channels: [] # Channel IDs where bot responds without threading
|
||||
history_backfill: true # Prepend recent channel scrollback on mention (default: true)
|
||||
history_backfill_limit: 50 # Max messages to scan backwards (default: 50)
|
||||
channel_prompts: {} # Per-channel ephemeral system prompts
|
||||
allow_mentions: # What the bot is allowed to ping (safe defaults)
|
||||
everyone: false # @everyone / @here pings (default: false)
|
||||
@@ -324,6 +330,20 @@ group_sessions_per_user: true # Isolate sessions per user in shared channels
|
||||
|
||||
When enabled, the bot only responds in server channels when directly `@mentioned`. DMs always get a response regardless of this setting.
|
||||
|
||||
#### `discord.thread_require_mention`
|
||||
|
||||
**Type:** boolean — **Default:** `false`
|
||||
|
||||
By default, once the bot has participated in a thread (auto-created on `@mention` or replied in once), it keeps responding to every subsequent message in that thread without needing to be `@mentioned` again. That's the right default for one-on-one conversations.
|
||||
|
||||
In **multi-bot threads** where users address one bot per turn, this default becomes a footgun — every other bot in the thread also fires on every message, burning credits and spamming the channel. Set `thread_require_mention: true` to disable the in-thread shortcut and gate threads the same way channels are gated. Explicit `@mentions` still work as before.
|
||||
|
||||
```yaml
|
||||
discord:
|
||||
require_mention: true
|
||||
thread_require_mention: true # multi-bot setup
|
||||
```
|
||||
|
||||
#### `discord.free_response_channels`
|
||||
|
||||
**Type:** string or list — **Default:** `""`
|
||||
@@ -350,7 +370,7 @@ Free-response channels also **skip auto-threading** — the bot replies inline r
|
||||
|
||||
**Type:** boolean — **Default:** `true`
|
||||
|
||||
When enabled, every `@mention` in a regular text channel automatically creates a new thread for the conversation. This keeps the main channel clean and gives each conversation its own isolated session history. Once a thread is created, subsequent messages in that thread don't require `@mention` — the bot knows it's already participating.
|
||||
When enabled, every `@mention` in a regular text channel automatically creates a new thread for the conversation. This keeps the main channel clean and gives each conversation its own isolated session history. Once a thread is created, subsequent messages in that thread don't require `@mention` — the bot knows it's already participating. Set [`thread_require_mention`](#discordthread_require_mention) to `true` to disable this in-thread shortcut for multi-bot setups.
|
||||
|
||||
Messages sent in existing threads or DMs are unaffected by this setting. Channels listed in `discord.free_response_channels` or `discord.no_thread_channels` also bypass auto-threading and get inline replies instead.
|
||||
|
||||
@@ -421,6 +441,47 @@ Behavior:
|
||||
- If a message arrives inside a thread or forum post and that thread has no explicit entry, Hermes falls back to the parent channel/forum ID.
|
||||
- Prompts are applied ephemerally at runtime, so changing them affects future turns immediately without rewriting past session history.
|
||||
|
||||
#### `discord.history_backfill`
|
||||
|
||||
**Type:** boolean — **Default:** `true`
|
||||
|
||||
When enabled, the bot recovers missed channel messages on each `@mention`. With `require_mention: true`, the bot only processes messages that tag it directly — everything else in the channel is invisible to the session transcript. History backfill scans backwards through recent channel history when triggered, collecting messages between the bot's last response and the current mention, and includes them as context.
|
||||
|
||||
Behavior by surface:
|
||||
|
||||
- **Server channels** (with `require_mention: true`): backfill scans the channel since the bot's last response. Useful when other participants posted while the bot wasn't addressed.
|
||||
- **Threads**: backfill scans the thread only — Discord's `channel.history()` on a thread returns only that thread's messages, not the parent channel. This is the right scope because threads are usually self-contained conversations.
|
||||
- **DMs**: skipped. Every DM message triggers the bot, so the session transcript is already complete — there's no mention gap to fill.
|
||||
- **Free-response channels** and **bot's own auto-created threads**: skipped for the same reason — no mention gating means no gap.
|
||||
|
||||
Per-user sessions (`group_sessions_per_user: true`, the default) also benefit: a user's session is missing the context posted by other channel participants and the user's own messages from before they tagged the bot. Backfill fills both gaps.
|
||||
|
||||
```yaml
|
||||
discord:
|
||||
history_backfill: true # default
|
||||
```
|
||||
|
||||
To turn it off:
|
||||
|
||||
```yaml
|
||||
discord:
|
||||
history_backfill: false
|
||||
```
|
||||
|
||||
> **Note:** Messages that arrive *while* the bot is processing (between a trigger and its response) are not captured. This is an accepted simplification — the user can re-send or tag again.
|
||||
|
||||
#### `discord.history_backfill_limit`
|
||||
|
||||
**Type:** integer — **Default:** `50`
|
||||
|
||||
Maximum number of messages to scan backwards when recovering channel context. In practice the scan usually stops much earlier — at the bot's own last message in the channel, which is the natural boundary between turns. This limit is a safety cap for cold starts and long gaps where no prior bot message exists in recent history.
|
||||
|
||||
```yaml
|
||||
discord:
|
||||
history_backfill: true
|
||||
history_backfill_limit: 50
|
||||
```
|
||||
|
||||
#### `group_sessions_per_user`
|
||||
|
||||
**Type:** boolean — **Default:** `true`
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# SimpleX Chat
|
||||
|
||||
[SimpleX Chat](https://simplex.chat/) is a private, decentralised messaging platform where users own their contacts and groups. Unlike other platforms, SimpleX assigns no persistent user IDs — every contact is identified by an opaque internal ID generated at connection time, which makes it one of the most private messengers available.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The **simplex-chat** CLI installed and running as a daemon
|
||||
- Python package **websockets** (`pip install websockets`)
|
||||
|
||||
## Install simplex-chat
|
||||
|
||||
Download the latest release from the [simplex-chat GitHub releases](https://github.com/simplex-chat/simplex-chat/releases) page, or via Docker:
|
||||
|
||||
```bash
|
||||
# Linux / macOS binary
|
||||
curl -L https://github.com/simplex-chat/simplex-chat/releases/latest/download/simplex-chat-ubuntu-22_04-x86-64 -o simplex-chat
|
||||
chmod +x simplex-chat
|
||||
|
||||
# Or Docker
|
||||
docker run -p 5225:5225 simplexchat/simplex-chat -p 5225
|
||||
```
|
||||
|
||||
## Start the daemon
|
||||
|
||||
```bash
|
||||
simplex-chat -p 5225
|
||||
```
|
||||
|
||||
The daemon listens on WebSocket at `ws://127.0.0.1:5225` by default.
|
||||
|
||||
## Configure Hermes
|
||||
|
||||
### Via setup wizard
|
||||
|
||||
```bash
|
||||
hermes setup gateway
|
||||
```
|
||||
|
||||
Select **SimpleX Chat** and follow the prompts.
|
||||
|
||||
### Via environment variables
|
||||
|
||||
Add these to `~/.hermes/.env`:
|
||||
|
||||
```
|
||||
SIMPLEX_WS_URL=ws://127.0.0.1:5225
|
||||
SIMPLEX_ALLOWED_USERS=<contact-id-1>,<contact-id-2>
|
||||
SIMPLEX_HOME_CHANNEL=<contact-id>
|
||||
```
|
||||
|
||||
| Variable | Required | Description |
|
||||
|---|---|---|
|
||||
| `SIMPLEX_WS_URL` | Yes | WebSocket URL of the simplex-chat daemon |
|
||||
| `SIMPLEX_ALLOWED_USERS` | Recommended | Comma-separated contact IDs allowed to use the agent |
|
||||
| `SIMPLEX_ALLOW_ALL_USERS` | Optional | Set `true` to allow every contact (use carefully) |
|
||||
| `SIMPLEX_HOME_CHANNEL` | Optional | Default contact ID for cron job delivery |
|
||||
| `SIMPLEX_HOME_CHANNEL_NAME` | Optional | Human label for the home channel |
|
||||
|
||||
## Find your contact ID
|
||||
|
||||
After starting the daemon, open a conversation with your agent contact. The contact ID will appear in session logs or via `hermes send_message action=list`.
|
||||
|
||||
## Authorization
|
||||
|
||||
By default **all contacts are denied**. You must either:
|
||||
|
||||
1. Set `SIMPLEX_ALLOWED_USERS` to a comma-separated list of contact IDs, or
|
||||
2. Use **DM pairing** — send any message to the bot and it will reply with a pairing code. Enter that code via `hermes gateway pair`.
|
||||
|
||||
## Using SimpleX with cron jobs
|
||||
|
||||
```python
|
||||
cronjob(
|
||||
action="create",
|
||||
schedule="every 1h",
|
||||
deliver="simplex", # uses SIMPLEX_HOME_CHANNEL
|
||||
prompt="Check for alerts and summarise."
|
||||
)
|
||||
```
|
||||
|
||||
Or target a specific contact:
|
||||
|
||||
```python
|
||||
send_message(target="simplex:<contact-id>", message="Done!")
|
||||
```
|
||||
|
||||
## Privacy notes
|
||||
|
||||
- SimpleX never reveals phone numbers or email addresses — contacts use opaque IDs
|
||||
- The connection between Hermes and the daemon is local WebSocket (`ws://127.0.0.1:5225`) — no data leaves your machine
|
||||
- Messages are end-to-end encrypted by the SimpleX protocol before reaching the daemon
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"Cannot reach daemon"** — Ensure `simplex-chat -p 5225` is running and the port matches `SIMPLEX_WS_URL`.
|
||||
|
||||
**"websockets not installed"** — Run `pip install websockets`.
|
||||
|
||||
**Messages not received** — Check that the contact's ID is in `SIMPLEX_ALLOWED_USERS` or approve them via DM pairing.
|
||||
@@ -264,6 +264,22 @@ For backward compatibility with older manifests, you can still type
|
||||
run the tests`. Free-form questions also work: `/hermes what's the
|
||||
weather?` is treated as a regular message.
|
||||
|
||||
### Using commands inside threads (the `!cmd` prefix)
|
||||
|
||||
Slack itself blocks native slash commands inside thread replies — try
|
||||
`/queue` in a thread and Slack responds with *"/queue is not supported
|
||||
in threads. Sorry!"* There is no app-side setting that re-enables them;
|
||||
Slack never delivers them to Hermes.
|
||||
|
||||
As a workaround, Hermes recognises a leading `!` as an alternate
|
||||
command prefix that works in threads (and anywhere else). Type
|
||||
`!queue`, `!stop`, `!model gpt-5.4`, etc. as a regular thread reply —
|
||||
Hermes treats it identically to the slash form and replies in the same
|
||||
thread.
|
||||
|
||||
Only the first token is checked against the known command list, so
|
||||
casual messages like `!nice work` pass through to the agent unchanged.
|
||||
|
||||
### Advanced: emit only the slash-commands array
|
||||
|
||||
If you maintain your Slack manifest by hand and just want the slash
|
||||
|
||||
@@ -25,6 +25,43 @@ The SQLite database stores:
|
||||
- Timestamps (started_at, ended_at)
|
||||
- Parent session ID (for compression-triggered session splitting)
|
||||
|
||||
### What Counts Toward Context
|
||||
|
||||
Hermes stores session history so it can resume conversations, but it does not
|
||||
keep re-sending every byte it has ever handled. On each turn, the model sees
|
||||
the selected system prompt, the current conversation window, and any content
|
||||
Hermes explicitly injects for that turn.
|
||||
|
||||
Media attachments are handled as turn-scoped inputs:
|
||||
|
||||
- Images may be attached natively to the next model call, or pre-analyzed into
|
||||
a text description when the active model does not support native vision.
|
||||
- Audio is transcribed into text when speech-to-text is configured.
|
||||
- Text documents can have their extracted text included; other document types
|
||||
are usually represented by a saved local path and a short note.
|
||||
- Attachment paths and extracted/derived text can appear in the transcript, but
|
||||
the raw image, audio, or binary file bytes are not repeatedly copied into
|
||||
future prompts.
|
||||
|
||||
For example, if a user sends an image and asks Hermes to make a meme from it,
|
||||
Hermes may inspect that image once with vision and run an image-processing
|
||||
script. Future turns do not automatically carry the original JPEG in context.
|
||||
They carry only whatever was written into the conversation, such as the user's
|
||||
request, a short image description, a local cache path, or the final assistant
|
||||
response.
|
||||
|
||||
The most common cause of context growth is not the media file itself. It is
|
||||
verbose text: pasted transcripts, full logs, large tool outputs, long diffs,
|
||||
repeated status reports, and detailed proof dumps. Prefer summaries, file
|
||||
paths, focused excerpts, and tool-backed lookups over copying large artifacts
|
||||
into chat.
|
||||
|
||||
:::tip
|
||||
Use `/compress` when a session gets long, `/new` for a fresh thread, and
|
||||
`hermes sessions prune` only when you want to delete old ended sessions from
|
||||
storage. Compression reduces the active context; it is not a privacy delete.
|
||||
:::
|
||||
|
||||
### Session Sources
|
||||
|
||||
Each session is tagged with its source platform:
|
||||
|
||||
@@ -1,249 +0,0 @@
|
||||
---
|
||||
title: "Base"
|
||||
sidebar_label: "Base"
|
||||
description: "Query Base (Ethereum L2) blockchain data with USD pricing — wallet balances, token info, transaction details, gas analysis, contract inspection, whale detect..."
|
||||
---
|
||||
|
||||
{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
|
||||
|
||||
# Base
|
||||
|
||||
Query Base (Ethereum L2) blockchain data with USD pricing — wallet balances, token info, transaction details, gas analysis, contract inspection, whale detection, and live network stats. Uses Base RPC + CoinGecko. No API key required.
|
||||
|
||||
## Skill metadata
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Source | Optional — install with `hermes skills install official/blockchain/base` |
|
||||
| Path | `optional-skills/blockchain/base` |
|
||||
| Version | `0.1.0` |
|
||||
| Author | youssefea |
|
||||
| License | MIT |
|
||||
| Platforms | linux, macos, windows |
|
||||
| Tags | `Base`, `Blockchain`, `Crypto`, `Web3`, `RPC`, `DeFi`, `EVM`, `L2`, `Ethereum` |
|
||||
|
||||
## Reference: full SKILL.md
|
||||
|
||||
:::info
|
||||
The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
|
||||
:::
|
||||
|
||||
# Base Blockchain Skill
|
||||
|
||||
Query Base (Ethereum L2) on-chain data enriched with USD pricing via CoinGecko.
|
||||
8 commands: wallet portfolio, token info, transactions, gas analysis,
|
||||
contract inspection, whale detection, network stats, and price lookup.
|
||||
|
||||
No API key needed. Uses only Python standard library (urllib, json, argparse).
|
||||
|
||||
---
|
||||
|
||||
## When to Use
|
||||
|
||||
- User asks for a Base wallet balance, token holdings, or portfolio value
|
||||
- User wants to inspect a specific transaction by hash
|
||||
- User wants ERC-20 token metadata, price, supply, or market cap
|
||||
- User wants to understand Base gas costs and L1 data fees
|
||||
- User wants to inspect a contract (ERC type detection, proxy resolution)
|
||||
- User wants to find large ETH transfers (whale detection)
|
||||
- User wants Base network health, gas price, or ETH price
|
||||
- User asks "what's the price of USDC/AERO/DEGEN/ETH?"
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
The helper script uses only Python standard library (urllib, json, argparse).
|
||||
No external packages required.
|
||||
|
||||
Pricing data comes from CoinGecko's free API (no key needed, rate-limited
|
||||
to ~10-30 requests/minute). For faster lookups, use `--no-prices` flag.
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
RPC endpoint (default): https://mainnet.base.org
|
||||
Override: export BASE_RPC_URL=https://your-private-rpc.com
|
||||
|
||||
Helper script path: ~/.hermes/skills/blockchain/base/scripts/base_client.py
|
||||
|
||||
```
|
||||
python3 base_client.py wallet <address> [--limit N] [--all] [--no-prices]
|
||||
python3 base_client.py tx <hash>
|
||||
python3 base_client.py token <contract_address>
|
||||
python3 base_client.py gas
|
||||
python3 base_client.py contract <address>
|
||||
python3 base_client.py whales [--min-eth N]
|
||||
python3 base_client.py stats
|
||||
python3 base_client.py price <contract_address_or_symbol>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Procedure
|
||||
|
||||
### 0. Setup Check
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
|
||||
# Optional: set a private RPC for better rate limits
|
||||
export BASE_RPC_URL="https://mainnet.base.org"
|
||||
|
||||
# Confirm connectivity
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py stats
|
||||
```
|
||||
|
||||
### 1. Wallet Portfolio
|
||||
|
||||
Get ETH balance and ERC-20 token holdings with USD values.
|
||||
Checks ~15 well-known Base tokens (USDC, WETH, AERO, DEGEN, etc.)
|
||||
via on-chain `balanceOf` calls. Tokens sorted by value, dust filtered.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py \
|
||||
wallet 0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045
|
||||
```
|
||||
|
||||
Flags:
|
||||
- `--limit N` — show top N tokens (default: 20)
|
||||
- `--all` — show all tokens, no dust filter, no limit
|
||||
- `--no-prices` — skip CoinGecko price lookups (faster, RPC-only)
|
||||
|
||||
Output includes: ETH balance + USD value, token list with prices sorted
|
||||
by value, dust count, total portfolio value in USD.
|
||||
|
||||
Note: Only checks known tokens. Unknown ERC-20s are not discovered.
|
||||
Use the `token` command with a specific contract address for any token.
|
||||
|
||||
### 2. Transaction Details
|
||||
|
||||
Inspect a full transaction by its hash. Shows ETH value transferred,
|
||||
gas used, fee in ETH/USD, status, and decoded ERC-20/ERC-721 transfers.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py \
|
||||
tx 0xabc123...your_tx_hash_here
|
||||
```
|
||||
|
||||
Output: hash, block, from, to, value (ETH + USD), gas price, gas used,
|
||||
fee, status, contract creation address (if any), token transfers.
|
||||
|
||||
### 3. Token Info
|
||||
|
||||
Get ERC-20 token metadata: name, symbol, decimals, total supply, price,
|
||||
market cap, and contract code size.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py \
|
||||
token 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913
|
||||
```
|
||||
|
||||
Output: name, symbol, decimals, total supply, price, market cap.
|
||||
Reads name/symbol/decimals directly from the contract via eth_call.
|
||||
|
||||
### 4. Gas Analysis
|
||||
|
||||
Detailed gas analysis with cost estimates for common operations.
|
||||
Shows current gas price, base fee trends over 10 blocks, block
|
||||
utilization, and estimated costs for ETH transfers, ERC-20 transfers,
|
||||
and swaps.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py gas
|
||||
```
|
||||
|
||||
Output: current gas price, base fee, block utilization, 10-block trend,
|
||||
cost estimates in ETH and USD.
|
||||
|
||||
Note: Base is an L2 — actual transaction costs include an L1 data
|
||||
posting fee that depends on calldata size and L1 gas prices. The
|
||||
estimates shown are for L2 execution only.
|
||||
|
||||
### 5. Contract Inspection
|
||||
|
||||
Inspect an address: determine if it's an EOA or contract, detect
|
||||
ERC-20/ERC-721/ERC-1155 interfaces, resolve EIP-1967 proxy
|
||||
implementation addresses.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py \
|
||||
contract 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913
|
||||
```
|
||||
|
||||
Output: is_contract, code size, ETH balance, detected interfaces
|
||||
(ERC-20, ERC-721, ERC-1155), ERC-20 metadata, proxy implementation
|
||||
address.
|
||||
|
||||
### 6. Whale Detector
|
||||
|
||||
Scan the most recent block for large ETH transfers with USD values.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py \
|
||||
whales --min-eth 1.0
|
||||
```
|
||||
|
||||
Note: scans the latest block only — point-in-time snapshot, not historical.
|
||||
Default threshold is 1.0 ETH (lower than Solana's default since ETH
|
||||
values are higher).
|
||||
|
||||
### 7. Network Stats
|
||||
|
||||
Live Base network health: latest block, chain ID, gas price, base fee,
|
||||
block utilization, transaction count, and ETH price.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py stats
|
||||
```
|
||||
|
||||
### 8. Price Lookup
|
||||
|
||||
Quick price check for any token by contract address or known symbol.
|
||||
|
||||
```bash
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py price ETH
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py price USDC
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py price AERO
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py price DEGEN
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py price 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913
|
||||
```
|
||||
|
||||
Known symbols: ETH, WETH, USDC, cbETH, AERO, DEGEN, TOSHI, BRETT,
|
||||
WELL, wstETH, rETH, cbBTC.
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **CoinGecko rate-limits** — free tier allows ~10-30 requests/minute.
|
||||
Price lookups use 1 request per token. Use `--no-prices` for speed.
|
||||
- **Public RPC rate-limits** — Base's public RPC limits requests.
|
||||
For production use, set BASE_RPC_URL to a private endpoint
|
||||
(Alchemy, QuickNode, Infura).
|
||||
- **Wallet shows known tokens only** — unlike Solana, EVM chains have no
|
||||
built-in "get all tokens" RPC. The wallet command checks ~15 popular
|
||||
Base tokens via `balanceOf`. Unknown ERC-20s won't appear. Use the
|
||||
`token` command for any specific contract.
|
||||
- **Token names read from contract** — if a contract doesn't implement
|
||||
`name()` or `symbol()`, these fields may be empty. Known tokens have
|
||||
hardcoded labels as fallback.
|
||||
- **Gas estimates are L2 only** — Base transaction costs include an L1
|
||||
data posting fee (depends on calldata size and L1 gas prices). The gas
|
||||
command estimates L2 execution cost only.
|
||||
- **Whale detector scans latest block only** — not historical. Results
|
||||
vary by the moment you query. Default threshold is 1.0 ETH.
|
||||
- **Proxy detection** — only EIP-1967 proxies are detected. Other proxy
|
||||
patterns (EIP-1167 minimal proxy, custom storage slots) are not checked.
|
||||
- **Retry on 429** — both RPC and CoinGecko calls retry up to 2 times
|
||||
with exponential backoff on rate-limit errors.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# Should print Base chain ID (8453), latest block, gas price, and ETH price
|
||||
python3 ~/.hermes/skills/blockchain/base/scripts/base_client.py stats
|
||||
```
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
title: "Evm — Read-only EVM client: wallets, tokens, gas across 8 chains"
|
||||
sidebar_label: "Evm"
|
||||
description: "Read-only EVM client: wallets, tokens, gas across 8 chains"
|
||||
---
|
||||
|
||||
{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
|
||||
|
||||
# Evm
|
||||
|
||||
Read-only EVM client: wallets, tokens, gas across 8 chains.
|
||||
|
||||
## Skill metadata
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Source | Optional — install with `hermes skills install official/blockchain/evm` |
|
||||
| Path | `optional-skills/blockchain/evm` |
|
||||
| Version | `1.0.0` |
|
||||
| Author | Mibayy (@Mibayy), youssefea (@youssefea), ethernet8023 (@ethernet8023), Hermes Agent |
|
||||
| License | MIT |
|
||||
| Platforms | linux, macos, windows |
|
||||
| Tags | `EVM`, `Ethereum`, `BNB`, `BSC`, `Base`, `Arbitrum`, `Polygon`, `Optimism`, `Avalanche`, `zkSync`, `Blockchain`, `Crypto`, `Web3`, `DeFi`, `NFT`, `ENS`, `Whale`, `Security` |
|
||||
| Related skills | [`solana`](/docs/user-guide/skills/optional/blockchain/blockchain-solana) |
|
||||
|
||||
## Reference: full SKILL.md
|
||||
|
||||
:::info
|
||||
The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
|
||||
:::
|
||||
|
||||
# EVM Blockchain Skill
|
||||
|
||||
Query EVM-compatible blockchain data across 8 chains with USD pricing.
|
||||
14 commands: wallet portfolio, token info, transactions, activity, gas tracker,
|
||||
network stats, price lookup, multi-chain scan, whale detection, ENS resolution,
|
||||
allowance checker, contract inspector, and transaction decoder.
|
||||
|
||||
Supports 8 chains: Ethereum, BNB Chain (BSC), Base, Arbitrum One, Polygon,
|
||||
Optimism, Avalanche (C-Chain), zkSync Era.
|
||||
|
||||
No API key needed. Zero external dependencies — Python standard library only
|
||||
(urllib, json, argparse, threading).
|
||||
|
||||
> **Supersedes the standalone `base` skill.** Base-specific tokens (AERO, DEGEN,
|
||||
> TOSHI, BRETT, WELL, cbETH, cbBTC, wstETH, rETH) and all Base RPC functionality
|
||||
> previously living under `optional-skills/blockchain/base/` have been folded
|
||||
> into this skill. Pass `--chain base` to any command for Base coverage.
|
||||
|
||||
---
|
||||
|
||||
## When to Use
|
||||
- User asks for a wallet balance or portfolio on any EVM chain
|
||||
- User wants to check the same wallet across ALL chains at once
|
||||
- User wants to inspect a transaction by hash (or decode what it did)
|
||||
- User wants ERC-20 token metadata, price, supply, or market cap
|
||||
- User wants recent transaction history for an address
|
||||
- User wants current gas prices or to compare fees across chains
|
||||
- User wants to find large whale transfers in recent blocks
|
||||
- User asks to resolve an ENS name (vitalik.eth) or reverse-lookup an address
|
||||
- User wants to check if a contract has dangerous token approvals
|
||||
- User wants to inspect a smart contract (proxy? ERC-20? ERC-721? bytecode size?)
|
||||
- User wants to compare gas costs across chains before a transaction
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
Python 3.8+ standard library only. No pip installs required.
|
||||
Pricing: CoinGecko free API (rate-limited, ~10-30 req/min).
|
||||
ENS: ensideas.com public API.
|
||||
Tx decoding: 4byte.directory public API.
|
||||
|
||||
Override RPC endpoint: `export EVM_RPC_URL=https://your-rpc.com`
|
||||
|
||||
Helper script path: `~/.hermes/skills/blockchain/evm/scripts/evm_client.py`
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```
|
||||
SCRIPT=~/.hermes/skills/blockchain/evm/scripts/evm_client.py
|
||||
|
||||
# Network & prices
|
||||
python3 $SCRIPT stats # Ethereum stats
|
||||
python3 $SCRIPT stats --chain arbitrum # Arbitrum stats
|
||||
python3 $SCRIPT compare # Gas + prices ALL 8 chains
|
||||
|
||||
# Wallet
|
||||
python3 $SCRIPT wallet 0xd8dA...96045 # Portfolio (ETH + ERC-20)
|
||||
python3 $SCRIPT wallet 0xd8dA...96045 --chain bsc
|
||||
python3 $SCRIPT multichain 0xd8dA...96045 # Same wallet on ALL chains
|
||||
|
||||
# Tokens & prices
|
||||
python3 $SCRIPT price ETH
|
||||
python3 $SCRIPT price 0xdAC1...1ec7 # By contract address
|
||||
python3 $SCRIPT token 0xdAC1...1ec7 # ERC-20 metadata + market cap
|
||||
|
||||
# Transactions
|
||||
python3 $SCRIPT tx 0x5c50...f060 # Transaction details
|
||||
python3 $SCRIPT decode 0x5c50...f060 # Decode input data (4byte.directory)
|
||||
python3 $SCRIPT activity 0xd8dA...96045 # Recent transactions
|
||||
|
||||
# Gas
|
||||
python3 $SCRIPT gas # Gas prices + cost estimates
|
||||
python3 $SCRIPT gas --chain optimism
|
||||
|
||||
# Security
|
||||
python3 $SCRIPT allowance 0xd8dA...96045 # Dangerous ERC-20 approvals
|
||||
python3 $SCRIPT contract 0xdAC1...1ec7 # Contract inspection (proxy? standards?)
|
||||
|
||||
# ENS
|
||||
python3 $SCRIPT ens vitalik.eth # Name -> address + profile
|
||||
python3 $SCRIPT ens 0xd8dA...96045 # Address -> ENS name
|
||||
|
||||
# Whale detection
|
||||
python3 $SCRIPT whale # Large transfers (last 20 blocks, >$10k)
|
||||
python3 $SCRIPT whale --blocks 50 --min-usd 100000 --chain arbitrum
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Procedure
|
||||
|
||||
### 0. Setup Check
|
||||
```bash
|
||||
python3 --version # 3.8+ required
|
||||
python3 ~/.hermes/skills/blockchain/evm/scripts/evm_client.py stats
|
||||
```
|
||||
|
||||
### 1. Wallet Portfolio
|
||||
Native balance + known ERC-20 tokens, sorted by USD value.
|
||||
```bash
|
||||
python3 $SCRIPT wallet 0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045
|
||||
python3 $SCRIPT wallet 0xd8dA... --chain bsc --no-prices # faster
|
||||
```
|
||||
|
||||
### 2. Multi-Chain Scan
|
||||
Scans all 8 chains simultaneously for the same address using threads.
|
||||
```bash
|
||||
python3 $SCRIPT multichain 0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045
|
||||
```
|
||||
Output: per-chain native balance + token holdings + grand total USD.
|
||||
|
||||
### 3. Compare (Gas + Prices)
|
||||
All 8 chains queried in parallel. Shows cheapest/most expensive chain.
|
||||
```bash
|
||||
python3 $SCRIPT compare
|
||||
```
|
||||
|
||||
### 4. Transaction Details & Decode
|
||||
```bash
|
||||
python3 $SCRIPT tx 0x5c504ed432cb51138bcf09aa5e8a410dd4a1e204ef84bfed1be16dfba1b22060
|
||||
python3 $SCRIPT decode 0x5c504ed... # Shows human-readable function signature
|
||||
```
|
||||
Decode uses 4byte.directory to translate 0xa9059cbb -> transfer(address,uint256).
|
||||
|
||||
### 5. ENS Resolution
|
||||
```bash
|
||||
python3 $SCRIPT ens vitalik.eth # -> 0xd8dA... + avatar + social links
|
||||
python3 $SCRIPT ens 0xd8dA...96045 # -> vitalik.eth
|
||||
```
|
||||
|
||||
### 6. Allowance Checker (Security)
|
||||
Checks ERC-20 approvals granted to known DEX/bridge contracts.
|
||||
```bash
|
||||
python3 $SCRIPT allowance 0xYourWallet
|
||||
```
|
||||
Flags UNLIMITED approvals as HIGH risk.
|
||||
|
||||
### 7. Contract Inspector
|
||||
```bash
|
||||
python3 $SCRIPT contract 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48 # USDC (proxy)
|
||||
python3 $SCRIPT contract 0xdAC17F958D2ee523a2206206994597C13D831ec7 # USDT (ERC-20)
|
||||
```
|
||||
Detects: proxy (EIP-1967/EIP-1167), ERC-20, ERC-721, ERC-165. Shows bytecode size and implementation address for proxies.
|
||||
|
||||
### 8. Whale Detection
|
||||
```bash
|
||||
python3 $SCRIPT whale # ETH, last 20 blocks, >$10k
|
||||
python3 $SCRIPT whale --blocks 50 --min-usd 50000 --chain bsc
|
||||
```
|
||||
|
||||
### 9. Gas Tracker
|
||||
```bash
|
||||
python3 $SCRIPT gas
|
||||
python3 $SCRIPT gas --chain polygon
|
||||
```
|
||||
Shows gwei price + USD cost for: transfer, ERC-20 transfer, approve, swap, NFT mint, NFT transfer.
|
||||
|
||||
---
|
||||
|
||||
## Supported Chains
|
||||
| Key | Name | Native | Chain ID |
|
||||
|-----------|----------------|--------|----------|
|
||||
| ethereum | Ethereum | ETH | 1 |
|
||||
| bsc | BNB Chain | BNB | 56 |
|
||||
| base | Base | ETH | 8453 |
|
||||
| arbitrum | Arbitrum One | ETH | 42161 |
|
||||
| polygon | Polygon | POL | 137 |
|
||||
| optimism | Optimism | ETH | 10 |
|
||||
| avalanche | Avalanche C | AVAX | 43114 |
|
||||
| zksync | zkSync Era | ETH | 324 |
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls
|
||||
- CoinGecko free tier: ~10-30 req/min. Use `--no-prices` for faster wallet scans.
|
||||
- Public RPCs may throttle. Set EVM_RPC_URL to a private endpoint for production.
|
||||
- `wallet` and `allowance` only check known token list (~30 tokens per chain). Use a block explorer for complete token discovery.
|
||||
- `activity` scans recent blocks only (max 200). For full history, use Etherscan API.
|
||||
- `multichain` runs 8 parallel threads — can trigger rate limits on public RPCs.
|
||||
- ENS resolution depends on a single public endpoint (ensideas.com / ens.vitalik.ca) with no fallback. If that endpoint is down, `ens` will fail — re-run later or use a block explorer.
|
||||
- Tx decoding depends on a single public endpoint (4byte.directory) with no fallback. Selectors not in their database show up as `unknown`.
|
||||
- **L2 gas estimates are L2-execution only.** On rollups like Base, Arbitrum, Optimism, and zkSync, the actual transaction cost also includes an L1 data-posting fee that depends on calldata size and current L1 gas prices. The `gas` command does not estimate that L1 component. For Base specifically, see the network's L1 fee oracle (contract `0x420000000000000000000000000000000000000F`).
|
||||
- Address / tx-hash inputs are validated for 0x-prefix + correct length + hex, but EIP-55 checksum casing is **not** enforced (RPC endpoints accept any-case hex).
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
# Should print current block, gas price, ETH price
|
||||
python3 ~/.hermes/skills/blockchain/evm/scripts/evm_client.py stats
|
||||
|
||||
# Should resolve vitalik.eth to 0xd8dA...
|
||||
python3 ~/.hermes/skills/blockchain/evm/scripts/evm_client.py ens vitalik.eth
|
||||
```
|
||||
@@ -1,323 +0,0 @@
|
||||
---
|
||||
title: "Hermes Atropos Environments — Build, test, and debug Hermes Agent RL environments for Atropos training"
|
||||
sidebar_label: "Hermes Atropos Environments"
|
||||
description: "Build, test, and debug Hermes Agent RL environments for Atropos training"
|
||||
---
|
||||
|
||||
{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
|
||||
|
||||
# Hermes Atropos Environments
|
||||
|
||||
Build, test, and debug Hermes Agent RL environments for Atropos training. Covers the HermesAgentBaseEnv interface, reward functions, agent loop integration, evaluation with tools, wandb logging, and the three CLI modes (serve/process/evaluate). Use when creating, reviewing, or fixing RL environments in the hermes-agent repo.
|
||||
|
||||
## Skill metadata
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Source | Optional — install with `hermes skills install official/mlops/hermes-atropos-environments` |
|
||||
| Path | `optional-skills/mlops/hermes-atropos-environments` |
|
||||
| Version | `1.1.0` |
|
||||
| Author | Hermes Agent |
|
||||
| License | MIT |
|
||||
| Platforms | linux, macos, windows |
|
||||
| Tags | `atropos`, `rl`, `environments`, `training`, `reinforcement-learning`, `reward-functions` |
|
||||
| Related skills | [`axolotl`](/docs/user-guide/skills/optional/mlops/mlops-training-axolotl), [`fine-tuning-with-trl`](/docs/user-guide/skills/optional/mlops/mlops-training-trl-fine-tuning), `lm-evaluation-harness` |
|
||||
|
||||
## Reference: full SKILL.md
|
||||
|
||||
:::info
|
||||
The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
|
||||
:::
|
||||
|
||||
# Hermes Agent Atropos Environments
|
||||
|
||||
Guide for building RL environments in the hermes-agent repo that integrate with the Atropos training framework.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
<!-- ascii-guard-ignore -->
|
||||
```
|
||||
Atropos BaseEnv (atroposlib/envs/base.py)
|
||||
└── HermesAgentBaseEnv (environments/hermes_base_env.py)
|
||||
├── Handles agent loop orchestration
|
||||
├── Handles tool resolution per group
|
||||
├── Handles ToolContext for reward verification
|
||||
└── YOUR ENVIRONMENT (environments/your_env.py)
|
||||
Only implements: setup, get_next_item, format_prompt,
|
||||
compute_reward, evaluate, wandb_log
|
||||
```
|
||||
<!-- ascii-guard-ignore-end -->
|
||||
|
||||
Hermes environments are special because they run a **multi-turn agent loop with tool calling** — not just single-turn completions. The base env handles the loop; you implement the task and scoring.
|
||||
|
||||
## File Locations
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `environments/hermes_base_env.py` | Base class with agent loop + tool resolution |
|
||||
| `environments/agent_loop.py` | `HermesAgentLoop` + `AgentResult` dataclass |
|
||||
| `environments/tool_context.py` | `ToolContext` for reward verification |
|
||||
| `environments/tool_call_parsers.py` | Phase 2 tool call parsers (hermes, mistral, etc.) |
|
||||
| `environments/your_env.py` | Your environment implementation |
|
||||
|
||||
## Inference Setup — Ask the User First
|
||||
|
||||
**IMPORTANT:** Before running any test, evaluation, or data generation command, always ask the user how they want to handle inference. Do NOT assume OpenRouter or any specific endpoint. Present these options:
|
||||
|
||||
1. **OpenRouter** — Ask which model they want to use (e.g., `anthropic/claude-sonnet-4.5`, `google/gemini-2.5-pro`, `meta-llama/llama-3.3-70b-instruct`, etc.). Requires `OPENROUTER_API_KEY` in environment.
|
||||
2. **Self-hosted VLLM endpoint** — Ask for their base URL (e.g., `http://localhost:8000/v1`) and model name. Set `--openai.server_type vllm`.
|
||||
3. **Other OpenAI-compatible API** — Ask for the base URL, model name, and any required API key. Set `--openai.server_type openai` and `--openai.health_check false`.
|
||||
4. **Local Atropos training server** — For `serve` mode with a live training loop. Default `http://localhost:8000/v1`.
|
||||
|
||||
Once the user tells you their setup, use those values in all CLI commands for that session. Example prompts:
|
||||
|
||||
> "Before I run this, how would you like to handle inference?
|
||||
> 1. OpenRouter (I'll need your preferred model, e.g. claude-sonnet-4.5)
|
||||
> 2. A self-hosted VLLM endpoint (give me the URL and model name)
|
||||
> 3. Another OpenAI-compatible API (give me the URL, model, and any auth details)
|
||||
> 4. Local Atropos training server (serve mode)"
|
||||
|
||||
### Key flags by provider:
|
||||
|
||||
| Provider | `--openai.server_type` | `--openai.health_check` | `--openai.api_key` |
|
||||
|----------|----------------------|------------------------|-------------------|
|
||||
| OpenRouter | `openai` | `false` | `$OPENROUTER_API_KEY` |
|
||||
| VLLM (self-hosted) | `vllm` | (default) | (not needed) |
|
||||
| Other OpenAI-compatible | `openai` | `false` | As needed |
|
||||
| Local Atropos | (default) | (default) | (not needed) |
|
||||
|
||||
## Required Methods
|
||||
|
||||
### 1. `setup()` — Load dataset and initialize state
|
||||
|
||||
```python
|
||||
async def setup(self) -> None:
|
||||
"""Called once at startup. Load datasets, initialize state."""
|
||||
# Try HuggingFace first, fallback to built-in samples
|
||||
try:
|
||||
from datasets import load_dataset
|
||||
ds = load_dataset("your/dataset", split="test")
|
||||
self._items = [...]
|
||||
except Exception:
|
||||
self._items = BUILTIN_SAMPLES
|
||||
|
||||
# Always split into train/eval
|
||||
random.shuffle(self._items)
|
||||
eval_size = max(20, int(len(self._items) * 0.1))
|
||||
self._eval_items = self._items[:eval_size]
|
||||
self._items = self._items[eval_size:]
|
||||
```
|
||||
|
||||
### 2. `get_next_item()` — Return next training item
|
||||
|
||||
```python
|
||||
async def get_next_item(self) -> dict:
|
||||
"""Return next item, cycling through dataset."""
|
||||
item = self._items[self._index % len(self._items)]
|
||||
self._index += 1
|
||||
return item
|
||||
```
|
||||
|
||||
### 3. `format_prompt(item)` — Convert item to user message
|
||||
|
||||
```python
|
||||
def format_prompt(self, item: dict) -> str:
|
||||
"""Convert a dataset item into the user-facing prompt."""
|
||||
return f"Research this question: {item['question']}"
|
||||
```
|
||||
|
||||
### 4. `compute_reward(item, result, ctx)` — Score the rollout
|
||||
|
||||
**CRITICAL**: `result` is an `AgentResult`, NOT a dict. It has these attributes:
|
||||
- `result.messages` — List of message dicts (OpenAI format)
|
||||
- `result.turns_used` — Number of LLM calls made
|
||||
- `result.finished_naturally` — True if model stopped voluntarily
|
||||
- `result.tool_errors` — List of ToolError objects
|
||||
|
||||
**AgentResult does NOT have**: `final_response`, `tool_calls`, `tools_used`.
|
||||
You must extract these from `result.messages`:
|
||||
|
||||
```python
|
||||
async def compute_reward(self, item, result: AgentResult, ctx: ToolContext) -> float:
|
||||
# Extract final response (last assistant message with content)
|
||||
final_response = ""
|
||||
tools_used = []
|
||||
for msg in reversed(result.messages):
|
||||
if msg.get("role") == "assistant" and msg.get("content") and not final_response:
|
||||
final_response = msg["content"]
|
||||
if msg.get("role") == "assistant" and msg.get("tool_calls"):
|
||||
for tc in msg["tool_calls"]:
|
||||
fn = tc.get("function", {}) if isinstance(tc, dict) else {}
|
||||
name = fn.get("name", "")
|
||||
if name:
|
||||
tools_used.append(name)
|
||||
|
||||
# Score using LLM judge, heuristic, or ToolContext verification
|
||||
correctness = await self._llm_judge(item, final_response)
|
||||
return correctness
|
||||
```
|
||||
|
||||
`ctx` (ToolContext) gives you terminal/file access to the agent's sandbox for verification:
|
||||
```python
|
||||
# Run tests in the agent's sandbox
|
||||
result = ctx.terminal("pytest /workspace/test.py")
|
||||
return 1.0 if result["exit_code"] == 0 else 0.0
|
||||
```
|
||||
|
||||
### 5. `evaluate()` — Periodic evaluation with full agent loop
|
||||
|
||||
**MUST use the full agent loop with tools**, not single-turn chat_completion.
|
||||
The whole point of hermes-agent environments is agentic evaluation:
|
||||
|
||||
```python
|
||||
async def evaluate(self, *args, **kwargs) -> None:
|
||||
import time, uuid
|
||||
from environments.agent_loop import HermesAgentLoop
|
||||
from environments.tool_context import ToolContext
|
||||
|
||||
start_time = time.time()
|
||||
tools, valid_names = self._resolve_tools_for_group()
|
||||
samples = []
|
||||
|
||||
for item in self._eval_items[:self.config.eval_size]:
|
||||
task_id = str(uuid.uuid4())
|
||||
messages = []
|
||||
if self.config.system_prompt:
|
||||
messages.append({"role": "system", "content": self.config.system_prompt})
|
||||
messages.append({"role": "user", "content": self.format_prompt(item)})
|
||||
|
||||
agent = HermesAgentLoop(
|
||||
server=self.server,
|
||||
tool_schemas=tools,
|
||||
valid_tool_names=valid_names,
|
||||
max_turns=self.config.max_agent_turns,
|
||||
task_id=task_id,
|
||||
temperature=0.0, # Deterministic for eval
|
||||
max_tokens=self.config.max_token_length,
|
||||
extra_body=self.config.extra_body,
|
||||
)
|
||||
result = await agent.run(messages)
|
||||
|
||||
ctx = ToolContext(task_id)
|
||||
try:
|
||||
reward = await self.compute_reward(item, result, ctx)
|
||||
finally:
|
||||
ctx.cleanup()
|
||||
|
||||
samples.append({"prompt": ..., "response": ..., "reward": reward})
|
||||
|
||||
eval_metrics = {"eval/mean_reward": ...}
|
||||
await self.evaluate_log(metrics=eval_metrics, samples=samples,
|
||||
start_time=start_time, end_time=time.time())
|
||||
```
|
||||
|
||||
### 6. `wandb_log()` — Custom metrics logging
|
||||
|
||||
Always call `super().wandb_log()` at the end:
|
||||
|
||||
```python
|
||||
async def wandb_log(self, wandb_metrics=None):
|
||||
if wandb_metrics is None:
|
||||
wandb_metrics = {}
|
||||
if self._reward_buffer:
|
||||
n = len(self._reward_buffer)
|
||||
wandb_metrics["train/mean_reward"] = sum(self._reward_buffer) / n
|
||||
self._reward_buffer.clear()
|
||||
await super().wandb_log(wandb_metrics) # MUST call super
|
||||
```
|
||||
|
||||
**Pitfall**: `compute_reward` appends to metric buffers. During eval, this pollutes training metrics. Roll back buffer entries added during eval.
|
||||
|
||||
## Config Class
|
||||
|
||||
Always create a custom config subclass with Pydantic Field descriptors. Key inherited fields you can tune: `enabled_toolsets`, `max_agent_turns`, `agent_temperature`, `system_prompt`, `terminal_backend`, `group_size`, `steps_per_eval`, `total_steps`.
|
||||
|
||||
## config_init() — Default Configuration
|
||||
|
||||
Classmethod returning `(YourEnvConfig, [APIServerConfig(...)])`. Set server_type to "openai" for OpenRouter/external APIs. Load API key from environment variable.
|
||||
|
||||
## Three CLI Modes
|
||||
|
||||
```bash
|
||||
# SERVE — Full training loop (connects to Atropos API server)
|
||||
python environments/my_env.py serve --openai.base_url http://localhost:8000/v1
|
||||
|
||||
# PROCESS — Offline data generation (saves JSONL)
|
||||
python environments/my_env.py process --env.total_steps 10 --env.group_size 1 \
|
||||
--env.use_wandb false --env.data_path_to_save_groups output.jsonl \
|
||||
--openai.base_url "<USER_BASE_URL>" \
|
||||
--openai.model_name "<USER_MODEL>" \
|
||||
--openai.server_type <USER_SERVER_TYPE> --openai.health_check false
|
||||
|
||||
# EVALUATE — Standalone eval (runs setup + evaluate only)
|
||||
python environments/my_env.py evaluate --env.eval_size 20 \
|
||||
--env.data_dir_to_save_evals /tmp/eval_results \
|
||||
--openai.base_url "<USER_BASE_URL>" \
|
||||
--openai.model_name "<USER_MODEL>" \
|
||||
--openai.server_type <USER_SERVER_TYPE> --openai.health_check false
|
||||
```
|
||||
|
||||
Config priority: CLI args > YAML file > config_init() defaults.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **AgentResult has .messages, not .final_response** — Extract the final response by iterating reversed(result.messages) looking for the last assistant message with content.
|
||||
|
||||
2. **evaluate() must use HermesAgentLoop, not chat_completion** — Single-turn chat_completion has no tools. The whole point of hermes-agent benchmarks is agentic evaluation with tool use.
|
||||
|
||||
3. **Don't call _llm_judge twice** — If compute_reward already calls it, extract the score from the buffer instead of calling judge separately in evaluate().
|
||||
|
||||
4. **Eval pollutes training buffers** — compute_reward appends to metric buffers. During eval, roll back buffer entries to keep training metrics clean.
|
||||
|
||||
5. **Always set health_check=false for OpenRouter** — OpenRouter has no /health endpoint.
|
||||
|
||||
6. **Set data_dir_to_save_evals in evaluate mode** — Without it, results aren't saved.
|
||||
|
||||
7. **default_toolsets class variable vs enabled_toolsets config** — The class variable is a hint; the config field is what actually controls tool resolution.
|
||||
|
||||
8. **Tool call parsing in messages** — Tool calls are dicts with `{"function": {"name": ..., "arguments": ...}}`. Always check `isinstance(tc, dict)`.
|
||||
|
||||
9. **ToolContext.cleanup()** — Always call in a finally block to release sandbox resources.
|
||||
|
||||
10. **server_type must be "openai" for external APIs** — Without it, Atropos assumes a local VLLM server.
|
||||
|
||||
11. **Always ask the user for their inference setup** — Never hardcode or assume a specific provider/model. See the "Inference Setup" section above.
|
||||
|
||||
## Reward Function Patterns
|
||||
|
||||
### LLM Judge (for open-ended tasks)
|
||||
Use `self.server.chat_completion()` with a scoring prompt. Parse JSON response for score float. Always include a heuristic fallback (keyword overlap) for when the judge call fails.
|
||||
|
||||
### Binary Verification (for code/terminal tasks)
|
||||
Use `ctx.terminal("pytest test.py -q")` to run tests in the agent's sandbox. Return 1.0 for pass, 0.0 for fail.
|
||||
|
||||
### Multi-Signal (combine multiple indicators)
|
||||
Weight correctness (0.6) + tool usage (0.2) + efficiency (0.2) + optional bonuses. Clamp to [0, 1].
|
||||
|
||||
## Testing Your Environment
|
||||
|
||||
1. **Import test**: `python -c "from environments.my_env import MyEnv; print('OK')"`
|
||||
2. **Ask the user for inference setup** (see "Inference Setup" section above)
|
||||
3. **Process mode** (1 item): Verify JSONL output has valid tokens, masks, scores
|
||||
4. **Evaluate mode**: Verify full agent loop runs with tools, metrics logged correctly
|
||||
5. **Check reward range**: Scores should be in [0, 1], not all identical
|
||||
|
||||
## Minimum Implementation Checklist
|
||||
|
||||
```python
|
||||
class MyEnv(HermesAgentBaseEnv):
|
||||
name = "my-env"
|
||||
env_config_cls = MyEnvConfig
|
||||
|
||||
@classmethod
|
||||
def config_init(cls): ... # Default server + env config
|
||||
async def setup(self): ... # Load dataset + train/eval split
|
||||
async def get_next_item(self): ... # Cycle through training items
|
||||
def format_prompt(self, item): ... # Item → user message string
|
||||
async def compute_reward(self, item, result, ctx): ... # Score rollout
|
||||
async def evaluate(self, *args, **kwargs): ... # Full agent loop eval
|
||||
async def wandb_log(self, metrics=None): ... # Custom metrics + super()
|
||||
|
||||
if __name__ == "__main__":
|
||||
MyEnv.cli()
|
||||
```
|
||||
Reference in New Issue
Block a user