seb/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(state.db): recover from malformed sqlite_master so hidden sessions reappear (#43149)

Browse Source 
* fix(state.db): recover from malformed sqlite_master so hidden sessions reappear

The corruption class behind "Desktop/Dashboard show no sessions while
hundreds of session files sit on disk" is a malformed sqlite_master — most
often a duplicate object row, e.g. two CREATE VIRTUAL TABLE messages_fts
entries — surfacing as:

    sqlite3.DatabaseError: malformed database schema (messages_fts) -
    table messages_fts already exists

SQLite parses the whole schema while preparing the FIRST statement on a
connection, so on this class every statement fails before it runs: PRAGMA
journal_mode (which is where SessionDB.__init__ actually trips, in
apply_wal_with_fallback, BEFORE _init_schema), PRAGMA integrity_check, and
even DROP TABLE. The only operations that still work are
PRAGMA writable_schema=ON plus direct sqlite_master surgery. A plain
FTS-index rebuild at the _init_schema layer therefore cannot reach or fix
this; the canonical sessions/messages rows are intact — only the derived
schema is broken.

Add a dedicated recovery that operates where the failure actually happens:

- hermes_state.repair_state_db_schema(): backs up the raw file first, then a
  least-destructive ladder — (1) de-duplicate sqlite_master keeping the
  lowest rowid per object (preserves the existing FTS index), escalating to
  (2) drop every messages_fts* schema object + VACUUM and let the next open
  rebuild the FTS index from messages. sessions/messages are never modified.
  Plus is_malformed_db_error() to discriminate this class.
- SessionDB.__init__ auto-heals: on a malformed-schema open error it repairs
  once (process-guarded against loops / concurrent web_server opens) and
  reopens, so Desktop/Dashboard recover on their own instead of silently
  showing "no sessions".
- hermes doctor --fix detects the malformed class and repairs it (reporting
  the recovered session count + backup name).
- hermes sessions repair [--check-only] [--no-backup] runs on the raw file
  path, since SessionDB() itself cannot open a malformed DB.

Supersedes #32589 and #33869: both targeted FTS corruption but gated their
repair behind statements (integrity_check / SELECT / DROP TABLE) that
themselves fail on this class, and neither addressed the apply_wal_with_fallback
open-time failure. Credit preserved via Co-authored-by.

Closes #33865.

Co-authored-by: João Vitor Cunha <145560011+plcunha@users.noreply.github.com>
Co-authored-by: Tuna Dev <273476039+tuancookiez-hub@users.noreply.github.com>

* test(state.db): cover strat-B escalation + unrepairable safe-fail paths

---------

Co-authored-by: João Vitor Cunha <145560011+plcunha@users.noreply.github.com>
Co-authored-by: Tuna Dev <273476039+tuancookiez-hub@users.noreply.github.com>
This commit is contained in:
brooklyn!
2026-06-09 18:49:08 -05:00
committed by GitHub
co-authored by João Vitor Cunha Tuna Dev
parent 72154ad879
commit 218452b050
4 changed files with 609 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hermes_cli/doctor.py
+47 -1
View File
@@ -1151,7 +1151,53 @@ def run_doctor(args):
conn.close()
check_ok(f"{_DHH}/state.db exists ({count} sessions)")
except Exception as e:
check_warn(f"{_DHH}/state.db exists but has issues: {e}")
from hermes_state import is_malformed_db_error, repair_state_db_schema
if is_malformed_db_error(e):
# sqlite_master itself is malformed (e.g. duplicate
# messages_fts) — every statement fails before it runs, so
# this is NOT a plain FTS-index rebuild. Repair sqlite_master
# in place (backup first; sessions/messages preserved).
check_warn(
f"{_DHH}/state.db schema is malformed (sessions hidden until repaired)",
f"({e})",
)
if should_fix:
report = repair_state_db_schema(state_db_path)
if report.get("repaired"):
try:
conn = sqlite3.connect(str(state_db_path))
count = conn.execute(
"SELECT COUNT(*) FROM sessions"
).fetchone()[0]
conn.close()
except Exception:
count = "?"
backup_name = (
Path(report["backup_path"]).name
if report.get("backup_path") else "n/a"
)
check_ok(
f"Repaired state.db schema ({count} sessions recovered)",
f"(strategy: {report.get('strategy')}; backup: {backup_name})",
)
fixed_count += 1
else:
check_warn(
"state.db schema repair did not recover automatically",
f"({report.get('error')}; backup: {report.get('backup_path')})",
)
issues.append(
"state.db schema malformed and auto-repair failed — "
"restore from the backup copy beside state.db"
)
else:
issues.append(
"state.db schema malformed — run 'hermes doctor --fix' "
"(or 'hermes sessions repair') to recover hidden sessions"
)
else:
check_warn(f"{_DHH}/state.db exists but has issues: {e}")
else:
check_info(f"{_DHH}/state.db not created yet (will be created on first session)")
hermes_cli/main.py
+68 -2
View File
@@ -11185,6 +11185,27 @@ def main():
help="Reclaim disk space: merge FTS5 segments + VACUUM (no data change)",
)
sessions_repair = sessions_subparsers.add_parser(
"repair",
help="Repair a malformed state.db schema so hidden sessions reappear",
description=(
"Recover a state.db whose schema is malformed (e.g. 'table "
"messages_fts already exists'), which makes Desktop/Dashboard show "
"no sessions. A backup is made first; sessions and messages are "
"preserved and the FTS search index is rebuilt if needed."
),
)
sessions_repair.add_argument(
"--check-only",
action="store_true",
help="Only report whether the database opens cleanly; do not modify it",
)
sessions_repair.add_argument(
"--no-backup",
action="store_true",
help="Skip the timestamped backup copy (not recommended)",
)
sessions_subparsers.add_parser("stats", help="Show session store statistics")
sessions_rename = sessions_subparsers.add_parser(
@@ -11214,6 +11235,53 @@ def main():
def cmd_sessions(args):
import json as _json
action = args.sessions_action
# 'repair' must run BEFORE opening SessionDB(): a malformed schema is
# exactly the case where SessionDB() can't open, so it operates on the
# raw file path instead.
if action == "repair":
from hermes_state import (
DEFAULT_DB_PATH,
_db_opens_cleanly,
repair_state_db_schema,
)
db_path = DEFAULT_DB_PATH
if not db_path.exists():
print(f"No session database at {db_path} (nothing to repair).")
return
reason = _db_opens_cleanly(db_path)
if reason is None:
print(f"✓ {db_path} opens cleanly — no repair needed.")
return
print(f"✗ {db_path} does not open cleanly: {reason}")
if getattr(args, "check_only", False):
return
print("Repairing (a backup copy is made first)…")
report = repair_state_db_schema(
db_path, backup=not getattr(args, "no_backup", False)
)
if report.get("repaired"):
if report.get("backup_path"):
print(f" backup: {report['backup_path']}")
print(f" strategy: {report.get('strategy')}")
try:
from hermes_state import SessionDB
n = SessionDB()._conn.execute(
"SELECT COUNT(*) FROM sessions"
).fetchone()[0]
print(f"✓ Repaired — {n} sessions recovered.")
except Exception:
print("✓ Repaired.")
else:
print(f"✗ Repair failed: {report.get('error')}")
if report.get("backup_path"):
print(f" A backup is preserved at: {report['backup_path']}")
print(" Keep state.db and the backup; do not delete them.")
return
try:
from hermes_state import SessionDB
@@ -11222,8 +11290,6 @@ def main():
print(f"Error: Could not open session database: {e}")
return
action = args.sessions_action
# Hide third-party tool sessions by default, but honour explicit --source
_source = getattr(args, "source", None)
_exclude = None if _source else ["tool"]