perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions optimize'
The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of incremental b-tree segments — one per trigger-driven insert batch. SQLite's automerge caps at ~16 segments, so a long-lived store keeps scanning many segments per MATCH and never collapses them unless the special 'optimize' command runs. Nothing in the codebase ever ran it: vacuum() only fired after a prune that deleted rows, and even then never merged FTS segments. Changes: - SessionDB.optimize_fts(): merges each FTS5 index to a single segment, probing for the (optional/lazy) trigram table first so it is safe to call unconditionally. Layout-only — search results and snippet() are unchanged. - vacuum() now calls optimize_fts() before VACUUM so freed index pages are returned to the OS in the same pass. - 'hermes sessions optimize' CLI subcommand for on-demand reclamation + segment compaction (previously there was no way to compact the store without a prune deleting rows), with before/after size reporting. Benchmark (8000 msgs, fragmented to 8 segments/index): - segments 8 -> 1 on both indexes - porter MATCH 5.5x faster (0.449 -> 0.081 ms/q) - trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q) - 8000 matches before == 8000 after, identical row ids (no functional change) Orthogonal to the structural FTS-size PRs (#20239 external-content, #27770 optional trigram) — segment merge helps regardless of those. Tests: TestOptimizeFts covers index count, search+snippet preservation, missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
This commit is contained in:
@@ -13390,6 +13390,11 @@ Examples:
|
||||
"--yes", "-y", action="store_true", help="Skip confirmation"
|
||||
)
|
||||
|
||||
sessions_subparsers.add_parser(
|
||||
"optimize",
|
||||
help="Reclaim disk space: merge FTS5 segments + VACUUM (no data change)",
|
||||
)
|
||||
|
||||
sessions_subparsers.add_parser("stats", help="Show session store statistics")
|
||||
|
||||
sessions_rename = sessions_subparsers.add_parser(
|
||||
@@ -13562,6 +13567,39 @@ Examples:
|
||||
relaunch(["--resume", selected_id])
|
||||
return # won't reach here after execvp
|
||||
|
||||
elif action == "optimize":
|
||||
db_path = db.db_path
|
||||
before_mb = (
|
||||
os.path.getsize(db_path) / (1024 * 1024)
|
||||
if db_path.exists()
|
||||
else 0.0
|
||||
)
|
||||
print("Optimizing session store (FTS merge + VACUUM)…")
|
||||
try:
|
||||
# vacuum() merges FTS5 segments (optimize_fts) then VACUUMs.
|
||||
# Probe the index count first for the summary line.
|
||||
n = sum(
|
||||
1
|
||||
for t in db._FTS_TABLES
|
||||
if db._fts_table_exists(t)
|
||||
)
|
||||
db.vacuum()
|
||||
except Exception as e:
|
||||
print(f"Error: optimization failed: {e}")
|
||||
db.close()
|
||||
return
|
||||
after_mb = (
|
||||
os.path.getsize(db_path) / (1024 * 1024)
|
||||
if db_path.exists()
|
||||
else 0.0
|
||||
)
|
||||
saved = before_mb - after_mb
|
||||
print(f"Optimized {n} FTS index(es).")
|
||||
print(
|
||||
f"Database size: {before_mb:.1f} MB -> {after_mb:.1f} MB "
|
||||
f"(reclaimed {saved:.1f} MB)"
|
||||
)
|
||||
|
||||
elif action == "stats":
|
||||
total = db.session_count()
|
||||
msgs = db.message_count()
|
||||
|
||||
Reference in New Issue
Block a user