seb/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(tools): downscale computer_use captures before aux-vision routing

Browse Source 
Full-resolution desktop captures (1920x1032+) tokenize to thousands
of vision tokens and overflow small local vision models' context
windows - the aux call came back "the vision API rejected the image"
and the model got no description at all. Cap the long side at 1456px
before writing the temp image for vision_analyze: SOM badges stay
legible, the request fits comfortably, and per-capture vision latency
drops roughly in half. Also drop the hardcoded "macOS" from the
describe prompt now that captures come from Windows hosts too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jeff 2026-06-10 21:40:24 -04:00 committed by Teknium
parent 42fc7d86d3
commit 52cf24e441
No known key found for this signature in database
1 changed files with 20 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
tools/computer_use/tool.py
View File

@ -710,10 +710,29 @@ def _route_capture_through_aux_vision(
cache_dir = get_hermes_dir("cache/vision", "temp_vision_images")
cache_dir.mkdir(parents=True, exist_ok=True)
temp_image_path = cache_dir / f"computer_use_{_uuid.uuid4().hex}{ext}"
# Downscale before handing to the aux vision model. Full-resolution
# desktop captures (e.g. 1920x1032) tokenize to thousands of vision
# tokens and overflow small local-model context windows ("the vision
# API rejected the image"); ~1456px keeps SOM badges legible while
# fitting comfortably and cutting latency.
_MAX_VISION_DIM = 1456
try:
from io import BytesIO as _BytesIO
from PIL import Image as _Image
img = _Image.open(_BytesIO(raw))
if max(img.size) > _MAX_VISION_DIM:
img.thumbnail((_MAX_VISION_DIM, _MAX_VISION_DIM))
out = _BytesIO()
img.save(out, format="JPEG" if ext == ".jpg" else "PNG")
raw = out.getvalue()
except Exception as exc:
logger.debug("computer_use: vision downscale skipped: %s", exc)
temp_image_path.write_bytes(raw)
prompt = (
"Describe what is visible in this macOS application screenshot in "
"Describe what is visible in this application screenshot in "
"concise but specific terms. Mention the app name and window "
"title if visible, the overall layout, any labelled buttons, "
"menus or text fields, and any prominent text content the user "