hermes-mobile/docs/COMPANION_SERVER.md

Companion Server Plan

Purpose

The companion server is the local service installed beside Hermes Agent. It turns Hermes into a phone-friendly app backend without forcing Hermes core to become a mobile backend.

Responsibilities:

• Authenticate/pair phones
• Accept prompts, voice recordings, and files
• Start/continue Hermes tasks
• Stream Hermes activity to the app
• Store app metadata
• Send push notifications
• Expose cron/job/session/file APIs
• Provide health checks and install/update status

Runtime

Preferred MVP runtime: Node.js + TypeScript.

Reasons:

• Great fit for HTTP/WebSocket/SSE/file uploads.
• Easy bundling for one-liner installs.
• PM2/systemd deployment is straightforward in Zeb's environment.
• Frontend and backend can share TypeScript event schemas.

Potential framework:

• Fastify: mature, fast, good plugins.
• Hono: simple, edge-like API, also good.

Initial pick: Fastify unless we want ultra-minimal.

Config

Config locations:

System mode:

/etc/hermes-mobile/config.yaml
/var/lib/hermes-mobile/

User mode:

~/.config/hermes-mobile/config.yaml
~/.local/share/hermes-mobile/

Example:

server:
  host: 0.0.0.0
  port: 8787
  public_url: https://hermes-mobile.molberg.cloud

auth:
  pairing_enabled: true
  token_ttl_days: 90
  passkeys_enabled: false

hermes:
  home: /root/.hermes
  mode: python # python | api | cli
  cli_bin: /root/.local/bin/hermes
  api_base_url: http://127.0.0.1:18793/v1
  api_key: ""

storage:
  data_dir: /var/lib/hermes-mobile
  max_upload_mb: 512
  retention_days: 30

notifications:
  provider: webpush # webpush | ntfy | gotify | none
  ntfy_url: https://ntfy.example.com/hermes

Hermes Adapter Strategy

Phase 1: CLI/API hybrid

• Use Hermes CLI for health tests and simple prompts.
• Use Hermes API server if available for OpenAI-compatible chat.
• Parse coarse progress where possible.

Pros:

• Requires few/no Hermes changes.
• Easy to prove app UX.

Cons:

• Tool-call visibility limited.

Phase 2: Direct Python adapter

Run a Python worker process that imports Hermes AIAgent and communicates with Node over stdio or local socket.

Benefits:

• Can pass stream_delta_callback and tool_progress_callback.
• More structured event stream.
• Better session control.

Potential shape:

Node companion <-- JSON lines --> Python hermes_worker.py <-- imports --> AIAgent

Worker commands:

• health
• start_task
• continue_session
• cancel_task
• list_sessions
• cron_action

Worker emits events:

• assistant.delta
• tool_call.started
• tool_call.output
• tool_call.finished
• assistant.final

Phase 3: Hermes-native plugin/API

If Hermes Agent grows a stable event API, switch adapter implementation without rewriting app/server.

Task Lifecycle

created -> queued -> running -> finished
                   -> failed
                   -> cancelled
                   -> waiting_approval

Task record:

type Task = {
  id: string
  sessionId?: string
  title?: string
  status: TaskStatus
  inputText: string
  attachmentIds: string[]
  createdAt: string
  startedAt?: string
  finishedAt?: string
  error?: string
}

Upload Handling

Use multipart upload endpoint.

Pipeline:

1. Validate file count/size/type.
2. Store under uploads/<task-id-or-upload-id>/original-name.
3. Record metadata in SQLite.
4. Include absolute file paths in Hermes prompt.

Prompt augmentation example:

The user uploaded these files:
- screenshot.png: /var/lib/hermes-mobile/uploads/abc/screenshot.png (image/png, 320 KB)
- source.zip: /var/lib/hermes-mobile/uploads/abc/source.zip (application/zip, 12 MB)

User prompt:
<actual user prompt>

Voice Handling

MVP options:

1. Client records .webm/.ogg, server stores it, then sends file path to Hermes with instruction to transcribe/analyze.
2. Server transcribes with local faster-whisper and sends text prompt.

Recommended MVP:

• Server-side transcription using local faster-whisper if available.
• Fallback: pass audio file to Hermes as attachment.

Notifications

Events that trigger notifications:

• task completed after app backgrounded
• task failed
• approval required
• cron job finished/failed

Use provider abstraction:

interface NotificationProvider {
  sendCompletion(task: Task): Promise<void>
  sendFailure(task: Task): Promise<void>
  sendApproval(approval: Approval): Promise<void>
  sendCron(job: CronEvent): Promise<void>
}

Pairing/Auth

MVP pairing:

1. Installer/server logs one-time code.
2. Phone opens /pair and enters code or uses link.
3. Server stores device and creates token.
4. Token stored in secure-ish PWA storage. Later use passkeys.

For public exposure, add:

• Rate limiting
• Token rotation
• Session revoke UI
• Optional Tailscale-only mode

Cron Integration

Initial implementation can shell out to Hermes CLI:

hermes cron list --json
hermes cron run <id>
hermes cron pause <id>
hermes cron resume <id>

If CLI JSON is not available, import cron modules from Hermes Python or parse job files carefully.

System Health

Health endpoint should check:

• companion server up
• DB writable
• upload dir writable
• Hermes CLI exists
• Hermes config exists
• Hermes API server reachable if configured
• Python worker reachable if configured
• notification provider configured

Logging

• Structured JSON logs for service.
• Human-readable recent logs in UI.
• Do not log secrets, bearer tokens, or full uploaded file content.

Open Questions

• Should companion run as same user as Hermes Agent for direct module/file access?
• Should public access be Cloudflare Access/Tailscale-first instead of app auth-first?
• How much Hermes core should be extended for better event streaming?