hermes-mobile/docs/COMPANION_SERVER.md

# Companion Server Plan

## Purpose

The companion server is the local service installed beside Hermes Agent. It turns Hermes into a phone-friendly app backend without forcing Hermes core to become a mobile backend.

Responsibilities:
- Authenticate/pair phones
- Accept prompts, voice recordings, and files
- Start/continue Hermes tasks
- Stream Hermes activity to the app
- Store app metadata
- Send push notifications
- Expose cron/job/session/file APIs
- Provide health checks and install/update status

## Runtime

Preferred MVP runtime: Node.js + TypeScript.

Reasons:
- Great fit for HTTP/WebSocket/SSE/file uploads.
- Easy bundling for one-liner installs.
- PM2/systemd deployment is straightforward in Zeb's environment.
- Frontend and backend can share TypeScript event schemas.

Potential framework:
- Fastify: mature, fast, good plugins.
- Hono: simple, edge-like API, also good.

Initial pick: Fastify unless we want ultra-minimal.

## Config

Config locations:

System mode:
```text
/etc/hermes-mobile/config.yaml
/var/lib/hermes-mobile/
```

User mode:
```text
~/.config/hermes-mobile/config.yaml
~/.local/share/hermes-mobile/
```

Example:

```yaml
server:
  host: 0.0.0.0
  port: 8787
  public_url: https://hermes-mobile.molberg.cloud

auth:
  pairing_enabled: true
  token_ttl_days: 90
  passkeys_enabled: false

hermes:
  home: /root/.hermes
  mode: python # python | api | cli
  cli_bin: /root/.local/bin/hermes
  api_base_url: http://127.0.0.1:18793/v1
  api_key: ""

storage:
  data_dir: /var/lib/hermes-mobile
  max_upload_mb: 512
  retention_days: 30

notifications:
  provider: webpush # webpush | ntfy | gotify | none
  ntfy_url: https://ntfy.example.com/hermes
```

## Hermes Adapter Strategy

### Phase 1: CLI/API hybrid

- Use Hermes CLI for health tests and simple prompts.
- Use Hermes API server if available for OpenAI-compatible chat.
- Parse coarse progress where possible.

Pros:
- Requires few/no Hermes changes.
- Easy to prove app UX.

Cons:
- Tool-call visibility limited.

### Phase 2: Direct Python adapter

Run a Python worker process that imports Hermes `AIAgent` and communicates with Node over stdio or local socket.

Benefits:
- Can pass `stream_delta_callback` and `tool_progress_callback`.
- More structured event stream.
- Better session control.

Potential shape:

```text
Node companion <-- JSON lines --> Python hermes_worker.py <-- imports --> AIAgent
```

Worker commands:
- `health`
- `start_task`
- `continue_session`
- `cancel_task`
- `list_sessions`
- `cron_action`

Worker emits events:
- `assistant.delta`
- `tool_call.started`
- `tool_call.output`
- `tool_call.finished`
- `assistant.final`

### Phase 3: Hermes-native plugin/API

If Hermes Agent grows a stable event API, switch adapter implementation without rewriting app/server.

## Task Lifecycle

```text
created -> queued -> running -> finished
                   -> failed
                   -> cancelled
                   -> waiting_approval
```

Task record:

```ts
type Task = {
  id: string
  sessionId?: string
  title?: string
  status: TaskStatus
  inputText: string
  attachmentIds: string[]
  createdAt: string
  startedAt?: string
  finishedAt?: string
  error?: string
}
```

## Upload Handling

Use multipart upload endpoint.

Pipeline:
1. Validate file count/size/type.
2. Store under `uploads/<task-id-or-upload-id>/original-name`.
3. Record metadata in SQLite.
4. Include absolute file paths in Hermes prompt.

Prompt augmentation example:

```text
The user uploaded these files:
- screenshot.png: /var/lib/hermes-mobile/uploads/abc/screenshot.png (image/png, 320 KB)
- source.zip: /var/lib/hermes-mobile/uploads/abc/source.zip (application/zip, 12 MB)

User prompt:
<actual user prompt>
```

## Voice Handling

MVP options:

1. Client records `.webm`/`.ogg`, server stores it, then sends file path to Hermes with instruction to transcribe/analyze.
2. Server transcribes with local faster-whisper and sends text prompt.

Recommended MVP:
- Server-side transcription using local faster-whisper if available.
- Fallback: pass audio file to Hermes as attachment.

## Notifications

Events that trigger notifications:
- task completed after app backgrounded
- task failed
- approval required
- cron job finished/failed

Use provider abstraction:

```ts
interface NotificationProvider {
  sendCompletion(task: Task): Promise<void>
  sendFailure(task: Task): Promise<void>
  sendApproval(approval: Approval): Promise<void>
  sendCron(job: CronEvent): Promise<void>
}
```

## Pairing/Auth

MVP pairing:
1. Installer/server logs one-time code.
2. Phone opens `/pair` and enters code or uses link.
3. Server stores device and creates token.
4. Token stored in secure-ish PWA storage. Later use passkeys.

For public exposure, add:
- Rate limiting
- Token rotation
- Session revoke UI
- Optional Tailscale-only mode

## Cron Integration

Initial implementation can shell out to Hermes CLI:

```bash
hermes cron list --json
hermes cron run <id>
hermes cron pause <id>
hermes cron resume <id>
```

If CLI JSON is not available, import cron modules from Hermes Python or parse job files carefully.

## System Health

Health endpoint should check:
- companion server up
- DB writable
- upload dir writable
- Hermes CLI exists
- Hermes config exists
- Hermes API server reachable if configured
- Python worker reachable if configured
- notification provider configured

## Logging

- Structured JSON logs for service.
- Human-readable recent logs in UI.
- Do not log secrets, bearer tokens, or full uploaded file content.

## Open Questions

- Should companion run as same user as Hermes Agent for direct module/file access?
- Should public access be Cloudflare Access/Tailscale-first instead of app auth-first?
- How much Hermes core should be extended for better event streaming?