hermes-mobile/docs/PRODUCT_PLAN.md

Product Plan

Working Name

Hermes Mobile

One-line Pitch

A private Android-first app for talking to Hermes Agent with voice, files, live tool calls, approvals, cron controls, and completion notifications.

Target User

Primary: Zeb / homelab power user who already runs Hermes Agent on a Linux machine and wants a better phone interface than Telegram.

Secondary later:

• Developers running Hermes locally/remotely
• Homelab admins
• People who want an agent command center rather than a chatbot

Non-goals

• Not a generic OpenAI chat frontend.
• Not a multi-provider model playground.
• Not a public SaaS.
• Not a replacement for Hermes Agent internals.
• Not a Matrix/Discord/Telegram clone.

Design Principles

1. Agent-first, not chat-first

   • The UI should show what Hermes is doing, not just the final response.
   • Tool calls are first-class objects.

2. Phone-native ergonomics

   • Native Android app first, PWA fallback second.
   • Thumb-reachable controls.
   • Large tap targets.
   • Voice and upload actions always nearby.
   • Background task status survives app close/reopen.
   • Haptics, native push, status bar/splash, and Android back behavior should be part of the product, not afterthoughts.

3. Private by default

   • Self-hosted companion server.
   • Local uploads.
   • Optional LAN/Tailscale-only mode.
   • No third-party notification requirement unless chosen.

4. Modular integrations

   • Hermes CLI bridge now.
   • Hermes API bridge where useful.
   • Direct Python/event bridge later.
   • Notification providers swappable.

5. Install should feel boring

   • One command.
   • Clear service status.
   • Easy update/uninstall.

Core Screens

1. Home / Ask

Purpose: send a prompt fast.

Features:

• Text box with multiline support
• Hold-to-record voice note
• Attach button
• Model/session selector compact chip
• “New task” vs “continue session” toggle
• Streaming answer
• Activity timeline under/alongside answer
• Completion toast/push

2. Activity

Purpose: see active and recent agent runs.

Features:

• Running tasks
• Tool-call cards
• Logs/output previews
• Generated files
• Error states
• Stop/cancel if supported

3. Sessions

Purpose: browse/resume history.

Features:

• Recent conversations
• Search
• Rename/pin/archive
• Session metadata: platform, started, last active, tool count, attachments

4. Cron

Purpose: manage scheduled Hermes tasks from phone.

Features:

• Job list grouped by active/paused/completed
• Run now
• Pause/resume/remove
• Edit prompt/schedule later
• Last output viewer
• Delivery target setting: in-app inbox, push, Matrix/Telegram/etc.

5. Files

Purpose: manage uploaded/generated artifacts.

Features:

• Upload inbox
• Generated media/files from responses
• Download/share
• Link file into a new prompt
• Retention controls later

6. Approvals

Purpose: approve dangerous actions safely.

Features:

• Pending approvals queue
• Command/action preview
• Risk label
• Approve once / deny
• Expiry countdown
• Audit log

7. Settings

Purpose: connect to companion server and configure app.

Features:

• Server URL
• Pairing code / passkey login
• Notification test
• Theme
• Hermes health
• Storage/retention
• STT/TTS preferences

Feature Details

Text Prompting

Modes:

• Quick ask: starts new task/session.
• Continue: appends to selected session.
• Background task: app can be closed, push when finished.

Voice Notes

MVP flow:

1. Browser MediaRecorder captures audio.
2. Upload audio file to companion.
3. Companion either:
   • passes audio path to Hermes gateway/STT path, or
   • transcribes with configured local Whisper/faster-whisper, then sends text prompt.
4. UI shows transcript for confirmation if desired.

Later:

• Voice-to-voice replies using Hermes TTS.
• Streaming partial transcript.

File Uploads

Supported MVP:

• png/jpg/webp/gif
• zip/tar/gz
• pdf
• txt/log/md/json/csv
• arbitrary binary as attachment

Prompt format to Hermes:

• Store file locally in companion upload store.
• Send prompt with absolute paths and metadata.
• Example: User uploaded files: /var/lib/hermes-mobile/uploads/.../archive.zip

Tool Calls

Event model:

• tool_call.started
• tool_call.delta
• tool_call.finished
• tool_call.failed

Tool cards show:

• Icon/name
• Arguments summary
• Status
• Duration
• Expandable result
• Copy/open file actions where relevant

Notifications

MVP notification events:

• task finished
• task failed
• approval required
• cron job completed/failed

Providers:

• Web Push for installed PWA
• ntfy fallback
• Gotify fallback
• Home Assistant persistent notification optional

Cron Triggers

The app should not reimplement Hermes cron. It should wrap Hermes cronjob functions/CLI:

• list jobs
• run job
• pause/resume/remove
• create/edit later

“Prompting itself” / Hermes test harness

Because Hermes Mobile runs next to Hermes Agent, the companion can test connectivity by sending a harmless prompt through:

• Hermes CLI: hermes chat -q "ping"
• API server: /v1/chat/completions
• Direct Python bridge later

The Settings health page should include a Test Hermes button.

MVP Acceptance Criteria

• Install companion server on same Linux machine as Hermes.
• Open mobile PWA on Android, pair to companion.
• Send text prompt to Hermes and receive answer.
• Upload a PNG or ZIP and include it in prompt context.
• Record a voice note and turn it into a prompt.
• See at least coarse tool activity while Hermes runs.
• Receive push/ntfy notification when task completes.
• View cron jobs and trigger one manually.

Risks

1. Hermes API server may not expose enough event detail.

   • Mitigation: start with CLI/direct Python bridge and add event callbacks.

2. PWA push notification quirks.

   • Mitigation: support ntfy/Gotify fallback.

3. Mobile audio and lifecycle quirks.

   • Mitigation: use Capacitor from the start; begin with WebView MediaRecorder and move to a native audio plugin if needed.

4. Tool-call streaming requires Hermes integration changes.

   • Mitigation: companion adapter interface; use available callbacks in AIAgent where possible.