hermes-mobile/docs/TESTING.md

Testing Strategy

Philosophy

Hermes Mobile must be tested as an app plus a local service plus a Hermes integration. Generic unit tests are not enough; the important thing is whether a phone can send work to Hermes, see activity, close the app, and get notified when done.

Test Layers

1. Unit Tests

Frontend:

• event reducer/idempotency
• upload tray state
• voice recorder state machine
• API client error handling
• cron job card states

Companion:

• config loading
• auth pairing tokens
• notification provider interface
• upload path normalization
• event bus fanout
• task state transitions

2. Integration Tests

Companion server with fake Hermes adapter:

• create task
• stream events
• upload file + attach to task
• completion notification queued
• cron list/run mocks

Hermes adapter contract tests:

• CLI adapter health
• API adapter health
• Python worker JSONL protocol

3. End-to-End Tests

Use Playwright against local dev server.

Scenarios:

• first-time pairing
• send text prompt
• upload PNG and send prompt
• record/upload fake audio blob
• watch tool timeline update
• cron run button
• notification permission flow mocked

4. Real Hermes Smoke Tests

On a machine with Hermes installed:

hermes chat -q "Reply with exactly: hermes-mobile-ok"

Companion health test should do the equivalent and verify response.

More advanced smoke:

• Ask Hermes to run a harmless terminal command: pwd.
• Verify a tool-call event appears.
• Upload a small text file and ask Hermes to summarize it.

Manual Android QA

Test on Android Chrome installed PWA:

• Install app to home screen.
• Launch standalone mode.
• Keyboard does not break composer layout.
• Back button closes sheets/modals first.
• Voice recording works.
• File picker works for image/zip/pdf.
• Push notification arrives after app is backgrounded.
• App reconnects to running task after being killed/reopened.

Compatibility Targets

• Android Chrome latest
• Android Firefox optional
• Desktop Chrome for dev
• iOS Safari later, not MVP priority

Observability for Testing

Companion should expose:

GET /api/health
GET /api/debug/events/recent
GET /api/debug/config/redacted
POST /api/hermes/test
POST /api/notifications/test

Only expose debug endpoints in dev mode or authenticated admin mode.

CI Plan

Later:

• pnpm lint
• pnpm typecheck
• pnpm test
• pnpm e2e with fake adapter
• build release artifact

Dogfooding Plan

The app should be used to prompt Hermes about its own repo.

Example dogfood tasks:

• “Run the test suite for Hermes Mobile and summarize failures.”
• “Inspect the companion logs and fix the upload bug.”
• “Create a cron job that reminds me if companion is down.”

This is important because Hermes is both the target agent and the tool used to build the app.