Hermes Agent
120 lines
2.8 KiB
Markdown
120 lines
2.8 KiB
Markdown
# Testing Strategy
|
|
|
|
## Philosophy
|
|
|
|
Hermes Mobile must be tested as an app plus a local service plus a Hermes integration. Generic unit tests are not enough; the important thing is whether a phone can send work to Hermes, see activity, close the app, and get notified when done.
|
|
|
|
## Test Layers
|
|
|
|
### 1. Unit Tests
|
|
|
|
Frontend:
|
|
- event reducer/idempotency
|
|
- upload tray state
|
|
- voice recorder state machine
|
|
- API client error handling
|
|
- cron job card states
|
|
|
|
Companion:
|
|
- config loading
|
|
- auth pairing tokens
|
|
- notification provider interface
|
|
- upload path normalization
|
|
- event bus fanout
|
|
- task state transitions
|
|
|
|
### 2. Integration Tests
|
|
|
|
Companion server with fake Hermes adapter:
|
|
- create task
|
|
- stream events
|
|
- upload file + attach to task
|
|
- completion notification queued
|
|
- cron list/run mocks
|
|
|
|
Hermes adapter contract tests:
|
|
- CLI adapter health
|
|
- API adapter health
|
|
- Python worker JSONL protocol
|
|
|
|
### 3. End-to-End Tests
|
|
|
|
Use Playwright against local dev server.
|
|
|
|
Scenarios:
|
|
- first-time pairing
|
|
- send text prompt
|
|
- upload PNG and send prompt
|
|
- record/upload fake audio blob
|
|
- watch tool timeline update
|
|
- cron run button
|
|
- notification permission flow mocked
|
|
|
|
### 4. Real Hermes Smoke Tests
|
|
|
|
On a machine with Hermes installed:
|
|
|
|
```bash
|
|
hermes chat -q "Reply with exactly: hermes-mobile-ok"
|
|
```
|
|
|
|
Companion health test should do the equivalent and verify response.
|
|
|
|
More advanced smoke:
|
|
- Ask Hermes to run a harmless terminal command: `pwd`.
|
|
- Verify a tool-call event appears.
|
|
- Upload a small text file and ask Hermes to summarize it.
|
|
|
|
## Manual Android QA
|
|
|
|
Test on Android Chrome installed PWA:
|
|
- Install app to home screen.
|
|
- Launch standalone mode.
|
|
- Keyboard does not break composer layout.
|
|
- Back button closes sheets/modals first.
|
|
- Voice recording works.
|
|
- File picker works for image/zip/pdf.
|
|
- Push notification arrives after app is backgrounded.
|
|
- App reconnects to running task after being killed/reopened.
|
|
|
|
## Compatibility Targets
|
|
|
|
- Android Chrome latest
|
|
- Android Firefox optional
|
|
- Desktop Chrome for dev
|
|
- iOS Safari later, not MVP priority
|
|
|
|
## Observability for Testing
|
|
|
|
Companion should expose:
|
|
|
|
```http
|
|
GET /api/health
|
|
GET /api/debug/events/recent
|
|
GET /api/debug/config/redacted
|
|
POST /api/hermes/test
|
|
POST /api/notifications/test
|
|
```
|
|
|
|
Only expose debug endpoints in dev mode or authenticated admin mode.
|
|
|
|
## CI Plan
|
|
|
|
Later:
|
|
- pnpm lint
|
|
- pnpm typecheck
|
|
- pnpm test
|
|
- pnpm e2e with fake adapter
|
|
- build release artifact
|
|
|
|
## Dogfooding Plan
|
|
|
|
The app should be used to prompt Hermes about its own repo.
|
|
|
|
Example dogfood tasks:
|
|
- “Run the test suite for Hermes Mobile and summarize failures.”
|
|
- “Inspect the companion logs and fix the upload bug.”
|
|
- “Create a cron job that reminds me if companion is down.”
|
|
|
|
This is important because Hermes is both the target agent and the tool used to build the app.
|