bonzi/hermes-mobile
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add companion API and Android APK shell

Browse Source
This commit is contained in:
Hermes Agent
2026-07-09 04:43:00 +00:00
parent bd0cf34aa3
commit 7eb9648eb7
92 changed files with 7907 additions and 877 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/COMPANION_SERVER.md
+46 -241
View File
@@ -1,254 +1,59 @@
# Companion Server Plan
# Companion Server
## Purpose
The companion server is a local Fastify HTTP API that the Android app talks to. It runs beside Hermes Agent and keeps Hermes core untouched.
The companion server is the local service installed beside Hermes Agent. It turns Hermes into a phone-friendly app backend without forcing Hermes core to become a mobile backend.
## Setup
Responsibilities:
- Authenticate/pair phones
- Accept prompts, voice recordings, and files
- Start/continue Hermes tasks
- Stream Hermes activity to the app
- Store app metadata
- Send push notifications
- Expose cron/job/session/file APIs
- Provide health checks and install/update status
## Runtime
Preferred MVP runtime: Node.js + TypeScript.
Reasons:
- Great fit for HTTP/WebSocket/SSE/file uploads.
- Easy bundling for one-liner installs.
- PM2/systemd deployment is straightforward in Zeb's environment.
- Frontend and backend can share TypeScript event schemas.
Potential framework:
- Fastify: mature, fast, good plugins.
- Hono: simple, edge-like API, also good.
Initial pick: Fastify unless we want ultra-minimal.
## Config
Config locations:
System mode:
```text
/etc/hermes-mobile/config.yaml
/var/lib/hermes-mobile/
```
User mode:
```text
~/.config/hermes-mobile/config.yaml
~/.local/share/hermes-mobile/
```
Example:
```yaml
server:
host: 0.0.0.0
port: 8787
public_url: https://hermes-mobile.molberg.cloud
auth:
pairing_enabled: true
token_ttl_days: 90
passkeys_enabled: false
hermes:
home: /root/.hermes
mode: python # python | api | cli
cli_bin: /root/.local/bin/hermes
api_base_url: http://127.0.0.1:18793/v1
api_key: ""
storage:
data_dir: /var/lib/hermes-mobile
max_upload_mb: 512
retention_days: 30
notifications:
provider: webpush # webpush | ntfy | gotify | none
ntfy_url: https://ntfy.example.com/hermes
```
## Hermes Adapter Strategy
### Phase 1: CLI/API hybrid
- Use Hermes CLI for health tests and simple prompts.
- Use Hermes API server if available for OpenAI-compatible chat.
- Parse coarse progress where possible.
Pros:
- Requires few/no Hermes changes.
- Easy to prove app UX.
Cons:
- Tool-call visibility limited.
### Phase 2: Direct Python adapter
Run a Python worker process that imports Hermes `AIAgent` and communicates with Node over stdio or local socket.
Benefits:
- Can pass `stream_delta_callback` and `tool_progress_callback`.
- More structured event stream.
- Better session control.
Potential shape:
```text
Node companion <-- JSON lines --> Python hermes_worker.py <-- imports --> AIAgent
```
Worker commands:
- `health`
- `start_task`
- `continue_session`
- `cancel_task`
- `list_sessions`
- `cron_action`
Worker emits events:
- `assistant.delta`
- `tool_call.started`
- `tool_call.output`
- `tool_call.finished`
- `assistant.final`
### Phase 3: Hermes-native plugin/API
If Hermes Agent grows a stable event API, switch adapter implementation without rewriting app/server.
## Task Lifecycle
```text
created -> queued -> running -> finished
-> failed
-> cancelled
-> waiting_approval
```
Task record:
```ts
type Task = {
id: string
sessionId?: string
title?: string
status: TaskStatus
inputText: string
attachmentIds: string[]
createdAt: string
startedAt?: string
finishedAt?: string
error?: string
}
```
## Upload Handling
Use multipart upload endpoint.
Pipeline:
1. Validate file count/size/type.
2. Store under `uploads/<task-id-or-upload-id>/original-name`.
3. Record metadata in SQLite.
4. Include absolute file paths in Hermes prompt.
Prompt augmentation example:
```text
The user uploaded these files:
- screenshot.png: /var/lib/hermes-mobile/uploads/abc/screenshot.png (image/png, 320 KB)
- source.zip: /var/lib/hermes-mobile/uploads/abc/source.zip (application/zip, 12 MB)
User prompt:
<actual user prompt>
```
## Voice Handling
MVP options:
1. Client records `.webm`/`.ogg`, server stores it, then sends file path to Hermes with instruction to transcribe/analyze.
2. Server transcribes with local faster-whisper and sends text prompt.
Recommended MVP:
- Server-side transcription using local faster-whisper if available.
- Fallback: pass audio file to Hermes as attachment.
## Notifications
Events that trigger notifications:
- task completed after app backgrounded
- task failed
- approval required
- cron job finished/failed
Use provider abstraction:
```ts
interface NotificationProvider {
sendCompletion(task: Task): Promise<void>
sendFailure(task: Task): Promise<void>
sendApproval(approval: Approval): Promise<void>
sendCron(job: CronEvent): Promise<void>
}
```
## Pairing/Auth
MVP pairing:
1. Installer/server logs one-time code.
2. Phone opens `/pair` and enters code or uses link.
3. Server stores device and creates token.
4. Token stored in secure-ish PWA storage. Later use passkeys.
For public exposure, add:
- Rate limiting
- Token rotation
- Session revoke UI
- Optional Tailscale-only mode
## Cron Integration
Initial implementation can shell out to Hermes CLI:
Install dependencies, generate an access key, and start the server:
```bash
hermes cron list --json
hermes cron run <id>
hermes cron pause <id>
hermes cron resume <id>
npm install
npm run companion:setup
npm run companion:dev
```
If CLI JSON is not available, import cron modules from Hermes Python or parse job files carefully.
`npm run companion:setup` runs the local `hermes-mobile-companion setup` command. It creates `~/.config/hermes-mobile/companion.json`, prints the generated `hm_...` access key, and stores the safe workspace root.
## System Health
Useful environment variables:
Health endpoint should check:
- companion server up
- DB writable
- upload dir writable
- Hermes CLI exists
- Hermes config exists
- Hermes API server reachable if configured
- Python worker reachable if configured
- notification provider configured
- `PORT` defaults to `8787`.
- `HOST` defaults to `0.0.0.0` so Android emulators/devices can connect.
- `HERMES_MOBILE_CONFIG` overrides the config JSON path.
- `HERMES_MOBILE_ACCESS_KEY` overrides the stored key.
- `HERMES_MOBILE_WORKSPACE_ROOT` sets the file/terminal allowlisted root.
## Logging
## Pairing
- Structured JSON logs for service.
- Human-readable recent logs in UI.
- Do not log secrets, bearer tokens, or full uploaded file content.
1. Run `npm run companion:setup` on the computer running Hermes.
2. Copy the printed access key.
3. Open Hermes Mobile Settings.
4. Use `http://10.0.2.2:8787` for the Android emulator, or `http://<LAN-IP>:8787` for a physical phone on the same network.
5. Paste the access key and tap **Test connection**.
## Open Questions
All non-health API requests require:
- Should companion run as same user as Hermes Agent for direct module/file access?
- Should public access be Cloudflare Access/Tailscale-first instead of app auth-first?
- How much Hermes core should be extended for better event streaming?
```http
Authorization: Bearer <access-key>
```
## API
- `GET /api/health` returns service, Hermes CLI, auth, uptime, and workspace status.
- `GET /api/auth/validate` validates the bearer token.
- `POST /api/chat` accepts `{ "prompt": "..." }` and invokes the local `hermes` CLI when it is on `PATH`.
- `GET /api/files?path=.` lists files under the configured workspace root.
- `GET /api/files/read?path=README.md` reads UTF-8 file content.
- `POST /api/files/write` writes `{ "path": "notes.txt", "content": "..." }` under the workspace root.
- `GET /api/files/metadata?path=README.md` returns file metadata for download-style clients.
- `POST /api/terminal/run` runs `{ "command": "pwd && ls", "cwd": ".", "timeoutMs": 10000 }` under the workspace root.
## Security Model
This is a private LAN/local companion, not an internet-facing service. Security controls are intentionally simple but strict for the MVP:
- Bearer token required for every non-health endpoint.
- File and terminal `cwd` access is constrained to `HERMES_MOBILE_WORKSPACE_ROOT` or the stored setup root.
- The mobile file explorer hides `.git`, `.dev`, `node_modules`, `dist`, and generated Android build folders.
- Path traversal outside the root is rejected.
- Terminal commands run with a configurable timeout capped at 30 seconds.
- Secrets live in local config or environment variables, never in the repo.
docs/MOBILE_APP.md
+30 -281
View File
@@ -1,297 +1,46 @@
# Mobile App Plan
# Mobile App Structure
## App Type Decision
The mobile app is a Vite React app wrapped by Capacitor for Android. It remains runnable as a browser dev app while producing an installable APK through the generated Android project.
Build Hermes Mobile as a native Android app from the start, using Capacitor around a shared React UI.
## Screens
PWA will still exist as a fallback/dev target, but it should not be the primary product experience.
- **Chat** sends prompts to `POST /api/chat` and displays Hermes replies.
- **Files** browses the companion workspace, opens UTF-8 files, edits content, and saves through HTTP.
- **Terminal** runs allowlisted workspace commands through the companion server.
- **Status** checks companion health, Hermes CLI availability, workspace root, and uptime.
- **Settings** stores the companion URL and access key in `localStorage` and validates pairing.
## Why Native Android First
## Companion Settings
A PWA can get surprisingly far on Android: installable icon, offline shell, camera/mic, file picker, Web Push, and a standalone window. But for this project, “suffices” is not the bar. The goal is that Hermes feels like a genuine phone app designed around agent work.
The default companion URL is `http://10.0.2.2:8787`, which is the standard Android emulator loopback address for a server running on the host computer.
Native Android via Capacitor gives us:
- Real APK install path.
- Native splash screen and status bar control.
- More reliable push notifications.
- Better haptics.
- Better file/share integration.
- Easier future “Share to Hermes” target from Android apps.
- Better control over permissions and app lifecycle.
- Native-feeling back button behavior.
- More credible “this is an app” feel than browser-installed PWA.
For a physical Android device, set the URL to the host machine LAN address, for example `http://192.168.1.50:8787`, and make sure the firewall allows port `8787`.
## Role of PWA
Keep PWA support for:
- Fast development loop.
- Desktop/LAN access.
- Emergency fallback if APK is not installed.
- Users who do not want to sideload.
But feature priority should be:
1. Android app / Capacitor
2. PWA fallback
3. Desktop web convenience
## Frontend Stack
Recommended:
- Vite + React + TypeScript
- Tailwind CSS
- Capacitor Android
- TanStack Query for server data
- Zustand for local UI/session state
- Framer Motion for app-like motion where useful
- Zod for API schema validation
- Vite PWA plugin for fallback web install
Capacitor plugins:
- `@capacitor/android`
- `@capacitor/app` for lifecycle/back button
- `@capacitor/haptics`
- `@capacitor/status-bar`
- `@capacitor/splash-screen`
- `@capacitor/push-notifications`
- `@capacitor/filesystem`
- `@capacitor/share`
- `@capacitor/preferences`
Potential later:
- Biometric auth plugin
- Camera plugin
- Background task/upload plugin if needed
## Build Targets
```text
apps/mobile/ shared React UI
apps/mobile/android/ Capacitor Android project
apps/mobile/dist/ web build consumed by Capacitor and PWA
```
Commands later:
## Development
```bash
pnpm mobile:dev # browser dev server
pnpm mobile:build # web build
pnpm android:sync # capacitor sync
pnpm android:apk # debug/release APK
npm run mobile:dev
```
## App Package Structure
## Android APK Build
After installing dependencies:
```bash
npm run cap:add:android --workspace @hermes-mobile/mobile
npm run android:build:debug --workspace @hermes-mobile/mobile
```
The debug APK is generated by Gradle at:
```text
apps/mobile/
capacitor.config.ts
android/ # generated Capacitor Android project
src/
app/
App.tsx
router.tsx
providers.tsx
native.ts # Capacitor runtime helpers
screens/
AskScreen/
ActivityScreen/
SessionsScreen/
CronScreen/
FilesScreen/
SettingsScreen/
components/
agent-status-orb/
composer/
tool-card/
event-timeline/
upload-tray/
voice-recorder/
cron-job-card/
approval-card/
bottom-nav/
app-shell/
features/
tasks/
sessions/
cron/
files/
approvals/
notifications/
pairing/
settings/
native/
haptics.ts
push.ts
share.ts
statusBar.ts
filesystem.ts
backButton.ts
lib/
api-client.ts
realtime.ts
audio.ts
file-utils.ts
format.ts
styles/
globals.css
tokens.css
assets/
icons/
splash/
main.tsx
public/
manifest.webmanifest
service-worker.ts
apps/mobile/android/app/build/outputs/apk/debug/app-debug.apk
```
## Navigation
If `android/` already exists, use:
Bottom tabs:
- Ask
- Activity
- Cron
- Files
- Settings
Native back behavior:
- If a sheet/modal is open, close it.
- Else if not on Ask, go back to previous tab/screen.
- Else prompt/minimize/exit according to Android convention.
## Ask Screen Behavior
States:
- idle
- composing
- uploading
- sending
- running
- waiting_approval
- finished
- failed
Composer:
- Text input
- Attach file button
- Hold/tap voice recording button
- Send button
- Optional mode chips: New / Continue / Background
When task starts:
- Input collapses to bottom.
- Task timeline appears.
- Assistant response streams.
- Tool cards appear in chronological order.
- Subtle haptic on send, completion, error, and approval required.
## Realtime
Use WebSocket first:
- `/api/realtime?token=...`
- Subscribe to task/session events.
Fallback:
- SSE `/api/tasks/:id/events/stream`
- Polling last resort.
Frontend event store should append events idempotently by event ID.
## Push Notifications
Native Android priority:
- Use Capacitor Push Notifications / FCM where practical.
- Companion server stores device push token.
- Notifications for task completion, failure, approval required, cron completion.
Fallbacks:
- Web Push for PWA.
- ntfy/Gotify if native push is too annoying for self-hosted early builds.
Important: notification provider must be abstracted so the app can start with ntfy/Web Push and later use FCM/native without rewriting task logic.
## Voice Recording
Prefer native-friendly implementation:
- First version can use browser MediaRecorder inside Capacitor WebView.
- If recording quality/lifecycle is poor, move to native audio recording plugin.
Preferred mime order for web implementation:
- `audio/webm;codecs=opus`
- `audio/ogg;codecs=opus`
- browser default fallback
UI:
- Press/tap mic to start.
- Big native-feeling recording sheet with timer/waveform.
- Haptic on start/stop.
- Cancel / send.
- Upload progress.
## File Upload UX
Priority:
- Use standard file input first inside WebView.
- Add Capacitor Filesystem/Share integration for better Android feel.
- Later add Android share target so user can share files/text/images into Hermes Mobile from other apps.
UI:
- Preview thumbnails for images.
- File chips for archives/docs.
- Upload before task send or as part of task multipart.
## Offline/Background Behavior
- App shell should load offline.
- Draft prompt persists locally via Capacitor Preferences/local storage.
- Running task state reloads from server after reconnect.
- Notifications tell user when background tasks finish.
- No attempt to run Hermes offline.
## Settings UX
First launch:
1. Enter companion URL or open pairing deep link/QR.
2. Pair device.
3. Test connection.
4. Enable notifications.
5. Land on Ask.
Settings fields:
- Companion URL
- Device name
- Notification status/test
- Theme
- Default task mode
- Upload retention display
- Hermes health
- App version/build channel
## Native Polish Checklist
- Proper Android package ID, e.g. `cloud.molberg.hermesmobile`.
- Adaptive icon.
- Native splash screen.
- Status/navigation bar colors match theme.
- Edge-to-edge layout with safe-area handling.
- Keyboard does not break composer layout.
- Native back button behavior.
- Haptics on important interactions.
- Notification permission onboarding.
- Share target later.
- Deep links for pairing.
- Avoid browser pull-to-refresh in PWA fallback; app should have explicit refresh controls.
## Accessibility
- Visible focus states.
- Reduced motion mode.
- ARIA labels for icon buttons.
- Captions/transcript for voice notes.
- Minimum 44px tap targets.
## PWA Sufficiency Summary
PWA is enough for a functional prototype.
Native Android is better for the product Zeb described.
Decision: build shared React app + Capacitor Android from day one, while keeping PWA fallback essentially free.
```bash
npm run cap:sync --workspace @hermes-mobile/mobile
cd apps/mobile/android
./gradlew assembleDebug
```