hermes-mobile/docs/MOBILE_APP.md

# Mobile App Plan

## App Type Decision

Build Hermes Mobile as a native Android app from the start, using Capacitor around a shared React UI.

PWA will still exist as a fallback/dev target, but it should not be the primary product experience.

## Why Native Android First

A PWA can get surprisingly far on Android: installable icon, offline shell, camera/mic, file picker, Web Push, and a standalone window. But for this project, “suffices” is not the bar. The goal is that Hermes feels like a genuine phone app designed around agent work.

Native Android via Capacitor gives us:
- Real APK install path.
- Native splash screen and status bar control.
- More reliable push notifications.
- Better haptics.
- Better file/share integration.
- Easier future “Share to Hermes” target from Android apps.
- Better control over permissions and app lifecycle.
- Native-feeling back button behavior.
- More credible “this is an app” feel than browser-installed PWA.

## Role of PWA

Keep PWA support for:
- Fast development loop.
- Desktop/LAN access.
- Emergency fallback if APK is not installed.
- Users who do not want to sideload.

But feature priority should be:
1. Android app / Capacitor
2. PWA fallback
3. Desktop web convenience

## Frontend Stack

Recommended:
- Vite + React + TypeScript
- Tailwind CSS
- Capacitor Android
- TanStack Query for server data
- Zustand for local UI/session state
- Framer Motion for app-like motion where useful
- Zod for API schema validation
- Vite PWA plugin for fallback web install

Capacitor plugins:
- `@capacitor/android`
- `@capacitor/app` for lifecycle/back button
- `@capacitor/haptics`
- `@capacitor/status-bar`
- `@capacitor/splash-screen`
- `@capacitor/push-notifications`
- `@capacitor/filesystem`
- `@capacitor/share`
- `@capacitor/preferences`

Potential later:
- Biometric auth plugin
- Camera plugin
- Background task/upload plugin if needed

## Build Targets

```text
apps/mobile/                  shared React UI
apps/mobile/android/          Capacitor Android project
apps/mobile/dist/             web build consumed by Capacitor and PWA
```

Commands later:

```bash
pnpm mobile:dev        # browser dev server
pnpm mobile:build      # web build
pnpm android:sync      # capacitor sync
pnpm android:apk       # debug/release APK
```

## App Package Structure

```text
apps/mobile/
  capacitor.config.ts
  android/                    # generated Capacitor Android project
  src/
    app/
      App.tsx
      router.tsx
      providers.tsx
      native.ts               # Capacitor runtime helpers
    screens/
      AskScreen/
      ActivityScreen/
      SessionsScreen/
      CronScreen/
      FilesScreen/
      SettingsScreen/
    components/
      agent-status-orb/
      composer/
      tool-card/
      event-timeline/
      upload-tray/
      voice-recorder/
      cron-job-card/
      approval-card/
      bottom-nav/
      app-shell/
    features/
      tasks/
      sessions/
      cron/
      files/
      approvals/
      notifications/
      pairing/
      settings/
    native/
      haptics.ts
      push.ts
      share.ts
      statusBar.ts
      filesystem.ts
      backButton.ts
    lib/
      api-client.ts
      realtime.ts
      audio.ts
      file-utils.ts
      format.ts
    styles/
      globals.css
      tokens.css
    assets/
      icons/
      splash/
    main.tsx
  public/
    manifest.webmanifest
    service-worker.ts
```

## Navigation

Bottom tabs:
- Ask
- Activity
- Cron
- Files
- Settings

Native back behavior:
- If a sheet/modal is open, close it.
- Else if not on Ask, go back to previous tab/screen.
- Else prompt/minimize/exit according to Android convention.

## Ask Screen Behavior

States:
- idle
- composing
- uploading
- sending
- running
- waiting_approval
- finished
- failed

Composer:
- Text input
- Attach file button
- Hold/tap voice recording button
- Send button
- Optional mode chips: New / Continue / Background

When task starts:
- Input collapses to bottom.
- Task timeline appears.
- Assistant response streams.
- Tool cards appear in chronological order.
- Subtle haptic on send, completion, error, and approval required.

## Realtime

Use WebSocket first:
- `/api/realtime?token=...`
- Subscribe to task/session events.

Fallback:
- SSE `/api/tasks/:id/events/stream`
- Polling last resort.

Frontend event store should append events idempotently by event ID.

## Push Notifications

Native Android priority:
- Use Capacitor Push Notifications / FCM where practical.
- Companion server stores device push token.
- Notifications for task completion, failure, approval required, cron completion.

Fallbacks:
- Web Push for PWA.
- ntfy/Gotify if native push is too annoying for self-hosted early builds.

Important: notification provider must be abstracted so the app can start with ntfy/Web Push and later use FCM/native without rewriting task logic.

## Voice Recording

Prefer native-friendly implementation:
- First version can use browser MediaRecorder inside Capacitor WebView.
- If recording quality/lifecycle is poor, move to native audio recording plugin.

Preferred mime order for web implementation:
- `audio/webm;codecs=opus`
- `audio/ogg;codecs=opus`
- browser default fallback

UI:
- Press/tap mic to start.
- Big native-feeling recording sheet with timer/waveform.
- Haptic on start/stop.
- Cancel / send.
- Upload progress.

## File Upload UX

Priority:
- Use standard file input first inside WebView.
- Add Capacitor Filesystem/Share integration for better Android feel.
- Later add Android share target so user can share files/text/images into Hermes Mobile from other apps.

UI:
- Preview thumbnails for images.
- File chips for archives/docs.
- Upload before task send or as part of task multipart.

## Offline/Background Behavior

- App shell should load offline.
- Draft prompt persists locally via Capacitor Preferences/local storage.
- Running task state reloads from server after reconnect.
- Notifications tell user when background tasks finish.
- No attempt to run Hermes offline.

## Settings UX

First launch:
1. Enter companion URL or open pairing deep link/QR.
2. Pair device.
3. Test connection.
4. Enable notifications.
5. Land on Ask.

Settings fields:
- Companion URL
- Device name
- Notification status/test
- Theme
- Default task mode
- Upload retention display
- Hermes health
- App version/build channel

## Native Polish Checklist

- Proper Android package ID, e.g. `cloud.molberg.hermesmobile`.
- Adaptive icon.
- Native splash screen.
- Status/navigation bar colors match theme.
- Edge-to-edge layout with safe-area handling.
- Keyboard does not break composer layout.
- Native back button behavior.
- Haptics on important interactions.
- Notification permission onboarding.
- Share target later.
- Deep links for pairing.
- Avoid browser pull-to-refresh in PWA fallback; app should have explicit refresh controls.

## Accessibility

- Visible focus states.
- Reduced motion mode.
- ARIA labels for icon buttons.
- Captions/transcript for voice notes.
- Minimum 44px tap targets.

## PWA Sufficiency Summary

PWA is enough for a functional prototype.

Native Android is better for the product Zeb described.

Decision: build shared React app + Capacitor Android from day one, while keeping PWA fallback essentially free.