hermes-mobile/docs/MOBILE_APP.md

Mobile App Plan

App Type

Start as a PWA optimized for Android Chrome.

Why PWA first:

• Fast iteration.
• Easy self-hosting.
• No app store.
• Installable to home screen.
• Web Push support on Android is good.
• Same frontend can later be wrapped with Capacitor.

Potential later native shell:

• Capacitor Android for native share target, richer notifications, and better file/mic APIs.

Frontend Stack

Recommended:

• Vite + React + TypeScript
• Tailwind CSS
• TanStack Query for server data
• Zustand for local UI/session state
• Framer Motion for motion where useful
• Workbox or Vite PWA plugin
• Zod for API schema validation

Alternative:

• Next.js if SSR/server components become useful, but for a companion-served app Vite is simpler.

App Package Structure

apps/mobile/
  src/
    app/
      App.tsx
      router.tsx
      providers.tsx
    screens/
      AskScreen/
      ActivityScreen/
      SessionsScreen/
      CronScreen/
      FilesScreen/
      SettingsScreen/
    components/
      agent-status-orb/
      composer/
      tool-card/
      event-timeline/
      upload-tray/
      voice-recorder/
      cron-job-card/
      approval-card/
      bottom-nav/
      app-shell/
    features/
      tasks/
      sessions/
      cron/
      files/
      approvals/
      notifications/
      pairing/
      settings/
    lib/
      api-client.ts
      realtime.ts
      push.ts
      audio.ts
      file-utils.ts
      format.ts
    styles/
      globals.css
      tokens.css
    assets/
      icons/
      splash/
    main.tsx
  public/
    manifest.webmanifest
    service-worker.ts

Navigation

Bottom tabs:

• Ask
• Activity
• Cron
• Files
• Settings

Sessions can be either:

• a sub-screen from Ask, or
• included in Activity initially.

Ask Screen Behavior

States:

• idle
• composing
• uploading
• sending
• running
• waiting_approval
• finished
• failed

Composer:

• Text input
• Attach file button
• Hold/tap voice recording button
• Send button
• Optional mode chips: New / Continue / Background

When task starts:

• Input collapses to bottom.
• Task timeline appears.
• Assistant response streams.
• Tool cards appear in chronological order.

Realtime

Use WebSocket first:

• /api/realtime?token=...
• Subscribe to task/session events.

Fallback:

• SSE /api/tasks/:id/events/stream
• Polling last resort.

Frontend event store should append events idempotently by event ID.

Push Notifications

PWA flow:

1. User enables notifications in Settings.
2. App registers service worker.
3. App subscribes to PushManager.
4. Subscription sent to companion server.
5. Server sends Web Push on completion/approval/cron events.

Fallback if Web Push fails:

• Configure ntfy link in Settings.

Voice Recording

Use MediaRecorder.

Preferred mime order:

• audio/webm;codecs=opus
• audio/ogg;codecs=opus
• browser default fallback

UI:

• Press/tap mic to start.
• Big recording sheet with timer/waveform.
• Cancel / send.
• Upload progress.

File Upload UX

• Android file picker through <input type=file multiple>.
• Share Target API later for “Share to Hermes”.
• Preview thumbnails for images.
• File chips for archives/docs.
• Upload before task send or as part of task multipart.

Offline/Background Behavior

• App shell should load offline.
• Draft prompt persists locally.
• Running task state reloads from server after reconnect.
• No attempt to run Hermes offline.

Settings UX

First launch:

1. Enter companion URL or scan QR/pairing link.
2. Pair device.
3. Test connection.
4. Enable notifications.
5. Land on Ask.

Settings fields:

• Companion URL
• Device name
• Notification status/test
• Theme
• Default task mode
• Upload retention display
• Hermes health

Android Polish Checklist

• display: standalone manifest.
• Proper app icon masks/adaptive icon assets.
• Theme color matching dark background.
• Avoid viewport resize bugs with keyboard.
• Safe-area padding.
• Disable browser pull-to-refresh where possible and add in-app refresh.
• Haptics via Vibration API where appropriate.
• Back button behavior for modals/sheets.

Accessibility

• Visible focus states.
• Reduced motion mode.
• ARIA labels for icon buttons.
• Captions/transcript for voice notes.

Future Capacitor Features

• Native push via FCM.
• Native share target.
• Background upload reliability.
• Local biometric unlock.
• Better notification actions: Approve/Deny from notification.