hermes-mobile/docs/MOBILE_APP.md

Mobile App Plan

App Type Decision

Build Hermes Mobile as a native Android app from the start, using Capacitor around a shared React UI.

PWA will still exist as a fallback/dev target, but it should not be the primary product experience.

Why Native Android First

A PWA can get surprisingly far on Android: installable icon, offline shell, camera/mic, file picker, Web Push, and a standalone window. But for this project, “suffices” is not the bar. The goal is that Hermes feels like a genuine phone app designed around agent work.

Native Android via Capacitor gives us:

• Real APK install path.
• Native splash screen and status bar control.
• More reliable push notifications.
• Better haptics.
• Better file/share integration.
• Easier future “Share to Hermes” target from Android apps.
• Better control over permissions and app lifecycle.
• Native-feeling back button behavior.
• More credible “this is an app” feel than browser-installed PWA.

Role of PWA

Keep PWA support for:

• Fast development loop.
• Desktop/LAN access.
• Emergency fallback if APK is not installed.
• Users who do not want to sideload.

But feature priority should be:

1. Android app / Capacitor
2. PWA fallback
3. Desktop web convenience

Frontend Stack

Recommended:

• Vite + React + TypeScript
• Tailwind CSS
• Capacitor Android
• TanStack Query for server data
• Zustand for local UI/session state
• Framer Motion for app-like motion where useful
• Zod for API schema validation
• Vite PWA plugin for fallback web install

Capacitor plugins:

• @capacitor/android
• @capacitor/app for lifecycle/back button
• @capacitor/haptics
• @capacitor/status-bar
• @capacitor/splash-screen
• @capacitor/push-notifications
• @capacitor/filesystem
• @capacitor/share
• @capacitor/preferences

Potential later:

• Biometric auth plugin
• Camera plugin
• Background task/upload plugin if needed

Build Targets

apps/mobile/                  shared React UI
apps/mobile/android/          Capacitor Android project
apps/mobile/dist/             web build consumed by Capacitor and PWA

Commands later:

pnpm mobile:dev        # browser dev server
pnpm mobile:build      # web build
pnpm android:sync      # capacitor sync
pnpm android:apk       # debug/release APK

App Package Structure

apps/mobile/
  capacitor.config.ts
  android/                    # generated Capacitor Android project
  src/
    app/
      App.tsx
      router.tsx
      providers.tsx
      native.ts               # Capacitor runtime helpers
    screens/
      AskScreen/
      ActivityScreen/
      SessionsScreen/
      CronScreen/
      FilesScreen/
      SettingsScreen/
    components/
      agent-status-orb/
      composer/
      tool-card/
      event-timeline/
      upload-tray/
      voice-recorder/
      cron-job-card/
      approval-card/
      bottom-nav/
      app-shell/
    features/
      tasks/
      sessions/
      cron/
      files/
      approvals/
      notifications/
      pairing/
      settings/
    native/
      haptics.ts
      push.ts
      share.ts
      statusBar.ts
      filesystem.ts
      backButton.ts
    lib/
      api-client.ts
      realtime.ts
      audio.ts
      file-utils.ts
      format.ts
    styles/
      globals.css
      tokens.css
    assets/
      icons/
      splash/
    main.tsx
  public/
    manifest.webmanifest
    service-worker.ts

Navigation

Bottom tabs:

• Ask
• Activity
• Cron
• Files
• Settings

Native back behavior:

• If a sheet/modal is open, close it.
• Else if not on Ask, go back to previous tab/screen.
• Else prompt/minimize/exit according to Android convention.

Ask Screen Behavior

States:

• idle
• composing
• uploading
• sending
• running
• waiting_approval
• finished
• failed

Composer:

• Text input
• Attach file button
• Hold/tap voice recording button
• Send button
• Optional mode chips: New / Continue / Background

When task starts:

• Input collapses to bottom.
• Task timeline appears.
• Assistant response streams.
• Tool cards appear in chronological order.
• Subtle haptic on send, completion, error, and approval required.

Realtime

Use WebSocket first:

• /api/realtime?token=...
• Subscribe to task/session events.

Fallback:

• SSE /api/tasks/:id/events/stream
• Polling last resort.

Frontend event store should append events idempotently by event ID.

Push Notifications

Native Android priority:

• Use Capacitor Push Notifications / FCM where practical.
• Companion server stores device push token.
• Notifications for task completion, failure, approval required, cron completion.

Fallbacks:

• Web Push for PWA.
• ntfy/Gotify if native push is too annoying for self-hosted early builds.

Important: notification provider must be abstracted so the app can start with ntfy/Web Push and later use FCM/native without rewriting task logic.

Voice Recording

Prefer native-friendly implementation:

• First version can use browser MediaRecorder inside Capacitor WebView.
• If recording quality/lifecycle is poor, move to native audio recording plugin.

Preferred mime order for web implementation:

• audio/webm;codecs=opus
• audio/ogg;codecs=opus
• browser default fallback

UI:

• Press/tap mic to start.
• Big native-feeling recording sheet with timer/waveform.
• Haptic on start/stop.
• Cancel / send.
• Upload progress.

File Upload UX

Priority:

• Use standard file input first inside WebView.
• Add Capacitor Filesystem/Share integration for better Android feel.
• Later add Android share target so user can share files/text/images into Hermes Mobile from other apps.

UI:

• Preview thumbnails for images.
• File chips for archives/docs.
• Upload before task send or as part of task multipart.

Offline/Background Behavior

• App shell should load offline.
• Draft prompt persists locally via Capacitor Preferences/local storage.
• Running task state reloads from server after reconnect.
• Notifications tell user when background tasks finish.
• No attempt to run Hermes offline.

Settings UX

First launch:

1. Enter companion URL or open pairing deep link/QR.
2. Pair device.
3. Test connection.
4. Enable notifications.
5. Land on Ask.

Settings fields:

• Companion URL
• Device name
• Notification status/test
• Theme
• Default task mode
• Upload retention display
• Hermes health
• App version/build channel

Native Polish Checklist

• Proper Android package ID, e.g. cloud.molberg.hermesmobile.
• Adaptive icon.
• Native splash screen.
• Status/navigation bar colors match theme.
• Edge-to-edge layout with safe-area handling.
• Keyboard does not break composer layout.
• Native back button behavior.
• Haptics on important interactions.
• Notification permission onboarding.
• Share target later.
• Deep links for pairing.
• Avoid browser pull-to-refresh in PWA fallback; app should have explicit refresh controls.

Accessibility

• Visible focus states.
• Reduced motion mode.
• ARIA labels for icon buttons.
• Captions/transcript for voice notes.
• Minimum 44px tap targets.

PWA Sufficiency Summary

PWA is enough for a functional prototype.

Native Android is better for the product Zeb described.

Decision: build shared React app + Capacitor Android from day one, while keeping PWA fallback essentially free.