FRIDAY documentation.
From setup to internals.

The technical guide to FRIDAY Desktop and Sentinel: installation, architecture, runtime behavior, tools, integrations, configuration, and operations.

FRIDAY is one Python package with three parts: friday.core holds shared settings, storage, the secrets vault, and the Gemini Live engine. friday.desktop is the Mac node. friday.sentinel is the headless daemon, running on Linux, Raspberry Pi OS, or macOS.

Each mode can run independently. When connected, Desktop sends heartbeats and telemetry to Sentinel, receives alerts, and pulls its model configuration at boot. A last-good local cache lets it continue if Sentinel is unreachable.

Choose your starting point.

Both modes need Python 3.11+ and a Google Gemini API key for AI features. Desktop requires macOS 12+ and audio/GUI dependencies. Sentinel installs without Mac-specific dependencies.

1. Prepare your checkout

From a checkout containing Sentinel, create a virtual environment. If cloning the project, use the repository URL from your source access and select the Sentinel-enabled branch.

python3 -m venv .venv
cp .env.template .env

Copy the template only on first setup; keep an existing .env if you already configured FRIDAY.

2A. Start Desktop

.venv/bin/pip install -e ".[desktop]"

For a standalone Desktop, set GEMINI_API_KEY in your .env. Then launch:

.venv/bin/python -m friday.desktop

Grant the permissions needed for the features you use in System Settings → Privacy & Security: Microphone and Camera, Screen Recording, Accessibility, Calendars, and Automation. Desktop uses PyAudio and PyWebView; install the platform prerequisites required by those packages if their installation fails.

To serve the desktop UI in a browser instead, run .venv/bin/python -m friday.desktop.hub and open http://127.0.0.1:8766. The desktop WebSocket gateway uses port 8765.

2B. Start Sentinel

.venv/bin/pip install -e ".[sentinel]"

# First setup only. Keep a secure backup of this key.
.venv/bin/python -m friday.sentinel keygen >> .env
.venv/bin/python -m friday.sentinel user set-password admin

# Start the daemon
.venv/bin/python -m friday.sentinel

Open http://127.0.0.1:8770/, sign in with the user you created, and enter your Gemini API key under Settings → Model & Core. Sources and channels need their own setup; monitors, triage, and the WhatsApp control ship off.

3. Connect Desktop to Sentinel

On the Sentinel host, create a token. Its plaintext is shown once:

.venv/bin/python -m friday.sentinel token create desktop

On the Mac, set these in .env. Use the loopback URL only when both modes run on the same Mac; for separate machines, use the private HTTPS URL from the deployment guide.

FRIDAY_SENTINEL_URL=http://127.0.0.1:8770
FRIDAY_SENTINEL_TOKEN=fn_replace_with_your_node_token

Restart Desktop to pull its scoped model configuration. Secrets come from the vault when present; legacy .env values are still used when the vault has no value. Process environment overrides .env.

Watching is only the beginning.

Sentinel normalises incoming items, persists them in SQLite, and publishes events to its bus. Rules handle clear urgency signals; Gemini triage handles the remaining items. A policy engine then chooses a call, alert, digest, or no interruption.

Connect your sources

Enter credentials in Settings → Sources & Monitors. For Google, create a Desktop app OAuth client and configure its ID and secret before authorising:

.venv/bin/python -m friday.sentinel google-auth

On a headless host, run authorisation on a machine with a browser using --print-only, then paste the token into Settings. Enable the required Google APIs in the same Cloud project. Switch on each source under Controls.

Decide what deserves attention

Switch on Controls → Triage after configuring sources. Settings under Alert Escalation & Channels control VIP senders, muted senders, critical keywords, quiet hours, digest times, suggested drafts, and model budget. Set sentinel.timezone so schedules use your local time.

• Rules first: VIPs, Jira priority, imminent meetings, and critical keywords can produce deterministic verdicts.
• Model second: remaining items get a validated structured verdict. Fallback verdicts cannot be critical.
• Policy last: urgency is gated by call mode, do not disturb, quiet hours, and the meeting guard. Suppressions are recorded too.
• Drafts are opt-in: policy.draft_replies can prepare suggested replies for urgent mail. It cannot send them.

Your dashboard

A shared assistant

Dashboard text, browser voice, and authorised WhatsApp conversations can inspect nodes, telemetry, recent events, settings, triage, and the digest. Ask “what did I miss?”, create a calendar reminder, or change operational controls. Control changes are audited with the channel that made them.

Browser voice

Start voice opens a Gemini Live session through Sentinel. Mic audio streams at 16 kHz; playback returns at 24 kHz. Transcripts join the same conversation as text. Mute mic leaves the session connected; End, leaving the page, or closing the tab ends it.

Use ⌘K or / to focus the message input, and ⌘/ to start or end voice. Set voice.enabled and voice.name in Settings → Telephony & Voice Agent. Browser mic access requires localhost or a secure HTTPS context.

Speech for intent. Screen for substance.

Desktop combines Gemini Live voice, a reactive Three.js orb, an asynchronous widget deck, and a persistent spatial workspace. Background agents handle multi-step work while the voice session remains available for interruption.

The orb is the status display

Colour stays tied to state. Motion follows audio playback. Tap the orb to interrupt. With no widgets, the orb sits centrally; as content arrives, it moves into the left rail.

Cards and 3D

The Cards tab holds generated HTML widgets: visual summaries, metric grids, charts, and feeds. A skeleton appears immediately and a background generator fills it. The hub manages up to eight cards and sanitises generated markup before display.

The 3D tab holds persistent editable SVE scenes and generated models. One surface is visible at a time; hidden ones retain their camera state while pausing rendering. Ask FRIDAY to show a scene or model by name.

• Spatial Visualization Engine: build and edit objects, labels, and scenes with incremental updates. Suitable for structural and editable visuals.
• Generated assets: Tripo turns a description into a textured GLB. Requires a separate TRIPO_API_KEY and generation credits. Saved models can be shown again without regenerating.
• Hand gestures: local MediaPipe tracking supports pointing, pinch-dragging, orbiting, and two-hand zoom. The active 3D surface receives gestures.

Native Mac agency

FRIDAY uses Quartz for multi-monitor capture, Accessibility for UI inspection, and CGEvent for mouse and keyboard input. It can execute shell and AppleScript commands, work with EventKit calendars and Apple Mail, and delegate multi-step operations to background agents.

Face recognition uses local YuNet and SFace models. Voice profiling uses local MFCC features. These local biometric paths are distinct from the cloud audio and image processing used by Gemini.

The conversation goes with you.

Connect WhatsApp

The Go sidecar pairs as a linked device. The Sentinel writes outbound messages to a durable SQLite outbox, then sends at your configured pace. Only whatsapp.to can receive messages or reach the inbound assistant; groups, self-messages, and unauthorised numbers are dropped.

# Requires Go on the build machine
deploy/build_sidecar.sh

# Pair on the sending host while the bridge is switched off
.venv/bin/python -m friday.sentinel whatsapp-pair

Scan the QR code in WhatsApp → Linked devices. On a separate host, copy the matching built binary to deploy/bin/friday-whatsapp as described in the repository’s deployment guide. Set your destination under Settings → Alert Escalation & Channels, then add the bridge to the host’s .env:

FRIDAY_BRIDGES=friday.sentinel.bridges.whatsapp.WhatsAppBridge

Restart Sentinel after changing FRIDAY_BRIDGES. Enable Controls → WhatsApp. Enable whatsapp.inbound for conversations and alert replies. Conversations are grouped by local day and visible in the dashboard Assistant.

Respond to an alert

Alerts include a short code, such as #3. With inbound messaging enabled, send an exact command, quote the alert, or include its code when several alerts are open.

A bare command works when exactly one alert is open. With several, FRIDAY asks which one instead of guessing. Unanswered critical alerts repeat every policy.realert_min minutes (default 30), up to policy.realert_max repeats (default 2). Alerts expire after policy.alert_ttl_h hours (default 24). DND holds reminders.

Optional phone call agent

Phone calls require a secondary Android phone, wireless adb, and an audio connection from its USB-C adapter to a USB sound card on the Linux host. ALSA’s arecord and aplay carry the audio.

sudo apt install android-tools-adb alsa-utils
arecord -l
aplay -l

# Replace with your phone's wireless-debugging addresses
adb pair PHONE_IP:PAIRING_PORT
adb connect PHONE_IP:DEBUG_PORT

Set telephony.adb_serial, telephony.audio.capture, telephony.audio.playback, and optionally telephony.call_to in Settings. Add friday.sentinel.telephony.bridge.TelephonyBridge to the comma-separated FRIDAY_BRIDGES list, restart, and enable Controls → Telephony.

Dial, answer, and hang-up commands depend on the Android ROM. A phone that refuses permissions cannot be assumed to work; editable command templates are under Advanced Hardware Templates. Failed calls fall back to WhatsApp with a reason, when that channel is configured.

Incoming calls from your number are read only because caller ID can be spoofed. Other callers reach a message-taking agent. Outgoing alert calls to the configured owner number support acknowledge and snooze. Answered calls feed the meeting-minutes pipeline.

The meeting ends. The useful part stays.

FRIDAY turns audio recordings and transcripts into decisions and action items, delivered through WhatsApp with a meeting code such as #M1. The persisted pipeline resumes across restarts: received → transcribed → ready → delivered.

Four ways to add a meeting

1. CLI: add a file on the Sentinel host.
2. Dashboard: upload on the Meetings page, optionally set a title and time, and retry failed items.
3. WhatsApp: send an audio or transcript document from your authorised number. Short uncaptioned voice notes are treated as questions instead.
4. Google Meet: enable Meet transcription and let the monitor read new transcript documents you own in Drive.

.venv/bin/python -m friday.sentinel minutes add \
  ~/Downloads/launch-sync.m4a --title "Launch sync"

.venv/bin/python -m friday.sentinel minutes add \
  meet-transcript.vtt --date 2026-10-02T10:00

A date without an offset uses sentinel.timezone. Audio supports M4A, MP3, WAV, OGG, OPUS, FLAC, AAC, and AIFF. Transcripts support VTT, SRT, and TXT. Dashboard uploads default to 200 MB; WhatsApp media defaults to 50 MB. Uncaptioned voice notes up to 120 seconds are answered as questions by default.

Get the next steps

Set minutes.my_names to the names people use for you. Your action items with future dates can be added to your calendar automatically; disable this with minutes.auto_calendar. Recordings are deleted after transcription. Transcripts and minutes default to 180-day retention; meeting codes are reusable after 7 days.

Enable Google Meet ingestion

Enable the Google Drive API in your existing Google Cloud project, rerun google-auth for Drive read access, and switch on Controls → Google Meet. Only transcript documents you own are ingested. Use uploads for meetings organised by others. The first poll looks back one day.

Give Sentinel a place to stay.

Sentinel defaults to 127.0.0.1:8770. State is stored under data/ or FRIDAY_DATA_DIR. The dashboard ships as static assets, so no frontend build is needed on a Raspberry Pi.

Run as a service

The repository includes deploy/systemd/friday-sentinel.service for Linux and deploy/launchd/com.friday.sentinel.plist for macOS development. Substitute the repository and user placeholders before installing. The systemd unit uses a readiness notification and a 90-second watchdog.

sed -e "s|__USER__|$USER|g" -e "s|__REPO__|$PWD|g" \
  deploy/systemd/friday-sentinel.service | \
  sudo tee /etc/systemd/system/friday-sentinel.service
sudo systemctl daemon-reload
sudo systemctl enable --now friday-sentinel
journalctl -u friday-sentinel -f

macOS development service

Use the provided launchd template for a Sentinel running on your Mac:

mkdir -p data/logs ~/Library/LaunchAgents
sed "s|__REPO__|$PWD|g" deploy/launchd/com.friday.sentinel.plist > \
  ~/Library/LaunchAgents/com.friday.sentinel.plist
launchctl bootstrap gui/$(id -u) \
  ~/Library/LaunchAgents/com.friday.sentinel.plist
tail -f data/logs/sentinel.log

Private remote access

Use Tailscale Serve on the Sentinel host to expose the loopback service through your private tailnet:

tailscale serve --bg --https=443 --set-path=/sentinel \
  http://127.0.0.1:8770

Behind Tailscale Serve, Caddy, or Nginx, set FRIDAY_TRUSTED_PROXY=true on the Sentinel host so forwarded HTTPS produces Secure session cookies. Restart after changing host bootstrap settings. Proxy the /ws and /voice/ws WebSockets too. Set Desktop’s FRIDAY_SENTINEL_URL to your actual private HTTPS address, including the /sentinel prefix when used.

Durable by design

POST /events commits events before returning 202. SQLite uses WAL with FULL synchronisation by default. Work interrupted while processing is requeued at boot. Event delivery is at least once with three handler attempts; handlers use idempotent operations. WhatsApp delivery has its own durable, paced outbox.

Background tasks restart with backoff. SIGINT and SIGTERM trigger an ordered shutdown bounded to ten seconds. Telemetry probes degrade gracefully when a sensor is unavailable.

API essentials

Node tokens use Authorization: Bearer fn_…. Dashboard routes use a signed-in session. State-changing dashboard calls also require X-FRIDAY-Client: dashboard.

Know what you are giving access to.

FRIDAY is self-hosted, but its AI features are cloud-powered. Source data used for triage, conversations, visual perception, and audio transcription may be sent to Gemini. Optional model generation sends prompts to Tripo. Local biometrics and gesture tracking stay on the device.

Secrets and identity

• Vault: secrets in Sentinel’s SQLite database are AES-GCM encrypted with a key derived from FRIDAY_MASTER_KEY. The setting name is authenticated with its ciphertext.
• Dashboard: a single-user login with hashed session storage and HttpOnly, SameSite=Lax cookies; Secure when served through correctly configured HTTPS.
• Node access: issue and revoke tokens from the dashboard. Token plaintext is shown once. Configuration pulls are scoped and audited.
• Desktop cache: the last-good decrypted configuration is stored locally in data/config-cache.json with mode 0600. Protect the Mac and its backups accordingly.
• Audit: sign-ins, settings changes, token events, and assistant control changes are recorded.

Source capabilities

Sentinel connectors enforce fixed capabilities in code: mail can read and draft but cannot send; Jira has no write method; calendar edits are restricted to events FRIDAY created. These limits describe the Sentinel connectors. Desktop’s native automation is a separate, broader capability.

Channels and caller identity

WhatsApp authorises the configured sender before dispatching assistant turns or downloading media, and deduplicates message IDs. It never allows an event payload to redirect the destination. Incoming phone calls are read only for the owner because caller ID is not a reliable authority to change state.

Operational checklist

• Back up FRIDAY_MASTER_KEY securely along with the persistent data you need.
• Keep local service bindings behind your intended private network or authenticated proxy.
• Enable only the sources and channels you intend to use.
• Review quiet hours, timezone, outbound pacing, and retention before relying on alerts.
• Verify the phone’s actual dial/answer behaviour before relying on telephony.

Package architecture & configuration authority

Source: readme.md; friday/core/config.py; friday/desktop/config_pull.py

FRIDAY is one installable package, friday, split into three sub-packages with a hard boundary between them:

• friday.core — everything every node shares: settings (.env + environment, per-role LLM routing), the SQLite/WAL store, event schemas, the AES-GCM secrets vault, the cross-platform telemetry collector and the Gemini provider. Imports only stdlib + psutil, aiohttp, python-dotenv, google-genai, cryptography. A test (tests/test_boundaries.py) imports every module in a poisoned interpreter and fails if anything macOS-only leaks in.
• friday.desktop — the macOS node: the Gemini Live hub, the HUD, audio, Quartz vision, CGEvent automation, EventKit, biometrics. Unchanged in behaviour; it now reads paths and model ids from core and heartbeats to the sentinel when FRIDAY_SENTINEL_URL is set.
• friday.sentinel — the headless 24/7 daemon. Identical code on macOS (launchd), Fedora and Raspberry Pi OS (systemd, Type=notify with a watchdog).

The sentinel is the configuration authority. .env holds host bootstrap only (FRIDAY_MASTER_KEY, bind address, data dir); everything else — API keys, per-role model routes, Jira/Google/telephony credentials — lives in data/sentinel.db, AES-GCM-encrypted under a key derived from FRIDAY_MASTER_KEY, with the setting key bound as associated data so a ciphertext cannot be moved between rows. A single dashboard user (created with friday-sentinel user set-password) signs in at /; sessions are HttpOnly/SameSite=Lax cookies stored hashed; every login, settings change, token issue and config pull is audited. Nodes authenticate with vault-managed tokens (friday-sentinel token create <name>, shown once). The desktop pulls its LLM configuration from GET /config?scope=desktop at boot, caches the last good copy at data/config-cache.json (mode 0600) and falls back to its .env when the sentinel is unreachable. Legacy .env values are still honoured when the vault has none and show with an env badge in the dashboard. The dashboard (Overview, Activity, Triage, Meetings, Controls, Assistant, Settings, Nodes & tokens) is vanilla ES modules with a committed Tailwind build — nothing compiles on the Pi. It is described in full under The dashboard.

            ┌─────────────────────────────────────────────────────────┐
            │                  USER INTERFACES                        │
            │  • Web GUI (Orb / Widget Deck / Spatial 3D / Gestures)  │
            │  • PyWebView desktop shell  • Phone via Tailscale HTTPS │
            │  • Voice In / Out (16 kHz PCM Mic / 24 kHz Speaker)     │
            └───────────────▲─────────────────────────▲───────────────┘
                            │ (WebSocket / Audio)     │ (Touch / Video)
                            ▼                         ▼
┌───────────────────────────────────────────────────────────────────────────────────────────┐
│                          DESKTOP NODE HUB (friday/desktop/hub.py)                         │
│                                                                                           │
│  ┌───────────────────────────┐  ┌───────────────────────────┐  ┌────────────────────────┐ │
│  │    Audio Pipeline         │  │   State & Event Hub       │  │  Gemini Live Session   │ │
│  │ • PyAudio 16kHz/24kHz     │  │ • Play Queue / Interrupts │  │ • Aoede voice, tools   │ │
│  │ • Voice Enrollment Buffer │  │ • Widget Deck Registry    │  │ • Per-turn receive loop│ │
│  │ • Remote Audio Routing    │  │ • Approval State Machine  │  │ • In-run resumption    │ │
│  └───────────────────────────┘  └───────────────────────────┘  └────────────────────────┘ │
└──────┬────────────────────┬───────────────────┬────────────────────────┬──────────────────┘
       │ Tool Calls         │ Dispatch          │ Scene Ops              │ Biometrics
       ▼                    ▼                   ▼                        ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ CAPABILITY       │ │ BACKGROUND AGENTS│ │ SPATIAL ENGINE (SVE) │ │ RECOGNITION & MEMORY │
│ SUBSYSTEMS       │ │ agents.py        │ │                      │ │                      │
│                  │ │                  │ │ • sentry_scene.py    │ │ • sentry_recognition │
│ • sentry_vision  │ │ • os tier        │ │   SceneGraph deltas  │ │   YuNet + SFace 128-d│
│ • sentry_action  │ │   (routed model) │ │ • web_gui/sve.js     │ │   Mel MFCC voice     │
│ • sentry_exec    │ │ • spatial tier   │ │   Three.js renderer  │ │ • friday_memory.json │
│ • sentry_personal│ │ • widget writer  │ │ • web_gui/gestures.js│ │   Persistent facts   │
│ • sentry_web     │ │   (routed model) │ │   MediaPipe hands    │ │                      │
└──────────────────┘ └──────────────────┘ └──────────────────────┘ └──────────────────────┘

Desktop interface & state transitions

Source: readme.md; friday/desktop/web_gui/

Workspace tabs

The workspace is split in two, behind a tab bar carrying live counts:

• Cards — HTML and component widgets, stacked newest-on-top and scrollable, exactly as before.
• 3D — every 3D surface, whether an SVE scene (3d_spatial) or a generated Tripo asset (3d_asset). Only one is visible at a time, filling the full workspace height. The rest stay mounted with their WebGL context and camera intact but stop rendering, so nothing is re-loaded when you switch back.

Switch by clicking a pill above the stage, or ask FRIDAY — show_3d_view takes 'spatial' for the scene or a model's name, and is the only thing that moves the selection. A newly arrived model claims the slot and raises the 3D tab, the same way a new card raises Cards. Re-showing a saved model reuses its card (ids are derived from the file), so asking for the same one twice never stacks a second pill.

Hand tracking follows the visible model: switching to the scene hands gestures back to the SVE, switching to a model points them at that model.

Both 3D types share one chrome: the pills name them, so neither card carries its own header, and both use the same floating Hands / Reset view pills over a full-height stage.

The GUI is a spatial workspace with Cards and 3D tabs. The ambient orb, telemetry, agent chips, camera preview, command lane, and sensor dock occupy reserved regions around the workspace.

┌──────────────────────────────────────────────────────────────────────────┐
│ 21:04:18 │ CPU 38% │ RAM 12.8/16 GB          ← ambient telemetry HUD     │
│ ◐ OS agent · Open Safari and summarise…      ← background agent chips    │
│                                                                          │
│      ╭─────────╮        ┌──────────────────────────────────────────────┐ │
│      │  ORB    │        │  ALPHABET INC.                            ×  │ │
│      │ (state) │        │  207.42  USD              ▲ +3.81 +1.87%     │ │
│      ╰─────────╯        │  ╭────────────────────────────────────────╮  │ │
│                         │  │  gradient area chart, dashed baseline  │  │ │
│    ← 260px rail         │  ╰────────────────────────────────────────╯  │ │
│                         │  OPEN 204.10   HIGH 209.94   LOW 203.44      │ │
│                         └──────────────────────────────────────────────┘ │
│  ┌───────────┐          ┌──────────────────────────────────────────────┐ │
│  │ camera PIP│          │  MARKET INTELLIGENCE                      ×  │ │
│  │ (mirrored)│          │  01 [ALERT] Antitrust ruling lands Tuesday   │ │
│  └───────────┘          └──────────────────────────────────────────────┘ │
│  ╭ Speak, or type… ╮                                   ╭─ 🎙 📷 🖥 ─╮     │
└──────────────────────────────────────────────────────────────────────────┘

Layout states

Driven purely by how many widgets are mounted:

Transitions interpolate over 0.6s cubic-bezier(0.16, 1, 0.3, 1), and both WebGL renderers are re-fitted during the animation so nothing stretches.

The orb as status

State drives colour; colour holds until the state changes. Nothing else can shift the hue — audio drives motion and brightness only.

Over all five, a fixed violet-to-blush accent (#A18CD1 → #FBC2EB) tints the rim highlight and the outer bloom. It carries no state — it is purely FRIDAY's finish, and the state hue stays exactly as readable as before.

The "speaking" state follows the hub's authoritative turn status and the playback timeline — audio arrives roughly 3× faster than it plays, so the orb animates in step with what you hear, not with what has downloaded. Tap the orb to interrupt.

Ambient furniture

• Telemetry HUD (top-left) — live clock, CPU and RAM, updated every 2 s from the hub.
• Agent chips — one per running background agent, with a spinner and its goal.
• Camera PIP (bottom-left) — mirrored; the hand-landmark overlay is mirrored with it so the skeleton stays registered.
• Command lane (bottom-centre) — ghosted until you hover, focus, or simply start typing.
• Sensor dock (bottom-right) — three icon toggles: mic, camera, screen. Active glows cyan; inactive is ghosted with a strike.

Desktop engines, agents & visual generation

Source: readme.md; friday/desktop/hub.py; agents.py; widget_generator.py; asset_generator.py

The model identifiers and timing measurements below describe this checkout, not guaranteed provider availability or performance. Configure model routes for the API access available to your installation.

1. Gemini Live Hub (friday/desktop/hub.py)

• Bidirectional voice over WebSockets to gemini-3.1-flash-live-preview, typed LiveConnectConfig with the Aoede prebuilt voice (override with FRIDAY_VOICE).
• Per-turn receive loop — session.receive() yields one conversational turn and ends, so the loop re-enters it. Without that the session tears down after every reply and the reconnect replays the last turn.
• Session lifecycle — the resumption handle is held in memory only, bridging GoAway rotations and drops within a run. A new process is always a new conversation.
• Persona — FRIDAY_SYSTEM_INSTRUCTION holds who she is and how she sounds: perceptive, effortlessly competent, dryly witty, subtly warm. It is a standalone constant, composed with the operational rules by build_system_instruction(), so the voice can be retuned without touching the tool, widget and safety instructions.
• Frontend assets are never cached stale — the GUI server sends Cache-Control: no-cache, must-revalidate. Without it WKWebView applies heuristic freshness and can keep rendering an old app.js/style.css across relaunches (the desktop shell runs private_mode=False, so its store survives restarts). ETags are kept, so unchanged files still cost only a 304.
• Concise by default — spoken replies stay under ten words; when a widget is mounted, one or two sentences carrying the key takeaway. Detail belongs on screen.

2. Background Agent Tiers (friday/desktop/agents.py)

Live must stay free for barge-in, so anything multi-step is dispatched off the audio path via dispatch_agent. FRIDAY acknowledges in the same turn ("Working on that now.") and announces the result when it lands.

Agents reuse the hub's own tool implementations, run up to 12 tool round-trips, and report back a single spoken sentence. Results are queued for a gap in the conversation — a finished agent can never cut FRIDAY off mid-sentence.

3. Async Widget Deck (hub.py, widget_generator.py, web_gui/app.js)

Composing a data-dense card takes seconds. Doing that inside the voice turn would stall the conversation, so the work is split in two:

  user speaks
       │
       ▼
  Gemini Live ──► speaks the takeaway (1-2 sentences)
       │
       └──► create_skeleton_widget(widget_id, title, query_context)   returns in ~0ms
                    │
                    ├──► broadcast "create_skeleton"  → card appears, shimmering
                    │
                    └──► background task: gemini-3.7-flash writes the card HTML
                                  │
                                  ▼
                         broadcast "patch_content"   → skeleton fades, card hydrates

Measured end to end: the tool returns at +0.00s, the skeleton is on screen in the same frame, and content patches in at ~4–12s depending on whether the generator needs to search for live figures.

• The hub owns the deck (max 8 cards). A client connecting late gets a full sync snapshot; a card dismissed mid-generation is never patched.
• query_context is the contract. The generator cannot see the conversation — it receives only that string, so Live is instructed to spell out every figure and section the card should carry.
• Everything generated is sanitised before it reaches the DOM. <script>, <iframe>, <style>, <link>, inline on* handlers and javascript: URLs are stripped hub-side. The generator is told not to emit them; the sanitiser is what guarantees it.
• The generator writes no CSS. It composes from a fixed set of hud-* classes defined in style.css, which is what keeps model-authored markup looking like the rest of the app.
• Charts are earned, not decorative. One is drawn only when there is a genuine series over time or a set of comparable magnitudes. A quote card gets a chart; a headlines card, a password or a weather summary does not.
• Images are fetched, verified, then proxied. Two failure modes are real and both were observed: a model reconstructing a plausible CDN path that answers 401, and a genuine article image that answers 403 to a direct browser request because of hotlink protection. So the hub fetches each candidate itself with a browser user-agent and the image's own origin as Referer, drops whatever cannot be retrieved, and rewrites the survivors to /img?u=… — a local endpoint that streams the bytes from the hub. The browser only ever loads images from localhost, and a card never shows a broken frame.

4. Multi-Monitor Vision & Spatial Targeting (sentry_vision.py)

• Quartz display enumeration — all monitors, scaling factors, desktop arrangement.
• Context-aware capture — the display holding the frontmost window, a specific display, or a labelled composite of all.
• Global coordinate tracking — LAST_CAPTURE_BOUNDS maps normalized 0–1000 model coordinates back to physical pixels.

5. Native macOS Desktop Automation (sentry_action.py, sentry_exec.py)

• Hardware-level input via CGEventPost (kCGHIDEventTap), bypassing PyAutoGUI's single-monitor clamping.
• Accessibility inspector (read_ui_elements) reads the frontmost app's AX tree for exact control positions.
• Window & Spaces enumeration (list_open_windows) across the Quartz Window Server.
• Shell and AppleScript execution with timeouts and the remote approval gate.

6. Biometric Face & Voice Recognition (sentry_recognition.py)

• OpenCV YuNet ONNX detection + SFace 128-d embeddings, cosine-matched against friday_profiles.json.
• Voice fingerprinting — pure-NumPy MFCCs pooled by mean and variance from the rolling mic buffer.

7. Spatial Visualization Engine (sentry_scene.py, web_gui/sve.js, web_gui/gestures.js)

• Persistent 3D workspace — scenes live in friday_scenes.json and stay on stage until dismissed.
• Incremental delta protocol — update, rotate, recolour, highlight, hide, or explode individual objects; never a full rebuild.
• Screen-sized labels — annotations are sized to a constant on-screen height, depth-tested so geometry occludes them, and decluttered by screen-space overlap (12 visible at once, nearest first).
• Markerless hand tracking — vendored MediaPipe HandLandmarker WASM: point to hover, pinch to grab, pinch empty space to orbit, two-hand pinch to zoom.

8. Generated 3D Assets (asset_generator.py, web_gui/asset_viewer.js)

• Text-to-3D via Tripo3D — generate_spatial_3d_asset turns a description into a textured .glb. Requires TRIPO_API_KEY; without it the SVE still works and only this tool is unavailable.
• Never on the audio path — generation takes tens of seconds, so the tool returns instantly, mounts a card, and runs the poll loop as a background task. The card shows live progress and the model drops in when it lands, exactly like the widget generator.
• PBR viewer — GLTFLoader with a RoomEnvironment image-based light plus cyan key and violet rim lights, orbit controls, and auto-framing (models arrive at arbitrary scale, so each is normalised to unit size and the camera fitted to it).
• Downloaded, not hotlinked — the hub fetches the .glb server-side and saves it to generated_assets/, then serves it from /assets. The CDN sends no CORS headers and signs its URLs with an expiry, so a card pointed straight at it fails even though generation succeeded.
• Models persist — every asset is content-addressed (<slug>_<sha1>.glb) and indexed in friday_assets.json. list_3d_assets shows what exists and show_3d_asset re-mounts one instantly for free, so asking for the same model twice never costs a second credit. Runtime assets are kept under the configured data directory and are excluded from version control.
• Hand-gesture control, on by default — a model takes gesture focus the moment it mounts and starts hand tracking itself, the way a live scene does. Pinch-drag to grab and move it, pinch empty space or open palm to orbit, two-hand pinch to zoom. The card carries the same Hands / Reset view pills as the spatial card, and fills the active 3D workspace. gestures.js now resolves a target rather than calling window.SVE directly, and a focused card implements the same surface (pickAt / select / moveSelectedTo / orbitCamera / dollyCamera / commitSelectedMove / register+unregisterInputSource). With no card focused the target is the SVE, exactly as before. The gesture cursor and HUD move into the focused card and back on release.
• Context-safe — every card owns a WebGL context and browsers cap those, so dismissing a card disposes its renderer, geometry, and textures.

This is not a replacement for the SVE. A generated asset is one photoreal object: no ids, no labels, no edits, variable generation time, and credits per call. Anything structural or editable — molecules, orbits, flowcharts, anatomy, data — stays an SVE scene, which is directly updatable and does not use Tripo credits; Gemini usage still applies. The persona instruction and the tool description both enforce that split.

9. Personal Productivity Suite (sentry_personal.py)

• EventKit calendars via PyObjC across iCloud, Google, and Exchange.
• Apple Mail via AppleScript — read recent mail, search sender/subject.

10. Zero-Trust Remote Access (deploy/setup_remote.sh, Tailscale)

• The hub binds only to 127.0.0.1; remote access is tunnelled through Tailscale Serve HTTPS.
• Human-in-the-loop approvals — while a remote client is connected, every shell and AppleScript call suspends for one-tap approval with a 45-second auto-deny.
• Smart sensor routing — the phone's mic and camera become the primary senses; the unattended Mac's webcam and screen are left alone unless asked for explicitly.

Widget design vocabulary

The generator is given this vocabulary and nothing else — no <style> blocks, no invented colours. Every class below is defined in web_gui/style.css:

Unanticipated markup still lands sensibly: tables, lists, headings, paragraphs and images inside .hud are given baseline styling rather than inheriting browser defaults.

Scene graph, rendering & gesture protocol

Source: SVE.md (updated to the current 3D workspace); friday/desktop/sentry_scene.py; web_gui/sve.js; web_gui/gestures.js

Architecture

User request (voice/text)
        │
        ▼
LLM (Gemini Live / Gemini agents)          ← visualization planner: decides
        │  tool calls                        objects, ids, layout, animation
        ▼
create_3d_scene / update_3d_scene / delete_3d_scene / list_3d_scenes / inspect_3d_scene
        │  JSON scene-graph specs & ops
        ▼
sentry_scene.SceneManager                 ← validation, persistence
        │                                    (friday_scenes.json), state, diffing
        │  incremental op broadcast (WebSocket)
        ▼
web_gui/sve.js                            ← rendering backend (Three.js)
        │                                    object cache, animation loop,
        ▼                                    interaction manager
3D workspace in FRIDAY GUI                 ← live scene; user actions stream
                                             back to SceneManager

Key property: object-level updates. "Highlight the left ventricle" becomes one {action:"highlight", id:"left_ventricle"} op — the renderer touches one mesh; nothing is regenerated.

Scene graph

A scene: {id, name, objects: {id → spec}, environment, selected}.

Object spec (renderer-neutral JSON):

Environment: {background, grid, stars, ambient, camera:{position,target}}.

Edit ops (update_3d_scene): add, update (merge changes), remove, highlight/unhighlight, hide/show, focus, camera, environment, explode (factor), style (wireframe/solid).

Persistence & memory

• Scenes persist in friday_scenes.json; reload on restart, pushed to every connecting GUI as an sve_workspace snapshot.
• The AI recalls stage state via list_3d_scenes / inspect_3d_scene — including which object the user selected by clicking, so "make it red" can resolve "it".
• User GUI actions (select, delete object, close scene) stream back via sve_user_action and update the same store — AI and GUI never diverge.

Interaction manager

Built-in sources (web_gui/sve.js): - Mouse/touch: OrbitControls (rotate/zoom/pan), raycast click-select. - Keyboard: F focus selected, Del remove selected. - Voice: inherently — any spoken edit becomes scene ops via the LLM.

Hand tracking (implemented — web_gui/gestures.js)

MediaPipe HandLandmarker, fully local (vendored wasm + model, ~27MB in web_gui/vendor/). Toggle with the ✋ Hands button in the 3D toolbar; shares the webcam stream with the preview card (refcounted FridayCamera).

Cursor is mirrored (hand right → cursor right) and EMA-smoothed.

Other input sources (extension point)

SVE.registerInputSource({update(dt), dispose()}) runs every frame. Helper API for sources: SVE.pickAt(ndc), SVE.select(obj), SVE.selected, SVE.moveSelectedTo(ndc), SVE.commitSelectedMove(), SVE.orbitCamera(dAz, dPolar), SVE.dollyCamera(factor). The same contract fits OpenXR hand tracking, Leap Motion, or game controllers; the scene graph does not know about them.

Rendering backend abstraction

The scene graph and op stream are engine-neutral JSON. sve.js is the Three.js implementation. To swap engines (Babylon.js, native Metal/Vulkan viewer, WebXR renderer): implement the same three entry points —

• consume sve_workspace (full state), sve_scene_create, sve_scene_update (ops), sve_scene_delete;
• emit sve_user_action messages;
• honor the object spec table above.

No Python changes needed.

Extending the vocabulary ("plugins")

Domain plugins are additions at two layers:

1. New primitive types (only when composition can't express it): add a case to _sanitize_object (sentry_scene.py) and to buildObject (sve.js). Example: molecule_bond, terrain, gltf (load external models).
2. Domain knowledge lives in the LLM prompt/tool description — e.g. a Chemistry preset is a prompt fragment teaching element colors (CPK) and bond conventions, not code. Add such fragments to the system instruction when needed.

Live data feeds (CPU, telemetry): broadcast sve_scene_update ops from any backend task (e.g. a psutil loop emitting {action:"update", id:"cpu_bar", changes:{scale:[1, load, 1]}}) — the renderer already applies streamed ops continuously; no new mechanism required.

Current limits (honest list)

• Hand tracking is single-viewport 2D-projected (no depth grab); rotation and scale gestures per-object not bound yet — two-hand zoom moves the camera.
• VR/AR/OpenXR, Unity/Unreal backends: out of scope; abstraction supports them.
• text rendering uses canvas sprites (billboards), not extruded 3D type.
• One viewport; scenes switch via tabs (all stay resident and animated state is preserved — only the active one renders).
• Drag-moving objects with the mouse: not bound yet (select/focus/delete are).

Scene and asset distinction

The scene-graph operation protocol applies to SVE objects. A Tripo-generated GLB is loaded by the asset viewer as a model, rather than being exposed as independently editable SVE object IDs. Input-source interfaces are shared so hand tracking follows whichever 3D surface is visible.

Dashboard rendering & interaction model

Source: readme.md; friday/sentinel/dashboard/

The sentinel serves its own admin console at /. It is static ES modules under friday/sentinel/dashboard/, with no bundler and no framework, and a Tailwind build committed alongside them, so nothing compiles on the Pi.

Pages:

• Overview: every node's heartbeat and telemetry tiles, and the Watching and WhatsApp cards.
• Activity: the event bus as it happens.
• Triage: what the monitors found and what was decided.
• Meetings: uploads, pipeline state, #M codes and retry.
• Controls, Assistant and Settings, described below.
• Nodes & tokens: issue and revoke node tokens.

The shell. The sidebar and header are pinned. The page is a fixed-height frame (h-dvh overflow-hidden, the dynamic viewport height, so a phone's browser toolbar never hides the bottom of the app), and only the content area (<main>) scrolls, so the navigation and the connection badge (live, reconnecting or offline) never scroll away. Both bars sit on a translucent, blurred background over zinc borders. Below the md breakpoint the sidebar becomes a tab strip under the header.

• Full-bleed pages. A page may declare itself full-bleed (export const layout = "full"). The router then gives it the whole content area with no padding and no page scroll; this is how the Assistant lays out its own columns.
• Scroll. Every navigation starts the new page at the top.
• Leaving a page that is still loading. Each page mounts into its own host element. If you leave before it has finished loading, it is undone completely: its late content, its listeners, any voice session and the orb's render loop.

Settings. Every dynamic setting in the vault appears on one page, declared in the settings registry, grouped into five domain cards:

• Rows. Each row reads left to right:
• On the left: a plain-language label; a one-line description; the config key as a small monospace badge; and where the value comes from (default, vault or env).
• On the right: the control.

At md and wider the two columns split 40/60; on a phone they stack. - Controls by type:

• Advanced blocks. Rarely touched or risky settings sit in each card's collapsed Advanced block. Telephony's is called Advanced Hardware Templates and holds the adb dial, answer and hang-up commands.

Edits are held until you save; nothing is saved as you type.

• Marking and counting. An edited row is marked with an accent. A bar pinned to the bottom of the page counts the edits ("3 unsaved changes") and offers Discard and Save changes.
• Save. It sends every change in one request.
• If the server refuses a value, its reason appears under that field. An Advanced block holding it opens, and the page scrolls to the first problem. Nothing is marked saved.
• A secret you typed is cleared from its field once it has been saved.
• Changes made elsewhere. If a setting changes while you are editing (another tab, a WhatsApp command, the CLI), untouched rows update in place and your edits are kept.
• Leaving. Navigating away or closing the tab with unsaved edits asks first.

Labels, units, the Advanced flag and the card layout are declared once, in friday/sentinel/settings_registry.py, and served by GET /api/settings/schema. A test fails if a setting ships without a label, or with a unit its key contradicts.

Controls. These are the live switches: call mode, do not disturb, WhatsApp, telephony, each monitor and triage, plus the sentinel's timing.

• Instant apply. Unlike Settings, every change takes effect the moment you make it. A switch flips and is saved at once.
• Refusals. If the server refuses a change, the switch moves back and the reason is shown.
• Timing. A timing number applies when you press Enter or leave the field.

Assistant. A split view: 40% visualiser on the left, 60% conversation on the right, divided by a single hairline and drawn on the page background without boxed cards.

• The visualiser.
• The orb: Three.js, shared with the desktop HUD.
• A status pill: Idle, Listening, Thinking, Speaking or Offline, with "· muted" added while the mic is muted.
• The voice controls: Start voice (which becomes End), and Mute mic, which keeps the session open but sends no audio.
• The conversation.
• Your messages are tinted bubbles.
• FRIDAY's replies are rendered as markdown: headings, bold and italic, lists, quotes, inline and fenced code, links, and tables with column alignment.
• Tools. Each tool FRIDAY used appears as a small name · ms chip.
• Voice. A voice session's transcripts land in the same stream as they are spoken, marked 🎙.
• Streaming. Tokens accumulate, and a reply is redrawn at most once per animation frame, so text appears smoothly without the content above it moving.
• Scrolling. The view follows new text while you are at the bottom. Scroll up even slightly and it stays put until you return to the bottom or send a message.
• Interruptions. If you interrupt FRIDAY by voice, her cut-off reply stays as it was and the next one starts a new bubble.
• Shortcuts. ⌘K or / focuses the message box; ⌘/ starts or ends voice.
• Leaving and returning. Leaving the page ends the voice session and stops the orb's render loop. Coming back re-attaches the same canvas, so the orb is always there.

How it is built:

md.js only ever creates elements and text nodes and never parses HTML, so nothing in a reply can become markup. Links must be http(s), and table alignment comes from a fixed list.

After changing any class, run deploy/build_css.sh.

• tests/sentinel/test_dashboard_files.py fails on a stale build. It also checks each page's wiring, and renders markdown over every prefix of a streamed reply.
• tests/sentinel/test_dashboard_js.py runs the dashboard's modules in node against a small DOM shim, and is skipped where node is not installed. It covers:
• dirty tracking, including an edit typed while a save is in flight;
• the controls by type;
• once-per-frame streaming and follow-the-tail;
• the voice session's barge-in and end signals;
• the orb's remount.

Monitoring, triage & channel internals

Source: readme.md; friday/sentinel/sources, triage, bridges/whatsapp, escalation, minutes, telephony; current policy.py

friday/sentinel/sources/ holds one capability object per surface, each exposing only the verbs it is allowed to use: GmailAccount (read + draft), ImapAccount (read + draft by APPEND), CalendarGuarded (read, create, and modify only what it stamped as its own), and JiraReadOnly (one _get, no write method to call). Two tests enforce the promises rather than trusting them: no module under friday/ may import smtplib, and no executable string in sources/ may name a send endpoint.

friday/sentinel/watch.py polls each source on its own interval, honours the controls.monitors.* switches live, deduplicates through watch_items (the INSERT is the dedupe) and publishes monitor.item. A source that fails goes degraded and backs off; a revoked credential goes needs_reauth and stops retrying, because only a human can fix it. Credentials live in the vault under sources.*; Google consent is a one-time friday-sentinel google-auth.

Deciding

friday/sentinel/triage/ is the deciding half: rules.py (deterministic, ordered, each rule names itself in the verdict), model.py (one forced record_triage call whose parameters are enums, a per-hour budget, and a fallback ladder that can never return critical), policy.py (one pure function from verdict plus controls to an action), handler.py (the bus handler that judges, claims and dispatches), drafts.py (a suggested reply written to Drafts, never sent) and digest.py (the scheduled pile).

The at-least-once bus means an item can arrive twice, so dispatch is guarded by watch_item_claim_decision — an UPDATE … WHERE decision = '' whose Boolean is the guard. One item yields at most one call, one alert and one draft, however many times it is redelivered.

Alerts travel over the existing /ws to friday/desktop/alerts.py, which shows a card on the HUD and queues the spoken line behind whatever FRIDAY is already saying.

Everything is configuration, live and unencrypted: policy.vip and policy.mute decide who matters, policy.keywords_critical decides what does, policy.quiet_hours (which may cross midnight) and policy.digest_times decide when, policy.draft_replies opts into writing drafts, and policy.model_budget_per_hour caps what the model may cost. controls.triage is the master switch and ships off; sentinel.timezone is the zone all of that is read in.

Sending

friday/sentinel/bridges/whatsapp/ is the outbound half: jid.py (one destination, spelled one way), protocol.py (newline-delimited JSON, both directions), sidecar.py (a child process kept alive through exits, hangs, garbage and lock-outs) and bridge.py (the bus handler, the outbox and the paced sender). sidecar/whatsapp/ is a separate Go module — go.mau.fi/whatsmeow plus a pure-Go SQLite driver so it cross-compiles for the Pi with no toolchain there — and nothing in friday/ imports it.

The handler's only job is to write the row down, because the bus gives it three attempts and a phone can be off for a day; the sender loop then owns delivery with its own pace and patience. The event id is the dedupe key, so an at-least-once redelivery queues one message. The paired session is the only credential: it lives in a file only the sidecar opens, guarded by an advisory flock so the daemon and whatsapp-pair can never fight over it.

The pace is deliberate, because a burst is what gets an account banned: whatsapp.min_gap_s and whatsapp.max_per_hour space the queue out, whatsapp.max_body_chars truncates a long digest below WhatsApp's own limit, and anything still undelivered after whatsapp.expire_after_h is expired and audited rather than arriving a day late. whatsapp.to is the single destination, validated and normalised to <countrycode><number>@s.whatsapp.net; controls.whatsapp is the live switch and ships off. This is an unofficial client, so the number it pairs is the secondary one.

Listening

bridges/whatsapp/inbound.py is the gate: authorise, then deduplicate before dispatching the turn. WhatsApp replays offline messages; already-answered IDs are ignored. The current bridge does not cap the authorised owner's inbound message count; outbound pacing and per-message input limits still apply. conversation.py runs the turn: one thread per local day, the per-conversation lock the dashboard already uses but awaited rather than refused, run_turn with via="whatsapp", and one reply through markdown.py into the outbox with a deterministic id. The turn runs as a tracked task rather than inside the bus handler: the bus cancels a handler after 30 s and retries it, so a slow tool loop run inline would be cut off and run again, tools and all. As a task it keeps its own 180 s budget, the event completes once, and alerts queued behind it are not held up.

Resolving who sent it happens in Go, because recent WhatsApp releases set the sender to a hidden <opaque>@lid address rather than a phone number: resolveSender tries the sender, then the alternative address whatsmeow supplies, then the LID-to-phone mapping store, and forwards nothing it cannot identify. Groups and FRIDAY's own messages are dropped before their text crosses the pipe.

Acting on alerts

friday/sentinel/escalation/ follows an alert through. notifier.py turns every call or alert decision into a WhatsApp message carrying a short code (#3), written in the same transaction as the escalation row and a reply ref. commands.py is a strict grammar: ack, dismiss, not urgent and snooze 2h | 90m | 1d | tomorrow, optionally with #3. Only a whole message counts, so "ack, on it" stays a sentence for the assistant. actions.py resolves which alert a command means and moves it with one conditional UPDATE. runner.py re-raises open critical alerts and wakes snoozed ones.

Resolution never guesses. An explicit code wins, then the alert the reply quoted (the sidecar forwards the quoted message id, and whatsapp_refs maps it back), then the only live alert. With two live alerts and no pointer, FRIDAY asks which one. Commands wait behind the same per-conversation lock as turns, never reach the model, and are audited as user:<name> via whatsapp (escalation action). ack, dismiss and not urgent also write a labelled row to triage_feedback for future policy tuning.

Minutes of meeting

friday/sentinel/minutes/ turns a recording or transcript into minutes:

• transcribe.py reads .vtt, .srt and .txt locally and sends audio to Gemini through the Files API, deleting the upload afterwards.
• extract.py makes one forced record_minutes call over a fenced, untrusted transcript and validates every field.
• render.py writes the WhatsApp text and splits it under whatsapp.max_body_chars as [1/3] parts.
• runner.py is the pipeline. Each meeting's state lives in its row (received → transcribed → ready → delivered), so a restart resumes and a step never runs twice.

Minutes arrive with a code (#M1). Quoting any part, or sending #M1 tasks, #M1 decisions, #M1 transcript or #M1 add <n>, routes to that meeting. A question asked by quoting the minutes runs one assistant turn with that meeting as context. Action items that are yours (minutes.my_names) and dated in the future go on the calendar once each.

Meetings arrive from four places, and all of them go through the same pipeline: - The minutes add CLI. - The dashboard's Meetings page, a streamed upload with the pipeline state, #M codes and retry. - WhatsApp. The sidecar reports a file as metadata, and only after the sender is authorised does Python ask it to download. Uncaptioned audio up to minutes.voice_note_max_s seconds is a voice note, transcribed and answered as a question. Longer audio, and any .vtt/.srt/.txt or audio document, becomes a meeting. - Google Meet. The meet monitor reads new transcript documents you own from Drive, read-only. A 403 is read by its reason: a disabled API or a rate limit is not needs_reauth, and a file Google will not export is skipped.

The call agent

friday/sentinel/telephony/ puts FRIDAY on a phone line:

• adb.py drives the secondary phone over wireless adb, using dial, answer and hang-up commands that are settings (Settings → Telephony & Voice Agent → Advanced Hardware Templates) because which one works depends on the ROM. It reads dumpsys telecom (has the outgoing call gone ACTIVE?) and dumpsys telephony.registry (ringing, and who). It turns every failure into a reason: unreachable, permission denied, or switched off.
• audio.py carries the call audio through a USB sound card with arecord and aplay. No Python audio library is needed.
• session.py runs the Live engine on that line. FRIDAY speaks first and the transcript is kept.
• bridge.py places alert calls, answers incoming ones, and keeps them to one line.

A critical alert whose policy decision is call rings your phone. Say "acknowledge" or "snooze two hours" and the alert is updated exactly as a WhatsApp reply would update it. Before every dial and redial, the bridge checks the alert, so anything you have already acked, dismissed or snoozed is never rung. Every way a call can fail sends the alert back to WhatsApp, saying why: the phone was unreachable, it refused to dial, or there is no audio. Incoming calls are answered when the controls allow it. Your number gets the owner persona, read-only because caller ID can be faked. Anyone else gets a message-taking agent with one tool that never says your number. Every answered call becomes minutes. A voice model that never connects, or an answer the phone never picks up, goes back to WhatsApp instead.

The Live engine

friday.core.live owns every Gemini Live session in the project: connect, resume, rotate on GoAway, decode the stream into typed events (AudioOut, TextOut, Transcript, Interrupted, TurnComplete, ToolStarted/ToolFinished, GoAway), and answer tool calls — including the ones that raise, because an unanswered turn never closes and the model re-issues it after every resume. Consumers push media with send_audio / send_video / send_text and react to session.events(). An optional AudioTransport lets the session pump audio itself: the dashboard passes a WebSocket transport, the desktop hub passes none and keeps its own mic/playback multiplexing, and the call agent uses an ALSA transport.

Event dispatcher and lifecycle

EventBus defaults to 16-event claim batches, a 1-second poll, three total handler attempts, and a 30-second timeout per handler. It processes events sequentially. All handlers whose glob patterns match are invoked; successful handlers can see an event again if another handler fails, so handlers must remain idempotent. These dispatcher retries are immediate, unlike the independent backoff used by sources and the WhatsApp outbox.

publish validates the event, writes it to SQLite, then wakes the dispatcher. Queue states are pending, processing, done, and failed; exhausted attempts park the item as failed. Shutdown stops acceptance, stops monitors and bridges, drains the current dispatch batch, checkpoints, and closes storage within the daemon's bounded shutdown window. systemd readiness and watchdog notifications follow the daemon's lifecycle and heartbeat.

Exact triage policy ordering

The current policy is a pure function; it takes a verdict, controls, meeting state, quiet-hours state, repeated-thread state, and whether the item is mail. A rule description alone does not bypass this final policy.

always permits calls for high and critical urgency. urgent_only permits calls only for critical urgency. mute prevents calls; it does not mean every possible alert is discarded. Gate precedence is DND, then quiet hours, then meeting. A digest created during quiet hours is held until the quiet period ends. WhatsApp notification and reminder handling are separate stages from the spoken-alert flag.

Assistant execution contract

The Sentinel assistant uses the assistant model route through the provider-neutral Message, Chunk, ToolCall, generate, and stream interfaces. The current tool set has ten declarations; the full catalog is below. Incoming-call tools are a restricted subset rather than the same authority as an authenticated dashboard session.

• Model/tool loop cap: 8 steps per turn.
• Model step timeout: 60 seconds. Tool timeout: 15 seconds.
• History character budget: 24,000. Tool result limit: 8,000 characters.
• Dashboard message content: 1–8,000 characters; simultaneous turns in one conversation receive HTTP 409.
• WhatsApp turns wait for the conversation lock and run as supervised tasks with a 180-second turn budget, outside the 30-second event-handler budget.
• set_controls exposes call mode, DND, email/calendar/Jira monitor switches, and an allowlist of telemetry, heartbeat, event retention, and chat retention settings. It does not expose arbitrary vault writes, secrets, or model-route editing.
• create_reminder resolves a supplied ISO datetime in the configured timezone, validates it, and rejects past/unparseable times before calendar creation.
• Gemini tool-call thought signatures are preserved in conversation history so a subsequent request can replay the call correctly.

Source failures and input boundaries

Each source polls on its own configured interval. Insertion into watch_items is the deduplication guard. Temporary failures produce a degraded state and backoff; revoked credentials stop retrying and require reauthorisation. A disabled Drive API is distinguished from expired authorisation. Export-disabled or oversized Drive transcripts are skipped and logged. IMAP drafts use APPEND, and the application has no SMTP sending path.

Only text from an authorised WhatsApp owner reaches conversation handling. Media begins as metadata; download happens after authorisation and size checks. Hidden @lid identities must resolve to a phone number before forwarding. Group and self messages are dropped. Unknown recipients cannot redirect outbound delivery.

Escalation state and idempotency

The initial WhatsApp alert, short-code escalation record, and reply reference are written together. The exact-command parser chooses an explicit code first, then a quoted message reference, then a single open alert. It never guesses between several alerts. Conditional updates make repeated commands safe, and actions are audited with the WhatsApp actor. Acknowledged, dismissed, or snoozed alerts are checked before every phone dial or redial and will not be rung again while resolved or deferred. DND-held alerts also fail the dial guard.

Meeting processing and call authority

Text transcripts are read locally. Audio is uploaded to Gemini's Files API for transcription, and that upload is deleted afterwards. The extractor fences the transcript as untrusted content, forces a record_minutes call, and validates the result. Each pipeline state is persisted; retry resumes work rather than recreating completed steps. Calendar insertions for action items are deduplicated. Failed recordings are retained for a configurable retry period; successful recordings are removed after transcription.

Outbound calls target the configured number; acknowledgements and snoozes can change the corresponding alert. Incoming owner calls provide read-only answers because caller ID can be spoofed. Other callers receive the message-taking persona. A call that fails to dial, answer, establish audio, or connect its voice model returns a reason through the configured WhatsApp channel. Ring timeout must be shorter than voicemail pickup, which otherwise looks like an answered call.

HTTP, events & streaming protocols

Source: readme.md; friday/sentinel/api.py; web.py; voice.py; friday/core/events.py

An event is a small JSON envelope; type is dotted lowercase and payload is free-form:

{"type": "sensor.update", "source": "office-esp32", "payload": {"temp_c": 24.5}, "priority": 0}

Deployment (systemd unit, launchd plist, Tailscale Serve) is documented in deploy/README.md.

Authentication and status codes

/health is public. Node APIs accept a dashboard session or a node token in Authorization: Bearer fn_…. Dashboard-management APIs require a user session. /auth/login creates that session from credentials; it does not require an existing session. Login and all state-changing dashboard endpoints require X-FRIDAY-Client: dashboard; an Origin header, when supplied, must match the expected host. The server does not expose CORS permission for arbitrary origins.

Typical failures are 400 for validation, 401 for missing or invalid identity, 403 for CSRF/origin rejection, 404 for an unknown resource, 409 for a conversation already running a turn or voice session, 413 for oversized input, 429 for login throttling, and 503 when browser voice is disabled.

Meeting routes

The multipart receiver applies minutes.upload_max_mb; this is independent of the event-ingestion JSON size limit. Upload limits and recognised formats are configurable or defined by the minutes transcriber.

Event envelope and acceptance

An event has id, ts, source, type, payload, and priority. The server generates a UUID-hex ID and Unix timestamp when absent. Type must match dotted lowercase components (^[a-z0-9_]+(\.[a-z0-9_]+)+$); source and ID must be nonempty strings; payload must be a JSON-serialisable object; timestamp must be numeric and priority an integer, with booleans rejected for both numeric fields.

POST /events accepts a single envelope or an array of up to 100, with a maximum body of 256 KiB. The full batch is validated before publication. Each event is committed before its ID is acknowledged. HTTP 202 means accepted into the queue, not that downstream work has finished.

{
  "type": "sensor.update",
  "source": "office-sensor",
  "payload": {"temp_c": 24.5},
  "priority": 0
}

{"ids":["server-generated-event-id"]}

Event WebSocket

GET /ws defaults to all events. Send a nonempty list of nonempty glob strings to change the subscription. The server acknowledges the patterns and pushes full event envelopes. This endpoint accepts the token header, session cookie, or a token query parameter; the query-token allowance is specific to this route. Prefer headers where the client supports them.

{"subscribe":["node.*","telemetry.sample","triage.*"]}

{"subscribed":["node.*","telemetry.sample","triage.*"]}

The socket heartbeat is 20 seconds. Fanout allows two seconds per socket send; a slow/dead socket is dropped without failing the persisted event. This is live fanout, not a replay cursor. Use /api/events for historical inspection after reconnecting.

Streaming text chat

Create a conversation with POST /api/chat and an optional title. Send {"content":"What did I miss?"} to /api/chat/{id}/messages. The response is newline-delimited JSON, with application/x-ndjson, Cache-Control: no-store, and X-Accel-Buffering: no. A client must parse complete newline-terminated JSON frames rather than assuming transport chunks match messages.

Frames include text delta, tool start/result records, errors, and a final done. Tool chips and text rendering update incrementally. Conversation and message history persist in SQLite; the configured chat-retention period prunes idle threads. Tool output is data rather than authority to issue further instructions.

Voice WebSocket frames

Connect to /voice/ws?conversation=<id> with a session or node-token header. An absent/unknown conversation ID creates a conversation. A second voice session for the same conversation is rejected with 409. This socket has a 20-second heartbeat and a 4 MiB frame-size ceiling. Binary input is mono PCM16 at 16 kHz; output is mono PCM16 at 24 kHz.

The client stops sending audio when muted. Closing the socket tears down the session. Transcript fragments accumulate by speaker; final fragments and turn completion flush them into the conversation with via=voice. Session duration is audited.

Shared Live engine

friday.core.live.LiveSession owns connection, turn receive loops, GoAway rotation, resumption, typed events, and tool answers. Consumers send audio, video, and text, and read an asynchronous event stream. Events include Connected, Disconnected, AudioOut, TextOut, Transcript, Interrupted, TurnComplete, ToolStarted, ToolFinished, and GoAway. A failing tool still receives a tool response so a turn can close. Resumption handles are held within a process run, not as cross-restart conversations.

The optional AudioTransport interface supplies start/read/write/clear/stop operations. Dashboard voice uses a WebSocket transport, the phone agent uses ALSA, and Desktop manages its own microphone/playback multiplexing. The text-provider abstraction does not make the Gemini Live wire protocol provider-neutral.

Complete tool catalog & parameter schemas

Source: Literal declarations in friday/desktop/hub.py and friday/sentinel/assistant.py

Desktop: 38 tools

Expand a tool to inspect its current declaration and complete parameter schema. These are model-facing contracts from the source; runtime permission gates still apply.

{
  "type": "OBJECT",
  "properties": {
    "command": {
      "type": "STRING",
      "description": "The shell/bash command to run."
    }
  },
  "required": [
    "command"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "script": {
      "type": "STRING",
      "description": "The AppleScript code to run."
    }
  },
  "required": [
    "script"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "display": {
      "type": "STRING",
      "description": "'active' (default, monitor with mouse), 'all' (composite of all monitors), or a display number like '1' or '2'."
    }
  }
}

{
  "type": "OBJECT",
  "properties": {
    "source": {
      "type": "STRING",
      "description": "'auto' (default: phone camera during a remote session, else Mac webcam) or 'mac' to force the Mac's webcam."
    }
  }
}

{
  "type": "OBJECT",
  "properties": {
    "reason": {
      "type": "STRING",
      "description": "The reason why you need to enable the camera feed."
    },
    "source": {
      "type": "STRING",
      "description": "'auto' (default: phone camera during a remote session, else Mac webcam) or 'mac' to force the Mac's webcam."
    }
  },
  "required": [
    "reason"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "reason": {
      "type": "STRING",
      "description": "The reason why you need to enable the screen feed."
    }
  },
  "required": [
    "reason"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "name": {
      "type": "STRING",
      "description": "The name of the person being registered (e.g. 'Vince', 'Anu')."
    }
  },
  "required": [
    "name"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "key": {
      "type": "STRING",
      "description": "The name/category of the fact (e.g. 'user_name', 'favorite_food')."
    },
    "value": {
      "type": "STRING",
      "description": "The detail/fact content to save."
    }
  },
  "required": [
    "key",
    "value"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "name": {
      "type": "STRING",
      "description": "Scene name shown in the workspace, e.g. 'Solar System'."
    },
    "objects_json": {
      "type": "STRING",
      "description": "JSON array of object specs (see tool description)."
    },
    "environment_json": {
      "type": "STRING",
      "description": "Optional JSON: {background '#hex', grid bool, stars bool, ambient 0-3, camera {position [x,y,z], target [x,y,z]}}."
    }
  },
  "required": [
    "name",
    "objects_json"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "scene": {
      "type": "STRING",
      "description": "Scene name or id."
    },
    "operations_json": {
      "type": "STRING",
      "description": "JSON array of operations (see tool description)."
    }
  },
  "required": [
    "scene",
    "operations_json"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "scene": {
      "type": "STRING",
      "description": "Scene name or id."
    }
  },
  "required": [
    "scene"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "scene": {
      "type": "STRING",
      "description": "Scene name or id."
    }
  },
  "required": [
    "scene"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "url": {
      "type": "STRING",
      "description": "The full http(s) URL to fetch."
    }
  },
  "required": [
    "url"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "x": {
      "type": "INTEGER",
      "description": "Normalized horizontal position 0-1000."
    },
    "y": {
      "type": "INTEGER",
      "description": "Normalized vertical position 0-1000."
    },
    "button": {
      "type": "STRING",
      "description": "'left' (default), 'right', or 'middle'."
    },
    "clicks": {
      "type": "INTEGER",
      "description": "1 = single click (default), 2 = double click."
    }
  },
  "required": [
    "x",
    "y"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "text": {
      "type": "STRING",
      "description": "The text to type."
    },
    "press_enter": {
      "type": "BOOLEAN",
      "description": "Press Enter after typing (default false)."
    }
  },
  "required": [
    "text"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "keys": {
      "type": "ARRAY",
      "items": {
        "type": "STRING"
      },
      "description": "Keys pressed together. Modifiers: command, option, ctrl, shift."
    }
  },
  "required": [
    "keys"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "amount": {
      "type": "INTEGER",
      "description": "Scroll units. e.g. -5 scrolls down a bit."
    },
    "x": {
      "type": "INTEGER",
      "description": "Optional normalized x to hover before scrolling."
    },
    "y": {
      "type": "INTEGER",
      "description": "Optional normalized y to hover before scrolling."
    }
  },
  "required": [
    "amount"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "x1": {
      "type": "INTEGER",
      "description": "Start normalized x."
    },
    "y1": {
      "type": "INTEGER",
      "description": "Start normalized y."
    },
    "x2": {
      "type": "INTEGER",
      "description": "End normalized x."
    },
    "y2": {
      "type": "INTEGER",
      "description": "End normalized y."
    }
  },
  "required": [
    "x1",
    "y1",
    "x2",
    "y2"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "days_ahead": {
      "type": "INTEGER",
      "description": "How many days ahead to include (default 7)."
    },
    "days_back": {
      "type": "INTEGER",
      "description": "How many past days to include (default 0)."
    }
  }
}

{
  "type": "OBJECT",
  "properties": {
    "title": {
      "type": "STRING",
      "description": "Event title."
    },
    "start_iso": {
      "type": "STRING",
      "description": "Start time as 'YYYY-MM-DD HH:MM' (24h, local time)."
    },
    "duration_minutes": {
      "type": "INTEGER",
      "description": "Duration in minutes (default 60)."
    },
    "notes": {
      "type": "STRING",
      "description": "Optional notes/description."
    }
  },
  "required": [
    "title",
    "start_iso"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "count": {
      "type": "INTEGER",
      "description": "Number of recent emails to fetch (default 10, max 25)."
    }
  }
}

{
  "type": "OBJECT",
  "properties": {
    "query": {
      "type": "STRING",
      "description": "Text to match against sender or subject."
    },
    "count": {
      "type": "INTEGER",
      "description": "Max results (default 8)."
    }
  },
  "required": [
    "query"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "target": {
      "type": "STRING",
      "description": "'spatial' for the SVE scene, or words naming a generated model, e.g. 'jet engine'."
    }
  },
  "required": [
    "target"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "query": {
      "type": "STRING",
      "description": "Words identifying the saved model, e.g. 'jet engine'. Omit for the most recent."
    },
    "widget_id": {
      "type": "STRING",
      "description": "Optional stable card id."
    }
  }
}

{
  "type": "OBJECT",
  "properties": {
    "prompt": {
      "type": "STRING",
      "description": "Vivid, concrete description of the single object, including material and finish. E.g. 'vintage brass astrolabe with engraved rings, studio lighting'. Describe one object, not a scene or an arrangement."
    },
    "widget_id": {
      "type": "STRING",
      "description": "Stable id for the subject, e.g. 'drone_model'. Reuse it to replace that card."
    },
    "title": {
      "type": "STRING",
      "description": "Card header, e.g. 'Recon Drone'."
    }
  },
  "required": [
    "prompt"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "widget_id": {
      "type": "STRING",
      "description": "Stable id for the subject, e.g. 'goog_quote' or 'sysmon'."
    },
    "title": {
      "type": "STRING",
      "description": "Card header, e.g. 'Alphabet Inc.'."
    },
    "query_context": {
      "type": "STRING",
      "description": "Everything the generator needs, in detail \u2014 it cannot see the conversation. Name the subject, every figure or section the card should carry, and any values you already know. E.g. 'Live GOOG quote: price, day change and percent, open/high/low, market cap, P/E, 52-week range, intraday trend chart, one-line sentiment.' Thin context makes a thin card."
    }
  },
  "required": [
    "widget_id",
    "title",
    "query_context"
  ]
}

{
  "type": "OBJECT",
  "properties": {
    "widget_id": {
      "type": "STRING",
      "description": "The widget to remove."
    }
  },
  "required": [
    "widget_id"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

{
  "type": "OBJECT",
  "properties": {
    "goal": {
      "type": "STRING",
      "description": "The complete task, self-contained. The agent runs without further input and cannot ask questions, so include every detail it needs."
    },
    "tier": {
      "type": "STRING",
      "enum": [
        "os",
        "spatial"
      ],
      "description": "'os' for macOS automation, shell, AppleScript and GUI control. 'spatial' for building or editing 3D SVE scenes. Default 'os'."
    }
  },
  "required": [
    "goal"
  ]
}

{
  "type": "OBJECT",
  "properties": {}
}

Sentinel assistant: 10 tools

Expand a tool to inspect its current declaration and complete parameter schema. These are model-facing contracts from the source; runtime permission gates still apply.

{
  "type": "object",
  "properties": {}
}

{
  "type": "object",
  "properties": {
    "node_id": {
      "type": "string",
      "description": "Only this node; omit for all nodes."
    }
  }
}

{
  "type": "object",
  "properties": {}
}

{
  "type": "object",
  "properties": {
    "type_glob": {
      "type": "string",
      "description": "Event type glob such as node.* (default *)."
    },
    "limit": {
      "type": "integer",
      "description": "1-50, default 20."
    }
  }
}

{
  "type": "object",
  "properties": {
    "limit": {
      "type": "integer",
      "description": "1-50, default 20."
    }
  }
}

{
  "type": "object",
  "properties": {}
}

{
  "type": "object",
  "properties": {
    "urgency": {
      "type": "string",
      "description": "critical, high, normal or low."
    },
    "limit": {
      "type": "integer",
      "description": "1-50, default 20."
    }
  }
}

{
  "type": "object",
  "properties": {}
}

{
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "What to be reminded of."
    },
    "when_iso": {
      "type": "string",
      "description": "ISO 8601 local datetime, e.g. 2026-10-01T10:00:00. Resolve relative phrases against the current time given in your instructions."
    },
    "duration_minutes": {
      "type": "integer",
      "description": "1-1440, default 30."
    },
    "notes": {
      "type": "string",
      "description": "Optional detail for the event body."
    }
  },
  "required": [
    "title",
    "when_iso"
  ]
}

{
  "type": "object",
  "properties": {
    "call_mode": {
      "type": "string",
      "description": "always, urgent_only or mute."
    },
    "dnd": {
      "type": "boolean",
      "description": "Do not disturb on/off."
    },
    "monitors": {
      "type": "object",
      "description": "Switches: email, calendar, jira \u2192 boolean.",
      "properties": {
        "email": {
          "type": "boolean"
        },
        "calendar": {
          "type": "boolean"
        },
        "jira": {
          "type": "boolean"
        }
      }
    },
    "timing": {
      "type": "object",
      "description": "Operational parameters, applied live without a restart.",
      "properties": {
        "telemetry_interval_s": {
          "type": "number",
          "description": "1-3600 seconds."
        },
        "heartbeat_interval_s": {
          "type": "number",
          "description": "1-3600 seconds."
        },
        "retention_days": {
          "type": "integer",
          "description": "1-365 days of events and telemetry."
        },
        "chat_retention_days": {
          "type": "integer",
          "description": "1-3650 days of conversations."
        }
      }
    }
  }
}

The incoming-call owner persona uses a restricted read-only subset; the message-taking persona has its own narrow tool. The portfolio website’s separate FRIDAY guide only exposes link navigation and does not inherit these project tools.

Configuration, routes & every setting

Source: friday/core/config.py; friday/sentinel/settings_registry.py; runtime_config.py

Bootstrap and precedence

Process configuration is loaded once from the repository .env, overlaid by the process environment. For dynamic keys the runtime then resolves vault → valid legacy environment value → registry default. An existing undecryptable vault value is reported as such instead of silently pretending it came from the environment.

Secrets are entered in Settings and AES-GCM encrypted. Other settings are stored as JSON; not every row is encrypted. Desktop pulls only settings declared for the desktop scope, overlays known fields, and stores a last-good cache with mode 0600. Changing bootstrap values or the bridge class list requires a process restart. Dashboard dynamic controls apply live; Desktop's scoped configuration is pulled at boot.

An explicit FRIDAY_LLM_LIVE takes precedence over the legacy GEMINI_MODEL alias. The route defaults below are exactly those declared by this project checkout, not a promise that a provider/model is available to every API account.

Validation and save behavior

The registry defines string, URL, enum, boolean, integer, float, and string-list controls. Lists may be provided as comma-separated text. Range validators constrain numeric values. Time validators accept HH:MM strings; quiet hours may cross midnight. WhatsApp numbers are normalised to a phone-number JID. Phone-call destinations are normalised to international dial strings. adb templates validate allowed placeholders and require {number} for a nonempty dial template.

Settings edits are held until Save. The server validates the batch and returns per-field errors; the UI opens the relevant Advanced group and focuses the problem. Null values in the Settings API request remove overrides rather than storing the string "null". Controls use the same declared keys but apply immediately. Secret controls show only masked status; the form clears a typed secret after saving. Model routes and secrets are outside the chat assistant's control-write allowlist.

93 declared dynamic settings. This inventory is generated from the registry, including defaults, types, legacy environment aliases, scope, and validation. Unset means the registry default is null. Values shown here are declaration defaults, never values from a running installation.

llm

desktop

sources

policy

minutes

whatsapp

telephony

voice

controls

sentinel

Persistence, migrations & recovery

Source: friday/core/storage.py; friday/core/vault.py; friday/sentinel/bus.py

Store and queue mechanics

The synchronous Store owns one SQLite connection. AsyncStore serialises calls onto a dedicated worker thread. SQLite 3.35+ is required for UPDATE…RETURNING. WAL journaling and FULL synchronisation are the defaults; NORMAL reduces synchronisation work but may lose recent commits after power loss. This is a durability setting, not a replacement for backups.

Forward-only schema migrations run statements inside a transaction with the schema-version update. Interrupted processing is returned to pending on startup. The event queue orders eligible work by priority and time, increments claim attempts, and persists errors. Event publication, node heartbeat upserts, telemetry, configuration, sessions, audit, conversations, watched items, outbox, escalation, meetings, and calls all use the same persistent store.

Data families

Desktop state is also local: memory facts, biometric profiles, persisted scene graphs, generated asset indexes and GLBs, and the config cache live beneath the configured data directory. Do not assume every Desktop artifact is a row in sentinel.db.

Vault cryptography and session storage

The master key decodes to at least 32 bytes. HKDF-SHA256 derives a 32-byte AES-GCM key with the friday-vault-v1 context. Encryption uses a new 12-byte nonce; the setting key is associated data. Stored ciphertext tokens are versioned (v1:nonce:ciphertext). Moving a ciphertext to another setting name fails authentication. Losing or replacing the master key makes existing secrets unreadable.

Passwords use scrypt with N=32768, r=8, p=1, a random 16-byte salt, and a 32-byte derived key. Verification uses constant-time comparison; unknown users still incur a hash verification. Session and node-token plaintext is random and only its SHA-256 hash is stored. The session cookie is friday_session, HttpOnly and SameSite=Lax, Secure over correctly detected HTTPS. Session TTL defaults to 30 days. Node tokens use the fn_ prefix and can be revoked independently.

Backup and recovery

For a simple consistent backup, stop Sentinel cleanly before copying its persistent data and store the master key separately in a secure backup. For a live backup, use SQLite's backup mechanism rather than copying only sentinel.db while WAL writes continue. Include the WhatsApp sidecar's pairing/session state if you need to preserve that pairing, and the Desktop data you want to retain. Protect plaintext config caches and biometric files too.

Restore the database with its original master key. Do not regenerate the key to repair an undecryptable vault. Confirm health, node authentication, source credentials, and channel pairing before re-enabling unattended alerts. Schema migrations are forward-only; keep a pre-upgrade backup when changing versions.

Exact migration definitions

The following SQL is generated from the application's migration declarations. It documents schema evolution; do not manually execute these statements against a running installation. FRIDAY applies them itself.

CREATE TABLE kv (  key TEXT PRIMARY KEY, value TEXT NOT NULL, updated_at REAL NOT NULL);

CREATE TABLE heartbeats (  node_id TEXT PRIMARY KEY, last_seen REAL NOT NULL,  status TEXT NOT NULL, meta TEXT NOT NULL);

CREATE TABLE events (  id TEXT PRIMARY KEY, ts REAL NOT NULL, source TEXT NOT NULL,  type TEXT NOT NULL, payload TEXT NOT NULL,  priority INTEGER NOT NULL DEFAULT 0,  status TEXT NOT NULL DEFAULT 'pending',  attempts INTEGER NOT NULL DEFAULT 0,  created_at REAL NOT NULL, claimed_at REAL, processed_at REAL, error TEXT);

CREATE INDEX events_status_ts ON events(status, priority DESC, ts);

CREATE TABLE telemetry (  ts REAL NOT NULL, node_id TEXT NOT NULL, snapshot TEXT NOT NULL);

CREATE INDEX telemetry_node_ts ON telemetry(node_id, ts DESC);

CREATE TABLE settings (  key TEXT PRIMARY KEY, value TEXT NOT NULL, secret INTEGER NOT NULL DEFAULT 0,  updated_at REAL NOT NULL, updated_by TEXT NOT NULL);

CREATE TABLE users (  username TEXT PRIMARY KEY, password_hash TEXT NOT NULL,  created_at REAL NOT NULL, password_changed_at REAL NOT NULL);

CREATE TABLE sessions (  id TEXT PRIMARY KEY, username TEXT NOT NULL, created_at REAL NOT NULL,  expires_at REAL NOT NULL, last_seen REAL NOT NULL, user_agent TEXT, ip TEXT);

CREATE INDEX sessions_expires ON sessions(expires_at);

CREATE TABLE node_tokens (  id TEXT PRIMARY KEY, name TEXT NOT NULL, token_hash TEXT NOT NULL UNIQUE,  created_at REAL NOT NULL, last_used REAL, revoked_at REAL);

CREATE TABLE audit (  id INTEGER PRIMARY KEY AUTOINCREMENT, ts REAL NOT NULL, actor TEXT NOT NULL,  action TEXT NOT NULL, target TEXT, detail TEXT NOT NULL);

CREATE INDEX audit_ts ON audit(ts DESC);

CREATE TABLE conversations (  id TEXT PRIMARY KEY, title TEXT NOT NULL,  created_at REAL NOT NULL, updated_at REAL NOT NULL);

CREATE INDEX conversations_updated ON conversations(updated_at DESC);

CREATE TABLE messages (  id INTEGER PRIMARY KEY AUTOINCREMENT, conversation_id TEXT NOT NULL,  seq INTEGER NOT NULL, role TEXT NOT NULL, content TEXT NOT NULL,  tool_name TEXT, tool_args TEXT, tool_result TEXT,  status TEXT NOT NULL DEFAULT 'complete', ts REAL NOT NULL);

CREATE UNIQUE INDEX messages_conv_seq ON messages(conversation_id, seq);

ALTER TABLE messages ADD COLUMN via TEXT NOT NULL DEFAULT 'text';

CREATE TABLE watch_items (  id TEXT PRIMARY KEY, source TEXT NOT NULL, external_id TEXT NOT NULL,  title TEXT NOT NULL, snippet TEXT NOT NULL, who TEXT NOT NULL, url TEXT NOT NULL,  ts REAL NOT NULL, first_seen REAL NOT NULL, meta TEXT NOT NULL);

CREATE INDEX watch_items_seen ON watch_items(first_seen DESC);

CREATE INDEX watch_items_source_ts ON watch_items(source, ts DESC);

ALTER TABLE watch_items ADD COLUMN urgency TEXT NOT NULL DEFAULT '';

ALTER TABLE watch_items ADD COLUMN category TEXT NOT NULL DEFAULT '';

ALTER TABLE watch_items ADD COLUMN reason TEXT NOT NULL DEFAULT '';

ALTER TABLE watch_items ADD COLUMN scored_by TEXT NOT NULL DEFAULT '';

ALTER TABLE watch_items ADD COLUMN decision TEXT NOT NULL DEFAULT '';

ALTER TABLE watch_items ADD COLUMN suppressed_by TEXT NOT NULL DEFAULT '';

ALTER TABLE watch_items ADD COLUMN digest_state TEXT NOT NULL DEFAULT '';

ALTER TABLE watch_items ADD COLUMN thread_key TEXT NOT NULL DEFAULT '';

CREATE INDEX watch_items_digest ON watch_items(digest_state, first_seen DESC);

CREATE INDEX watch_items_thread ON watch_items(thread_key, first_seen DESC);

CREATE TABLE whatsapp_outbox (  id TEXT PRIMARY KEY, body TEXT NOT NULL, priority INTEGER NOT NULL DEFAULT 0,  item_id TEXT NOT NULL DEFAULT '', state TEXT NOT NULL DEFAULT 'pending',  attempts INTEGER NOT NULL DEFAULT 0, next_attempt_at REAL NOT NULL,  created_at REAL NOT NULL, sent_at REAL NOT NULL DEFAULT 0,  last_error TEXT NOT NULL DEFAULT '', message_id TEXT NOT NULL DEFAULT '');

CREATE INDEX whatsapp_outbox_due ON whatsapp_outbox(state, priority DESC, next_attempt_at);

CREATE TABLE whatsapp_inbound (id TEXT PRIMARY KEY, ts REAL NOT NULL);

CREATE INDEX whatsapp_inbound_ts ON whatsapp_inbound(ts DESC);

CREATE TABLE escalations (  id TEXT PRIMARY KEY, code INTEGER NOT NULL, state TEXT NOT NULL DEFAULT 'open',  urgency TEXT NOT NULL, title TEXT NOT NULL, source TEXT NOT NULL DEFAULT '',  sender TEXT NOT NULL DEFAULT '', category TEXT NOT NULL DEFAULT '',  scored_by TEXT NOT NULL DEFAULT '', thread_key TEXT NOT NULL DEFAULT '',  created_at REAL NOT NULL, updated_at REAL NOT NULL,  wake_at REAL NOT NULL DEFAULT 0, realerts INTEGER NOT NULL DEFAULT 0,  acted_by TEXT NOT NULL DEFAULT '', acted_via TEXT NOT NULL DEFAULT '',  held INTEGER NOT NULL DEFAULT 0);

CREATE UNIQUE INDEX escalations_live_code ON escalations(code) WHERE state IN ('open', 'snoozed');

CREATE INDEX escalations_due ON escalations(state, wake_at);

CREATE TABLE whatsapp_refs (  outbox_id TEXT PRIMARY KEY, kind TEXT NOT NULL, ref_id TEXT NOT NULL,  created_at REAL NOT NULL);

CREATE INDEX whatsapp_outbox_message ON whatsapp_outbox(message_id);

CREATE TABLE triage_feedback (  id INTEGER PRIMARY KEY AUTOINCREMENT, ts REAL NOT NULL, item_id TEXT NOT NULL,  label TEXT NOT NULL, source TEXT NOT NULL, sender TEXT NOT NULL,  thread_key TEXT NOT NULL, category TEXT NOT NULL, urgency TEXT NOT NULL,  scored_by TEXT NOT NULL, actor TEXT NOT NULL);

CREATE INDEX triage_feedback_ts ON triage_feedback(ts DESC);

CREATE TABLE meetings (  id TEXT PRIMARY KEY, code INTEGER, source TEXT NOT NULL, external_id TEXT NOT NULL,  title TEXT NOT NULL, occurred_at REAL NOT NULL, duration_s REAL NOT NULL DEFAULT 0,  state TEXT NOT NULL DEFAULT 'received', attempts INTEGER NOT NULL DEFAULT 0,  error TEXT NOT NULL DEFAULT '', media_path TEXT NOT NULL DEFAULT '',  mime TEXT NOT NULL DEFAULT '', transcript TEXT NOT NULL DEFAULT '',  minutes TEXT NOT NULL DEFAULT '', created_at REAL NOT NULL, updated_at REAL NOT NULL,  delivered_at REAL NOT NULL DEFAULT 0, next_at REAL NOT NULL DEFAULT 0,  UNIQUE (source, external_id));

CREATE INDEX meetings_state ON meetings(state, next_at);

CREATE UNIQUE INDEX meetings_live_code ON meetings(code) WHERE code IS NOT NULL;

CREATE TABLE meeting_actions (  meeting_id TEXT NOT NULL, idx INTEGER NOT NULL, text TEXT NOT NULL,  owner TEXT NOT NULL DEFAULT '', due_at REAL NOT NULL DEFAULT 0,  mine INTEGER NOT NULL DEFAULT 0, calendar_event_id TEXT NOT NULL DEFAULT '',  PRIMARY KEY (meeting_id, idx));

CREATE TABLE calls (  id TEXT PRIMARY KEY, direction TEXT NOT NULL, number TEXT NOT NULL DEFAULT '',  persona TEXT NOT NULL DEFAULT '', item_id TEXT NOT NULL DEFAULT '',  state TEXT NOT NULL, outcome TEXT NOT NULL DEFAULT '', started_at REAL NOT NULL,  answered_at REAL NOT NULL DEFAULT 0, ended_at REAL NOT NULL DEFAULT 0,  meeting_id TEXT NOT NULL DEFAULT '');

CLI, development & troubleshooting

Source: deploy/README.md; readme.md; friday/sentinel/cli.py; tests/

Command-line entry points

The installed entry points are friday-desktop and friday-sentinel. Their module equivalents are python -m friday.desktop and python -m friday.sentinel; use the virtual environment's Python. Desktop can also run its browser-facing hub with python -m friday.desktop.hub.

Bootstrap commands use the SQLite store directly and audit as the CLI actor. A clean shutdown, persistent data directory, and matching master key are required for upgrades and service relocation. Existing installation commands are in the setup and deployment chapters above.

Development and dependencies

Install .[desktop,dev] on the Mac or .[sentinel,dev] on the server for test tooling. Core dependencies are python-dotenv, psutil, aiohttp, google-genai, and cryptography. Desktop adds the audio, image, macOS bridge, GUI, and websocket dependencies declared in pyproject.toml. The sentinel extra deliberately excludes macOS-only libraries.

The Sentinel dashboard is vanilla ES modules with a committed Tailwind stylesheet. After changing dashboard class names, run deploy/build_css.sh; it downloads the pinned standalone Tailwind binary into the local cache on first use. This is the application dashboard build, distinct from this static documentation website, which needs no frontend build.

deploy/build_sidecar.sh builds the Go WhatsApp sidecar for the Mac and Raspberry Pi targets. The destination Pi does not need the Go toolchain. Rebuild and deploy the current sidecar when using quoted alerts, media ingestion, or identity-resolution changes; an old binary may lack the corresponding protocol features.

Troubleshooting

Limits and extension points

The event dispatcher is sequential; slow handlers can delay later events. Text assistant and Desktop agents have finite step budgets. WhatsApp is unofficial and subject to account restrictions. Telephony depends on external hardware, adb permissions, and carrier behavior. SVE uses a single active viewport and projected hand input; VR/AR, per-object hand rotation/scale, and mouse object dragging are not implemented by the documented interface.

A bridge implements start(bus, ctx) and stop(), publishes inbound events, and subscribes to command patterns. Add the class to FRIDAY_BRIDGES and restart. A new monitor uses a narrowly scoped source capability and emits normalised watch items. New SVE primitives require both Python validation and renderer support; external renderers consume the same scene/op messages and emit user actions. Extend the settings registry with types, validation, labels, units, grouping, and node scopes rather than adding unrelated hard-coded configuration paths.

Repository map

friday-ai-assistant/
├── pyproject.toml                 # one distribution "friday"; extras: desktop, sentinel, dev
├── .env.template                  # host bootstrap (master key, bind, data dir) + legacy values
├── SVE.md                         # Spatial Visualization Engine specification
├── friday/
│   ├── core/                      # shared, platform-neutral
│   │   ├── config.py              # Settings from env + .env; per-role LLM routes
│   │   ├── platform.py            # OS/arch/Pi/distro probes
│   │   ├── storage.py             # SQLite/WAL Store + AsyncStore: kv, heartbeats, events, telemetry, settings, users, sessions, tokens, audit
│   │   ├── vault.py               # AES-GCM secrets under HKDF(FRIDAY_MASTER_KEY); setting key as AAD
│   │   ├── events.py              # Event envelope, Heartbeat, TelemetrySnapshot schemas
│   │   ├── telemetry.py           # sensor-tolerant collector
│   │   ├── logsetup.py            # stdout logging, journald-aware
│   │   ├── live/                  # Gemini Live: session, events, audio transports
│   │   └── llm/                   # routing, provider protocol + conversation API (Message, Chunk, stream), Gemini adapter
│   ├── desktop/                   # the macOS node
│   │   ├── hub.py                 # Async hub: audio, WebSocket, Live session, widget deck
│   │   ├── app.py                 # PyWebView desktop shell + process lifecycle
│   │   ├── agents.py              # Background agent tiers (os / spatial) and their tool loop
│   │   ├── widget_generator.py    # Card HTML synthesis + output sanitiser
│   │   ├── asset_generator.py     # Tripo3D text-to-3D
│   │   ├── sentinel_client.py     # heartbeat poster + config pull from the sentinel
│   │   ├── alerts.py              # /ws subscriber: alert cards and the HUD speech queue
│   │   ├── config_pull.py         # overlay sentinel config onto Settings; 0600 cache; .env fallback
│   │   ├── sentry_vision.py       # Quartz multi-monitor capture & coordinate tracking
│   │   ├── sentry_action.py       # CGEvent mouse/keyboard automation & click mapping
│   │   ├── sentry_exec.py         # Shell & osascript execution
│   │   ├── sentry_recognition.py  # Face (YuNet+SFace) & voice (MFCC) biometrics
│   │   ├── sentry_scene.py        # SVE scene graph manager, validation, persistence
│   │   ├── sentry_personal.py     # EventKit calendars & Apple Mail
│   │   ├── sentry_web.py          # Async webpage reader
│   │   ├── models/                # YuNet + SFace ONNX weights
│   │   └── web_gui/               # index.html, style.css, app.js, orb.js, sve.js, gestures.js, vendor/
│   └── sentinel/                  # the headless node
│       ├── daemon.py              # boot / supervise / ordered shutdown, sd_notify
│       ├── bus.py                 # durable dispatcher over the store
│       ├── handlers.py            # Handler protocol + heartbeat / telemetry / log handlers
│       ├── api.py                 # aiohttp: /health /telemetry /nodes /events /ws
│       ├── web.py                 # /auth/* /api/settings /api/tokens /api/audit /api/events /api/telemetry /api/watch /api/chat /config + shell
│       ├── audit.py               # audit row + audit.entry event
│       ├── voice.py               # /voice/ws bridge: Live session ↔ browser audio
│       ├── assistant.py           # tool set and streaming turn loop
│       ├── services.py            # Services bundle handed to every handler
│       ├── principals.py          # who is asking: session cookie (user) or bearer token (node)
│       ├── auth.py                # scrypt passwords, hashed sessions, node tokens, login lockout
│       ├── settings_registry.py   # declared shape of every dynamic setting
│       ├── runtime_config.py      # vault → legacy .env → default; publishes config.changed
│       ├── cli.py                 # friday-sentinel: run, keygen, user set-password, token create/list/revoke, google-auth
│       ├── dashboard/             # index.html, app.js, api.js, socket.js, ui.js, md.js, fields.js, form.js, chat.js, views/ (incl. voice.js), tailwind.{src.css,config.js,css}
│       ├── monitors.py            # telemetry sampler, self-heartbeat, housekeeping
│       ├── sources/               # least-privilege capabilities: gmail, imap, calendar, jira
│       ├── watch.py               # Monitor protocol, WatchRunner, the three monitors
│       ├── triage/                # rules, model, policy, handler, drafts, digest
│       ├── bridges/whatsapp/      # jid, protocol, sidecar, outbox sender, inbound gate, turns
│       └── bridges/               # Bridge protocol + FRIDAY_BRIDGES loader
│   └── webassets/                 # shared browser assets: orb.js + three.module.min.js (served at /shared)
├── sidecar/whatsapp/              # Go: the only process that speaks WhatsApp
├── tests/                         # pytest: core, sentinel, desktop import smoke, boundary guard
├── deploy/                        # systemd unit, launchd plist, setup_remote.sh, build_css.sh, README
├── docs/superpowers/              # design spec and implementation plan
└── data/                          # ALL runtime state, git-ignored (FRIDAY_DATA_DIR)

Generated at runtime, not in the repo. Everything below lives under data/ (or FRIDAY_DATA_DIR), is git-ignored, and is created on first run:

Test coverage

.venv/bin/pytest

core and sentinel are covered end to end: SQLite pragmas, crash recovery and the v1 → v2 migration, the event queue's claim/retry/park lifecycle, every telemetry probe with its sensor missing, the vault (wrong key, wrong setting key, tampered ciphertext), settings validation and vault → env → default precedence, scrypt/session/token/lockout logic, every dashboard route (CSRF header, Origin check, cookie flags, masked secrets, audit trail), the CLI, the desktop's config pull with cache and .env fallback, the daemon's full boot → SIGTERM → clean-exit path, the assistant loop against a scripted provider (tool round-trips, step cap, timeouts, prompt-injection, disconnects), NDJSON streaming through the aiohttp test client, a dashboard build check that fails when tailwind.css is stale or a module does not parse, the dashboard's modules driven in node (dirty tracking and rebasing, controls by type, once-per-frame streaming and follow-the-tail, markdown tables over every prefix of a streamed reply, the orb re-attaching to a new stage and never mounting after the page is left), the Live engine against a scripted fake client (turn loop, interruption, tool answers including a raising tool, GoAway rotation, resume-handle rules, backoff), the voice bridge end to end (auth, frame routing, transcript persistence), the source capabilities against fake Google, Jira and IMAP servers (HTML-only mail, RFC 2047 headers, all-day events, oversized bodies), the two send guardrails, the watch runner's dedupe, switch, backoff and degrade/recover paths, the rule table and its ordering promises, the forced function call with every unusable answer it can return, the policy engine over every combination of urgency, call mode, DND, quiet hours and meeting, quiet-hours spans that cross midnight, a replayed item proving one call and one draft, the digest schedule across a restart, the HUD speech queue waiting for a turn to finish, the outbox's dedupe, ordering, backoff and expiry, the line codec against blank, oversized and non-object lines, a real child process that crashes, hangs, prints garbage and holds a lock, the pacing under a fake clock, the guarantee that a payload carrying its own recipient still goes to the configured number, the sender resolver over hidden @lid addresses with and without a mapping, the gate's authorisation and message-id deduplication, two messages proving turns run sequentially, the markdown translation table, and create_reminder refusing a past or unparseable time. tests/test_boundaries.py imports every core/sentinel module in a subprocess where Quartz, pyaudio, cv2, webview, EventKit, numpy and friends are blocked — the guarantee that the sentinel really does run on a headless Linux box. Desktop modules get an import smoke that is skipped where the desktop extra is absent.

The actual interfaces.

These development screenshots show the Desktop workspace and Sentinel dashboard. Displayed records and telemetry are captured examples, not live information.

Sentinel dashboard

A LOOK INSIDE SENTINEL

OverviewActivityAssistant

↗ Expand screenshot

Desktop workspace

FRIDAY / SPATIAL WORKSPACEmacOS

↗ Expand screenshot

01 / SPATIALIdeas with another dimension.Persistent 3D scenes, generated models, and local hand-gesture controls.02 / WIDGETSThe detail belongs on screen.Live data cards, charts, and visual summaries built in the background.03 / PRESENCEA conversation that flows.Full-duplex Gemini Live voice. Natural interruption. An orb that reacts.