FRIDAY Spatial OS v2.4
Private Preview
Gemini Live Powered Zero-Latency Voice OS Native macOS

Speech is for intent.
The screen is for substance.

Project FRIDAY is a personal multimodal operating system for macOS. You speak; it answers in under ten words and projects rich, live visual substance on screen: interactive 3D spatial scenes, asynchronous SVG charts, and deep desktop agency.

Explore 3D Engine Quickstart Guide
🔒 Codebase Status: Private Preview (Open-source release coming soon)
HOLOGRAPHIC PRESENCE 60 FPS GLSL SHADER
IDLE (CONNECTED)
< 100 ms Audio Latency Gemini Live WebSocket
38 Tools Native macOS Agency Quartz, AX & CGEvent
60 FPS Spatial 3D Engine Incremental SVE Deltas
Zero-Trust Remote Security Tailscale Mesh + Approvals
Core Foundations

What is Project FRIDAY?

Most voice assistants trap you in endless spoken monologues. FRIDAY redefines the human-computer interface by keeping conversational speech under ten words while orchestrating a dynamic visual workspace.

🎙️

Gemini Live Voice Streaming

Full-duplex conversational audio over Google Gemini Live WebSocket API (`gemini-3.1-flash-live-preview`). Streams 16 kHz PCM microphone capture and plays back 24 kHz Aoede voice with natural barge-in interruptions.

  • Sub-second turn-taking latency
  • Instant barge-in queue purging
  • Natural articulate feminine Aoede persona
🔮

Ambient Holographic Presence

A Three.js plasma core that *is* the status display. Strict color-encoded states (Idle cyan, Listening blue, Thinking amber, Speaking emerald) with zero distracting UI chrome, gliding smoothly into spatial mode.

  • Custom Simplex 3D noise shaders
  • Additive fresnel halo & outer bloom
  • 1,200 dynamic drifting orbital particles
🧩

Async Widget Deck

Asking for data mounts a shimmering skeleton card instantly. An asynchronous background agent synthesizes verified HTML — hero figures, inline SVG charts, metric matrices — and hydrates the card in seconds.

  • Zero voice latency on complex layout
  • Strict tokenized design system (`.hud-*`)
  • Verified server-side image proxying
🌐

Spatial Visualization Engine

Live, persistent, interactive 3D scene graphs in Three.js with object-level delta updates (`update_3d_scene`). Plus markerless hand tracking via local MediaPipe HandLandmarker WASM and Tripo3D text-to-3D asset generation.

  • Object-level delta protocol (never full rebuilds)
  • Local point, pinch-to-orbit, two-hand zoom
  • Generative textured PBR 3D asset caching
🖥️

Native macOS Desktop Agency

Hardware-level input automation via Quartz CoreGraphics (`CGEventPost`), multi-monitor coordinate projection, Accessibility tree inspection (`AXUIElement`), EventKit calendars, Apple Mail, and shell execution.

  • Quartz multi-monitor display enumeration
  • Local ONNX face & MFCC voice biometrics
  • Tailscale zero-trust 45s remote approval gate
🤖

Tiered Background Agents

Gemini Live stays 100% responsive for voice turns by delegating heavy multi-step automation to specialized background models: Gemini 3.1 Pro for desktop workflows and Gemini 3.7 Flash for spatial scene planning.

  • Non-blocking background delegation loop
  • Live agent telemetry chips with progress spinners
  • Graceful cancellation and rollback
Under the Hood

System Architecture

How FRIDAY orchestrates low-latency voice, asynchronous widget generation, spatial 3D rendering, and native OS automation in a unified asynchronous event loop.

1. Perception & Senses
Audio PipelinePyAudio
16 kHz 16-bit mono mic input, rolling voice enrollment buffer.
Hand TrackingWASM
Local MediaPipe HandLandmarker running at 30+ FPS in-browser.
Multi-MonitorQuartz
CGDisplay capture, coordinate projection, frontmost app tracking.
BiometricsONNX
OpenCV YuNet face detection + SFace 128-d cosine matching.
2. Friday Engine Hub
Core Event Loopfriday_hub.py
Asyncio runtime managing live sockets, queues, and turns.
WebSocket Gateway:8765
Bidirectional event bus for audio, widgets, SVE ops, and telemetry.
Audio Playback24 kHz
High-fidelity Aoede output with instant barge-in queue purging.
3. Gemini Intelligence
Live Voice Tier3.1-flash
Sub-second reasoning, conversational speech, native tool decisions.
Widget Generator3.1-pro
Synthesizes token-constrained SVG charts and metric cards off-audio.
Desktop OS Agent3.1-pro
Autonomous multi-step execution across macOS apps.
Spatial 3D Planner3.7-flash
Generates renderer-neutral JSON scene graphs & Tripo assets.
4. Execution Engines
SVE Scene Managersentry_scene
Delta diffing, persistence (friday_scenes.json), selection state.
OS Automationsentry_action
CGEventPost mouse/key injection, AXUIElement accessibility.
Shell & AppleScriptsentry_exec
Subprocess execution with 45s human-in-the-loop approval gate.
5. Client Surfaces
Spatial Web GUIweb_gui
Zero-chrome spatial HUD, Holographic Orb, Widget Deck, 3D tab.
Desktop ShellPyWebView
Native macOS window with persistent camera/mic authorizations.
Remote MobileTailscale
Encrypted HTTPS mesh routing phone mic/camera to desktop hub.

Life of a Query: Zero Voice Latency

01
Intent Spoken
User says "Show Alphabet stock and compare with market trends." Streamed at 16 kHz PCM over Gemini Live WebSocket.
02
Instant Skeleton
Live engine responds concisely in 6 words and fires create_skeleton_widget. Shimmering placeholder mounts in 0 ms.
03
Async Card Synthesis
Background agent fetches market data, generates tokenized SVG gradient chart, sanitizes markup, and hydrates card in place.
04
Workspace Evolution
Orb glides smoothly to the 260 px left rail. The interactive card settles into the deck with zero stutter on the voice audio.
Spatial Visualization Engine

Interactive 3D Spatial Canvas

SVE renders live, persistent 3D scene graphs in Three.js. Rather than regenerating scenes, FRIDAY broadcasts incremental delta operations—highlighting, exploding, recoloring, or transforming objects on the fly.

Scene: friday_drone_hex01 (6 objects, 4 animations)
DRAG TO ORBIT · SCROLL TO ZOOM

Object-Level Delta Protocol

Spoken commands like "Highlight the rotors and explode the hull" are converted into JSON delta operations:

{ "action": "highlight", "id": "rotor_alpha", "color": "#00F2FE", "emissive": 2.0 }

Zero scene teardown. WebGL geometries and animation mixers persist continuously across turns.

Markerless Hand Gestures

MediaPipe WASM

Track hands through the webcam feed locally without specialized hardware or wearables:

👆 Point Raycast hover & inspect object attributes.
🤏 Pinch + Drag Grab and reposition selected 3D object in space.
🤏 Empty Pinch Pinch empty space to orbit camera around target.
👐 Two-Hand Pinch Pinch with both hands to smoothly zoom and dolly.
Data Visualisation

The Async Widget Deck

How FRIDAY mounts live information cards without stalling conversational audio turns. Test the skeleton-to-card hydration flow below.

Interactive Deck Simulator

When a user asks for complex data, FRIDAY emits a skeleton placeholder in 0 milliseconds. While the user hears the brief verbal answer, a background agent composes strict .hud-* tokenized HTML with SVG charts and hydrates the card.

Security Guardrails: Hub strips all <script>, <iframe>, inline styles, and event handlers. All remote images are verified and proxied via local /img?u=... endpoint.
ACTIVE WIDGET DECK HYDRATION ENGINE ACTIVE
Hardware-Level Agency

Native macOS Desktop Integration

FRIDAY executes directly against macOS Quartz Window Server and Accessibility APIs—not through slow sandboxed browser emulation.

🖱️

CGEventPost Hardware Injection

Directly injects mouse clicks, drags, smooth trackpad scrolling, and keyboard shortcuts at the kCGHIDEventTap hardware layer. Operates seamlessly across multi-monitor setups without single-screen clamping.

🔍

Quartz Multi-Monitor Vision

Enumerates all active displays, Retina scaling factors, and Desktop Spaces. Maps normalized model coordinates (0–1000) back to physical pixels for pinpoint click accuracy on any monitor.

🛡️

Zero-Trust Approval Gate

When connecting remotely from an iPhone or iPad over Tailscale HTTPS, all shell commands and AppleScript executions suspend for one-tap biometric approval with a 45-second auto-deny timeout.

Capability Matrix

Native Tool Registry

38 native tools exposed directly to the model. Search and filter by category to inspect parameters, descriptions, and execution context.

38 Tools
Deployment & Setup

How to Setup & Use FRIDAY

Follow the step-by-step instructions below to configure permissions, initialize your environment, and launch FRIDAY on macOS.

macOS Permissions & System Requirements

FRIDAY requires macOS 12 Monterey or higher (tested on Sonoma & Sequoia) and Python 3.10+.

Accessibility

Enable in System Settings → Privacy & Security → Accessibility for PyObjC CGEvent automation and UI element inspection.

Screen Recording & Camera

Grant access for Quartz multi-monitor capture, FaceTime camera input, and MediaPipe hand gesture tracking.

Installation & Codebase Access

🔒
Repository Not Yet Public

The Project FRIDAY codebase is currently maintained as a private repository while core multimodal features and security sandboxing are finalized. The repository will be made publicly accessible on GitHub in an upcoming open-source release.

Terminal — Setup Workflow (Upcoming Public Release)
# 1. Once public access is opened: git clone https://github.com/vincecyriac/friday-ai-assistant.git cd friday-ai-assistant # 2. Create and activate virtual environment python3 -m venv .venv source .venv/bin/activate # 3. Install core dependencies (pyaudio, google-genai, opencv, websockets, pyobjc) pip install -r requirements.txt

Environment Configuration (.env)

Copy the template and insert your Gemini API Key:

.env
# Gemini Developer or Vertex API Key GEMINI_API_KEY=your_gemini_api_key_here # Gemini Live Realtime Model GEMINI_MODEL=gemini-3.1-flash-live-preview # FRIDAY Voice: Aoede (feminine, articulate) or Kore FRIDAY_VOICE=Aoede # Optional Tripo3D API Key for generative text-to-3D models TRIPO_API_KEY=your_tripo_key_here

Launch Modes

Desktop App (PyWebView)
# Launches native window with saved permissions .venv/bin/python app_desktop.py
Headless Hub Server
# Runs hub & serves GUI at http://localhost:8766 .venv/bin/python friday_hub.py

Zero-Trust Remote Access via Tailscale

Access FRIDAY securely from your iPhone or Android phone without opening public router ports:

Tailscale Configuration
# 1. Start Tailscale Serve HTTPS gateway ./setup_remote.sh # 2. Open printed HTTPS URL on your phone # Smart sensor routing activates: phone mic/camera become primary sensors # Shell/AppleScript calls prompt 45s one-tap approval

Example Conversational Voice Prompts

Friday, show me Alphabet's stock and key metrics.
Mounts instant skeleton and hydrates animated SVG financial chart in <1.5s.
Render a 3D model of a drone and highlight the rotors.
Loads interactive Three.js SVE scene and executes delta highlight op.
Inspect what's on my second monitor and click the Submit button.
Quartz display capture + AX element inspect + CGEventPost click.
Who is standing in front of the camera right now?
OpenCV YuNet face detection matched against friday_profiles.json embeddings.