Controlling coding agents from Ray-Ban Meta Display glasses
A weekend experiment from the BionicOS playground — my Claude coding agents now reach me on smart glasses. When an agent on my Mac needs approval, the question pops up in my field of view and I answer with a finger gesture, from anywhere in my apartment.
A weekend experiment from my BionicOS playground: my Claude coding agents („Marvin“) now reach me on the Ray-Ban Meta Display. When an agent on my Mac needs approval, the question pops up right in my field of view — and I answer with a finger gesture, from anywhere in my apartment.
Controlling coding agents from smart glasses? Works. 🤖🕶️
Under the hood
- A Mac daemon (Node.js + Claude Agent SDK) runs parallel coding sessions and broadcasts events via WebSocket.
- The glasses app: pure vanilla JS, 600×600 additive display (black = transparent), fully navigable via the EMG wristband.
- Vision Pro and the glasses share one session pool — start on the glasses, continue on Vision Pro. Two ports, one state.
- Voice input: speak a task → Web Audio API → 16 kHz WAV → whisper.cpp transcribes locally on the Mac → confirm → the agent gets to work. No cloud STT.
The bigger picture
BionicOS is the layer where agents do the work and humans just direct — from spatial computing down to everyday glasses. The agent works, I live. The interface is just the back channel for „may I?“ and „go ahead.“
A bicycle for the mind
And the best part: this very weekend experiment is what I’m now using to write this entry. Steve Jobs once called the computer a „bicycle for the mind“ — a tool that multiplies your own capability. We humans are tool builders; it’s our oldest skill. The agent builds, I think — the tool recedes into the background, the idea remains. A weekend experiment becomes the tool the next idea is built with.
This entry captures an experiment from the BionicOS Lab — raw, built over a weekend, right in the field of view. And written with that very tool.