§1The problem
In a shop, checking prices or comparing products means taking out the phone, opening an app, typing and taking a photo. There was nothing that worked hands-free.
§2What I built
- PWAInstallable web app with the live session: ready, listening, searching, answering, paused.
- Double-tap detectorUses the phone's motion sensor, no shortcut needed.
- CaptureRecording that ends automatically on silence, 2 to 4 photos, spoken answer.
- Transcription serverSpeech-to-text on the GPU.
- Choice of AIOn the home screen: an LLM through the command line, or an open model on a GPU server rented by the hour, started on demand.
- AI zoomUltra-wide camera and a tool that crops the original photo.
- DiagnosticsNumbers only, to tune the tap detector.
§3The hard parts
5 entries- 01
Recognising taps from the sensor without a clean signal
Thresholds tuned with real phone measurements, logging every decision.
- 02
iOS permissions
The sensor has to be requested before the camera.
- 03
Giving vision to a local language model
Loading the image module, and doing web search for it without exposing the photos.
- 04
Transcription sharing the GPU with another model
Loads on use and leaves the GPU after 5 minutes idle; 0.25 s with the model loaded.
- 05
Privacy by design
Photos in a temp folder deleted at the end, audio never touches the disk, only the text goes to history.
§4Result
- transcription with the model loaded
- 0.25 s
- answer with web search
- ~17-30 s
Running on my home server. Answers take ~17 to 30 s with web search. The double tap already works on a real iPhone, but real-world testing on the iPhone isn't finished yet.
§5Stack
- Next.js 16 + TypeScript
- PWA (motion sensor, camera, microphone, browser speech)
- Python + faster-whisper on the GPU
- LLM via command-line agent
- Open LLM on a rented GPU (RunPod)
- Own MCP server
- pm2
- Tailscale (HTTPS)