§1The problem
Chord sites have mistakes, don't have the exact version you want, and the screen doesn't keep up with what you're playing. There was no app that generated the chart from the audio and moved along with the song.
§2What I built
- Audio analysisPython on the GPU: separates instruments, finds beats, chords, key, sections (verse, chorus, solo), timed lyrics and notes.
- Web appSong library, processing queue, review screen (fix a chord/section/lyric with the audio playing) and a play screen.
- ToolsChord dictionary (guitar diagrams) and a tuner.
- Microphone followerWorks out where you are in the song, playing along with the recording or alone.
- Song by linkYouTube downloads directly; Spotify, Apple and Deezer links only identify the song.
- Native shellFor iPhone/iPad (stores on the device, works offline), plus an Apple Watch version.
§3The hard parts
5 entries- 01
Following the music live
Choosing the “most likely path” instead of reacting to every instant; several references (mix, backing without vocals, chords) so an intro isn't confused with an instrumental passage; tuned with a bench test, not by eye.
- 02
Telling verse from chorus
The strongest signal was repeated lyrics, compared word by word, because transcription splits the same chorus in different ways.
- 03
Old audio models on new Python
An ONNX model installed without its old dependencies, and avoiding hangs (an infinite transcription loop at the end of a song; a time limit on the queue).
- 04
Platform details
WKWebView inside SwiftUI crashing on iOS 26 (replaced with UIKit), M4A audio converted before analysis, an intermittent 403 on downloads solved by trying several formats.
- 05
No database
Each song is a folder of files; the AI output is never edited by hand and the reviewed version is kept separately.
§4Result
The first three stages of the plan and the web part of the fourth are done. The web app is in production on my home server, and I use it over a private network. The native shell runs on a real iPhone (not just the simulator), and the Apple Watch version works.
§5Stack
- Python + GPU (Demucs, Whisper, beat_this, basic-pitch)
- Next.js 16 + TypeScript
- Own design system
- Swift / UIKit (native shell)
- Service worker (offline)
- pm2
- Tailscale (HTTPS for the microphone)
- yt-dlp