$ show windhide/SkyMusicPlay-for-Windows
SkyMusicPlay-for-Windows is a Windows desktop application that automates piano playing inside the game Sky: Children of the Light. It is built as a two-process localhost system: an Electron + Vue 3 (Naive UI) frontend for the music library, playback controls, settings, i18n, and a custom canvas-based sheet editor; and a Python FastAPI backend (port 9899) that converts ordinary audio/MIDI files into game-compatible key sheets and replays them by injecting synthetic keyboard events into the game window. Conversion of audio uses a real pipeline: demucs source separation, Spotify basic_pitch audio-to-MIDI, pretty_midi pitch analysis, and BPM-based dynamic merge thresholds, producing JSON sheets consumed by the player. A YOLO-based vision layer (ultralytics with two custom .pt models) detects on-screen keys and friend/UI elements to power a follow-along tutorial mode, and a separate Tkinter transparent overlay (draw-follow-window) draws guidance boxes over the game. Shortcut keys arrive over a WebSocket server on port 11452. The architecture is a localhost monitor-and-worker design: Electron spawns and supervises the packaged Python server, the Vue renderer talks to it over REST, and Python runs multiple background threads (playback queue, HWND check, shortcut handling, OCR/follow detection, frame-alive watchdog), with separate Intel and AMD keyboard-robot implementations. The project is feature-rich and actively developed, but carries hobby-grade technical debt: hardcoded absolute Windows paths and fixed localhost ports, a security-sensitive eval() in the config getter, wide-open CORS, duplicated Intel/AMD modules, swallowed exceptions, stale commented-out AI providers, version inconsistencies, machine-translated locale files, and at least one corrupted macro template asset. Most automation code is Sky-specific (window class names, key maps, OCR labels), so it is not portable as-is; nonetheless, several modules illustrate clean, adaptable patterns such as producer-consumer task queues, Win32 input injection, audio-to-discrete-note mapping, and socket-driven screen overlays.
The LLM identified 4 files worth highlighting for reuse:
Core audio-to-game-sheet transcription pipeline: demucs separates the instrumental stem, Spotify basic_pitch converts it to MIDI, and pretty_midi maps pitches onto the game's discrete key grid. Non-obvious craftsmanship includes BPM-based dynamic merge thresholds, velocity filtering, semitone transposition, and detail-granularity controls. The pattern (polyphonic audio to MIDI to a constrained discrete key map) transfers to any fixed-layout instrument or game, although the pitch-to-key tables themselves are Sky-specific.
Self-contained transparent Tkinter overlay that draws colored guidance boxes around detected game elements, driven by a tiny socket protocol on localhost:12345. It is intentionally decoupled from the rest of the codebase and demonstrates a reusable pattern for real-time visual guidance: any detection pipeline can emit box coordinates and this server renders them over any desktop window with minimal modification.
A producer-consumer worker that decouples API requests from long-running playback operations: commands are enqueued and executed sequentially by a background thread, preventing concurrent playback-state corruption. This is a clean, adaptable concurrency pattern for any single-resource automation service, even though the task payloads reference app-specific global state.
Renders a transcribed sheet's JSON note data into a self-contained visual HTML page via a Jinja2 template, giving an inspectable and printable representation of the song. The data-to-render boundary is clean, and the pattern (structured timeline/note data to human-readable HTML) is reusable for any sheet-music or event-timeline visualization.