$ show metabrainz/listenbrainz-server
ListenBrainz Server is the backend and frontend for ListenBrainz, an open-source music listening history platform operated by the MetaBrainz Foundation. In production it ingests listening events from multiple sources (Spotify, Last.fm, Libre.fm, Maloja, file uploads, and direct API submissions), stores them in TimescaleDB, enriches them with MusicBrainz identifiers via an MBID mapping pipeline, and exposes them through a public REST API, a Last.fm-compatible API, WebSocket streams, and a React single-page application. The system also computes per-user and site-wide statistics, artist/recording/user similarity, collaborative-filtering recommendations, and annual 'Year in Music' summaries using Apache Spark batch jobs orchestrated through RabbitMQ and HDFS. The architecture is a hybrid of a traditional Flask web application and a set of background data-processing services. The main web server handles user authentication, API endpoints, playlist management, social features, and page rendering, while separate workers consume RabbitMQ queues for timescale writes, Spark job requests, metadata cache crawling (Spotify, Apple Music, SoundCloud, Internet Archive), and background imports/exports. The frontend is a large TypeScript/React application with React Router, TanStack Query, Jotai state, and a custom multi-source audio player called BrainzPlayer. Although the repository is primarily an application, it contains substantial data-engineering machinery—Spark incremental stats engines, dump generation/import, MBID canonicalization, and similarity algorithms—making it a genuinely hybrid codebase with the application character dominant. The codebase is well-structured for its size: it has clear module boundaries, abstract base classes for importers and OAuth services, a reusable incremental stats query-provider pattern, and extensive unit and integration tests. However, it also carries significant complexity and technical debt: multiple database backends, duplicated SQL across Spark entity stats, legacy compatibility endpoints, hardcoded external URLs, and some skipped or heavily mocked tests. The project is actively maintained, with a steady stream of schema migrations and feature additions through 2026.
The LLM identified 5 files worth highlighting for reuse:
Defines the QueryProvider abstraction used by all incremental Spark statistics jobs. Subclasses implement get_aggregate_query, get_combine_aggregates_query, get_filter_aggregate_query, and get_stats_query, allowing sitewide and per-user entity stats (artists, releases, recordings, listening activity, etc.) to share a single incremental processing engine. This is a clean template-method pattern that could be extracted for any time-series aggregation pipeline needing incremental recomputation over partitioned data.
Provides a reusable BulkInsertTable base class for creating, populating, and atomically swapping large PostgreSQL tables. It handles batched inserts and post-processing hooks, which is a common pattern for building derived data tables (canonical mappings, metadata caches) without downtime. The abstraction is generic enough to be lifted into other projects that need reliable bulk table rebuilds.
Implements RecordingSimilarityBase, an abstract base class for session-based recording similarity computed from listening events. It orchestrates chunked Spark processing, writes intermediate Parquet results to HDFS, combines chunks, and emits result messages. The session-detection SQL uses window functions and configurable thresholds, demonstrating a non-trivial, reusable pattern for large-scale pairwise similarity computation.
Defines the BaseListensImporter abstract class that standardizes how listens are fetched from external services (Spotify, Last.fm, Libre.fm, etc.) and submitted into ListenBrainz. It encapsulates pagination, error handling, and progress tracking, leaving subclasses to implement service-specific API calls. This is a solid template-method pattern for any recurring data-import worker that must pull from rate-limited external APIs.
Provides a BaseBrainzService base class for OAuth2-based external service integrations (Spotify, CritiqueBrainz, MusicBrainz, etc.). It centralizes token storage, refresh logic, and API request handling, so new music-service integrations can be added by subclassing rather than duplicating OAuth boilerplate. This is a clean, reusable pattern for any application that integrates multiple OAuth2 providers.