$ show TeamNewPipe/NewPipeExtractor
NewPipe Extractor is a Java library that provides a unified abstraction for extracting structured metadata (streams, channels, playlists, comments, search results) from multiple streaming platforms including YouTube, SoundCloud, Bandcamp, PeerTube, and media.ccc.de. It reverse-engineers private APIs and scrapes HTML/JSON to produce normalized data models. The library is designed as a headless component for the NewPipe Android app but can be used independently. Its architecture features a service registry, abstract extractor classes, link handler factories for URL normalization, and a collector pattern for batch extraction with error accumulation. The codebase is well-tested with JUnit 5 and a mock/recording downloader system, but YouTube-specific extraction is fragile due to reliance on hardcoded client versions, JavaScript deobfuscation, and frequently changing InnerTube API schemas.
The LLM identified 5 files worth highlighting for reuse:
Defines the Collector interface for batch extraction with error accumulation. Used by all list extractors (streams, channels, playlists, comments) to commit items and collect errors. The pattern is self-contained and could be extracted for any batch processing pipeline that needs to aggregate results and errors.
Immutable value object representing an image with multiple resolution levels (LOW, MEDIUM, HIGH, UNKNOWN). Provides static factory methods and resolution-level ordering. Reusable in any application needing multi-resolution image handling, independent of the extraction context.
Generic serializable cache with configurable maximum size and clear factor for eviction. Used to cache generated DASH manifests. The implementation is clean, thread-safe (via synchronized), and could be reused in any Java project needing a simple size-bounded cache.
Utility to extract function bodies from JavaScript source code using brace matching. Handles nested braces, strings, and comments. Used for YouTube signature deobfuscation. The algorithm is self-contained and could be reused for any JavaScript source analysis task.
A smooth rate limiter adapted from Google Guava's SmoothRateLimiter, providing bursty and smooth rate limiting with configurable permits per second. Used in test infrastructure to throttle requests to YouTube and Bandcamp. The implementation is standalone and could be reused in any Java project needing rate limiting.