$ show freelawproject/courtlistener
CourtListener is a comprehensive legal research platform operated by Free Law Project. It aggregates U.S. court opinions, dockets, oral arguments, and judge profiles from multiple sources including PACER/RECAP, Harvard Caselaw, Columbia, state courts (Florida, Texas, New York), and SCOTUS. The system provides full-text search via Elasticsearch, real-time and scheduled alerts, REST API with OAuth2, webhook notifications, citation analysis, and audio transcription. It also includes tools for bulk data import, data cleaning, and embedding generation for LLM training. Architecturally, it is a Django 6.0+ monolith with a Celery task queue (Redis broker), PostgreSQL database, Elasticsearch search engine, and S3-compatible storage. The frontend uses Django templates with Alpine.js and some React components, built via Webpack. The system runs as multiple Docker containers (web, Celery workers, scrapers, daemons). Key data flows include nightly PACER scraping, email-based RECAP document ingestion, citation extraction using eyecite, and state-specific docket merging pipelines. The system is heavily event-driven with pgtrigger-based audit trails and Redis-based throttling. While primarily an application, it has a strong data research character: extensive ETL pipelines, machine learning integration (LLMs for text extraction, OpenAI Whisper for transcription), citation network analysis, and state court data modeling. The codebase demonstrates high craftsmanship in reusable patterns like the CeleryThrottle, tiered caching, and generic merger classes, but also carries some technical debt from many SQL migrations and tightly coupled scrapers.
The LLM identified 3 files worth highlighting for reuse:
CeleryThrottle is a Redis-based rate limiter for Celery task dispatch. It provides a context manager and decorator to control how often a task can be submitted (e.g., once per N seconds per unique key). This is a self-contained utility that can be extracted into any Celery project to prevent API rate limits or resource overload. It demonstrates non-obvious skill by combining Redis atomic operations with Celery's task flow.
tiered_cache is a multi-level caching decorator that checks an in-memory cache (function attribute), then Django's cache backend, then calls the underlying function. It supports configurable timeouts and cache key generation. This pattern is reusable across any Django project needing efficient caching with fallback. It showcases advanced Python decorator techniques and cache hierarchy design.
The Merger base class provides a generic pattern for merging scraped or imported data into existing database records. It defines Attribute, OneToOneRelation, and ManyToManyRelation descriptors that map source fields to model fields, with support for dry-run mode (database rollback), result tracking, and idempotent updates. This component is reused across Florida, New York, and Texas importers and could be extracted for any incremental data sync task. It demonstrates clean separation of concerns and careful transaction management.