$ show xai-org/x-algorithm
This is a large, multi-language monorepo from X (formerly Twitter) containing the core code for the For You feed recommendation system. It includes real-time Rust services (home-mixer, thunder, vm-ranker), a Scala rule engine (botmaker) for spam/abuse detection, Python ML training infrastructure (phoenix/xrex), content safety pipelines (grox), and batch feature extraction jobs (simclusters, agatha). The system is composed of loosely coupled services communicating via thrift/gRPC and Kafka, with a strong emphasis on modular pipeline architectures (sources, hydrators, filters, scorers, selectors). While the code demonstrates high craftsmanship with clean abstractions and extensive use of traits/interfaces, it is deeply integrated with internal infrastructure (Manhattan, Strato, Finagle) and requires significant adaptation for external reuse. The repo is hybrid, containing both production application services and substantial ML research/training code.
The LLM identified 3 files worth highlighting for reuse:
Defines an abstract TaskGenerator with start/stop/poll/ack, rate limiting via the `limits` library, and terminal failure hooks. The PriorityTaskGenerator multiplexes weighted sub-generators with random selection and payload-to-origin tracking. This is a reusable pattern for asynchronous task generation in any pipeline system, easily extractable to a standalone library.
Implements a generic Filter trait with async `filter` method, integrated with stats counters (request total, filtered counts). The trait is used across home-mixer for dedup, holdout, keyword, and other filters. This is a clean, extractable abstraction for any sequential filtering pipeline, with minimal internal dependencies.
Defines a `param!` macro that generates strongly-typed parameters with deserialization from string sources. It provides a succinct way to declare feature switches with defaults, experiment buckets, and client mapping. While X-specific, the macro pattern is reusable for any product needing dynamic configuration without a framework.