This is a large, multi-language monorepo from X (formerly Twitter) containing the core code for the For You feed recommendation system. It includes real-time Rust services (home-mixer, thunder, vm-ranker), a Scala rule engine (botmaker) for spam/abuse detection, Python ML training infrastructure (phoenix/xrex), content safety pipelines (grox), and batch feature extraction jobs (simclusters, agatha). The system is composed of loosely coupled services communicating via thrift/gRPC and Kafka, with a strong emphasis on modular pipeline architectures (sources, hydrators, filters, scorers, selectors). While the code demonstrates high craftsmanship with clean abstractions and extensive use of traits/interfaces, it is deeply integrated with internal infrastructure (Manhattan, Strato, Finagle) and requires significant adaptation for external reuse. The repo is hybrid, containing both production application services and substantial ML research/training code.
This repository is an educational tutorial for Apache Spark using Scala, designed to teach distributed data processing concepts through practical examples. It functions as a data research and engineering learning resource, featuring Jupyter notebooks and Scala source code that demonstrate RDD operations, SparkSQL, and streaming pipelines. The project includes bundled datasets such as the King James Version Bible and the Enron Spam/Ham email corpus, along with golden output files for test verification. While it serves as a comprehensive guide for Spark development, it primarily focuses on teaching data transformation patterns rather than deploying a production service. The codebase demonstrates solid engineering practices for an educational context, including unit tests, custom utility classes for CLI argument parsing, and structured data schemas. However, the technical stack is significantly outdated, relying on Spark 2.3.0 and Scala 2.11.8, which are no longer supported. The examples frequently use deprecated APIs, such as the RDD-based DStream for streaming, which has been superseded by Structured Streaming in modern Spark versions. This creates a high technical drift, making the code less suitable for new projects without significant refactoring. Despite the outdated dependencies, the repository remains a valuable resource for understanding Spark fundamentals. The inclusion of golden files and test utilities provides a reproducible framework for validating data processing logic. The tutorial structure progresses logically from basic word counts to complex SQL and streaming operations, making it effective for learning. Users should be aware that while the patterns are sound, the specific API implementations will require updates to align with current Spark LTS versions.
This repository is the source code for the 'Scala็ ไฟฎใใญในใ' (Scala Training Textbook), a comprehensive Japanese-language textbook for learning the Scala programming language. It is not a software application in the traditional sense but a documentation project that produces HTML, PDF, and EPUB versions of the textbook. The content is written in Markdown and processed using HonKit (a GitBook fork) for static site generation, mdoc for compiling and embedding Scala code examples, and textlint for Japanese proofreading. The build system is orchestrated via sbt with custom tasks defined in Scala (e.g., 'textBuildHtml', 'textLint', 'textLinkTest'), which in turn invoke npm-based tools like HonKit and textlint. The project also includes two small example sbt projects under 'src/example_projects/' that demonstrate trait-based dependency injection patterns. Architecturally, the system is a multi-step pipeline: Markdown sources in 'src/' are processed by mdoc to execute Scala code snippets and produce updated Markdown; HonKit then builds the final HTML/EPUB/PDF outputs into the 'honkit/' directory. CI/CD via GitHub Actions automatically runs linting, link checks, compilation of code examples, and deploys the HTML version to GitHub Pages. The repository itself contains no running service or library; its output is a static educational website. Key entry points are the sbt tasks defined in 'build.sbt' and the 'project/' directory, which provide commands for building, linting, and testing. External dependencies include sbt 2.0.9, Scala 3.8.4, HonKit 6.2.2, various npm packages (e.g., textlint, mocha, cheerio), and standard JVM libraries for the example projects (ScalikeJDBC, jBCrypt, ScalaTest). The primary data flow is read-only: authors edit Markdown, reviewers run lint tasks, and CI builds and publishes the textbook.
Polynote is a polyglot notebook server that enables interactive data analysis in Scala, Python, SQL, and Vega. It provides a web-based IDE with Monaco editor, real-time kernel execution, rich visualizations (Vega-Lite, Matplotlib), and tight Apache Spark integration. The architecture is a monolith: a Scala server using ZIO effects, a custom binary WebSocket protocol, and a TypeScript frontend with its own reactive state management. Key entry points are the server main class (Main.scala) and the headless notebook runner (NotebookRunner.scala). Data flows from cell edits through WebSocket messages to the kernel, which executes code and streams results back. External dependencies include JVM, Python via Jep, Spark, Coursier for dependency resolution, and various frontend libraries. While primarily an application, it also serves as a data exploration platform with streaming data abstractions and visualization capabilities.
This repository is a curated list of interesting open source projects organized by programming language. It contains markdown files with links and descriptions for each language. There is no application code, data pipelines, or analytical models. It serves as a reference directory for developers seeking open source projects to contribute to or learn from. The repository is well-maintained with recent updates and a large community following.