Made With ML is an educational data science project that implements an end-to-end ML pipeline for classifying GitHub repositories into categories (NLP, computer vision, MLOps, other) based on their titles and descriptions. The core data lineage flows from raw CSV datasets (dataset.csv, holdout.csv) through text cleaning, tokenization via SciBERT, and stratified splitting into train/val/test sets, followed by distributed training with PyTorch and Ray Train, hyperparameter tuning via HyperOpt, evaluation with slice-based metrics using Snorkel, and serving through a FastAPI/Ray Serve deployment. The project demonstrates MLOps patterns including MLflow experiment tracking, data validation with Great Expectations, behavioral model testing, and CI/CD via GitHub Actions with Anyscale for cloud execution. The codebase is well-structured with clear separation of concerns across data.py, train.py, tune.py, evaluate.py, predict.py, serve.py, models.py, and utils.py. It includes comprehensive tests at the code, data, and model levels. However, the project shows signs of staleness with pinned dependencies from 2023 (Ray 2.7.0, transformers 4.28.1, torch 2.0.0) that have known security vulnerabilities and compatibility issues with current Python versions. The data pipeline components—particularly the CustomPreprocessor, stratified splitting, and text cleaning—are genuinely reusable patterns. The evaluation framework with slice-based metrics and behavioral tests (invariance, directional expectations, minimum functionality) represents solid ML engineering practice. The deployment configuration is tightly coupled to Anyscale's proprietary platform, limiting portability.
BLURR is a library that integrates Hugging Face transformers with fastai for NLP tasks. It provides high-level, mid-level, and low-level APIs to train and deploy transformer models for text classification, token classification, question answering, summarization, translation, and language modeling. The library leverages fastai's data block API and learner system, wrapping Hugging Face models to work seamlessly within the fastai training loop. Key components include custom data blocks for text processing, model wrappers for Hugging Face integration, specialized loss functions, and callbacks for optimization. The repository includes extensive examples and notebooks demonstrating usage with standard datasets like GLUE, SQuAD, and CNN/DailyMail.
FastText.js is a Node.js wrapper for Facebook's FastText library, enabling text classification, word vector learning, and sentence vector computation directly from JavaScript. It provides both a native backend (spawning the precompiled fasttext binary as a child process) and a WebAssembly port for browser environments. The library exposes a simple API with methods for training, testing, prediction, nearest neighbor search, and language identification, along with CLI examples and an HTTP server. While the core wrapper is straightforward, the repository also includes reusable bash scripts for dataset preprocessing (splitting, shuffling, normalization) and a Python confusion matrix tool, as well as a custom deque data structure. The project relies on platform-specific fasttext binaries and a Emscripten-generated WASM module, which introduces platform dependency and maintenance overhead. Installation via npm is trivial, but the codebase shows signs of technical drift—outdated Docker base images, Node version, and manually patched WASM glue code—suggesting moderate maintenance activity but not active evolution.
This repository is a curated collection of five large-scale Chinese NLP datasets (Wikipedia articles, news, QA pairs, community Q&A, and translation pairs) with descriptions and download links. It functions as a data catalog rather than a software project: there is no executable code, no ETL scripts, and no pipelines. The datasets are hosted externally (Google Drive, Baidu) and require manual download. The repository provides clear formatting examples and suggested use cases, but lacks any automation or reproducibility support such as download scripts, environment definitions, or data processing code. It is a valuable reference for researchers needing Chinese text corpora, but it is not a reusable data engineering artifact.