$ show GokuMohandas/Made-With-ML
Made With ML is an educational data science project that implements an end-to-end ML pipeline for classifying GitHub repositories into categories (NLP, computer vision, MLOps, other) based on their titles and descriptions. The core data lineage flows from raw CSV datasets (dataset.csv, holdout.csv) through text cleaning, tokenization via SciBERT, and stratified splitting into train/val/test sets, followed by distributed training with PyTorch and Ray Train, hyperparameter tuning via HyperOpt, evaluation with slice-based metrics using Snorkel, and serving through a FastAPI/Ray Serve deployment. The project demonstrates MLOps patterns including MLflow experiment tracking, data validation with Great Expectations, behavioral model testing, and CI/CD via GitHub Actions with Anyscale for cloud execution. The codebase is well-structured with clear separation of concerns across data.py, train.py, tune.py, evaluate.py, predict.py, serve.py, models.py, and utils.py. It includes comprehensive tests at the code, data, and model levels. However, the project shows signs of staleness with pinned dependencies from 2023 (Ray 2.7.0, transformers 4.28.1, torch 2.0.0) that have known security vulnerabilities and compatibility issues with current Python versions. The data pipeline components—particularly the CustomPreprocessor, stratified splitting, and text cleaning—are genuinely reusable patterns. The evaluation framework with slice-based metrics and behavioral tests (invariance, directional expectations, minimum functionality) represents solid ML engineering practice. The deployment configuration is tightly coupled to Anyscale's proprietary platform, limiting portability.
The LLM identified 5 files worth highlighting for reuse:
Implements a reusable data pipeline with stratified train/test splitting that preserves class distribution using Ray Datasets, a clean_text function with stopword removal and regex-based normalization, and a CustomPreprocessor class with fit/transform interface for label encoding and tokenization. The stratify_split function handles the common ML problem of maintaining class balance across splits using groupby operations, which is directly transferable to any classification dataset. The preprocessor pattern (fit on training data, transform any data) follows scikit-learn conventions and avoids data leakage.
Provides a comprehensive evaluation framework that goes beyond aggregate metrics to include per-class performance, slice-based metrics using Snorkel's slicing functions (e.g., NLP projects mentioning LLMs, short text), and custom slicing logic. This pattern of evaluating model performance on meaningful data subgroups rather than just overall metrics is crucial for production ML systems and directly applicable to any classification task where certain data slices matter more than others.
Implements a behavioral testing framework for ML models inspired by CheckList, including invariance tests (output shouldn't change under semantically equivalent inputs), directional expectations (known input-output relationships), and minimum functionality tests (simple cases that must work). This pattern of testing model behavior rather than just accuracy metrics is a reusable approach to catching model regressions and is applicable to any NLP classification model.
Contains well-crafted utility functions including set_seeds for full reproducibility (NumPy, PyTorch, CUDA, Python hash seed), pad_array for handling variable-length sequences in batches, and collate_fn for converting numpy batches to properly typed and padded PyTorch tensors on the correct device. The set_seeds function in particular addresses the common pain point of reproducibility in ML experiments and can be dropped into any PyTorch project.
Uses Great Expectations to define a comprehensive data validation suite including schema adherence (expected columns in order), value set validation (allowed labels), uniqueness constraints (no duplicate IDs, no data leaks via compound columns), null checks, and type validation. This pattern of codifying data quality expectations as testable assertions is directly reusable for any tabular dataset and catches data pipeline issues early.