Mimesis is a Python library for generating high-quality, locale-aware fake data, intended for use in testing, database seeding, API mocking, and data anonymization. Unlike simpler lorem-ipsum generators, it produces realistic names, addresses, dates, financial data, internet-related strings, and more, across 47 locale variants. The architecture is modular: locale-independent and locale-dependent providers (e.g., Person, Address, Internet) load curated JSON datasets, while a suite of schema tools (Field, Fieldset, Schema, SchemaBuilder) allows declarative generation of flat or relational synthetic datasets with lazy evaluation, dependency resolution, and export to CSV/JSON/pickle. A custom Random class with weighted selection and deterministic seeding ensures reproducibility, and optional plugins integrate with factory_boy and pytest-randomly. The codebase is well-structured, extensively typed (mypy strict), and thoroughly tested, with a strong emphasis on performance (~20-30x faster than comparable libraries like Faker). External runtime dependencies are minimal; the only optional ones are pytz and factory-boy.
Kiln is a full-stack AI development platform for building, evaluating, and optimizing LLM-based tasks and agents. It provides a desktop application (Tkinter tray + FastAPI server) with a Svelte web UI, plus a core Python library and a cloud server component. The system supports the complete AI workflow: project/task management, multi-provider LLM orchestration via LiteLLM, evaluation with multiple judge types (LLM-as-judge, code eval, exact match, pattern match, etc.), synthetic data generation, RAG pipelines (chunking, embedding, vector stores), fine-tuning, prompt optimization, and Git-based project versioning. It also includes a 'Kiln Pro' cloud service for advanced features like copilot-assisted spec refinement and batch generation. The architecture is a monorepo with uv workspace, featuring extensive test coverage (unit, integration, e2e) and CI/CD pipelines. While the primary character is an application, it incorporates significant data engineering patterns for evaluation pipelines, synthetic data generation, and statistical analysis.
Promptomatix is a framework for automating and optimizing large language model (LLM) prompts through iterative refinement. It provides both CLI and API interfaces for users to input task descriptions, generate synthetic data, optimize prompts using DSPy or meta-prompt backends, and evaluate results with custom metrics. The system features modular architecture separating language model management, configuration, session tracking, feedback handling, and metrics evaluation. Key workflows include prompt optimization with feedback incorporation, batch processing capabilities, and extensibility for custom metrics and optimization backends. The framework targets researchers and developers seeking to reduce manual prompt engineering effort while improving LLM output consistency and quality.
The SmartNoise SQL/Synth/Eval monorepo is a differentially private (DP) data analysis toolkit. It provides three main packages: `snsql` (DP SQL query rewriting with calibrated noise), `snsynth` (DP synthetic data generation via MWEM, GANs, MST, and other mechanisms), and `sneval` (evaluation of synthetic data quality). The SQL package includes a custom ANTLR-based SQL parser (`SqlSmall.g4`) that rewrites queries to enforce DP, supports multiple backends (Pandas, Spark, PostgreSQL, SQL Server, BigQuery), and tracks privacy budgets via an odometer. The synth package offers a rich set of synthesizers (DPCTGAN, PATECTGAN, MWEM, MST, AIM, etc.) with a consistent `fit`/`sample` API and a `TableTransformer` pipeline for preprocessing. The eval package provides metrics for comparing real vs. synthetic data, including accuracy, bias, and suppression/fabrication counts. The repository includes extensive tests (SQL parsing, mechanism accuracy, synthesizer round-trips) and sample notebooks demonstrating DP workflows.
VOID is a Netflix research codebase for removing objects from videos while also erasing their physical side effects, such as shadows, reflections, and secondary motion caused by the removed object. The core runtime is a two-pass diffusion inpainting system built on CogVideoX: pass 1 performs coarse object removal guided by a quadmask, and pass 2 refines temporal consistency using warped noise and optical-flow-inspired conditioning. The repo is primarily an application/research library rather than a pure data project: it ships a `videox_fun` Python package with model definitions, custom diffusion pipelines, inference and training scripts, a FastAPI/Ray inference API, Gradio UIs, and a Colab notebook, alongside synthetic data-generation code that uses Blender and Kubric.