MLflow is an open-source ML lifecycle platform that provides experiment tracking, model registry, model deployment, evaluation, and observability for both classical ML and GenAI/LLM workflows. In production, it operates as a modular monolith: a Python backend (Flask/FastAPI) with SQLAlchemy-backed storage (SQLite, PostgreSQL, MySQL, MSSQL), a React/TypeScript web UI, and client SDKs for Python, R, Java, and TypeScript. The system exposes a REST API and CLI for logging experiments, registering models, collecting traces via OpenTelemetry, and managing an AI Gateway that routes requests to dozens of LLM providers with budget controls, guardrails, and traffic splitting. The architecture includes a plugin system for custom store backends, deployment targets, and authentication providers, as well as a custom linting tool (clint) and extensive CI/CD pipelines. Key entry points are the `mlflow` CLI, `MlflowClient` Python API, tracking server (`mlflow server`), gateway server (`mlflow gateway`), and model scoring server (`mlflow models serve`). Data flows from user instrumentation through asynchronous logging to the backend store, with traces and evaluations queryable via the UI or API. External dependencies span ML frameworks (scikit-learn, PyTorch, TensorFlow, LangChain, etc.), LLM provider SDKs, and infrastructure tools (Docker, Kubernetes, Helm).
Langfuse is an open-source LLM observability and evaluation platform. It provides tracing, prompt management, evaluations, datasets, experiments, monitors, and a public API for integrating with AI applications. The architecture is a monorepo with pnpm workspaces and Turborepo, consisting of a Next.js frontend (web), a background worker (Express + BullMQ), a shared package with types and server utilities, and an enterprise edition. Data flows from ingestion (REST API, OpenTelemetry, SDKs) into ClickHouse for high-volume analytics and PostgreSQL for relational metadata. Background jobs handle evaluations, batch exports, cloud billing, and integrations. The codebase is well-structured with extensive testing, a custom ESLint plugin, and reusable UI components. It is actively maintained with frequent migrations and feature additions.
Kiln is a full-stack AI development platform for building, evaluating, and optimizing LLM-based tasks and agents. It provides a desktop application (Tkinter tray + FastAPI server) with a Svelte web UI, plus a core Python library and a cloud server component. The system supports the complete AI workflow: project/task management, multi-provider LLM orchestration via LiteLLM, evaluation with multiple judge types (LLM-as-judge, code eval, exact match, pattern match, etc.), synthetic data generation, RAG pipelines (chunking, embedding, vector stores), fine-tuning, prompt optimization, and Git-based project versioning. It also includes a 'Kiln Pro' cloud service for advanced features like copilot-assisted spec refinement and batch generation. The architecture is a monorepo with uv workspace, featuring extensive test coverage (unit, integration, e2e) and CI/CD pipelines. While the primary character is an application, it incorporates significant data engineering patterns for evaluation pipelines, synthetic data generation, and statistical analysis.
This repository is a research codebase for training and evaluating draft models for speculative decoding, supporting three algorithms: DSpark, DFlash, and Eagle3. It provides a full pipeline from data preparation (downloading prompts, regenerating answers with a target model via SGLang, and precomputing a large binary cache of target hidden states) to distributed training with FSDP and BF16, and evaluation on standard LLM benchmarks (GSM8K, MATH500, AIME, HumanEval, etc.). The code is well-structured with reusable components for configuration, distributed utilities, checkpointing, metrics, and custom attention masks. While primarily a research artifact, it exhibits strong engineering practices and several modules that could be extracted for other large-scale ML projects. The repository is hybrid in nature: it contains both application-like training/evaluation scripts and data_research elements such as data pipelines and evaluation datasets.
AG2 is a production-grade Python framework for building multi-agent AI systems. It provides a unified agent abstraction with support for multiple LLM providers (OpenAI, Anthropic, Gemini, Mistral, Bedrock, Ollama, xAI, Z.AI, DashScope), protocol integrations (A2A, A2UI, MCP, ACP), and sandboxed code execution (Docker, Daytona, local). The architecture is event-driven with a modular middleware pipeline, dependency injection, and a streaming event bus. It includes a full evaluation suite with trace reconstruction, scorers, pairwise comparison, and result persistence. The network layer enables distributed agent coordination via a hub-and-channel model with WAL persistence, identity management, and transport-agnostic envelopes. The framework also supports real-time voice interaction, skills-based toolkits, knowledge stores, and human-in-the-loop workflows. While the surface area is large, the codebase demonstrates high craftsmanship with clean abstractions, protocol-based interfaces, and extensive test coverage across all modules.
CodeStable is a framework for AI coding assistants (Claude Code, Codex CLI) that provides a set of 'skills' — thin harnesses that guide AI agents through software development tasks such as feature implementation, bug fixing, refactoring, epic decomposition, code review, and lesson management. The system emphasizes risk-based assurance, evidence-driven decisions, and minimal project memory, with a core philosophy of 'thin harness, thick context'. It is distributed as plugins for both Claude Code and Codex CLI, and includes a meta-skill (eval-cs-skill) for authoring, evaluating, optimizing, and releasing skills. The repository contains extensive experiment infrastructure with 15+ structured experiments covering routing accuracy, end-to-end outcomes, planted-defect recall, and learning transfer across multiple models. It features a sophisticated evaluation pipeline with deterministic oracles, hidden tests, regression tests, and checkpointed resume. The codebase demonstrates high craftsmanship with clean abstractions, strict validation, and cognitive honesty tagging (measured/soft/underpowered). Key architectural patterns include the thin skill pattern (compact SKILL.md with progressive disclosure via references), adapter pattern for AI model execution (Claude Code, Codex CLI, API, mock), deterministic seed building for reproducible experiments, and a risk-to-assurance mapping used generically by all task skills. The project has 8 ADRs documenting design decisions and a v1-to-v2 migration path that reduced 32 skills to 8. While the codebase is well-maintained and actively developed, some experiments are marked underpowered and the learning transfer epic was cancelled without completing primary hypotheses. The evaluation system is complex and tightly coupled to specific model names and macOS sandbox profiles, which may limit portability.
PLINDER is a comprehensive protein-ligand interaction dataset and evaluation resource, providing over 400,000 annotated systems with 750+ annotation columns derived from the PDB. The repository contains a full data generation pipeline (Metaflow-based, configurable via YAML) that downloads, annotates, scores, clusters, and splits structures into train/val/test sets with leakage control. It also includes a Python API (PlinderSystem, PlinderDataset) for accessing the dataset, querying similarity scores, and loading structures, as well as an evaluation harness (ModelScores, StratifiedTestSet) for benchmarking docking algorithms using metrics like lDDT-PLI, DockQ, and posebusters checks. The codebase is hybrid: primarily a data research project with strong software engineering practices (Docker, CI/CD, extensive tests, config-driven pipelines), but the core value is the curated dataset and the tools to generate and evaluate it.
This repository is a collection of Claude/Anthropic skills and platform documentation rather than a standalone application. It contains skill definitions (SKILL.md files), MCP server evaluation guidance, managed-agents documentation, and supporting tooling such as schema definitions and build scripts. The dominant character is developer-facing content and reusable patterns for building LLM agents, with some executable scripts for validation and report generation. It provides structured output schemas for evaluation pipelines (grading, benchmark, comparison, analysis), a validator hierarchy for XML/XSD-based skill files, and a security-conscious file I/O example in build-report-lite.mjs. It also documents managed agents, platform availability, and MCP best practices. The repository is not a data research project; there are no notebooks, data pipelines, or model training scripts. Maintenance is active, but technical drift is medium due to version-sensitive SDK references, manually maintained platform availability tables, and multiple language-specific examples that must be kept in sync. The code and schemas are generally well-structured and reusable, though some paths are hardcoded and some documentation is tightly coupled to generator scripts.
This repository is a comprehensive educational cookbook for building with Anthropic's Claude API and Agent SDK. It contains over 60 Jupyter notebooks demonstrating capabilities such as classification, retrieval-augmented generation (RAG), summarization, text-to-SQL, knowledge graph construction, content moderation, tool use, multimodal processing, extended thinking, and agent orchestration. The repo also includes production-oriented hosting patterns (Docker, Kubernetes, Modal, Cloudflare Workers) for deploying agents, and extensive evaluation frameworks using promptfoo with custom Python evaluators (BLEU, ROUGE, LLM-as-judge). While primarily a tutorial collection, it exhibits hybrid characteristics with reusable infrastructure code for agent hosting, content moderation pipelines, and notebook validation. The codebase is actively maintained with CI/CD workflows, but relies on beta APIs and hardcoded model names that introduce technical drift. Data assets are synthetic but realistic, enabling reproducible demonstrations of Claude's capabilities.
This repository is a research evaluation harness for Visual Place Recognition (VPR) models. It wraps a wide range of state-of-the-art VPR methods behind a common CLI, allowing users to run any supported model on standard VPR datasets (e.g., St Lucia) or a bundled toy dataset. The main entry point is test_dataset.py, which loads database/query image folders, extracts descriptors using the selected model, computes recall@N metrics, and saves logs and optional visualizations. The repo is designed to work with the companion VPR-datasets-downloader to standardize dataset format. Although it is a CLI tool, its primary purpose is research evaluation, so it is classified as data_research.
The QwenLM/Qwen3 repository provides resources for the Qwen3 family of large language models, including model cards, documentation, and usage examples. It serves as an application-focused hub for inference, deployment, training, and evaluation of Qwen3 models. The repository contains extensive multilingual documentation, inference demos (web and CLI), deployment guides for various frameworks (vLLM, SGLang, etc.), training examples (using Axolotl, LLaMA-Factory, etc.), quantization techniques, and evaluation harnesses (notably for ARC-AGI). While the core model weights are hosted externally, this repo enables end-to-end usage through well-organized examples and scripts.
JiuwenSwarm is a multi-agent AI orchestration platform that runs as a set of distributed services: an AgentServer (WebSocket-based), a Gateway for multi-channel IM integration, a sandboxed execution environment (JiuwenBox), and a skill development pipeline. It enables teams of AI agents (leader/teammates) to collaborate on complex tasks, with skill retrieval and orchestration via the Symphony system. The system supports multiple frontends: web (React), TUI (terminal), and IM channels (Feishu, WeCom, DingTalk, Telegram, Discord, Slack, WeChat, Xiaoyi). The architecture is modular with an extension system, but many components are tightly coupled and some parts remain incomplete (e.g., SkillDev stages marked as '待实现'). The codebase is actively maintained but shows signs of drift: multiple monkeypatches, hardcoded paths, and a large dependency surface. The core strength lies in its agent orchestration logic, sandbox security, and multi-channel integration, while the skill evaluation and retrieval subsystems are functional but not yet production-ready across all stages.
Paper2Code is a research system that automatically generates runnable code repositories from machine learning research papers. The core pipeline, PaperCoder, uses a multi-agent LLM architecture (planning→analysis→coding) implemented as a sequence of Python scripts orchestrated via shell commands. It accepts papers in PDF (converted to JSON via s2orc-doc2json), JSON, or LaTeX format, and produces a complete Python project with config, model definitions, training loops, and evaluation scripts. The codebase supports both OpenAI API (o3-mini by default) and open-source models via vLLM (e.g., DeepSeek-Coder-V2-Lite-Instruct). It includes a debug stage that iteratively fixes runtime errors using LLM-generated SEARCH/REPLACE patches, and an evaluation module that scores generated repositories (reference-free or reference-based) using LLM judgment. The repository also bundles the Paper2Code benchmark dataset (90 papers from top ML conferences with gold repositories) and prompts for evaluation. Although primarily an application, it has research artifacts (evaluation scripts, prompt templates, benchmark data) that give it a hybrid character.
This repository provides a comprehensive CLI toolkit and programmatic interface (Python SDK and notebook clients) for creating, deploying, managing, monitoring, and evaluating AI agents on Amazon Bedrock AgentCore. The system supports multiple agent frameworks (Strands, LangChain/LangGraph, CrewAI, AutoGen, OpenAI Agents, Google ADK) and infrastructure-as-code options (CDK, Terraform). Key capabilities include project scaffolding via Jinja2 templates, runtime lifecycle management (configure, launch, destroy, invoke), memory management (short-term/long-term, semantic, user preference), observability (CloudWatch-based trace and log querying with OpenTelemetry span parsing), evaluation (LLM-as-a-judge with built-in and custom evaluators), policy engine CRUD, gateway management (MCP, API key, Cognito OAuth), and identity federation setup. The codebase is well-structured with a clear three-layer architecture (CLI -> operations -> services), extensive test coverage, and consistent error handling. However, the project is explicitly deprecated in favor of the @aws/agentcore NPM CLI, meaning future development is minimal and users are encouraged to migrate. Despite the deprecation, the reusable abstractions (e.g., CloudWatch query builder, trace processor, strategy models, configuration schemas) demonstrate solid software craftsmanship.
This repository is a literature survey on recent advances in large language model benchmarks against data contamination, specifically the transition from static to dynamic evaluation. It contains a single README.md file that lists and categorizes numerous research papers, benchmarks, and methods. There is no executable code, data pipelines, notebooks, or any software artifacts. The repository serves as a curated reference list and taxonomy for researchers, but offers no reusable code or data assets. It is not a functional application or a data research project with ETL or model training components.