thehardwarestore.ai

MLflow is an open-source ML lifecycle platform that provides experiment tracking, model registry, model deployment, evaluation, and observability for both classical ML and GenAI/LLM workflows. In production, it operates as a modular monolith: a Python backend (Flask/FastAPI) with SQLAlchemy-backed storage (SQLite, PostgreSQL, MySQL, MSSQL), a React/TypeScript web UI, and client SDKs for Python, R, Java, and TypeScript. The system exposes a REST API and CLI for logging experiments, registering models, collecting traces via OpenTelemetry, and managing an AI Gateway that routes requests to dozens of LLM providers with budget controls, guardrails, and traffic splitting. The architecture includes a plugin system for custom store backends, deployment targets, and authentication providers, as well as a custom linting tool (clint) and extensive CI/CD pipelines. Key entry points are the `mlflow` CLI, `MlflowClient` Python API, tracking server (`mlflow server`), gateway server (`mlflow gateway`), and model scoring server (`mlflow models serve`). Data flows from user instrumentation through asynchronous logging to the backend store, with traces and evaluations queryable via the UI or API. External dependencies span ML frameworks (scikit-learn, PyTorch, TensorFlow, LangChain, etc.), LLM provider SDKs, and infrastructure tools (Docker, Kubernetes, Helm).

Langfuse is an open-source LLM observability and evaluation platform. It provides tracing, prompt management, evaluations, datasets, experiments, monitors, and a public API for integrating with AI applications. The architecture is a monorepo with pnpm workspaces and Turborepo, consisting of a Next.js frontend (web), a background worker (Express + BullMQ), a shared package with types and server utilities, and an enterprise edition. Data flows from ingestion (REST API, OpenTelemetry, SDKs) into ClickHouse for high-volume analytics and PostgreSQL for relational metadata. Background jobs handle evaluations, batch exports, cloud billing, and integrations. The codebase is well-structured with extensive testing, a custom ESLint plugin, and reusable UI components. It is actively maintained with frequent migrations and feature additions.

Kiln is a full-stack AI development platform for building, evaluating, and optimizing LLM-based tasks and agents. It provides a desktop application (Tkinter tray + FastAPI server) with a Svelte web UI, plus a core Python library and a cloud server component. The system supports the complete AI workflow: project/task management, multi-provider LLM orchestration via LiteLLM, evaluation with multiple judge types (LLM-as-judge, code eval, exact match, pattern match, etc.), synthetic data generation, RAG pipelines (chunking, embedding, vector stores), fine-tuning, prompt optimization, and Git-based project versioning. It also includes a 'Kiln Pro' cloud service for advanced features like copilot-assisted spec refinement and batch generation. The architecture is a monorepo with uv workspace, featuring extensive test coverage (unit, integration, e2e) and CI/CD pipelines. While the primary character is an application, it incorporates significant data engineering patterns for evaluation pipelines, synthetic data generation, and statistical analysis.

VoltAgent is a large TypeScript monorepo for building, deploying, observing, and evaluating AI agents. The core package provides an agent runtime with tool calling, memory, subagents, guardrails, workflow orchestration, MCP client integration, A2A protocol support, voice, and OpenTelemetry-based observability. Framework adapters expose this runtime as HTTP/SSE/WebSocket servers via Hono, Elysia, and serverless function providers, while a CLI scaffolds projects and manages VoltOps deployment/auth flows. The repo is fundamentally an application framework, though it contains hybrid data-oriented subsystems: a RAG package with many document chunking strategies, vector adapters, and BM25 search, plus evaluation/scoring packages used for LLM benchmark experiments.

Typed Japanese is a TypeScript type-level library that models Japanese grammar as composable generic types, enabling the construction of grammatically valid Japanese sentences entirely within the type system. The core library defines interfaces for verbs (godan, ichidan, irregular), adjectives (i-adjectives, na-adjectives), copulas, particles, and sentence patterns, using template literal types and conditional types to encode conjugation rules. This allows developers to write Japanese sentences as type aliases that resolve to the exact Japanese string, with the TypeScript compiler acting as a grammar validator. The repository also includes an interactive playground web application (React + Vite + Monaco Editor) where users can explore the type system, write and analyze sentences, and follow a bilingual (English/Chinese) tutorial course spanning 47 chapters from elementary to advanced levels. A macOS app (TypedTranslate) provides a native interface for grammar annotation and chat, using a Bun bridge to integrate with the type system and LLMs (Codex/Claude). The project is hybrid in nature: while primarily an application/library, it includes a substantial data/research component in the form of curated evaluation datasets, an LLM annotation pipeline, and a rigorous iterative eval loop that measures conformance, reliability, and capability across grammar categories. The eval system produces quantifiable scoreboards and drives systematic improvements to the library's type definitions, prompt templates, and tutorial content.

garak is a CLI tool and Python library for automated red-teaming and vulnerability scanning of large language models (LLMs). It systematically probes models with adversarial inputs (jailbreaks, prompt injections, encoding attacks, etc.) and uses detectors to classify model responses for safety failures. It supports multiple model backends (OpenAI, Anthropic, HuggingFace, etc.) and generates structured reports with severity scoring. The system follows a modular plugin architecture with base classes for probes, detectors, generators, harnesses, evaluators, and buffs. Plugins are dynamically loaded and configured via a hierarchical config system. The default harness runs probes with their associated detectors, while a cross-product harness allows all probes against all detectors. Results are logged as JSONL and can be visualized via a React-based web UI (garak-report) that displays DEFCON-level severity, calibration Z-scores, and technique-intent matrices. While primarily an application, garak incorporates significant research elements: it implements state-of-the-art attack techniques (GCG, AutoDAN, BEAST, TAP) and includes curated datasets of adversarial prompts and calibration data. The codebase serves both as a practical security tool and as a research platform for LLM vulnerability assessment.

Prompt Optimizer is a multi-platform AI prompt optimization tool that helps users craft better prompts for large language models (LLMs) and image generation models. The system supports web (Vue 3 + Naive UI), desktop (Electron), Chrome extension, MCP server, and Docker deployment via a pnpm monorepo. Core workflows include system/user prompt optimization, evaluation with structured compare (pair-judge, synthesis), image generation (text2image, image2image, multi-image), variable extraction/value generation, a template system with 40+ built-in bilingual (zh/en) templates, and smart favorites with resource-aware asset management. The architecture cleanly separates framework-agnostic core services (LLM adapters, evaluation, storage, template processing) from UI composables and Electron proxies, using adapter, registry, and factory patterns extensively. A notable strength is the comprehensive test infrastructure spanning unit (Vitest), integration (real LLM API helpers gated by environment flags), and E2E (Playwright with VCR record/replay). Weaknesses include significant template duplication (bilingual files maintained separately), IPC fragility between Electron main/renderer, provider adapter maintenance overhead (16+ text adapters, 6+ image adapters), and some dead/deprecated code paths carrying legacy migration layers.

LLM Foundry is a production-grade framework for training, finetuning, evaluating, and serving large language models, with a focus on the MPT architecture and HuggingFace integration. It provides a full pipeline from data preparation (MDS streaming), to model definition (custom MPT, HF wrappers, embedding models), to distributed training (Composer, FSDP, TP), and to evaluation (ICL benchmarks, gauntlets). The architecture is modular, using YAML-driven configuration and a central registry pattern for extensibility. The codebase includes dozens of callbacks for monitoring and training stabilization, custom optimizers, and efficient data collation (bin packing). While tightly integrated with MosaicML's Composer and platform, the core registry and layer abstractions are reusable. The repository is actively maintained with comprehensive test suites, though some features remain experimental and several dependencies (flash attention, megablocks) require careful version management.

SIA (Self-Improving AI) is a framework that orchestrates a three-agent loop (meta-agent, target agent, feedback/improvement agent) to iteratively improve an AI system's performance on a benchmark task. In production, it runs as a CLI tool (`sia run`) that sets up a virtual environment, loads task files, and executes generations: the meta-agent reads `task.md` and generates `target_agent.py`, the target agent executes the task and logs results, and the feedback agent reviews the execution and rewrites the target agent for the next generation. A FastAPI web dashboard auto-launches to visualize per-generation accuracy, agent code, trajectories, and improvement plans. The architecture is a CLI-driven monolith with a plugin system for agent backends (Claude, OpenHands, PydanticAI), JSON-based provider/profile configuration, and Docker sandboxing for secure target-agent execution. It relies on external LLM APIs (Anthropic, OpenAI, Gemini, Nebius, OpenRouter, Together) and optionally on Modal/SandboxFusion for the RL-based weights mode. The repository also bundles four benchmark tasks (GPQA, lawbench, longcot-chess, spaceship-titanic) with public/private data splits and evaluators, giving it a hybrid character that includes data/research components. However, its primary function is as an application for running and visualizing self-improvement experiments.

Promptimize is a prompt engineering evaluation and testing toolkit implemented as a Python library with CLI interface. It enables users to define prompt cases (test cases for LLM prompts) as code, associate evaluation functions that score responses on a 0-1 scale, execute test suites across different models and configurations, and generate performance reports. The architecture centers around BasePromptCase and its derivatives (PromptCase, TemplatedPromptCase, LangchainPromptCase) which encapsulate prompt logic and evaluation, Suite classes that manage collections of prompt cases and handle execution/reporting, and utility modules for templating, caching, and result serialization. The toolkit minimizes redundant API calls by caching prompt executions and only re-running when prompts change, supports human review for edge cases, and integrates with Langchain/OpenAI for LLM interactions.

Youtu-Agent is a comprehensive AI agent framework developed by Tencent Cloud's Youtu Lab, designed for building, orchestrating, and evaluating LLM-powered agents. It provides a modular toolkit system, multi-agent orchestration patterns (simple, orchestra, orchestrator, workforce), a reusable configuration system (Hydra+YAML), and support for various environments (local shell, Docker, E2B sandbox, browser). The framework also includes a research-grade evaluation pipeline with benchmarks like GAIA, SWE-bench, WebWalkerQA, and a training-free GRPO module for agent self-improvement through experience distillation. Two front-end applications accompany the backend: a React-based chat UI and a Next.js evaluation analysis dashboard, both communicating via WebSocket and REST APIs. While the primary character is an application framework, Youtu-Agent has a strong data/research dimension: it ingests benchmark datasets, runs model-based judges, stores trajectories in SQL/PostgreSQL, and implements a meta-learning loop that extracts 'experiences' from agent behaviors. This hybrid nature makes it suitable both for production agent deployment and for conducting RL/benchmarking experiments. The codebase is well-organized into distinct modules (agents, tools, config, eval, practice, tracing, utils), uses modern Python packaging (uv, pyproject.toml), and includes extensive example configs and scripts. External dependencies include the openai-agents SDK, various LLM providers (OpenAI, Anthropic, etc. via litellm), search APIs (Serper, Jina, Google), document parsers (Chunkr, Unstructured), and MCP servers. Deployment supports Docker and multiple environment types. The repository is actively maintained with CI/CD, documentation (mkdocs), and a growing set of features.

QwenLM/Qwen-AgentWorld is a research benchmark evaluation toolkit for Language World Models (LWMs). The repository contains the AgentWorldBench evaluation infrastructure: a standalone CLI script (`eval/eval.py`) that orchestrates a two-stage evaluation pipeline — first calling an OpenAI-compatible inference endpoint to generate environment predictions from a world model, then using an LLM judge (e.g., GPT) to score the predictions across five dimensions (format, factuality, consistency, realism, quality). The code covers seven agent domains (MCP, Search, Terminal, SWE, Android, Web, OS) through per-domain prompt templates and task configurations. The repository is not a model serving system or training framework; it is a reproducible evaluation harness published alongside the Qwen-AgentWorld model weights and benchmark dataset (hosted separately on Hugging Face). The core value is in the robust parsing utilities for extracting structured output from LLM responses, including sophisticated handling of think/response tag removal and multi-strategy JSON extraction from judge outputs.

Agentic Security is an open-source LLM vulnerability scanner that probes AI endpoints for security weaknesses such as jailbreaks, prompt injection, data leakage, and sandbox escapes. It provides both a FastAPI web server (port 8718) with a Vue.js frontend and a Fire-based CLI (server, ci, init, ls, scan). The core scan engine sends adversarial prompts (text, image, audio) to target LLMs via configurable HTTP specs, classifies responses using a multi-layered refusal detection system (phrase-based, OneClassSVM, PII detector, LLM-as-judge, hybrid voting), and generates detailed reports with failure rates and cost metrics. The architecture is modular: LLM provider abstraction (OpenAI, Anthropic, LiteLLM), resilience primitives (rate limiter, circuit breaker, concurrent executor), plugin-based refusal classifiers, and a dataset registry supporting HuggingFace, CSV, Google Sheets, and dynamic transformations (stenography, encoding). It also integrates external red-teaming tools (Garak, Inspect AI) and includes reinforcement learning and Bayesian optimization for adaptive prompt selection. While primarily an application, it contains significant data/research components (ML models, datasets, refusal classification research) that support its scanning mission.

DeepTeam is a production-oriented Python library and CLI for red-teaming LLM applications. It does not merely document vulnerabilities; it executes automated adversarial campaigns against a user-supplied model callback, generating attack prompts through LLM-driven templates (single-turn, multi-turn, tree/crescendo jailbreaks), scoring target responses with LLM-as-judge metrics, and aggregating outcomes into risk assessments with CVSS scoring. It also ships a code scanner for static AI-security review, a trace scanner for agent executions, and deployable input/output guardrails. Main entry points are the red_team() Python API and the deepteam CLI (login, run, scan), with YAML configuration driving model selection, vulnerability types, and attack budgets. Architecturally, the repo is a modular Python package built on deepeval: vulnerabilities inherit from BaseVulnerability, all metrics inherit from BaseRedTeamingMetric, attacks implement configurable templates and schemas, and frameworks (OWASP, MITRE ATLAS, NIST, EU AI Act, BeaverTails, Aegis) map risks to pre-built categories. Results flow through RTTestCase objects and can be saved locally as JSON or pushed to the Confident AI platform. The library is dual sync/async and includes progress reporting, cost accumulation, and retry logic. Although it embeds research-style synthetic data generation and dataset-driven framework integrations, the dominant character is an application library/CLI rather than a data research project. Strengths include broad vulnerability and attack coverage, clear base-class contracts, a substantial test suite, and active CI. Weaknesses include heavy copy-paste duplication across 30+ vulnerability, metric, and template modules, tight coupling to deepeval internals, reliance on hardcoded model defaults such as gpt-3.5-turbo-0125, and a known logic bug where is_successful in the metric base class compares instead of assigns, always returning False unless an error is set. Telemetry code also contains hardcoded service keys. Overall, it is a functional, well-documented framework with real reuse value, but it would benefit from deduplication and dependency hardening.

MMSearch is a framework for evaluating large multimodal models (LMMs) as multi-modal search engines. It provides a pipeline (MMSearch-Engine) that enables any LMM to perform multimodal search by generating queries, retrieving web content, reranking results, and summarizing answers. The repository includes evaluation scripts for four tasks (end2end, requery, rerank, summarization) and a demo for running the engine on custom queries. The system relies on external dependencies like DuckDuckGo search, Playwright for web scraping, and Hugging Face datasets. The architecture is modular with separate components for model loading, prompt templating, web retrieval, content processing, and scoring.

This repository is a curated collection of 93+ production-ready AI engineering projects, spanning beginner to advanced levels. It serves as a practical resource for building AI applications, with a strong emphasis on agent orchestration (CrewAI, AutoGen, OpenAI Swarm), Retrieval-Augmented Generation (RAG) pipelines, Model Context Protocol (MCP) servers, and LLM evaluation frameworks. The dominant character is application development: most projects are deployable apps using Streamlit, Next.js, FastAPI, or as standalone MCP servers. Notable sub-projects include a full visual workflow builder (open-agent-builder) with Convex backend, a multi-modal RAG pipeline (notebook-lm-clone), and a comprehensive audio analysis toolkit. While primarily application-focused, the repository also contains data/research elements such as model fine-tuning scripts (GRPO, DPO, SFT), synthetic data generation (SDV), and LLM comparison dashboards with Opik/DeepEval evaluation. The codebase is heterogeneous in quality: some projects exhibit clean architecture and reusable patterns, while others are simple demos with boilerplate code and hardcoded dependencies. Overall, it is an excellent educational resource for AI engineers seeking practical, runnable examples, but not a single cohesive product.

This repository is a lightweight, reference implementation of several standard LLM benchmarks (MMLU, GPQA, MATH, HumanEval, DROP, MGSM, SimpleQA, BrowseComp, HealthBench) developed by OpenAI. It provides a unified evaluation framework based on an `Eval` base class and `SamplerBase` that supports OpenAI Chat Completions API, Anthropic Claude API, OpenAI Responses API, and reasoning models (o1, o3, o4-mini). The main entry point is `simple_evals.py` which accepts command-line arguments to select models and evals, runs them, and outputs HTML reports and JSON metrics. The code is cleanly factored into per-benchmark modules, a shared utility module (`common.py`), and sampler adapters. The framework is designed for reproducibility of OpenAI's published benchmark numbers, with all data loaded from public Azure blob storage URLs. The repository is deprecated as of July 2025 and will not receive updates for new models or benchmarks, though it will continue to host reference implementations for HealthBench, BrowseComp, and SimpleQA.