Kiln AI is a full-stack, local-first AI development platform for building, evaluating, and iterating on LLM-based systems. It provides a desktop application (FastAPI server + SvelteKit web UI) and a core Python library (published on PyPI) that together enable users to define tasks, run models via LiteLLM across 190+ providers, evaluate outputs with a rich framework (LLM-as-judge, programmatic checks, code evals), fine-tune models, generate synthetic data, manage RAG pipelines (chunking, embedding, vector stores, reranking), and integrate tools (MCP, code tools, skills). The architecture is a monolith with a local server serving a single-page web app, with all data stored on the local filesystem and optionally synced to Git. The codebase is well-structured with clean abstractions, extensive test coverage (unit, integration, e2e with Playwright), and several reusable patterns (sandboxed code execution, git sync middleware, job system, eval plugin architecture). While primarily an application, it includes significant data/research capabilities for synthetic data generation and evaluation.
IMDLBenCo is a comprehensive framework for image manipulation detection and localization (IMLD) research. It provides a unified interface for training, evaluating, and benchmarking deep learning models on standard forensic datasets. The architecture centers around a modular design with registries for models, datasets, and post-processing functions, enabling easy experimentation. Key components include dataset handlers with built-in augmentation and edge mask generation, a distributed training loop with mixed precision support, and evaluation metrics compatible with multi-GPU synchronization. The project serves as both a library for researchers and a reproducible benchmark for state-of-the-art IMLD methods.
Agno is a Python framework for building, orchestrating, and deploying AI agents. It provides a unified abstraction over dozens of LLM providers (OpenAI, Anthropic, Gemini, Groq, DeepSeek, etc.) and supports rich tool integration, multi-agent teams, workflow engines, RAG knowledge bases, persistent session storage, and human-in-the-loop patterns. The architecture centers on core classes: `Agent`, `Team`, `Workflow`, `Knowledge`, and `Toolkit`, all composable through a consistent configuration interface. Agents can be served via the `AgentOS` runtime, which exposes REST, MCP, A2A, Slack, and WhatsApp endpoints with JWT-based auth, scheduling, and observability. The framework includes a comprehensive evaluation suite (`AccuracyEval`, `PerformanceEval`, `ReliabilityEval`), a learning machine for persistent user profiling and memory, and an environments module for reproducible agent rollouts and scorer-based export. While primarily an application framework, it also exhibits data/research characteristics through its evaluation infrastructure, knowledge chunking/embedding pipelines, and systematic experiment tooling. The extensive cookbook (300+ examples) doubles as integration tests, demonstrating patterns ranging from simple single-agent tasks to complex distributed teams and nested workflows. Key entry points include `agno.agent.Agent`, `agno.team.Team`, `agno.workflow.Workflow`, `agno.os.AgentOS` (`agno serve`), and the `agno` CLI. The codebase is large and actively maintained, with a broad integration surface across models, vector databases, storage backends, and third-party tools. Technical drift is moderate, with some experimental model wrappers and rapidly evolving model IDs that may require updates.
Dexter is a CLI-based AI agent framework specifically designed for deep financial research. It implements a sophisticated agent loop that combines task planning, autonomous tool execution, and self-validation to answer complex financial queries. The system features a modular architecture with specialized systems for memory management, tool execution, permissions, and skills. Key capabilities include access to real-time financial data through multiple APIs, web search functionality, file system operations, browser automation, and specialized workflows like DCF valuation. The agent maintains persistent memory across sessions using vector embeddings and keyword search with temporal decay weighting. Dexter can be deployed via CLI or as a WhatsApp bot, and includes an evaluation framework for testing agent performance against financial questions. The system emphasizes security through a robust permission system that controls access to potentially dangerous operations like shell commands.
Cognee is an open-source AI memory platform that enables agents and applications to build, store, and query persistent knowledge graphs from documents, code, and conversations. The system provides a Python SDK, a FastAPI REST API, a CLI, a Next.js frontend, and an MCP server for agent integration. Core operations include ingesting data (remember), extracting entities and relationships via LLMs or GLiNER (cognify), retrieving context (recall), and self-improvement loops (improve). The architecture is modular with adapter/registry patterns for graph databases (Ladybug, Neo4j, Turso), vector stores (LanceDB, PGVector), LLM providers (LiteLLM, OpenAI, Anthropic, Ollama), and integrations (Slack, GitHub, Google Drive). The frontend provides a dashboard for dataset management, knowledge graph visualization, and business analytics. The project also includes an evaluation framework for benchmarking retrieval strategies against standard datasets like HotpotQA and BEAM. While production-grade in many areas, some components are marked as demo-quality or have cloud-only stubs.
OpenAgents Control (OAC) is a framework for managing, testing, and orchestrating AI coding agents. It provides a CLI tool (`oac`) to install, update, and manage agents, context files, and skills from a registry. The system includes a plugin architecture for OpenCode and a compatibility layer that converts agent definitions between OAC, Claude Code, Cursor, and Windsurf formats. A comprehensive evaluation framework tests agent behavior against critical rules (approval gates, context loading, stop-on-failure, delegation, tool usage) using YAML-defined test suites and timeline-based analysis. The repository is primarily a software development toolchain for AI-assisted coding workflows, with strong emphasis on deterministic safety rules and multi-agent coordination.
AssetOpsBench is a unified open framework for building, orchestrating, and evaluating domain-specific AI agents in industrial asset operations and maintenance (Industry 4.0). It provides a suite of MCP (Model Context Protocol) servers that expose domain tools for IoT telemetry, work order lifecycle, failure mode analysis, time-series forecasting/anomaly detection, vibration diagnostics, and utility functions. The framework includes multiple agent runners (plan-execute, Claude, OpenAI, Deep, Stirrup, OpenCode, direct LLM) that consume these tools, and a comprehensive evaluation pipeline with LLM-as-judge and deterministic structured-answer scorers. The system is designed to run reproducible benchmark scenarios with ground truth, producing leaderboard reports for agent comparison. It is accepted at KDD 2026 and powers multiple competitions. Architecturally, the system is a collection of Python packages with 14 CLI entry points defined in pyproject.toml. Each MCP server is a standalone FastMCP application using stdio transport, backed by CouchDB for persistence. Agent runners share a common trajectory model and observability layer (OpenTelemetry with file-based export). The evaluation pipeline is pluggable with a scorer registry. The codebase also includes a significant data/research component: a time-series feature extraction library (130+ extractors), a model catalog with foundation models (TinyTimeMixer, sktime wrappers), synthetic data generation for vibration signals, and showcase notebooks demonstrating end-to-end model lifecycle. The hybrid nature makes it both a software benchmark framework and a research platform for industrial AI.
This repository implements a multi-agent orchestrator system for healthcare workflows, specifically designed for cancer care/tumor board scenarios. The system coordinates specialized AI agents (Patient History, Radiology, Patient Status, Clinical Guidelines, Report Creation, Clinical Trials, Medical Research) via Semantic Kernel group chat, with an Orchestrator agent facilitating collaboration. It provides both Microsoft Teams integration and a React/Vite web interface for user interaction. The architecture includes a Python/FastAPI backend handling WebSocket connections and Bot Framework messages, agent-specific tools for data access and external API integration, and an evaluation framework for assessing agent performance. Infrastructure is provisioned via Azure Bicep templates deploying AI services, storage, compute, and networking resources.
Vizra ADK is a retired Laravel package (final release v0.0.48) that provides a comprehensive framework for building AI agents with LLM orchestration, tool systems, memory, workflows, evaluations, tracing, and MCP integration. It is not a standalone application but a library designed to be installed into Laravel projects. The architecture follows Laravel package conventions: service providers, facades (Agent, Workflow), Eloquent models, Artisan commands, Livewire components, and database migrations. Key entry points include the Agent facade for running agents, the Workflow facade for orchestrating multi-agent workflows, and a set of CLI generators for scaffolding agents, tools, and evaluations. The system supports multiple LLM providers via Prism PHP, persistent memory (session, user, vector), tool chaining, sub-agent delegation, and an evaluation framework with LLM-as-a-Judge assertions. The codebase is well-structured with strong test coverage (Pest + Mockery), but the package is officially retired with no further development, leading to high technical drift as dependencies age out. Despite retirement, many components (e.g., ToolChain, VersionablePrompts, AgentVectorProxy, HasLogging) are self-contained and demonstrate reusable patterns suitable for extraction into new projects.