Khoj is an open-source personal AI assistant functioning as a 'second brain' for interacting with personal knowledge bases. It provides a chat interface allowing users to converse with various AI models about documents, notes, and connected data sources (local files, GitHub, Notion). The system supports multimodal capabilities including image generation, code execution, voice input/output, and computer operation through a modular agent architecture. Khoj features a Python/Django backend with FastAPI for API endpoints, a PostgreSQL database with pgvector for vector storage and search, and multiple frontends including a web application (React/Next.js), desktop client (Electron), mobile app (Android), and plugins for Obsidian and Emacs. The architecture emphasizes separation of concerns with distinct modules for file processing, embedding generation, conversation handling, tool integration, and user management.
The repository provides a comprehensive ecosystem for the MiniCPM-V and MiniCPM-o series of multimodal large language models (MLLMs). It enables efficient on-device inference and real-time full-duplex multimodal interaction (vision + audio + text) with low latency. The core contributions include the OmniLMM architecture with intra-ViT early compression for visual encoding, mixed 4x/16x visual token compression, and support for streaming audio/video inputs. The system includes evaluation frameworks for multimodal benchmarks (MMBench, Video-MME, OmniDocBench), fine-tuning utilities (LoRA and full model), and deployment tools for various platforms (iOS, Android, HarmonyOS, web) and inference backends (vLLM, SGLang, llama.cpp, Ollama). Key entry points are the model inference code in omnilmm/model/, evaluation scripts in eval_mm/vlmevalkit/, fine-tuning in finetune/, and web demos in web_demos/ demonstrating real-time chat with multimodal inputs.
PyOD is a comprehensive Python library for anomaly/outlier detection offering 60+ detectors across tabular, time series, graph, text, image, and audio modalities. It provides a classic scikit-learn-compatible API (fit/predict/decision_function) via pyod.models, a high-level orchestration engine (ADEngine) for automatic profiling, detector selection, consensus scoring, and reporting, and an agentic workflow layer for LLM-driven anomaly detection through CLI, MCP server, and packaged AI skills. The library is primarily a software library (application) with strong data research influence: it comes with benchmark-backed routing rules, academic references, and Jupyter notebooks for research reprocibility. Its architecture follows a three-layer design (classic detectors, ADEngine orchestration, agentic skills) that allows progressive engagement from simple model fitting to full automated pipelines.
AgentScope is a full-stack, production-grade multi-agent orchestration platform. The core library (`src/agentscope/`) provides a FastAPI-based agent service with CRUD endpoints for agents, sessions, credentials, channels, knowledge bases, schedules, and SOPs. It features a modular middleware onion pattern for agent lifecycle hooks (reply, reasoning, acting, tool execution), a pluggable storage abstraction (Redis, SQL with Alembic migrations, S3 blob store), and a message bus (in-memory or Redis) for inter-session communication and distributed coordination. The system supports real-time voice (WebRTC, DashScope/OpenAI/Gemini/xAI), RAG pipelines with document parsing and vector stores (Qdrant, Milvus, MongoDB, Elasticsearch), channel integrations (Feishu, DingTalk, Discord), and sandboxed workspace backends (Docker, Kubernetes, E2B, Daytona, Apple Containers, Bubblewrap, OpenSandbox). A full React/TypeScript web UI (`examples/web_ui/frontend`) provides chat, channel management, schedule calendar, knowledge base browsing, and MCP hub browsing. Additionally, a terminal console and a goal-oriented pipeline are provided for developer workflows. The framework emphasizes transparency, controllability, and separation between agent logic and infrastructure, supporting multi-agent collaboration, human-in-the-loop permissions, and long-term memory via ReMe and mem0 middleware.
Hermes Agent is a self-improving AI agent platform with a sophisticated data architecture centered around session management, memory systems, and tool execution pipelines. The primary data lineage flows from user interactions (CLI, TUI, messaging platforms) through a gateway layer that manages session state, tool execution, and model communication. Raw conversation data is ingested via multiple transports (stdio, WebSocket, HTTP) and persisted in SQLite databases (state.db for sessions, kanban.db for task orchestration). The system transforms conversations through context compression, memory extraction (via plugins like Hindsight, Mem0, Honcho), and skill-based procedural memory. Analytical schemas expose session analytics, token usage, model performance, and skill effectiveness through REST endpoints and a web dashboard. The repository answers research questions about autonomous agent behavior, context management strategies, and multi-modal tool orchestration patterns.
Pollinations is a large monorepo hosting a multi-modal generative AI platform. It provides a unified API gateway (gen.pollinations.ai) for text, image, video, audio, 3D, and embeddings generation, routing requests to dozens of providers (OpenAI, Anthropic, Google, Fal, Replicate, ElevenLabs, etc.) with fallback chains, caching, rate limiting, and billing. A separate enterprise service (enter.pollinations.ai) handles authentication (GitHub OAuth, API keys, device code flow), user dashboards, API key management, Stripe/Pollen billing, community-hosted model endpoints, and a quest/reward gamification system. The platform also includes a design system UI library (@pollinations/ui), a Node.js SDK, a CLI tool (polli-cli), an MCP server, and over 20 community apps (chat, AI dungeon master, image reimagine, Discord bots, etc.). Infrastructure includes Cloudflare Workers, Durable Objects, D1, R2, Tinybird for analytics, and GPU inference servers (Flux, Dreamshaper, Z-Image) deployed on Vast/IO.net. The architecture is service-oriented within a monorepo: each major service (gen, enter, media, pollinations.ai) is a separate Cloudflare Worker with its own wrangler.toml and dependencies. Shared packages (shared/, packages/) provide cross-cutting concerns like billing logic, model registry, auth middleware, HTTP error handling, and UI components. Data flows from user requests through middleware (auth, balance, cache, rate-limit, safety, model resolution) to provider-specific adapters, with usage events published to Tinybird for analytics and billing. The system is heavily tested with Vitest, including integration tests with Cloudflare Workers test environment and VCR snapshots. External dependencies include Cloudflare (Workers, D1, R2, KV, Durable Objects, Vectorize), Tinybird (analytics), Stripe (payments), GitHub (OAuth, quests), and numerous AI model providers. The codebase is predominantly TypeScript with some Python (Floret agent, GPU servers, social automation) and Rust (polli-core for Discord bot). While the core platform is well-maintained, there is significant drift in community apps and infrastructure scripts, and the sheer number of provider integrations creates maintenance surface area.
Xinference is a production-grade model serving platform that provides OpenAI-compatible REST APIs for a wide range of AI models including LLMs, embeddings, rerank, image generation, video, audio/TTS, and world models. It supports multiple inference backends (vLLM, SGLang, transformers, MLX, llama.cpp) and can be deployed as a single process or a distributed cluster using an actor-based supervisor/worker architecture (xoscar). The system includes a web UI for model management, authentication/authorization with OAuth2 scopes, token-aware routing for load balancing, and monitoring capabilities. It bundles many third-party model implementations for turnkey deployment, which reduces setup friction but introduces maintenance overhead from vendored code. The architecture is a hybrid of a monolithic FastAPI server and a distributed actor system. Key entry points are CLI commands (xinference, xinference-supervisor, xinference-worker) and the REST API. The frontend is a Next.js application statically exported and served by the backend. The codebase demonstrates strong patterns in modular route registration, plugin-style model registries, and streaming response handling, but also carries significant technical debt from heavy vendoring, duplicated code across third-party integrations, and version-sensitive patches to upstream libraries like vLLM.
ModelScope is a unified framework for model inference, training, and deployment across computer vision, NLP, audio, multi-modal, and scientific domains. It provides a registry-based architecture where models, pipelines, preprocessors, trainers, and metrics are registered by task and can be instantiated from configuration or directly via Python APIs. The framework integrates tightly with the ModelScope Hub for model and dataset distribution, supporting versioned model IDs and snapshot downloads. Key entry points include the `pipeline()` function for inference, `build_trainer` for training, and CLI tools for cache management and server deployment. The architecture is modular with well-defined base classes (TorchModel, Pipeline, Preprocessor, Metric, Hook) and a configuration system that drives component construction. External dependencies span PyTorch, TensorFlow, HuggingFace Transformers, MMDetection, Diffusers, and numerous domain-specific libraries. The codebase is large and actively developed, but shows signs of technical drift through duplicated research code, skipped tests, and legacy TensorFlow 1.x components.
The Multimodal Universe is a large-scale astronomical data repository that aggregates ~30 surveys (Legacy Survey, HSC, JWST, Gaia, SDSS, DESI, TESS, etc.) into a unified, HEALPix-partitioned HDF5 format, exposed through Hugging Face datasets. It covers multiple modalities: images (multi-band), spectra (1D), time series (light curves), and tabular catalogs. The repository provides dataset builder scripts (GeneratorBasedBuilder subclasses) per survey, cross-matching utilities, and benchmark training code (PyTorch Lightning) for downstream tasks like photometric redshift estimation and galaxy property regression. While primarily a data research project, it also includes application-like components such as baseline classifiers (Informer for PLAsTiCC) and experimental benchmark pipelines. The data is designed for training foundation models in astronomy, with streaming support and standardized column naming. Notable patterns include HEALPix-based spatial indexing, parallel processing in build scripts, and a consistent Hugging Face dataset interface across all surveys. The codebase is well-structured for its purpose, though many scripts are tightly coupled to specific surveys and external data sources, limiting direct reuse without adaptation.
Caesar.jl is a Julia package for factor graph-based simultaneous localization and mapping (SLAM) and state estimation. It provides a framework for building factor graphs with variables (poses, landmarks) and factors (constraints), solving via incremental inference (iSAM2) and Bayes tree, and retrieving beliefs for non-Gaussian and multimodal distributions. The library supports integration with various sensors (AprilTags, lidar, sonar, GPS, IMU) through custom factors and middleware (ROS, LCM, ZMQ). Examples demonstrate applications in marine (ASV, AUV, ROV), wheeled (racecar, Victoria Park), tracking (acoustic, radar), and learning (neural network factors) domains. The architecture is modular, with core packages (RoME, IncrementalInference, DistributedFactorGraphs) handling specific aspects of the SLAM pipeline.
This repository contains the research code for Video-SALMONN-2, a family of audio-visual large language models for video understanding, developed by Tsinghua University and ByteDance. It includes three generations: Video-SALMONN-2 (based on LLaVA), Video-SALMONN-2+ (based on Qwen2.5-VL), and Video-SALMONN-2 Pro (based on Qwen3-VL). The core contribution is a caption-enhanced training paradigm that tightly aligns audio and visual signals. The codebase provides full training pipelines for supervised fine-tuning (SFT) and direct preference optimization (DPO), along with evaluation scripts, a demo mode, and a vLLM-based serving plugin for deployment. While the project is predominantly research-focused, it includes application-like components such as CLI inference and OpenAI-compatible serving. The code exhibits high duplication across the three model variants, hardcoded paths to internal cluster storage, and brittle dependency pinning, making it challenging to reproduce outside the original environment. However, several components—such as the multimodal batch sampler, audio-video processing utilities, and QFormer-based fusion—are well-structured and potentially reusable for other multimodal research projects.