Khoj is an open-source personal AI assistant functioning as a 'second brain' for interacting with personal knowledge bases. It provides a chat interface allowing users to converse with various AI models about documents, notes, and connected data sources (local files, GitHub, Notion). The system supports multimodal capabilities including image generation, code execution, voice input/output, and computer operation through a modular agent architecture. Khoj features a Python/Django backend with FastAPI for API endpoints, a PostgreSQL database with pgvector for vector storage and search, and multiple frontends including a web application (React/Next.js), desktop client (Electron), mobile app (Android), and plugins for Obsidian and Emacs. The architecture emphasizes separation of concerns with distinct modules for file processing, embedding generation, conversation handling, tool integration, and user management.
This repository is a comprehensive, curated bibliography and knowledge base for the field of active learning. It organizes hundreds of research papers by conference (ICML, NeurIPS, CVPR, etc.), journal (JMLR, TPAMI, etc.), thesis, and content topic (problem settings, deep active learning, applications, etc.). The repository also provides taxonomies, summaries, and links to papers. It is not a software application or data pipeline; it is a structured reference list intended for researchers and practitioners. The content is entirely in Markdown files, making it trivially reproducible and easy to navigate. The repository is actively maintained with recent commits and a growing star count.
GPT Researcher is an autonomous research agent that produces comprehensive, cited reports from web and local sources. In production, it operates as a full-stack web application: a FastAPI backend (main.py, backend/server/app.py) exposes REST and WebSocket endpoints for initiating research, streaming progress, and retrieving reports. A Next.js frontend (frontend/nextjs/) provides the user interface, and a lightweight vanilla JS frontend (frontend/scripts.js) is also available. The core Python package (gpt_researcher/) can be used programmatically via the GPTResearcher class or as a CLI tool (cli.py). Architecture follows a planner-executor pattern: given a research task, the system generates sub-queries, executes them in parallel across 20+ search engine wrappers (Tavily, Google, Bing, Brave, DuckDuckGo, ArXiv, PubMed, etc.), scrapes the resulting URLs using multiple backends (BeautifulSoup, browser automation, PyMuPDF, FireCrawl), filters and compresses the context using embedding or keyword-based methods, and finally generates a structured report with citations. The system supports multiple report types (summary, detailed, deep research) and multi-agent orchestrations (LangGraph, AG2) for more complex workflows. Key data flows: user query to sub-query generation (LLM) to parallel search across configured retrievers to URL scraping to context compression to source curation to report generation (LLM) to output as markdown/PDF/DOCX. Real-time progress is streamed via WebSocket. External dependencies include LLM providers (OpenAI, Anthropic, Google, Ollama, etc.), search APIs (Tavily, SerpAPI, etc.), and optional services (MCP servers, vector stores, image generation APIs). The codebase demonstrates strong engineering practices: extensive test suite (200+ tests), plugin architecture for retrievers, configurable context filters, cost tracking, and evaluation frameworks for quality and hallucination. However, it carries maintenance risks from its large dependency surface, multiple parallel agent implementations, and reliance on rapidly-changing external APIs.
This is a mature, actively maintained AI agent skill called 'last30days' that enables agents (Claude Code, Codex, Grok, Gemini CLI, MCP hosts) to research any topic by aggregating content from 20+ social and news platforms (Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, arXiv, Techmeme, Digg, and more) over a rolling 30-day window. The system is primarily a CLI engine (Python) with a companion Go MCP server for Claude Desktop. It employs a sophisticated pipeline: query planning by intent, parallel retrieval from multiple backends (keyless APIs, paid scrapers, CLI tools, LLM-based search), source-specific normalization, engagement-weighted relevance scoring, cross-source clustering, and deterministic fusion (weighted RRF) with per-author caps and diversity floors. Output is rendered as structured JSON, compact text, or self-contained HTML with optional publishing and Atom feeds. The codebase emphasizes security (credential hygiene, no browser cookie reads in unattended mode), determinism (fixture-based evaluation harness with grounding/recency/coherence scores), and graceful degradation (per-source fallback chains, never-raise error envelopes). While classified as an application (it's a skill/plugin for agent hosts), it exhibits strong data engineering traits: source adapters, normalization pipelines, scoring models, and an offline evaluation framework.
Protenix is a production-oriented PyTorch implementation of AlphaFold 3 for biomolecular structure prediction, maintained by ByteDance. It is best classified as an application rather than a pure research codebase: it ships a `protenix` CLI with commands for prediction, input conversion, MSA/template/RNA-MSA search, and data preparation, plus training/fine-tuning entry points, batch inference runners, Docker deployment, web-service helpers, and PyPI packaging. The code ingests PDB/CIF, FASTA, SMILES/SDF, and JSON descriptors covering proteins, DNA, RNA, ligands, ions, modified residues, covalent bonds, and contact/pocket constraints, then outputs predicted 3D CIF structures with confidence metrics such as pLDDT, PAE, pTM, and ipTM.
ipatool is a cross-platform command-line tool that acts as an unofficial Apple App Store client. It enables searching, purchasing, and downloading iOS/iPadOS/tvOS/watchOS/visionOS/macOS applications and packages directly from Apple's servers. The tool implements Apple's private SAP authentication and signing protocol by emulating Apple's StoreAgent binary in-process using the Unicorn CPU emulator, allowing it to generate kbsync blobs and decrypt macOS packages. It also includes low-level parsers for Apple's proprietary formats (AEA encrypted archives, APFS filesystems, dyld shared caches) to extract assets from macOS restore images. The architecture is modular: a CLI layer (using cobra) dispatches to domain logic in pkg/appstore, which relies on a generic HTTP client (pkg/http), keychain storage (pkg/keychain), and platform abstractions (pkg/util). An MCP (Model Context Protocol) server subcommand exposes the same functionality over stdio. The codebase is heavily tested with unit and integration tests, and CI builds for macOS, Linux, Windows, and iOS. While primarily an application, it contains significant reverse-engineering and protocol-analysis components, giving it a hybrid character.
This repository from Google DeepMind implements a family of point tracking models (TAP-Net, TAPIR, BootsTAP, TAPNext, TAPNext++, TRAJAN) and the associated TAP-Vid and TAPVid-3D benchmarks. It is primarily a research codebase for tracking arbitrary points in video, with both JAX/Haiku and PyTorch implementations. The code includes training pipelines (Jaxline), evaluation scripts, data generation for synthetic (Kubric) and real datasets (DAVIS, Kinetics, ADT, DriveTrack, Panoptic Studio), and interactive Colab demos. While it has application-like components such as live webcam demos and a VOT challenge tracker, the dominant character is research: the code is organized around benchmarking and model experimentation rather than production deployment. The repository is actively maintained with recent commits and provides extensive documentation for reproducing results.
Qlib is an open-source, AI-oriented quantitative investment platform developed by Microsoft. It covers the full pipeline for quantitative research and trading: data ingestion and feature engineering (Alpha158, Alpha360, high-frequency), model training (traditional ML like LightGBM/XGBoost, deep learning in PyTorch including LSTM, GRU, Transformer, GATs, and reinforcement learning), backtesting with realistic costs and slippage, portfolio optimization, and online serving for production-like deployment. The platform is highly modular and config-driven, with a core library (qlib) and a separate contrib layer for community extensions. It includes a CLI (qrun) for running full workflows from YAML configs, extensive experiment tracking via MLflow, and a rich set of evaluation metrics. While primarily a library/application, it also functions as a research benchmark suite with many example models and data collectors for global markets. The codebase is active and well-maintained, though some components (e.g., TFT model, older RL code) show signs of technical drift due to dependency version locks and deprecated patterns.
This repository is a research codebase for training and evaluating draft models for speculative decoding, supporting three algorithms: DSpark, DFlash, and Eagle3. It provides a full pipeline from data preparation (downloading prompts, regenerating answers with a target model via SGLang, and precomputing a large binary cache of target hidden states) to distributed training with FSDP and BF16, and evaluation on standard LLM benchmarks (GSM8K, MATH500, AIME, HumanEval, etc.). The code is well-structured with reusable components for configuration, distributed utilities, checkpointing, metrics, and custom attention masks. While primarily a research artifact, it exhibits strong engineering practices and several modules that could be extracted for other large-scale ML projects. The repository is hybrid in nature: it contains both application-like training/evaluation scripts and data_research elements such as data pipelines and evaluation datasets.
This repository provides the infrastructure for the Anticipatory Music Transformer, a research project for music generation using autoregressive transformers with anticipatory infilling. The codebase covers the full pipeline: preprocessing of MIDI files (Lakh MIDI dataset) into a compound token representation, tokenization into arrival-time or interarrival-time sequences with optional augmentation for anticipation training, and sampling from trained models using custom generation logic. It also includes scripts for human evaluation reproduction and model benchmarking. While primarily a research codebase for a specific paper, the library components in the `anticipation` package (particularly the conversion, ops, and sampling modules) are well-structured and could be reused in other music tokenization or generation projects. The repository does not include the training loop itself but provides configuration files for use with the Levanter training framework.
Favicorn is a command-line OSINT tool that searches for websites by their favicon hashes. It accepts input via favicon URL, file, or domain, computes multiple cryptographic and perceptual hashes (MMH3, MD5, SHA256, etc.), and generates search URLs for 11 security/search platforms (Shodan, ZoomEye, Censys, etc.). Optionally, it fetches and displays preview results from Shodan, ZoomEye, and Netlas using API keys. The architecture features a modular design: the Favicon class handles hash computation and platform-specific URL generation, while specialized Fetcher classes (Shodan, ZoomEye, Netlas) encapsulate API interactions, caching, and response parsing. The CLI orchestrates workflow, parallel execution, and output formatting. Key dependencies include requests, mmh3, dnspython, and platform-specific SDKs.
This repository implements a multi-agent AI system for generating comprehensive company research reports. The backend is a FastAPI application that orchestrates a LangGraph-based pipeline of specialized research agents: a grounding node that optionally crawls the company's website, parallel research nodes (company, industry, financial, news) that use Tavily search and LLM-generated queries to gather data, a curator that filters documents by relevance score, an enricher that fetches raw content, a briefing node that generates category-specific summaries using Gemini, and an editor that compiles the final report using GPT-4o. The frontend is a React application with TypeScript and Tailwind CSS that communicates via SSE streaming. The system is primarily a web application with a modular agent pipeline, not a data analysis project.
Audiocraft is a research codebase from Meta (Facebook Research) for state-of-the-art audio generation, including MusicGen, AudioGen, MAGNeT, JASCO, MultiBand Diffusion, and AudioSeal watermarking. It provides training and inference pipelines for text-to-music, text-to-sound, music continuation, style transfer, and symbolic conditioning (chords, drums, melody). The repository is primarily a research library with extensive configuration-driven training (Hydra/Dora), evaluation metrics (FAD, KLD, CLAP, ViSQOL), and interactive Gradio demos. It is not a production application but a collection of models, solvers, and utilities for audio generation research. The codebase is well-structured with modular components (conditioners, codebook patterns, audio effects) and includes unit tests, but relies on internal datasets and cluster paths that limit full reproducibility. Pretrained models are available via HuggingFace Hub for inference.
TRELLIS.2 is a research codebase for state-of-the-art image-to-3D generation using a novel sparse voxel representation (O-Voxel) and flow matching with 4B parameters. It provides full training and inference pipelines for generating high-fidelity 3D meshes with PBR materials from single images. The repository includes custom CUDA extensions for voxel processing, a flexible sparse tensor library, and comprehensive dataset tooling for large-scale 3D asset curation (Objaverse-XL, Sketchfab). While primarily a research project with training code and data pipelines, it also offers Gradio web demos and Python APIs for inference. The codebase demonstrates strong engineering in its modular design (pipelines, mixins, config-driven training) but suffers from some research-typical issues like code duplication, hardcoded paths, and fragile dependency management.
Local Deep Research is a self-hosted, privacy-focused AI research assistant that performs deep, agentic research using multiple LLMs and web searches. It provides a full-stack web application (FastAPI + JavaScript), a programmatic API, and a Model Context Protocol (MCP) server. The system ingests user queries, decomposes them, executes parallel searches across web and academic sources (SearXNG, Google, arXiv, PubMed, Semantic Scholar, etc.), fetches and extracts content, evaluates evidence, and synthesizes reports with citations. It features per-user encrypted SQLCipher databases, a settings system with environment variable overrides, a RAG library with FAISS vector store, journal quality scoring, news subscriptions, and extensive security controls (egress policy, SSRF protection, credential scrubbing). The codebase is heavily tested with unit, integration, UI, accessibility, and security tests. It is a hybrid of a web application and a research data pipeline, but the dominant character is a production-grade application.
This repository is a massive, multi-disciplinary research codebase from Google Research, primarily focused on machine learning, optimization, and computer science experiments. While it contains a few application-like components (e.g., a Next.js frontend for model exploration, a Rust-based 3D splatting renderer), the dominant character is data_research, hosting thousands of independent experimental projects. The codebase serves as a central hub for publishing reproducible research in domains including Reinforcement Learning, Computer Vision (NeRF, 3D), NLP (Transformers, LLMs), and Privacy. It relies heavily on shell scripts for environment setup and experiment orchestration, with a mix of Python, C++, Rust, and Java implementations. The infrastructure is robust for research (e.g., `scann` for search, `lila` for experiment management) but exhibits significant fragmentation and technical debt due to the rapid pace of research and mixed framework usage.
OASIS is a scalable, open-source framework for simulating social media platforms (Twitter, Reddit) using LLM-powered agents. The system models up to one million agents with realistic actions (post, like, follow, comment, report, etc.) and supports configurable recommendation systems, agent graphs, and time scaling. It is designed for research into information spread, misinformation, group polarization, and herd behavior, as evidenced by extensive experiment scripts and real-world datasets included in the repository. The architecture follows a PettingZoo-style environment loop with a Platform, SocialAgent, AgentGraph, and RecSys modules, all built on top of the CAMEL framework. While the core is a reusable simulation library, the repo also contains research data, visualization scripts, and one-off experiment configurations, making it a hybrid of software application and research project.
Razix is a multi-platform 3D game engine built in C++ with a focus on modern graphics APIs (Vulkan, DirectX 12, OpenGL). It features a custom rendering hardware interface (RHI), a frame graph system for efficient render pass scheduling, an entity-component system (ECS) for scene management, and a built-in editor with ImGui integration. The engine includes a Lua scripting system, physics integration via Jolt Physics, and a comprehensive memory management layer with custom allocators. Despite its ambitious scope and active development, the codebase exhibits a mix of well-engineered subsystems and areas of legacy code (e.g., LIMBO_STATE). The project provides testing frameworks for both engine core and graphics pipelines, and maintains a dedicated game project ('Tanu') as a showcase.
MiniMind-O is a research codebase for training and evaluating a small-scale (0.1B parameter) omni-modal language model that processes text, audio, and image inputs and generates text or streaming speech. The architecture uses a Thinker–Talker dual-path design with frozen external encoders (SenseVoice, SigLIP2, Mimi codec, CAM++) and trainable projectors, plus a Multi-Token Prediction (MTP) head for audio codec tokens. The repository includes a complete training pipeline (SFT with DDP, mixed precision, checkpointing), a comprehensive evaluation CLI, a Gradio web demo, and a real-time Flask WebSocket demo with VAD and ASR for near-duplex interaction. Data is stored in parquet format with pre-encoded audio codes, images, and speaker embeddings. The project is primarily research-oriented, aiming to provide a transparent, reproducible baseline for omni-modal models, but also includes application-like components for interactive demos.
This repository is a curated, actively-maintained personal knowledge base of academic paper reading notes, focused on systems for machine learning, cloud computing, and related areas. It is organized by conference venue (e.g., OSDI, SOSP, MLSys) and by topic (e.g., LLM systems, GPU sharing, memory disaggregation). The core content consists of markdown files with structured summaries (TL;DR, background, design, evaluation) and cross-linked topic aggregation pages. The repository is the source for a GitBook static site and includes scripts for building and serving that site, as well as utilities for PDF text extraction. While not a data pipeline or software application, it functions as a high-quality research resource with consistent formatting, curation guidelines, and reusable templates for note-taking and paper list organization.