Khoj is an open-source personal AI assistant functioning as a 'second brain' for interacting with personal knowledge bases. It provides a chat interface allowing users to converse with various AI models about documents, notes, and connected data sources (local files, GitHub, Notion). The system supports multimodal capabilities including image generation, code execution, voice input/output, and computer operation through a modular agent architecture. Khoj features a Python/Django backend with FastAPI for API endpoints, a PostgreSQL database with pgvector for vector storage and search, and multiple frontends including a web application (React/Next.js), desktop client (Electron), mobile app (Android), and plugins for Obsidian and Emacs. The architecture emphasizes separation of concerns with distinct modules for file processing, embedding generation, conversation handling, tool integration, and user management.
Ollama is a local LLM serving platform written in Go. It provides a REST API (OpenAI-compatible), CLI, TUI chat, and desktop application for running large language models locally. The system supports multiple inference backends: llama.cpp (CPU/CUDA/ROCm/Vulkan) and MLX (Apple Silicon). It includes a model conversion pipeline from HuggingFace safetensors to GGUF format, GPU hardware discovery, and an agent framework with tool execution (bash, file operations, web search). The architecture is a monolith with subprocess management for inference backends. Key entry points include the CLI (main.go), HTTP server (server/routes.go), runner binary (cmd/runner/main.go), and desktop UI (app/ui/app/). The codebase is actively maintained with extensive test coverage and support for dozens of model architectures. While primarily an application, it has significant data/research components in model conversion, tokenization, and speculative decoding research.
LLaMA-Factory is a production-grade, unified framework for fine-tuning, evaluating, and serving 100+ large language and multimodal models. It provides a CLI (`llamafactory-cli`), a Gradio web UI (LLaMA Board), and an OpenAI-compatible API server. The architecture is monolithic but highly modular, built on Hugging Face Transformers/TRL/PEFT with optional integration of vLLM, SGLang, DeepSpeed, FSDP, Megatron Bridge, and hardware-specific backends (CUDA, ROCm, NPU, XPU). Training pipelines support full-parameter, LoRA, QLoRA, freeze-tuning, and alignment methods (DPO, PPO, KTO, etc.) with extensive data processing (multimodal, packing, streaming, template-based formatting). The codebase features a plugin system (v1) for kernels, quantization, distributed backends, and batching strategies, though it maintains a parallel legacy (v0) path. Reusable components include dataset processors, chat template hierarchy, multimodal plugin architecture, custom trainer callbacks, and batching strategy plugins. While the framework is actively maintained and well-tested, it suffers from some technical drift due to a dual-codebase structure, hardcoded model registries, version pinning, and fragile monkey-patching in distributed integrations.
GenieX is a production-grade, on-device generative AI inference SDK and CLI developed by Qualcomm for Snapdragon platforms. It provides a C core with a plugin architecture supporting two inference backends: llama.cpp (GGUF models on CPU/GPU/NPU) and QAIRT (pre-compiled bundles on Hexagon NPU). The system includes a Rust-based model manager that downloads, caches, and resolves models from Hugging Face, Qualcomm AI Hub, Docker Hub, and local filesystem, with resumable chunked downloads and progress reporting. It exposes multiple client interfaces: a Go CLI with subcommands for inference, an OpenAI-compatible HTTP server (streaming chat/completions, tool calls), Python bindings with high-level AutoModel factories, Android bindings via JNI/Kotlin, and Go bindings. The architecture is cleanly layered: C SDK → plugin backends → hardware, with a separate model manager process. The codebase demonstrates strong engineering practices: extensive test suites (unit, integration, device-level), CI/CD pipelines, versioned releases, and cross-platform support (Linux ARM64, Windows ARM64, Android). While primarily an application, it includes benchmark infrastructure and quality evaluation tools that could be used for research purposes.
This repository provides a comprehensive toolkit for Chinese Llama-3 models, including base and instruction-tuned versions. It offers scripts for pre-training, fine-tuning, model merging, inference (via Hugging Face transformers, llama.cpp, Ollama, and OpenAI API-compatible server), and evaluation on standard benchmarks (CEval, CMMLU, MMLU, LongBench). The repository follows a modular architecture where each functionality is implemented in separate scripts under the scripts/ directory. Key entry points include training scripts (run_clm_pt_with_peft.py, run_clm_sft_with_peft.py), inference scripts (inference_hf.py, llama_cpp/chat.sh, openai_api_server.py), and evaluation scripts (ceval/eval.py, cmmlu/eval.py, etc.). The model supports LoRA adapters for efficient fine-tuning and provides merging utilities to combine base models with LoRA weights. The code demonstrates good practices in handling model quantization, PEFT integration, and multi-platform deployment.
This repository is a comprehensive collection of tutorials and code examples for deploying, fine-tuning, and integrating open-source large language models (LLMs), with a strong focus on the Chinese LLM ecosystem. It covers over 50 models (Qwen, ChatGLM, DeepSeek, InternLM, etc.) and provides step-by-step guides for setting up FastAPI services, Streamlit web demos, LangChain knowledge base assistants, and LoRA/QLoRA fine-tuning using PEFT. The repository also includes example applications such as role-playing chatbots and digital persona creation. While primarily educational, the code patterns are directly applicable to production scenarios, including model serving with vLLM/SGLang, experiment tracking with SwanLab, and deployment on diverse hardware (NVIDIA, AMD ROCm, Ascend NPU, Apple Silicon). The codebase is highly repetitive across models, with significant copy-paste, but the underlying patterns (fine-tuning pipeline, deployment scaffold, LangChain wrapper) are well-established and reusable.