$ show SPThole/CoexistAI
CoexistAI is a research assistant framework implemented as a FastAPI application that provides a suite of API endpoints for web search, YouTube and Reddit retrieval, geospatial mapping, git repository analysis, text-to-speech/podcast generation, and knowledge base creation from local files or crawled websites. It integrates with an MCP server for agentic workflows and supports pluggable LLMs (Google, OpenAI, Ollama, local) and embedders (Google, HuggingFace, Infinity). The architecture is monolithic with modular utility functions, containerized via Docker with a companion SearxNG instance for privacy-respecting search. The codebase demonstrates a hybrid nature: while primarily an application with explicit API entry points, its core purpose is to support research and data-gathering tasks, making it equally relevant for data research scenarios. Configuration is centralized, and the system includes a web-based admin UI for runtime parameter changes. Active development is evident from recent commits and numerous utility modules, though some global mutable state and fallback-heavy code indicate where the system has evolved rapidly.
The LLM identified 3 files worth highlighting for reuse:
WebSearchProfiler class provides a decorator for timing and profiling async functions, tracking execution duration, URL processing counts, and estimated time saved versus manual reading. It uses asyncio.Lock for thread safety and can be extracted and reused in any async Python project needing lightweight performance instrumentation without external dependencies.
A set of functions for converting HTML pages, PDFs, and image files into clean markdown text. It handles various content types via libraries like markdownify, pdftotext, and OCR (via PIL/pytesseract). The functions are self-contained and can be isolated for use in data ingestion pipelines or ETL processes that require uniform text extraction from mixed document formats.
Implements a structured approach to generating answers by decomposing user queries into sub-questions, retrieving relevant context for each, and synthesizing a final response using LLM calls. It uses iterative summarization and cross-validation of generated text. This pattern can be extracted into standalone libraries for question answering over retrieval augmented generation (RAG) pipelines, though it currently references a configurable LLM factory that would need to be adapted.