$ show 666ghj/BettaFish
BettaFish (åūŪč) is a multi-agent public opinion analysis system that automates the end-to-end process of gathering, analyzing, and reporting on public sentiment across 30+ Chinese social media platforms. The system is built around four specialized agents: QueryEngine (news breadth search via Tavily), MediaEngine (multimodal search via Bocha), InsightEngine (database mining via SQLAlchemy), and ReportEngine (template-based report generation with intermediate representation and HTML/PDF/Markdown rendering). A ForumEngine coordinates agent collaboration through log monitoring and LLM-generated moderator comments. The architecture is a monolithic Flask application that manages multiple Streamlit sub-applications and CLI tools, with a separate MindSpider crawler subsystem for data collection. While primarily an application, it includes significant data/research components such as sentiment analysis model training scripts (BERT, GPT-2, Qwen3, traditional ML), a Weibo sentiment dataset, and data pipeline scripts. The system relies heavily on external LLM APIs (OpenAI-compatible), search APIs (Tavily, Bocha, Anspire), and system dependencies (Playwright, WeasyPrint). The codebase demonstrates solid engineering with reusable patterns like robust JSON parsing, chart validation/repair, and a hardened GitHub Actions star-history script, but suffers from significant code duplication across engines and tight coupling to external APIs.
The LLM identified 5 files worth highlighting for reuse:
A robust JSON parser that handles common LLM output errors: markdown code block wrapping, missing commas, unbalanced brackets, trailing commas, and single-quote strings. Uses a multi-strategy approach: regex cleanup, local syntax repair, json_repair library, and optional LLM-based repair as last resort. Self-contained with minimal dependencies, making it easily extractable for any project that needs to parse potentially malformed JSON from LLM responses.
A Chart.js data validator and repairer that checks for required fields, data type correctness, and structural validity of chart configurations. Implements a local repair strategy (fixing common issues like missing datasets, invalid colors, incorrect axis types) before falling back to LLM-based repair. Results are cached to avoid redundant API calls. The pattern of 'validate locally, repair with LLM if needed' is reusable for any data validation task where the schema is well-defined but inputs may be malformed.
A self-contained script for tracking GitHub repository star history. It backfills historical data via GitHub GraphQL API, records daily snapshots, and generates dependency-free SVG charts with light/dark themes. The script is hardened for CI use: anonymous fetches (no credentials), SHA256 pinning of embedded assets, strict allowlist of changed files, and multi-stage verification before creating PRs. This is a reusable pattern for any project wanting automated, secure star-history tracking without external services.
A unified LLM client wrapper that supports OpenAI-compatible APIs with streaming, retry logic (with_retry decorator), byte-safe response concatenation, and configurable model selection. Provides both synchronous and asynchronous interfaces. The abstraction layer allows easy swapping of LLM providers (DeepSeek, Gemini, Qwen, etc.) via configuration. While it imports internal config and retry utilities, the core pattern of a retry-capable, streaming LLM client is reusable and demonstrates good error handling for production LLM usage.
A sentiment analysis abstraction that supports multiple model backends (multilingual DistilBERT, BERT-Chinese, traditional ML) via a configurable model_type parameter. Provides a unified predict() interface with confidence thresholds and batch processing. The design allows easy addition of new models without changing the calling code. This pattern is reusable for any NLP task where multiple models need to be compared or used interchangeably, and the abstraction layer cleanly separates model loading from inference logic.