$ account royerlab
napari-chatgpt (Omega) is an LLM-powered autonomous agent designed as a napari plugin for interactive image analysis. It enables users to interact with the napari viewer via natural language through a chat interface, either embedded in napari or via a web-based UI. The system processes user commands by leveraging LLMs (via LiteMind) to select and execute appropriate tools for tasks such as image loading, segmentation, denoising, widget creation, viewer control, web search, and code execution. The architecture centers around a plugin-contributed dock widget that launches a chat interface, which communicates with a FastAPI/WebSocket backend. The backend hosts the Omega agent, which uses a modular tool system where each tool (e.g., for file operations, segmentation, or web requests) inherits from a base class and is responsible for executing specific actions. The agent can iteratively refine its actions through dialogue, enabling complex workflows like 'segment nuclei and show measurement statistics'. Key external dependencies include napari (>=0.5), LiteMind, LLM provider libraries (OpenAI, Anthropic, Google), FastAPI, Qt bindings, and scientific libraries like scikit-image, Cellpose, and StarDist.