$ show shibing624/text2vec
text2vec is a Python library for converting text (words, sentences, paragraphs) into dense vector embeddings, with a strong focus on Chinese semantic similarity. It provides multiple embedding backends: Word2Vec (Tencent embeddings), Sentence-BERT, CoSENT, BGE, and BM25 for lexical ranking. The library offers a unified API via `SentenceModel` and `Similarity` classes, a command-line interface for batch embedding, and training/fine-tuning scripts for supervised and unsupervised text matching. It also includes deployment examples (FastAPI, Jina, Gradio) and evaluation benchmarks (C-MTEB, MTEB). While primarily an application/library, it has strong research components with detailed model comparison reports and dataset building scripts. The codebase is modular and well-documented, though some training code contains boilerplate repetition and dependency version ranges are broad, introducing potential drift.
The LLM identified 2 files worth highlighting for reuse:
A standalone implementation of the BM25 ranking algorithm with three variants (Okapi, L, Plus). It is self-contained, requires no internal imports from the rest of the library, and can be extracted and reused in any Python project needing lexical search. The code is clean, parameterized, and demonstrates non-obvious skill in implementing ranking functions.
A small utility module providing cosine similarity, Euclidean distance, and other vector distance functions. It is isolated, has no dependencies on the rest of the library, and can be reused directly. The implementation is efficient and handles edge cases (e.g., zero vectors) gracefully.