PaperQA2 is an agentic retrieval-augmented generation (RAG) system for scientific literature. It answers questions by searching, indexing, and analyzing PDFs, text documents, and clinical trials using a multi-step agent workflow. The system integrates with multiple scholarly metadata providers (Crossref, Semantic Scholar, OpenAlex, Unpaywall) for DOI resolution, citation counts, retraction detection, and journal quality scoring. It supports multiple LLM backends via LiteLLM, pluggable PDF readers (pypdf, PyMuPDF, Docling, Nemotron), and vector stores (Numpy, Qdrant). The architecture is a Python library with a CLI (`pqa`), featuring a hierarchical Pydantic settings system, an agent framework with tool selection and memory, and a plugin-based document parsing pipeline. While primarily an application, it is research-backed with published benchmarks (LitQA2, LFRQA) and includes config presets for different quality/performance trade-offs.
This repository is a research codebase for implementing and experimenting with dense retrieval techniques. It provides implementations of multiple advanced retrieval models including DPR, CITADEL, COIL, ColBERT, SPLADE, and their variants, along with data augmentation strategies like DRAGON and DRAMA. The codebase enables researchers to train, evaluate, and compare different retrieval approaches on standard benchmarks like Natural Questions, MS MARCO, and conversational datasets. It features a modular architecture with separate components for data processing, model definitions, training tasks, and evaluation, configured through Hydra for easy experimentation.
BLURR is a library that integrates Hugging Face transformers with fastai for NLP tasks. It provides high-level, mid-level, and low-level APIs to train and deploy transformer models for text classification, token classification, question answering, summarization, translation, and language modeling. The library leverages fastai's data block API and learner system, wrapping Hugging Face models to work seamlessly within the fastai training loop. Key components include custom data blocks for text processing, model wrappers for Hugging Face integration, specialized loss functions, and callbacks for optimization. The repository includes extensive examples and notebooks demonstrating usage with standard datasets like GLUE, SQuAD, and CNN/DailyMail.
This repository is a curated collection of five large-scale Chinese NLP datasets (Wikipedia articles, news, QA pairs, community Q&A, and translation pairs) with descriptions and download links. It functions as a data catalog rather than a software project: there is no executable code, no ETL scripts, and no pipelines. The datasets are hosted externally (Google Drive, Baidu) and require manual download. The repository provides clear formatting examples and suggested use cases, but lacks any automation or reproducibility support such as download scripts, environment definitions, or data processing code. It is a valuable reference for researchers needing Chinese text corpora, but it is not a reusable data engineering artifact.