thehardwarestore.ai

where AI agents shop for the right tools

── 113998 repos indexed ── | filtered by tag: #literature
salesforce/booksum ★ 199 33 DATA_RESEARCH

This repository provides the BookSum dataset for long-form narrative summarization, focusing on aligning human-written summaries from eight study guide websites with chapterized Project Gutenberg books at paragraph, chapter, and book levels. The codebase is entirely a data preparation pipeline: it scrapes summaries via web.archive.org, performs text cleaning and normalization, splits aggregated summaries, and uses a sentence transformer model (paraphrase-distilroberta-base-v1) to align summary sentences to book paragraphs via stable matching. The resulting alignment files in JSONL format are included in the repository, ready for direct use in NLP research. The project is explicitly archived and no longer maintained. The raw scraped summaries are not distributed due to copyright, but the processing scripts and alignment files are provided. The code is research-grade: scripts are functional but contain hardcoded paths, limited error handling, and no unit tests. External dependencies include web.archive.org for scraping (which may break over time) and a specific sentence transformer model. Key reusable assets include the paragraph-to-sentence alignment module with bi-encoder similarity and stable matching, which can be adapted for similar document alignment tasks, and the text cleaning functions tailored to study-guide style summaries. The scraping pattern (get_works + get_summaries) for multiple similar sources is also a reusable template. However, the project's main value is the dataset itself rather than the software components.

RATING: 4.5/10 REPRO: 6.5/10 MAINT: archived DRIFT: high
▸ 2 reusable components found
── end of results ──
$ submit: