$ show manodeep/Corrfunc
Corrfunc is a high-performance scientific computing library for cosmology, specifically for computing two-point correlation functions and the void probability function from galaxy catalogs. It provides both a C library/command-line interface and Python bindings through C extensions. The code implements pair counting for real-space (DD, xi, wp), redshift-space (DDrppi, DDsmu), angular (DDtheta), and mock/survey geometries, with support for periodic simulation boxes and line-of-sight survey data. The architecture is a hybrid C core with Python wrappers: OpenMP parallelized C kernels, runtime dispatch across SSE4.2/AVX/AVX2/AVX512 intrinsics, grid-based cell decomposition, and a custom fast-food binary format for fast I/O. The project also includes a substantial test suite with reference outputs, CI workflows, and benchmark scripts used in a peer-reviewed performance paper. Despite being packaged as an application/library, the repository contains research artifacts: benchmark .npz files, paper scripts, and astrophysical mock-catalog test data. These are support materials for validating and advertising performance, not the primary deliverable. The dominant character is clearly a reusable scientific library, not a data-analysis pipeline. The codebase is mature and well-engineered for its niche, but shows some drift: Python 2 compatibility remnants, a complex dual Makefile/setup.py build, heavy code duplication between theory and mock modules, and platform-specific SIMD code that requires careful maintenance. It remains actively maintained and is the de facto standard CPU tool for fast correlation functions in cosmology.
The LLM identified 4 files worth highlighting for reuse:
One of a set of SIMD abstraction headers (sse_calls.h, avx_calls.h, avx2_calls.h, avx512_calls.h) that expose vector load/store/comparison/broadcast operations as macros. This allows the same pair-counting kernel source to be compiled for different ISAs by simply including the appropriate header. It is self-contained, portable, and could be extracted into any C/C++ performance-critical project needing clean vectorization without compiler-specific spaghetti.
A self-contained polynomial approximation for the inverse cosine, designed to avoid expensive libm calls in angular correlation computations. It demonstrates a non-obvious performance/accuracy tradeoff pattern that can be reused in any geometry or angular-metric code where fast approximate acos is acceptable. Isolated and easily extracted.
Runtime x86 CPU feature detection (SSE, AVX, AVX2, AVX512) with graceful fallback for unsupported platforms. This is the backbone of Corrfunc's runtime ISA dispatch and is implemented as a reusable utility separate from the pair-counting logic. Any C/C++ library that wants to select optimized kernels based on host capabilities can lift this module directly.
A set of pure-Python, numpy-based utility functions for converting raw pair counts into standard correlation-function estimators (e.g., Landy-Szalay, projected wp), parsing bin files, handling RA/DEC coordinate fixes, and preprocessing weight arrays. These transformations are largely independent of Corrfunc's C internals and represent a reusable data-processing layer for any clustering pipeline using pair-count outputs.