$ show huawei-noah/noah-research
This is a sprawling multi-project monorepo from Huawei Noah's Ark Lab, containing dozens of independent research codebases rather than a single unified application. The dominant content is ML/NLP/CV research artifacts: object detection (CPNDet, RAOD, Focus-DETR), multilingual machine translation (a vendored fairseq fork in noahnmt/multiuat), semantic segmentation and domain adaptation (uni-uvpt), model quantization and compression (QuantWM, PocketLLM, S3-Training), OOD detection (Maha_OOD), text-to-code (NLP/text2code_mrpt), and many other experimental efforts. Most of these directories contain training/evaluation scripts, shell launchers, configs, and data-preparation utilities with hardcoded absolute paths, old pinned dependencies, and inconsistent maintenance. There is no central build, shared versioning, or global environment definition; each subproject is effectively its own repository with its own requirements and conventions. Embedded in this research collection are several production-shaped frameworks that stand apart in quality: MindMemOS (a memory-as-a-service backend with FastAPI, Kafka workers, Qdrant/Neo4j persistence, and a Python SDK), EvoFabric (an LLM agent graph framework with tools, memory, and multi-agent support), and ScienceFlow (an agentic scientific solver with resource safety, evaluation gates, and LNR orchestration). These components have real tests, typed configs, and cleaner abstractions, making them more reusable than the surrounding research scripts. However, they are still tightly coupled to MindMemOS/EvoFabric/ScienceFlow internal infrastructure and are not packaged as standalone products. The repository as a whole answers a wide range of research questions across language transfer, efficient model architectures, agent memory and skill evolution, benchmark evaluation, and computer vision. From a data engineering perspective, it offers useful patterns for corpus preparation, sharding, perplexity scoring, retrieval fusion, and vector-database batch ingestion, but reproducibility across subprojects is generally low: data is often not bundled, scripts reference personal paths, dependencies are stale or version-pinned to long-outdated releases, and several vendored frameworks (e.g., fairseq 1.0.0a0) show clear signs of bit-rot. The hybrid nature is real: the repo is primarily a research-code collection, with a few application-quality frameworks that are actively maintained.
The LLM identified 6 files worth highlighting for reuse:
Generates deterministic UUID5 identifiers from business keys. This enables idempotent inserts, stable cross-system references, and reproducible record identities. The module is self-contained and can be extracted into any data pipeline or service that needs collision-safe, deterministic primary keys.
A generic Reciprocal Rank Fusion implementation for combining multiple ranked result lists into a single ranking. It is algorithmically self-contained, dependency-free, and directly reusable in search, RAG, or recommender systems that need to merge outputs from diverse retrievers.
Async batched upsert writer for Qdrant using per-collection queues and semaphore-limited inflight operations. It demonstrates a high-throughput, backpressure-aware ingestion pattern for vector databases. The architecture can be adapted to other vector stores or batch-write workloads where unordered asynchronous writes must be coalesced.
Detects potential training/evaluation data leakage, such as exact text overlap or contamination patterns, before running model evaluations. This is a recurring data-quality problem in LLM and ML benchmark work. The detection logic is modular enough to be reused as a pre-evaluation gate in other benchmark pipelines.
Implements standard out-of-distribution detection metrics such as FPR@TPR, AUROC, and AUPR. These are self-contained metric functions that can be extracted and reused in any classification or OOD evaluation script, avoiding the need to reimplement benchmark scoring logic.
Computes masked-language-model perplexity for individual dataset examples, supporting data-quality filtering and curriculum-learning analysis. Although it contains hardcoded CUDA assumptions and per-item Python loops, the core scoring approach is reusable for anyone needing per-example LM difficulty scores.