This repository provides the Bench2Drive benchmark for closed-loop end-to-end autonomous driving, accepted at NeurIPS 2024 Datasets and Benchmarks Track. It includes a large-scale multi-modal dataset (up to 7.3TB) collected in the CARLA simulator, an evaluation suite with metrics like Driving Score and Success Rate, and baseline results for over 10 methods. The codebase is a hybrid of research data pipeline and application: the dataset generation and evaluation scripts form the core data_research component, while the B2DVisualize Flask app, scenario runner, and leaderboard evaluator provide application-like interfaces for interaction and reproducibility. Data lineage: raw simulation data (sensor recordings, ground truth annotations) is collected via data_collect.py, transformed into compressed archives with JSON manifests, and evaluated using the leaderboard pipeline that computes multiple ability metrics. The repository answers research questions about the effectiveness of end-to-end driving policies under diverse urban scenarios.
This repository generates rule sets (RULE-SET) for the Clash Premium proxy tool, specifically tailored for users in China. The core of the repo is a GitHub Actions workflow (run.yml) that daily fetches domain and IP lists from multiple upstream sources (e.g., v2ray-rules-dat, domain-list-custom, dnsmasq-china-list, geoip) and transforms them into Clash-compatible YAML payload files. The output rule sets categorize domains into direct, proxy, reject, private, Apple, Google, GFW, etc. The codebase itself is minimal (only CI configuration) and the actual data artifacts are stored in a separate 'release' branch. The repository acts as a data pipeline, not an application, with the workflow serving as the sole processing logic. The pipeline is straightforward but effective, relying on standard Unix tools (curl, grep, perl, sed) for data transformation.
This repository is an educational tutorial series that teaches how to build a nanoGPT-like transformer language model from scratch using PyTorch. It consists of over 30 Jupyter notebooks (with both TODO exercises and answer keys) designed for Google Colab, covering topics from token embeddings and positional encodings to multi-head attention, transformer blocks, training loops, scaling experiments, and instruction tuning. The code is modular, with separate classes for each component (TokenEmbedding, PositionEmbedding, AttentionHead, MultiHeadAttention, FeedForward, TransformerBlock, VocabularyLogits, nanoGPT, DataLoader, Trainer, ModelConfig), making it a valuable learning resource. Training is demonstrated on the Shakespeare dataset and later on a sample of FineWeb-Edu, with experiments on model size, learning rate, and tokenization. While the repository is not a production data pipeline or deployable application, it exhibits strong data/research characteristics through its focus on model training, scaling laws, and loss analysis. The notebooks are well-structured for self-paced learning but rely on Colab-specific workflows and hardcoded data URLs, limiting portability.
SimpleFold is a research repository focused on protein structure prediction using flow-matching generative models. While it includes application-like features such as a CLI and inference scripts, its core architecture is designed for ML research, specifically training and evaluating large-scale diffusion transformers (FoldingDiT) on structural biology data. The codebase is hybrid in nature, combining rigorous data engineering pipelines (parsing MMCIF, filtering, tokenization) with standard deep learning frameworks (PyTorch Lightning, Hydra). It supports dual backends (PyTorch and MLX) for inference, indicating a focus on accessibility across hardware platforms. The repository's primary value lies in its data processing pipeline, which is heavily adapted from the Boltz-1 project. It handles complex structural data (PDB, SwissProt, AFESM) through a modular pipeline including parsing, filtering, and tokenization. The model architecture implements flow-matching with ESM-2 embeddings for sequence context. Evaluation metrics include pLDDT confidence, TM-score, and RMSD, typical for structural biology research. However, the codebase exhibits signs of research-to-production transition challenges. Hardcoded paths (e.g., `/mnt/task_runtime/...`) and reliance on external binaries (TMscore, OpenStructure) reduce portability. The data pipeline is a fork of an external project (`boltz`), introducing maintenance overhead for divergence. Despite these issues, the use of Hydra for configuration management and PyTorch Lightning for training orchestration demonstrates solid engineering practices for an ML research project.
This repository is an interactive educational resource (Jupyter notebooks) teaching prompt engineering for Anthropic's Claude LLM. It is not a software application or a data pipeline; it is a notebook collection, so it is classified as data_research by the notebook signal, but its 'data' is synthetic/demo content (animal names, user/product records, arithmetic operations, legal tax code excerpts) used to illustrate prompt patterns. The primary entry points are the notebooks themselves, with parallel implementations for the Anthropic direct API and AWS Bedrock (both boto3 and anthropic Bedrock clients). Lessons progress from basic prompt structure, clarity, system prompts, XML tags, few-shot examples, step-by-step thinking ('Precognition'), output formatting, avoiding hallucinations, to complex/industry use cases. The notebooks use get_completion wrappers around client.messages.create(...), %store magic to share variables, and a tool-use appendix that implements prompt-injected function calling via XML-like tags and regex parsing. Exercises include golden answers and rubric-based grading (e.g., summary quality scored 1-5). Reusable patterns include the tool-use prompt-injection function construct_successful_function_run_injection_prompt(invoke_results), system-prompt composition (general tool-use explanation + specific tool definitions), and a grader prompt pattern. However, the code is tightly coupled to notebook execution order, API details may drift, and duplicated content across anthropic/boto3 notebooks creates maintenance risk.
This repository is a data engineering pipeline that produces a comprehensive, unified country codes dataset (CSV/JSON) by merging information from multiple authoritative sources: ISO 3166, ISO 4217, UN Statistics, UN Protocol (UNTERM), CLDR, statoids, Geonames, EDGAR, FIFA, and Wikidata. The pipeline is orchestrated via a Makefile that sequentially runs Python scripts to fetch, parse, clean, and join data. The final output contains dozens of columns covering country names in multiple languages, ISO codes, currency codes, dialing codes, and more. The repository is primarily a data transformation project rather than a software application, though it includes sophisticated web scraping and fuzzy matching. The code is modular and well-structured, but reproducibility is hampered by a manual step requiring users to manually download and rename UNSD CSV files. The included GitHub Actions workflow automates monthly updates. Overall, the project is a solid reference data curation effort with moderate complexity.
This repository contains the LatentNeeds-Bench benchmark and evaluation framework for proactive AI demand detection, as described in the research paper 'Pask: Toward Intent-Aware Proactive Agents with Long-Term Memory'. The project focuses on creating a benchmark to evaluate whether AI models can detect latent user needs in ongoing conversations without explicit prompting. It includes a comprehensive data processing pipeline that transforms raw speech transcripts into annotated benchmark datasets, an evaluation framework to test various LLMs on this benchmark, and tools to generate research paper tables and figures. The system implements the DD-MM-PAS paradigm (Demand Detection, Memory Modeling, Proactive Agent System) with IntentFlow as the specialized demand detection model.
OpenHGNN is a comprehensive research framework for Heterogeneous Graph Neural Networks (HGNNs) built on PyTorch and DGL. The repository implements 83+ graph neural network models across 17+ tasks including node classification, link prediction, recommendation, knowledge graph completion, and temporal graph learning. It serves as a benchmarking platform for academic research, providing standardized implementations of state-of-the-art architectures like HGT, MAGNN, CompGCN, and HERO. The framework uses a registration pattern for dynamic model/trainer discovery and supports distributed training, hyperparameter optimization via Optuna, and GraphBolt integration for large-scale graphs. While it has CLI entry points and setup.py, its primary purpose is research reproducibility rather than production deployment. The codebase demonstrates strong architectural patterns with BaseDataset, BaseTask, and BaseFlow abstractions that enforce consistent interfaces across 50+ model implementations. However, significant technical debt exists including version mismatches (setup.py declares 0.7.0 while README claims 0.9.0), hardcoded dataset paths, mixed DGL/PyG usage, and monolithic configuration files. The repository contains extensive training artifacts in output/ directories and documentation drift between README, setup.py, and Sphinx docs. Despite these issues, it remains a valuable resource for graph learning researchers seeking reproducible implementations of heterogeneous GNN architectures.
The Multimodal Universe is a large-scale astronomical data repository that aggregates ~30 surveys (Legacy Survey, HSC, JWST, Gaia, SDSS, DESI, TESS, etc.) into a unified, HEALPix-partitioned HDF5 format, exposed through Hugging Face datasets. It covers multiple modalities: images (multi-band), spectra (1D), time series (light curves), and tabular catalogs. The repository provides dataset builder scripts (GeneratorBasedBuilder subclasses) per survey, cross-matching utilities, and benchmark training code (PyTorch Lightning) for downstream tasks like photometric redshift estimation and galaxy property regression. While primarily a data research project, it also includes application-like components such as baseline classifiers (Informer for PLAsTiCC) and experimental benchmark pipelines. The data is designed for training foundation models in astronomy, with streaming support and standardized column naming. Notable patterns include HEALPix-based spatial indexing, parallel processing in build scripts, and a consistent Hugging Face dataset interface across all surveys. The codebase is well-structured for its purpose, though many scripts are tightly coupled to specific surveys and external data sources, limiting direct reuse without adaptation.
The banking-apps-compat-report repository is a crowdsourced initiative that tracks the compatibility of international banking apps with GrapheneOS by leveraging GitHub issues as a data collection mechanism. Rather than containing traditional source code or data processing pipelines, the project relies on users submitting detailed compatibility reports through a standardized issue form, which are then curated and published by maintainers on the PrivSec.dev website. The cornerstone of this repository is the GitHub issue template located at .github/ISSUE_TEMPLATE/app_report.yml. This template defines a comprehensive form for collecting structured data on banking app compatibility, including fields for app metadata, device and profile specifications, security settings (such as exploit protection and memory tagging), installation methods, NFC payment functionality, and user-reported issues. Its careful design ensures consistent and high-quality data input, making it a reusable asset for similar crowdsourced data collection efforts in other domains. The project is actively maintained, with recent commits and ongoing issue triage, and exhibits low technical drift due to its minimal dependencies and use of stable GitHub features. However, reproducibility of the published compatibility lists is limited because the raw data resides exclusively in the GitHub issue tracker and is not bundled within the repository. Despite this, the initiative successfully provides a valuable, community-driven resource for GrapheneOS users seeking reliable information on banking app functionality.
This repository is a curated collection of wordlists for bug bounty hunting and web application security testing. It aggregates URL paths, payloads, and vulnerability probes from multiple public sources (SecLists, OneListForAll, dirsearch, nuclei-templates, etc.) into a unified, organized dataset. The wordlists are categorized by technology (e.g., Apache, WordPress, Nginx) and severity (info, low, medium, high, critical), making them directly usable with tools like Nuclei, ffuf, and dirsearch. The repository includes two automation scripts: a shell script that fetches and deduplicates wordlists from external URLs, and a Go program that parses nuclei-templates YAML files to generate technology-specific wordlists. These scripts are run weekly via GitHub Actions to keep the dataset current. The repository is purely a data asset; it contains no application code, APIs, or services. Its primary value is as a ready-to-use, aggregated wordlist source for security researchers and penetration testers.
This repository is a curated list of npm resources and tips, organized into sections such as Articles, Tools, Packages, Clients, Tips, FAQ, Community, Documentation, Support, and Related. It serves as a reference for developers seeking npm-related tools, packages, and best practices. The maintenance is driven by community contributions via pull requests, which are reviewed according to the contributing guidelines. While there is no automated data processing, the repository functions as a static dataset of npm resources that is updated manually.
This repository accompanies a Korean-language book on practical LLM application development, containing hands-on code examples organized by chapter. It demonstrates the full LLM lifecycle including foundational concepts (transformers), model optimization (quantization, PEFT), application building (RAG, agents, multimodal systems), and deployment techniques. The primary value lies in its educational notebooks that show how to implement LLM-powered solutions for tasks like Text2SQL, retrieval-augmented generation, and AI agents using frameworks such as LlamaIndex, AutoGen, and vLLM. While designed for Google Colab execution, the code illustrates reusable patterns for LLM engineering.
This repository is a curated collection of rules for the ananicy-cpp daemon, which automatically adjusts process nice/ionice/scheduling priorities on Linux. It is primarily a dataset of process name to type mappings, organized by category (audio-video, browsers, games, desktop environments, services, tools, etc.), with additional cgroup CPU quota definitions. The rules are manually curated by the community and cover thousands of applications, including Linux native and Wine/Proton games, with detailed per-game executable entries. The repository includes JSON schemas for validation, a linting script, a sorting script for game rules, and a Makefile for installation. It is actively maintained and serves as the default ruleset for the CachyOS Linux distribution's ananicy-cpp package. No ETL pipelines or analytical models are present; the primary data product is the ruleset itself.
This repository implements FiveThirtyEight's NFL Elo forecasting model and evaluation framework. The data lineage flows from two raw CSV sources: `data/nfl_games.csv` containing historical game results (1920-present) with pre-computed Elo win probabilities, and `data/initial_elos.csv` providing starting Elo ratings for each franchise. The `forecast.py` module implements the core Elo engine: it initializes team ratings, applies season-to-season reversion (retaining 2/3 of prior rating, reverting 1/3 toward 1505), incorporates home-field advantage (65 Elo points), and updates ratings post-game using a margin-of-victory multiplier (log(pd+1) * 2.2/(elo_diff*0.001+2.2)). Special hardcoded reversions handle franchise relocations/expansions. The `util.py` module handles CSV parsing with type coercion and evaluates forecasts using a Brier-score-based points system (25 - 100*brier, doubled for playoffs). The `eval.py` script orchestrates the full pipeline: load games, generate forecasts, evaluate against FiveThirtyEight's baseline. The repo answers the research question: 'Can alternative forecasting models beat FiveThirtyEight's Elo model on historical NFL data?'
This repository is a curated literature review (awesome list) focused on the progression from video generation to world models in AI research. It systematically organizes and categorizes recent academic papers (primarily from 2022-2026) into a hierarchical structure covering technical foundations, application domains, evaluation methodologies, and theoretical perspectives. The main value lies in its comprehensive paper collection with structured metadata including publication dates, venues, paper links, project pages, and official code repositories. The repository follows a standard awesome-list format using markdown tables with shields.io badges for easy navigation to paper resources. Content is organized into eight main sections: Generation 1-4 (covering faithfulness, interactiveness, planning, and randomness aspects), Evaluation and Datasets, Study and Rethinking, Downstream Tasks, and World Models for Other Applications. Each section contains subsections detailing specific technical approaches (e.g., GAN-based, U-Net-based, DiT-based video generation) and application areas (robotics, autonomous driving, gaming). The list is actively maintained with frequent updates incorporating very recent pre-print papers. As a static reference resource, the repository contains no executable code, data processing pipelines, or computational components. Its reproducibility is high because the curated content is directly available in the README upon cloning, requiring no setup or dependencies to access the paper list. The organizational pattern demonstrates good practices for academic literature curation, though it does not contain novel reusable data engineering components beyond the standard markdown documentation format.
Scal3R is a research codebase for CVPR 2026 Highlight paper on large-scale 3D reconstruction from unordered images using test-time training. The core contribution is a novel bidirectional attention mechanism with fast weight updates (TTT layers) that adapts during inference to improve reconstruction quality for large scenes. The repository provides an end-to-end inference pipeline: given a folder of images, it estimates camera poses, depth maps, and produces a colored point cloud. The architecture builds on a DINOv2 backbone and VGGT-based aggregator with custom attention heads, loop detection via SALAD, and pose graph optimization with submap management. The codebase is well-structured with separate modules for data loading, model definition, inference orchestration, and visualization tools. It includes fallback implementations for key components (e.g., Python-only loop solver, simple correlation-based loop detector) to improve portability, though GPU and CUDA are assumed. The repository is primarily a research artifact with clean utilities that could be extracted for other 3D vision projects. Evaluation code and dataset scripts are explicitly marked as not yet released, limiting reproducibility until those are provided.
OpenUPM is a community-driven curated registry of Unity packages, hosted as a YAML metadata repository. The repository contains structured package manifests (name, description, license, source URL, topics, etc.) for thousands of Unity UPM packages, organized into released and unreleased categories. It includes a CI pipeline for automated validation of data changes, a PR comment bot for contributor guidance, and issue templates for governance (rename, replace, unpublish). The actual validation logic lives in a separate repository (openupm-next), making this repo primarily a data store with orchestration scripts. While not a traditional data research project with notebooks or models, it exemplifies a well-governed curated dataset with automated quality checks.
This repository is a manually curated awesome-list/registry of open-source security scanners and offensive/defensive tools. It spans traditional security testing categories (web, mobile, binary, smart-contract, IoT, APT) and an actively expanded section for AI/LLM-agent security. The actual content is a set of near-identical English, Chinese, and Spanish README files containing hundreds of one-line tool entries with shields.io badges for language, last commit, stars, and license. There is no application runtime, no build pipeline, and no test suite; the product is a navigable knowledge base rather than software. The repo is therefore classified as data_research only in the sense that it is a structured, human-maintained metadata registry, not a data pipeline. The primary data lineage is manual: maintainers sample external GitHub projects, annotate each with a one-sentence description and category, and publish the result as markdown. The taxonomy serves as an analytical schema, answering questions such as which open-source scanners exist for a given domain, which are active, and what maturity signals (stars, license, last commit) they carry. It does not provide quantitative benchmarks or automated validation, and the data is not exposed in machine-readable form beyond the markdown itself. Reuse is straightforward for human readers, but automation requires parsing the markdown and accepting that some badges or links may be stale or inconsistent. The repository is actively curated, with recent emphasis on AI security, a documented commit convention, and multilingual READMEs. Its main risks are extrinsic: link rot, renamed or abandoned upstream tools, and the CC BY-NC-ND license, which restricts derivative redistribution. Technical drift is moderate because some older categories and tools are outdated even though the overall list is kept current.
CARI4D is the official CVPR 2026 research codebase for category-agnostic 4D reconstruction of human-object interaction from monocular video. It is fundamentally a research pipeline rather than a production application: raw RGB-D video (primarily BEHAVE, plus HODome/InterCap and custom clips) is processed through a chain of external foundation models - UniDepth for metric depth, NLF/SAM3D for human pose, FoundationPose for object tracking - then aligned, rendered into H5 training files, refined by a learned temporal transformer (CoCoNet), and finally optimized at test time to output temporally consistent SMPL-H body parameters, object 6D pose, and hand contact labels. The repo contains training, inference, preprocessing, and evaluation entry points, but no API/service layer; the runnable scripts are research pipeline drivers.