$ show NVIDIA-NeMo/Skills
This repository, NVIDIA-NeMo/Skills, is a production-grade framework for evaluating, training, and serving large language models (LLMs), with a strong emphasis on reasoning (math, code, proofs), tool calling, and speech tasks. It provides a CLI (`ns`) that orchestrates distributed pipelines for synthetic data generation, model inference across multiple backends (vLLM, SGLang, TensorRT-LLM, Megatron, OpenAI-compatible APIs), evaluation on 80+ benchmarks (e.g., AIME, SWE-bench, MMLU, BFCL), and reinforcement learning (GRPO, PPO, SFT). The architecture is built around reusable abstractions: configurable dataset preparers, prompt templates, metric collectors, inference servers, code execution sandboxes, and MCP-based tools, all integrated with Slurm/Ray cluster management. While the core is an application toolkit, it also contains numerous research recipes (e.g., OpenMathReasoning, SHERLOC, Nemotron-IMO-TTS) that exemplify its use in generating training data and reproducing published results. The codebase is well-structured with unit tests, GPU integration tests, and Slurm CI, though some components have hardcoded paths and dependencies that limit portability outside NVIDIA's infrastructure.
The LLM identified 4 files worth highlighting for reuse:
Defines `BaseMetrics`, an abstract class for evaluation metrics that implements `update()`, `get_metrics()`, and `reset()` lifecycle. It provides reusable infrastructure for computing any metric (e.g., pass@k, F1) from generation outputs, with built-in support for aggregating across seeds. This pattern is used by all concrete metric classes in the repository and can be extracted for use in any LLM evaluation framework.
Provides robust JSON file I/O (`jload`, `jdump`) with optional orjson acceleration, automatic line-delimited JSON detection, and utilities for chunking large files and calculating chunk indices. Also includes `unroll_files` for glob expansion. This module has zero internal dependencies and can be reused directly in any Python project needing performant JSONL handling.
Implements `math_equal`, a symbolic math answer comparison function that handles boxed expressions, floating point tolerance, and set equivalence using the `math_verify` library. Also includes `extract_answer` regex-based parser for extracting answers from model outputs. These functions are self-contained and can be reused for any math evaluation pipeline.
Contains `Prompt`, `PromptManager`, and `get_prompt` function that load prompt templates from YAML configs via Hydra/OmegaConf, with caching support. It merges system/user prompts from model-specific and generic config directories and provides few-shot example loading via a ChainMap over concatenated dicts. This component decouples prompt engineering from model logic and can be ported to other LLM applications needing structured prompt templates.