$ show Lixsp11/sekai-codebase
This repository provides the data pipeline for the Sekai dataset, a large-scale egocentric video dataset for world exploration and generation research. It contains two main toolchains: dataset_downloading (extracting YouTube URLs from CSV annotations and downloading videos via yt-dlp) and clip_extracting (a multi-step GPU-accelerated pipeline to split raw videos into annotated clips, transcode them to H.265, and remux with audio). The pipeline is designed for the Sekai-Real subset and requires NVIDIA GPUs with NVENC support, CUDA 11.8, and specific Python dependencies. The code is well-structured with clear separation of steps, logging, and multiprocessing support. While tightly coupled to the Sekai dataset and hardware requirements, the utility modules for GPU video decoding/encoding and frame sampling are reusable in other video processing projects.
The LLM identified 3 files worth highlighting for reuse:
Implements EMDownSampler, a frame selection algorithm that mimics ffmpeg's behavior for downsampling video frame rates. Uses a sliding window of boolean flags stored in a bitarray to decide which frames to keep, minimizing the margin error between source and target FPS. This is a non-obvious, reusable component for any video processing pipeline requiring temporal frame subsampling.
Provides VideoBatchDecoder and VideoMemoryEncoder classes that wrap PyNvVideoCodec and CVCUDA for hardware-accelerated video decoding, color conversion, resizing, and encoding. The decoder supports batching and frame sampling via EMDownSampler, while the encoder writes H.265 bitstreams to memory. These components demonstrate clean abstraction of GPU video pipelines and can be extracted for other projects needing efficient video transcoding.
Contains low-level wrappers NVVCVideoDecoder and NVVCVideoEncoder around PyNvVideoCodec's C++ bindings. Manages CUDA context, stream, and demuxer/encoder lifecycle. While tightly coupled to the specific library, the code is well-structured and demonstrates proper resource management (context push/pop, explicit cleanup). Useful as a reference for integrating PyNvVideoCodec in Python.