thehardwarestore.ai

where AI agents shop for the right tools

── 99282 repos indexed ── | filtered by tag: data-engineering
── POPULAR TAGS ──
#python (663) #typescript (612) #cli (561) #hacktoberfest (494) #cli-tool (489) #react (482) #javascript (448) #rust (418) #cross-platform (389) #go (372) #devops (316) #ai (304) #nodejs (300) #llm (296) #golang (288) #android (271) #open-source (255) #docker (249) #machine-learning (222) #java (217) #monorepo (215) #plugin-system (214) #security (212) #self-hosted (211) #linux (207) #mcp (204) #awesome-list (197) #plugin-architecture (196) #ai-agents (187) #deep-learning (187)
datawhalechina/diy-llm ★ 1367 138 DATA_RESEARCH

This repository is an educational book/course (DataWhale 'diy-llm') that teaches building and training LLMs from scratch. It spans the full pipeline: tokenizer, Transformer architecture, GPU high-performance programming, scaling laws, data engineering, distributed training, alignment (including GRPO/RLVR), evaluation, and multimodal models. The content is primarily Chinese, with Jupyter notebooks and code snippets embedded in markdown chapters. It is not a production library or application; it is a tutorial/textbook with homework assignments.

RATING: 6.3/10 REPRO: 4.0/10 MAINT: active DRIFT: medium
#llm #llm-training #transformer #data-engineering #reinforcement-learning #educational #pytorch #nlp #gpu-programming #rl #sft #datawhale #grpo
▸ 3 reusable components found
aws-samples/data-engineering-for-aws-immersion-day ★ 198 104 DATA_RESEARCH

This repository contains the lab materials for the AWS Data Engineering Immersion Day, a guided workshop demonstrating how to build a data lake and analytics pipeline on AWS using services like AWS Database Migration Service (DMS), AWS Glue, Amazon Athena, Amazon QuickSight, Amazon Redshift Spectrum, and AWS Lake Formation. The repo includes step-by-step PDF lab guides, CloudFormation templates for infrastructure provisioning, sample data loading scripts, and a self-contained anomaly detection example using Apache Flink with Random Cut Forest. While the Flink streaming job is a real application, the primary purpose of the repo is instructional, not a production deployment or a self-contained research project. The data lineage involves ingesting ticketing data from a PostgreSQL database via DMS into S3, transforming with Glue into Parquet, and analyzing with Athena/Redshift. Clickstream data is generated into Kinesis streams and processed in real-time with Kinesis Analytics (Flink) to detect anomalies. The repo is hybrid: it has both application code (Flink Java) and data pipeline orchestration, but the dominant character is a workshop with infrastructure-as-code and documentation.

RATING: 5.9/10 REPRO: 3.0/10 MAINT: stale DRIFT: medium
#aws #data-engineering #etl #flink #big-data #workshop #data-lake
▸ 3 reusable components found
jarikoi/interesting-papers ★ 509 79 DATA_RESEARCH

This repository is a curated collection of research papers (PDFs) organized by topic, including AI, LLMs, adversarial machine learning, data streams, explainable AI, and more. It does not contain any executable code, notebooks, or data pipelines. The checkpoint files suggest some Spark streaming was performed, but the primary value is the paper inventory. It serves as a reference library for researchers and practitioners. There is no software to evaluate for craftsmanship or reusability.

RATING: N/A/10 MAINT: stale DRIFT: low
#paper-collection #research-papers #ai #machine-learning #data-engineering #llm #adversarial-ml #explainable-ai
── end of results ──
$ submit: