This repository is a comprehensive survey paper on the intersection of knowledge graphs and multi-modal learning. It organizes the literature into two main directions: KG-driven multi-modal learning (KG4MM) and multi-modal knowledge graphs (MM4KG). The survey covers various tasks such as visual question answering, image classification, multi-modal named entity recognition, and knowledge graph completion, among others. Each section includes a list of relevant papers with links to their sources (e.g., arXiv, conference proceedings). The repository is maintained through updates that include adding new papers and fixing links, as evidenced by the recent commit in 2026 and the news items up to 2024-12.
This repository is an awesome list curating research papers on Video-LMM Post-Training for video reasoning. It maintains a JSON dataset of paper metadata (title, release date, links, tags, etc.) and uses a Python script to generate a categorized README.md with markdown tables organized by research paradigm (Reinforced Video-LMMs, SFT for Reasoning, Test-Time Scaling, Benchmarks, Surveys). The repository serves as a reference tracker for the latest advancements in enhancing reasoning capabilities of video-focused multimodal models through post-training techniques. The hybrid nature lies in its dual role as both a data collection (the JSON paper metadata) and a lightweight data presentation system (the generation script).
This repository is a curated 'awesome list' of research resources on multimodal learning in medical imaging. It contains no application code, tests, or data pipelines; its sole substantive file is a large README.md that catalogs papers, datasets, benchmarks, and code repositories. Topics span medical vision-language models, report generation, visual question answering, retrieval-augmented generation (RAG), and multimodal LLMs. The list is organized by task and includes links to arXiv PDFs, GitHub code, and dataset sources, with badges indicating publication venues.
P-HAR is a video analysis application that performs human action recognition specifically tailored for adult content. It processes input videos through three complementary modalities: visual frames (using TimeSformer), human pose sequences (using PoseC3D), and audio features (using a ResNet-based model). The system extracts features from each modality, runs inference through specialized deep learning models, and fuses the results using a weighted late fusion approach that handles heterogeneous output spaces. The architecture follows a modular pipeline: video input undergoes frame extraction and preprocessing, then each modality is processed independently through its respective model pipeline. The RGB stream processes spatiotemporal visual features, the skeleton stream first performs human detection and pose estimation before action recognition, and the audio stream extracts loudness-normalized spectral features. Results from all three streams are aligned to a common label space and combined using weighted averaging to produce final action predictions. Output can be generated as an annotated video with overlayed action labels and confidence scores, or as a JSON file containing temporal predictions for further processing. Key entry points include the multimodal demo script for end-to-end video processing, utility scripts for dataset preparation and model evaluation, and specialized tools for audio filtering and data pipeline management. The application relies on the MMAction2 framework ecosystem (including MMDetection and MMPose) for core computer vision capabilities, supplemented by standard Python libraries for media processing and data handling.
This repository is primarily an extensive educational compendium covering mathematics, computer science, and artificial intelligence from first principles to state-of-the-art topics. It contains over 100 markdown files organized into 20 chapters, each with detailed explanations, diagrams, and interactive coding tasks (primarily in Python/JAX). The repo also includes auxiliary tooling: a MkDocs-based documentation site with MathJax and a Pyodide in-browser Python runner that includes a JAX compatibility shim, and a TypeScript MCP (Model Context Protocol) server that provides programmatic access to the compendium content for AI assistants. While the vast majority of the repository is educational content rather than runnable software, the MCP server and interactive runner are functional application components. The compendium is well-structured and current, but the code examples are pedagogical (simplified, not production-ready) and the JAX shim is a fragile polyfill. The project is not a data research pipeline—it contains no datasets, experiment logs, or trained models.