feat: Event-Driven Document/Video Ingestion Pipeline#351

Merged

shubhadeepd merged 19 commits intorelease-v2.5.0from

minh/continuous-ingestion-notebook

Feb 27, 2026

Collaborator

minhngu-glitch commented Feb 12, 2026 •

edited

Loading

Description

Add rag_event_ingest.ipynb notebook and supporting changes for an end-to-end continuous ingestion pipeline that monitors object storage (MinIO) for new document and video uploads, automatically routes them to the appropriate AI services, and makes all content searchable via RAG.

Notebook walkthrough:

Deploy NVIDIA RAG stack (NIMs, Milvus, Ingestor, RAG Server)
Deploy NVIDIA VSS stack (VLM, LLM, Embedding, Reranker NIMs)
Deploy continuous ingestion pipeline (Kafka, MinIO, Kafka Consumer)
Configure video analysis prompts for the Kafka consumer
Upload documents and videos to MinIO with storage verification
Verify ingestion status via consumer logs
Query ingested content via RAG API or Frontend UI

Checklist

I am familiar with the Contributing Guidelines.
All commits are signed-off (git commit -s) and GPG signed (git commit -S).
New or existing tests cover these changes.
The documentation is up to date with these changes.
If adjusting docker-compose.yaml environment variables have you ensured those are mimicked in the Helm values.yaml file.

minhngu-glitch requested review from ShevaNguyen and sumitkbh

February 12, 2026 09:50

smasurekar requested review from nv-pranjald, shubhadeepd and smasurekar

February 13, 2026 06:58

smasurekar reviewed

View reviewed changes

notebooks/rag_event_ingest.ipynb Show resolved Hide resolved

smasurekar reviewed

View reviewed changes

notebooks/rag_event_ingest.ipynb Show resolved Hide resolved

nv-pranjald reviewed

View reviewed changes

examples/rag_event_ingest/deploy/docker-compose.yaml Outdated Show resolved Hide resolved

examples/rag_event_ingest/deploy/docker-compose.yaml Outdated Show resolved Hide resolved

smasurekar reviewed

View reviewed changes

examples/rag_event_ingest/deploy/docker-compose.yaml Outdated Show resolved Hide resolved

smasurekar reviewed

View reviewed changes

notebooks/rag_event_ingest.ipynb Show resolved Hide resolved

smasurekar reviewed

View reviewed changes

examples/rag_event_ingest/kafka_consumer/services/document_indexer.py Show resolved Hide resolved

nv-pranjald reviewed

View reviewed changes

notebooks/rag_event_ingest.ipynb Outdated Show resolved Hide resolved

notebooks/rag_event_ingest.ipynb Outdated Show resolved Hide resolved

notebooks/rag_event_ingest.ipynb Outdated Show resolved Hide resolved

ShevaNguyen requested a review from anngu-2xx3

February 23, 2026 03:38

minhngu-glitch changed the base branch from develop to release-v2.5.0

February 26, 2026 09:46

nv-pranjald approved these changes

View reviewed changes

smasurekar approved these changes

View reviewed changes

minhngu-glitch force-pushed the minh/continuous-ingestion-notebook branch from 116e7eb to 0e0d7bb Compare

February 27, 2026 05:26

minhngu-glitch added 13 commits

February 27, 2026 05:34


          feat: add rag_event_ingest example - event-driven document/video inge…

91108e3

…stion pipeline

- Kafka consumer that monitors MinIO object storage for new uploads
- Routes documents to RAG Ingestor, videos to VSS for analysis
- Docker Compose deployment for Kafka, MinIO, and consumer
- Jupyter notebook for end-to-end deployment and testing
- Sample test data (PDF document, MP4 video) tracked via Git LFS

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: polish rag_event_ingest notebook - fix sections, descriptions, TOC

01aed10

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          refactor: consolidate Setup into single cell - clone, deps, API keys

f2a8394

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          refactor: inline check_rag/vss/aidp_status into their usage cells

a88a176

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          docs: add markdown description before every code cell

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: notebook is standalone entry point, clones RAG repo to ~/rag

4ed922b

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: set COLLECTION_NAME, load .env, simplify query_rag, add expected…

eb8d480

… logs

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: left-align markdown tables in notebook

a1c7a89

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: use HTML tables to force left alignment

05d6eb7

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: replace API Keys markdown table with HTML for left alignment

af81e4d

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          feat: add storage verification, RAG frontend hints, and configurable …

276fccd

…consumer prompts

- Add verify_file_in_storage() helper to confirm files landed in MinIO
- Merge storage verification into document/video ingestion checks
- Add RAG Frontend UI link (port 8090) to query sections
- Make Kafka consumer VSS prompts configurable via env vars in docker-compose
- Install git/git-lfs in notebook setup cell
- Index cells in Deploy Continuous Ingestion section

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: skip RAG clone if directory already exists

3b82e71

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: url

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor

minhngu-glitch and others added 6 commits

February 27, 2026 05:35


          feat: add continuous ingestion notebook for video and document pipeline

Add rag_event_ingest.ipynb notebook that provides an end-to-end walkthrough for:
- Deploying NVIDIA RAG stack (NIMs, Milvus, Ingestor, RAG Server)
- Deploying NVIDIA VSS stack (VLM, LLM, Embedding, Reranker NIMs)
- Deploying continuous ingestion pipeline (Kafka, MinIO, Kafka Consumer)
- Configurable video analysis prompts for the Kafka consumer
- Uploading documents and videos to MinIO with storage verification
- Verifying ingestion via consumer logs
- Querying ingested content via RAG API or Frontend UI

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: gpu

7eac754

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: add ensurepip, fix VSS tag to v2.4.1, use GPUs 2-3 for VSS, upda…

79115e7

…te hw req to 4 GPUs

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          fix: tag

1699ca2

Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor


          feat: resolve comment

e2c72fd


          fix: patch VSS config to use host-mapped ports for shared RAG embeddi…

6d9f001

…ng/reranker

The via-server runs on the local_deployment_single_gpu_default network,
not nvidia-rag, so it cannot resolve nemoretriever-embedding-ms or
nemoretriever-ranking-ms. Route through host.docker.internal with the
correct host-mapped ports instead (9080 for embedding, 1976 for reranker).

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor
Signed-off-by: Minh Nguyen <minhngu@nvidia.com>
Made-with: Cursor

minhngu-glitch force-pushed the minh/continuous-ingestion-notebook branch from 0e0d7bb to 6d9f001 Compare

February 27, 2026 05:38

shubhadeepd merged commit 1944a7e into release-v2.5.0

5 of 6 checks passed

shubhadeepd deleted the minh/continuous-ingestion-notebook branch

February 27, 2026 06:17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

nv-pranjald nv-pranjald approved these changes

smasurekar smasurekar approved these changes

sumitkbh Awaiting requested review from sumitkbh

ShevaNguyen Awaiting requested review from ShevaNguyen

shubhadeepd Awaiting requested review from shubhadeepd

anngu-2xx3 Awaiting requested review from anngu-2xx3

Labels

None yet