ts-virtual-data-engine

A framework-agnostic TypeScript utility designed to handle very large datasets (millions, billions, or even sizes like 10^27) without freezing the UI or exhausting memory.

---

🎯 Problem

When working with large datasets, common issues include:

• Rendering thousands or millions of DOM nodes causes UI freezes
• Loading all data into memory is not feasible
• Client-side sorting and filtering block the main thread
• Scrolling performance degrades dramatically

These problems are especially critical in:

• GIS and map-based systems
• Log viewers and monitoring tools
• Financial and banking dashboards
• Enterprise data tables with infinite scrolling

---

💡 Solution Overview

This library solves the problem at the architecture level, not with UI tricks.

Core principle:

Never load all data. Never render all data.

The architecture is intentionally simple and scalable:

▶️ How to Run and Use This Project

This repository is a library, not a standalone application. It does not have a traditional npm start command.

You can run and use it in three different ways.

---

🧩 Alternative Tools vs This Project

Below is a comparison of commonly used tools for handling large datasets and how they compare to ts-virtual-data-engine.

Category	Tool / Library	Framework-dependent	Handles Huge Data	Virtual Scroll	Data Loading Logic	Lightweight	Notes
Virtualization	react-window	✅ React	⚠️ Partial	✅ Yes	❌ No	✅ Yes	Only solves rendering, data logic is external
Virtualization	TanStack Virtual	✅ React	⚠️ Partial	✅ Yes	❌ No	✅ Yes	UI-level virtualization only
Virtualization	Angular CDK Virtual Scroll	✅ Angular	⚠️ Partial	✅ Yes	❌ No	❌ Medium	Angular-only solution
Enterprise Grid	AG Grid	✅ React/Angular/Vue	✅ Yes	✅ Yes	✅ Yes	❌ Heavy	Very powerful, but large bundle and complex
Enterprise Grid	MUI Data Grid Pro	✅ React	✅ Yes	✅ Yes	⚠️ Partial	❌ Heavy	Paid features, React-only
Data Fetching	TanStack Query	✅ React	⚠️ Partial	❌ No	✅ Yes	⚠️ Medium	Fetching & caching only, no windowing
Data Fetching	SWR	✅ React	⚠️ Partial	❌ No	✅ Yes	✅ Yes	Simple cache layer
Backend-heavy	Elasticsearch	❌ No	✅ Yes	❌ No	✅ Yes	❌ Heavy	Backend solution, UI logic still needed
Custom Engine	ts-virtual-data-engine	❌ No	✅ Yes (even 10^27)	✅ Yes	✅ Yes	✅ Yes	Framework-agnostic core data engine

---

🏆 Why ts-virtual-data-engine?

• Framework-agnostic (works with React, Angular, Vue, Vanilla)
• Handles extremely large datasets safely
• Combines data loading + windowing + load-more logic
• Very small and reusable core
• Designed as a data engine, not a UI component

This makes it ideal as a foundation layer on top of which any UI framework can be built.

1️⃣ Run the Local Test / Demo

This project includes a local test that simulates very large datasets and verifies pagination, windowing, and load-more logic.

Steps:

npm install
npm run test:local