DataFlow

A web-based reactive document platform with real-time computation and collaboration features.

Features

• Reactive Documents: Real-time document editing with live computation
• Kanban Boards: Interactive project management with drag-and-drop
• Variable System: Dynamic variables with formula support
• Real-time Collaboration: WebSocket-based real-time updates
• Responsive Design: Mobile-friendly interface with touch support
• Offline Support: Service worker for offline functionality
• Type Safety: Full TypeScript support across the stack
• Testing: Comprehensive test coverage with Vitest and Jest
• Production Ready: Docker containerization with security best practices

Project Structure

dataflow/
├── apps/
│   ├── api/          # NestJS backend API (Node.js 22 + TypeScript)
│   └── web/          # React frontend application (Vite + TypeScript)
├── packages/
│   ├── shared/       # Shared utilities and constants
│   └── types/        # TypeScript type definitions
├── docker/
│   └── postgres/     # Database initialization scripts
├── nginx/            # Nginx configuration for SSL (optional)
├── docker-compose.yml          # Development environment
├── docker-compose.prod.yml     # Production environment
├── Makefile
└── package.json

Prerequisites

• Node.js 22+ and npm 10+
• Docker and Docker Compose (v2.0+)
• Make (optional, for using Makefile commands)
• Git for version control

Quick Start

Development Setup

1. Clone the repository:

git clone <repository-url>
cd dataflow

2. Install dependencies:

# Using npm workspaces
npm install --legacy-peer-deps

# Or using Make
make install

3. Start the development environment:

# Start all services (recommended)
make dev

# Or manually
docker-compose up -d postgres redis
npm run dev

4. Access the applications:

• Web app: http://localhost:3001
• API: http://localhost:3000
• API docs: http://localhost:3000/api
• Adminer (DB UI): http://localhost:8080

Production Deployment

For production deployment, see the Docker Production Deployment Guide.

Quick production setup:

# Copy environment template
cp .env.production.template .env.production

# Edit environment variables
nano .env.production

# Deploy to production
docker-compose -f docker-compose.prod.yml --env-file .env.production up -d

Available Commands

Using Make (recommended)

# Show all available commands
make help

# Development
make dev              # Start all services in dev mode
make dev-api          # Start only API in dev mode
make dev-web          # Start only web app in dev mode

# Building
make build            # Build all packages
make build-api        # Build only API
make build-web        # Build only web app

# Docker Development
make docker-up        # Start all Docker services
make docker-down      # Stop all Docker services
make docker-logs      # View Docker logs
make docker-logs-api  # View API logs only

# Production
make prod-build       # Build production Docker images
make prod-up          # Start production services
make prod-down        # Stop production services
make prod-logs        # View production logs

# Database
make db-migrate       # Run migrations
make db-migrate-generate  # Generate new migration
make shell-db         # Open PostgreSQL shell

# Testing & Quality
make test             # Run all tests
make test-coverage    # Run tests with coverage
make lint             # Run linter
make typecheck        # Run type checking
make clean            # Clean build artifacts

Using npm scripts

# Development
npm run dev           # Start all services
npm run api:dev       # Start API only
npm run web:dev       # Start web only

# Building
npm run build         # Build all packages

# Testing & Quality
npm run test          # Run tests
npm run test:e2e      # Run end-to-end tests
npm run lint          # Run linter
npm run typecheck     # Type check

Technology Stack

Backend (API)

• NestJS - Node.js framework
• TypeScript - Type-safe JavaScript
• PostgreSQL - Primary database
• Redis - Caching and real-time features
• TypeORM - Database ORM
• WebSocket - Real-time communication
• JWT - Authentication
• Jest - Testing framework

Frontend (Web)

• React 18 - UI framework
• TypeScript - Type-safe JavaScript
• Vite - Build tool and dev server
• TanStack Query - Data fetching
• Zustand - State management
• React Router - Client-side routing
• CodeMirror - Code editor
• Tailwind CSS - Styling
• Vitest - Testing framework
• Playwright - E2E testing

Infrastructure

• Docker - Containerization
• Docker Compose - Multi-container orchestration
• Nginx - Reverse proxy and static file serving
• Alpine Linux - Minimal container base

Environment Configuration

Development Environment

The development environment uses docker-compose.yml with default settings that work out of the box.

Production Environment

For production, copy the environment template:

cp .env.production.template .env.production

Edit the production environment variables:

# Database Configuration
POSTGRES_USER=dataflow
POSTGRES_PASSWORD=your_secure_database_password_here
POSTGRES_DB=dataflow_db

# Redis Configuration
REDIS_PASSWORD=your_secure_redis_password_here

# JWT Configuration
JWT_SECRET=your_jwt_secret_key_at_least_32_characters_long
JWT_EXPIRES_IN=24h

# CORS Configuration
CORS_ORIGIN=https://yourdomain.com

# Port Configuration
WEB_PORT=80

Important: Never use default passwords in production!

Docker Setup

Development Docker Setup

The project includes a complete Docker setup with:

• PostgreSQL database
• Redis for caching and real-time features
• Adminer for database management
• API container with health checks

To run everything in Docker:

make docker-up
# or
docker-compose up -d

Production Docker Setup

Production deployment uses docker-compose.prod.yml with:

• Security hardening (non-root users, health checks)
• Environment variable configuration
• Nginx for static file serving
• SSL support (optional)

See the Docker Production Deployment Guide for detailed instructions.

Testing

Running Tests

# Run all tests
npm run test

# Run tests with coverage
npm run test:coverage

# Run specific package tests
npm run test --workspace=@dataflow/api
npm run test --workspace=@dataflow/web

# Run e2e tests
npm run test:e2e --workspace=@dataflow/web

Test Structure

• Unit Tests: Jest/Vitest for component and service testing
• Integration Tests: API endpoint testing
• E2E Tests: Playwright for full application testing
• Performance Tests: Load testing for critical paths

Development

Code Quality

The project enforces code quality through:

• ESLint: Code linting with TypeScript support
• Prettier: Code formatting
• TypeScript: Type checking
• Husky: Git hooks for pre-commit checks

Project Scripts

# Development
npm run dev              # Start all services
npm run api:dev          # Start API only
npm run web:dev          # Start web only

# Building
npm run build            # Build all packages
npm run clean            # Clean build artifacts

# Code Quality
npm run lint             # Run linter
npm run typecheck        # Run TypeScript checks
npm run test             # Run tests

Troubleshooting

Common Issues

Port Conflicts

If you get port conflict errors, make sure no other services are running on ports:

• 3000 (API)
• 3001 (Web dev server)
• 5432 (PostgreSQL)
• 6379 (Redis)
• 8080 (Adminer)

Build Errors

If you encounter build errors:

1. Clean the project: make clean
2. Reinstall dependencies: npm install --legacy-peer-deps
3. Rebuild: npm run build

Database Connection Issues

Ensure PostgreSQL is running:

docker-compose ps

Check logs:

make docker-logs

Test database connection:

docker exec -it dataflow-postgres psql -U dataflow -d dataflow_db

Node.js Version Issues

Ensure you're using Node.js 22+:

node --version  # Should be 22.x.x

Use nvm to switch versions:

nvm use 22

Performance Troubleshooting

Slow Database Queries

• Check query performance in adminer
• Review API logs for slow queries
• Consider adding database indexes

Memory Issues

• Monitor container memory usage: docker stats
• Adjust container memory limits if needed
• Check for memory leaks in application logs

Getting Help

1. Check the logs: make docker-logs
2. Review the Docker Production Deployment Guide
3. Verify environment configuration
4. Check GitHub issues for similar problems

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit your changes: git commit -m 'Add amazing feature'
4. Push to the branch: git push origin feature/amazing-feature
5. Open a Pull Request

Development Guidelines

• Follow TypeScript best practices
• Write tests for new features
• Update documentation as needed
• Follow the existing code style
• Run linting and tests before committing

License

Nobody cares