DataInfraPilot is a web application that helps small data engineering teams easily deploy and manage selected data engineering tools on Kubernetes clusters running on cost-effective cloud providers.
DataInfraPilot/
├── backend/ # FastAPI backend API
├── frontend/ # React frontend web app
├── demo/ # Example demo project
├── docker-compose.yml
└── README.md
- Backend: REST API for orchestration and management
- Frontend: User interface for deployment and monitoring
- Demo Project: Example pipeline and usage
- K8s cluster provisioning
- Selected Data Engineering tools deployment
- Cluster autoscaling through the Cluster Autoscaler
- Automated SSL certificate provisioning through the Cert Manager
- Hetzner
- Apache Airflow
- Apache Spark
- Grafana
| Part | Main Technologies |
|---|---|
| Backend | Python, FastAPI |
| Frontend | React, TypeScript, Vite, Tailwind CSS, shadcn-ui |
| Demo Project | Airflow, PostgreSQL, Python, Grafana |
pre-commit installBackend
cd backend
uv sync
source .venv/bin/activate
uvicorn src.api.main:app --reloadFrontend
cd frontend
npm install
npm run devdocker-compose upAdditionally, an SSH key pair is required to use it for server access. Create one under ~/.ssh directory (this directory will be mounted to the Docker container), and you will be prompted for the path when creating a new cluster.
