DataFlowX

Ingestion: Apache Airflow, Kafka Connect
Processing: Apache Spark, AWS Glue
Streaming: Apache Kafka, AWS Kinesis
Storage: PostgreSQL, S3, Redis
Analytics: Apache Spark, AWS Athena

A comprehensive data engineering platform for ingestion, processing, streaming, and AI analytics.

Architecture

DataFlowX implements a modern data pipeline architecture:

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
config		config
scripts		scripts
src		src
README.md		README.md
docker-compose.yml		docker-compose.yml
generate_batch_data.py		generate_batch_data.py
generate_data_aws.py		generate_data_aws.py
lambda_function.py		lambda_function.py
requirements.txt		requirements.txt
start.sh		start.sh