To run it locally, do the following:
- Install and set up minikube v0.22.3 (google for more details)
- run
minikube start --kubernetes-version v1.7.5 --cpus 7 --memory 12360 --disk-size 100g - run
make sidecars(to set up dependencies) - run
make publish(to build and publish all the docker images needed) - run
make submit JOB=<job_name>(to submit the spark job twitter_follower_jdbc_logging or reliable_s3_parquet_logging to local minikube cluster) - run
kubectl get podsto find the spark driver pod - run
kubectl logs -f <driver-pod-name>to check the driver log - run
kubectl exec -it cockroachdb-0 bashto log into cockroachdb - run
./cockroach sql --insecure --host=cockroachdb-public -e '<any_sql_query>'query cockroachDB - have fun
- run
kubectl delete pod <driver-pod-name>to stop the spark job - try re-installing vm and minikube if running into weird kubernetes related issues