Connectors sync worker

Prerequisites

• Docker Engine version 19.03.13
• Docker Compose version 1.27.4
• pyenv
• poetry

Getting Started

Install

source ./init.sh

Get Diffs from Latest Cookiecutter

./cookiecutter-diff.sh

Run CI tasks locally

$ inv --list
Available tasks:
  build          build a package
  call           calls a given task with given args
  clean          clean (remove) packages
  lint           python lint
  outdated       outdated packages
  test           run unit tests
  update         update packages
  audit          run safety checks on project dependencies
  start-services starts a redis broker and an instance of flower
  stop-services  stops the dependent services
  run-worker     starts the # connectors_sync_worker celery worker process. Note that ai21-celery adds  metrics endpoint for prometheus
  build-image    build docker image

Build Image

inv build-image

Run Celery tasks locally

• Execute the celery worker through redis broker

inv run-worker
inv call connectors_sync_worker.task_name --args "your_args"

• Another option is to run it as http service:

connectors_sync_worker.//tasks.py
curl http://localhost:8080/multiply -d '{"x": 3, "y":5}'

Navigate to http://localhost:3000 in your web browser and use the login credentials specified in the compose file to access Grafana. It is already configured with prometheus as the default datasource.

Navigate to http://localhost:9090 in your web browser to access directly the web interface of prometheus.

Stop and remove the containers. Use -v to remove the volumes if looking to erase all data.

inv stop_services

Contribute

1. Run validation by leveraging pre-commit
   1. Install pre-commit install --install-hooks -t pre-commit -t pre-push -t commit-msg
   2. To run on-demand pre-commit run -a
      • pre-commit run shellcheck -a --verbose --hook-stage manual for recommendation
2. Submit a pull-request