From 6ff904e4118bb993e8b20d3a7e4a57e03586f6b4 Mon Sep 17 00:00:00 2001 From: Anton Gulenko Date: Tue, 26 May 2020 14:47:51 +0200 Subject: [PATCH 1/4] Update index.md --- docs/index.md | 54 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 47 insertions(+), 7 deletions(-) diff --git a/docs/index.md b/docs/index.md index e0cd396..927e8a4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -5,15 +5,55 @@ # go-bitflow **go-bitflow** is a Go (Golang) library for sending, receiving and transforming streams of data. -The basic data entity is a `bitfolw.Sample`, which consists of a `time.Time` timestamp, a vector of `float64` values, and a `map[string]string` of tags. +`go-bitflow` is mainly used in a Kubernetes cluster, managed by the [Bitflow K8s Operator](https://github.com/bitflow-stream/bitflow-k8s-operator), but can also be executed standalone either in a Docker container or as a native binary. + +The basic data entity is a [`bitfolw.Sample`](bitflow/sample.go), which consists of a `time.Time` timestamp, a vector of `float64` values, and a `map[string]string` of tags. Samples can be (un)marshalled in CSV and a dense binary format. The marshalled data can be transported over files, standard I/O channels, or TCP. -A `SamplePipeline` can be used to pipe a stream of Samples through a chain of transformation or analysis steps implementing the `SampleProcessor` interface. +A [`bitflow.SamplePipeline`](bitflow/pipeline.go) can be used to pipe a stream of Samples through a tree of transformation or analysis steps implementing the [`bitflow.SampleProcessor`](bitflow/sample_processor.go) interface. +The [`cmd/bitflow-pipeline`](cmd/bitflow-pipeline) sub-package provides an executable with the same name. +The dataflow graph (including data sources, data sinks, tree of operators) is expressed in the simple domain specific language [`Bitflowscript`](https://bitflow.readthedocs.io/projects/bitflow-antlr-grammars/en/latest/bitflow-script). +Some aspects of the execution can also be configured through additional command line flags. + +Go requirement: at least version 1.11. Use the `--help` flag for a list of command line flags. + +# Installation + +### Native installation + +``` +go install github.com/bitflow-stream/go-bitflow/cmd/bitflow-pipeline +``` + +### Build Docker container + +There are Dockerfiles for different processors: `AMD64` (using Alpine Linux), `arm32v7` and `arm64v8`. +In addition, there are Dockerfiles for "static" builds, that produce a container from `scratch`, that contains only the executable `bitflow-pipeline` file. +All containers can be either built fully with one Dockerfile (clean but slow), or use a local build and then simply copy the resulting binary in the container (utilized Go mod and build cache). + +All commands below are executed in the repository root. + +#### Full build (slow) + +Select one of the files in `build/multi-stage/[alpine|arm23v7|arm64v8]-[full|static].Dockerfile` and run: + +``` +docker build -t [IMAGE_NAME] -f build/multi-stage/[...].Dockerfile +``` + +#### Cached static build (fast & small container) + +``` +./build/native-static-build.sh +docker build -t [IMAGE_NAME] -f build/static-prebuilt.Dockerfile build/_output/static +``` -The `cmd/bitflow-pipeline` sub-package provides an executable with the same name. -The pipeline (including data sources, data sinks, pipeline of transformations steps) is defined by a lightweight, domain-specific scripting language (see subpackage `script`). -Some aspects of the pipeline can also be configured through additional command line flags. +#### Cached build (ARM or Alpine) -Run `bitflow-pipeline --help` for a list of command line flags. +Select a build target `arm32v7|arm64v8|alpine` and a directory to store the Go build cache (`/tmp/go-cache-directory` in this example). -Go requirement: at least version 1.11 +``` +TARGET=[arm32v7|arm64v8|alpine] +./build/containerized-build.sh $TARGET /tmp/go-cache-directory +docker build -t [IMAGE_NAME] -f build/$TARGET-prebuilt.Dockerfile build/_output/$TARGET +``` From 50be6e8453d8ed0507d964a40b58faa25d01d2c5 Mon Sep 17 00:00:00 2001 From: Anton Gulenko Date: Wed, 27 May 2020 12:28:34 +0200 Subject: [PATCH 2/4] Update index.md --- docs/index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/index.md b/docs/index.md index 927e8a4..002ca20 100644 --- a/docs/index.md +++ b/docs/index.md @@ -57,3 +57,14 @@ TARGET=[arm32v7|arm64v8|alpine] ./build/containerized-build.sh $TARGET /tmp/go-cache-directory docker build -t [IMAGE_NAME] -f build/$TARGET-prebuilt.Dockerfile build/_output/$TARGET ``` + +# Dockerhub + +Docker container images are available in the [`bitflowstream/bitflow-pipeline`](https://hub.docker.com/repository/docker/bitflowstream/bitflow-pipeline) Dockerhub repository: + +``` +docker pull bitflowstream/bitflow-pipeline +docker run -ti bitflowstream/bitflow-pipeline --help +``` + +The Docker manifest will select the appropriate platform (amd64/arm32v7/arm64v8) automatically. You can choose between the `static` and the `latest` tags, where the `static` image is smaller and contains a single file that can easily be integrated in other container images. From a2cd546fd74348c049ff8dd24d913daa8dc1b951 Mon Sep 17 00:00:00 2001 From: Anton Gulenko Date: Wed, 27 May 2020 12:29:20 +0200 Subject: [PATCH 3/4] Update index.md --- docs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 002ca20..1322b13 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,7 +15,7 @@ The [`cmd/bitflow-pipeline`](cmd/bitflow-pipeline) sub-package provides an execu The dataflow graph (including data sources, data sinks, tree of operators) is expressed in the simple domain specific language [`Bitflowscript`](https://bitflow.readthedocs.io/projects/bitflow-antlr-grammars/en/latest/bitflow-script). Some aspects of the execution can also be configured through additional command line flags. -Go requirement: at least version 1.11. Use the `--help` flag for a list of command line flags. +Go version: 1.14. Use the `--help` flag for a list of command line flags. # Installation From 5339c4a88a353ed7f936d2012adad8f012df3537 Mon Sep 17 00:00:00 2001 From: Anton Gulenko Date: Wed, 27 May 2020 12:38:23 +0200 Subject: [PATCH 4/4] Update index.md --- docs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 1322b13..8b05427 100644 --- a/docs/index.md +++ b/docs/index.md @@ -38,7 +38,7 @@ All commands below are executed in the repository root. Select one of the files in `build/multi-stage/[alpine|arm23v7|arm64v8]-[full|static].Dockerfile` and run: ``` -docker build -t [IMAGE_NAME] -f build/multi-stage/[...].Dockerfile +docker build -t [IMAGE_NAME] -f build/multi-stage/[...].Dockerfile . ``` #### Cached static build (fast & small container)