Skip to content

Extended README, added installation instructions #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
antongulenko wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Extended README, added installation instructions #79

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6ff904e
Update index.md
antongulenko May 26, 2020
50be6e8
Update index.md
antongulenko May 27, 2020
a2cd546
Update index.md
antongulenko May 27, 2020
5339c4a
Update index.md
antongulenko May 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 58 additions & 7 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,66 @@

# go-bitflow
**go-bitflow** is a Go (Golang) library for sending, receiving and transforming streams of data.
The basic data entity is a `bitfolw.Sample`, which consists of a `time.Time` timestamp, a vector of `float64` values, and a `map[string]string` of tags.
`go-bitflow` is mainly used in a Kubernetes cluster, managed by the [Bitflow K8s Operator](https://github.com/bitflow-stream/bitflow-k8s-operator), but can also be executed standalone either in a Docker container or as a native binary.

The basic data entity is a [`bitfolw.Sample`](bitflow/sample.go), which consists of a `time.Time` timestamp, a vector of `float64` values, and a `map[string]string` of tags.
Samples can be (un)marshalled in CSV and a dense binary format.
The marshalled data can be transported over files, standard I/O channels, or TCP.
A `SamplePipeline` can be used to pipe a stream of Samples through a chain of transformation or analysis steps implementing the `SampleProcessor` interface.
A [`bitflow.SamplePipeline`](bitflow/pipeline.go) can be used to pipe a stream of Samples through a tree of transformation or analysis steps implementing the [`bitflow.SampleProcessor`](bitflow/sample_processor.go) interface.
The [`cmd/bitflow-pipeline`](cmd/bitflow-pipeline) sub-package provides an executable with the same name.
The dataflow graph (including data sources, data sinks, tree of operators) is expressed in the simple domain specific language [`Bitflowscript`](https://bitflow.readthedocs.io/projects/bitflow-antlr-grammars/en/latest/bitflow-script).
Some aspects of the execution can also be configured through additional command line flags.

Go version: 1.14. Use the `--help` flag for a list of command line flags.

# Installation

### Native installation

```
go install github.com/bitflow-stream/go-bitflow/cmd/bitflow-pipeline
```

### Build Docker container

There are Dockerfiles for different processors: `AMD64` (using Alpine Linux), `arm32v7` and `arm64v8`.
In addition, there are Dockerfiles for "static" builds, that produce a container from `scratch`, that contains only the executable `bitflow-pipeline` file.
All containers can be either built fully with one Dockerfile (clean but slow), or use a local build and then simply copy the resulting binary in the container (utilized Go mod and build cache).

All commands below are executed in the repository root.

#### Full build (slow)

Select one of the files in `build/multi-stage/[alpine|arm23v7|arm64v8]-[full|static].Dockerfile` and run:

```
docker build -t [IMAGE_NAME] -f build/multi-stage/[...].Dockerfile .
```

#### Cached static build (fast & small container)

```
./build/native-static-build.sh
docker build -t [IMAGE_NAME] -f build/static-prebuilt.Dockerfile build/_output/static
```

#### Cached build (ARM or Alpine)

Select a build target `arm32v7|arm64v8|alpine` and a directory to store the Go build cache (`/tmp/go-cache-directory` in this example).

```
TARGET=[arm32v7|arm64v8|alpine]
./build/containerized-build.sh $TARGET /tmp/go-cache-directory
docker build -t [IMAGE_NAME] -f build/$TARGET-prebuilt.Dockerfile build/_output/$TARGET
```

# Dockerhub

The `cmd/bitflow-pipeline` sub-package provides an executable with the same name.
The pipeline (including data sources, data sinks, pipeline of transformations steps) is defined by a lightweight, domain-specific scripting language (see subpackage `script`).
Some aspects of the pipeline can also be configured through additional command line flags.
Docker container images are available in the [`bitflowstream/bitflow-pipeline`](https://hub.docker.com/repository/docker/bitflowstream/bitflow-pipeline) Dockerhub repository:

Run `bitflow-pipeline --help` for a list of command line flags.
```
docker pull bitflowstream/bitflow-pipeline
docker run -ti bitflowstream/bitflow-pipeline --help
```

Go requirement: at least version 1.11
The Docker manifest will select the appropriate platform (amd64/arm32v7/arm64v8) automatically. You can choose between the `static` and the `latest` tags, where the `static` image is smaller and contains a single file that can easily be integrated in other container images.