Feature Request: Additional Batch Modes

[antongulenko]

The BatchPipelineStep should support following modes:

• flush after a fixed number of samples
• flush after a fixed time period

These modes and the names of the parameters etc. should follow the conventions here: https://bitflow.readthedocs.io/en/latest/naming/

[kevinstyp]

In addition to that, the "flush after a fixed number of samples" should additionally include an optional separationTag, where the batch is only flushed, when a fixed number of samples with the same tag value of the separationTag have been collected.

[antongulenko]

That sounds like you want samples with different values for the separationTag to go through the same batch step. That should be handled through forking by the separationTag instead. A batch step should assume that all samples going through it belong to one single data stream.

[kevinstyp]

Yes and no, yes i would like samples with different separationTags to go through the same batch step as they belong into the same pipeline. And No, forking is not possible for my use case. E.g. 3 Batches (belonging to the same pipeline) come in at the same time, this means all of their samples come in unsorted / undeterministic. Now, the collector shall collect all the samples of the respective batch by looking at the separationTag until the fixed number is met, thus bring order into the stream of mixed (in term of respective batches) samples.