Design Proposal: Integration with DataFlow (TPL)

[branimirangelov]

Background

DataFlow (TPL) is high level library for writing data flow pipelines (based on TPL). The library is not part of .NET Framework, but it is distributed as an external NuGet. However it is well supported by Microsoft with mid-term roadmap to .NET 6.0 and beyond. ISourceBlock and ITargetBlock abstractions fit naturally with Perper and Perper fits well with DataFlow further extending its parallelization capabilities beyond a single machine.

Proposal

1. Support ISourceBlock and ITargetBlock in addition to IAsyncEnumerable as function signatures.

public ITargetBlock<string> RunAsync([PerperTrigger] ISourceBlock<string> input)
{
    // create arbitrary data flow network 
}

Note: Providing custom ISourceBlock implementation will be very similar to IAsyncEnumerable implementation in Perper. Receiving ITargetBlock messages will be part of the binders / converters logic in the extension (similarly to handling IAsyncEnumerable)

2. Add extension method AsSourceBlock() of Perper IStream abstraction.

var realTime = await _context.StreamFunctionAsync<dynamic>(nameof(RealTimeMinuteBars), symbols);
// realTime.AsSourceBlock() can be used to incorporate it in a data flow network

3. Add ITargetBlock as function signature for blank streams in addition to IAsyncCollector.

public async Task<string> RunAsync(
            [PerperTrigger(ParameterExpression="{\"stream\":0}")] (string, string, long) parameters, 
            [Perper] ITargetBlock<string> symbols)