Streamz.farm/4

[hamiltop]

Spec:

@spec farm(Enumerable.t, (Enumerable.t -> [Enumerable.t]), (any -> any), ([Enumerable.t] -> Enumerable)) :: Enumerable.t
def farm(stream, distribute, map, reduce)

Example code:

common_friends = Streamz.farm(
  users,
  &hashed_distribute/1,
  &fetch_friends/1,
  &find_intersection/1
)

Explanation:
In this example, users would be bucketed together based on hash. Each bucket would then enumerate its users and fetch the list of friends for each user. The friend list would be combined back together via the intersection of elements.

The fetching the list stage will be in parallel. The distribute function can be custom, but we probably want to provide a bunch of useful built-ins (round robin, group by, etc.).

As for initial implementation, this should be easy to build on top of group_by and merge. The question is whether performance is going to be sufficient.