-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Issue
com.workflowfm.pew.metrics.MetricsAggregator stores all metrics from all workflows executed in memory.
This means it will keep consuming memory for each executed workflow, which is bad, particularly for long-term, persistent executions.
Suggestion 1
We need to introduce 2 implementations:
- An extension of the current in-memory implementation which stores metrics of completed processes and workflows in persistent storage (Mongo?) and clears them from memory.
- A fully persistent version that stores everything in persistent storage (Mongo?) and retrieves stuff it needs to update (we might also need CAS checking for conflicts).
We need both implementations to be available, as the second one is probably not good for fast workflows executing in memory.
The action of storing to persistent storage must be handled by a concurrent actor to avoid delaying workflow execution.
Suggestion 2
We might want to update the way outputs work so that they have the option to:
- Clear completed workflows from the aggregator.
- Only function on completed workflows.
The current implementation mirrors the behaviour of the simulation metrics, where it makes sense to collect all the metrics and run the output once when the simulation is finished. This, however, does not make sense for long-term or persistent execution.