Skip to content

MetricsAggregator keeps everything in memory #51

@PetrosPapapa

Description

@PetrosPapapa

Issue

com.workflowfm.pew.metrics.MetricsAggregator stores all metrics from all workflows executed in memory.

This means it will keep consuming memory for each executed workflow, which is bad, particularly for long-term, persistent executions.

Suggestion 1

We need to introduce 2 implementations:

  1. An extension of the current in-memory implementation which stores metrics of completed processes and workflows in persistent storage (Mongo?) and clears them from memory.
  2. A fully persistent version that stores everything in persistent storage (Mongo?) and retrieves stuff it needs to update (we might also need CAS checking for conflicts).

We need both implementations to be available, as the second one is probably not good for fast workflows executing in memory.

The action of storing to persistent storage must be handled by a concurrent actor to avoid delaying workflow execution.

Suggestion 2

We might want to update the way outputs work so that they have the option to:

  1. Clear completed workflows from the aggregator.
  2. Only function on completed workflows.

The current implementation mirrors the behaviour of the simulation metrics, where it makes sense to collect all the metrics and run the output once when the simulation is finished. This, however, does not make sense for long-term or persistent execution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions