First idea in adding histogram metric

[andsel]

Release notes

[rn:skip]

What does this PR do?

Create a new histogram metric type. Updated all metric infrastructure code, the core class is HistogramMetric which uses HdrHistogram Recorder to track the measurements and produces a new HistogramSnapshot every time getValue is invoked, clearing the measurements stored in the Recorder.
HistogramSnapshot is a data class that exposes 75Percentile and 90Percentile (at the moment).

To verify the effectiveness of this, the memory queue read client was updated to expose a metric to track the batch size.

Created a new setting named pipeline.batch.metrics with values "true" and "false" (at the moment), to enable and disable the computation of such metrics. This setting is a string because in a follow up PR will become a tri-state flag.

Why is it important/What is the impact to the user?

This is an intermediate step, it has to proof the exposition of new metric section under the _node/stats API endpoint like:

"pipelines": {
    "main": {
      ...
      "batch": {
        "event_size": {
          "p75": 12.4
          "p90": 13.1
        }
    }
  }
}

Percentiles are also pushed down to ES when leveraging the monitoring.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• Check with xpack monitoring

How to test this PR locally

Run Logstash with the setting pipeline.batch.metrics with values "true" and verify that node stats exposes the batch_size histogram.

curl http://localhost:9600/_node/stats | jq .pipelines.main.events

Related issues

• Relates Create new histogram metric to expose batch size percentiles amongst a time windows (1 min, 5 min, 15min, 1h, 1d) #17895
• Closes Implement histogram metric with flow retention windows for 1, 5, and 15 minutes. #17973

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label. Could you fix it @andsel? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[andsel]

Note for reviewer

Implements Serializable so that Valuefier can use the identity converter.

[andsel]

Note for reviewer

The x-pack monitoring pipeline create Logstash events which contains this snapshot, and need to be converted to Ruby object in the Rubyfier.deep method.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: edcf61c

Failed CI Steps

• 🥼 x-pack unit tests - FIPS mode

History

• 💔 Build #3300 failed d028816
• 💚 Build #3299 succeeded 1e27899
• 💚 Build #3295 succeeded 506c59c
• 💚 Build #3288 succeeded 66cd2b3
• 💚 Build #3287 succeeded 5932065

cc @andsel

[andsel]

Super seeded by #18503