qa: fairly split integration tests using hints

[yaauie]

Release notes

[rn:skip]

What does this PR do?

Refactors test-spliting code to improve the fairness of test partitions using hints

Why is it important/What is the impact to the user?

No user impact.

Because CI time-to-result is delayed until the slowest split of integration tests completes, we will have net improvement by ensuring that the tests are split more evenly with regard to each file's execution time.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

• observe that CI tests splits are relatively similar in duration
• run ci/partition-files.lib.sh test to run permutation validation on the file partitioner.

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

[mergify]

This pull request does not have a backport label. Could you fix it @yaauie? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 6b082d6

Failed CI Steps

• Observability SRE container smoke test
• Observability SRE container smoke test
• Observability SRE container smoke test

History

• 💚 Build #4058 succeeded c4afd39
• 💚 Build #4055 succeeded c4afd39
• 💔 Build #4054 failed 6f6c35c

[yaauie]

scenario	waterfall
round-robin
fair-split with hints

[donoghuc]

Love this idea. From https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/4064#019bb399-033e-4505-82b0-f5c68cfe5d77 it looks like the estimate/actual match relative to themselves but the actual observed values are a bit off. not sure if this is just difference between running on a laptop vs CI (or maybe even variance in CI runners).

Not sure if its worth it, but theoretically we could probably have rspec write the data to files and then export those files as build artifacts to track over time. That being said, the relative times probably mattern more than absolute so that might not be worth it.

spec	estimate	actual
cli/install_spec	284	556
cli/remove_spec	240	409
deprecation_log_spec	8	15
fatal_error_spec	15	26
cli/keystore_spec	10	20
cli/list_spec	24	47
env_variables_config_spec	7	13
mixed_codec_spec	26	46
monitoring_api_spec	180	384
reload_config_spec	20	23
01_logstash_bin_smoke_spec	98	124
install_java_plugin_spec	40	44
logstash_to_logstash_spec	20	21
multiple_pipeline_spec	30	42
secret_store_spec	41	61
slowlog_spec	12	13
beats_input_spec	80	85
cli/update_spec	40	81
env_variables_condition_spec	19	38
pipeline_log_spec	50	90
plugin_name_log_spec	11	17
webserver_spec	5	4
cli/prepare_offline_pack_spec	60	81
command_line_spec	12	16
direct_heap_allocator_flag_spec	40	41
dlq_spec	30	37
java_api_spec	20	19
kafka_input_spec	8	10
reserved_tags_field_spec	21	27
settings_spec	76	99

[yaauie]

running on a laptop vs CI

Likely. I recorded them all in 5 runs on my laptop (with FEATURE_FLAG=persistent_queues), and used the median value.

theoretically we could probably have rspec write the data to files and then export those files as build artifacts to track over time

I agree, but avoided expanding the scope. This would be a good follow-up though.