Feature: Metrics for s3 operations

[klaudworks]

I'd like to collect s3 operation metrics to evaluate performance issues of s3 compatible storages in Prometheus.

Here some example metrics:

s3proxy_request_duration_seconds{method="PUT",bucket="opensearch-backups",status="200"} 
s3proxy_request_duration_seconds{method="GET",bucket="opensearch-backups",status="200"}
s3proxy_backend_duration_seconds{provider="s3",operation="putBlob"}
s3proxy_active_requests
s3proxy_errors_total{type="timeout"}

I'd just use Micrometer to implement it.

@gaul would you be open to a PR?

[gaul]

@klaudworks Definitely, although I'd like to align with some other S3 implementation instead of creating something bespoke for only S3Proxy. #780 and #923 have some other background on this.

[klaudworks]

okay, I will follow the OTEL spec first. They have a single required metric: http.server.request.duration. This actually covers a lot of request related use cases by using different aggregations over the attributes.

Apart from the OTEL spec's attributes, I'll use an additional custom attribute to store the s3 operation. This is way more elegant than reimplementing minio metrics or implementing and extending default s3 metrics to make them useful.

I'll focus on http.server.request.duration first. Later on, we could add additional metrics as needed. For those we could follow the aws / minio metrics for s3 e.g. for a time to first byte metric.

[klaudworks]

I'd go for the OTEL SDK instead of micrometer unless you have a strong preference. Should be roughly the same effort.