-
Notifications
You must be signed in to change notification settings - Fork 72
Description
Today, metrics have the following structured key.
long bindingId
int metricId
...where bindingId encapsulates namespace and local name.
This means we cannot currently collect metrics for arbitrary attributes.
We can enhance the metrics structured key as follows.
int metricId
int attributesId
...where metricId remains unchanged, and attributesId represents a unique combination of dimension name value pairs.
We can encode the bindingId into the attributesId to remain compatible with today's behavior, and this still lets us add more attributes as needed.
Note: exporters don't need to be binding aware etc, so we could collect metrics for anything.
We would need a standard format for attributesId label so it is parseable back into structured form
Metrics handling frames themselves would need to parse attributes to apply filtering for observed traffic.
To avoid re-parsing protocol frames, we might need to supply the recorder dynamically by metricId and attributesId, then based on observed protocol values, compute the attributes and lookup the recorder.
Specific metrics would know if a particular dimension is relevant to them.
For example http metrics with attributes for method, path or status.
Some bindings may need to handle metrics directly instead of deferring to external frame handling.
This can be supported by marking the binding to exclude auto-telemetry handling for specific binding type(s),
then the XxxBindingConfig would have TelemetryRefConfig with List<MetricRefConfig> and (new) List<DimensionConfig>.
The internal EngineContext API would need to support enough primitives for stream handler to implement consistent behavior for metrics when handled by a binding directly.
zilla metrics should resolve metricId and attributesId plus counter/gauge/histogram types to output values
support generic filtering by metric(s) and/or attribute(s).
otlp exporter
- needs concept of kind: server or client, to transform metric names appropriately
otlp exporter can lookup binding type based on namespace and binding attributes
no need to change how metrics are recorded generically just to accommodate name transformation for otlp - metrics reader approach needs to be more dynamic, support evolving set of metrics
this may already be a bug if otlp exporter is already attached when a new namespace is added
due to cached metrics records - export via OTLP with attributes for aggregated metrics with appropriate striping
Initially we can preallocate enough space to dynamically handle all distinct metrics for lifetime of gateway.
Later we can phase out unused metrics slots until needed again.
Static metrics ones would stay automatically because they are never released.
Knowing that a dynamic metric has been exported by all exporters makes its storage eligible for reuse.