Skip to content

Enhance metrics recording to support arbitrary attributes #1577

@jfallows

Description

@jfallows

Today, metrics have the following structured key.

long bindingId
int metricId

...where bindingId encapsulates namespace and local name.

This means we cannot currently collect metrics for arbitrary attributes.

We can enhance the metrics structured key as follows.

int metricId
int attributesId

...where metricId remains unchanged, and attributesId represents a unique combination of dimension name value pairs.

We can encode the bindingId into the attributesId to remain compatible with today's behavior, and this still lets us add more attributes as needed.

Note: exporters don't need to be binding aware etc, so we could collect metrics for anything.

We would need a standard format for attributesId label so it is parseable back into structured form

Metrics handling frames themselves would need to parse attributes to apply filtering for observed traffic.

To avoid re-parsing protocol frames, we might need to supply the recorder dynamically by metricId and attributesId, then based on observed protocol values, compute the attributes and lookup the recorder.

Specific metrics would know if a particular dimension is relevant to them.
For example http metrics with attributes for method, path or status.

Some bindings may need to handle metrics directly instead of deferring to external frame handling.

This can be supported by marking the binding to exclude auto-telemetry handling for specific binding type(s),
then the XxxBindingConfig would have TelemetryRefConfig with List<MetricRefConfig> and (new) List<DimensionConfig>.

The internal EngineContext API would need to support enough primitives for stream handler to implement consistent behavior for metrics when handled by a binding directly.

zilla metrics should resolve metricId and attributesId plus counter/gauge/histogram types to output values
support generic filtering by metric(s) and/or attribute(s).

otlp exporter

  • needs concept of kind: server or client, to transform metric names appropriately
    otlp exporter can lookup binding type based on namespace and binding attributes
    no need to change how metrics are recorded generically just to accommodate name transformation for otlp
  • metrics reader approach needs to be more dynamic, support evolving set of metrics
    this may already be a bug if otlp exporter is already attached when a new namespace is added
    due to cached metrics records
  • export via OTLP with attributes for aggregated metrics with appropriate striping

Initially we can preallocate enough space to dynamically handle all distinct metrics for lifetime of gateway.
Later we can phase out unused metrics slots until needed again.
Static metrics ones would stay automatically because they are never released.
Knowing that a dynamic metric has been exported by all exporters makes its storage eligible for reuse.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions