The github.com/docker/go-metrics
library is used for metrics collection in Go applications. Metrics can be of different types and units, which are defined by the OpenTelemetry specification. The following are the types of metrics supported by the OpenTelemetry specification:
- Counter: A counter is a metric that reports an instantaneous measurement and can only increase or stay the same. Counter values cannot be negative.
- UpDownCounter: An UpDownCounter is a metric that reports an instantaneous measurement and can increase or decrease.
- Histogram: A histogram is a metric that reports the distribution of values in a set of data.
- Gauge: A gauge is a metric that reports an instantaneous measurement of a numeric value. Gauges can go up and down, and their values can be negative.
The units of a metric can be defined using the metric.WithUnit
function in the github.com/docker/go-metrics
library. The following are the base units used by Prometheus for better compatibility:
- Time: seconds
- Temperature: celsius
- Length: meters
- Bytes: By
When creating a new metric, it is recommended to follow the Metrics Semantic Conventions defined by OpenTelemetry. A new metric should not have the same name as a metric that existed in the past but was renamed. Units should not be included in the metric name unless it provides additional meaning to the metric name.
In manual instrumentation, it is important to explicitly copy the name, description, and unit when creating a new Stream
for the instrument. If the instrument does not have a unit suffix defined, but has a dimensional unit defined, the name should be updated with a unit suffix.
When transforming metrics, it is useful to include the name of the collector or exporter, the ID of any rule that was applied, and the name and details of the original metric into the help string. This will aid users in tracking back to what the original was and what rules were in play that caused the transformation.
In Prometheus, every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Therefore, it is important to not use labels to store dimensions with high cardinality, such as user IDs, email addresses, or other unbounded sets of values.
Sources:
- Manual instrumentation of Go applications with OpenTelemetry Metrics SDK | Opentelemetry documentation: https://grafana.com/docs/opentelemetry/instrumentation/go/manual-instrumentation
- HTTP Requests Latency | Kyverno: https://kyverno.io/docs/monitoring/http-requests-latency
- Overview | OpenTelemetry: https://opentelemetry.io/docs/specs/otel/overview
- Writing exporters | Prometheus: https://prometheus.io/docs/instrumenting/writing_exporters
- Metrics Semantic Conventions | OpenTelemetry: https://opentelemetry.io/docs/specs/semconv/general/metrics
- Introduction to open source observability on Kubernetes | Opensource.com: https://opensource.com/article/19/10/open-source-observability-kubernetes
- azure_exporter_config | Grafana Cloud documentation: https://grafana.com/docs/grafana-cloud/monitor-infrastructure/agent/static/configuration/integrations/azure-exporter-config
- azure_exporter_config | Grafana Agent documentation: https://grafana.com/docs/agent/latest/static/configuration/integrations/azure-exporter-config
- Manual Instrumentation | OpenTelemetry: https://opentelemetry.io/docs/instrumentation/go/manual
- Metric and label naming | Prometheus: https://prometheus.io/docs/practices/naming