distribution / distribution

What is Metrics and Observability?

Metrics and observability are crucial for gaining insights into the health, performance, and usage patterns of a system. Metrics involve collecting numerical data points, often aggregated over time, to track key aspects of a system’s behavior. Observability, on the other hand, refers to the ability to understand the internal workings of a system through the analysis of various data sources, including logs, traces, and metrics.

Why is Metrics and Observability important?

Metrics and observability play a vital role in:

  • Identifying and resolving issues: By monitoring key metrics, developers can quickly detect anomalies and potential problems, enabling prompt intervention and resolution.
  • Performance optimization: Analyzing metrics and logs allows developers to identify performance bottlenecks and optimize system behavior for improved efficiency.
  • Capacity planning: Understanding system usage patterns through metrics provides valuable data for capacity planning, ensuring adequate resources are available to meet future demands.
  • Feature development and validation: Metrics can be used to track the impact of new features and validate their effectiveness.

Metrics Implementation

The Distribution registry utilizes Prometheus as its primary metrics collection and aggregation tool.

Prometheus documentation

Prometheus provides a powerful and flexible framework for collecting and visualizing metrics. The registry exposes a variety of metrics, grouped into different categories:

Registry Operations:

  • registry_http_requests_total: Total number of HTTP requests received by the registry.
  • registry_http_request_duration_seconds: Distribution of HTTP request durations.
  • registry_http_response_status_count: Number of requests for each HTTP status code returned.
  • registry_gc_runs_total: Total number of garbage collection runs.
  • registry_gc_duration_seconds: Distribution of garbage collection durations.

Blob Storage Operations:

  • registry_blob_uploads_total: Total number of blob uploads.
  • registry_blob_downloads_total: Total number of blob downloads.
  • registry_blob_upload_duration_seconds: Distribution of blob upload durations.
  • registry_blob_download_duration_seconds: Distribution of blob download durations.

Image Storage Operations:

  • registry_image_pulls_total: Total number of image pulls.
  • registry_image_pushes_total: Total number of image pushes.
  • registry_image_pull_duration_seconds: Distribution of image pull durations.
  • registry_image_push_duration_seconds: Distribution of image push durations.

Registry Health:

  • registry_errors_total: Total number of errors encountered by the registry.
  • registry_uptime_seconds: Registry uptime in seconds.
  • registry_memory_usage_bytes: Registry memory usage in bytes.
  • registry_disk_usage_bytes: Registry disk usage in bytes.

Additional Metrics:

  • registry_storage_driver_operations_total: Total number of operations performed by the storage driver.
  • registry_storage_driver_operation_duration_seconds: Distribution of storage driver operation durations.

Note: These are just a few examples, and the registry exposes numerous other metrics. Refer to the codebase for a complete list.

Observability Implementation

The Distribution registry utilizes Jaeger for distributed tracing.

Jaeger Documentation

Jaeger enables tracing requests through the registry, providing detailed insights into request flows and potential bottlenecks.

Key Features:

  • Span Tracing: Jaeger allows developers to track individual requests, tracing their paths through the system.
  • Distributed Tracing: Jaeger supports tracing across multiple services and applications, providing a holistic view of request flow.
  • Analysis and Visualization: Jaeger provides tools for analyzing and visualizing traces, identifying performance issues and understanding system behavior.

Top-Level Directory Explanations

metrics/ - This directory is related to the metrics functionality of the distribution project, specifically for Prometheus.

registry/ - This directory is related to the GitHub Package Registry functionality of the distribution project. It includes various subdirectories for API, auth, handlers, listener, middleware, proxy, storage, and more.