Shoulder.dev Logo Shoulder.dev

Monitoring and Logging with FluxCD

Scenario: A developer wants to monitor and log FluxCD deployments using FluxCD’s built-in features or external tools. In this example, we will cover how to use FluxCD’s logging and monitoring capabilities, focusing on displaying formatted logs for Flux components and monitoring Flux events with Prometheus and Grafana.

First, let’s explore FluxCD’s logging capabilities. FluxCD provides a flux logs command to display formatted logs from various Flux components. This command is in preview and under development, but it can be used to print logs from different Flux controllers.

Codebase Files:

  • cmd/flux/logs.go
  • cmd/flux/logs_unit_test.go

To use the flux logs command, run the following command:

flux logs [flags]

Some common usage examples include:

  • Print the reconciliation logs of all Flux custom resources in your cluster: flux logs --all-namespaces
  • Print all logs of all Flux custom resources newer than 2 minutes: flux logs --all-namespaces --since=2m
  • Stream logs for a particular log level: flux logs --follow --level=error --all-namespaces
  • Filter logs by kind, name, and namespace: flux logs --kind=Kustomization --name=podinfo --namespace=default

Next, let’s discuss monitoring Flux events with Prometheus and Grafana. Flux controllers emit Kubernetes events during the reconciliation operation to provide information about the object being reconciled. Events are supplemental data that can be used along with logs to provide a complete picture of controllers’ operations.

Documentation:

To monitor Flux events, you can use Prometheus and Grafana. First, set up the monitoring example provided in the fluxcd/flux2-monitoring-example repository. This example setup includes Promtail, Loki, and Grafana. Promtail aggregates the logs from all the Pods in every node and sends them to Loki. Grafana can be used to query the logs from Loki and analyze them.

Codebase Files:

  • cmd/flux/bootstrap_prometheus.go
  • cmd/flux/bootstrap_grafana.go
  • manifests/monitoring/configs/dashboards/logs.json

Once the monitoring setup is complete, you can access the Grafana dashboard to browse logs from all the Flux controllers in a centralized manner. This can be useful for monitoring the overall state of the Flux Sources and Cluster Reconcilers.

Tests:

To verify the monitoring and logging setup, you can perform the following tests:

  1. Run flux logs --all-namespaces and check the logs displayed in the terminal.
  2. Access the Grafana dashboard and verify that logs from all the Flux controllers are displayed.
  3. Use the Grafana query editor to search for specific logs based on the kind, name, and namespace.
  4. Use the Prometheus query editor to retrieve metrics related to Flux controllers.

By following this example, you will have a better understanding of how to monitor and log FluxCD deployments using FluxCD’s built-in features and external tools like Prometheus and Grafana.