fluxcd / flux2-multi-tenancy

This documentation focuses on the strategies and methods for monitoring the fluxcd/flux2-multi-tenancy project in a production environment. The emphasis is on using various Kubernetes tools and Flux capabilities to ensure that production workloads are continuously operating as expected.

Monitoring Overview

Proper monitoring of production environments requires validating Helm releases, Kubernetes manifests, and CI processes. The following sections detail the techniques employed to monitor the fluxcd/flux2-multi-tenancy deployment effectively.

Prerequisites

Ensure that the necessary tools are installed:

  • Flux CLI
  • Kustomize
  • kubeconform
  • yq

Continuous Integration Validation

To enforce proper versioning and quality of manifests, GitHub CI workflows are set up to validate Kubernetes manifests before merges. Examples of validation scripts include:

Validate Helm Releases

# Validate kustomize overlays
echo "INFO - Validating kustomize overlays"
find . -type f -name "$kustomize_config" -print0 | while IFS= read -r -d $'\0' file; do
  echo "INFO - Validating kustomization ${file/%$kustomize_config}"
  kustomize build "${file/%$kustomize_config}" "${kustomize_flags[@]}" | \
  kubeconform "${kubeconform_flags[@]}" "${kubeconform_config[@]}"
  if [[ ${PIPESTATUS[0]} != 0 ]]; then
    exit 1
  fi
done

This script ensures that all kustomize configurations are valid before they are applied.

Monitoring Helm Releases

Monitor the health of Helm releases deployed in the production cluster with the following command:

# Verify installed Helm releases
watch flux -n apps get helmreleases

Expected output:

NAME   	READY	MESSAGE                         	REVISION	SUSPENDED
podinfo	True 	Release reconciliation succeeded	5.0.3   	False

The command above repeatedly checks the status of Helm releases and confirms whether they are successfully reconciled.

Kustomization Configuration for Monitoring

The Kustomization resources in the production cluster describe the interval for monitoring and the source reference for the configurations. An example kustomization configuration is shown below:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenants
  namespace: flux-system
spec:
  dependsOn:
    - name: kyverno-policies
  interval: 5m
  serviceAccountName: kustomize-controller
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./tenants/production
  prune: true

Ensure Provenance of Container Images

It is crucial to verify the provenance of container images utilized within the cluster. The following Kyverno policy can be defined to ensure that only signed images are deployed in the cluster:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-flux-images
spec:
  validationFailureAction: enforce
  background: false
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
rules:
- name: verify-cosign-signature
  match:
    resources:
      kinds:
        - Pod
  verifyImages:
    - imageReferences:
        - "ghcr.io/fluxcd/source-controller:*"
        - "ghcr.io/fluxcd/kustomize-controller:*"
        - "ghcr.io/fluxcd/helm-controller:*"
        - "ghcr.io/fluxcd/notification-controller:*"
      attestors:
        - entries:
            - keyless:
                subject: "https://github.com/fluxcd/*"
                issuer: "https://token.actions.githubusercontent.com"
                rekor:

This policy will enforce image signing during deployment, enhancing the security posture of the cluster.

Validate Cluster Configuration

Validate all manifest configurations periodically to maintain cluster health:

# Download Flux OpenAPI schemas for validation
mkdir -p /tmp/flux-crd-schemas/master-standalone-strict
curl -sL https://github.com/fluxcd/flux2/releases/latest/download/crd-schemas.tar.gz | tar zxf - -C /tmp/flux-crd-schemas/master-standalone-strict

# Validate each YAML file
find . -type f -name '*.yaml' -print0 | while IFS= read -r -d $'\0' file; do
  echo "INFO - Validating $file"
  yq e 'true' "$file" > /dev/null
done

echo "INFO - Validating clusters"
find ./clusters -maxdepth 2 -type f -name '*.yaml' -print0 | while IFS= read -r -d $'\0' file; do
  kubeconform "${kubeconform_flags[@]}" "${kubeconform_config[@]}" "${file}"
  if [[ ${PIPESTATUS[0]} != 0 ]]; then
    exit 1
  fi
done

This automated validation script checks all YAML files in the repository and verifies their compliance with the expected formats.

Conclusion

By following the outlined procedures for monitoring within production for fluxcd/flux2-multi-tenancy, teams can maintain high levels of observability over their workloads, ensuring stable and secure operations.

Sources:

  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • README.md
  • scripts/validate.sh
  • clusters/production/tenants.yaml
  • clusters/production/flux-system/kustomization.yaml
  • clusters/production/infrastructure.yaml
  • clusters/staging/infrastructure.yaml
  • tenants/production/dev-team-patch.yaml
  • clusters/production/flux-system/gotk-sync.yaml
  • scripts/validate.sh