Testing and Debugging @ fluxcd/flux2-multi-tenancy

Testing and Debugging

This repository uses several testing and debugging techniques to ensure the stability and security of the multi-tenancy system.

CI Workflows

This repository includes the following GitHub CI workflows:

Test workflow: This workflow validates the Kubernetes manifests and Kustomize overlays using kubeconform.
E2E workflow: This workflow starts a Kubernetes cluster in CI and tests the staging setup by running Flux in Kubernetes Kind.

Testing Techniques

1. Manifest Validation with Kubeconform:

The scripts/validate.sh script uses kubeconform to validate the Kubernetes manifests and Kustomize overlays before merging them into the main branch. This script:

Downloads the Flux OpenAPI schemas.
Validates the Kubernetes manifests for syntax errors.
Validates the Kustomize overlays using kubeconform.

Example:

# Validate Kustomize overlays
          kustomize build "${file/%$kustomize_config}" "${kustomize_flags[@]}" | \
            kubeconform "${kubeconform_flags[@]}" "${kubeconform_config[@]}"

2. End-to-End Testing with Kubernetes Kind:

The E2E workflow utilizes Kubernetes Kind to create a test environment.

3. Kyverno Policy Enforcement:

The Kyverno policy engine is used to enforce security and governance policies, including:

Enforcing Git repository sources across all tenants.
Validating the use of approved Flux image tags.

Example:

The infrastructure/kyverno-policies/verify-git-repositories.yaml policy enforces that all Git repositories must originate from a specific GitHub organization.

apiVersion: kyverno.io/v1
          kind: ClusterPolicy
          metadata:
            name: verify-git-repositories
          spec:  
            # This provides users a working example of how an admin
            # would be able to enforce git repository sources across
            # all tenants.
            validationFailureAction: Audit # Change to 'Enforce' once the specific org url is set.
            rules:
              - name: github-repositories-only
                match:
                  any:
                  - resources:
                      kinds:
                        - GitRepository
                exclude:
                  any:
                  - resources:
                      namespaces:
                        - flux-system
                validate:
                  message: ".spec.url must be from a repository within the organisation X"
                  pattern:
                    spec:
                      url: https://github.com/fluxcd/?* |

4. Securing Tenant Credentials:

The multi-tenancy system encrypts tenant credentials using SOPS with a GPG key. This ensures that the credentials are stored securely in the Kubernetes cluster.

Example:

The tenants/base/dev-team/auth.yaml file is encrypted using SOPS:

sops --encrypt \
          --pgp=1F3D1CED2F865F5E59CA564553241F147E7C5FA4 \
          --encrypted-regex '^(data|stringData)$' \
          --in-place ./tenants/base/dev-team/auth.yaml

5. Flux Bootstrap Command:

The flux bootstrap command creates a deploy key with read-only access to the GitHub repository, enabling Flux to pull changes within the cluster.

6. Flux CLI Usage:

The Flux CLI is used for interacting with the Flux instances and managing resources within the cluster.

7. Multi-tenancy Lockdown:

Multi-tenancy lockdown provides a robust mechanism for isolating tenant workloads and enhancing security.

Example:

The flux-system/kustomization.yaml file contains patches that enable cross-namespace reference restrictions.

apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          resources:
            - gotk-components.yaml
            - gotk-sync.yaml
          patches:
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-cross-namespace-refs=true
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-remote-bases=true
              target:
                kind: Deployment
                name: "kustomize-controller"
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --default-service-account=default
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller)"
            - patch: |
                - op: add

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.