fluxcd / flux2-multi-tenancy

Kyverno Policy Engine

This repository utilizes Kyverno to enforce security policies and control access in a multi-tenant Kubernetes environment.

Kyverno Integration:

  • Setup: Kyverno is installed as a custom resource definition (CRD) in the flux-system namespace.
    • File: clusters/production/infrastructure.yaml
    • File: clusters/staging/infrastructure.yaml
  • Policy Enforcement: Kyverno policies are defined and managed in the infrastructure/kyverno-policies directory.
    • File: infrastructure/kyverno-policies/verify-git-repositories.yaml
    • File: infrastructure/kyverno-policies/verify-flux-images.yaml
  • Reconciliation: The Kyverno validation webhook is configured to reject resources that don’t meet the defined policies.
    • File: clusters/staging/tenants.yaml
    • File: clusters/production/tenants.yaml

Policies in Use:

  • Verify Git Repositories: Ensures Git repositories used in the cluster originate from a specific GitHub organization.
    • File: infrastructure/kyverno-policies/verify-git-repositories.yaml
    • Example: “.spec.url must be from a repository within the organisation X”
  • Verify Flux Images: Validates the images used by Flux components, ensuring they are signed with Cosign.
    • File: infrastructure/kyverno-policies/verify-flux-images.yaml
    • Example: Attestors: https://github.com/fluxcd/* with issuer https://token.actions.githubusercontent.com and https://rekor.sigstore.dev as rekor url.
  • Other Policies:
    • Restrict repositories: Define which repositories can be accessed per environment.
    • Align image policies: Ensure images used by highly privileged pods adhere to strict security requirements.

Kyverno Policy Examples:

  • Policy Name: verify-git-repositories.yaml
    • Description: Ensures Git repositories used in the cluster originate from a specific GitHub organization.
    • File: infrastructure/kyverno-policies/verify-git-repositories.yaml
  • Policy Name: verify-flux-images.yaml
    • Description: Validates the images used by Flux components, ensuring they are signed with Cosign.
    • File: infrastructure/kyverno-policies/verify-flux-images.yaml

Key Points:

  • Multi-Tenancy Lockdown: Flux’s built-in multi-tenancy lockdown features enhance tenant isolation without external admission controllers.
  • Least Privilege: The default service account used by tenants should have minimal privileges.
  • Security Posture: The Platform Admin can impose additional policies to meet specific security requirements.

Further Exploration:

  • Kyverno Documentation
  • Expanding Policies: Consider extending policies to cover HelmRepository, Bucket, and GitRepository resources with restrictions on protocols and providers.

Note: This outline provides a basic overview of Kyverno’s role in this multi-tenant Flux deployment. For more detailed information, refer to the specific policy files and the official Kyverno documentation.

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.