Kubernetes Admission Controllers

This repository leverages Kubernetes admission controllers to enforce policies and validate resource requests within a Kubernetes cluster. It’s a key element in implementing multi-tenancy, ensuring that tenants operate within defined boundaries and adhere to specific configuration rules.

Admission Controller Configuration

The Admission Controllers configuration within this repository is achieved primarily through kustomization patches applied to various resources.

• Default Service Account:

  The --default-service-account flag is used to set a default service account for kustomize-controller and helm-controller.

  Example:

  - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --default-service-account=default
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller)"
            

  Note: This default service account is recommended to have no privileges, and each named service account used should follow the least privilege principle.

• Cross-namespace References:

  The --no-cross-namespace-refs flag is used to prevent controllers from referencing resources in different namespaces. This enforces tenant isolation.

  Example:

  - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-cross-namespace-refs=true
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
            

• Remote Bases:

  The --no-remote-bases flag is used to disallow the use of remote Kustomize bases, ensuring that all resources refer to local files and controlled Flux sources.

  Example:

  - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-remote-bases=true
              target:
                kind: Deployment
                name: "kustomize-controller"
            

• Service Account Name in Kustomization:

  The serviceAccountName field in Kustomization resources is set to the kustomize-controller service account, ensuring proper impersonation for tenant resource reconciliation.

  Example:

  - patch: |
                - op: add
                  path: /spec/serviceAccountName
                  value: kustomize-controller
              target:
                kind: Kustomization
                name: "flux-system"
            

Tenant Isolation and Reconciliation

The implementation of tenant isolation revolves around service account impersonation and the restricted capabilities of the default service account.

• Tenant Service Accounts:

  Each tenant should be assigned a specific service account with appropriate permissions to manage their resources within their assigned namespaces.

• Default Service Account:

  The default service account exists in all namespaces and should be kept without any privileges. This ensures that tenants with missing service account configurations cannot perform unauthorized actions.

• Reconciliation with Tenant Service Accounts:

  Flux controllers, when applying changes to a tenant’s namespace, will impersonate the tenant’s service account, ensuring that only authorized actions are performed.

Kyverno Policies for Additional Enforcement

This repository uses Kyverno policies to further enforce configuration rules and validation:

• Cluster Policies:

  Cluster policies can be defined to validate resource requests, enforce image integrity, and control resource creation based on specified criteria.

• Resource Generation:

  Kyverno’s resource generation feature can be used to automate the creation of tenant resources like namespaces, service accounts, and role bindings.

CI/CD Validation and Testing

This repository contains GitHub CI workflows to ensure the integrity and functionality of the codebase:

• Test Workflow:

  This workflow validates Kubernetes manifests and Kustomize overlays using kubeconform.

• E2E Workflow:

  This workflow starts a Kubernetes cluster in CI and tests the staging setup by running Flux in Kubernetes Kind.

Conclusion

This repository demonstrates a robust multi-tenancy implementation using Kubernetes admission controllers, Kyverno policies, and carefully controlled service account permissions. This approach enforces strict tenant isolation and ensures secure resource management within a Kubernetes cluster.

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.