Service Accounts and RBAC

This document describes the use of Service Accounts and Role-Based Access Control (RBAC) within Kubernetes and how they are used to manage tenant permissions in a multi-tenant environment.

Motivation

This repository implements a best practice of locking down access to the cluster by applying the least privilege model. README.md This means that by default, Flux resources such as Kustomization and HelmRelease are not allowed to apply changes to the cluster.

Implementation

To enable tenants to operate, a service account must be created with the required permissions and its name set to the spec.serviceAccountName of all Kustomization and HelmRelease resources the tenant has. README.md

Best Practices

• Least Privilege Model: Each service account used should observe the least privilege model. README.md
• Default Service Account: The default service account exists in all namespaces and should always be kept without any privileges. README.md

Example:

The following example shows the creation of a service account named dev-team with access to the apps namespace.

flux create tenant dev-team --with-namespace=apps \
          --export > ./tenants/base/dev-team/rbac.yaml
          

RBAC Manifest:

The rbac.yaml file defines the namespace, service account, and role binding for the dev-team tenant.

---
          apiVersion: v1
          kind: Namespace
          metadata:
            labels:
              toolkit.fluxcd.io/tenant: dev-team
            name: apps
          
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            labels:
              toolkit.fluxcd.io/tenant: dev-team
            name: dev-team
            namespace: apps
          
          ---
          apiVersion: rbac.authorization.k8s.io/v1
          kind: RoleBinding
          metadata:
            labels:
              toolkit.fluxcd.io/tenant: dev-team
            name: gotk-reconciler
            namespace: apps
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: ClusterRole
            name: cluster-admin
          subjects:
          - kind: User
            name: gotk:apps:reconciler
          - kind: ServiceAccount
            name: dev-team
            namespace: apps
          

Kustomization Manifest:

The kustomization.yaml file defines the resources for the dev-team tenant.

apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
            - rbac.yaml
            - sync.yaml
          

Flux CLI Commands:

The Flux CLI provides commands to generate the Kubernetes manifests needed to define tenants. README.md

Tenant Roles:

• Platform Admin: Manages cluster-wide resources, onboards tenants, and assigns namespaces, service accounts, and role bindings. README.md
• Tenant: Manages app deployments and releases within their assigned namespaces. README.md

Security Policies:

The platform admin can impose additional security policies to enforce specific behaviors. README.md

Example:

apiVersion: kyverno.io/v1
          kind: ClusterPolicy
          metadata:
            name: verify-flux-images
          spec:
            validationFailureAction: Audit
            background: false
            webhookTimeoutSeconds: 30
            failurePolicy: Fail
            rules:
              - name: verify-cosign-signature
                match:
                  any:
                  - resources:
                      kinds:
                        - Pod
                verifyImages:
                  - imageReferences:
                      - "ghcr.io/fluxcd/source-controller:*"
                      - "ghcr.io/fluxcd/kustomize-controller:*"
                      - "ghcr.io/fluxcd/helm-controller:*"
                      - "ghcr.io/fluxcd/notification-controller:*"
                      - "ghcr.io/fluxcd/image-reflector-controller:*"
                      - "ghcr.io/fluxcd/image-automation-controller:*"
                      - "docker.io/fluxcd/source-controller:*"
                      - "docker.io/fluxcd/kustomize-controller:*"
                      - "docker.io/fluxcd/helm-controller:*"
                      - "docker.io/fluxcd/notification-controller:*"
                      - "docker.io/fluxcd/image-reflector-controller:*"
                      - "docker.io/fluxcd/image-automation-controller:*"
                    mutateDigest: false
                    verifyDigest: false
                    attestors:
                      - entries:
                          - keyless:
                              subject: "https://github.com/fluxcd/*"
                              issuer: "https://token.actions.githubusercontent.com"
                              rekor:
                                url: https://rekor.sigstore.dev
          

This policy verifies the signature of images from ghcr.io/fluxcd and docker.io/fluxcd. infrastructure/kyverno-policies/verify-flux-images.yaml

Note:

This repository uses a combination of Kubernetes Service Accounts, RBAC, and Kyverno policies to implement a secure multi-tenant environment. This approach ensures that each tenant only has access to the resources they need, while also providing flexibility and extensibility.

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.