Authentication
The flux CLI can be used to create secrets for authentication with private repositories. This example uses git for a Git repository, but the same pattern can be used for other source types.
Filename: README.md
flux -n apps create secret git dev-team-auth \
\
--export > ./tenants/base/dev-team/auth.yaml
SSH Keys
This method requires generating a SSH key pair and storing the private key in a Kubernetes secret. The public key is added as a read-only deploy key to the repository.
flux -n apps create secret git dev-team-auth \
\
--export > ./tenants/base/dev-team/auth.yaml
yq eval 'data."identity.pub"' git-auth.yaml | base64 --decode
GPG Keys
For secure storage, you can encrypt secrets with GPG. Generate a GPG key pair.
$ gpg --full-generate-key
Email address:
Create a Kubernetes secret in the flux-system namespace with the GPG private key.
gpg --export-secret-keys \
--armor 1F3D1CED2F865F5E59CA564553241F147E7C5FA4 |
kubectl create secret generic sops-gpg \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin
Store the GPG private key safely for disaster recovery. The GPG public key can be shared with the platform team for encrypting secrets.
Filename: README.md
Basic Auth
To authenticate with basic auth credentials, use the following command.
flux -n apps create secret git dev-team-auth \
--url=https://github.com// \
--username=$GITHUB_USERNAME \
--password=$GITHUB_TOKEN \
--export > ./tenants/base/dev-team/auth.yaml
The GitHub token must have read-only access to the dev-team repository.
Filename: README.md
Encryption
Secrets stored in Git repositories can be encrypted with SOPS to protect them. This example demonstrates encrypting the dev-team-auth secret using GPG.
sops --encrypt \
--pgp=1F3D1CED2F865F5E59CA564553241F147E7C5FA4 \
--encrypted-regex '^(data|stringData)$' \
--in-place ./tenants/base/dev-team/auth.yaml
Image Verification
The verify-flux-images policy ensures that all Flux images used are the ones built and signed by the Flux team. It uses Cosign to verify the images’ attestations.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-flux-images
spec:
validationFailureAction: enforce
background: false
webhookTimeoutSeconds: 30
failurePolicy: Fail
rules:
- name: verify-cosign-signature
match:
resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "ghcr.io/fluxcd/source-controller:*"
- "ghcr.io/fluxcd/kustomize-controller:*"
- "ghcr.io/fluxcd/helm-controller:*"
- "ghcr.io/fluxcd/notification-controller:*"
attestors:
- entries:
- keyless:
subject: "https://github.com/fluxcd/*"
issuer: "https://token.actions.githubusercontent.com"
rekor:
rekor:
url: https://rekor.sigstore.dev
Flux Configuration
The Flux configuration is set up to decrypt secrets using the sops-gpg key.
flux create kustomization tenants \
--depends-on=kyverno-policies \
--source=flux-system \
--path="./tenants/staging" \
--prune=true \
--interval=5m \
--validation=client \
--decryption-provider=sops \
--decryption-secret=sops-gpg \
--export > ./clusters/staging/tenants.yaml
Kyverno Policies
Kyverno policies are used to enforce specific GitHub organizations for GitRepository resources.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: kyverno-policies
namespace: flux-system
spec:
dependsOn:
- name: kyverno
interval: 5m
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure/kyverno-policies
prune: true
Kustomization
The following Kustomizations are used for the dev-team tenant.
Base Kustomization:
flux create kustomization dev-team \
--namespace=apps \
--service-account=dev-team \
--source=GitRepository/dev-team \
--path="./" \
--export >> ./tenants/base/dev-team/sync.yaml
cd ./tenants/base/dev-team/ && kustomize create --autodetect --namespace apps
Staging Overlay:
cat << EOF | tee ./tenants/staging/dev-team-patch.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: dev-team
namespace: apps
spec:
path: ./staging
EOF
Tenant Sync Manifest:
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: dev-team
namespace: apps
spec:
interval: 1m
url: https://github.com/fluxcd/flux2-multi-tenancy
ref:
branch: dev-team
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: dev-team
namespace: apps
spec:
serviceAccountName: dev-team
interval: 5m
sourceRef:
kind: GitRepository
name: dev-team
prune: true
Filename: tenants/base/dev-team/sync.yaml
Other Policies
The following policies are suggested for further consideration:
- Restricting repositories accessible in each cluster. This can be useful for environment-specific deployments.
- Aligning image policies with pods requiring highly privileged
securityContext. Filename: README.md
Service Account Defaulting
The kustomize-controller and helm-controller deployments have a default service account set via --default-service-account. This means that if a tenant does not specify a service account in a Flux Kustomization or HelmRelease, it will automatically default to the specified account.
It is recommended that the default service account has no privileges and that each named service account observes the least privilege model.
This repository applies the following patch automatically to ensure this configuration:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- patch: |
- op: add
path: /spec/template/spec/containers/0/args/-
value: --no-cross-namespace-refs=true
target:
kind: Deployment
Top-Level Directory Explanations
clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.
clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.
clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.
clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.
clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.
infrastructure/ - This directory contains infrastructure-related configuration files and scripts.
infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.
infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.
scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.
tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.
tenants/base/ - This directory contains configuration and scripts for the base tenant.
tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.
tenants/production/ - This directory contains configuration and scripts for the production tenant.
tenants/staging/ - This directory contains configuration and scripts for the staging tenant.
