This documentation focuses on the steps and code involved in scaling FluxCD with a multi-tenancy approach in production. It includes configurations and practices that expert developers can use to ensure robust and efficient management of multiple tenants in a Kubernetes environment.

Prerequisites

Ensure the Flux CLI is installed and the staging cluster meets all necessary prerequisites:

flux check --pre

Bootstrapping Flux in Production

Before scaling, bootstrap your production environment. Set the necessary environment variables for your GitHub repository:

export GITHUB_USER=your-github-username
export GITHUB_REPO=your-repository-name

Bootstrap Flux for the production context:

flux bootstrap github \
--context=your-production-context \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--branch=main \
--personal \
--path=clusters/production

You will be prompted for your GITHUB_TOKEN.

Directory Structure

The typical directory structure for the Flux configuration is as follows:

├── clusters
│   ├── production
│   └── staging
├── infrastructure
│   ├── kyverno
│   └── kyverno-policies
└── tenants
    ├── base
    ├── production
    └── staging

Tenant Management

Create the kustomization for a development team:

flux create kustomization dev-team \
--namespace=apps \
--service-account=dev-team \
--source=GitRepository/dev-team \
--path="./" \
--export >> ./tenants/base/dev-team/sync.yaml

Next, create the kustomization.yaml for this tenant:

cd ./tenants/base/dev-team/ && kustomize create --autodetect --namespace apps

Create the staging overlay:

cat << EOF | tee ./tenants/staging/dev-team-patch.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dev-team
  namespace: apps
spec:
  path: ./staging
EOF

Reconciliation Setup

In the clusters/production directory, set up the reconciliation for namespace policies:

Infrastructure Configuration

Configure the infrastructure.yaml for the production cluster:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: kyverno
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  serviceAccountName: kustomize-controller
  path: ./infrastructure/kyverno
  prune: true
  wait: true
  timeout: 5m

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: kyverno-policies
  namespace: flux-system
spec:
  dependsOn:
    - name: kyverno
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  serviceAccountName: kustomize-controller
  path: ./infrastructure/kyverno-policies
  prune: true

Tenant Configuration

Define the tenants.yaml configuration for production:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenants
  namespace: flux-system
spec:
  dependsOn:
    - name: kyverno-policies
  interval: 5m
  serviceAccountName: kustomize-controller
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./tenants/production
  prune: true

Deploying Workloads

Deploy the necessary workloads for each tenant by defining a Kustomization and associating it with the specific tenant repository. For example, to deploy a dev-team application:

Production Patch

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dev-team
  namespace: apps
spec:
  path: ./production

Key Considerations

1. Ensure that service accounts and role bindings are set up correctly for each tenant.
2. Regularly monitor the reconciliation intervals to maintain cluster health and responsiveness.
3. Regular reviews of policies and infrastructure setups are essential to adapt to changing production requirements.

By following the outlined steps and utilizing the provided code snippets, a smooth scaling process for FluxCD multi-tenancy can be achieved in a production environment.

Sources:

• README.md (1.2121469180224305)
• README.md (1.252702461980479)
• clusters/production/infrastructure.yaml (1.1880307266423242)
• clusters/production/tenants.yaml (1.1956402696989412)