Manage kubernetes manifest with Kustomize

Manage Kubernetes manifest using Kustomize

In this blog have demonstrated, how we can use Kustomize to manage Kubernetes manifest with simple example and also how to render Kubernetes manifest using Kustomize CLI.

Pre-requisites:

Basic understanding of Kubernetes and how resources are deployed using yaml
- Kustomize CLI installed in the machine. To install refer the documentation for more details

What is Kustomize?

Kustomize provides a solution for customizing Kubernetes resource configuration free from templates and DSLs. Refer kustomize.io
- Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.
kubectl client already supports kustomize
Kustomize configuration can applied directly to Kubernetes cluster using kubectl apply -k <directory\to\kustomization.yaml>

High level overview

In enterprise project to deploy the application into Kubernetes cluster the resources will be defined in many of YAML manifest file. If there are lots of manifest file and in case we need to apply different configuration for each environment like dev, test or production we can use helm in order to render and deploy the manifest we need helm command. Kustomize can helps in managing the environment specific configuration in a manageable way.
With Kustomize we not need to template the manifest, the environment specific configuration can be applied to cluster directly using the kubectl command. The manifest can also be rendered using kustomize cli.

With the kustomize structure the base or raw manifest will never change.

How kustomize works?

Say, we have a manifest for our application in a directory named base, kustomize uses kustomization.yaml file that configures the path these manifest under the same directory using resources tag

Info:- The resources tag is used to specify the path of the application manifest yaml this tag is identified by kustomize framework

In order to configure the environment specific configuration overlay folder is created, which contains folders specific to environment. With reference to the image below, we use dev and test folders refers the environment.

Info:- Per Kustomize documentation example have used the folder name as overlay, this folder name can be different.

overlay\dev folder contains kustomization.yaml file which defines the path to the base folder (in resources tag), and dev environment specific changes. For dev the configuration adds "-dev" to the name using nameSuffix and also updates the deployment replicas which is defined inline in the kustomization.yaml
overlay\test folder contains kustomization.yaml file which defines the path to the base folder (in resources tag), and path of the patch file using patches tag. The deployment_patch.yaml use patch transform to update the replicas of the deployment. There are other options to update the configuration of resources, using inline patch configuration where the deployment image tag and the secret properties are updated.
In order to render the environment specific manifest we can use the kustomize CLI command kustomzie build /environment/directory/of/kustomization.yaml or use kubectl apply -k /directory/of/kustomization.yaml.In both case the command takes in the environment specific folder where the kustomization.yaml exists.

The framework will use the kustomize.yaml configuration to identify the base manifest.

Kustomize project folder structure

Dev environment specific representation

Test environment specific representation

With reference to the dev, the base folder is same for the test as well.

Code sample

Below are the YAML file used for this demonstration
base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

base/secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: demo-app-secret
type: Opaque
data:
  APP_NAME: demo_app
  ENV_CODE: DEV   # For test we need to update this to TEST

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
   - deployment.yaml
   - secret.yaml

Dev environment specific configuration

overlay/dev/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
 - ../../base/

nameSuffix: -dev

# example of updating deployment replica from kustomization
# Note:- Kustomize supports patch update where the same 
# changes can be created as a patch, will see this example in 
# overlay/test directory for demo
# when using the replica in this kustomization.yaml, any change 
# to deployment replica will not be applied, noticed this update
# takes precedence
replicas:
- name: demo-app
  count: 1

Render dev environment manifest

Command to generate dev specific manifest

# navigate to specific directory where the manifest is stored
# in this case kustomization foder contains the base and overlay folder
# refer the image above where the project structure is deplicted (fig 1)

> kustomize build overlay/dev

INFO:- The manifest will be displayed in the stdout, we can redirect it to specific directory using the -o switch, usage kustomize build overlay/dev -o output/

Output of rendered dev manifest

apiVersion: v1
data:
  APP_NAME: demo_app
  ENV_CODE: DEV
kind: Secret
metadata:
  name: demo-app-secret-dev
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: demo-app-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:latest
        name: nginx
        ports:
        - containerPort: 80

Test environment specific configuration

overlay/test/deployment_patch.yaml

- op : replace
  path: /spec/replicas
  value: 3

INFO:- There are couple ways to specify the patch files
The target can be specified directly in the deployment_patch.yaml file like
target: 
group: apps
version: v1
kind: Deployment
name: demo-app
This patch file can directly define the deployment manifest yaml, with the test specific configuration. The kustomization.yaml file under test, will refer to the pathc using patches tag.

kustomize framework will automatically infer the type of patch to be used when using patches tag, like patchesStrategicMerge or patchTransform

overlay/test/kustomization.yaml
- Below yaml file defines different options to apply the patch, using external file and inline in the configuration yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
 - ../../base/

nameSuffix: -test

patches:
  - path: deployment_patch.yaml
    target: 
      group: apps
      version: v1
      kind: Deployment
      name: demo-app
  - patch: |-
     - op: replace
       path: /spec/template/spec/containers/0/image
       value: nginx:1.21.0
    target:
      kind: Deployment
      name: demo-app
  - patch: |-
      apiVersion: v1
      kind: Secret
      metadata:
        name: demo-app-secret
        type: Opaque
      data:
        APP_NAME: demo_app
        ENV_CODE: TEST ## UPDATE TO TEST IN THIS PATCH

Render test environment manifest

To render test environment specific manifest, the command is kustomzie build overlay/test

Output of rendered test environment manifest

apiVersion: v1
data:
  APP_NAME: demo_app
  ENV_CODE: TEST
kind: Secret
metadata:
  name: demo-app-secret-test
  type: Opaque
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: demo-app-test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.21.0
        name: nginx
        ports:
        - containerPort: 80

Info:- The rendered YAML file can be directly deployed to the cluster using kubectl command. We can also directly use CD (continuous delivery) tool like Argo CD to use the kustomize configuration to deploy the resources.

Additional Info

Kustomize do supports different options to customize the configuration like

replacements
ConfigMap generator
SecretGenerator
built-in plugin
etc...

Refer the kustomize documentation for details.

Manage Kubernetes Manifest with Kustomize

Table of contents

Manage Kubernetes manifest using Kustomize

What is Kustomize?

High level overview

How kustomize works?

Kustomize project folder structure

Dev environment specific representation

Test environment specific representation

Code sample

Dev environment specific configuration

Render dev environment manifest

Output of rendered dev manifest

Test environment specific configuration

Render test environment manifest

Output of rendered test environment manifest

Additional Info