Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite

Does writing Kubernetes Manifests count as writing code? Should we still bother to test it? Sure! And with the Kubernetes Test Tool (kuttl) there's great tooling available. Let's explore how to use it with Crossplane.

Crossplane – blog series

1. Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3
2. Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite
3. Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert
4. Create, build & publish Crossplane Configuration Packages with GitHub Actions & Container Registry

Should you test Crossplane Compositions?

If you're new to Crossplane, you might want to learn about all the fancy concepts such as Composite Resource Definitions (XRDs), Compositions and how to use them. And that's great! Crossplane provides a fantastic tooling to drive your Platform Engineering initiative. I wrote about how to get started here. Digging deeper, you learn about more advanced topics such as how to write your own Compositions and how to package them as Configurations. But isn't there something missing?

Lately I gave a Crossplane training at a customer and was surprised to hear that testing infrastructure code is often limited to linting. But regardless of the tooling: If we do Infrastructure-as-Code, we shouldn't forget the principles of modern software engineering, right? And one of these is:

code that is not automatically and constantly executed and tested will eventually rot sooner or later!

This also applies to Kubernetes manifests. Especially since the world around our manifests doesn't stand still. For example, there are constantly new Crossplane releases or Crossplane Provider releases coming. How do you make sure your Crossplane Composition is still compatible with these and provisions the same infrastructure as with the previous releases?

I wrote about the topic of testing infrastructure code with Molecule back in 2018. The meme I used in my post back then just applies today:

Meme source: imgflip.com

So the question is: how can we test our Crossplane Compositions (and Composite Resource Definitions)?

Why Kubernetes Test Tool (kuttl) for Crossplane testing?

You might ask why this post isn't based on uptest, which was donated to the CNCF a while ago. That's because uptest seems to be a work in progress. See this issue comment from an Upbound employee (the company behind Crossplane):

"I think we have to be honest and document somewhere that currently uptest is not really usable without surrounding make targets and the build module :)"

I tried to set up uptest in a Crossplane Configuration repository and was stuck in the middle of nowhere. Documentation is nearly non-existent and I finally resigned – especially when I stumbled upon the mentioned GitHub issue.

But luckily uptest itself is based on the Kubernetes Test Tool (kuttl) and generates a kuttl test case under the hood. You're even encouraged to inspect the generated kuttl tests in the troubleshooting section of the uptest documentation. So why shouldn't we use native kuttl for our Crossplane tests directly?!

Getting started with kuttl

So what is the Kubernetes Test Tool? The kuttl docs have a great explanation for us:

"KUTTL is a declarative integration testing harness for testing operators, KUDO, Helm charts, and any other Kubernetes applications or controllers. Test cases are written as plain Kubernetes resources and can be run against a mocked control plane, locally in kind, or any other Kubernetes cluster."

In other words: kuttl is a test harness for any Kubernetes application. That also makes it a great fit for Crossplane!

Logo sources: Crossplane logo, kuttl logo

So before talking too long, let's dive into the first steps with kuttl. As kuttl uses the Kubernetes in Docker (kind) tooling to run local Kubernetes clusters, we need to have a local Docker installation in place. Also, be sure to have the following command line tools installed: kubectl, helm & kind. As we will be using AWS as our infrastructure resource provider later, there should also be a working installation of the awscli. You can testdrive your awscli configuration with the command aws configure.

Installing kuttl kubectl plugin

Having Docker and the mentioned CLI tools in place, we need to install the kuttl kubectl plugin. On a Mac (or Linux with brew installed) you can simply use the homebrew package:

1brew tap kudobuilder/tap
2brew install kuttl-cli

An alternative way is to leverage the kubectl plugins package manager krew to install the kuttl kubectl plugin. Therefore have a look into the krew installation docs. Installing krew might be a bit more work compared to the homebrew variant, but it also works for other kubectl plugins. If you have krew in place, install kuttl via the command kubectl krew install kuttl:

1$ kubectl krew install kuttl
2
3WARNING: To be able to run kubectl plugins, you need to add
4the following to your ~/.zshrc:
5
6    export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
7
8and restart your shell.

Add the export PATH... statement to your shell configuration as mentioned in the installation log output.

Now the kubectl kuttl plugin should work as expected:

1$ kubectl kuttl --version
2kubectl-kuttl version 0.15.0

How to use kuttl with Crossplane

kuttl defines the following building blocks:

• A "test suite" (aka TestSuite) is comprised of many test cases that are run in parallel.
• A "test case" is a collection of test steps that are run serially – if any test step fails then the entire test case is considered failed.
• A "test step" (aka TestStep) defines a set of Kubernetes manifests to apply and a state to assert on (wait for or expect).

In order to use Crossplane with kuttl, we need to install Crossplane and a Crossplane Provider (like AWS) in the kuttl TestSuite. We also need to configure the Crossplane Provider with a ProviderConfig and its matching credentials via a Secret. It depends on the kind of testing we want to do if the Secret needs to hold only fake (Rendering-only tests) or real (Integration Tests) credentials:

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo, AWS logo, Azure logo, Google Cloud logo

With that configured in our kuttl TestSuite, we can create a kuttl test case, containing some kuttl TestSteps. These will be shown in the next part of this blog post series.

Now before we can start writing our first kuttl tests, we need to have an example Crossplane Composition in place. As I needed it for a meetup talk and an magazine article, I created a simple Composition to provision a public accessible S3 Bucket. The example code used throughout this blog post is fully available on GitHub.

An example Composition

The example project provides a Crossplane Configuration that will make use of the Upbound Provider for AWS S3 and create a public accessible S3 Bucket in AWS. The Composite Resource Definition defines a Composite Resource Claim called ObjectStorage with the two parameters bucketName and region. It can be found in the file apis/objectstorage/definition.yaml in the example repository:

1apiVersion: apiextensions.crossplane.io/v1
2kind: CompositeResourceDefinition
3metadata:
4  name: xobjectstorages.crossplane.jonashackt.io
5spec:
6  group: crossplane.jonashackt.io
7  names:
8    kind: XObjectStorage
9    plural: xobjectstorages
10  claimNames:
11    kind: ObjectStorage
12    plural: objectstorages
13
14  versions:
15  - name: v1alpha1
16    served: true
17    referenceable: true
18    schema:
19      openAPIV3Schema:
20        type: object
21        properties:
22          spec:
23            type: object
24            properties:
25              parameters:
26                type: object
27                properties:
28                  bucketName:
29                    type: string
30                  region:
31                    type: string
32                required:
33                  - bucketName
34                  - region

The Composition is defined in the file apis/objectstorage/composition.yaml. Since it became way more complex to setup a S3 Bucket in AWS that is publicly accessible somewhere in April 2023 , the Composition needs to use a set of Managed Resources. According to this issue and these Terraform docs we need to define the Bucket creation along with a definition of a BucketPublicAccessBlock, BucketOwnershipControls, the BucketACL and a BucketWebsiteConfiguration.

1apiVersion: apiextensions.crossplane.io/v1
2kind: Composition
3metadata:
4  name: objectstorage-composition
5  labels:
6    crossplane.io/xrd: xobjectstorages.crossplane.jonashackt.io
7    provider: aws
8spec:
9  compositeTypeRef:
10    apiVersion: crossplane.jonashackt.io/v1alpha1
11    kind: XObjectStorage
12  
13  writeConnectionSecretsToNamespace: crossplane-system
14  
15  resources:
16    - name: bucket
17      base:
18        apiVersion: s3.aws.upbound.io/v1beta1
19        kind: Bucket
20        metadata: {}
21        spec:
22          deletionPolicy: Delete
23      
24      patches:
25        - fromFieldPath: "spec.parameters.bucketName"
26          toFieldPath: "metadata.name"
27        - fromFieldPath: "spec.parameters.region"
28          toFieldPath: "spec.forProvider.region"
29
30    - name: bucketpublicaccessblock
31      base:
32        apiVersion: s3.aws.upbound.io/v1beta1
33        kind: BucketPublicAccessBlock
34        spec:
35          forProvider:
36            blockPublicAcls: false
37            blockPublicPolicy: false
38            ignorePublicAcls: false
39            restrictPublicBuckets: false
40
41      patches:
42        - fromFieldPath: "spec.parameters.bucketName"
43          toFieldPath: "metadata.name"
44          transforms:
45            - type: string
46              string:
47                fmt: "%s-pab"
48        - type: PatchSet
49          patchSetName: bucketNameAndRegionPatchSet
50
51    - name: bucketownershipcontrols
52      base:
53        apiVersion: s3.aws.upbound.io/v1beta1
54        kind: BucketOwnershipControls
55        spec:
56          forProvider:
57            rule:
58              - objectOwnership: ObjectWriter
59
60      patches:
61        - fromFieldPath: "spec.parameters.bucketName"
62          toFieldPath: "metadata.name"
63          transforms:
64            - type: string
65              string:
66                fmt: "%s-osc"
67        - type: PatchSet
68          patchSetName: bucketNameAndRegionPatchSet
69
70    - name: bucketacl
71      base:
72        apiVersion: s3.aws.upbound.io/v1beta1
73        kind: BucketACL
74        spec:
75          forProvider:
76            acl: "public-read"
77
78      patches:
79        - fromFieldPath: "spec.parameters.bucketName"
80          toFieldPath: "metadata.name"
81          transforms:
82            - type: string
83              string:
84                fmt: "%s-acl"
85        - type: PatchSet
86          patchSetName: bucketNameAndRegionPatchSet
87  
88    - name: bucketwebsiteconfiguration
89      base:
90        apiVersion: s3.aws.upbound.io/v1beta1
91        kind: BucketWebsiteConfiguration
92        spec:
93          forProvider:
94            indexDocument:
95              - suffix: index.html
96
97      patches:
98        - fromFieldPath: "spec.parameters.bucketName"
99          toFieldPath: "metadata.name"
100          transforms:
101            - type: string
102              string:
103                fmt: "%s-websiteconf"
104        - type: PatchSet
105          patchSetName: bucketNameAndRegionPatchSet
106
107  patchSets:
108  - name: bucketNameAndRegionPatchSet
109    patches:
110    - fromFieldPath: "spec.parameters.bucketName"
111      toFieldPath: "spec.forProvider.bucketRef.name"
112    - fromFieldPath: "spec.parameters.region"
113      toFieldPath: "spec.forProvider.region"

If you have your own Composition in place, you can start from there. Or you just take the example Composition provided here. Anyway, we can now start writing our first Crossplane powered kuttl test suite.

Creating a kuttl TestSuite: The kuttl-test.yaml

For this, we need to create a kuttl-test.yaml defining our TestSuite in the root of our project :

1apiVersion: kuttl.dev/v1beta1
2kind: TestSuite
3testDirs:
4  - tests/e2e/
5startKIND: true
6kindContext: crossplane-test

This file is the starting point of the kuttl configuration. Right now it mainly defines two things: The directory where our tests will be stored and if kuttl should fire up a kind cluster for us. Since we defined our testDirs to be tests/e2e/, we now also need to create a new directory tests containing another directory e2e in the root of our project:

1mkdir -p tests/e2e

We should also add the following lines to our .gitignore to prevent us from checking in temporary kind logs or kubeconfig files to Git:

1kubeconfig
2kind-logs-*

Installing Crossplane in kuttl TestSuite

To be able to write tests for Crossplane, we need to have it installed in our cluster first. Luckily kuttl has a commands keyword ready for us in the TestSuite and TestStep objects. Starting the command with a binary, we can execute anything we'd like.

Since we need Crossplane installed and ready for all our tests, we will install it in the TestSuite instead of every TestStep. Therefore, inside our kuttl-test.yaml we add command statements to install Crossplane into the kind test cluster:

1apiVersion: kuttl.dev/v1beta1
2kind: TestSuite
3commands:
4  # Install crossplane via Helm Renovate enabled (see https://stackoverflow.com/a/71765472/4964553)
5  - command: helm dependency update crossplane/install
6  - command: helm upgrade --install crossplane --namespace crossplane-system crossplane/install --create-namespace --wait
7testDirs:
8  - tests/e2e/
9startKIND: true
10kindContext: crossplane-test

The installation of Crossplane works "Renovate" enabled via a local Helm Chart. You might recall the intro to this post, where we stated that our Compositions should be tested automatically when new Crossplane versions arrive. Here's how to make sure the update will be triggered by Renovate automatically. For this purpose we define a Chart.yaml in a new directory crossplane/install:

1apiVersion: v2
2type: application
3name: crossplane
4version: 0.0.0 # unused
5appVersion: 0.0.0 # unused
6dependencies:
7  - name: crossplane
8    repository: https://charts.crossplane.io/stable
9    version: 1.15.1

Additionally, we add the following to our .gitignore to prevent us from checking in generated Helm files:

1# Exclude Helm charts lock and packages
2**/**/charts
3**/**/Chart.lock

That's it. If we configure Renovate in the repository later, our Composition will be tested for new Crossplane versions automatically :)

Already at this point, we can use the kubectl kuttl test command to verify if our Crossplane installation works as expected:

1kubectl kuttl test --skip-cluster-delete

The --skip-cluster-delete flag comes in handy, since it will preserve our crossplane-test kind cluster for later runs (without the flag it would be deleted everytime). You may double-check if the kind cluster still runs after the kubectl kuttl test command. A docker ps should show the cluster also:

1docker ps
2CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS              PORTS                       NAMES
3782fa5bb39a9   kindest/node:v1.25.3   "/usr/local/bin/entr…"   2 minutes ago   Up About a minute   127.0.0.1:34407->6443/tcp   crossplane-test-control-plane

You can even connect to the kind cluster directly setting the KUBECONFIG env variable like this (simply replace your profile and repository path):

1export KUBECONFIG="/home/yourProfileNameHere/yourRepositoryPathHere/kubeconfig"

With this KUBECONFIG you can access the kuttle created kind cluster in your current shell:

1$ kubectl get nodes
2NAME                            STATUS   ROLES           AGE     VERSION
3crossplane-test-control-plane   Ready    control-plane   4m25s   v1.25.3

To get your normal kubectx config working again, simply run unset KUBECONFIG.

Since kuttl doesn't remove its kind cluster when we use the --skip-cluster-delete flag, we also need to know how to delete the kind cluster ourselves:

1kind delete clusters crossplane-test

Installing AWS Provider in kuttl TestSuite

Now that we successfully installed Crossplane into our kuttl kind cluster, we also need to install a Provider. As our Composition is based on AWS and provisions a publicly accessible S3 Bucket, we need to use the Provider upbound/provider-aws-s3. To install it we should first create a new directory provider inside our already existant crossplane folder:

1mkdir -p crossplane/provider

Inside of crossplane/provider we create our Provider specification in the file upbound-provider-aws-s3.yaml:

1apiVersion: pkg.crossplane.io/v1
2kind: Provider
3metadata:
4  name: upbound-provider-aws-s3
5spec:
6  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.2.1
7  packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache.
8  revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate & healthy
9  revisionHistoryLimit: 1

Having the Provider spec in place, we can integrate it into our kuttl TestSuite. Thus inside our kuttl-test.yaml we add additional command statements to install the Provider into the kind test cluster and wait for it to become healthy:

1apiVersion: kuttl.dev/v1beta1
2kind: TestSuite
3commands:
4  # Install crossplane via Helm Renovate enabled (see https://stackoverflow.com/a/71765472/4964553)
5  - command: helm dependency update crossplane-install
6  - command: helm upgrade --install crossplane --namespace crossplane-system crossplane-install --create-namespace --wait
7
8  # Install the crossplane Upbound AWS S3 Provider Family
9  - command: kubectl apply -f crossplane/provider/upbound-provider-aws-s3.yaml
10  # Wait until AWS Provider is up and running
11  - command: kubectl wait --for=condition=healthy --timeout=180s provider/upbound-provider-aws-s3
12testDirs:
13  - tests/e2e/
14startKIND: true
15kindContext: crossplane-test

As you can see we also wait until the Provider reached a healthy state, before we proceed. That's crucial for the next steps to work.

Configuring AWS Provider in kuttl for testing Resource rendering (without AWS access)

Often we do not need to really create resources on AWS throughout our tests. It might be enough to just verify if the Managed Resources are rendered correctly. Just as we intended while writing our Composition. Or we can even start with a test-driven approach and start with our kuttl test right away.

To get the Crossplane AWS Provider to render the Managed Resources without actual AWS connectivity, we use the trick described here and create a Secret without actual AWS creds. Let's therefore create a file non-access-secret.yaml in the crossplane/provider/ directory:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: aws-creds
5  namespace: crossplane-system
6type: Opaque
7stringData:
8  key: nocreds

Now inside our kuttl-test.yaml we add additional command statements to create the Secret and configure the AWS Provider without actual AWS access:

1apiVersion: kuttl.dev/v1beta1
2kind: TestSuite
3commands:
4  # Install crossplane via Helm Renovate enabled (see https://stackoverflow.com/a/71765472/4964553)
5  - command: helm dependency update crossplane/install
6  - command: helm upgrade --install --force crossplane --namespace crossplane-system crossplane/install --create-namespace --wait
7
8  # Install the crossplane Upbound AWS S3 Provider Family
9  - command: kubectl apply -f crossplane/provider/upbound-provider-aws-s3.yaml
10  # Wait until AWS Provider is up and running
11  - command: kubectl wait --for=condition=healthy --timeout=180s provider/upbound-provider-aws-s3
12
13  # Create AWS Provider secret without AWS access
14  - command: kubectl apply -f crossplane/provider/non-access-secret.yaml
15  # Create ProviderConfig to consume the Secret containing AWS credentials
16  - command: kubectl apply -f crossplane/provider/provider-config-aws.yaml
17testDirs:
18  - tests/e2e/
19startKIND: true
20kindContext: crossplane-test

As you can see, we added another line. In it, we configure the AWS Provider via a ProviderConfig. For this to work, we need to create a file provider-config-aws.yaml inside the crossplane/provider directory:

1apiVersion: aws.upbound.io/v1beta1
2kind: ProviderConfig
3metadata:
4  name: default
5spec:
6  credentials:
7    source: Secret
8    secretRef:
9      namespace: crossplane-system
10      name: aws-creds
11      key: creds

Now we should be able to successfully run kubectl kuttl test --skip-cluster-delete:

1$ kubectl kuttl test
2=== RUN   kuttl
3    harness.go:462: starting setup
4    harness.go:249: running tests with KIND.
5    harness.go:173: temp folder created /tmp/kuttl1667306899
6    harness.go:155: Starting KIND cluster
7    kind.go:66: Adding Containers to KIND...
8    harness.go:275: Successful connection to cluster at: https://127.0.0.1:43619
9    logger.go:42: 10:54:17 |  | running command: [helm dependency update crossplane-install]
10    logger.go:42: 10:54:17 |  | WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
11    logger.go:42: 10:54:17 |  | WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
12    logger.go:42: 10:54:17 |  | Getting updates for unmanaged Helm repositories...
13    logger.go:42: 10:54:18 |  | ...Successfully got an update from the "https://charts.crossplane.io/stable" chart repository
14    logger.go:42: 10:54:18 |  | Saving 1 charts
15    logger.go:42: 10:54:18 |  | Downloading crossplane from repo https://charts.crossplane.io/stable
16    logger.go:42: 10:54:18 |  | Deleting outdated charts
17    logger.go:42: 10:54:18 |  | running command: [helm upgrade --install crossplane --namespace crossplane-system crossplane-install --create-namespace --wait]
18    logger.go:42: 10:54:18 |  | WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
19    logger.go:42: 10:54:18 |  | WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
20    logger.go:42: 10:54:18 |  | Release "crossplane" does not exist. Installing it now.
21    logger.go:42: 10:54:41 |  | NAME: crossplane
22    logger.go:42: 10:54:41 |  | LAST DEPLOYED: Tue Apr  9 10:54:18 2024
23    logger.go:42: 10:54:41 |  | NAMESPACE: crossplane-system
24    logger.go:42: 10:54:41 |  | STATUS: deployed
25    logger.go:42: 10:54:41 |  | REVISION: 1
26    logger.go:42: 10:54:41 |  | TEST SUITE: None
27    logger.go:42: 10:54:41 |  | running command: [kubectl apply -f crossplane/provider/upbound-provider-aws-s3.yaml]
28    logger.go:42: 10:54:41 |  | provider.pkg.crossplane.io/upbound-provider-aws-s3 created
29    logger.go:42: 10:54:41 |  | running command: [kubectl wait --for=condition=healthy --timeout=180s provider/upbound-provider-aws-s3]
30    logger.go:42: 10:55:50 |  | provider.pkg.crossplane.io/upbound-provider-aws-s3 condition met
31    logger.go:42: 10:55:50 |  | running command: [kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf]
32    logger.go:42: 10:55:50 |  | secret/aws-creds created
33    logger.go:42: 10:55:50 |  | running command: [kubectl apply -f crossplane/provider/provider-config-aws.yaml]
34    logger.go:42: 10:55:50 |  | providerconfig.aws.upbound.io/default created
35    ...

If your output looks similar to this, everything should be prepared for testing resource rendering with Crossplane!

Next up: kuttl TestSteps with Crossplane

In this post, we learned about kuttl and that it can be a great match for Crossplane. We fully set up kuttl to have Crossplane installed and a Crossplane Provider configured. With this in place, we are now able to render our Composition with AWS-based Managed Resources. There's also a full example Composition ready that we will be able to test with kuttl.

So the next part of this blog series will go into the details of how to create test steps with kuttl. The next post explores how to structure the kuttl test steps into a scheme lend from the Behavior-driven Development (BDD). In the end, everybody in the project should be able to read the tests structure and log output. The next post also outlines how we can create Integration Tests with kuttl and provision real infrastructure in them using Crossplane. Finally everything will be packed into a GitHub Actions pipeline.