Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert

In the first part of this blog series we learned about kuttl and why it's a great idea to write tests for your Crossplane Compositions. Now it's time to set up the kuttl test steps to finally verify our Composition renders correctly.

Crossplane – blog series

1. Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3
2. Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite
3. Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert
4. Create, build & publish Crossplane Configuration Packages with GitHub Actions & Container Registry

BDD-style Crossplane testing

We already configured a kuttl TestSuite that has a Crossplane installation and Provider configuration ready to use. Also, there was an example Crossplane Composition provided that waits to be tested. Based on that, we can now create a kuttl test case, featuring the kuttl TestSteps. Just as we're used to from writing Unit Tests in any other language, we can leverage the naming scheme lend from the Behavior-driven Development (BDD): Given, When, Assert.

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo

I know, in BDD the third step is commonly named "Then". But as this step is always named "Assert" per default in kuttl, we need to arrange ourselves with this slight change :)

Creating a kuttl test case

Before we can craft test steps in kuttl, we need to create a test case. Creating a kuttl test case is fairly simple since it's only defined by the next directory level inside of tests/e2e. So let's create a test case for our objectstorage composition:

1mkdir -p tests/e2e/objectstorage

Creating a kuttl test step: the "Given" installing XRD & Composition

Now we're where we wanted to be in the first place: writing our first Crossplane-enabled kuttl TestStep. As already stated above, we borrowed the structure of our tests from the BDD-style Given, When, Then syntax. We start with the Given test step, where we install our Composite Resource Definition (XRD) followed by our Composition under test.

To be able to create our Given test step, we need to have a look into how kuttl handles files inside a test case directory:

"In a test case's directory, each file that begins with the same index is considered a part of the same test step. All objects inside of a test step are operated on by the test harness simultaneously, so use separate test steps to order operations."

As kuttl executes every 00-* prefixed test step found in the folder before proceeding to the 01-* one, we can have the 00-given-install-xrd-composition working as our preparation step for the other steps to come. Therefore let's create a new file called 00-given-install-xrd-composition.yaml inside the tests/e2e/objectstorage directory:

1apiVersion: kuttl.dev/v1beta1
2kind: TestStep
3commands:
4  # Keep in mind that the apis dir is way up the folder hierachy relative to this TestStep!
5  # Install the XRD
6  - command: kubectl apply -f ../../../apis/objectstorage/definition.yaml
7  # Install the Composition
8  - command: kubectl apply -f ../../../apis/objectstorage/composition.yaml
9  # Wait for XRD to become "established"
10  - command: kubectl wait --for condition=established --timeout=20s xrd/xobjectstorages.crossplane.jonashackt.io

As you can see, we install the XRD and Composition first. After that it's a good idea to wait for the XRD to become established before proceeding to the next step.

If the kuttl logs show errors like the path "apis/objectstorage/definition.yaml" does not exist, check the paths in your command statements:

1=== RUN   kuttl/harness
2=== RUN   kuttl/harness/objectstorage
3=== PAUSE kuttl/harness/objectstorage
4=== CONT  kuttl/harness/objectstorage
5    logger.go:42: 11:24:22 | objectstorage | Creating namespace: kuttl-test-hopeful-mustang
6    logger.go:42: 11:24:22 | objectstorage/0-given-install-xrd-composition | starting test step 0-given-install-xrd-composition
7    logger.go:42: 11:24:22 | objectstorage/0-given-install-xrd-composition | running command: [kubectl apply -f apis/objectstorage/definition.yaml]
8    logger.go:42: 11:24:22 | objectstorage/0-given-install-xrd-composition | error: the path "apis/objectstorage/definition.yaml" does not exist
9    logger.go:42: 11:24:22 | objectstorage/0-given-install-xrd-composition | command failure, skipping 1 additional commands
10   ...
11--- FAIL: kuttl (127.38s)
12    --- FAIL: kuttl/harness (0.00s)
13        --- FAIL: kuttl/harness/objectstorage (5.41s)
14FAIL

First I also missed this, since in the TestSuite at kuttl-test.yaml everything worked relatively well from the root dir. BUT remember, we're inside tests/e2e/objectstorage now! So we need to go up 3 dirs like ../../../apis/objectstorage/definition.yaml to fetch the correct file.

Creating another kuttl test step: the "When" applying XR or Claim

Now that we have installed our XRD and Composition in the Given step, it's time to apply our XR or Claim (XRC) in a When step.

Therefore let's created a file 01-when-applying-claim.yaml inside the tests/e2e/objectstorage directory:

1apiVersion: kuttl.dev/v1beta1
2kind: TestStep
3commands:
4  # Create the XR/Claim
5  - command: kubectl apply -f ../../../examples/objectstorage/claim.yaml

In this test step's configuration we apply a Composite Resource Claim that should reside in the examples dir. If we don't already have it in place, we should now also create an examples directory in the root of our project.

The examples directory is the Crossplane default to place XRs or Claims in a configuration repository.

Inside the newly created examples folder we also create a objectstorage directory to reflect the folder structure of our Composition and XRD.

Insideexamples/objectstorage we finally create an example Claim in the file claim.yaml. It could look like this, for example:

1apiVersion: crossplane.jonashackt.io/v1alpha1
2kind: ObjectStorage
3metadata:
4  namespace: default
5  name: managed-upbound-s3
6spec:
7  compositionRef:
8    name: objectstorage-composition
9  
10  parameters:
11    bucketName: kuttl-test-bucket
12    region: eu-central-1

The third kuttl test step: Validate / Assert Resource rendering (without AWS access)

We finally hit our third kuttl test step: The "Then" or Assert step, where we verify our Crossplane resource is rendered correctly. With kuttl we need to adhere to a specific name scheme here:

It's crucial to use 01-assert as the name here, to get the assertion started after our Claim has been applied!

As kuttl always searches for assert in the file name of our validation test step, we sadly can't use the BDD term "Then" here directly. But that shouldn't prevent us from writing our first kuttl assertion! Let's create a file 01-assert.yaml inside the tests/e2e/objectstorage directory:

1apiVersion: s3.aws.upbound.io/v1beta1
2kind: Bucket
3metadata:
4  name: kuttl-test-bucket
5spec:
6  forProvider:
7    region: eu-central-1
8---
9apiVersion: s3.aws.upbound.io/v1beta1
10kind: BucketACL
11metadata:
12  name: kuttl-test-bucket-acl
13spec:
14  forProvider:
15    acl: public-read
16    bucketRef:
17      name: kuttl-test-bucket
18    region: eu-central-1

This test step will be considered completed once our Managed Resources rendered are matching the state that we have defined.

If the state is not reached by the time the assert's timeout has expired, then the test step and case will be considered failed by kuttl.

Be sure to define the exact metadata like in your Claim! Otherwise kuttl won't find it and will show an error like the following:

1logger.go:42: 15:54:03 | objectstorage/1-when-applying-claim | test step failed 1-when-applying-claim
2    ...
3    logger.go:42: 15:54:03 | objectstorage/1-when-applying-claim | objectstorage.crossplane.jonashackt.io "managed-upbound-s3" deleted
4    case.go:364: failed in step 1-when-applying-claim
5    case.go:366: no resources matched of kind: crossplane.jonashackt.io/v1alpha1, Kind=ObjectStorage

Now we're finally able to run our test suite with our already known kubectl kuttl test command:

1$ kubectl kuttl test --start-kind=false
2=== RUN   kuttl
3    harness.go:462: starting setup
4    harness.go:252: running tests using configured kubeconfig.
5    harness.go:275: Successful connection to cluster at: https://127.0.0.1:42415
6    logger.go:42: 12:58:54 |  | running command: [helm dependency update crossplane/install]
7    logger.go:42: 12:58:54 |  | WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
8    logger.go:42: 12:58:54 |  | WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
9    logger.go:42: 12:58:54 |  | Hang tight while we grab the latest from your chart repositories...
10    logger.go:42: 12:58:55 |  | ...Successfully got an update from the "crossplane-stable" chart repository
11    logger.go:42: 12:58:55 |  | Update Complete. ⎈Happy Helming!⎈
12    logger.go:42: 12:58:55 |  | Saving 1 charts
13    logger.go:42: 12:58:55 |  | Downloading crossplane from repo https://charts.crossplane.io/stable
14    logger.go:42: 12:58:55 |  | Deleting outdated charts
15    logger.go:42: 12:58:55 |  | running command: [helm upgrade --install --force crossplane --namespace crossplane-system crossplane/install --create-namespace --wait]
16    logger.go:42: 12:58:55 |  | WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
17    logger.go:42: 12:58:55 |  | WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/jonashackt/dev/crossplane-kuttl/kubeconfig
18    logger.go:42: 12:58:57 |  | Release "crossplane" has been upgraded. Happy Helming!
19    logger.go:42: 12:58:57 |  | NAME: crossplane
20    logger.go:42: 12:58:57 |  | LAST DEPLOYED: Wed May 15 12:58:55 2024
21    logger.go:42: 12:58:57 |  | NAMESPACE: crossplane-system
22    logger.go:42: 12:58:57 |  | STATUS: deployed
23    logger.go:42: 12:58:57 |  | REVISION: 2
24    logger.go:42: 12:58:57 |  | TEST SUITE: None
25    logger.go:42: 12:58:57 |  | running command: [kubectl apply -f crossplane/provider/upbound-provider-aws-s3.yaml]
26    logger.go:42: 12:58:58 |  | provider.pkg.crossplane.io/upbound-provider-aws-s3 unchanged
27    logger.go:42: 12:58:58 |  | running command: [kubectl wait --for=condition=healthy --timeout=180s provider/upbound-provider-aws-s3]
28    logger.go:42: 12:58:58 |  | provider.pkg.crossplane.io/upbound-provider-aws-s3 condition met
29    logger.go:42: 12:58:58 |  | running command: [kubectl apply -f crossplane/provider/non-access-secret.yaml]
30    logger.go:42: 12:58:59 |  | secret/aws-creds configured
31    logger.go:42: 12:58:59 |  | running command: [kubectl apply -f crossplane/provider/provider-config-aws.yaml]
32    logger.go:42: 12:58:59 |  | providerconfig.aws.upbound.io/default unchanged
33    harness.go:360: running tests
34    harness.go:73: going to run test suite with timeout of 30 seconds for each step
35    harness.go:372: testsuite: tests/e2e/ has 1 tests
36=== RUN   kuttl/harness
37=== RUN   kuttl/harness/objectstorage
38=== PAUSE kuttl/harness/objectstorage
39=== CONT  kuttl/harness/objectstorage
40    logger.go:42: 12:58:59 | objectstorage | Creating namespace: kuttl-test-subtle-shad
41    logger.go:42: 12:58:59 | objectstorage/0-given-install-xrd-composition | starting test step 0-given-install-xrd-composition
42    logger.go:42: 12:58:59 | objectstorage/0-given-install-xrd-composition | running command: [kubectl apply -f ../../../apis/objectstorage/definition.yaml]
43    logger.go:42: 12:58:59 | objectstorage/0-given-install-xrd-composition | compositeresourcedefinition.apiextensions.crossplane.io/xobjectstorages.crossplane.jonashackt.io unchanged
44    logger.go:42: 12:58:59 | objectstorage/0-given-install-xrd-composition | running command: [kubectl apply -f ../../../apis/objectstorage/composition.yaml]
45    logger.go:42: 12:58:59 | objectstorage/0-given-install-xrd-composition | composition.apiextensions.crossplane.io/objectstorage-composition configured
46    logger.go:42: 12:58:59 | objectstorage/0-given-install-xrd-composition | running command: [kubectl wait --for condition=established --timeout=20s xrd/xobjectstorages.crossplane.jonashackt.io]
47    logger.go:42: 12:59:00 | objectstorage/0-given-install-xrd-composition | compositeresourcedefinition.apiextensions.crossplane.io/xobjectstorages.crossplane.jonashackt.io condition met
48    logger.go:42: 12:59:00 | objectstorage/0-given-install-xrd-composition | test step completed 0-given-install-xrd-composition
49    logger.go:42: 12:59:00 | objectstorage/1-when-applying-claim | starting test step 1-when-applying-claim
50    logger.go:42: 12:59:00 | objectstorage/1-when-applying-claim | running command: [kubectl apply -f ../../../examples/objectstorage/claim.yaml]
51    logger.go:42: 12:59:00 | objectstorage/1-when-applying-claim | objectstorage.crossplane.jonashackt.io/managed-upbound-s3 unchanged
52    logger.go:42: 12:59:01 | objectstorage/1-when-applying-claim | test step completed 1-when-applying-claim
53    logger.go:42: 12:59:01 | objectstorage | objectstorage events from ns kuttl-test-subtle-shad:
54    logger.go:42: 12:59:01 | objectstorage | Deleting namespace: kuttl-test-subtle-shad
55=== CONT  kuttl
56    harness.go:405: run tests finished
57    harness.go:513: cleaning up
58    harness.go:570: removing temp folder: ""
59--- PASS: kuttl (13.52s)
60    --- PASS: kuttl/harness (0.00s)
61        --- PASS: kuttl/harness/objectstorage (7.75s)
62PASS

The --skip-cluster-delete will preserve the kind cluster if our tests failed and thus speed up our development cycle. As already explained, kind and the Crossplane installation/configuration will otherwise take place in every test run. Since kuttl will create a local kubeconfig file, it can reuse the kind cluster automatically in subsequent runs. Therefore, be sure to define export KUBECONFIG="/home/jonashackt/dev/crossplane-kuttl/kubeconfig" once and append --start-kind=false in the following commands (a sole kubectl kuttl test would otherwise give KIND is already running, unable to start errors):

1kubectl kuttl test --start-kind=false

Yay! We just wrote our first complete kuttl TestSuite verifying that our Crossplane Composition renders its Managed Resources exactly as we intended them to! A great advantage of these rendering tests is that they run completely in isolation inside our CI system. And their execution time is relatively fast. Thus just like Unit Tests, they can (and should) be executed very often!

As a bonus with kuttl we're not limited to assert on Crossplane's Managed Resources render correctly. We can even assert on Kubernetes events! Since Crossplane utilizes many Kubernetes events, we can assert on any specific condition in our Crossplane setup. Pretty cool!

Integration Testing: Configuring AWS Provider in kuttl for testing actual infrastructure provisioning (with real AWS access)

Now that we have a full kuttl cycle running and validating our Crossplane resource rendering, there is another scenario we can use this exact tooling: making sure through Integration Testing that our Crossplane Compositions provision real infrastructure correctly:

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo, AWS logo, Azure logo, Google Cloud logo

To get this scenario working, we only need to tweak some things a bit we already created.

First we need to create a Secret containing our AWS credentials that Crossplane can later use to access AWS. Therefore we simply create an aws-creds.conf file (remember to have the aws CLI correctly installed and configured) in the root of our project:

1echo "[default]
2aws_access_key_id = $(aws configure get aws_access_key_id)
3aws_secret_access_key = $(aws configure get aws_secret_access_key)
4" > aws-creds.conf

ATTENTION: Don't ever check aws-creds.conf into version control. The best is to add it to your .gitignore file right now.

Inside our kuttl-test.yaml we add another command statements to create the Secret and configure the AWS Provider inside our kuttl kind cluster:

1apiVersion: kuttl.dev/v1beta1
2kind: TestSuite
3timeout: 300 # We definitely need a higher timeout for the external AWS resources to become available
4commands:
5  # Install crossplane via Helm Renovate enabled (see https://stackoverflow.com/a/71765472/4964553)
6  - command: helm dependency update crossplane/install
7  - command: helm upgrade --install --force crossplane --namespace crossplane-system crossplane/install --create-namespace --wait
8
9  # Install the crossplane Upbound AWS S3 Provider Family
10  - command: kubectl apply -f crossplane/provider/upbound-provider-aws-s3.yaml
11  # Wait until AWS Provider is up and running
12  - command: kubectl wait --for=condition=healthy --timeout=180s provider/upbound-provider-aws-s3
13
14  # Create AWS Provider secret (pre-deleting it to prevent errors like this while re-using the kuttl kind cluster)
15  - command: kubectl delete secret aws-creds -n crossplane-system --ignore-not-found
16  - command: kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf
17  # Create ProviderConfig to consume the Secret containing AWS credentials
18  - command: kubectl apply -f crossplane/provider/provider-config-aws.yaml
19testDirs:
20  - tests/e2e/
21startKIND: true
22kindContext: crossplane-test

You might wonder why we delete the Secret before we even created it. Why is that? Because we want to omit errors like error: failed to create secret secrets "aws-creds" already exists.

See https://stackoverflow.com/a/45881324/4964553 - The best approach using dry-run=client sadly doesn't work with kuttl producing a error: unknown shorthand flag: 'f' in -f error.

You may also have noticed that we configured a higher timeout for resources to become available via the timeout configuration of our TestSuite. This is because provisioning real infrastructure takes its time to be provisioned. And sadly, we can't configure the timeout directly in the 01-when-applying-claim.yaml TestStep. But without the setting we would otherwise run into errors like this one soon:

1case.go:364: failed in step 1-when-applying-claim
2    case.go:366: command "kubectl wait --for condition=Ready --timeout=180s objectstorage.crossplane.jonashackt.io/managed-upbound-s3" exceeded 30 sec timeout, context deadline exceeded

The final bit is to configure the AWS Provider via a ProviderConfig that actually makes use of the Secret featuring the AWS credentials. Therefore we need to change the provider-config-aws.yaml located in crossplane/provider:

1apiVersion: aws.upbound.io/v1beta1
2kind: ProviderConfig
3metadata:
4  name: default
5spec:
6  credentials:
7    source: Secret
8    secretRef:
9      namespace: crossplane-system
10      name: aws-creds
11      key: creds

Now our kuttl setup should be able to provision real infrastructure with Crossplane.

Validate / Assert for testing actual infrastructure provisioning (with real AWS access)

But before actually running our kuttl setup, we also need to enhance our 01-assert.yaml located in tests/e2e/objectstorage:

1apiVersion: kuttl.dev/v1beta1
2kind: TestAssert
3timeout: 30 # override test suite's long timeout again to have fast results in assertion
4# Clean up AWS resources if something goes wrong, see https://kuttl.dev/docs/testing/reference.html#collectors
5collectors:
6- type: command
7  command: kubectl delete -f ../../../examples/objectstorage/claim.yaml
8---
9apiVersion: s3.aws.upbound.io/v1beta1
10kind: Bucket
11metadata:
12  name: kuttl-test-bucket
13spec:
14  forProvider:
15    region: eu-central-1
16---
17apiVersion: s3.aws.upbound.io/v1beta1
18kind: BucketACL
19metadata:
20  name: kuttl-test-bucket-acl
21spec:
22  forProvider:
23    acl: public-read
24    bucket: kuttl-test-bucket # This is only testable with real AWS infrastructure
25    bucketRef:
26      name: kuttl-test-bucket
27    region: eu-central-1

Using an explicit TestAssert definition here we're able to override the TestSuite's timeout again to enable a faster test cycle. Otherwise the assertion would also wait for 300 seconds as defined in the test suite above.

Additionally we use a collector to make sure a cleanup step is also run in case of an error. Without that collector, infrastructure provisioned in our kuttl test steps wouldn't be cleansed and thus create unnecessary costs.

Now run our test suite with a kubectl kuttl test command.

You may even watch your AWS console, where the bucket gets created:

Pretty neat! Our setup with kuttl and Crossplane is now also ready to do Integration Testing with real infrastructure! Just be careful though how many Integration Tests you create and want to maintain. Just as the TDD testing Pyramid states, there should be far more Unit Tests than Integration Tests.

Cleanup after assertion (with real AWS access)

You might already have thought of it: In case of a successful test run, the provisioned infrastructure would also be preserved! Therefore, in case of Integration Tests with real AWS access, we should also clean up all resources after the last assertion ran. Therefore let's create a 02-* step called 02-cleanup.yaml in the tests/e2e/objectstorage dir:

1apiVersion: kuttl.dev/v1beta1
2kind: TestStep
3commands:
4  # Cleanup AWS resources
5  - command: kubectl delete -f ../../../examples/objectstorage/claim.yaml

This cleanup step should make sure that our infrastructure gets deleted in the end when everything went fine. If not, we have configured our collector inside the TestAssert config above.

Running Crossplane-featured kuttl tests in GitHub Actions

As mentioned at the beginning of this post, the holy grail is to have your tests being run automatically. Either scheduled or whenever a new Crossplane version or Provider version is released (triggered by Renovate for example). In order to achieve this, we can leverage a CI tooling like GitHub Actions:

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo, AWS logo, Azure logo, Google Cloud logo, GitHub Actions logo, Renovate logo

Thus, the following YAML shows a full GitHub Actions workflow executing the Crossplane featured kuttl tests (see the example project workflow in .github/workflows/kuttl-crossplane-aws.yml):

1name: kuttl-crossplane-aws
2
3on: [push]
4
5# Secrets configuration is only needed for real external AWS infrastructure
6env:
7  # AWS
8  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
9  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
10  AWS_DEFAULT_REGION: 'eu-central-1'
11
12jobs:
13  run-kuttl:
14    runs-on: ubuntu-latest
15    steps:
16      - name: Checkout
17        uses: actions/checkout@master
18
19      # Secrets configuration is only needed for real external AWS infrastructure
20      - name: Prepare AWS access via aws-creds.conf
21        run: |
22          echo "### Create aws-creds.conf file"
23          echo "[default]
24          aws_access_key_id = $AWS_ACCESS_KEY_ID
25          aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
26          " > aws-creds.conf
27
28      - name: Install kuttl & run Crossplane featured kuttl tests
29        run: |
30          echo "### Add homebrew to path as described in https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#notes"
31          eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
32        
33          echo "### Install kuttl via brew"
34          brew tap kudobuilder/tap
35          brew install kuttl-cli
36
37          echo "### Let's try to use kuttl"
38          kubectl kuttl --version
39
40          echo "### Run Crossplane featured kuttl tests"
41          kubectl kuttl test

Be sure to have the repository secrets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in place in case you want to do Integration Testing with kuttl and GitHub Actions! As the example repository on GitHub has Renovate configured, you can see it automerging Crossplane and Crossplane Provider updates after successful kuttl test runs:

Testing Crossplane Compositions with kuttl is a great combination!

In this blog series we saw how to leverage kuttl to test our Crossplane Compositions. We installed and configured Crossplane in kuttl and provided two options for testing: Starting with the Crossplane resource rendering tests, we have a sort of Unit Testing harness available for our Crossplane Configurations! These can be run fast and in isolation in our CI system.

The second option is to use kuttl for Integration Testing Crossplane Compositions. This is also a great option if we really want to make sure our provisioned infrastructure looks exactly the way we intended it to.

The overall structure of our kuttl tests, borrowed from the Behavior-driven Development scheme, will make sure that our tests stay readable and maintainable – which is overall one of the biggest success factors while using tests in our project. Even Test-driven Development of Crossplane components becomes possible with kuttl! And leveraging CI systems such as GitHub Actions and Renovate will automatically make sure our Crossplane Compositions run with future versions of Crossplane and its Providers.

I would be greatly interested to read about your experiences with Crossplane testing in the comments!