Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3

What if learning the Kubernetes API is all you need to provision any infrastructure? And we’re not only talking about AWS, Azure & Google – but also IONOS, DigitalOcean and even vSphere. Let’s have a look at Crossplane and how we can create an S3 Bucket on AWS step by step.

No code required, it’s all declarative!

Crossplane claims to be the “The cloud native control plane framework”. It introduces a new way for managing any cloud resource (be it Kubernetes-native or not). It’s an alternative Infrastructure-as-Code tooling to Terraform , AWS CDK /Bicep or Pulumi and introduces a different level of abstraction, based on Kubernetes CRDs . One could name it the cloud native way to do GitOps.

Crossplane – blog series

1. Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3
2. Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite
3. Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert
4. Create, build & publish Crossplane Configuration Packages with GitHub Actions & Container Registry

Why Crossplane is different

Crossplane can be also compared to tools which enable the management of cloud resources through the Kubernetes API, like AWS Controllers for Kubernetes (ACK) , Azure Service Operator for Kubernetes (ASO) or Google Config Connector . Crossplane providers are even generated from ACK and ASO . But the CNCF incubating project Crossplane adds on top of these:

The Crossplane community believes that the typical developer using Kubernetes to deploy their application shouldn’t have to deal with low level infrastructure APIs.

Crossplane uses the Kubernetes API and extends it with a set of Custom Resource Definitions (CRDs) to abstract from actual cloud provider APIs. Additionally, these CRDs are a great foundation to build an Internal Developer Platform (IDP) . Crossplane promotes self-service by introducing building blocks like Composite Resource Claims (XRCs) that are a great API for the typical application developer. On the other hand, Composite Resources (XR) and Composites are a magnitude more powerful and ideal for platform operators.

Drawing on our experiences as platform builders, SREs, and application developers, we’ve designed Crossplane as a toolkit to build your own custom resources on top of any API – often those of the cloud providers. We think this approach is critical to enable usable self-service infrastructure in Kubernetes.

Finally, Crossplane also enables effective multi-cloud interoperability . But don’t worry, you can even order Pizza using Crossplane :).

Crossplane basic concepts

The crossplane concept of a Composite Resource is composed out of several building blocks :

• Composite Resources (XR) : they compose Managed Resources into higher level infrastructure units (especially interesting for platform teams) and an optional CompositeResourceClaim (XRC) (which is also referred to as ‘Claim’ and can be seen as an abstraction of the XR for the application team to consume)
• a CompositeResourceDefinition (XRD) which defines an OpenAPI schema the Composition needs to be conform to (think of Kubernetes CRDs)
• a Composition that describes the actual infrastructure primitives aka Managed Resources used to build the Composite Resource. One XRD could have multiple Compositions – e.g. one for every environment like development, stating and production

Composite Resources can also be nested together, which allows for higher level abstractions and better separation of concerns. Think of an AWS EKS cluster where you need lot’s of network/subnetting setup (which can form one XR) and the actual EKS cluster creation with NodeGroups etc (which would be the XR using the networking XR). More in-depth details on nested XRs can be found here .

As an optional step, you can package these Composite Resource artifacts using a Configuration into an OCI container image. Now your custom Configuration package can be installed easily in any other Crossplane clusters.

Composite Resources are composed of infrastructure building blocks called Managed Resources (MR) that are bundled by a Provider:

• Managed Resourced (MR) : a Kubernetes custom resources (CRDs) that represent infrastructure primitives (mostly in cloud providers). All Crossplane Managed Resources could be found via https://doc.crds.dev
• Providers : are Packages that bundle a set of Managed Resources and a Controller to provision infrastructure resources – all providers can be found on GitHub, e.g. provider-aws or on docs.crds.dev . A list of all available Providers can also be found on GitHub.

You may also stumble upon Packages . They were formerly named Stacks and are simply OCI container images. Packages handle the distribution, version updates, dependency management and permissions for Providers and Configurations.

Getting started with Crossplane: fire up a k8s cluster with kind

In order to use Crossplane, we’ll need any kind of Kubernetes cluster to let it operate in. This management cluster with Crossplane installed will then provision the defined infrastructure. Using any managed or local Kubernetes cluster like EKS, AKS, Minikube or k3d is suitable. In my example project (which is available on GitHub ), I used kind to host the management cluster.

Using kind is pretty easy. Let’s get our hands dirty! Just be sure to have kind, the package manager Helm and kubectl installed. On a Mac you can use brew like this (have a look at the docs for other systems ):

We should also install the crossplane CLI:

The kubectl crossplane --help command should be ready to use.

Now spin up a local kind cluster with:

Install Crossplane with Helm

The Crossplane docs tell us to use Helm for installation :

As a Renovate -powered alternative we can create our own simple Chart.yaml to enable automatic updates of our installation if new crossplane versions get released:

To install Crossplane using our own Chart.yaml simply run:

Be sure to exclude /charts directories and Chart.lock files via .gitignore. Now Renovate will have an eye on our Crossplane versions:

Before we can actually apply a Provider, we have to make sure that Crossplane is actually healthy and running. Therefore we can use the kubectl wait command like this:

Otherwise we will run into errors like this when applying a Provider:

Finally check the Crossplane status with kubectl get all -n crossplane-system:

Configure Crossplane to access AWS: create aws-creds.conf

I assume that you have aws CLI installed and configured . The command aws configure should work on your system. With this prepared we can create an aws-creds.conf file as described in the docs :

Don’t ever check this file into source control – it holds your AWS credentials! In the example project I added *-creds.conf to the .gitignore file.

If you’re using a CI system like GitHub Actions (as this repository is also based on), you need to have both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY configured as repository secrets:

Also make sure to have your default region configured locally, or as a env: variable in your CI system. All three needed variables in GitHub Actions for example look like this:

Create AWS Provider secret

Now we need to use the aws-creds.conf file to create the Crossplane AWS Provider secret:

If everything went well there should be a new aws-creds secret ready:

Install the Crossplane AWS Provider

To be able to provision infrastructure on a cloud provider like AWS, we need to install the corresponding Crossplane Provider first. Remember: the Provider packages all needed Managed Resources (MRs) and their respective controllers. As a Provider is a Crossplane Package, we can install it using the Crossplane CLI like this:

Alternatively we can create our own provider-aws.yaml file like this:

This kind: Provider with apiVersion: pkg.crossplane.io/v1 is completely different from the kind: Provider which we want to consume. The latter uses apiVersion: meta.pkg.crossplane.io/v1.

Don’t forget to install the AWS provider using kubectl:

The package version in combination with the packagePullPolicy configuration here is crucial, since we can configure an update strategy for the Provider here. A full table of all possible fields can be found in the docs . We can also let Crossplane manage the upgrade to new versions for us. If you installed multiple package versions, you’ll see them prefixed with providerrevision.pkg.x when running kubectl get providerrevision:

Now our first Crossplane Provider has been installed. You may check it with kubectl get provider:

Before we can actually apply a ProviderConfig to our AWS provider we have to make sure that it’s actually healthy and running. Therefore we can use the kubectl wait command again like this:

Otherwise we may run into errors when applying the ProviderConfig right after the Provider.

Create ProviderConfig to consume the Secret containing AWS credentials

Now we need to create a ProviderConfig object that will tell the AWS Provider where to find it’s AWS credentials . Therefore we create a provider-config-aws.yaml :

Crossplane resources use the ProviderConfig named default if no specific ProviderConfig is specified, so this ProviderConfig will be the default for all AWS resources.

The secretRef.name and secretRef.key have to match the fields of the already created Secret. Apply the ProviderConfig with:

Provision a S3 Bucket with Crossplane

The Crossplane core controller and the Provider AWS controller should now be ready to provision any infrastructure component in AWS! So let’s start with a simple S3 Bucket.

The first step towards using Composite Resources is configuring Crossplane so that it knows what XRs you’d like to exist, and what to do when someone creates one of those XRs. This is done using a CompositeResourceDefinition (XRD) resource and one or more Composition resources.

So in order to provision a S3 Bucket in AWS using Crossplane we have to craft three building blocks:

1. CompositeResourceDefinition (XRD) (reference docs )
2. Composition (reference docs )
3. Composite Resource (XR) or Claim (XRC) (reference docs )

1. Defining a CompositeResourceDefinition (XRD) for our S3 Bucket

All possible fields a XRD may consist of are documented here . The field spec.versions.schema must contain a OpenAPI schema , which is similar to the ones used by any Kubernetes CRDs. They determine what fields the XR (and Claim) will have. The full CRD documentation and a guide on how to write OpenAPI schemas could be found in the Kubernetes docs .

Note that Crossplane will automatically extend this section. These includes the following fields, which will be ignored if they’re found in the schema:

• spec.resourceRef
• spec.resourceRefs
• spec.claimRef
• spec.writeConnectionSecretToRef
• status.conditions
• status.connectionDetails

The example project on GitHub hosts an a Composite Resource Definition (XRD) for an S3 Bucket. The definition.yaml could look like this:

The file uses lots of comments in the attempt to give some assistance for crafting your own XRD. A CompositeResourceDefinition can be roughly divided into the top area featuring Crossplane specific configuration (with spec.group, spec.names, spec.claimNames and a compositionRef using spec.defaultCompositionRef) and the bottom section leveraging the OpenAPI schema defining the parameters of our resources. In our case in which we want to provision an S3 Bucket, there are only two parameters: bucketName and region.

We need to install the XRD into our cluster with:

Optionally one can check the CRDs beeing created with kubectl get crds and filter them using grep to our group name crossplane.jonashackt.io:

2. Craft a Composition to provision a S3 Bucket for static website hosting

The main work in Crossplane has to be done crafting the Compositions. This is because they interact with the infrastructure primitives the cloud provider APIs provide.

There are detailed docs to many of the possible manifest configurations . A Composition to manage an S3 Bucket in AWS with public access for static website hosting could for example look like the example project’s composition.yaml :

Again this example uses lots of comments in the attempt to give some assistance for crafting your own Composition. After defining some metadata labels, spec.compositeTypeRef and spec.writeConnectionSecretsToNamespace the actual resource configuration happens in spec.resources. Since we want to provision a S3 Bucket in AWS we definitely should have a look into the Crossplane AWS provider API docs . It also helps to have the Terraform docs opened up to clarify some questions or to skim the internet for some example code. The latter wasn’t that comprehensive when writing this post. I hope to see more Crossplane example implementations around in the near future.

The Composition also needs to be installed with kubectl apply -f aws/s3/composition.yaml.

3. Craft a Composite Resource (XR) or Claim (XRC)

Only the platform team itself typically has the permissions to create XRs directly. Everyone else uses a lightweight proxy resource called CompositeResourceClaim (XRC or simply “Claim”) to create them with Crossplane. If you’re familiar with Terrafrom you can think of an XRD as similar to variable blocks of a Terrafrom module. The Composition could then be seen as the rest of the HCL code describing how to instrument those variables to create actual resources.

Regardless of the role you only have to write a Composite Resource (XR) or Claim (XRC) ! You don’t need to craft both, since the XR will be automatically generated from the XRC by Crossplane. If you step into the role of an platform engineer, you can start crafting the XR directly and no Claim will be generated.

Since we want to create a S3 Bucket, the example project hosts a claim.yaml :

As you can see the Claim is one of the simpler building blocks in Crossplane. You need to look at the correct apiVersion and name the kind exactly to what you defined in the CompositeResourceDefinition (XRD). Also a namespace is needed to define a valid Claim (which isn’t true for XRs, since they’re cluster scoped). Also a compositionRef or alternatively compositionSelector need to be defined to reference the Composition the Claim should use. Finally both our parameters bucketName and region must be defined.

Now kubectl applying our Claim will finally trigger the provisioning of our S3 Bucket in AWS! If you went through all the steps, you can now test-drive your setup with kubectl apply -f aws/s3/claim.yaml.

The CLI validation will have your back, if you held the YAML ruler incorrectly like me:

This will also help you debug your configuration – it hints for the actual problems that are still present.

Waiting for resources to become ready

There are some possible things to check while your resources get deployed after running a kubectl apply -f aws/s3/claim.yaml. The best overview gives a kubectl get crossplane which will simply list all the crossplane resources:

There are also some other useful commands you should know:

• kubectl get claim: get all resources of all Claim kinds, like PostgreSQLInstance.
• kubectl get composite: get all resources that are of kind Composite (aka XR), like XPostgreSQLInstance.
• kubectl get composition: don’t confuse with composite! Get’s you all Compositions.
• kubectl get managed: get all resources that represent a unit of external infrastructure.
• kubectl get name-of-provider: get all resources related to the Provider.

Troubleshooting your Crossplane configuration

There’s a great explanation in the docs on how to efficiently track down errors in Crossplane configuration:

Per Kubernetes convention, Crossplane keeps errors close to the place they happen. This means that if your Claim is not becoming ready due to an issue with your Composition or with a composed resource you’ll need to “follow the references” to find out why. Your Claim will only tell you that the XR is not yet ready.

The docs also tell us what they mean by “follow the references”:

• Find your XR by running kubectl describe claim-kind claim-metadata.name and look for its “Resource Ref” (aka spec.resourceRef).
• Run kubectl describe on your XR. This is where you’ll find out about issues with the Composition you’re using, if any.
• If there are no issues but your XR doesn’t seem to be becoming ready, take a look for the “Resource Refs” (or spec.resourceRefs) to find your composed resources.
• Run kubectl describe on each referenced composed resource to determine whether it is ready and what issues, if any, it is encountering.

Inspect the S3 Bucket and deploy a static website

Now let’s check our Claim with kubectl get claim-kind claim-metadata.name like this:

To watch the provisioned resources become ready, we can run kubectl get crossplane -l crossplane.io/claim-name=claim-metadata.name:

We can also check if the S3 Bucket has been created successfully via aws CLI with aws s3 ls.

Our bucket should be provisioned by now! It will then also be visible in the AWS console:

To really “proof” it’s working, let’s deploy a website (the example project holds a simple index.html ) to our S3 Bucket using the aws CLI like this:

Now we can open up microservice-ui-nuxt-js-static-bucket.s3-website.eu-central-1.amazonaws.com in our Browser and should see our website already deployed:

To delete the S3 Bucket again, we simply need to remove the Claim. But before deleting the Claim, we should remove our index.html. Otherwise we’ll run into errors like BucketNotEmpty: The bucket you tried to delete is not empty:

Using kubectl delete claim we can remove our S3 Bucket again: