Seven Ways to Replace Kaniko in your Container Image Builds

Seven Ways to Replace Kaniko in your Container Image Builds

Modern CI pipelines have largely abandoned specialized, custom-configured build servers in favor of reproducible, code-defined environments. In GitLab CI and other popular platforms, this is provided through a container-based approach, where each CI job runs in its own Linux container. This provides a clean, reproducible, and portable build environment, which is a massive win for consistency.

However, things start to get tricky when you want to build container images for your application within these CI job containers. Kaniko, an open source project originally released by Google, has been a popular solution for this problem for quite a while. However, its recent deprecation has left many teams looking for alternatives.

In this post, we will explore seven options to replace Kaniko in container-based CI jobs, with a focus on GitLab CI.

Background: Container isolation trade-offs

Why are image builds such a big issue in the first place?

In a container-based CI environment, jobs should not interfere with each other. Even more so, job containers should not be able to modify their host machine: Because of their vital role, CI servers enable straightforward supply-chain attacks and are an attractive target for attackers. Even though this risk is particularly acute on shared CI servers, isolation between jobs and the host system is still desirable on project-specific servers. Providing this isolation is the job of the container runtime (typically Docker, containerd, or CRI-O).

In order to build container images, we must (at least to some degree) launch another container within the CI job container. This is largely because of the various Linux kernel features that power containers. I will not cover them in detail in this blog post. If you're interested, check out these resources:

• Jessie Frazelle's 2018 blog post on Building Container Images Securely on Kubernetes
• Alban Crequy's ContainerDays 2018 talk Towards unprivileged container builds: Video, accompanying blog post
• Andrew Martin's All Systems Go 2019 talk Rootless, Reproducible & Hermetic: Secure Container Build Showdown
• My ContainerDays 2024 talk Unprivileged Image Builds: What are the Challenges and Where are we Today?: Video, slides
• Its slightly updated 2025 version Beyond Kaniko: Navigating Unprivileged Container Image Creation

It is relatively easy to omit the isolation between the CI job container and the image being built, which is the approach taken by Kaniko. This omission is also not as concerning from a security perspective, since the job container is ephemeral and in most cases, both originate from the same codebase. It is also easy to omit the isolation between the host and the CI job container. That should be avoided because it does allow breaches of the host from a single CI job. Maintaining both layers of isolation is possible, but requires some effort.

Kaniko provides no isolation between the CI job container and the image being built. Other options do not isolate between the host and the CI job container instead. Ideally, we would like to keep both layers intact.

We're going to learn how to do that in practice later in this blog post. But first, let's explore some simpler options.

Option 1: Keep using Kaniko

OK, this is not really an alternative to Kaniko, but let's not dismiss this option prematurely.

While the original image by Google (gcr.io/kaniko-project/executor) is still available and probably won't go anywhere soon, it is not a good idea to base critical build steps on unmaintained software.

Luckily, Chainguard has stepped in and now develops a fork of Kaniko. The catch is that while development happens in the open and source releases are provided, binary releases and container images are only available to paying customers.

Alternatively, you can build an image yourself or look for unofficial, community-provided images. GitLab is currently also planning to provide images built by them. Furthermore, there is at least one other, community-maintained fork of Kaniko that provides container images free of charge.

As for me personally, I had been unhappy with Kaniko for quite a while, even before Google's discontinuation. Perhaps now that Chainguard has taken over development, they will address some long-standing, fundamental bugs. Still, I can't help but feel like it's a tool whose time has passed.

Option 2: Privileged containers or mounted sockets

For the sake of completeness, let's also mention ways without isolation between the host and the CI job container. While these are probably the oldest ways to build container images within CI containers, they are still somewhat common and recommended by the GitLab docs.

You either launch a Docker in Docker ("dind") job container in privileged mode, effectively giving it full control of the host. Or you mount the Docker socket (/var/run/docker.sock) into the job container. While the latter option may look more secure at first glance, it means that the job container can spawn arbitrary other containers, including privileged ones. Ultimately, this also grants full control of the host.

If you followed my argument from above, you will not be surprised to hear that I consider this unacceptable from a security perspective.

Option 3: Buildah with chroot isolation

Buildah is a container image build tool by Red Hat. It is strongly affiliated with Podman and the rest of Red Hat's container world. In fact, it gets used internally when you run podman build, but it can also be used as a stand-alone tool.

What makes Buildah particularly interesting for us is its support for different isolation modes: You can adjust the level of isolation between the image being built and its environment (i.e. the CI job container in our case) through the --isolation flag or the BUILDAH_ISOLATION environment variable.

Buildah's least-isolated mode is called chroot isolation. According to the manpage, this means that Buildah will be:

reusing the host's control group, network, IPC, and PID namespaces, and creating private mount and UTS namespaces, and creating user namespaces only when they're required for ID mapping

Unlike with Kaniko, we need to ensure the container has the ability to perform mounts and create User Namespaces. On the other hand, procfs masking and read-only cgroups will not be issues in chroot isolation mode.

Getting it working with Docker

What is actually required to get Buildah with chroot isolation running in practice? Let's first consider the case of the GitLab Docker Executor.

By default, Docker will apply a seccomp profile that prohibits required syscalls such as mount() and unshare(). The easiest way to work around this is to disable seccomp filtering completely in your GitLab Runner's config.toml file:

1[runners.docker]
2# ...
3security_opt= ["seccomp=unconfined"]

The cleaner way is to provide a custom seccomp profile allowing just the required syscalls in addition to the defaults. I have assembled an adjusted version of Docker's default profile and am providing it here. Note that in order to provide it through the GitLab Runner, you have to provide it as a long, single-line JSON string inline within the config.toml:

1[runners.docker]
2# ...
3security_opt = ['seccomp={"defaultAction": ...

On Debian and Ubuntu, Docker is typically further restricted through AppArmor in enforce mode. Other distributions may enforce SELinux rules in a similar fashion. AppArmor and SELinux are additional layers that, in our case, will also prohibit some required syscalls. We can deal with them in much the same way as seccomp: Either disable them or provide custom profiles. In order to disable AppArmor, you add the "apparmor:unconfined" option to config.toml:

1[runners.docker]
2# ...
3security_opt= ["seccomp=unconfined", "apparmor=unconfined"]

Just as for seccomp, I'm providing an adjusted version of Docker's default AppArmor profile. Profiles are stored in /etc/apparmor.d and loaded once through:

1apparmor_parser -r -W '/etc/apparmor.d/<profile-name>'

Afterward, you reference the profile from config.toml like this:

1[runners.docker]
2# ...
3security_opt= ["apparmor=<profile-name>"]

This minimal example does not require Docker itself to be running with User Namespaces. However, Buildah will set up a User Namespace internally. For this to work, unprivileged users need the ability to create User Namespaces. That should be the case on most modern distributions, but note that recent versions of Debian and Ubuntu introduced AppArmor restrictions around it. Both disabling AppArmor and using my custom profile handle it appropriately.

Both seccomp and AppArmor are defense-in-depth mechanisms that we disabled or at least somewhat weakened through our adjustments. Given that, I recommend compensating for that by running Docker itself with User Namespaces. You can follow the instructions from the official Docker docs for that. The rest of the configuration should continue to work as before.

Unfortunately, all of the settings – seccomp, AppArmor / SELinux, and User Namespaces – require adjustments to the Runner or CI host config. This means you cannot utilize them as a mere user of GitLab CI, but need to be able to administer your GitLab Runner.

It also means all changes apply to all jobs on the respective Runner, not just container image builds. You may, of course, set up a special Runner and designate it for image builds using tags. However, the security benefits are marginal if this Runner can still be used by the same range of projects and developers.

Getting it working on Kubernetes

With the GitLab Kubernetes Executor, the general approach is similar to Docker. Since vanilla Kubernetes does not apply seccomp or AppArmor profiles, there is a significant chance that Buildah with chroot isolation will work out of the box. For example, it will just work in an Amazon EKS cluster with default options.

Restrictions may apply if you have a hardened Kubernetes cluster and your ability to overwrite these as a cluster tenant (e.g. setting a custom seccomp profile) may be restricted through Pod Security Standards or an Admission Controller such as Kyverno or OPA Gatekeeper. This means that depending on your exact setup, some configuration changes might be required at the Kubernetes cluster level.

Options for the GitLab Runner on Kubernetes are typically not set directly in a config.toml file, but through Helm values. For example, let's look at how to apply a custom seccomp profile to Pods launched through GitLab CI. With Kubernetes, profiles get loaded from JSON files in /var/lib/kubelet/seccomp on the Node's file system. You would add this to your Helm values.yml file:

1runners:
2  config: |
3    [[runners]]
4      environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true"]
5
6      [runners.kubernetes]
7        namespace = "{{ default .Release.Namespace .Values.runners.jobNamespace }}"
8        image = "alpine"
9
10        # See https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3904#note_1322875439
11        [[runners.kubernetes.pod_spec]]
12        patch = '''
13          containers:
14            - name: build
15              securityContext:
16                seccompProfile:
17                  type: Localhost
18                  localhostProfile: <profile-name>.json
19        '''
20        patch_type = "strategic"

Regarding User Namespaces, the same considerations apply as with the Docker Runner Executor: Buildah will use them internally, for which unprivileged User Namespaces need to be available on the cluster Nodes. Recent versions of Kubernetes also support running the Buildah pod itself in a User Namespace. But getting this to work for image builds can be a delicate endeavor. We'll discuss it for option 4 below.

The CI job

What does a GitLab CI job for Buildah with chroot isolation actually look like?

In our example, we will use the official quay.io/buildah/stable container image. You may of course use any other image and, for example, install Buildah through package management. One downside of the official image is that unlike Buildah's general default, it is pre-configured to use fuse-overlayfs.

I assume this is mostly for historic reasons, since at one point in time, fuse-overlays used to be supported in User Namespaces, while regular OverlayFS was not. However, this is not the case anymore since Linux kernel 5.13, so these days there is little reason to deal with the intricacies of getting FUSE to work. Other guides such as GitLab's Buildah rootless tutorial opt to use Buildah's VFS storage driver, but that can result in lower performance and higher disk usage.

The easiest way to get the container to use OverlayFS is to comment out the mount_program = and mountopt = options from /etc/containers/storage.conf. Including this, a full .gitlab-ci.yml file appears as follows:

1stages:
2  - build
3
4build_image:
5  stage: build
6  image: quay.io/buildah/stable
7  before_script:
8    - buildah login -u "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
9    - sed -i 's/^mount_program =/#&/' /etc/containers/storage.conf
10    - sed -i 's/^mountopt =/#&/' /etc/containers/storage.conf
11  script:
12    - buildah build --isolation chroot -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
13    - buildah push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

All in all, Buildah with chroot isolation is a strong option currently available to many people. It provides a bit more isolation than Kaniko while being based on an established, well-maintained tool. Depending on the Runner setup, it may work out of the box or require adjustments to the Runner config, to the host machine, or the Kubernetes cluster.

But now that we got Buildah running, why stick to chroot isolation? Can we achieve even more isolation between the CI job container and the image being built?

Option 4: Buildah with rootless isolation

One isolation level up from chroot isolation, Buildah offers rootless isolation. Compared to chroot isolation, we now also get private IPC and PID namespaces for the image being built.

A separate PID namespace requires mounting a new procfs, which means the /proc shadowing performed by container runtimes becomes an issue. In essence, since the CI job container does not have access to the full /proc in the first place, it is not allowed to mount a new procfs.

In general, we can disable this behavior for Docker by launching containers with --security-opt systempaths=unconfined. Without the masking, enabling user namespaces is strictly advised to prevent non-isolated privileged operations from within the image build.

Unfortunately, there is currently no way to tell the GitLab Docker Executor to launch job containers with the systempaths=unconfined option. This is known as GitLab Runner issue #36810. It means that as of now, we cannot use rootless isolation with the Docker Executor.

For the Kubernetes Executor, proc masking can be disabled through the Helm values.yml file. The full Runner settings, including enabled User Namespaces and the seccomp profile from above, then looks like this:

1runners:
2  config: |
3    [[runners]]
4      environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true"]
5      builds_dir = "/tmp/builds"
6
7      [runners.kubernetes]
8        namespace = "{{ default .Release.Namespace .Values.runners.jobNamespace }}"
9        image = "alpine"
10
11        # See https://docs.gitlab.com/runner/executors/kubernetes/#user-namespaces
12        privileged = false
13        allowPrivilegeEscalation = false
14        logs_base_dir = "/tmp"
15        scripts_base_dir = "/tmp"
16
17        [[runners.kubernetes.pod_spec]]
18        name = "hostUsers"
19        patch = '''
20          [{"op": "add", "path": "/hostUsers", "value": false}]
21        '''
22        patch_type = "json"
23
24        # See https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3904#note_1322875439
25        [[runners.kubernetes.pod_spec]]
26        patch = '''
27          containers:
28            - name: build
29              securityContext:
30                procMount: Unmasked
31                seccompProfile:
32                  type: Localhost
33                  localhostProfile: <profile-name>.json
34        '''
35        patch_type = "strategic"

However, as mentioned above, getting User Namespaces to actually work in this environment can be more complex. The exact steps heavily depend on your Kubernetes cluster components, their configuration, and the amount of hardening.

Aspects to consider include:

• You need at least Kubernetes 1.33 for User Namespaces to be generally available.
• Due to Kubernetes' usage of ID-mapped mounts for tmpfs, you will also need at least Linux kernel 6.3.
• With the containerd CRI, at least containerd 2.0 is required.
• The interaction between Kubernetes, containerd, and User Namespaces exposes an unsolved corner case affecting the /var/lib/containers volume defined by the quay.io/buildah image.

The latter part can be solved by manually mounting volumes through an additional piece of config in your values.yml. However, this specifically tailors your generic Runner config for Buildah usage:

1runners:
2  config: |
3        # ...
4        # (See above)
5
6        # Workaround for https://github.com/containerd/containerd/discussions/12212
7        # because "'overlay' is not supported over overlayfs"
8        # See also: https://github.com/containers/image_build/blob/943ddba/buildah/Containerfile
9        [[runners.kubernetes.volumes.empty_dir]]
10        name = "var-lib-containers"
11        mount_path = "/var/lib/containers"
12
13        [[runners.kubernetes.volumes.empty_dir]]
14        name = "home-local-share-containers"
15        mount_path = "/home/build/.local/share/containers"

Rootless isolation might provide you some increased isolation, but is even more intricate to set up. I would suggest to first start with getting chroot isolation running and then, if your environment allows it, increase it to rootless.

Option 5: BuildKit in rootless mode

In many ways, BuildKit is to Docker what Buildah is to Podman. In fact, BuildKit gets invoked internally when you run docker build these days. While BuildKit is less designed to be invoked as an individual utility than Buildah, it can still be invoked in a stand-alone way.

BuildKit's rootless mode behaves similarly to Buildah's rootless isolation by default. PID namespaces can optionally be disabled through the --oci-worker-no-process-sandbox flag, making the behavior more akin to Buildah with chroot isolation.

The requirements are the same as for Buildah (as in option 3): seccomp and AppArmor / SELinux need to be disabled or adjusted. User Namespaces for the BuildKit container itself may be used optionally. Without --oci-worker-no-process-sandbox, proc masking needs to be disabled (as in option 4).

Based on the example from the GitLab docs, we can create a .gitlab-ci.yml file for BuildKit like this:

1stages:
2  - build
3
4build_image:
5  stage: build
6  image: moby/buildkit:rootless
7  variables:
8    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
9  before_script:
10    - mkdir -p ~/.docker
11    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > ~/.docker/config.json
12  script:
13    - |
14      buildctl-daemonless.sh build \
15        --frontend dockerfile.v0 \
16        --local context=. \
17        --local dockerfile=. \
18        --output type=image,name=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA,push=true

If I recall correctly, I successfully used BuildKit with this setup in the past. In my recent tests, though, the BuildKit build would always hang after logging "running server on /run/user/1000/buildkit/buildkitd.sock". However, this also happened in a privileged container, so I assume it is unrelated to the isolation.

If you're able to find the cause for that issue and, for whatever reason, do not want to use Buildah, BuildKit might be a viable alternative.

Option 6: Virtual Machines

Finally, after our deep dive into the internals of containers and isolation, let's take a step back and consider why all this effort is required. So far, we've been trying to solve the problem from inside the container world. But what if we took a step back and questioned the premise? After all, GitHub Actions and other CI platforms completely avoid the problem by using full-blown Virtual Machines instead of containers. Could we do the same thing with GitLab CI?

In order for this to work, we would need a GitLab CI Executor that launches an ephemeral VM per build job. I would expect this to be built upon a virtualization technology such as libvirt, VMware, or OpenStack, or even a cloud service like Amazon EC2.

The closest option to fulfill these requirements are the GitLab docs on Using libvirt with the Custom executor. This solution uses a surprisingly simple collection of shell scripts to launch VMs using libvirt. I cannot assess its production readiness, so I'd be interested to hear from anyone who has experience with it. It should be obvious that this cannot be done as a mere user of GitLab CI, but requires significant changes to the Runner infrastructure.

Option 7: Advanced container runtimes

Taking it one step further, there are some projects that combine the deployment model of Docker and Kubernetes with the isolation of Virtual Machines. If done right, this could, for example, allow running privileged containers that are still isolated from the host machine.

These projects can roughly be divided into two categories: Those fully utilizing hardware-based virtualization and those sticking to a software solution.

In the hardware-based category, Kata Containers is a widely used project. Kata will launch a lightweight VM per container respectively Pod that isolates it from the host system. It can be plugged into containerd as a container runtime and therefore be used within Kubernetes.

Firecracker is an open source technology by AWS that popularized the notion of micro VMs for container isolation. For example, it powers AWS Lambda. While the firecracker-containerd project does not yet seem ready for production usage with Kubernetes, Firecracker may be used as the underlying hypervisor in Kata Containers.

gVisor by Google is a software-based solution that gets inserted between the container and the host kernel. It intercepts all system calls and implements a substantial subset of the kernel API in user space. Similar to Kata Containers, gVisor can be plugged into containerd and is also readily available on the Google Kubernetes Engine as GKE Sandbox.

Finally, sysbox promises to enable all kinds of container workloads, including nested containers, through a sophisticated arrangement of Linux kernel features and custom code. In fact, sysbox faces the same challenges we discussed above and is powered by many of the same technologies: User Namespaces, ID-mapped mounts, and strategic mounts of volumes to avoid OverlayFS on OverlayFS. What sets sysbox apart is its virtualization of procfs and sysfs to avoid masking issues. The company behind sysbox got acquired by Docker in 2022 and Docker offers its capabilities commercially as Enhanced Container Isolation.

All of these advanced container runtimes offer powerful capabilities for building container images. However, I have not evaluated their practicality and compatibility with GitLab CI.

While these options may require significant investment, they are solid and put an emphasis on security. If security is your number one priority and you're building a dedicated CI platform to last for years, exploring them might be worth the effort. Let me know if you have any experience with getting them working!

Summary

We have comprehensively explored a variety of options, from sticking with Kaniko (for now) to completely redesigning your CI Runner infrastructure. Each option has different trade-offs and varies in its ease of implementation.

What works best for you heavily depends on your existing infrastructure and the amount of control you have over your CI environment. To provide you with some practical guidance:

• If you need a quick fix and are willing to spend some money or trust a community fork, stick with Kaniko (option 1).
• If you have full admin access to your own Runners, use Buildah with chroot isolation (option 3).
• If your Runners are powered by a bleeding-edge Kubernetes cluster, you may also consider Buildah with rootless isolation (option 4).
• If you're on a shared Runner with no admin rights, your best bet is to continue using a Kaniko image (option 1) and hope for the other options to become more accessible on shared platforms in the future.

Thanks to Laura Spork and Simon Ruderich for providing valuable feedback on an earlier revision of this post.