GitLab security scanning – part 3: Kubernetes deployments

In part 1 and part 2 , we focused on different types of security scanning practices. In this article we will take a look at Kubernetes deployments with Helm and Helmfile. In particular, we are interested in how to ensure that objects deployed to Kubernetes follow security best practices.

1) Scanning Helm Charts

Let’s first look at how security best practices can be ensured in deployments of Helm Charts using GitLab pipelines. As described in helm charts , Helm Charts summarise YAML manifest files in a specific format. In particular, these manifests describe deployments, StatefulSets or other Kubernetes objects that ultimately create pods in Kubernetes. For pods to run securely in Kubernetes, pod and container definitions, among other things, must define certain properties. For example, the SecurityContext should be set as restrictively as possible and ServiceAccounts should be used for pods. GitLab uses kubesec for scanning Helm Charts. Now let us look at the directory structure:

1.
2├── .gitlab-ci.yml
3├── helm-chart
4│  └── vuln-project
5│     ├── Chart.yaml
6│     ├── README.md
7│     ├── templates
8│     │  ├── _helpers.tpl
9│     │  ├── deployment.yaml
10│     │  └── service.yaml
11│     └── values.yaml
12└── README.md

The root directory contains the .gitlab-ci.yml file. Parallel to this is the directory helm-chart, which contains the helm chart vuln-project. This path must be given to the pipeline as variable KUBESEC_HELM_CHARTS_PATH (line 8). To enable kubesec analyzer, set SCAN_KUBERNETES_MANIFESTS: "true". Scanning a Helm Chart is one of the static analyses. Therefore, it can be activated by inserting the SAST job (line 2+3).

1# Enable SAST scanning
2include:
3  - template: Security/SAST.gitlab-ci.yml
4
5# Enable manifest scanning on helm chart directory
6variables:
7  SCAN_KUBERNETES_MANIFESTS: "true"
8  KUBESEC_HELM_CHARTS_PATH: $CI_PROJECT_DIR/helm-chart/vuln-project

Once the sources have been committed and pushed to the GitLab server, kubesec starts analysing.

GitLab then reports any vulnerabilities found in the security tab of the pipeline:

Details of vulnerabilities found and links to further information can be found in the popup window:

Findings can be rated in the pop-up window in the same way as in the overview. If this assessment requires mitigation, an issue can also be created here for tracking.

2) Scanning Helmfiles

As seen in Part 1 of this article, GitLab natively supports security scanning of Helm Charts. For the declarative deployment of Helm Charts with Helmfile , this support is missing. However, to still be able to check Helmfile sources for vulnerabilities, a small additional step is sufficient. Let’s first look at the directory structure:

1.
2├─ .gitlab-ci.yml
3├─ helmfile
4  └── vuln-project
5     ├── data
6     │  ├── helmchart
7     │  │  └── vuln-project
8     │  └── helmfile
9     │     ├── helmfile.d
10     │     │  ├── 00-namespaces.yaml
11     │     │  ├── 10-network-policies.yaml
12     │     │  └── 20-vuln-project-install.yaml
13     │     ├── helmfile.yaml
14     │     └── values
15     │        └── vuln-project.yaml.gotmpl
16     └── env
17        ├── prod
18        └── test

Again the root directory contains the .gitlab-ci.yml file. The helmfile.yaml file contains the inclusion of the various stages from the env directory and defines which objects are installed. In order for SAST to have files for a security analysis, Helmfile must first be executed. This happens in the .gitlab-ci.yml in an additional job build_helm_manifests (line 5-16). This job creates a directory $CI_PROJECT_DIR/k8s-manifests and updates the helm repos used. Subsequently, helmfile template --output-dir $CI_PROJECT_DIR/k8s-manifests generates YAML manifests (lines 10-12). In order for these generated files to be passed to the kubesec-sast job for analysis, these manifests are stored in an artifact (lines 14-16). In order for the kubesec-sast job to access the artifacts, it is given a dependency on the build job (lines 19-21). Finally, the kubesec analyser must be activated with SCAN_KUBERNETES_MANIFESTS: "true" (line 29) and by inserting the SAST job .gitlab-ci.yml (line 24+25).

1stages:
2  - build
3  - test
4
5build_helm_manifests:
6  stage: build
7  image:
8    name: image.registry/k8s-cicd-tools-image:0.0.1
9  script:
10    - mkdir -p $CI_PROJECT_DIR/k8s-manifests
11    - helmfile --environment test -f $CI_PROJECT_DIR/helmfile/vuln-project/data/helmfile/helmfile.yaml repos
12    - helmfile --environment test -f $CI_PROJECT_DIR/helmfile/vuln-project/data/helmfile/helmfile.yaml template --output-dir "$CI_PROJECT_DIR/k8s-manifests"
13  artifacts:
14    paths:
15      - $CI_PROJECT_DIR/k8s-manifests
16    expire_in: 10 minutes
17
18# Extend job kubesec-sast by dependency to pass artifacts
19kubesec-sast:
20  dependencies:
21  - build_helm_manifests
22
23# Enable SAST scanning
24include:
25  - template: Security/SAST.gitlab-ci.yml
26
27# Enable manifest scanning
28variables:
29  SCAN_KUBERNETES_MANIFESTS: "true"

The pipeline starts as soon as the code is pushed to GitLab. As expected, two jobs are now running one after the other to create the artifact and to run the security scanning procedures.

After the security scan has been completed, the findings can be viewed in the vulnerability report. Here it is possible to apply various filters to the results and sort them if necessary. Details about vulnerabilities can be displayed via links within the findings.

Summing up

GitLab can also scan deployments to Kubernetes for security best practices using tolls such as Helm Charts or even Helmfile. If vulnerabilities are discovered, they can be fixed before deployment.

This article concludes the series of articles (GitLab security scanning – part 1 , GitLab security scanning – part 2 ) on security scanning with GitLab.