In part 1 and part 2 , we focused on different types of security scanning practices. In this article we will take a look at Kubernetes deployments with Helm and Helmfile. In particular, we are interested in how to ensure that objects deployed to Kubernetes follow security best practices.
1) Scanning Helm Charts
Let’s first look at how security best practices can be ensured in deployments of Helm Charts using GitLab pipelines. As described in helm charts , Helm Charts summarise YAML manifest files in a specific format. In particular, these manifests describe deployments, StatefulSets or other Kubernetes objects that ultimately create pods in Kubernetes. For pods to run securely in Kubernetes, pod and container definitions, among other things, must define certain properties. For example, the SecurityContext should be set as restrictively as possible and ServiceAccounts should be used for pods. GitLab uses kubesec for scanning Helm Charts. Now let us look at the directory structure:
1. 2├── .gitlab-ci.yml 3├── helm-chart 4│ └── vuln-project 5│ ├── Chart.yaml 6│ ├── README.md 7│ ├── templates 8│ │ ├── _helpers.tpl 9│ │ ├── deployment.yaml 10│ │ └── service.yaml 11│ └── values.yaml 12└── README.md
The root directory contains the
.gitlab-ci.yml file. Parallel to this is the directory
helm-chart, which contains the helm chart
vuln-project. This path must be given to the pipeline as variable
KUBESEC_HELM_CHARTS_PATH (line 8). To enable kubesec analyzer, set
SCAN_KUBERNETES_MANIFESTS: "true". Scanning a Helm Chart is one of the static analyses. Therefore, it can be activated by inserting the SAST job (line 2+3).
1# Enable SAST scanning 2include: 3 - template: Security/SAST.gitlab-ci.yml 4 5# Enable manifest scanning on helm chart directory 6variables: 7 SCAN_KUBERNETES_MANIFESTS: "true" 8 KUBESEC_HELM_CHARTS_PATH: $CI_PROJECT_DIR/helm-chart/vuln-project
Once the sources have been committed and pushed to the GitLab server, kubesec starts analysing.
GitLab then reports any vulnerabilities found in the security tab of the pipeline:
Details of vulnerabilities found and links to further information can be found in the popup window:
Findings can be rated in the pop-up window in the same way as in the overview. If this assessment requires mitigation, an issue can also be created here for tracking.
2) Scanning Helmfiles
As seen in Part 1 of this article, GitLab natively supports security scanning of Helm Charts. For the declarative deployment of Helm Charts with Helmfile , this support is missing. However, to still be able to check Helmfile sources for vulnerabilities, a small additional step is sufficient. Let’s first look at the directory structure:
1. 2├─ .gitlab-ci.yml 3├─ helmfile 4 └── vuln-project 5 ├── data 6 │ ├── helmchart 7 │ │ └── vuln-project 8 │ └── helmfile 9 │ ├── helmfile.d 10 │ │ ├── 00-namespaces.yaml 11 │ │ ├── 10-network-policies.yaml 12 │ │ └── 20-vuln-project-install.yaml 13 │ ├── helmfile.yaml 14 │ └── values 15 │ └── vuln-project.yaml.gotmpl 16 └── env 17 ├── prod 18 └── test
Again the root directory contains the
.gitlab-ci.yml file. The
helmfile.yaml file contains the inclusion of the various stages from the
env directory and defines which objects are installed. In order for SAST to have files for a security analysis, Helmfile must first be executed. This happens in the
.gitlab-ci.yml in an additional job
build_helm_manifests (line 5-16). This job creates a directory
$CI_PROJECT_DIR/k8s-manifests and updates the helm repos used. Subsequently,
helmfile template --output-dir $CI_PROJECT_DIR/k8s-manifests generates YAML manifests (lines 10-12). In order for these generated files to be passed to the
kubesec-sast job for analysis, these manifests are stored in an artifact (lines 14-16). In order for the
kubesec-sast job to access the artifacts, it is given a dependency on the build job (lines 19-21). Finally, the kubesec analyser must be activated with
SCAN_KUBERNETES_MANIFESTS: "true" (line 29) and by inserting the SAST job
.gitlab-ci.yml (line 24+25).
1stages: 2 - build 3 - test 4 5build_helm_manifests: 6 stage: build 7 image: 8 name: image.registry/k8s-cicd-tools-image:0.0.1 9 script: 10 - mkdir -p $CI_PROJECT_DIR/k8s-manifests 11 - helmfile --environment test -f $CI_PROJECT_DIR/helmfile/vuln-project/data/helmfile/helmfile.yaml repos 12 - helmfile --environment test -f $CI_PROJECT_DIR/helmfile/vuln-project/data/helmfile/helmfile.yaml template --output-dir "$CI_PROJECT_DIR/k8s-manifests" 13 artifacts: 14 paths: 15 - $CI_PROJECT_DIR/k8s-manifests 16 expire_in: 10 minutes 17 18# Extend job kubesec-sast by dependency to pass artifacts 19kubesec-sast: 20 dependencies: 21 - build_helm_manifests 22 23# Enable SAST scanning 24include: 25 - template: Security/SAST.gitlab-ci.yml 26 27# Enable manifest scanning 28variables: 29 SCAN_KUBERNETES_MANIFESTS: "true"
The pipeline starts as soon as the code is pushed to GitLab. As expected, two jobs are now running one after the other to create the artifact and to run the security scanning procedures.
After the security scan has been completed, the findings can be viewed in the vulnerability report. Here it is possible to apply various filters to the results and sort them if necessary. Details about vulnerabilities can be displayed via links within the findings.
GitLab can also scan deployments to Kubernetes for security best practices using tolls such as Helm Charts or even Helmfile. If vulnerabilities are discovered, they can be fixed before deployment.
Your job at Codecentric?
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.