Manage a secure AKS Cluster with Security Tower & GitOps

Introduction

In this tutorial you will learn:

  • How to setup AKS using CLI
  • Prepare your GitHub Repository and install Security Tower
  • Install a GitOps Tool (Flux or ArgoCD)
  • Deploy OPA Gatekeeper
  • Create an application and see how policies are working

After completing this tutorial you will understand the basic principles of GitOps and combine them with highest security principles using policies managed by Security Tower.

Prerequisites

To successfully complete this tutorial, the following requirements are necessary:

  1. Microsoft Azure Account
  2. GitHub Account
  3. Security Tower CLI installed
  4. Azure CLI installed

Step 1: Create the AKS Cluster

To get started you need to login into your Azure Account using the Azure CLI

az login

Set your Subscription you want to use

az account set -s <subscription-name>

Verify the current deployments of your selected subscription

az resource list

Create a resource group you want to deploy your AKS Cluster

az group create --name myAKS --location germanywestcentral

Change the Azure location if required.

Create the AKS Cluster

az aks create --resource-group myAKS \
  --name myAKSCluster1 \
  --node-count 1 \
  --enable-addons monitoring \
  --generate-ssh-keys

Connect to your Cluster

If not done yet, you can install kubectl with this command:

az aks install-cli

Get the configuration for the AKS cluster by running:

az aks get-credentials --resource-group myAKS --name myAKSCluster1

And verify your connection:

$ kubectl get nodes
NAME                                STATUS   ROLES   AGE     VERSION
aks-nodepool1-16727520-vmss000000   Ready    agent   3h32m   v1.19.9

Step 2: Set up Security Tower

  1. Go to Syncier Security Tower on the GitHub Marketplace
  2. Select an appropriate plan and click the green install button and then confirm your choice on the next screen
  3. Select the repository you created and click "Install"
  4. After you've been redirected to Syncier Security Tower, click on "Authorize" to log in with your GitHub account

Step 3: Prepare the GitHub repository

Create a new repository aks-gitops on GitHub. We will use this repository to store all manifests which are going to be deployed to the AKS cluster.

Add a deploy key for the repository on GitHub

Generate a SSH key using the following command:

ssh-keygen -t ed25519 -C '<your github email address>'

Go to https://github.com/<github account>/aks-gitops/settings/keys and click "Add deploy key". Add the public key you just created, and save the private key file for later.

Please see the GitHub documentation for more details.

Create GitOps directory layout

Now we will prepare the GitHub repository so we can use it for GitOps. We will use a simplified version of our recommended layout in this demonstration. See the template repository on GitHub for a complete example.

Clone the repository and change into this directory:

git clone https://github.com/<github account>/aks-gitops
cd aks-gitops

We need a directory where the policies created by Security Tower can be stored:

mkdir -p cluster/global/policies
touch cluster/global/policies/.gitkeep

Next let's create a demo application. It contains a nginx webserver which is exposed via a LoadBalancer service.

curl --fail --create-dirs -so cluster/namespaces/demo-app/hello-world.yaml \
  https://raw.githubusercontent.com/securitytower/downloads/main/examples/hello-world.yaml

Commit and push the changes:

git add cluster
git commit -m "Initialize with GitOps layout"
git push

Add the Security Tower configuration files

Let's tell Security Tower about our cluster and where the applications are deployed. Inside the repository, run the following command to add the cluster configuration:

securitytower cluster add --name=cluster1 --policies=cluster/global/policies

Commit all changes and push them to GitHub.

git add .securitytower
git commit -m "Add cluster configuration"
git push

Add now the application configuration:

securitytower application add --name policies
securitytower application stage add --name production \
  --app policies --cluster cluster1 --namespace policies --path cluster/global/policies

securitytower application add --name demo-app
securitytower application stage add --name production \
  --app demo-app --cluster cluster1 --namespace demo-app --path cluster/namespaces/demo-app

Commit all changes and push them to GitHub.

git add .securitytower
git commit -m "Add Security Tower configuration"
git push

If you have done everything correctly, your cluster should now appear in the list on https://app.securitytower.io!

Step 3: Install a GitOps Tool

You have two options in this guide:

  1. Managing your cluster using Argo CD (https://argoproj.github.io/argo-cd/)
  2. Managing your cluster using Flux (https://fluxcd.io/)

Step 3a: Argo CD

Install Argo CD into your cluster

First create a namespace for Argo CD

kubectl create namespace argocd

Then install the Argo CD resources into your cluster:

kubectl apply \
  -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Check if the installation was successful:

$ kubectl get pods -n argocd
NAMESPACE           NAME                                             READY   STATUS    RESTARTS   AGE
argocd              argocd-application-controller-0                  1/1     Running   0          3h28m
argocd              argocd-dex-server-5dd657bd9-kvrzx                1/1     Running   0          3h28m
argocd              argocd-redis-759b6bc7f4-pdnxt                    1/1     Running   0          3h28m
argocd              argocd-repo-server-6c495f858f-8hbgt              1/1     Running   0          3h28m
argocd              argocd-server-859b4b5578-rtpmp                   1/1     Running   0          3h28m

Expose the endpoint of the app, here we are using a LoadBalancer:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

To get your external IP address you can use following command:

$ kubectl describe service argocd-server -n argocd | grep Ingress
LoadBalancer Ingress:     20.79.97.98

Next get the initial password for Argo CD:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" \
  | base64 -d

With the result you can can login to Argo CD via web UI and the command line. Please ensure that the Argo CD CLI is installed. The username is admin.

argocd login <ip-address>

Finally add the cluster to Argo CD

argocd cluster add myAKSCluster1

Connect Argo CD to the GitHub repository

Argo CD needs access to the repository on GitHub. Now we need the SSH key you created in step 2:

argocd repo add git@github.com:<github account>/aks-gitops \
  --ssh-private-key-path <private key file>

Remember the demo application we created in cluster/namespaces/demo-app? We need to tell Argo CD about this application so it can deploy it to the cluster:

argocd app create demo-app \
  --repo git@github.com:<github account>/aks-gitops \
  --path cluster/namespaces/demo-app \
  --dest-namespace=demo-app \
  --directory-recurse \
  --sync-option CreateNamespace=true \
  --sync-policy automated \
  --dest-server https://kubernetes.default.svc

Wait for the demo-app to get ready. You can check the Argo CD web interface for details about the status in case something goes wrong.

argocd app wait demo-app

Step 3b: Flux

Install Flux 2 in the cluster:

flux bootstrap github \
   --owner=<github account> \
   --repository=aks-gitops \
   --personal \
   --path=cluster/global/flux

# pull the changes flux made to the repository
git pull

Remember the demo application we created in cluster/namespaces/demo-app? We need to tell Flux about this application so it can deploy it to the cluster.

# flux will not create the namespace for us
kubectl create namespace demo-app

flux create kustomization demo-app \
    --target-namespace=demo-app \
    --source=flux-system \
    --path="./cluster/namespaces/demo-app" \
    --prune=true \
    --validation=none \
    --export > ./cluster/global/flux/demo-app.yaml

git add cluster/global/flux/demo-app.yaml
git commit -m "Deploy demo app with flux"
git push

Check the output of flux get kustomization:

$ flux get kustomization
NAME                READY   MESSAGE                                                             REVISION                                        SUSPENDED
demo-app            True    Applied revision: master/4e6b29611e87127627c814454dea962c68598cf3   master/4e6b29611e87127627c814454dea962c68598cf3 False
flux-system         True    Applied revision: master/4e6b29611e87127627c814454dea962c68598cf3   master/4e6b29611e87127627c814454dea962c68598cf3 False

In case the demo-app does not appear, you can force a refresh with the following command:

flux reconcile kustomization flux-system

Check the demo application

Finally we can get the endpoint exposed by the demo-app and see if we can reach it:

kubectl describe service hello-world -n demo-app | grep Ingress
curl -v http://<ip-address>:80

Step 5: Activate a policy

Now we are going to activate a security policy and see how Security Tower provides pull request feedback.

  1. Go to https://app.securitytower.io
  2. Click on your cluster in the list
  3. Open the "Policies" tab
  4. Click on "Edit Policies"
  5. Find the "EnforceImageVersion" policy in the list, and change the status to "Active"
  6. Click on "Update Policies"
  7. A pull request with the policy definition files will now be created by Security Tower in the repository. Merge the pull request on GitHub.

The policy we just activated enforces the best practice of specifying an explicit image version for all Pods in the cluster. Now let's create a manifest which violates this policy and open up a pull request via the official GitHub CLI:

git checkout -b violation-test
curl --fail -so cluster/namespaces/demo-app/nginx-latest.yaml \
    https://raw.githubusercontent.com/securitytower/downloads/main/examples/nginx-latest.yaml
git add cluster
git commit -m "Test violation"
gh pr create --web

Security Tower will detect the violation, and provide feedback directly on GitHub. Even with the failed check, you can still decide to merge the pull request. To enforce the policies by making the check mandatory, please follow these instructions.

Step 6: Deploy OPA Gatekeeper

All resources added to the GitOps repository are guarded by Security Tower. But what happens if someone manually creates a resource via kubectl? To establish another line of defense, install OPA Gatekeeper with this command:

securitytower policies gatekeeper install --path=cluster/namespaces/gatekeeper-system
git add cluster/namespaces/gatekeeper-system
git commit -m "Install gatekeeper"
git push

Argo CD

If you are using Argo CD, create an Argo CD app to deploy the resources:

argocd app create gatekeeper-system \
    --repo git@github.com:<github account>/aks-gitops \
    --dest-namespace gatekeeper-system \
    --path cluster/namespaces/gatekeeper-system \
    --sync-policy automated \
    --dest-server https://kubernetes.default.svc

Run the following command to wait for gatekeeper to get ready:

argocd app wait gatekeeper-system

Now we can create another Argo CD application which will deploy the activated policies in the form of Gatekeeper CRDs:

argocd app create policies \
    --repo git@github.com:<github account>/aks-gitops \
    --path cluster/global/policies \
    --directory-recurse \
    --sync-policy automated \
    --dest-server https://kubernetes.default.svc

Flux

Create the namespace and the kustomization for Gatekeeper with the flux CLI:

# flux will not create the namespace for us
kubectl create namespace gatekeeper-system

flux create kustomization gatekeeper-system \
    --target-namespace=gatekeeper-system \
    --source=flux-system \
    --path="./cluster/namespaces/gatekeeper-system" \
    --prune=true \
    --validation=none \
    --export > ./cluster/global/flux/gatekeeper.yaml

git add cluster/global/flux/gatekeeper.yaml
git commit -m "Deploy gatekeeper with flux"
git push

Now we can create another Flux kustomization which will deploy the activated policies in the form of Gatekeeper CRDs:

flux create kustomization policies \
    --target-namespace=policies \
    --source=flux-system \
    --path="./cluster/global/policies" \
    --prune=true \
    --validation=none \
    --export > ./cluster/global/flux/policies.yaml

git add cluster/global/flux/policies.yaml
git commit -m "Deploy policies with flux"
git push

Try it in action

Let's try to apply the nginx manifest from above with kubectl:

curl --fail -s https://raw.githubusercontent.com/securitytower/downloads/main/examples/nginx-latest.yaml \
    | kubectl apply -f -

If everything is configured correctly, you should now see an error message from Gatekeeper's kubernetes admission webhook that the requested changes have been denied.

Now we have two lines of defense for policy violations:

  1. Security Tower is checking on pull requests if the change is violating our defined policies
  2. Gatekeeper ensures during runtime that there are no policy violations

All done!

Congratulations, you have successfully completed this guide and deployed a secure cluster via GitOps on AKS!

If you would like to learn more about this setup, have a look at our reference architecture GitOps for Azure Kubernetes Service.