Run self-hosted CI/CD agents on Azure Kubernetes Service - Part 1 - CircleCI + Buildah

We use self-hosted CI/CD when we want to run and manage our own continuous integration and delivery (CI/CD) pipeline, rather than using a cloud-based service like CircleCI, Azure DevOps, or GitHub Actions. One way to do this is to host the agents on a Kubernetes cluster, which can provide scalability and resource isolation for your build processes. By self-hosting our CI/CD agents on Kubernetes, we have greater control over the infrastructure and can customize the build process to fit our specific needs. However, it also requires more setup and maintenance effort on our part.

Overview

We will:

Create AKS cluster with workload identity enabled. We touched on that topic in this post so please take the time to read it if you are not familiar with this setup.
Create Azure Container Registry to store our images
Create Persistent Volume Claim
Create CircleCI Agent and simple config

Video walkthrough

Prerequisites

Kubernetes 1.12+
Helm 3.x
Token for a resource class
jq tool if you want to run all commands
Dockerfile

Create an AKS cluster with workload identity

I created script with necessary commands to provision a basic setup. This is the same setup as in last post. Let’s run it.

chmod +x aks.sh
./aks.sh 'add user id, for me, it is my email of AAD user'

After running above script, if there were no errors, variables should be available in terminal. If you will have any problem running it, let me know and I will try to help. I use zsh terminal.

Create a user-assigned managed identity for workload identity

  
clientId=$(az identity create --name $workloadIdentity --resource-group $resourceGroup --query clientId -o tsv)

Grant permission to access the secret in Key Vault

  
az keyvault set-policy --name $kvName \
--secret-permissions get \
--spn $clientId

Create Azure Container Registry

We will create ACR and we will enable admin login. We will add these credentials to Key Vault. Later in this post, we will push an image to ACR using Buildah which does not work currently with Azure Managed Identity setup without Docker.

  
acrName='circleciacr'
acrId=$(az acr create --name $acrName --resource-group $resourceGroup --sku Basic --admin-enabled --query id -o tsv)

Get credentials for ACR and add them to Key Vault

  
ACRUSER=$(az acr credential show -n $acrName | jq -r .username)
ACRPASSWORD=$(az acr credential show -n $acrName | jq -r '.passwords | .[0].value')
az keyvault secret set --vault-name $kvName \
--name "ACRUSER" \
--value $ACRUSER
az keyvault secret set --vault-name $kvName \
--name "ACRPASSWORD" \
--value $ACRPASSWORD

Create SecretProviderClass

We need this object if we want to access Key Vault from AKS cluster.

Create file secretprovider.yaml and add missing variables and apply.

  
apiVersion: v1
kind: Namespace
metadata:
  name: circleci
  labels:
    name: circleci
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aks-kv-workload-identity
  namespace: circleci
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"          
    clientID: # az identity show --name $workloadIdentity --resource-group $resourceGroup --query clientId -o tsv
    keyvaultName: # echo $kvName
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: ACRUSER
          objectType: secret
          objectVersion: ""
        - |
          objectName: ACRPASSWORD
          objectType: secret
          objectVersion: ""
    tenantId: # az account show --query tenantId -o tsv

kubectl apply -f secretprovider.yaml

Create Service Account, Role and Rolebinding for CircleCI agent

Create file sa.yaml, add missing variable and apply.

  
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id:  # az identity show --name $workloadIdentity --resource-group $resourceGroup --query clientId -o tsv
  labels:
    azure.workload.identity/use: "true"
  name: circleci-sa
  namespace: circleci
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: circleci
  name: circleci-role
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec"]
  verbs: ["get", "watch", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["watch"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: circleci-rolebinding
  namespace: circleci
subjects:
- kind: ServiceAccount
  name: circleci-sa
  namespace: circleci
roleRef:
  kind: Role
  name: circleci-role
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f sa.yaml

Create federated identity credentials

  
az identity federated-credential create \
--name "aks-federated-credential" \
--identity-name $workloadIdentity \
--resource-group $resourceGroup \
--issuer "${oidcUrl}" \
--subject "system:serviceaccount:circleci:circleci-sa"

Setup CircleCI

Enable Self-Hosted Runners in CircleCI and create resource class. During that process You will optain access token. Follow documentation.

Create secret object with CircleCI token

  
kubectl -n circleci create secret generic circleci-token-secrets --from-literal=circleci-runner.resourceClass='ADD TOKEN'

Create Persistent Volume Claim

We will use storage class that is already created in AKS: azurefile-csi.

Create file pvc.yaml and apply.

  
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-circleci
  namespace: circleci
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: azurefile-csi

kubectl apply -f pvc.yaml

Create CircleCI agent

Install fuse-overlay program according to documentation.

We will create custom values.yaml file.

  
agent:
  customSecret: circleci-token-secrets
  serviceAccount.create: false
  rbac.create: false
  resourceClasses:
   circleci-runner/resourceClass:
    metadata:
    annotations:
      custom.io: circleci-runner
    spec:
      serviceAccountName: circleci-sa
      containers:
        - resources:
            limits:
              github.com/fuse: 1
          volumeMounts:
            - name: agent-store
              mountPath: /home/build/
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true
          securityContext:
            privileged: true
            runAsUser: 1000
      volumes:
        - name: agent-store
          persistentVolumeClaim:
            claimName: pvc-circleci
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aks-kv-workload-identity"

Install agent with Helm.

  
helm install container-agent container-agent/container-agent --namespace circleci -f values.yaml

Confirm if deployment is created.

Create config.yaml for CircleCI pipeline

  
version: 2.1

jobs:
  build-push:
    docker:
      - image: quay.io/buildah/stable:v1.28.0
    resource_class: azuretour/aksrunner
    steps:
      - checkout
      - run:
          name: Build and Push
          command: |
            ACRUSER=$(cat /mnt/secrets-store/ACRUSER)
            ACRPASSWORD=$(cat /mnt/secrets-store/ACRPASSWORD)
            buildah bud --format docker --layers -f ./Dockerfile -t circleciacr.azurecr.io/demo-image:v1.0.0 .
            buildah push --creds=${ACRUSER}:${ACRPASSWORD} circleciacr.azurecr.io/demo-image:v1.0.0
workflows:
  build-push-workflow:
    jobs:
      - build-push

Push our changes to the repository and check the job result.

Confirm that pod for circleci job is created.

Confirm that CircleCI job is complete.

Confirm that image is in ACR.

According to circleci documentation we can add `runAsNonRoot: true ` to `securityContext` but during my tests I was unable to mount fuse device plugin without --priviliged flag in AKS with default settings. We need to be able to modify pod that circle agent creates.

Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied

Here are great articles from RedHat for further reading:

Conclusion

I would like to have more control over pod which is created by circleci-agent. This solution is not flexible enough for me yet and there are security concerns.