Home Run self-hosted CI/CD agents on Azure Kubernetes Service - Part 3 - GitHub Actions
Post
Cancel

Run self-hosted CI/CD agents on Azure Kubernetes Service - Part 3 - GitHub Actions

Posted Jan 25, 2023 Updated Feb 5, 2023
By Adam Kielar
9 min read

This post is a continuation of our journey with self-hosted CI/CD agents. I encourage you to check part 1 and part 2 if you want to see a different approach to that topic. In this post, we will focus on GitHub Actions.

GitHub-hosted vs self-hosted runners

The documentation says that GitHub-hosted runners offer a simple way to run workflows, while self-hosted runners are a great solution if you want to create a more configurable set up in your own environment.

GitHub-hosted runners:

  • Receive automatic updates for the operating system and tools.
  • Are managed and maintained by GitHub.
  • Provide a clean instance for every job execution.
  • Use free minutes on your GitHub plan, with per-minute rates applied after surpassing the free minutes.

Self-hosted runners:

  • Receive automatic updates for the self-hosted runner application only.
  • You are responsible for updating the operating system and all other software.
  • Can use cloud services or local machines that you already pay for.
  • Are customizable to your hardware, operating system, software, and security requirements.
  • Don’t need to have a clean instance for every job execution.
  • Are free to use with GitHub Actions, but you are responsible for the cost of maintaining your runner machines.

Overview

We will create and configure the following resources:

  • AKS cluster with workload identity and kubelet identity.
  • Azure Container Registry.
  • Role assignment for kubelet identity and workload identity.
  • Kubernetes objects for workload identity and GitHub runner.

We will also check how to se up a runner with actions runner controller.

Here is a link to GitHub repo with all files for reference.

Video walkthrough

Create AKS cluster and ACR

  1. Run script.
    1
2
3

    # After running above script, if there were no errors, variables should be available in terminal.
chmod +x aks.sh
./aks.sh 'add user id, for me, it is my email of AAD user'
  2. Get credentials to AKS, oidcUrl and test connection.
    1
2
3
4
5

    az aks get-credentials --resource-group $resourceGroup --name $aksName
export oidcUrl="$(az aks show --name $aksName \
--resource-group $resourceGroup \
--query "oidcIssuerProfile.issuerUrl" -o tsv)"
kubectl get nodes

Set up workload identity

  1. Create workload identity.
    1
2
3
4
5

    workloadIdentity="workload-identity"
workloadClientId=$(az identity create --name $workloadIdentity \
--resource-group $resourceGroup --query clientId -o tsv)
workloadPrincipalId=$(az identity show --name $workloadIdentity \
--resource-group $resourceGroup --query principalId -o tsv)
  2. Create Kubernetes Service Account in GitHub namespace.
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

    # github-runner-sa.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: github
  labels:
 name: github
---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
 azure.workload.identity/client-id: <insert workloadClientId> # echo $workloadClientId
  labels:
 azure.workload.identity/use: "true"
  name: workload-sa
  namespace: github

    1

    kubectl apply -f github-runner-sa.yaml
  3. Create ClusterRole and ClusterRoleBinding for Service Account. Update workloadPrincipalId in file.
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

    # github-roles.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
 rbac.authorization.kubernetes.io/autoupdate: "true"
  name: github
rules:
- apiGroups:
  - '*'
  resources:
  - statefulsets
  - services
  - replicationcontrollers
  - replicasets
  - podtemplates
  - podsecuritypolicies
  - pods
  - pods/log
  - pods/exec
  - podpreset
  - poddisruptionbudget
  - persistentvolumes
  - persistentvolumeclaims
  - endpoints
  - deployments
  - deployments/scale
  - daemonsets
  - configmaps
  - events
  - secrets
  verbs:
  - create
  - get
  - watch
  - delete
  - list
  - patch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
 rbac.authorization.kubernetes.io/autoupdate: "true"
  name: github
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: github
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:github
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: # echo $workloadPrincipalId

    1

    kubectl apply -f github-roles.yaml
  4. Create federated identity credentials.
    1
2
3
4
5
6

    az identity federated-credential create \
--name "aks-federated-credential" \
--identity-name $workloadIdentity \
--resource-group $resourceGroup \
--issuer "${oidcUrl}" \
--subject "system:serviceaccount:github:workload-sa"
  5. Create custom role for workload identity. Create acrbuild.json file with following definition. Replace {YOUR SUBSCRIPTION} with your subscription id.
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

    {
  "Name": "AcrBuild",
  "IsCustom": true,
  "Description": "Can read, push, pull and list builds.",
  "Actions": [
 "Microsoft.ContainerRegistry/registries/read",
 "Microsoft.ContainerRegistry/registries/pull/read",
 "Microsoft.ContainerRegistry/registries/push/write",
 "Microsoft.ContainerRegistry/registries/scheduleRun/action",
 "Microsoft.ContainerRegistry/registries/runs/*",
 "Microsoft.ContainerRegistry/registries/listBuildSourceUploadUrl/action"
  ],
  "AssignableScopes": [
 "/subscriptions/{YOUR SUBSCRIPTION}"
  ]
}

    1

    az role definition create --role-definition acrbuild.json
  6. Assign AcrBuild role to workload identity.
    1
2

    az role assignment create --assignee $workloadClientId \
--role 'AcrBuild' --scope $acrId
  7. Assign Azure Kubernetes Service Cluster User Role and Azure Kubernetes Service RBAC Writer to workload identity.
    1
2
3
4

    az role assignment create \
--role "Azure Kubernetes Service Cluster User Role" \
--assignee $workloadPrincipalId \
--scope $aksId

    1
2
3
4

    az role assignment create \
--role "Azure Kubernetes Service RBAC Writer" \
--assignee $workloadPrincipalId \
--scope "$aksId/namespaces/github"

Install GitHub Actions self-hosted runner

GitHub create runner

Create docker image for GitHub runner.

We will use information from above instructions. We will add az cli, kubectl. Size of image is far from perfect. Build and push it to our ACR.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

FROM ubuntu:22.04
USER root
RUN apt-get -y update && apt-get install -y curl && \
    curl -sL https://aka.ms/InstallAzureCLIDeb | bash && az aks install-cli && \
    curl -fsSL https://get.docker.com -o get-docker.sh && sh ./get-docker.sh && \
    mkdir actions-runner && cd actions-runner && \
    curl -o actions-runner-linux-x64-2.301.1.tar.gz -L https://github.com/actions/runner/releases/download/v2.301.1/actions-runner-linux-x64-2.301.1.tar.gz && \
    tar xzf ./actions-runner-linux-x64-2.301.1.tar.gz && ./bin/installdependencies.sh && \
  apt-get clean

RUN addgroup --gid 106 github && adduser github --uid 105 --system && adduser github github && \
  chown -R github:github actions-runner

USER github

EXPOSE 8080

1

az acr build -f Dockerfile.runner -t github-runner:v1.0.0 -r $acrName -g $resourceGroup .

Create Storage Class and Persistent Volume Claim

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# github-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: github-azurefile
provisioner: file.csi.azure.com
mountOptions:
  - uid=105
  - gid=106
allowVolumeExpansion: true
volumeBindingMode: Immediate
reclaimPolicy: Delete
parameters:
  skuName: Standard_LRS

1
2
3
4
5
6
7
8
9
10
11
12
13

# github-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: github-pvc
  namespace: github
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: github-azurefile

1
2

kubectl apply -f github-storageclass.yaml
kubectl apply -f github-pvc.yaml

Create StatefulSet and Service for runner

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

apiVersion: v1
kind: Service
metadata:
  name: github-runner
  namespace: github
  labels:
    app: github-runner
spec:
  ports:
  - port: 8080
    name: github-runner-port
  clusterIP: None
  selector:
    app: github-runner
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: github-runner
  namespace: github
spec:
  replicas: 1
  minReadySeconds: 10
  serviceName: github-runner
  selector:
    matchLabels:
      app: github-runner
      azure.workload.identity/use: "true"
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Delete
  template:
    metadata:
      labels:
        app: github-runner
        azure.workload.identity/use: "true"
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: workload-sa
      containers:
      - image: githubacr14149.azurecr.io/github-runner:v1.0.0 # Add your ACR respository
        name: github-runner
        imagePullPolicy: Always
        command:
        - sleep
        args:
        - 99d
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        volumeMounts:
          - mountPath: "/home/github"
            name: runner-data
      volumes:
        - name: runner-data
          persistentVolumeClaim:
            claimName: github-pvc

1

kubectl apply -f statefulset.yaml

Set up runner inside pod

  1. Check if pod is running and connect to container.
1
2

kubectl -n github get pods
kubectl -n github exec -it github-runner-0 -- sh
  1. Configure runner
    1

    ./actions-runner/config.sh --url https://github.com/adamkielar/github-runner --token <YOU TOKEN>

    Github runner 2

  2. Start runner
    1

    ./actions-runner/run.sh

    Github runner 3

Create GitHub workflow

  1. In .github/workflows create deploy-app.yaml.

Update name of ACR registry and optionally AKS name and resource group.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# deploy-app.yaml
name: Build image, push to ACR, deploy to AKS
concurrency: 
  group: deployment

on:
  push:
  workflow_dispatch:

jobs:
  deploy:
    name: Deploy application
    runs-on: self-hosted
    steps:
      - name: Checkout repo
        uses: actions/checkout@v2
      
      - name: Login with workload identity
        run: az login --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --federated-token $(cat $AZURE_FEDERATED_TOKEN_FILE)

      - name: Build image with AZ ACR
        run: |
          az acr build -f /actions-runner/_work/github-runner/github-runner/Dockerfile -t githubacr14149.azurecr.io/github-demo:$ -r githubacr14149 .
      
      - name: Get AKS credentials
        run: |
          az aks get-credentials --resource-group github-runner-rg --name aks-github-runner --overwrite-existing
          kubelogin convert-kubeconfig -l workloadidentity

      - name: Deploy application
        run: |
          sed 's|IMAGE|githubacr14149.azurecr.io/github-demo|g; s/TAG/$/g' /actions-runner/_work/github-runner/github-runner/pod.yaml | kubectl apply -f -
  1. Push changes to GitHub and check status of each resource.

Job output

GitHub output

Kubectl output

Set up self-hoster runner with Actions Runner Controller

Let’s follow the documentation and see if the steps are straightforward.

  1. Install cert-manager.
    1

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
  2. Generate a Personal Access Token in GitHub for your repo.
    • repo (all)
    • admin:public_key - read:public_key
    • admin:repo_hook - read:repo_hook
    • admin:org_hook
    • notifications
    • workflow
  3. Deploy controller using helm.
    1

    helm repo add actions-runner-controller https://actions-runner-controller.github.io/actions-runner-controller

    1
2
3
4

    helm upgrade --install --namespace actions-runner-system --create-namespace\
  --set=authSecret.create=true\
  --set=authSecret.github_token="REPLACE_YOUR_TOKEN_HERE"\
  --wait actions-runner-controller actions-runner-controller/actions-runner-controller
  4. Create runnerdeployment.yaml.
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

    # runnerdeployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-runnerdeploy
  namespace: github
spec:
  replicas: 1
  selector:
 matchLabels:
   app: github-runner-v2
  template:
 metadata:
   labels:
     app: github-runner-v2
     azure.workload.identity/use: "true"
 spec:
   repository: adamkielar/github-runner # change to your repository
   labels:
     - github-runner
   ephemeral: true
   serviceAccountName: workload-sa

    1

    kubectl apply -f runnerdeployment.yaml
  5. Confirm pods are running. Controller pods
  6. Create .github/workflow/deploy-app-v2.yaml file. We will modify our pipeline a bit. We will add install_libs script.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

name: Build image, push to ACR, deploy to AKS
concurrency: 
  group: deployment

on:
  push:
  workflow_dispatch:

jobs:
  deploy:
    name: Deploy application
    runs-on: github-runner
    steps:
      - name: Checkout repo
        uses: actions/checkout@v2
      
      - name: Install Azure CLI, kubectl
        run: chmod +x install_libs.sh && ./install_libs.sh

      - name: Install kubelogin
        run: sudo az aks install-cli
      
      - name: Login with workload identity
        run: az login --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --federated-token $(cat $AZURE_FEDERATED_TOKEN_FILE)

      - name: Build image with AZ ACR
        run: |
          az acr build -f /runner/_work/github-runner/github-runner/Dockerfile -t githubacr14149.azurecr.io/github-demo:$ -r githubacr14149 .
      
      - name: Get AKS credentials
        run: |
          az aks get-credentials --resource-group github-runner-rg --name aks-github-runner --overwrite-existing
          export KUBECONFIG=/home/runner/.kube/config
          kubelogin convert-kubeconfig -l workloadidentity

      - name: Deploy application
        run: |
          sed 's|IMAGE|githubacr14149.azurecr.io/github-demo|g; s/TAG/$/g' /runner/_work/github-runner/github-runner/pod.yaml | kubectl apply -f -
  1. Delete github-demo-pod.
    1

    kubectl -n github delete pod github-demo-pod
  2. Push changes to GitHub and check result.

GitHub deploy

We successfully managed to deploy our application.

azure aks github kubernetes
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents