ArgoCD backup cronjob — DRP

Photo by Kelly Sikkema on Unsplash

When shit will hit the fan, you better be ready. Having a disaster recovery plan will save your ass one day in the future. For example, what will happen if someone will accidentally delete your k8s cluster ? If your organization is using GitOps deployment style, which by itself is a DRP friendly way of deploying stuff, ( all k8s/helm yaml files are safely stored in git ). And ArgoCD applies those files on the cluster for you. Argo is another thing that might get screwed / deleted. It stores its state in k8s config maps , secrets and CRD’s called argocd-apps. Backing them up will enable you to restore argo and all apps installed by it with a single command.

In this post I will describe how to setup a cronjob that will backup argo daily on GCP. We will get into k8s Service Accounts, linking GCP SA to KSA to allow the backup pod to have access to both k8s config maps and google cloud storage. ( Sorry EKS users … )

argocd-utils is a handy tool made by argo team that can export/import state to/from a yaml file . Its not bundled with argocd binary, and in order to run it locally you will need a docker image.

running locally

In order for it to work localy you need to have a kubeconfig with access to the cluster argocd installed on.

This was easy… now lets put it in a cronjob and run daily ! To achieve that we will first need a Dockerfile that will extend argocd image and install gcloud cli on it.

installing gcloud cli on argocd image

A small comment about the above, ArgoCD image is based on Ubuntu, and USER root is required to run apt.

the backup script

Notice there is nothing related to authentication in the backup script, We will use a nifty way to allow our backup pod to be authorized to access both k8s api and GCS, called GCP workload identity. In short. We will create a k8s Service Account and bind it to a role that has access to k8s api, and we will also create a GCP service account with admin access to the GCS bucket that will contain our backups. And bind the KSA to GSA to allow the pod identify with both Service Accounts as one.

binding GSA to KSA

Lets break down the above:

• create GCS bucket
• create a GCP service account
• Assign the GCP SA objectAdmin role on the bucket
• Bind the GCP SA to the KSA SA ( k8s service account )
• Test assignment worked

Before you can test the magic works, you first need to create a k8s SA and annotate it with the same name of the GCP SA.

sa+role+tolebinding

The important prats above are the ClusterRole rules that give argocd-util access to relevant k8s api, and the ServiceAccount annotation witch will be used by GCP workload identity to link the KSA to the GSA we created earlier.

If everything was done correctly till this point, you should be able to run the Test command and list the empty GCS bucket from the test pod.

Last thing is of course linking the CronJob.yaml to the KSA , and tell it to run daily .

# cronjob.yaml
...
schedule: "0 0 * * *"
...
serviceAccount: stg-argocd-backup-eu-west1

And we are done !

To recover ArgoCD from the backup run:

argocd-util -n argocd import < /backup/backup.yaml

If this article helped you leave, me a comment.

And don’t forget to water the cactus.

Cheers.