Low-Cost Kubernetes on DigitalOcean

In a previous post I was able to decrease my Azure AKS cost to ~$66AUD/month. In this post I’ll migrate to Kubernetes on DigitalOcean in an attempt to reduce costs considerably more.

Resource Comparison

For both providers I’ve gone single node, lowest compatible VM. On Azure the Basic Load Balancer is free, but a Small Load Balancer is 10USD on DigitalOcean and required only if you wish to use an Ingress Controller.

DigitalOcean’s price is 10USD/mo for the Droplet, 10USD/mo for the Load Balancer and 0.8/mo for the persistent volume. 20.80USD/mo equates to ~27AUD/mo. Both Azure and DigitalOcean offer the control plane for free.

Migrating from AKS to DigitalOcean

I’ve only got a few things in my cluster:

• Digital Icebreakers
• PostgreSQL & Adminer
• Some twitter bots and other CronJobs

Digital Icebreakers is built and deployed via Azure Pipeline so it was just a matter of changing the Kubernetes Service Connection in Azure DevOps and re-deploying.

I use nginx-ingress and Let’s Encrypt to terminate TLS. Here’s a great tutorial on implementing this in your cluster. I have my Ingress and Issuer in source, so for me it came down to:

# install ingress controller
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx

# install cert-manager
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.1.0 \
  --set installCRDs=true

# deploy issuer & ingress
kubectl apply -f cluster-issuer-letsencrypt-prod.yml
kubectl apply -f ingress.yml

I have an existing postgres backup CronJob which can be executed immediately by using --from syntax:

kubectl create job --from=cronjob/my-postgres-backup backup-now

During migration I halted all my CronJobs by deleting them via script in bulk from the old cluster:

#!/usr/bin/env bash

kubectl get cronjob --all-namespaces | sed '1d' | awk '{ print $2, "--namespace", $1 }' | while read line; do
  echo "Running with: ${line}"
  kubectl delete cronjob ${line} &
  sleep 0.05
done

I have the CronJobs in a single folder in source so I can apply them quickly against the new cluster:

kubectl apply -f cron-job-folder

A number of secrets needed migrating and can be done directly between clusters using --context syntax:

kubectl get secret my-secret --context sourceCluster -o yaml | kubectl apply --context destCluster -f -

Getting Postgres and Adminer up in the new cluster is simple using helm:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add cetic https://cetic.github.io/helm-charts                                                                                             
helm repo update
helm install postgres bitnami/postgresql
helm install --set service.type=ClusterIP adminer cetic/adminer

Conclusion

DigitalOcean’s pricing is less than half of Azure’s for similar functionality. As their product offering is much less than Azure’s, I find navigating their website and using their CLI much simpler than Azure. Migration between clusters is simplified by using helm and having most of my kubernetes resources defined in source. Those that aren’t in source (like secrets), can be migrated directly between clusters with shell commands. Latency however is slightly increased due to the move from Sydney to Singapore datacenters.