Highly available Pi-hole setup in Kubernetes with secure DNS over HTTPS DoH

Welcome! I've been a fan of Pi-hole for some time now. Not only is this an incredibly useful project, with great community support, but it was also my first Homelab project. With an undisclosed amount of time and money later, I'm still going strong with many projects and a rack full of hardware. For those in the know, having a Homelab looks a lot like a side gig as an IT consultant / Platform Engineer in my own home 😀. To add insult to injury, I don't get paid for this; it costs me money. But I digress.

Previously, I wrote an article about how to set up two or more Pi-hole instances using Docker Swarm, which has worked quite well for me up to this point. I've since decided this setup was too easy and wanted to move from Normal to Hard mode and bring Kubernetes into the mix. Not only that, but I wanted true high availability. It was hard at first, but once you cross that initial learning threshold and bring in some tools to help, you'll be glad you did. I'll assume you are new to Kubernetes, or at least new to running it on your own metal (like I was), so we'll start there.

Kubernetes with K3s

Install k3s w/ etcd, HAProxy, and keepalived to support high-availability

I've found this a dead simple, effective, and powerful way to start at home. K3s is an open-source, well-maintained, well-documented, compliant distribution of Kubernetes that is lightweight and designed to run on various hardware and environments...Including your old laptop or an ARM-based Raspberry Pi 4. I'm using a mix of Proxmox VMs running on some Dell hardware and 4 Raspberry Pi 4's running SSDs (not SD Cards). You can start like I did and use the k3s install script.

Prerequisites

1. At least 3 physical computers (recommended) or virtual machines with their own IP addresses. The more physical separation, the more resilient your setup will be. If you use Pi's for your k3s server nodes, you'll want to use USB SSDs or NVME; SD cards will be too slow using the embedded etcd DB.
2. Some familiarity with Linux, ideally Ubuntu, and comfortable running commands in your favorite terminal.
3. A text file, notes app, or something to keep track of commands, IP addresses, and configuration.

Let's install k3s

Check your cluster

Let's make sure your cluster is up and running. There are many ways to do this; primarily, you'll need the kubectl CLI, which is how you'll interface with k3s. Once that works, we can move to some cool visual tools. Install the following tools and copy the Kubernetes config you captured earlier (/etc/rancher/k3s/k3s.yaml on server-1) to your working directory or the default Mac directory ~/.kube/ with the filename config (no extension)

Now run kubectl get nodes and you should see an output like this:

HAProxy Load Balancer

Now that your cluster is highly available, meaning that you can still access your cluster if 1 of your 3 server nodes is down, we can move on to setting up our load balancer. We'll follow the docs for setting up HAProxy and layering in our specific node IP addresses established in the previous step.

TIP: use a separate VM's or physical computers that are running Ubuntu server or desktop, avoid using an esisting k3s server or agent nodes as they must have unique IP Addreses.

On the Ubuntu servers or VMs that will run your Load Balancer, install: sudo apt-get install haproxy keepalived

You'll also need to update your keepalived config on each LB server. This is what will make your HAProxy load balancer highly available under your VIP. On each server, edit the keepalive config: sudo nano /etc/keepalived/keepalived.conf Add the config below and save ctrl-x , then y.

For lb-1 192.168.1.20:

For lb-2 192.168.1.21:

Finally, restart the services sudo systemctl restart haproxy && sudo systemctl restart keepalived

Optional: Add additional agent nodes to your cluster

Pi-hole

Deploy Pi-hole w/ DoH support via helm

Okay, the hard part is over. k3s is up and running, and we're now ready to deploy Pi-hole, enterprise style! Thankfully, there is a strong community around Pi-hole, filled with homelabbers like us who have already built and actively support a way to do this easily via Helm. Think of Helm as a package manager for Kubernetes. Helm packages are called Charts, and we are now going to install one for Pi-hole, which you can check out here on GitHub.

First, let's add the chart repository so that we'll be able to deploy it using the helm command:

Before we install the chart, let's customize our installation. You can do this by setting/overriding the chart's available values. For this chart by mojo2600, you can see some recommended values in the README and the full values file in the chart here.

Start by creating your own values file in your working directory, called pi-hole-values.yaml. I'll break it down by section and then pull it all together at the end.

Configure DNS

We need our pi-hole DNS service available outside the cluster so that we can assign it via DHCP and use it to start blocking ads across the network.

Configure DoH

We want to ensure all of our upstream DNS requests use DNS over HTTPS (DoH) via a Cloudflare tunnel.

Configure Pi-hole web admin

We want to ensure we can access the web interface using a local IP Address or local DNS. This helps with viewing logs and manually running backups.

TIP: Consider using a persistent volume storage provider, like NFS or Longhorn, for your pi-hole volume. I highly recommend it, but I won't cover it in this walkthrough. The good news is that once you decide to add a PV storage provider to your cluster, it will be very easy to move Pi-hole over to it via a helm upgrade. For now, ensure you backup via teleporter regularly or leverage something like Orbital-Sync, to synchronize your configuration between this instance and perhaps a pi-hole server running directly on a Raspberry Pi.

Pulling it all together

Here is our complete pi-hole-values.yaml configuration file we'll use when installing the chart:

Install Pi-hole chart

I'm using the helm upgrade command with the --install flag, setting my release name to my-pihole (choose your own), the repository and chart name mojo2600/pihole, and finally, our custom values configuration file -f pi-hole-values.yaml

helm upgrade --install my-pihole mojo2600/pihole -f pi-hole-values.yaml

Successful deployment checklist:

1. kubectl get deployments should show my-pihole as ready and available.
2. http://pihole-k3s.mylocaldomain.org/admin loads the web interface and accepts my!admin3password to login.
3. dig @192.168.1.100 google.com answers successfully on port 53

Get visual with Lens

An outstanding Kubernetes management interface that is free for personal use. It's saved me quite a few times and helped me learn Kubernetes. I'm sure it will help you as well: Install Lens Desktop - Lens Documentation (k8slens.dev)

Is the internet down?

Homelabs are fun, but stuff happens. Do yourself a favor and set up a separate static install of Pi-hole on a cheap Raspberry Pi, and make this your secondary DNS server. I know we just worked hard to make Pi-hole highly available, but let's face it: you have a day job. So now you have two separate ad lists, whitelists, local dns, etc, to manage, which is where orbital-sync comes in. It's dead simple to set up on that same raspberry-pi via docker-compose. It creates a backup of your "primary" pi-hole, and restores that backup to any number of "secondary" pi-holes. You'll thank me on this one.

Join and Subscribe

For help, fun, and inspiration

• r/pihole, r/homelab
• Techno Tim, Network Chuck, Jeff Geerling, Christian Lempa