Home Assistant on Kubernetes: The Latest

In which I iterate and refine.

May 15, 2023

So long-term readers may recall that I posted an article a while back on how I run Home Assistant in Kubernetes, because I refuse to dedicate a freakin’ VM (or worse, hardware) to such things when I have a perfectly good cluster sitting right here.

Which was not terribly satisfactory because not everything was yet clusterified, to an extent which included the recorder database. So then I made things a little better by including the recorder database and the MySQL instance supporting it into the Home Assistant pod (here) and then properly setting things up such that the recorder database would always start up first (here).

Well, I’ve refined things even more since then, so here’s some details on the latest iteration.

Recorder Database

First up, the recorder database, which is now in a separate pod (or rather pods) of its own, because there is no particular need for it and Home Assistant to run on the same node when they don’t have to. Also, to create the recorderdb MySQL instance, I’m now using the MySQL Operator for Kubernetes, which is both a very convenient way to set up such instances (as many things you may wish to run on a cluster require) and backups for them, and means that it’s set up in the same way as my other MySQL instances.

I won’t include here details on how to set up the MySQL Operator itself, because you can read the documentation at the link and install the Helm chart just as well as I can.

So the configuration files for the recorder database look like this (I’m omitting recorderdb-secret.yaml, which has the root passwords for the instance in it, because once they’re redacted there’s nothing left. See the above documentation for the format.):

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	---
	apiVersion: v1
	kind: PersistentVolume
	metadata:
	name: recorderdb-pv
	namespace: homeassistant
	spec:
	storageClassName: ""
	capacity:
	storage: 48Gi
	accessModes:
	- ReadWriteOnce
	persistentVolumeReclaimPolicy: Retain
	nodeAffinity:
	required:
	nodeSelectorTerms:
	- matchExpressions:
	- key: kubernetes.io/hostname
	operator: In
	values:
	- princess-luna
	hostPath:
	path: /opt/k3store/recorderdb

view raw recorderdb-pv.yaml hosted with ❤ by GitHub

First, the definition of the persistent volume which holds the recorder database. I define this directly rather than using a normal PersistentVolumeClaim because the volume of activity of my recorder database makes putting it on an NFS share from my NAS impractical, and as well as being local, I need it tied to a specific node so that Home Assistant doesn’t lose track of existing recorder data when it restarts (as would happen if a new database data volume was assigned). This is handled by the nodeAffinity section. The storageClassName is deliberately blank.

Show hidden characters

	---
	apiVersion: mysql.oracle.com/v2
	kind: InnoDBCluster
	metadata:
	name: recorderdb
	namespace: homeassistant
	spec:
	secretName: recorderdb-pwd
	instances: 1
	router:
	instances: 1
	datadirVolumeClaimTemplate:
	storageClassName: ""
	volumeName: recorderdb-pv
	accessModes:
	- ReadWriteOnce
	tlsUseSelfSigned: true
	podSpec:
	nodeSelector:
	kubernetes.io/hostname: princess-luna

view raw recorderdb-db.yaml hosted with ❤ by GitHub

This, meanwhile, is the custom InnoDBCluster resource that instructs the MySQL Operator to create the recorder database. You can consult the operator documentation for the details, but essentially, we’re creating a single-instance cluster (equivalent to what we were doing before), and using the persistent volume we created above (the cluses under datadirVolumeClaimTemplate). We also use nodeSelector to tie the recorder database pods to the same cluster node, princess-luna, that the persistent volume was tied to.

And that gets the instance up and running on Kubernetes. At this point, since it’s a clean and new MySQL instance, you will need to set up the actual Home Assistant user and database on that instance (as described here) before you proceed. There are several ways to do this - accessing the container directly, running a MySQL client pod with kubectl, and so forth - of which my personal choice is Telepresence. I’ll describe how to do this in detail in a future article.

If you have a currently running previous recorder database instance, you can transfer the data across using mysqldump, or if you have a backup of a previous recorder database, you can restore the backup when creating the instance (see the operator documentation for how).

But for now, let’s move on to setting up Home Assistant on top of this. The first thing you will need to know is the service created for the MySQL database instance you’ve made. For the above, since I’m using the homeassistant namespace, this is easily obtained by kubectl get services -n homeassistant, which produces the following output:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

…

recorderdb ClusterIP 10.43.209.27 <none> 3306/TCP,33060/TCP,6446/TCP,6448/TCP,6447/TCP,6449/TCP,8443/TCP 7d14h

recorderdb-instances ClusterIP None <none> 3306/TCP,33060/TCP,33061/TCP 7d14h

…

The relevant one for you is the one without -instances at the end, which is the via-the-router access you want. What’s even more relevant is that you can access this service from inside the cluster using the default hostname

recorderdb.homeassistant.svc.cluster.local.

You need, first, to put this into your Home Assistant configuration files for the Recorder integration to tell Home Assistant where to find your database. Since I have created a user ha, with a password <REDACTED>, and a database named ha_recorder, the relevant section for me looks like this:

Show hidden characters

	recorder:

	# We store this in an external MySQL database elsewhere on the cluster.
	db_url: mysql://ha:<REDACTED>@recorder.homeassistant.svc.cluster.local/ha_recorder?chartset=utf8mb4

view raw recorder.yml hosted with ❤ by GitHub

Home Assistant

Now we get to the complicated bit. There are three main configuration files involved in the Home Assistant configuration - the deployment itself, plus the service and the ingress that let you access it - but first, there’s the elephant in the room that makes deploying Home Assistant on Kubernetes a little painful.

Kubernetes is stateless. In an ideal Kubernetes setup, if an application needs a database, it will wait for it if it hasn’t started up yet, fail gracefully if the database isn’t available, and reconnect to it automatically if it becomes available again.

Home Assistant is not. If the recorder database isn’t available at startup, it will just fail messily. (It also won’t reconnect automatically if the recorder database goes unavailable for a while, but that’s harder to fix and outside the scope of this.)

This means that we have to ensure that the Home Assistant pod does not start until the recorder database service is available. (When they lived in the same pod, this what we used OpenKruise for.) In this iteration, we’re using stackanetes/kubernetes-entrypoint to achieve this, and that means a little more configuration up front:

Show hidden characters

	---
	kind: Role
	apiVersion: rbac.authorization.k8s.io/v1
	metadata:
	name: read-ha-dependency
	namespace: homeassistant
	rules:
	- apiGroups: [""]
	resources: ["endpoints"]
	verbs: ["get"]
	---
	kind: RoleBinding
	apiVersion: rbac.authorization.k8s.io/v1
	metadata:
	name: read-ha-dependency-binding
	namespace: homeassistant
	subjects:
	- kind: ServiceAccount
	name: default
	namespace: homeassistant
	roleRef:
	kind: Role
	name: read-ha-dependency
	apiGroup: rbac.authorization.k8s.io

view raw dependency-check-rbac.yaml hosted with ❤ by GitHub

Basically, in order to achieve this, the Home Assistant pod - or rather, the service account for the homeassistant namespace, under which that pod runs - needs permission to check if the recorderdb service we mentioned earlier currently has a valid endpoint (i.e., a functioning MySQL instance). The above file gives it permission to know that.

So now, the Home Assistant deployment itself:

Show hidden characters

	---
	apiVersion: apps/v1
	kind: Deployment
	metadata:
	labels:
	app: homeassistant
	name: homeassistant
	spec:
	replicas: 1
	selector:
	matchLabels:
	app: homeassistant
	strategy:
	type: Recreate
	template:
	metadata:
	labels:
	app: homeassistant
	spec:
	nodeSelector:
	arkane-systems.lan/bluez: "true"
	arkane-systems.lan/zigbee: "true"
	volumes:
	- name: ha-storage
	nfs:
	server: mnemosyne.arkane-systems.lan
	path: "/swarm/harmony/homeassistant/ha"
	# For Bluetooth support
	- name: dbus
	hostPath:
	path: "/var/run/dbus"
	# For Zigbee support
	- name: skyconnect
	hostPath:
	path: "/dev/ttyUSB0"
	type: CharDevice
	initContainers:
	- image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
	name: dependency-check
	env:
	- name: NAMESPACE
	valueFrom:
	fieldRef:
	apiVersion: v1
	fieldPath: metadata.namespace
	- name: DEPENDENCY_SERVICE
	value: 'recorderdb'
	- name: COMMAND
	value: echo done
	securityContext:
	privileged: true
	runAsUser: 0
	containers:
	- image: homeassistant/home-assistant:2023.5.3
	name: homeassistant
	securityContext:
	privileged: true
	volumeMounts:
	- mountPath: "/config"
	name: ha-storage
	# For Bluetooth support
	- mountPath: "/var/run/dbus"
	name: dbus
	# For Zigbee support
	- mountPath: "/dev/ttyUSB0"
	name: skyconnect

view raw homeassistant-deployment.yaml hosted with ❤ by GitHub

This is a little complex compared to a vanilla Kubernetes deployment, so I’m going to walk you through it pointing out the interesting bits from top to bottom.

1. First, you’ll notice the strategy: Recreate line up in the specification. This ensures that when you update the deployment (say, for a new version of Home Assistant), or when for some other reason the Home Assistant pod is recreated, that Kubernetes waits for the old one to shut down before starting the new one, instead of following its default strategy of allowing them to overlap. This is important because Home Assistant assumes that there will be only one of it talking to the recorder database at a given time, not to mention other integrations that may fight over who’s talking to a particular service or piece of hardware.

2. The nodeSelector clause requires the Home Assistant pod run on a node labeled with both arkane-systems.lan/zigbee and arkane-systems.lan/bluez as “true”. I use these labels in my cluster to identify the nodes which, for the former, have a Zigbee adapter (the SkyConnect, as it happens) connected to them, and for the latter, which have BlueZ installed for Bluetooth support. These are required by integrations I use.

3. The ha-storage volume contains all the Home Assistant configuration and log files, etc., other than the recorder database. As in previous versions of this configuration, it’s an NFS share on my NAS. You could use a local path for this, but if you do, don’t forget to tie the Home Assistant pod to the relevant node.

4. The other volumes expose the dbus socket (so that the Bluetooth integration can talk to BlueZ) and the serial port used by the SkyConnect USB device (for the ZHA integration) to the Home Assistant pod.

5. The init container, dependency-check, is the one using stackanetes/kubernetes-entrypoint (documented here) that we talked about before to prevent Home Assistant from starting up before the recorder database is ready.

The configuration you can see here sets it to run in the relevant namespace, to wait for the recorderdb service in that namespace to be ready (i.e., have an endpoint) before continuing, and then to just echo “done” to the log once the service is ready. Since it’s an init container, the actual Home Assistant container below won’t start up until this happens.

Show hidden characters

	---
	apiVersion: networking.k8s.io/v1
	kind: Ingress
	metadata:
	name: homeassistant-ingress
	annotations:
	traefik.ingress.kubernetes.io/router.entrypoints: 'websecure'
	traefik.ingress.kubernetes.io/router.tls: 'true'
	spec:
	rules:
	- host: jeeves.harmony.arkane-systems.lan
	http:
	paths:
	- pathType: Prefix
	path: /
	backend:
	service:
	name: homeassistant
	port:
	number: 8123
	- host: automation.arkane-systems.net
	http:
	paths:
	- pathType: Prefix
	path: /
	backend:
	service:
	name: homeassistant
	port:
	number: 8123

view raw homeassistant-ingress.yaml hosted with ❤ by GitHub

Show hidden characters

	---
	apiVersion: v1
	kind: Service
	metadata:
	name: homeassistant
	spec:
	selector:
	app: homeassistant
	ports:
	- protocol: TCP
	port: 8123
	name: http
	ipFamilyPolicy: PreferDualStack

view raw homeassistant-service.yaml hosted with ❤ by GitHub

Last and least interestingly, the service and ingress configuration, which publishes Home Assistant’s front end so things outside the cluster can see it. There’s nothing really interesting to note here, except that I’m using Traefik as my ingress controller (obviously things will be different if you’re not), and that I have ingress rules for Home Assistant under two different names, one for my internal network, and one for a public DNS name (which comes through a proxy), such that I can access it conveniently from both inside and outside my local network.

And that’s it! Hope this has been interesting, and if you are looking to run Home Assistant on Kubernetes yourself, helpful, although it will obviously require considerable tweaking for your local network setup and requirements.

Until next time!

Random Bytes

Discussion about this post