So when I originally set up my smol me-and-some-bots Mastodon instance, I had it running on a Raspberry Pi, for ‘tis what I had, and my shipment of additional cluster nodes had not yet arrived.
This presented me with the issue of needing to move my existing Mastodon instance over to the cluster, which seemed simple enough. I had these excellent instructions from Funky Penguin’s Geek Cookbook on setting up Mastodon in Kubernetes, I had a perfectly cromulent container for Mastodon Glitch Edition, I had the most current Helm chart for Mastodon I could dig up, and I was pretty darn confident that all I should need to do was move the database over to the Postgres instance running on my NAS (the media already being on S3), clone the configuration, and I should be in business, right?
Oh, boy.
Helm Chart
Of course, that Helm chart doesn’t work, because it generates the same annotation twice. Argh!
Hackaround here (the changes to templates/deployment-sidekiq), in my own personal version.
IPv6
So here’s the thing.
Mastodon supports IPv6.
Kubernetes supports IPv6. (In fact, my cluster specifically prefers IPv6, but keeps IPv4 around for the less enlightened clients.)
The container image built for Mastodon, and by inheritance the Glitch edition, do not support IPv6.
See, as it tells you here, the BIND environment variable is how you tell Mastodon what address it should bind to, and that Docker container? Well, if you look at the Dockerfile, it sets BIND to 0.0.0.0. Every IP address, but IPv4 only.
“But Alistair,” I hear you cry, “this is so simple! All you have to do is change that to the IPv6 universal binding, ::, rebuild the image, and everything will be fine!”
“Seriously,” I respond, “do you really think I didn’t think of that?”
You see, there is one image. However, in your Mastodon instance on Kubernetes, this same image is used in three containers inside three pods, to give you the familiar services (mastodon-sidekiq, mastodon-streaming, and mastodon-web) that make up a Mastodon instance.
mastodon-sidekiq and mastodon-streaming both work perfectly when you set BIND to ::. (In the former’s case, that’s because it doesn’t bind to anything at all and so simply ignores it.) mastodon-web, however, does not. It crashes on startup.
To make mastodon-web work, you can set BIND to [::], the square brackets indicating to the relevant code that the contents are an IPv6 address, and it then starts up perfectly. But - you can see this coming, can’t you? - mastodon-streaming doesn’t recognize this format, and now it crashes on startup.
So much for simply rebuilding the container.
As I don’t speak either Ruby or JavaScript well enough to unbreak this the way it ought to be unbroken, I hacked the relevant BIND values into a personal copy of the Helm chart. The relevant commits are here and here.
(You will notice that I also added ipFamilyPolicy: PreferDualStack to the Mastodon web and streaming services. This shouldn’t be necessary, since Traefik, my ingress provider, also speaks perfectly good IPv6 - but I have some legacy bots and such that it’s worth providing for in service-land.
Let me also take a moment here to say that not only should every container image bind by default to IPv6 addresses instead of 0.0.0.0 these days, but every Helm chart should include PreferDualStack by default. IPv6 has been out for a long time now, people. Get with the program! It’s not like PreferDualStack hurts anything on a single-stack cluster!)
Glitch Configuration
And finally, you can’t strictly-speaking configure various Mastodon features through the default Helm chart, because it doesn’t include them. For me, this means the features that Glitch Edition adds, because I’m a wordy sort and like my long toots and bio, which I previously configured by defining the following environment variables:
MAX_TOOT_CHARS=4096
MAX_BIO_CHARS=4096
MAX_PROFILE_FIELDS=8
MAX_POLL_OPTIONS=6
in the non-clustered .env configuration file.
Now, I could have added these to the Helm chart as new values to submit to it, but the Helm chart I’m working off is for regular Mastodon, not Glitch. Also, depending on which of the various Mastodon forks someone might want to run, there could be all sorts of configuration options out there I’m not familiar with. So I went for a generic solution. Basically, I define my additional configuration in a very simple ConfigMap:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: mastodon-supplementary
data:
MAX_TOOT_CHARS: "4096"
MAX_BIO_CHARS: "4096"
MAX_PROFILE_FIELDS: "8"
MAX_POLL_OPTIONS: "6"
and I made some modifications to the Helm chart to define an extraEnvironment: option which lets you reference that ConfigMap, and have the environment variables defined in it added to all of the Mastodon pods. See this commit for details of those changes.
Working! Working at Last!
Having done all that, I could finally just feed the relevant values into my personal Helm chart copy, per the Geek Cookbook article at the top, and have a working system.
| --- | |
| apiVersion: v1 | |
| kind: ConfigMap | |
| metadata: | |
| name: mastodon-chart-values | |
| data: | |
| values.yaml: |- | |
| # Configuration values for Mastodon | |
| image: | |
| repository: ghcr.io/glitch-soc/mastodon | |
| tag: latest | |
| pullPolicy: Always | |
| mastodon: | |
| local_domain: social.arkane-systems.net | |
| s3: | |
| enabled: true | |
| access_key: REDACTED | |
| access_secret: REDACTED | |
| bucket: REDACTED | |
| endpoint: https://s3.us-east-1.amazonaws.com | |
| hostname: s3.us-east-1.amazonaws.com | |
| region: us-east-1 | |
| secrets: | |
| secret_key_base: REDACTED | |
| otp_secret: REDACTED | |
| vapid: | |
| private_key: REDACTED | |
| public_key: REDACTED | |
| streaming: | |
| workers: 1 | |
| smtp: | |
| server: mail.arkane-systems.lan | |
| login: nobody | |
| password: nonesuch | |
| extraEnvironment: mastodon-supplementary | |
| ingress: | |
| enabled: true | |
| ingressClassName: traefik | |
| annotations: | |
| traefik.ingress.kubernetes.io/router.entrypoints: 'websecure' | |
| traefik.ingress.kubernetes.io/router.tls: 'true' | |
| hosts: | |
| - host: social.arkane-systems.net | |
| paths: | |
| - path: '/' | |
| tls: [] | |
| postgresql: | |
| enabled: false | |
| postgresqlHostname: mnemosyne.arkane-systems.lan | |
| auth: | |
| database: mastodon_production | |
| username: mastodon | |
| password: REDACTED | |
| redis: | |
| replica: | |
| replicaCount: 0 | |
| elasticsearch: | |
| enabled: false |
(Of possible interest here, if you’re comparing it to the article’s:
I don’t have a createAdmin: section because I’m using my existing database.
The blank tls: clause in the ingress eliminates default values from the chart that were interfering with the way I prefer to run things: no TLS inside the cluster, the ingress controller handles TLS outside the cluster, and the border reverse proxy rewrites it for the outside world.
Setting enabled: false in the postgresql: section and then specifying auth details is, for this chart, how you specify using a Postgres database outside the cluster. I run Postgres on my NAS for this because it only supports NFS, which isn’t really suitable for remote database files, and doesn’t support something more useful for that like Ceph.
I force the redis replica count to 0 because otherwise I get a full three-instance replica as well as the master, and frankly, that’s very much overkill for my little personal instance. As is ElasticSearch, which I disable because it eats much more memory that I feel like spending on a search option I rarely, if ever, use.)
But that was quite a bit of work along the way.