Migrating from systemd-genie to native WSL systemd

Because WSL has native systemd support now!

Yes, it does. Look, it’s right here!

(And it is very cool. Much more elegant than the assorted hacks we’ve put together so far. I mention this specifically because in this migration guide I bring up a whole bunch of differences and potential issues and I want to be clear that you should not let them put you off; use this instead of genie if you can.)

So here’s the big list of differences and issues. Some of them come with notes, some of them come with workarounds, and some of them are going to be addressed in my upcoming genie spinoff, bottle-imp, specifically designed to help out users of native WSL systemd.

1. Can make the resolv.conf symlink for systemd-resolved

genie does this and native systemd does not. That being said, this is an ugly kluge that went in to solve a particular problem in the early days of genie and I’m not aware of anyone still using it. And if you are, please don’t. There are better ways and this should never have been solved there in the first place.

2. Can update the hostname to distinguish the host from the WSL distribution

genie does this and native systemd does not. You couldn’t do it in the genie way, actually, since that relies on changing the hostname before systemd fires up and it fires up first in this scenario.

On the other hand, we have this entry in /etc/wsl.conf now that didn’t exist when I wrote this feature. Use that instead:

[network]
hostname = pallas-wsl

(Incidentally, if you’re using systemd-resolved, it should notice that there are two instances of, say, “pallas” on the network and quietly rename your WSL distribution “pallas2” for network purposes only. Convenient, but confusing.)

3. Checks for appropriate default target

genie does this and native systemd does not. Just be careful. Remember, graphical.target is not your friend.

Unfortunately, the current release of native systemd hard-codes graphical.target, but a fix for that issue is already on its way.

(I may put in a check for this if I add a imp-check script to bottle-imp.)

4. Configures systemd-tmpfiles to restore the X11 socket symlink to /tmp/.X11-unix after systemd clears out /tmp on startup

genie does this and native systemd does not. I’m implementing the same fix in bottle-imp that genie uses. Until then, you’ll need to recreate it manually or place instructions in /usr/lib/tmpfiles.d to automatically recreate the link.

5. Create a user session for you, along with a user systemd and session dbus

It would be a little unfair to claim that native systemd does not do this, because really, it’s just an artifact of how user sessions are created (i.e., via login, or systemd-machined, or other bits of Linux that invoke PAM and therefore the systemd PAM module that creates user sessions and thus the user systemd to manage them and the session dbus associated with that.

genie had this happen almost as a side-effect, simply because it used systemd-machined and machinectl to get inside the bottle, which triggered all the relevant mechanisms. WSL, which by default just creates a process with the appropriate uid, does not cause this to happen. If you want or need a user session and to be associated with that user session, you’ll need to do something like that to trigger the PAM mechanisms.

Greatly simplified versions of genie -s, -l, and -c are going into bottle-imp to provide easy access to this functionality.

This is also Microsoft/WSL issue #8842, for those interested.

6. Mounts securityfs (for AppArmor and other LSMs) and pstorefs (for debugging panics) automatically

genie does this; native systemd does not. Quite understandably, really, since to use an LSM under WSL already means you’re heading off to the magical land of custom kernels.

Anyway, the issue here is that native systemd launches systemd in container mode, and these two “API filesystems” - which are normally automatically mounted by systemd early on - aren’t mounted in container mode.

If you need them, either you can use the fix that will soon be in bottle-imp, or you can create your own systemd units to manually mount them. A word of warning: because systemd considers them API filesystems, you can’t create mount units to do this. You have to kluge it with service units which call mount.

Gritty details at Microsoft/WSL issue #8840, including an example unit.

7. Mounts the user runtime directory already created by WSLg over that created by systemd (for user with uid=1000), ensuring that everything is gathered in one, correct place

genie does that; native systemd does not, although it does create its own user runtime directory (i.e., /run/user/1000). The fix for this will be in bottle-imp, or if you want to implement it on your own you can find it here (plus the two dependent scripts a little way up the repo).

8. Passes the WSL_INTEROP environment variable through to systemd-sourced sessions, enabling interop in them

When I say systemd-sourced sessions, I mean those created by machinectl (as above), or ssh, or other things that put the session as a direct child of pid 1 (systemd) rather than the Microsoft init.

You see, the process tree looks something like this:

systemd─┬─accounts-daemon───2*[{accounts-daemon}]
├─agetty
├─atd
├─automount───3*[{automount}]
├─containerd───9*[{containerd}]
├─dbus-daemon
├─dockerd───9*[{dockerd}]
├─init─┬─init───init───bash
│ ├─init───init───machinectl
│ ├─login───bash
│ └─2*[{init}]
├─msmtpd

…and when you run something, like a shell, under WSL the init in the second column spawns the ones in the third and fourth columns to handle it. The one in the fourth column, right before bash, handles the interop. In this example, it’s pid 91, and the bash and its child processes get an environment variable pointing to a socket that that init has opened to receive interop messages, namely:

WSL_INTEROP=/run/WSL/91_interop

You see that machinectl, though? The shell it spawned is way down at the other end of the process tree, here:

├─upowerd───2*[{upowerd}]
├─watchdog
└─zsh─┬─(sd-pam)
└─tmux: client

…where it’s a direct child of systemd, and the tmux server that client is talking to is a service managed by the user systemd in yet another location on the process tree, and none of them are in any position to inherit the WSL_INTEROP variable that makes interop work.

In genie, I used to solve this by passing the WSL_INTEROP variable through with the other environment variables cloned from outside the bottle, which worked fine. The obvious solution here would be to copy the WSL_INTEROP from a long-running WSL session into the systemd base environment with:

systemctl import-environment WSL_INTEROP

…at which point it’s inherited by all systemd services and sessions.

But the spoiler in all this is that it only stays valid for as long as that long-running WSL session stays running. In genie, this was not an issue, because the session running genie itself would obviously stay around for as long as systemd did, but there’s no guarantee of that in this scenario.

A workaround for this is going into bottle-imp, but it’s more or less intrinsically linked to the one for (13), below.

The issue is at Microsoft/WSL issue #8846, again for those interested.

9. Provides a command to cleanly shutdown systemd/the system

genie provided the genie -u command to shut down the bottle (and hence systemd, and all of its services, cleanly) from outside. You don’t need to do this under native systemd; you’re always inside the bottle, so you can just run systemctl poweroff.

(You may not even need to do this, since per the blog post, “Additional modifications had to be made to ensure a clean shutdown (as that shutdown is controlled by systemd now)”, which sure sounds like WSL does its own systemctl poweroff when closing down, but if you feel paranoid, running your own doesn’t hurt.)

10. Provides check functions to discover if systemd is running and if you’re “inside the bottle”

Well, obviously, these aren’t needed any more. Because it is, and you are.

11. Restores the WSLInterop binary format in case it is lost when systemd updates binfmts

The genie package includes this and so will the bottle-imp one, but if you find yourself losing the WSLInterop binary format under native systemd, you just need to create this file in /usr/lib/binfmt.d.

12. Sets up AppArmor in a namespace to prevent profile leaks between distributions

genie did this; native systemd does not appear to. I don’t have a workaround for this one, as you have to create and enter the AppArmor namespace before systemd starts up.

Expect profile leaks if you compile AppArmor into your kernel and use multiple distributions.

13. Stays running even when you have no active sessions.

Oh, boy. Now this is a big change in behavior; as the blog post says, “It is also important to note that with these change, systemd services will NOT keep your WSL instance alive.”

Also, systemd user sessions, including systemd user sessions commanded to linger with loginctl, will also not keep your WSL instance alive.

Basically, if it’s not a child of init - the Microsoft init, not pid 1 - it won’t keep your instance running. This is probably going to be quite the gotcha for folks used to the genie behavior, in which the genie process itself kept everything up and running for as long as systemd was up.

I’m putting a workaround for this in bottle-imp, with a hold-instance-open process that also provides the workaround to (8). Until then… start a spare console?

14. Unmounts and remounts the binfmts filesystem to avoid later problems

genie did this; native systemd doesn’t have to because the problems don’t exist.

15. Waits for systemd to have initialized before proceeding

Native systemd does not do this. The usual wait/timeout will be in bottle-imp, but without it, you can have a console all the way through initializing and starting including the various phases of startup where native Linux wouldn’t let you log in yet.

Be careful that

systemctl is-system-running

reports that it’s running (or at least finished with startup) before you start doing things that might need systemd. Which includes starting systemd user sessions, per (5), because those have dependencies.

16. Warns you if systemd is in a degraded state after starting up

Since the answer to (15) is no, it’s probably no surprise to you that the answer to this one is no, too. Until bottle-imp is available, you’ll need to keep an eye on this one the old-fashioned way, too.

Other warnings…

Most of the things that could be problematic with systemd units under genie can be likewise problematic under native systemd support. If you run into issues, this page can still be helpful.

And that’s all she wrote for now! More as more is learnt, or at least when bottle-imp is released.