The day I broke my Nix store (and my NixOS)

Nix and NixOS are reputed for their reproducibility. I've been using it for a few years with great satisfaction, never believing that somehow the Nix store (holding Nix packages and binaries) could somehow be corrupted into not following this sacro-saint reproducibility.

Updating NixOS, what could go wrong?

NixOS system update or upgrade is as simple as editing a few .nix files and running

nixos-rebuild switch

Which builds .nix configurations into the Nix store. This is supposedly entirely reproducible: if I build on a system then wipe that system and do it again, I should get the exact same result.

But not this time.

Here's what happened when I tried to upgrade my NixOS from 23.05 to 23.11:

• I updated my .nix config and ran nixos-rebuild switch. It started copying and building lots of new files into the Nix store (Nix will build everything here before finally applying changes)
• This was a pretty long update but I forgot to plug-in my computer to power. Of course it ran out of battery and crashed 😮
• Fortunately it still worked after reboot. Running nixos-rebuild switch again, Nix simply continued where it left off.

After the update and another reboot... I was greeted by a blank screen. Oops.

Reboot again. Use last previously working config. Pray. It worked! Well, back so something stable. Let's see what went wrong.

Did Nix failed somewhere?

My first reflex was to run nixos-rebuild switch again. Unsurprinsigly Nix reported nothing was built since everything was already. After a bit of googling around I found out that the Nix store could actually be corrupted, but Nix didn't complain.

A few dead-ends and some reboot later I finally bumped into strange entries in the journalctl of a failed boot:

error while loading shared libraries: /nix/store/[...]/libmutter-clutter-13.so.0: file too short

Well. This looks like a corrupted .so... It was probably damaged by my previous system crash during update ! Let's try to clean that up and build it again.

Cleaning the corrupted Nix store

Following a few sources I tried to clean my Nix store with:

nix-collect-garbage -d

And updated my system again:

nixos-rebuild switch

This time making sure power was plugged-in (duh.) After reboot, it worked and the *.so: file too short was gone from journalctl. Yay !