brokenSUSE
It finally happened: my openSUSE Tumbleweed install broke! I was not ready for it.
Background
I've been using this same Tumbleweed install since 2022, it's a solid distribution, YaST is a handy little tool, it has a nice selection of packages, and I hadn't had any major problems with it... until last week.
It's so over
I had just gotten home from my third day of college, I turned on my PC and went to get changed, then checked my phone to get caught up with messages I'd received. Some ten minutes later, I check the monitor... Blank screen.
"Huh?"
I hit the "Any" key. Wiggle the mouse. Nothing. That's not good.
"Maybe something went wrong during the boot, somehow?"
I hit the reset button. Motherboard logo, alright. GRUB message, okay. Blank screen. Little cursor blinking at the top. I leave it be. A couple minutes later, it restarts on its own. The same thing happens. And then again. It was not going to fix itself.
But that's fine, it was probably just a bad update, by nature of being a rolling distribution, Tumbleweed is very prone to breakage. Thankfully, openSUSE, by default, saves snapshots before and after each update, so there is no breakage without salvation! Let me just get to the GRUB menu and...
Oh... Right.
Some months ago, just to get a quicker boot time, I set the GRUB timeout to zero. The menu does not appear. I only see a brief message and it attempts to boot into the installed system.
...Well, surely there is a way to make it show the menu regardless, right? It would be silly not to! After all, I do still see a GRUB message for a split second before the OS starts. Yeah, right! It says online I can use "Shift" or "Escape" to make it show a boot prompt aaaaaand that does not work.
Well, this is quickly turning out to be an exciting night, which I didn't sign up for at all. Time for a little rescue job.
I always keep a little bootable drive handy in case I need a Linux environment. Recently, it's been KDE Neon, because I was itching to give Plasma 6 a try before it officially came out.
So it was just a matter of plugging in a thumb drive with KDE Neon and I would be able to better assess the situation.
Rescue the chameleon
And so I got to work, once I was in the live environment, the first thing I did was to mount my main drive and chroot into it, everything did seem in order, so at least I could be sure I didn't lose anything.
My main focus right now was to set the boot timeout for GRUB, so that I could roll back, I would be able to find out afterwards what broke the system. Since I was already in a chroot environment anyways, I could simply try and use YaST to set up GRUB, instead of fiddling with the configuration files. First, however, I needed to actually mount some additional partitions under the chroot environment so that:
- YaST would be able to launch properly
- The system would be able to actually write the reconfigured GRUB to the EFI parition
I stumbled around a bit for this part, since I hadn't ever needed to mess around with mounting BTRFS subvolumes.
mount /dev/nvme0p2 /mnt # Mount the root fs
mount --bind /dev /mnt/dev # Mount a few partitions
mount --bind /proc /mnt/proc # required by YaST
mount --bind /sys /mnt/sys
mount /dev/nvme0p1 /boot/efi # Mount the boot partition
# Mount the directories which store GRUB components
# referencing the subvolume names from fstab
mount /dev/nvme0p2 -o subvol=/@/boot/grub2/x86_64-efi /boot/grub2/x86_64-efi
mount /dev/nvme0p2 -o subvol=/@/boot/grub2/i386-pc /boot/grub2/i386-pc
I could have probably modified the grub configuration directly and built a configuration manually, but I always leave that to YaST under openSUSE because then things are less likely to go wrong. YaST actually recognised being under a chroot environment, so I figure it would be able to handle the task of reconfiguring GRUB.
After all of that was setup, I just hopped back into the chroot environment, ran YaST, and set the bootloader timeout to five seconds, and so a few agonizing seconds pass...
...And it was successful!
Well.... That was actually pretty easy. Time to reboot.
We're so back
Booting again, motherboard logo, and then GRUB appears! Nice! I immediately hop into advanced options and pick an older snapshot to boot from. Success, I'm in my desktop. I make sure to make this snapshot the active one with snapper rollback, and then reboot again.
We're all good. I decided to investigate.
Apparently the culprit was some package called virtiofsd, a dependency of qemu-tools, an update had changed a file into a directory, and rpm did not know how to deal with that, causing an error during installation (never a good thing), which I hadn't noticed because I left it updating through the graphical updater at some point.
So in the end, it was just a matter of me not paying attention to what a system update did.
Still, maybe I did need the scare, Tumbleweed had kept me way too comfy for a while, I needed something to get me back into hacking around in my system to fix problems.