How gnome-software/systemd software upgrade dropped me to the grub command line, and how I resolved it

Earlier I was messing around in one of my Fedora Linux virtual machines. I’ve had this particular install since Fedora 23 was the latest release. I’ve upgraded it from 23 to 24 and then again from 24 to 25. I completed the upgrade a few weeks back and decided to come back to it. Of course, since it had been a few weeks, after I loaded it up and logged in, GNOME was telling me there were updates available in the Software (gnome-software) application.

Brief warning: this article contains a ton of screenshots as I worked the problem.

Mindlessly, I opened the gnome-software utility and decided to update the system through there. The updates looked pretty benign, whatever. I clicked ‘Restart & Install’ and confirmed. The system rebooted and brought me to the systemd installing updates prompt, awesome. It finished and rebooted once more, however I was immediately dumped to a grub command line. Uh oh.

So I took the #fedora IRC channel on Freenode, explained what had happened, and got some interesting feedback.

First, we tried to identify the problem. We started by trying to figure out why grub command line was coming up immediately while booting rather than the grub boot menu. I found that the grub config file was in fact empty, pretty straightforward then why we’re met with the grub command line.

My initial thought is the gnome-software utility told systemd to reboot for updates, but systemd didn’t install updates correctly. There was probably a kernel update and the grub config got caught in the mess. A little more trial and error with the fellas in #fedora and I was able to tell grub to boot the vmlinuz and initramfs files still visibly present.

Booting didn’t work though. A kernel panic came up while booting, Linux couldn’t mount the root filesystem.

Now I’m wondering if my hard disk is somehow corrupted after the updates. Obviously an update didn’t finish correctly, that much is obvious.

I head back to the #fedora IRC channel and suggest that I try booting the Live CD iso instead to dig around any further, I was tired of messing around in grub’s command line. They agreed this would in fact aid diagnosing the issue, so off I went.

I identified that the hard disk is using a single LVM partition, there are no other partitions on the Master Boot Record (MBR) of the disk. And the LVM partition only has a single logical volume in it called root with the / mount point and is xfs formatted. Pretty strange, and I don’t remember why I chose this layout so long ago.

I decide to remove the Live CD iso from the machine and reboot. What happened next though was pretty weird. I had looked away for a bit to check the IRC channel, when I came back, the machine had booted!

What the hell. Alright, so might as well browse around then. I logged in. I even was greeted with the message that updates were installed successfully!

I decided to open Terminal and verify the disk layout:

Yup. That’s a lvm-xfs partitioned disk. There is in fact no /boot partition, which means the single / partition is the /boot partition. By this point, the people over in #fedora were pretty grateful grub has matured to be able to understand booting LVM partitions, but they were at a loss for words for what was going on as well.

After talking with the #fedora IRC channel some more, I agree to figure out what state causes the machine to drop to the grub command line after upgrading.

I began going through each variable in my tests:

  • Default partitioning format
  • Custom partitioning format
  • Upgrading system using gnome-software
  • Upgrading system using dnf

The default partitioning format is just as you would expect–you load up the Live CD iso with a blank hard disk, you install Fedora to that hard disk using the default partitioning that anaconda chooses for you. No modifications.

The custom partitioning format is a bit different. You start with a single LVM partition and then you create an xfs partitioned with the / mount point:

I and the people in the #fedora IRC both found that installing Fedora with the custom partitioning format and upgrading using gnome-software would cause Fedora to be dropped to the grub command line every time, and only repairable by booting the Live CD iso again and mounting the filesystem so that the xfs journal is replayed after the updates.

Getting to that diagnosis though took several hours, and after confirming with others I opened the bug report over at Red Hat: Bug 1416650 – Upgrading using gnome-software/systemd with lvm-xfs custom partitioning format causes grub boot failure

Hopefully I get an answer back from the Fedora and/or Red Hat teams. Quite a head scratcher at first, but obviously a bug since it’s supposed to be supported and doesn’t indicate otherwise.