I got my car back today

I woke up at 5pm Sunday evening, thinking alright I’ll spend the night chatting with friends and playing games, no big deal. That night, I try to go to sleep at 3am, and I just sort of lay there in my bed until 7am rolled around, and I decided I’m not going to sleep for 1 hour until my alarm goes off. Insomnia is terrible. So, I decide since it’s Monday, before I go in to work at 9am, I’m going to go pick up my car, cause it’s ready to be picked up so why not?

Well, let me tell you. What a day today has been.

A little background: I drive a 2014 Honda Civic. I absolutely love the car since the day I picked it out and drove it off the lot. Fast forward two years later, I was hit by someone on my way to work, to the driver-side rear bumper. The damage wasn’t too bad, the car still drove and functioned as it should, and it didn’t even leak when wet, but still it needed to get repaired since there’s still payments left and there’s an insurance claim. I take it in to my local Honda Collision Center and get situated with a 2017 Toyota Corolla rental.

The Corolla only had around 6,200 miles on it when they handed the keys to me. My Civic had about 21,000 miles. There’s several reasons I chose my Civic over the other entry-level sedans like the Corolla, most of them are preference but a few things stood out immediately even on the 2017 Corolla when compared to my 2014 Civic:

  • The headlights. My Civic has halogen bulbs. The Corolla has LED bulbs. This was a huge difference I wasn’t immediately used to but enjoyed as I rode the rental for a while. LED bulbs are actually pretty neat.
  • The engine, accelerator, and brake. They’re just not the same. I don’t expect them to be, they’re two different cars after all. But, just after being stopped at a red light accelerating to 35 mph, the Corolla would accelerate up to 3500 rpm easily, however my Civic would only need 1500 rpm to do the same task.
  • The rear-view back-up camera. My 2014 Civic’s camera has a HUD that aligns itself with the steering wheel, allowing ease of movement while geared in reverse. The Corolla’s camera does not have a HUD that steers with the steering wheel. In addition, the Corolla’s camera was very foggy even after rubbing and cleaning, yet my Civic has a crystal clear picture.

With that out of the way, it’s 7am, I get dressed for the day and head to the collision center to pick up my car and drop off the rental. They had the car out to me within 5 minutes, pretty speedy no issue at all. Only immediately notice one problem and that was on the driver-side rear seatbelt, the plastic chassis behind the seatbelt wasn’t clipped into the interior wall. They took it back, had it fixed within 5 minutes, and back out to me again. I give it a good look over, I’ve even been described as conscientious before. No other problems that I see. The interior smells like fresh paint, of course. The paint job looks fresh, I can tell they waxed it and put a new clear coat on. I sign the paperwork that I received the car. Cool, I have my car back! But wait, I still have to turn in the rental…

[This post was never fully drafted. It ends there. Sorry.]

Replacing your server is not always necessary after encountering boot failure

First, a little background context before I dive in to how I recovered the virtual machine from boot failure…

  • The host has three physical disks mapped to the virtual machine which I’ll be recovering.
    • One disk is a 256 GB SSD and is unused/unallocated, the other two are 5 TB HDDs paired together via LVM to create a single 10 TB disk inside the guest’s filesystem.
  • The host runs Windows 7 64-bit, runs VirtualBox 5.1.10, and the guest runs Fedora 25 64-bit on Linux kernel 4.8.15-300.fc25.x86_64.
  • The guest uses SystemD and is referred to as ‘seedbox’ in my logs and snapshots.

Earlier this morning, the host completely froze, not even so much as a BSOD. There is a hardware problem with the host’s RAM that I have yet to completely diagnose. I ended up pushing the reset button on the chassis. All running guests were immediately aborted due to my power reset.

The host system rebooted just fine. Upon turning back on all of the guests, the ‘seedbox’ guest entered into SystemD emergency mode while it was booting. Uh oh.

Turns out that when the host rebooted, the physical disk mappings present for the guest were changed by either Windows or the host’s BIOS itself, and VirtualBox was not aware or did not track the difference. The SSD and one of the HDDs were no longer the correct physical disks. The LVM volume group was therefore invalid and auto-mounting the LVM logical volume failed. This was the root cause of boot failure within the guest.

Okay, so we’ve now identified the cause of boot failure. What about recovery? Is all the data corrupt now?

The guest failed to mount the LVM logical volume, which prevented it from possibly writing to the disks, causing corruption. The SSD and HDDs were never mounted by Linux, not even in read-only, so they were preserved during the boot failure. This is a good indicator that the data should be completely intact if we can rebuild the mapping.

Looking into correcting the physical to virtual disk mapping, I found out that due to the shift in mappings, the newly assigned physical disks were actually possibly catastrophic. One of them was the host’s boot drive, which even VirtualBox says NEVER to mount in a guest: “Most importantly, do not attempt to boot the partition with the currently running host operating system in a guest. This will lead to severe data corruption” [VirtualBox Manual 9.9.1]. I quickly commented out the auto-mount for that LVM logical volume and powered off the virtual machine, then looked into correcting the mapping via the host.

In VirtualBox, if you want to map physical disks to virtual ones, you need to run a utility called VBoxManage as root/administrator, and call some internals of the utility that are vaguely documented in section 9.9.1 in the advanced topics chapter of the VirtualBox manual (link). This is non-trivial for those just getting started with virtual machines and disk management, since there’s a high risk for corruption, as the manual indicates. If you map your host’s physical boot disk to the guest, then try to boot off of it, you’ll almost always cause severe corruption, this is what the manual is warning everyone about.

So off I went. I opened up the Command Prompt in administrator mode on the host and went to the directory the virtual disk files were in. In the VirtualBox window, I removed the disks from VirtualBox’s Media Manager, then switched back to Command Prompt. I then replaced the .vmdk files with new copies that had corrected physical drive ids. I then re-added the virtual disk files into VirtualBox, attached them to the seedbox guest, and booted up the guest.

The guest booted up successfully. Since I uncommented the failing drive configuration, it was able to boot normally, with only services relying on that mountpoint failing to run correctly. The core system was intact.

I uncommented the auto-mount line for the LVM logical volume I commented out earlier, re-mounted all auto-mounts via mount -a, and then listed the directory inside the LVM logical volume. Success!

I then rebooted the virtual machine and it came up all on its own in normal boot mode, no more emergency mode or boot failure. It acted as if it never fell ill.

To anyone this helped out or if this was a great story to read, I encourage you to follow me on social media or this blog, see you next time!