Automating a Factorio Server with Docker Containerization

Factorio Gameplay Trailer Thumbnail

I recently began playing Factorio earlier this year, a game about building and creating automated factories to produce items of increasing complexity. I have had a love for real-time strategy and simulation games since I was a kid, and this game has been a joy to play. Here’s a link to the gameplay trailer of the older 0.16.x branch of the game.

It only seems more enjoyable then to gather friends and family and play this game on multiplayer, with everyone collaborating to build a bigger, faster factory in shorter time. The first step to do that is getting a Factorio server setup. The current 0.17.x branch is the latest version of the game. This is how I went about setting up for that.

Setup a Cloud Server

First, create a cloud server that will be used to install docker and will host the factorio server container. I already have a few servers with a variety of cloud hosting providers, but if you need one, I recommend Vultr Cloud (use this link to receive $50 to test out their platform).

Alternatively, if you use Kubernetes or already have a container platform, you may use that. Keep in mind, Factorio needs to be able to resolve its public IP address from inside the container, in order to let people play on it. If behind a NAT, there may be some difficulties getting the server working, including port forwarding through the firewall. Viewing the server logs after creating the container will help identify if there’s an issue.

The OS I chose for my cloud server was Fedora 30. I’m personally not a fan of Debian/Ubuntu or the DEB-formatted packages, especially how services automatically start after apt-get’ing them. I prefer RPM-formatted packages, but CentOS is too old for what I feel I need, so Fedora is the best choice for me. I’m also not widely supportive of CoreOS, so that was a non-starter.

After the cloud server provider gives you root access to the host, login as root and immediately configure it as you would any of your other servers. At minimum, do a full package update: dnf upgrade -y --refresh --setopt=install_weak_deps=False --best. Also note that it is assumed all further actions will be done as root, if that’s not the case then prefix sudo to each command further below.

Install Docker

The next step is to install docker. The version that comes with the OS is typically too old to be of any practical use, but luckily the Docker Team maintains their own repository with better packages. Follow the Docker Engine Install Guide, or run: dnf install -y --releasever=29 docker-ce docker-ce-cli containerd.io. At time of this post, they don’t have a Fedora 30 version of Docker, so adding --releasever=29 is required during the dnf install step. This extra step shouldn’t be required for too long, once they support the newer Fedora version.

Make sure after installing Docker that you do another full package update (dnf update -y) to ensure any dependent packages it installed get updated to the Fedora 30 release version.

Note: Installing the previous Fedora release’s Docker packages has had no ill effects for me. It’s not recommended to do this for just any package that you can’t install otherwise. I’ve tested this out and it’s been working stable enough for me to justify doing it. Definitely use your best judgment here and do what makes you comfortable, even if that means using Fedora 29 instead for your OS with the cloud server provider.

Once everything’s installed and updated, enable and start docker: systemctl enable --now docker.service.

Install Factorio

This step’s pretty quick and simple. These steps are copied directly from the factoriotools/factorio-docker Readme and will install the latest 0.17.x branch of the Factorio server.

mkdir -p /opt/factorio
chown 845:845 /opt/factorio
docker run -d \
  -p 34197:34197/udp \
  -p 27015:27015/tcp \
  -v /opt/factorio:/factorio \
  --name factorio \
  --restart=always \
  factoriotools/factorio

Note: You can omit port 27015 if you have no plan to use the RCON commands. More info here: https://wiki.factorio.com/Console

The container will start but it won’t be playable yet. We’ll need to do a few more steps before we can start playing on it. Next, shutdown the container: docker stop factorio.

Edit the server settings file here: /opt/factorio/config/server-settings.json. You will need to enter your Factorio username/password, or alternatively your token, and this is also where you’ll decide how to title and describe your new Factorio server. To get your server’s token, go to factorio.com/profile and copy it from there.

Once you’ve sufficiently changed that file, you’re pretty much ready to play. The only thing left would be to alter how the map is generated, but this is optional.

Change the Map Settings (optional)

When the container started, it generated a new map with the settings in /opt/factorio/config/map-gen-settings.json and /opt/factorio/config/map-settings.json.

Modify those two files to your own liking, then delete all zip files under /opt/factorio/saves.

The steps above will cause the server to regenerate a map with the new settings you’ve applied.

Play Factorio

The server’s setup and ready to go. All what’s left to do is start up the container again: docker start factorio.

One last note: If you’re using the Steam version of Factorio, you will need to switch your Factorio installation to the beta release. Right-click Factorio from the Steam Games Library, choose Properties, go to the Betas tab, and in the dropdown select the latest 0.17.x. This will let you match the version of Factorio you just installed on your new server.

You may now launch Factorio and connect to your new server. Share the name of it in the public server list with your friends and family, or ask them to enter the hostname/IP address directly, depending on the settings you chose.

Have fun!

Troubleshooting

If there’s an issue, the factoriotools/factorio-docker repository is a good place to start. There are sections in their Readme for certain scenarios you may encounter.

How gnome-software/systemd software upgrade dropped me to the grub command line, and how I resolved it

Earlier I was messing around in one of my Fedora Linux virtual machines. I’ve had this particular install since Fedora 23 was the latest release. I’ve upgraded it from 23 to 24 and then again from 24 to 25. I completed the upgrade a few weeks back and decided to come back to it. Of course, since it had been a few weeks, after I loaded it up and logged in, GNOME was telling me there were updates available in the Software (gnome-software) application.

Brief warning: this article contains a ton of screenshots as I worked the problem.

Mindlessly, I opened the gnome-software utility and decided to update the system through there. The updates looked pretty benign, whatever. I clicked ‘Restart & Install’ and confirmed. The system rebooted and brought me to the systemd installing updates prompt, awesome. It finished and rebooted once more, however I was immediately dumped to a grub command line. Uh oh.

So I took the #fedora IRC channel on Freenode, explained what had happened, and got some interesting feedback.

First, we tried to identify the problem. We started by trying to figure out why grub command line was coming up immediately while booting rather than the grub boot menu. I found that the grub config file was in fact empty, pretty straightforward then why we’re met with the grub command line.

My initial thought is the gnome-software utility told systemd to reboot for updates, but systemd didn’t install updates correctly. There was probably a kernel update and the grub config got caught in the mess. A little more trial and error with the fellas in #fedora and I was able to tell grub to boot the vmlinuz and initramfs files still visibly present.

Booting didn’t work though. A kernel panic came up while booting, Linux couldn’t mount the root filesystem.

Now I’m wondering if my hard disk is somehow corrupted after the updates. Obviously an update didn’t finish correctly, that much is obvious.

I head back to the #fedora IRC channel and suggest that I try booting the Live CD iso instead to dig around any further, I was tired of messing around in grub’s command line. They agreed this would in fact aid diagnosing the issue, so off I went.

I identified that the hard disk is using a single LVM partition, there are no other partitions on the Master Boot Record (MBR) of the disk. And the LVM partition only has a single logical volume in it called root with the / mount point and is xfs formatted. Pretty strange, and I don’t remember why I chose this layout so long ago.

I decide to remove the Live CD iso from the machine and reboot. What happened next though was pretty weird. I had looked away for a bit to check the IRC channel, when I came back, the machine had booted!

What the hell. Alright, so might as well browse around then. I logged in. I even was greeted with the message that updates were installed successfully!

I decided to open Terminal and verify the disk layout:

Yup. That’s a lvm-xfs partitioned disk. There is in fact no /boot partition, which means the single / partition is the /boot partition. By this point, the people over in #fedora were pretty grateful grub has matured to be able to understand booting LVM partitions, but they were at a loss for words for what was going on as well.

After talking with the #fedora IRC channel some more, I agree to figure out what state causes the machine to drop to the grub command line after upgrading.

I began going through each variable in my tests:

  • Default partitioning format
  • Custom partitioning format
  • Upgrading system using gnome-software
  • Upgrading system using dnf

The default partitioning format is just as you would expect–you load up the Live CD iso with a blank hard disk, you install Fedora to that hard disk using the default partitioning that anaconda chooses for you. No modifications.

The custom partitioning format is a bit different. You start with a single LVM partition and then you create an xfs partitioned with the / mount point:

I and the people in the #fedora IRC both found that installing Fedora with the custom partitioning format and upgrading using gnome-software would cause Fedora to be dropped to the grub command line every time, and only repairable by booting the Live CD iso again and mounting the filesystem so that the xfs journal is replayed after the updates.

Getting to that diagnosis though took several hours, and after confirming with others I opened the bug report over at Red Hat: Bug 1416650 – Upgrading using gnome-software/systemd with lvm-xfs custom partitioning format causes grub boot failure

Hopefully I get an answer back from the Fedora and/or Red Hat teams. Quite a head scratcher at first, but obviously a bug since it’s supposed to be supported and doesn’t indicate otherwise.

Cheers.

Replacing your server is not always necessary after encountering boot failure

First, a little background context before I dive in to how I recovered the virtual machine from boot failure…

  • The host has three physical disks mapped to the virtual machine which I’ll be recovering.
    • One disk is a 256 GB SSD and is unused/unallocated, the other two are 5 TB HDDs paired together via LVM to create a single 10 TB disk inside the guest’s filesystem.
  • The host runs Windows 7 64-bit, runs VirtualBox 5.1.10, and the guest runs Fedora 25 64-bit on Linux kernel 4.8.15-300.fc25.x86_64.
  • The guest uses SystemD and is referred to as ‘seedbox’ in my logs and snapshots.

Earlier this morning, the host completely froze, not even so much as a BSOD. There is a hardware problem with the host’s RAM that I have yet to completely diagnose. I ended up pushing the reset button on the chassis. All running guests were immediately aborted due to my power reset.

The host system rebooted just fine. Upon turning back on all of the guests, the ‘seedbox’ guest entered into SystemD emergency mode while it was booting. Uh oh.

Turns out that when the host rebooted, the physical disk mappings present for the guest were changed by either Windows or the host’s BIOS itself, and VirtualBox was not aware or did not track the difference. The SSD and one of the HDDs were no longer the correct physical disks. The LVM volume group was therefore invalid and auto-mounting the LVM logical volume failed. This was the root cause of boot failure within the guest.

Okay, so we’ve now identified the cause of boot failure. What about recovery? Is all the data corrupt now?

The guest failed to mount the LVM logical volume, which prevented it from possibly writing to the disks, causing corruption. The SSD and HDDs were never mounted by Linux, not even in read-only, so they were preserved during the boot failure. This is a good indicator that the data should be completely intact if we can rebuild the mapping.

Looking into correcting the physical to virtual disk mapping, I found out that due to the shift in mappings, the newly assigned physical disks were actually possibly catastrophic. One of them was the host’s boot drive, which even VirtualBox says NEVER to mount in a guest: “Most importantly, do not attempt to boot the partition with the currently running host operating system in a guest. This will lead to severe data corruption” [VirtualBox Manual 9.9.1]. I quickly commented out the auto-mount for that LVM logical volume and powered off the virtual machine, then looked into correcting the mapping via the host.

In VirtualBox, if you want to map physical disks to virtual ones, you need to run a utility called VBoxManage as root/administrator, and call some internals of the utility that are vaguely documented in section 9.9.1 in the advanced topics chapter of the VirtualBox manual (link). This is non-trivial for those just getting started with virtual machines and disk management, since there’s a high risk for corruption, as the manual indicates. If you map your host’s physical boot disk to the guest, then try to boot off of it, you’ll almost always cause severe corruption, this is what the manual is warning everyone about.

So off I went. I opened up the Command Prompt in administrator mode on the host and went to the directory the virtual disk files were in. In the VirtualBox window, I removed the disks from VirtualBox’s Media Manager, then switched back to Command Prompt. I then replaced the .vmdk files with new copies that had corrected physical drive ids. I then re-added the virtual disk files into VirtualBox, attached them to the seedbox guest, and booted up the guest.

The guest booted up successfully. Since I uncommented the failing drive configuration, it was able to boot normally, with only services relying on that mountpoint failing to run correctly. The core system was intact.

I uncommented the auto-mount line for the LVM logical volume I commented out earlier, re-mounted all auto-mounts via mount -a, and then listed the directory inside the LVM logical volume. Success!

I then rebooted the virtual machine and it came up all on its own in normal boot mode, no more emergency mode or boot failure. It acted as if it never fell ill.

To anyone this helped out or if this was a great story to read, I encourage you to follow me on social media or this blog, see you next time!