Home Servers, AArch64, and You (well, me)

Recently, my SoftIron OverDrive1000 arrived, and I’ve finally given myself some time to sit down and implement the project I purchased it for. First, one may ask, why AArch64? The answer lies in my history of installing Linux-based machines at home, the number of PowerPC machines still exceeds the number of x86/x86-64 machines that I have done. So when I got the chance to pick up some AArch64 hardware that came in a regular form factor, I decided to jump on it. The other reason is that, as I have joked with some people, I like to do things the hard way. What do I mean by that? Well, I’ll explain. But first, some background.

The hardware comes with openSUSE Leap 42.2 preinstalled. I decided to take the fact that this is the least familiar distribution to me as a challenge. At the same time, I also decided that it would be worth evaluating other distributions that I’m familiar with, such as CentOS and Debian, on AArch64. Most of what I do with my home server today is related to locally streaming media. So while I had been installing and running various applications directly (making use of crontab’s @reboot keyword to launch them), I decided I should move on to modern best practices and use Docker to isolate, control, and update these applications. Finally, while I’ve been using Rygel for a few months, I want to get back to using Plex Media Server to serve the content. This in turn introduces the constraint of needing to use a 32-bit ARM binary, as Plex currently does not have a 64-bit build available.

The first bit of fun I found was that while the Docker project has community provided support for AArch64, Docker, Inc does not actually provide AArch64 builds. So if you want to have the most up to date Docker installation and wish to have Docker be managed by your distribution packaging manager, there’s some work to be done. The Docker project does have some contrib scripts to create packages of Docker for many distributions, but they’re written assuming that you’re running on x86-64. The good news is that all 3 of the distributions mentioned above do ship a version of Docker. For my needs (a well firewalled internal server), the versions they provide are OK.

Testing on CentOS proved interesting. It was easy to get it installed and running from a spare drive on the real hardware. This was just as easy and boring as advertised. After checking basic functionality, I switched to running it in a virtual machine for the rest of my tests. Here’s where things got difficult. First, as of today, CentOS uses 64K pages rather than 4K pages. This in turn disallows the possibility of having 32-bit binary compatibility. One can recompile the CentOS kernel to change the page size and get 32-bit compatibility working. I did this and it proved straightforward yet time consuming. After confirming that this was enough to enable at least basic 32-bit compatibility, I decided to move on.

Debian was the next distribution that I tried out. While the Debian wiki says that you need a newer image to install on AMD Seattle hardware such as the Overdrive 1000, the latest Jessie images are actually new enough to boot and start the installer. At this point in my experiments I didn’t want do another install on the hardware, so the rest of my tests were done on a virtual machine install. This installation didn’t work out well for me and my needs, unfortunately. While there is a docker-engine package available which is used by 96boards.org, I didn’t want to introduce such a large deviation from upstream Debian. The docker.io package exists as a backport to Jessie, but not for arm64. Further, at the time that I was testing things, Sid was in an incomplete state for the docker.io package. While I was able to resolve the dependencies manually, Docker didn’t want to start. I should note that I’m doing my reviews here out of order slightly. At this point I had openSUSE with Docker running, and didn’t want to further dive into this problem, which I’m sure could be resolved. One last note is that I did test 32-bit compatibility, and it works fine on Debian out of the box.

Which brings me back to openSUSE. The main repository has Docker available, and the kernel is already built with 32-bit compatibility enabled. This was the easiest distribution for getting the first order of problems solved on, as everything just worked, and I was also able to easily set up VMs for testing the other distributions. The biggest hurdle I faced on openSUSE is that I didn’t find a lot of documentation about installing Docker, but instead about setting up various flavours of LXC. Fortunately, Docker is not very host distribution specific, so this was not a big problem.

All of the above distributions, and other efforts such as the Linaro distributions for the Dragonboard, suffer from one more issue with respect to running Docker containers consisting of 32-bit binaries. Namely, that we’re not using binaries that we’ve controlled the build process of, we’re just picking up whatever is in various public containers from Docker. Without getting too deep into the technical details of what instructions AArch64 will emulate, not all instructions required to run “armhf” binaries are required. For example, a number of Docker images are built optimized to the ARMv6 architecture as found in some models of the Raspberry Pi, and these instructions require additional Linux kernel support to be enabled. For more details see here.

So, where does that leave us? Well, first of all, I’ve completed the software side of the desired migration. The new server is doing everything the old server was doing before. As far as my end users go, it’s all working just as well as before too. In doing this migration, I’ve been reminded of the phrase “the more things change, the more they stay the same”. Back during the early Linux on PowerPC days, one would often have to deal with the problems induced by “Linux” really meaning “Linux on 32-bit Intel/x86 compatible CPUs” to some project or maintainer. Today one has to deal with the problem of “Linux” often meaning “Linux on 64-bit Intel/AMD x86-64 compatible systems” with an occasional “Linux on the Raspberry Pi” instead. Both of these assumptions make life interesting at times. With the former, for example, it is possible to have Docker images deal with multiple architectures, but that’s just not done today. I suspect that while it will be done for the core Docker images at some point, one is always going to need to take care when using arbitrary community images, setting aside any other concerns one might have in that area. With the latter assumption, we run into the kernel issue mentioned above, as without the Pi, “armhf” would likely mean “ARMv7” in practice. In the end, the cool thing here is that the distributions have achieved pretty close to parity between x86-64 and AArch64 in terms of the offered software. However, once you start to move out of that garden and into the world at large, you’re going to start to find some oddities. Now, if you’re like me and find the challenge fun, or are looking for a reason to dive deeper into how things work, it’s a great time and a great idea. But if you’re looking for a turn-key solution on an AArch64 platform, things aren’t quite there yet.

Konsulko Group is going to FOSSASIA!

Konsulko Group is attending FOSSASIA 2017 in Singapore on March 17-19. We have a presentation on Linux kernel support for sensor hardware:

Look for us at embedded and kernel talks as well as the hallway track. See you there!