Building a DIY SOHO router, Part 3

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 3

In part two of this series I created a local configuration layer for OpenEmbedded, and had the build target core-image-minimal producing an image. The image that was produced wasn’t really a router, but did let us bring up our board and look around. In this article, I’m going to create a custom image and populate it with additional software packages configured to my  requirements. I’m also going to get started using Over-The-Air (OTA) software updates on the device.

Now that I’ve proven that the image works on the hardware, I can really get down to implementing the project of making a router.  While I could continue to add things to core-image-minimal, it really makes sense at this point to stop and create my own image. Since I want something relatively small, I will still start with core-image-minimal as the base.  Moving back over to meta-local-soho, I’m creating the recipes-core/images directory and then populating core-image-minimal-router.bb with:

require recipes-core/images/core-image-minimal.bb

DESCRIPTION = "Small image for use as a router"

IMAGE_FEATURES += "ssh-server-openssh"
IMAGE_FEATURES += "empty-root-password allow-empty-password allow-root-login"

IMAGE_INSTALL += "\
    "
MENDER_STORAGE_TOTAL_SIZE_MB = "4096"

This tells bitbake that it must have core-image-minimal.bb available and to include it. I then provide a new DESCRIPTION to describe the new image. Next, I include a number of new features in the image. First, I’ll use the normal hook for adding a SSH server. Then I’ll add a line of features for development mode that I’ll remove later.  These features, as their names imply, allow for root to login without a password. This is quite handy for development and quite unwise for production. I’ll circle back and remove these development features later. Next, I give myself an empty list of additional packages to be filled out later. Finally, I tell Mender that it has 4096 megabytes of disk space to work with.  I’m going to hide space from Mender so that I can entirely control that part myself instead. At this point I can build core-image-minimal-router, and it will complete very quickly as I’ve not yet added any packages that have not been previously built. So it’s time to once again git add, git commit, and bitbake these changes.

At this point, I want to flash the new image onto the device and boot it up. The reason for this is that the new image can be used with Mender to test any subsequent image builds. The system is now functional enough to support delivering new image updates via Mender, so it’s good to get into the habit of using the OTA update workflow. It also forces me to treat the device as if it’s really stateless. I’ll talk about how to apply an OTA update when I make the next set of changes.

Now it’s time to begin adding content to the custom image. The first thing I’m going to do is borrow some logic from packagegroup-machine-base. I don’t want to use this packagegroup directly because it will cause bitbake to build a lot of extra stuff that I don’t end up installing. This is due to the fact that it’s part of packagegroup-base.bb (because it’s needed to resolve dependencies of other parts of the packagegroup). Instead, I’m going to add:

    ${MACHINE_EXTRA_RDEPENDS} \
    ${MACHINE_EXTRA_RRECOMMENDS} \

to IMAGE_INSTALL so that any additional machine-specific functionality that’s been specified is installed to the image. Next, I’ll add in kernel-modules to the list so that all of the modules that have been built for the kernel are installed to the image. This will be a lot easier than listing out every module I may need, especially for later on when it comes to various firewall rules I want to use.  On top of all of this, I also want to drop in a bunch of full-versions of common packages I use, and then let busybox fill in the rest.

    bind-utils \
    coreutils \
    findutils \
    iputils-ping \
    iputils-tracepath \
    iputils-traceroute6 \
    iproute2 \
    less \
    ncurses-terminfo \
    net-tools \
    procps \
    util-linux \

Almost everything in this list can be tweaked as desired. There are a couple items that serve a critical purpose and deserve an explanation:

  1. systemd calls out to $PAGER for many functions, including browsing logs with journalctl. If I don’t have the full version of less available, I won’t have a fully functional pager and browsing the output is extremely difficult.
  2. I don’t use xterm for my terminal emulator anymore so I want ncurses-terminfo installed. This ensures that the right terminfo is available and terminal output is correct.

At this point it’s time for a git add, git commit, and then a bitbake of our image too.

Now that I have a new image with additional content to try out, I want to put it on the device and confirm things work. As mentioned before, I’m using Mender in standalone mode since I have a single deployed device.  It’s very simple to serve the new image and then apply it. On the build machine, I do the following (change qemux86-64 to match the machine in use):

$ (cd tmp-glibc/deploy/images/qemux86-64; python3 -m http.server)

And then on the device:

# mender -rootfs http://build-server.local:8000/core-image-minimal-router-qemux86-64.mender
... wait while it downloads and applies ...
# reboot

Once the device comes back up, I’ve logged back in, and confirmed I’m satisfied with my changes, I do:

# mender -commit

This will mark what I am now running as the valid rootfs. However, if the device didn’t boot up or I couldn’t log in, I would simply not commit the changes. To do that I would then just reboot or otherwise power-cycle the device. If I don’t commit the changes to Mender then I get an automatic rollback to the previous install.  Of course, it’s also possible to use any HTTP server on the build machine.

At this point, it’s time to iterate over adding a number of different features that require little more than adding to IMAGE_INSTALL. Since I’ve talked about LXC, I need to add in lxc and gnupg (for verification of containers used from the download template). Once that’s added, I do the git add, git commit, bitbake, and then mender -rootfs cycle again and confirm LXC is working. One thing I noticed when doing this was that containers didn’t autostart because the service isn’t enabled by default.  Since I’m keeping this stateless, I changed that behavior with a bbappend file.  I also ended up installing e2fsprogs-mke2fs to be able to further partition my device to give LXC some room to work with.  This also means that I needed to have base-files provide the fstab that matches my setup, rather than the stock one.  Another small thing to cover is if your hardware does, or does not have a hardware random number genreator available.  If you do have one, you should pull in rng-tools on the image.  If you don’t have one however, you should install haveged to help feed the entropy pool instead.

Now I need to enable a functional access point. This is the first case where it’s really non-trivial to write up the config file to use, so it’s done a little bit differently.  The first step is to install hostapd and iw and boot that.  Now, on the device, edit /etc/hostapd.conf and iterate on editing and testing it on the device until everything is set up as desired. The iw tool can be helpful here to do things like perform a site scan to see what frequencies are already in use.  Once I’m done with the config, I copy the file out from the target and over to my build server with scp as /tmp/hostapd.conf. Then it’s time to make it stateless:

$ mkdir -p recipes-connectivity/hostapd/hostapd
$ cp /tmp/hostapd.conf recipes-connectivity/hostapd/hostapd/

And then I edit recipes-connectivity/hostapd/hostapd_%.bbappend to look like this:

FILESEXTRAPATHS_prepend := ":${THISDIR}/${PN}"

SRC_URI += "file://hostapd.conf"

do_install_append() {
    install -m 0644 ${WORKDIR}/hostapd.conf ${D}${sysconfdir}
}

SYSTEMD_AUTO_ENABLE_${PN} = "enable"

This will do two things. Everything except that last line is to tell bitbake to look in my layer for hostapd.conf and then to install it. The last thing is that now that we have a configured AP we want to start it automatically so have it be an enabled systemd service. Now it’s time once again for the git add, git commit, and so forth cycle.

The next step is to do the same kind of thing to dnsmasq. The good news that this time, the dnsmasq_%.bbappend file only needs one line:

FILESEXTRAPATHS_prepend := ":${THISDIR}/${PN}"

This is because the rest of the recipe already knows to grab dnsmasq.conf from a local file. In the case of my network, I need to pass in a few special options to some DHCP clients and have certain clients be given certain IP addresses, so I’ve gone with dnsmasq as my light-weight, but still fully featured IPv4 configuration server. I could have just as easily gone with ISC DHCPD instead, and it would look much the same as the above.  Conversely, if I didn’t need those few extra rules, I could just let systemd handle DHCP serving.  I left out IPv6 from my statement there as I am letting systemd handle that.

The only thing missing at this point from a router, aside from turning off developer mode features, is to add in a firewall. There are a few ways to go about this.  I already have systemd handling one of the aspects that is often associated with a firewall, setting up IPv4 NAT.  If the only other thing I needed on top of this is to shut the rest of the world out, I can use ufw and potentially even leverage its features that allow for adding iptables commands directly for slight enhancements.  While I have gone that direction for some projects, it’s not a good fit for this one. Instead, I chose to go with arno-iptables-firewall because I’m going to have a more complex setup. The process of customizing the firewall configuration is similar to how I customized hostapd and dnsmasq. That is, I iteratively configure it on the device, test for functionality, and copy the configuration files to my host.  This time, however, the arno-iptables-firewall_%.bbappend will look a little different:

FILESEXTRAPATHS_append := ":${THISDIR}/files"

SRC_URI += "file://firewall.conf \
            file://custom-rules \
"

do_install_append() {
    install -m 0644 ${WORKDIR}/firewall.conf \
    ${D}${sysconfdir}/arno-iptables-firewall/
    install -m 0644 ${WORKDIR}/custom-rules \
    ${D}${sysconfdir}/arno-iptables-firewall/
}

I have two files this time. The first one is the main config file, and the second one is the file that contains my custom rules. This is only necessary because I have a number of custom rules, otherwise it could be omitted.

At this point, looking back at the feature list I laid out in part one, I believe I can check all of my items off now.  I have the following all operational:

  1. access point
  2. firewall
  3. IPv4 and IPv6 network configuration
  4. containers 
  5. OTA software update

I’m building all of my software as hardened as my compiler will allow.  There’s very little state on the router itself to worry about backing up, and everything else is handled by my build server being backed up.  I’m confident in my OTA configuration as I’ve been using it for some time now in the development workflow. I’ve also tweaked the installed package list so that all of my favorite sysadmin tools are available.

At this point, it’s time to lock things down. First up, it’s time to go back to core-image-minimal-router.bb and remove that second line worth of IMAGE_FEATURES. Instead, I’m going to create a new local-user.bb recipe with my own user and SSH key. After listing local-user in IMAGE_INSTALL, I copy meta-skeleton/recipes-skeleton/useradd/useradd-example.bb to somewhere in meta-local-soho, and change it to look like this:

SUMMARY = "SOHO router user"
DESCRIPTION = "Add our own user to the image"
SECTION = "examples"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"

SRC_URI = "file://authorized_keys"

S = "${WORKDIR}"

inherit useradd

# You must set USERADD_PACKAGES when you inherit useradd. This
# lists which output packages will include the user/group
# creation code.
USERADD_PACKAGES = "${PN}"

USERADD_PARAM_${PN} = "-u 1200 -d /data/trini -r -s /bin/bash trini"

do_install () {
install -d -m 0755 ${D}/data/trini
install -d -m 0700 ${D}/data/trini/.ssh

install -m 0600 ${WORKDIR}/authorized_keys ${D}/data/trini/.ssh/

# The new users and groups are created before the do_install
# step, so you are now free to make use of them:
chown -R trini ${D}/data/trini
chgrp -R trini ${D}/data/trini
}

FILES_${PN} = "/data/trini"

# Prevents do_package failures with:
# debugsources.list: No such file or directory:
INHIBIT_PACKAGE_DEBUG_SPLIT = "1"

Now, there’s one slight problem. I added myself with a user under /data which is excluded from Mender updates. The good thing is I get persistent history and so forth.  The bad thing is I’m not installed there yet.  So I either need to re-flash one last time or manually copy the files over from the filesystem image to the device before I reboot.  Finally, I need to enable myself to use sudo. In addition to adding sudo to IMAGE_INSTALL I also need to either tweak the sudo recipe so that /etc/sudoers.d/ is looked under, tweak it so that anyone in the wheel group can use sudo and add a wheel group, or borrow the example from meta/recipes-core/images/build-appliance-image_15.0.0.bb and do the following in core-image-minimal-router.bb:

# Take the example from recipes-core/images/build-appliance-image_15.0.0.bb
# on adding more sudoers
fakeroot do_populate_poky_src () {
    echo "trini ALL=(ALL) NOPASSWD: ALL" >> ${IMAGE_ROOTFS}/etc/sudoers
}
IMAGE_PREPROCESS_COMMAND += "do_populate_poky_src; "

With all of that built, deployed, and unit tested, it’s time to go live.  My SOHO router is done and ready for production.  It’s now on me to make sure this stays up to date, which in some ways is a lot better than the alternative.  With my previous router, I only had an non-volatile RAM dump specific to the model of router as a backup. I now have my complete configuration containing firewall rules, DHCP options, and more saved. Since starting on the project I have even braved a few OTA updates and had minimal downtime.

This concludes the walk through of building a SOHO router with OpenEmbedded. In the final part of this series, I will describe some of the lessons I learned while designing and implementing this project.

[Go to Part Four of the series.]

Building a DIY SOHO router, Part 2

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 2

In part one of this series I explained some of my motivations for this project. Now it’s time to start on implementing the project itself. At this point I’m going to assume the reader has basic familiarity with using OpenEmbedded.  Otherwise, the Yocto Project (YP) quickstart guide and OpenEmbedded getting started pages are useful to bring yourself up to speed.  I assume that you’ve followed these instructions on how to prepare your build host and gone so far as to have completed a build for some target previous to this.  In this guide, I’m going to do my best to follow common best practices. Whenever I deviate from best practices, I’ll explain why we want something a bit different. After all, best practices are supposed to be taken as guidelines, not absolutes. Finally, I’m going to be working against the Yocto Project thud code-name release as that’s what is current as of this writing.

The first thing I’m going to do in my project is create a build directory now so that I can easily get access to various tools.  The next thing I’m going to do is use those tools to make a layer to store our configuration in.

$ . oe-core/oe-init-build-env
You had no conf/local.conf file. This configuration file has therefore been
created for you with some default values. You may wish to edit it to, for
example, select a different MACHINE (target hardware). See conf/local.conf
for more information as common configuration options are commented.
...
You can also run generated qemu images with a command like 'runqemu qemux86'
$ bitbake-layers create-layer ../meta-local-soho NOTE: Starting bitbake server... Add your new layer with 'bitbake-layers add-layer ../meta-local-soho' $ bitbake-layers add-layer ../meta-local-soho NOTE: Starting bitbake server... $

Be sure to change oe-core to wherever the core layer was checked out.  Now that the layer has been created, let’s go ahead and start by putting it into git. Why? I am a firm believer in “commit early and commit often” as well as “cleanup and rebase once you’re done”. To me, one of the big selling points of git is that you can track your work incrementally, and when you notice unexpected breakage later on you can easily go back in time and locate the bugs.

$ cd ../meta-local-soho
$ git init .
$ git add *
$ git commit -s -m "Initial layer creation"
$ git branch -m thud

With the first commit in place, it’s time to start customizing the layer. We don’t need the example recipe, so let’s remove it.

$ git rm -r recipes-example/example
$ git commit -s -m "Remove example recipe"

Next, it’s time for a more substantive set of customizations. I’ll start by editing the README. Why? To start with, I’m going to list all of the layers I know of that are dependencies at this point. The README will also be a handy place to note which physical port is for WAN, which port(s) are for LAN, and the network devices that each port is associated with. For now, I add the following to the README:

  URI: git://git.openembedded.org/meta-openembedded
  branch: thud
  layers: meta-oe, meta-python, meta-networking, meta-filesystems

  URI: git://git.yoctoproject.org/meta-virtualization
  branch: thud

  URI: https://github.com/mendersoftware/meta-mender
  branch: thud

Now that I’ve documented these requirements, I’ll also enforce them in code. I edit conf/layer.conf and document these requirements too. The end of the file should look like:

LAYERDEPENDS_meta-local-soho = "core"
LAYERDEPENDS_meta-local-soho += "openembedded-layer meta-python"
LAYERDEPENDS_meta-local-soho += "networking-layer filesystems-layer"
LAYERDEPENDS_meta-local-soho += "virtualization-layer mender"
LAYERSERIES_COMPAT_meta-local-soho = "thud"

Since there are so many layers in use, I will make use of the TEMPLATECONF functionality so that the build directory will be populated correctly to start with. Next, I copy over meta/conf/bblayers.conf.sample, meta/conf/local.conf.sample and meta/conf/conf-notes.txt from the core layer over to the conf directory and commit them without change. Why? This will isolate my local edits later on and make my life easier next year when I decide it’s time to update to a current release. After I’ve committed those files, I the edit conf/bblayers.conf.sample and change it to look like:

BBLAYERS ?= " \
  ##OEROOT##/meta \
  ##OEROOT##/../meta-openembedded/meta-oe \
  ##OEROOT##/../meta-openembedded/meta-python \
  ##OEROOT##/../meta-openembedded/meta-networking \
  ##OEROOT##/../meta-openembedded/meta-filesystems \
  ##OEROOT##/../meta-virtualization \
  ##OEROOT##/../meta-mender/meta-mender-core \
  ##OEROOT##/../meta-local-soho \

Note that this assumes a directory structure where I’ve put all of the layers I will use in the same base directory. If I didn’t do that then I would need to adjust all of the paths to match how to get from the core layer to where the layers are stored. Now I want to use git add and commit all of these changes, so I can move on to the next step.

I will enable systemd next, and while there’s a number of places I could do this, including going so far as to create my own “distro” policy file, for this article I’ll just use conf/local.conf.sample to store these changes. While the core layer’s conf/local.conf.sample.extended has an example on switching to systemd, I’ll do it slightly differently. At the end of the conf/local.conf.sample insert the following, git add and then commit:

# Switch to systemd
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = ""
VIRTUAL-RUNTIME_syslog = ""
VIRTUAL-RUNTIME_login_manager = "shadow-base"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"

This differs from the core example in a few ways. First, I blank out pulling in an initscripts compat package. While this can be useful, for these images it’s going to end up being redundant. Next, I blank out a syslog provider as I’ll be letting systemd handle all of the logging. Finally, while busybox can be available and provide the login manager, I’ll be using shadow-base instead. This particular change is something now done upstream and will be in later releases, so keep that in mind for the future.

The next functional chunk for conf/local.conf.sample is enabling support for virtualization through meta-virtualization. That layer has its own well documented README file, and taking the time to go over it is a great idea. In my configuration, I’m just going to enable a few things that give me the minimal functionality. So add, git add and git commit:

# Add virtualization support
DISTRO_FEATURES_append = " virtualization aufs kvm"

When talking about system security there are many aspects. One aspect is to harden the system as much as possible by having the compiler apply various build-time safety measures that turn certain classes of attack from “exploit the system” to “Denial of Service by crashing the application”, or even “this is wildly unsafe code, fail to build”. To enable those checks in the build, add, git add and git commit:

# Security flags
require conf/distro/include/security_flags.inc

There’s one last bit of functionality I want in the conf/local.conf.sample file, and that’s support for Mender. How do I go about that? That’s going to depend a bit on what hardware you’re going to do this project on, as it’s a little bit different for something like the APU2 than on an ARM platform. Fortunately, there’s great documentation on how to integrate Mender here. While reading over all of the information there is important, it’s best to focus on the Configuring the build section to understand all of the required variables. In fact, now is a good time to talk more about how I’m going to use Mender in this particular setup. Looking at the overall Mender documentation, it’s very flexible. For this very small deployment scenario, I’ll use standalone mode rather than managed mode. This lets me skip all of the things about setting up a host of other services to make upgrades happen automatically. So, we’ll follow the instructions for configuring Mender in standalone mode. The next thing to touch on is how to handle persistent data. One way to do this would be to make sure that anything which needs to be manually customized or that will be persistently changed at runtime is written somewhere under /data on the device rather than at its normal location. This is because the normal location is going to change when I apply a new update but /data will always be the same. For a lot of deployments, this idea works best, as there are likely many users, and beyond our initial configuration the end user will make changes I know nothing about. For my use case, I am the user, and making the system as stateless as possible will in turn make backing the system up as easy as possible. So instead I will be modifying recipes to include our local configuration when needed. Having an easy to access and modify backup of the various configuration files and so forth will make my life easier in the long run if, for example, something happens to the hardware and it needs to be replaced.

Now that I have everything configured, it’s time to create the build directory. This will let me take advantage of all of the configuration work I just did. At this point, I can go back to the shell, leave meta-local-soho and do this:

$ TEMPLATECONF=../meta-local-soho/conf . oe-core/oe-init-build-env build-router

Again, be sure to change oe-core to wherever the core layer was checked out. Running cat conf/bblayers.conf will show how the changes that were made were now expanded.

What to do next? Well, that depends a little on what I already know about the system. The next step in this project is to configure systemd to take care of creating the networks. If I knew what all of the devices will be named, I could move on to that step. But if I didn’t know, or wasn’t sure, the next step is to build core-image-minimal. That’s as easy as doing:

$ bitbake core-image-minimal

and waiting for the final result, assuming that local.conf.sample file was configured to default to the appropriate target MACHINE. Otherwise I’ll need to pass that in on the command line above. Once that completes, take the appropriate image and boot it. The reward should be a root login prompt.  In this configuration, there’s no root password right now. Login and do:

# ip link

and this will show me what the interface names are. Assuming there is a DHCP server somewhere already, I can then use udhcpc -i NAME after plugging in an Ethernet cable to see which interface is which and note it down for the next step.

Now that I have my network interface names, I can configure systemd to handle them. In my specific case I have 4 Ethernet ports, and coincidentally the one closest to the physical console port (enp1s0) is the one I wanted to call the WAN port. This lets me put the other 3 ports into a bridge for my LAN. Now, to configure these interfaces, I’m going to dump some files under /etc/systemd/network/, and I will use the base-files recipe to own these files, rather than systemd itself. Why? Whenever a change forces a rebuild of systemd that forces a rebuild of a lot of other packages, so this allows me to isolate that kind of build churn. Now I go to meta-local-soho and enter the following:

$ mkdir -p recipes-core/base-files/base-files
$ cd recipes-core/base-files/base-files

I’m going to create four different files. Note that all of the filenames are arbitrary and are meant to help poor humans figure out what file does what. If another naming scheme is helpful, it can be used just as easily. First I’ll take care of the WAN port by creating wan-ethernet.network with the following content:

# Take the eth port closest to the console port for WAN
[Match]
Name=enp1s0

[Network]
DHCP=yes

Next, I’ll create our bridge for the other ports and this is done with two files. First I need a lan-bridge.network with:

# Bridge the other 3 remaining ports into one.
[Match]
Name=enp2s0 enp3s0 enp4s0

[Network]
Bridge=br0

Second I create br0.netdev with:

[NetDev]
Name=br0
Kind=bridge

And now I have a bridge device that I can use to manage all of our LAN ethernet ports. Later I’ll even put the AP on this bridge for simplicity. Now that I have a bridge, I need to configure it, so create bridge-ethernet.network and add:

[Match]
Name=br0

[Network]
Address=192.168.0.1
IPForward=yes
IPMasquerade=yes
IPv6AcceptRA=false
IPv6PrefixDelegation=dhcpv6

[IPv6PrefixDelegation]
RouterLifetimeSec=1800

There’s a lot in there, but it can be broken down pretty easily. Working my way up from the bottom, the first portion is needed to have the IPv6 address on the WAN port pass along what’s needed to the LAN to have devices configure themselves. Since we still live in a world with IPv4, I need to enable masquerade and forwarding. Finally, I configure a static IP for the interface that matches to br0. I haven’t talked about configuring the LAN for IPv4 yet. This is going to be a bit more complex and while systemd supports a trivial DHCP server, I want to do more complex things like assign specific addresses, so that will come later.

Finally, it’s time to make use of these new files. I’ll head up a level and back to recipes-core/base-files and create base-file_%.bbappend with the following:

FILESEXTRAPATHS_prepend := ":${THISDIR}/${PN}"

SRC_URI += "file://wan-ethernet.network \
            file://lan-bridge.network \
            file://bridge-ethernet.network \
            file://br0.netdev \
" do_install_append() { # Add custom systemd conf network files install -d ${D}${sysconfdir}/systemd/network # Add custom systemd conf network files install -m 0644 ${WORKDIR}/*.network ${D}${sysconfdir}/systemd/network/ install -m 0644 ${WORKDIR}/*.netdev ${D}${sysconfdir}/systemd/network/ }

What this does is to look in that directory where I created those 4 files, add them to the recipe and then finally install them on the target where systemd will want them. At this point, I can git add the whole of recipes-core/base-files, and git commit.

There’s one last thing I want to configure right now. I’m going to have systemd handle things like turning on NAT, and doing some other basic iptables work. As a result, I need to enable that part of systemd. Another thing I’ll want to do here is work around a current systemd issue with respect to bridges. To do so, I need to make the directory recipes-core/systemd/ and create the file systemd_%.bbappend in that directory with the following:

do_install_append() {
    # There are problems with bridges and this service, see
    # https://github.com/systemd/systemd/issues/2154
    rm -f ${D}${sysconfdir}/systemd/system/network-online.target.wants/systemd-networkd-wait-online.service
}

PACKAGECONFIG_append = " iptc"

The first part of these changes is to work around the github issue mentioned in the comment. While it’s quite frustrating that the issue in question has been open since the end of 2015, I can easily work around it by just deleting the service for my use case. The second part is to add iptc to the PACKAGECONFIG options, and in turn that part of systemd will be enabled and it can support iptables so the IPMasquerade flag above will work.

At this point, I can now build core-image-minimal again and have a system that will automatically bring up the network. However, it is not a router yet. In the next part of the series I will create a new image that is intended to be a router and customize that.

[Go to Part Three of the series.]

Looking forward to the OSLS Half Moon Bay

This week, Konsulko Group CEO Pete Popov will be attending the Linux Foundation’s Open Source Leadership Summit in Half Moon Bay, California. The OSLS has always been a premier forum for open source leaders convene to drive digital transformation with open source technologies, and learn how to collaboratively manage the largest shared technology investment of our time. An intimate event, OSLS fosters innovation, growth and partnerships among the leading projects and corporations working in open technology development. Hope to see you there.

Building a DIY SOHO router, Part 1

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 1

I spend my days working on embedding Linux in devices where the end user doesn’t typically think about what’s running inside. As a result, I became motivated to embed Linux in a device that’s a little more visible, even if only to myself. To that end, in this series of articles I will discuss how to build your own SOHO router using the Yocto Project build system, OpenEmbedded.

I realize that many people may be asking the question, “Why build our own router when there are any number of SOHO routers available on the market that are specifically built using good Open Source technologies?”. Commonly, when this question is answered in other guides the major reasons for Roll Your Own (RYO) tend to be better performance and enhanced security. While these are both true, that’s not the primary motivation behind this series. While projects like pfSense and OpenWrt are wonderful for a “turn-key solution”, they don’t meet one of my primary requirements. My requirement is to use my favorite Linux distribution creation tools, and gain a stronger understanding of the inner working of these tools when building a system from the ground up. This is why I’ve chosen to build my router with OpenEmbedded.

What are the Yocto Project and OpenEmbedded? To quote from the former’s website:

The Yocto Project (YP) is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture.

The OpenEmbedded Project (OE) is the maintainer of the core metadata used to create highly flexible and customized Linux distributions and a member of the Yocto Project. As a long time developer and user of YP and OE, these projects have become my favorite tools for creating customized distributions. It’s also something that we frequently use and support commercially at Konsulko Group.

Let’s consider what software is required to meet the functional requirements of the SOHO router. My high-level requirements are:

  • Wired/Wireless networking – basic network connectivity
  • Firewall – necessary when a device is on the Internet
  • Wireless access point – required to support the end user’s many WiFi devices
  • Over-The-Air (OTA) software update – My initial software load will be improved and bug fixed in the same manner as any commercial product and requires a simple update path

OpenEmbedded provides all of these features as well as other advanced features I’d like to leverage such as container virtualization.

With the question of software features settled, the next issue is hardware selection. One of the major advantages of OpenEmbedded is that it runs on a wide range of processors and boards, so there are many options. Depending on one’s specific use case, one option is to use any x86-64 based system with two or more ethernet ports. For example, the SolidPC Q4 or the apu2 platform. Another direction would be to use an ARM CPU and look at the HummingBoard Pulse, NXP i.MX6UL EVK, or even the venerable Raspberry Pi 3 B, and making use of a USB ethernet adapter for secondary ethernet. If you have a specific piece of hardware in mind, so long as there’s Linux support for it, you can use it. I did pick something from the above list for my specific installation. However, this guide is intended to be hardware agnostic and so I will highlight any hardware-dependent portions along the way.

Now it’s necessary to further develop detailed requirements and expectations that we have for the router. First, the device is expected to be able to perform all of the standard duties of a modern SOHO router. It’s not just a WiFi access point and IPv4 NAT. It also must handle complicated firewall rules, and support public IPv6 configuration of the LAN (when the ISP supports IPv6). This is an advanced router, so it’s not enough to simply pass along the ISP-defined DNS servers. I may like to push my outbound traffic, with a few exceptions for streaming services, perhaps, through a VPN. OpenEmbedded, via various metadata collections stored in layers, supports all of these features with its core layer and a few of the main additional layers.

I’m selecting Mender for self-managed A/B style OTA upgrades. Why? I don’t have redundant routers, and I’d like to be able to both experiment with enabling new features and updates (such as a new Linux kernel version or other core component), firewall changes, and minimize downtime if anything goes wrong. Neither myself nor my users (my family) are tolerant of much in the way of Internet outages, so the ability to fall back to a known good state with just a reboot is a major feature. I want the router to be kept as current as possible, and with A/B style updates there’s much less state to maintain from release to release. As a result, the router installation will be made as stateless as possible. If a component parameter is configured by the router administration, it comes in the installation image. This will not only help with rollback, but also make it much easier to back up the router. Having the exact config file for something like dnsmasq is a lot more portable than an nvram export that’s tied to a specific hardware model, just in case of lightning strikes.

As a framework component for enabling some advanced router features, LXC will be included via the meta-virtualization layer to support containers. I realize that many people will be asking, “Why would anybody want containers on a router?”. The answer is, quite simply, that there are many good examples of router-appropriate software that’s well managed by containers rather than directly on the installation itself. In this guide, I use the example of pi-hole as an advanced router feature that I want to enable. A container allows us keep it current in the manner which the project itself recommends. The container also supports the goal of keeping a minimum amount of state on the device that must be backed up elsewhere. Could Docker have been used here instead? Potentially, yes. Due to some peculiarities of the IPv6 deployment by my ISP, LXC is much easier to deploy than Docker.

Finally, lets talk about security. For this first guide, what that means is that YP does its best to keep software up to date with respect to known security issues as well as making it easy to enable compiler-generated safeguards such as -fstack-protector-strong and string format protections. On top of that there are layers available which support various forms of owner-controlled measured boot and various Linux Security Models such as Integrity Measurement Architecture (IMA). As the former is hardware dependent we’re going to leave that to a follow-on series as the specific hardware I’m using lacks certain hardware components to make that useful. As for IMA, it can be difficult to combine that with containers so it too will be covered in another series.

[Go to Part Two of the series.]

The Year in Review

2018 has been a very important year for Konsulko Group.

We are privileged to work with outstanding customers, helping them build the software for exciting and essential devices and vehicles. From Level 5 autonomous taxis and open source automotive platforms to innovative consumer devices, from lifesaving medical devices and advanced robotic surgery tools to the high-end networking equipment that powers the Internet, these are all products that impact and even shape our lives, now and into the future.

In addition to our commercial activity, we continued our community work in key open source projects, including the Linux kernel, Yocto Project, OpenEmbedded and Automotive Grade Linux, and participated at open source conferences around the world, like FOSDEM, ELC/ELCE, FOSSASIA, TuxCon and OpenFest.

We accelerated our work with Automotive Grade Linux and the Linux Foundation, supporting AGL development, demos and member meetings in North America, Europe and Japan, as well as presenting multiple technical talks at all four Embedded Linux Conferences, developing the AGL Deep Dive workshop and providing expert training, both as a Linux Foundation Authorized Training Partner and via the Embedded Apprentice Linux Engineer program found at leading conferences (e-ale.org).

Finally, we continued to grow our company, welcoming new engineers, increasing the size of our European development center in Bulgaria and opening a new branch in Sweden.

Hopefully, 2018 has been merely a prelude to great things coming in 2019. We look forward to working with all of you in the coming year.

2 new articles help engineers new to Automotive Grade Linux

Konsulko Group’s Leon Anavi has made a couple of recent contributions to the Automotive Linux (AGL) wiki:
* ConnMan (how to use the cli tool to connect to a network)
* Media Browser and Player (playing songs from a USB stick)

We hope that this technical information will help beginners to get started easier and faster with AGL. We encourage you to participate in the AGL wiki. Feel free to update and improve either or both articles.

Don’t be afraid to get help with the basics

Fifteen years ago, we used to say that getting an embedded Linux kernel and drivers successfully running on your target hardware got you about 10% of the way to your finished product. Now, thanks to OpenEmbedded and the Yocto project, you can do a lot better than that, but there are usually a few technical challenges along the way.

At Konsulko Group, we’ve seen many of these issues (often repeatedly) in the world of Yocto/OE, and we are often called upon by our clients for this “basic” work. If you are moving to new hardware or building the next generation of your device, we can help with your hardware bring-ups, “upreving” your software stacks, porting your unique OS work, determining if rearchitecting is necessary, and providing on-going maintenance.

Don’t hesitate to contact us, no matter how large or small your technical challenge. Chances are that our past two decades of experience can solve your problems very efficiently, so you can spend your time focussing on the things that make your product unique.

What We Learned from Jim Ready

When Jim Ready passed away at the end of December, he left a legacy in the world of software development that continues to this day and will continue for many years to come. He drove the commercial acceptance of Linux as an embedded operating system in its early days, taking it all the way from “how can this work?” to widespread adoption across one industry after another.

Jim was also an articulate (if occasionally reluctant) spokesman for both Embedded Linux as a community and using the open source methodology for commercial software — the symbiotic parts of a never-ending process.

With his pioneering work with VRTX and Ready Systems, Jim was already well respected in embedded systems when he founded MontaVista Software in the final months of the last century (1999) with the Internet bubble bursting and Linux still very young (and very PC and server focused).

He had the crazy idea that open source software could be a viable commercial solution for every conceivable embedded device. This was a huge leap, basing a business not on the tightly controlled “precious jewel” world of RTOSs, but instead on the constantly evolving, messy organism that was the Linux community. That required looking at commercial software development in the open source model, moving beyond static releases, updates and lifecycle programs, focusing on process instead of product, on outcomes instead of systems, on management and integration instead of authorship.

Many of us at the Konsulko Group worked at MontaVista in those early years. We experienced first hand the paradigm shift and the hard work it took to make it happen. Jim gave us the chance to do something we loved (and in most cases already pursued as a hobby) and turn it into our life’s work. Almost two decades later, through the lasting relationships we formed and the lessons we learned working with him, we are thankful for the opportunities, mentoring and friendship Jim gave us, and that’s part of his legacy as well.

YAML and Devicetree

Introduction

This document attempts to explain the rationale behind using a YAML based data model instead of the standard devicetree source (DTS). It assumes a working knowledge of devicetree, so readers are expected to have perused the devicetree specification located here.

Devicetree and its underlying concepts

While device tree deals with describing hardware devices, at its core it is a method of declaring a hierarchical structure as defined in the Devicetree Specification:

“A devicetree is a tree data structure with nodes that describe the devices in a system. Each node has property/value pairs that describe the characteristics of the device being represented. Each node has exactly one parent except for the root node, which has no parent.”

This structure is familiar to anyone with a passing knowledge of programming languages with rich data structures: nodes can be hashes keyed by their name, properties can be either scalars or sequences of scalars, and labels of nodes and phandles can be references/pointers.

Unfortunately, device tree is not orthogonal enough for this mapping to work. Namely, properties are irregular in the following ways:

  1. Boolean values cannot be part of a sequence, since a named property is defined as false if it doesn’t exist in a node and as true otherwise.
  2. Phandles are encoded as integer (cell) scalar values and are allowed in any property that contains cell values.
  3. While properties are defined either as a single value or a sequence of values, their type information is thrown away. The importance of this is that the property accessors must have an out-of-band way to be informed of the type(s) used in the property, i.e. property type information is not discoverable.

Lifecycle of a Devicetree.

The purpose of the device tree is (or at least was until recently) to be provided to an operating system at boot time. This was done by the following steps.

  1. Device Tree source files (DTS) are processed by a compiler to generate an in-memory tree structure. This structure is dynamically created at compile time by editing operations of the the compile tree sources which are:
    • Device tree sources are usually now pre-processed using the C proprocessor, but the built-in source include directive is still supported. Note that mixing them is permitted although it can lead to the unexpected behaviour of the base source file being preprocessed while the included one is not.
    • Declaration of a device node results in the creation of new in-memory device node if it doesn’t exist, or reusing it if it does.
    • Declaration of a property results in the creation of a new in-memory property containing the new property values if it doesn’t exist or replacing it if it does.
    • Node and property removal directives remove nodes and properties of the runtime tree structure as appropriate.
    • Node labels and references to them in properties are tracked. Note that references are the only scalar values that are tracked in the in-memory property data structure.
    • phandle references editing operations of the form ‘&label’ & ‘&{/path}’ are processed. These reference nodes with labels declared earlier in the main tree source. This form is typically used when compiling a device tree comprised of a main source file and a number of included files because it lends well to the a pattern of incremental change.
    • The special /memreserve/ directive is parsed and processed.
  2. The in-memory tree structure is ‘flattened’, i.e. it is serialized to create a device tree blob (DTB). It is in this stage that the symbolic references to node labels are resolved to integer/cell phandle values, with references to them being replaced by a cell value of the node’s phandle. Special ‘automatic’ properties (named phandle) containing the assigned phandle values are created for nodes that are referenced.
  3. This device tree blob file is placed in the applicable device and the bootloader is informed about how to retrieve it. This may be done by placing it in non-volatile storage at a specific byte offset, or being put in a boot-loader accesible filesystem with a specific name, etc.
  4. The bootloader starts, retrieves the device tree blob, and either passes it unchanged to the operating system or performs minor modification (i.e. altering the boot command line in the chosen node or enabling/disabling devices by modifying the status properties of some nodes). The bootloader typically does not create an in-memory tree structure at this step, it operates on the DTB blob level.
  5. The operating system starts and ‘unflattens’ the device tree blob which the bootloader has passed to it using the agreed upon architecture specific interface. The in-memory tree structure created is the same as the one created at the end of the compilation step, but with any changes that the boot-loader performed. The kernel at this point starts using the in-memory data structure, and it is referred to as the live-tree going forward. Note that while node phandles are discovered and tracked by the ‘phandle’ properties, their references cannot be deduced at this time.
  6. The operating system (including any device drivers) scans the live-tree and performs initialization and configuration of the hardware described there. Note that the operating system must have complete knowledge of the nodes and properties of an active node. This is evident by the use of access methods that include type information (e.g. of_property_read_u32() ), node references needing to be explicitly discovered by converting cell phandle value to a reference to a node, etc. Unfortunately, this is very error-prone since the type information has been discarded. There is no way to disallow access to a property using a different method than what was declared in the original source file.

The steps above are applicable to the simple case of a single platform, and up to a few years ago used to be the norm. In contemporary systems the situation is more complex for the following reasons:

  1. A common requirement is for a single image to be used for a number of different (but sufficiently similar) platforms. The number of stored DTBs would match the number of supported platforms, even if their changes are minimal.
  2. Hardware is no longer static. The proliferation of FPGAs and add-on expansion boards requires runtime device tree modification using device tree overlays. Those overlays are extremely similar to the way in-memory tree modification is performed at compile time but it is different in subtle ways.
  3. The device tree lifecycle expects perfect coordination across all the steps without the possibility of errors. This is troublesome in practice since every step in the sequence is part of a different project (compiler, bootloader, operating system). Errors can easily creep in and are usually not detected until the last step of the sequence, the operating system boot process. In case of an error, the result is usually a hung system without any indication what might have gone wrong.

YAML as a source format alternative

YAML is a human-readable data serialization language which is expressive enough to cover all DTS features. Simple YAML files are just key-value pairs that are very easy to parse, even without using a formal YAML parser. YAML streams are containing documents separated with a — marker. This model is a good fit for device tree since one may simply append a few lines of text to a given YAML stream to modify it.

YAML parsers are very mature, as YAML was first released in 2001. It is currently in wide-spread use and schema validation tools are available and common. Additionally, YAML support is available for many major programming languages.

Mapping of DTS constructs to YAML

The mapping of DTS constructs to YAML is relatively straightforward since they are both key-value declaration languages.

  • Comments in YAML are done using the # character instead of the C-like comments that DTS uses.
/* dts comment */
# YAML comment
  • DTS is a free form language using braces for denoting nest level while YAML is indentation sensitive in standard YAML encoding. Fortunately YAML is a superset of JSON which can be used as a valid free form.
node {
   property = "foo";
}
node:
  property: "foo"
{ "node": { "property": "foo" } }
  • There is no explict root in YAML encoding. Top level nodes & properties are taken to be located in the root.
/ {
    property;
    subnode {
       another-property;
    };
};
property: true
  subnode:
    another-property: true
  • Sequences in YAML may be denoted either by a single line starting with a hyphen ‘-‘, or bracketed JSON form. The following are equivalent.
property = "a", "b", "c";
property:
  - "a"
  - "b"
  - "c"
property: [ "a", "b", "c" ]
  • Values that may be evaluated as numeric scalars are used as cells.
property = <10>;
property: 10

Note that this includes integer expressions as well

property = <(5 + 5)>;
property: 5+5
  • String property values are enclosed in double quotes, although this is optional if the value cannot be expressed as a numeric scalar.
property = "string";
property: "string"
  • Boolean values are encoded as true and false. This is not implicit like in DTS.
property;
property: true

Note that it is possible to declare a property as false but you will get a warning about it being removed when generating the DTB.

property: false
  • It is possible to explictly declare the type of a scalar using the standard ‘!’ method of YAML. For instance this is how byte properties are supported.
property = [0124AB];
property: !int8 [ 0x01, 0x24, 0xab ]
  • Similarly the /bits/ directive is supported by explicit tagging.
property = /bits/ 64 <100>;
property: !int64 100
  • Labels are named anchors and are referenced by a ‘*’. Note that references are typed as such in YAML, they are transformed to phandle cells only on DTB generation.
label: node {
   property;
};

ref = <&label>;

&label {
    foo;
};
node: &label
  property: true

ref: *label

*label:
  foo: true
  • The delete node and properties directives are replaced with assignment to null/~. It works the same for both properties and nodes.

/ {
    node {
        property;
    };
};

/ {
    /delete-node/ node;
};
node:
  property: true

node: ~
  • There is no source /include/ directive in YAML. It is expected that thet C preprocessor will be used as is the norm with DTS.
  • Similarly there is no /include-bin/ directive, YAML can relatively easily include binary data as base64 string properties.
  • To easily support pre-processor macros from a DTS environment, scalars that are detected to be space separated integer expressions are transparently converted to scalar integer sequences.
#define MACRO(x, y) x y (x + y)
property = ;
#define MACRO(x, y) x y (x + y)
property: MACRO(10, 5)

Will result in

property: [ 10, 5, 15 ]

The YAML advantage

Radical changes are seldom worth it without bringing in significant benefits. Switching to YAML instead of DTS is indeed a radical change, but it does carry benefits, namely:

  1. YAML is a well known and mature technology which is supported by many programming languages and environments.
  2. YAML’s original purpose was data serialization. Therefore it is orthogonal and supports high-level language data structures well.
  3. It is suited for the description of graph structures, since it supports references and anchors.
  4. With its mature parsers and tools, it easily supports the human edit and compile cycle that is now common with device tree development. Since all property values are potentially typed, it is possible to track type information in order to perform thorough validation and checking against device tree bindings (once the bindings are converted to a machine readable format, preferably YAML). As well, this allows the reporting of accurate error messages and warnings at any stage of the compilation process.
  5. It is possible to generate YAML as an intermediate format with references not resolved, in a similar way that object files are used. Those intermediate files can them be compiled/linked again to generate the final DTB/YAML file. For example, instead of compiling into a single output file, one could generate intermediate YAML files, similar in every way to device tree overlays, and then perform the final ‘linking’ step at either compile time or the bootloader.
  6. It is relatively easy to parse, and a resource limited parser that can be included in bootloaders or the kernel is possible.
  7. Data in YAML can easily be converted to and from other formats making it convertable to formats which future tools may understand.

The yamldt compiler

yamldt is a YAML/DTS to DT blob generator/compiler and validator. The YAML schema is functionally equivalent to DTS and supports all DTS features, while as a DTS compiler it is bit-exact compatible with DTC. yamldt parses a device tree description (source) file in YAML/DTS format and outputs a device tree blob (which can be bit-exact to the one generated from the reference dtc compiler if the -C option is used).

Validation is performed against a YAML schema that defines properties and constraints. A checker uses this schema to generate small code fragments that are compiled to eBPF and executed for the specific validation of each DT node the rule selects in the output tree.

Validation

As mentioned above, yamldt is capable of performing validation of DT constructs using a C-based eBPF checker. eBPF code fragments are assembled that can perform type checking of properties and enforce arbitrary value constraints while fully supporting inheritance.

As an example, here’s how the validation of a given fragment works using on a jedec,spi-nor node:

m25p80@0:
  compatible: "s25fl256s1"
  spi-max-frequency: 76800000
  reg: 0
  spi-tx-bus-width: 1
  spi-rx-bus-width: 4
  "#address-cells": 1
  "#size-cells": 1

The binding for this is:

%YAML 1.1
---
jedec,spi-nor:
  version: 1

  title: >
    SPI NOR flash: ST M25Pxx (and similar) serial flash chips

  maintainer:
    name: Unknown

  inherits: *spi-slave

  properties:
    reg:
      category: required
      type: int
      description: chip select address of device

    compatible: &jedec-spi-nor-compatible
      category: required
      type: strseq
      description: >
        May include a device-specific string consisting of the
        manufacturer and name of the chip. A list of supported chip
        names follows.
        Must also include "jedec,spi-nor" for any SPI NOR flash that can
        be identified by the JEDEC READ ID opcode (0x9F).
      constraint: |
        anystreq(v, "at25df321a") ||
        anystreq(v,  "at25df641") ||
        anystreq(v, "at26df081a") ||
        anystreq(v,   "mr25h256") ||
        anystreq(v,    "mr25h10") ||
        anystreq(v,    "mr25h40") ||
        anystreq(v, "mx25l4005a") ||
        anystreq(v, "mx25l1606e") ||
        anystreq(v, "mx25l6405d") ||
        anystreq(v,"mx25l12805d") ||
        anystreq(v,"mx25l25635e") ||
        anystreq(v,    "n25q064") ||
        anystreq(v, "n25q128a11") ||
        anystreq(v, "n25q128a13") ||
        anystreq(v,   "n25q512a") ||
        anystreq(v, "s25fl256s1") ||
        anystreq(v,  "s25fl512s") ||
        anystreq(v, "s25sl12801") ||
        anystreq(v,  "s25fl008k") ||
        anystreq(v,  "s25fl064k") ||
        anystreq(v,"sst25vf040b") ||
        anystreq(v,     "m25p40") ||
        anystreq(v,     "m25p80") ||
        anystreq(v,     "m25p16") ||
        anystreq(v,     "m25p32") ||
        anystreq(v,     "m25p64") ||
        anystreq(v,    "m25p128") ||
        anystreq(v,     "w25x80") ||
        anystreq(v,     "w25x32") ||
        anystreq(v,     "w25q32") ||
        anystreq(v,     "w25q64") ||
        anystreq(v,   "w25q32dw") ||
        anystreq(v,   "w25q80bl") ||
        anystreq(v,    "w25q128") ||
        anystreq(v,    "w25q256")

    spi-max-frequency:
      category: required
      type: int
      description: Maximum frequency of the SPI bus the chip can operate at
      constraint: |
        v > 0 && v < 100000000

    m25p,fast-read:
      category: optional
      type: bool
      description: >
        Use the "fast read" opcode to read data from the chip instead
        of the usual "read" opcode. This opcode is not supported by
        all chips and support for it can not be detected at runtime.
        Refer to your chips' datasheet to check if this is supported
        by your chip.

  example:
    dts: |
      flash: m25p80@0 {
          #address-cells = <1>;
          #size-cells = <1>;
          compatible = "spansion,m25p80", "jedec,spi-nor";
          reg = <0>;
          spi-max-frequency = <40000000>;
          m25p,fast-read;
      };
    yaml: |
      m25p80@0: &flash
        "#address-cells": 1
        "#size-cells": 1
        compatible: [ "spansion,m25p80", "jedec,spi-nor" ]
        reg: 0;
        spi-max-frequency: 40000000
        m25p,fast-read: true

Note the constraint rule matches on any compatible string in the given list. This binding inherits from spi-slave as indicated by the line: inherits: *spi-slave

*spi-slave is standard YAML reference notation which points to the spi-slave binding, pasted here for convenience:

%YAML 1.1
---
spi-slave: &spi-slave
  version: 1

  title: SPI Slave Devices

  maintainer:
    name: Mark Brown <[email protected]>

  inherits: *device-compatible

  class: spi-slave
  virtual: true

  description: >
    SPI (Serial Peripheral Interface) slave bus devices are children of
    a SPI master bus device.

  # constraint: |+
  #  class_of(parent(n), "spi")

  properties:
    reg:
      category: required
      type: int
      description: chip select address of device

    compatible:
      category: required
      type: strseq
      description: compatible strings

    spi-max-frequency:
      category: required
      type: int
      description: Maximum SPI clocking speed of device in Hz

    spi-cpol:
      category: optional
      type: bool
      description: >
        Boolean property indicating device requires
        inverse clock polarity (CPOL) mode

    spi-cpha:
      category: optional
      type: bool
      description: >
        Boolean property indicating device requires
        shifted clock phase (CPHA) mode

    spi-cs-high:
      category: optional
      type: bool
      description: >
        Boolean property indicating device requires
        chip select active high

    spi-3wire:
      category: optional
      type: bool
      description: >
        Boolean property indicating device requires
        3-wire mode.

    spi-lsb-first:
      category: optional
      type: bool
      description: >
        Boolean property indicating device requires
        LSB first mode.

    spi-tx-bus-width:
      category: optional
      type: int
      constraint: v == 1 || v == 2 || v == 4
      description: >
        The bus width(number of data wires) that
        used for MOSI. Defaults to 1 if not present.

    spi-rx-bus-width:
      category: optional
      type: int
      constraint: v == 1 || v == 2 || v == 4
      description: >
        The bus width(number of data wires) that
        used for MISO. Defaults to 1 if not present.

  notes: >
    Some SPI controllers and devices support Dual and Quad SPI transfer mode.
    It allows data in the SPI system to be transferred in 2 wires(DUAL) or
    4 wires(QUAD).
    Now the value that spi-tx-bus-width and spi-rx-bus-width can receive is
    only 1(SINGLE), 2(DUAL) and 4(QUAD). Dual/Quad mode is not allowed when
    3-wire mode is used.
    If a gpio chipselect is used for the SPI slave the gpio number will be
    passed via the SPI master node cs-gpios property.

  example:
    dts: |
      spi@f00 {
          ethernet-switch@0 {
              compatible = "micrel,ks8995m";
              spi-max-frequency = <1000000>;
              reg = <0>;
          };

          codec@1 {
              compatible = "ti,tlv320aic26";
              spi-max-frequency = <100000>;
              reg = <1>;
          };
      };
    yaml: |
      spi@f00:
        ethernet-switch@0:
          compatible: "micrel,ks8995m"
          spi-max-frequency: 1000000
          reg: 0

        codec@1:
          compatible: "ti,tlv320aic26"
          spi-max-frequency: 100000
          reg: 1

Note the &amp;spi-slave anchor, this is what it is used to refer to other parts of the schema.

The SPI slave binding defines a number of properties that all inherited bindings include. This in turn inherits from device-compatible, which is this:

%YAML 1.1
---
device-compatible: &device-compatible
  title: Contraint for devices with compatible properties
  # select node for checking when the compatible constraint and
  # the device status enable constraint are met.
  selected: [ "compatible", *device-status-enabled ]

  class: constraint
  virtual: true

Note that device-compatible is a binding that all devices defined with the DT schema will inherit from.

The selected property will be used to generate a select test that will be used to to find out whether a node should be checked against a given rule.

The selected rule defines two constraints. The first one is the name of a variable in a derived binding that all its constraints must satisfy; in this case it’s the jedec,spi-nor compatible constraint in the binding above. The selected constraint is a reference to the device-status-enabled constraint defined at:

%YAML 1.1
---
device-enabled:
  title: Contraint for enabled devices

  class: constraint
  virtual: true

  properties:
    status: &device-status-enabled
      category: optional
      type: str
      description: Marks device state as enabled
      constraint: |
        !exists || streq(v, "okay") || streq(v, "ok")

The device-enabled constraint checks where the node is enabled in DT parlance.

Taking those two constraints together, yamldt generates an enable method filter which triggers on an enable device node that matches any of the compatible strings defined in the jedec,spi-nor binding.

The check method will be generated by collecting all the property constraints (category, type and explicit value constraints).

Note how in the above example a variable (v) is used as the current property value. The generated methods will provide it, initialized to the current value to the constraint.

Note that custom, manually written select and check methods are possible but their usage is not recommended for simple types.

Installation

Install libyaml-dev and the standard autoconf/automake generation tools, then compile with the standard ./autogen.sh, ./configure, and make cycle. Note that the bundled validator requires a working eBPF compiler and libelf. Known working clang versions with eBPF support are 4.0 and higher.

For a complete example of a port of a board DTS file to YAML take a look in the port/ directory

Usage

The yamldt options available are:

yamldt [options] <input-file>
 options are:
   -q, --quiet           Suppress; -q (warnings) -qq (errors) -qqq (everything)
   -I, --in-format=X     Input format type X=[auto|yaml|dts]
   -O, --out-format=X    Output format type X=[auto|yaml|dtb|dts|null]
   -o, --out=X           Output file
   -c                    Don't resolve references (object mode)
   -g, --codegen         Code generator configuration file
       --schema          Use schema (all yaml files in dir/)
       --save-temps      Save temporary files
       --schema-save     Save schema to given file
       --color           [auto|off|on]
       --debug           Debug messages
   -h, --help            Help
   -v, --version         Display version

   DTB specific options

   -V, --out-version=X   DTB blob version to produce (only 17 supported)
   -C, --compatible      Bit-exact DTC compatibility mode
   -@, --symbols         Generate symbols node
   -A, --auto-alias      Generate aliases for all labels
   -R, --reserve=X       Make space for X reserve map entries
   -S, --space=X         Make the DTB blob at least X bytes long
   -a, --align=X         Make the DTB blob align to X bytes
   -p, --pad=X           Pad the DTB blob with X bytes
   -H, --phandle=X       Set phandle format [legacy|epapr|both]
   -W, --warning=X       Enable/disable warning (NOP)
   -E, --error=X         Enable/disable error (NOP)
   -b, --boot-cpu=X      Force boot cpuid to X

-q/--quiet suppresses message output.

The -I/--in-format option selects the input format type. By default it is set to auto which is capable of selecting based on file extension and input format source patterns.

The -O/--out-format option selects the output format type. By default it is set to auto which uses the output file extension.

-o/--out sets the output file.

The -c option causes unresolved references to remain in the output file resuling in an object file. If the output format is set to DTB/DTS it will generate an overlay, if set to yaml it results in a YAML file which can be subsequently recompiled as an intermediate object file.

The -g/--codegen option will use the given YAML file(s) (or dir/ as in the schema option) as input for the code generator.

The --schema option will use the given file(s) as input for the checker. As an extension, if given a directory name with a terminating slash (i.e. dir/) it will recursively collect and use all YAML files within.

The --save-temps option will save all intermediate files/blobs.

--schema-save will save the processed schema and codegen file including all compiled validation filters. Using it speeds validation of multiple files since it can be used as an input via the –schema option.

--color controls color output in the terminal, while --debug enables the generation of a considerable amount of debugging messages.

The following DTB specific options are supported:

-V/--out-version selects the DTB blob version; currently only version 17 is supported.

The -C/--compatible option generates a bit-exact DTB file as the DTC compiler.

The -@/--symbols and -A/--auto-alias options generate a symbols and alias entries for all the defined labels in the source files.

The -R/--reserve, -S/--space, -a/--align and -p/--pad options work the same way as in DTC. -R add reserve memreserve entries, -S adds extra space, -a aligns and -p pads extra space end of the DTB blob.

The -H/--phandle option selects either legacy/epapr or both phandle styles.

The -W/--warning and -E/--error options are there for command line compatibility with dtc and are ignored.

Finally -d/--boot-cpu forces the boot cpuid.

Automatic suffix detection does what you expect, i.e. an output file ending in .dtb if selecting the DTB generation option, .yaml if selecting the yaml generation option, and so on.

Given a source file in YAML foo.yaml, you generate a DTB file with:

# foo.yaml
foo: &foo
  bar: true
  baz:
   - 12
   - 8
   - *foo
  frob: [ "hello", "there" ]

To process it with yamldt:

$ yamldt -o foo.dtb foo.yaml
$ ls -l foo.dtb
-rw-rw-r-- 1 panto panto 153 Jul 27 18:50 foo.dtb
$ fdtdump foo.dtb
/dts-v1/;
// magic:       0xd00dfeed
// totalsize:       0xe1 (225)
// off_dt_struct:   0x38
// off_dt_strings:  0xc8
// off_mem_rsvmap:  0x28
// version:     17
// last_comp_version:   16
// boot_cpuid_phys: 0x0
// size_dt_strings: 0x19
// size_dt_struct:  0x90

/ {
    foo {
        bar;
        baz = <0x0000000c 0x00000008 0x00000001>;
        frob = "hello", "there";
        phandle = <0x00000001>;
    };
    __symbols__ {
        foo = "/foo";
    };
};

dts2yaml

dts2yaml is an automatic DTS to YAML conversion tool, that works on standard DTS files which use the preprocessor. It is capable of detecting macro usage and advanced DTS concepts, like property/node deletes, etc. Conversion is accurate as long as the source file still looks like DTS source (i.e. it is not using extremely complex macros).

dts2yaml [options] [input-file]
 options are:
   -o, --output        Output file
   -t, --tabs       Set tab size (default 8)
   -s, --shift      Shift when outputing YAML (default 2)
   -l, --leading    Leading space for output
   -d, --debug      Enable debug messages
       --silent        Be really silent
       --color         [auto|off|on]
   -r, --recursive     Generate DTS/DTSI included files
   -h, --help       Help
       --color         [auto|off|on]

All the input files will be converted to yaml format. If no output option is given, the output will be named according to the input filename. So foo.dts will yield foo.yaml and foo.dtsi will yield foo.yamli.

The recursive option converts all included files that have a dts/dtsi extension as well.

Test suite

To run the test-suite you will need a relatively recent DTC compiler, YAML patches are no longer required.

The test-suite first converts all the DTS files in the Linux kernel for all architectures to YAML format using dts2yaml. Afterwards, it compiles the YAML files with yamldt and the DTS files with dtc. The resulting dtb files are bit-exact because the -C option is used.

Run make check to run the test suite.
Run make validate to run the test suite and perform schema validation checks. It is recommended to use the --keep-going flag to continue checking even in the presence of validation errors.

Currently out of 1379 DTS files, only 6 fail conversion:

exynos3250-monk exynos4412-trats2 exynos3250-rinato exynos5433-tm2
exynos5433-tm2e

All 6 use a complex pin mux macro declaration that it is not possible to automatically convert.

Workflow

It is expected that the first thing a user of yamldt would want to do is to convert an existing DTS configuration to YAML.

The following example uses the beaglebone black and the am335x-boneblack.dts source as located in the port/ directory.

Compile the original DTS source with DTC

$  cc -E  -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input 
    -nostdinc -undef -x assembler-with-cpp -D__DTS__ am335x-boneblack.dts 
    | dtc -@ -q -I dts -O dtb - -o am335x-boneblack.dtc.dtb

Use dts2yaml to convert to yaml

$ dts2yaml -r am335x-boneblack.dts
$ ls *.yaml*
am335x-boneblack-common.yamli  am335x-bone-common.yamli  am33xx-clocks.yamli
am33xx.yamli  tps65217.yamli

Note the recursive option automatically generates the dependent include files.

$ cc -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input 
    -nostdinc -undef -x assembler-with-cpp -D__DTS__ am335x-boneblack.yaml | 
    ../../yamldt -C -@ - -o am335x-boneblack.dtb 
$ ls -l *.dtb
-rw-rw-r-- 1 panto panto 50045 Jul 27 19:10 am335x-boneblack.dtb
-rw-rw-r-- 1 panto panto 50045 Jul 27 19:07 am335x-boneblack.dtc.dtb
$ md5sum *.dtb
3bcf838dc9c32c196f66870b7e6dfe81  am335x-boneblack.dtb
3bcf838dc9c32c196f66870b7e6dfe81  am335x-boneblack.dtc.dtb

Compiling without the -C option results in a file with the same functionality, but it is slightly smaller due to better string table optimization.

$ yamldt am335x-boneblack.dtc.yaml -o am335x-boneblack.dtb
$ ls -l *.dtb
-rw-rw-r-- 1 panto panto 50003 Jul 27 19:12 am335x-boneblack.dtb
-rw-rw-r-- 1 panto panto 50045 Jul 27 19:07 am335x-boneblack.dtc.dtb

Note that the CPP command line is the same, so no changes to header files are required. dts2yaml will detect macro usage and convert from the space delimited form that DTC uses to the comma delimted form used by YAML.

yamldt as a DTC compiler

yamldt supports all dtc options, so using it as a dtc replacement is straightforward.

Using it for compiling the Linux Kernel DTS files is as simple as:

$ make DTC=yamldt dtbs

Note that by default the compatibility option (-C) is not used, so if you need to be bit-compatible with DTC pass the -C flag as follows:

$ make DTC=yamldt DTC_FLAGS="-C"

Generally, yamldt is a little bit faster than dtc and generates somewhat smaller DTB files (if not using the -C option). However, due to internally tracking all parsed tokens and their locations in files, it is capable of generating accurate error messages that are parseable by text editors for automatic movement to the error.

For example, with this file containing an error:

/* duplicate label */
/dts-v1/;
/ {
    a: foo { foo; };
    a: bar { bar; };
};

yamldt will generate the following error:

$ yamldt -I dts -o dts -C duplabel.dts
duplabel.dts:8:2: error: duplicate label a at "/bar"
  a: bar {
  ^
duplabel.dts:4:2: error: duplicate label a is defined also at "/foo"
  a: foo {
  ^

while dtc will generate:

$ yamldt -I dts -o dts -C duplabel.dts
dts: ERROR (duplicate_label): Duplicate label 'a' on /bar and /foo
ERROR: Input tree has errors, aborting (use -f to force output)

Known features of DTC that are not available are:

  • Only version 17 DT blobs are supported. Passing a -V argument requesting a different one will result in error.
  • Assembly output is not supported.
  • Assembly and filesystem inputs are not supported.
  • The warning and error options are accepted, but they don’t do anything. yamldt uses a validation schema for application specific errors and warnings so those options are superfluous.

Notes on DTS to DTS conversion

The conversion from DTS is straight forward:

For example:

/* foo.dts */
/ {
    foo = "bar";
    #cells = <2>;
    phandle-ref = <&ref 1>;
    ref: refnode { baz; };
};
# foo.yaml
foo: "bar"
"#cells": 2
phandle-ref: [ *ref 1 ]
refnode: &ref
  baz: true

Major differences between DTS & YAML:

  • YAML is using # as a comment marker, therefore properties with a # prefix get converted to explicit string literals:
#cells = <0>;

to YAML

"#cells": 0
  • YAML is indentation sensitive, but it is a JSON superset. Therefore the following are equivalent:
foo: [ 1, 2 ]
foo:
 - 1
 - 2
  • The labels in DTS are defined and used as:
foo: node { baz; };
bar = <&foo>;

In YAML the equivalent methods are called anchors and are defined as follows:

node: &foo
  baz: true
bar: *foo
  • Explicit tags in YAML are using !, so the following:
mac = [ 0 1 2 3 4 5 ];

is used like this in YAML:

mac: !int8 [ 0, 1, 2, 3, 4, 5 ]
  • DTS uses spaces to seperate array elements, YAML uses either indentation or commas in JSON form. Note that yamldt is smart enough to detect the DTS form and automatically convert in most cases:
pinmux = <0x00 0x01>;

In YAML:

pinmux:
  - 0x00
  - 0x01

or:

pinmux: [ 0x00, 0x01 ]
  • Path references () automatically are converted to pseudo YAML anchors (of the form yaml_pseudo__n__):
/ {
    foo { bar; };
};
ref = <&/foo>;

In YAML:

foo: &yaml_pseudo__0__
ref: *foo
  • Integer expression evaluation, similar in manner to that which the CPP preprocessor performs, is available. This is required in order for macros to work. For example, given the following two files:
/* add.h */
#define ADD(x, y) ((x) + (y))
# macro-use.yaml

#include "add.h"

result: ADD(10, 12)

The output after the cpp preprocessor pass:

result: ((10) + (12))

Parsing with yamldt to DTB will generate a property:

result = <22>;

Validation example

For this example we’re going to use port/am335x-boneblack-dev/. An extra rule-check.yaml file has been added where validation tests can be performed.

That file contains a single jedec,spi-nor device:

*spi0:
  m25p80@0:
    compatible: "s25fl256s1"
    spi-max-frequency: 76800000
    reg: 0
    spi-tx-bus-width: 1
    spi-rx-bus-width: 4
    "#address-cells": 1
    "#size-cells": 1

This is a valid device node, so running validate produces the following:

$ make validate
cc -E -MT am33xx.cpp.yaml -MMD -MP -MF am33xx.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ am33xx.yaml >am33xx.cpp.yaml
cc -E -MT am33xx-clocks.cpp.yaml -MMD -MP -MF am33xx-clocks.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ am33xx-clocks.yaml >am33xx-clocks.cpp.yaml
cc -E -MT am335x-bone-common.cpp.yaml -MMD -MP -MF am335x-bone-common.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ am335x-bone-common.yaml >am335x-bone-common.cpp.yaml
cc -E -MT am335x-boneblack-common.cpp.yaml -MMD -MP -MF am335x-boneblack-common.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ am335x-boneblack-common.yaml >am335x-boneblack-common.cpp.yaml
cc -E -MT am335x-boneblack.cpp.yaml -MMD -MP -MF am335x-boneblack.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ am335x-boneblack.yaml >am335x-boneblack.cpp.yaml
cc -E -MT rule-check.cpp.yaml -MMD -MP -MF rule-check.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ rule-check.yaml >rule-check.cpp.yaml
../../yamldt  -g ../../validate/schema/codegen.yaml -S ../../validate/bindings/ -y am33xx.cpp.yaml am33xx-clocks.cpp.yaml am335x-bone-common.cpp.yaml am335x-boneblack-common.cpp.yaml am335x-boneblack.cpp.yaml rule-check.cpp.yaml -o am335x-boneblack-rules.pure.yaml
jedec,spi-nor: /ocp/spi@48030000/m25p80@0 OK

Note the last line. It means the node was checked and was found OK.

Editing the rule-check.yaml file, let’s introduce a couple of errors. The following output is generated by commenting out the reg property # reg: 0:

$ make validate
cc -E -MT rule-check.cpp.yaml -MMD -MP -MF rule-check.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ rule-check.yaml &gt;rule-check.cpp.yaml
../../yamldt  -g ../../validate/schema/codegen.yaml -S ../../validate/bindings/ -y am33xx.cpp.yaml am33xx-clocks.cpp.yaml am335x-bone-common.cpp.yaml am335x-boneblack-common.cpp.yaml am335x-boneblack.cpp.yaml rule-check.cpp.yaml -o am335x-boneblack-rules.pure.yaml
jedec,spi-nor: /ocp/spi@48030000/m25p80@0 FAIL (-2004)
../../validate/bindings/jedec,spi-nor.yaml:15:5: error: missing property: property was defined at /jedec,spi-nor/properties/reg
     reg:
     ^~~~

Note the descriptive error and the pointer to the missing property in the schema.

Making another error, assign a string to the reg property reg: "string":

$ make validate
$ make validate
cc -E -MT rule-check.cpp.yaml -MMD -MP -MF rule-check.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ rule-check.yaml &gt;rule-check.cpp.yaml
../../yamldt  -g ../../validate/schema/codegen.yaml -S ../../validate/bindings/ -y am33xx.cpp.yaml am33xx-clocks.cpp.yaml am335x-bone-common.cpp.yaml am335x-boneblack-common.cpp.yaml am335x-boneblack.cpp.yaml rule-check.cpp.yaml -o am335x-boneblack-rules.pure.yaml
jedec,spi-nor: /ocp/spi@48030000/m25p80@0 FAIL (-3004)
rule-check.yaml:8:10: error: bad property type
     reg: "string"
          ^~~~~~~~
../../validate/bindings/jedec,spi-nor.yaml:15:5: error: property was defined at /jedec,spi-nor/properties/reg
     reg:
     ^~~~

Note the message about the type error, and the pointer to the location where the reg property was defined.

Finally, let’s make an error that violates a constraint.

Change the spi-tx-bus-width value to 3:

$ make validate
cc -E -MT rule-check.cpp.yaml -MMD -MP -MF rule-check.o.Yd -I ./ -I ../../port -I ../../include -I ../../include/dt-bindings/input -nostdinc -undef -x assembler-with-cpp -D__DTS__ -D__YAML__ rule-check.yaml &gt;rule-check.cpp.yaml
../../yamldt  -g ../../validate/schema/codegen.yaml -S ../../validate/bindings/ -y am33xx.cpp.yaml am33xx-clocks.cpp.yaml am335x-bone-common.cpp.yaml am335x-boneblack-common.cpp.yaml am335x-boneblack.cpp.yaml rule-check.cpp.yaml -o am335x-boneblack-rules.pure.yaml
jedec,spi-nor: /ocp/spi@48030000/m25p80@0 FAIL (-1018)
rule-check.yaml:9:23: error: constraint rule failed
     spi-tx-bus-width: 3
                       ^
../../validate/bindings/spi/spi-slave.yaml:77:19: error: constraint that fails was defined here
       constraint: v == 1 || v == 2 || v == 4
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~
../../validate/bindings/spi/spi-slave.yaml:74:5: error: property was defined at /spi-slave/properties/spi-tx-bus-width
     spi-tx-bus-width:

Note how the offending value is highlighted. The offending constraint and property definition are aslo listed.

Home Servers, AArch64, and You (well, me)

Recently, my SoftIron OverDrive1000 arrived, and I’ve finally given myself some time to sit down and implement the project I purchased it for. First, one may ask, why AArch64? The answer lies in my history of installing Linux-based machines at home, the number of PowerPC machines still exceeds the number of x86/x86-64 machines that I have done. So when I got the chance to pick up some AArch64 hardware that came in a regular form factor, I decided to jump on it. The other reason is that, as I have joked with some people, I like to do things the hard way. What do I mean by that? Well, I’ll explain. But first, some background.

The hardware comes with openSUSE Leap 42.2 preinstalled. I decided to take the fact that this is the least familiar distribution to me as a challenge. At the same time, I also decided that it would be worth evaluating other distributions that I’m familiar with, such as CentOS and Debian, on AArch64. Most of what I do with my home server today is related to locally streaming media. So while I had been installing and running various applications directly (making use of crontab’s @reboot keyword to launch them), I decided I should move on to modern best practices and use Docker to isolate, control, and update these applications. Finally, while I’ve been using Rygel for a few months, I want to get back to using Plex Media Server to serve the content. This in turn introduces the constraint of needing to use a 32-bit ARM binary, as Plex currently does not have a 64-bit build available.

The first bit of fun I found was that while the Docker project has community provided support for AArch64, Docker, Inc does not actually provide AArch64 builds. So if you want to have the most up to date Docker installation and wish to have Docker be managed by your distribution packaging manager, there’s some work to be done. The Docker project does have some contrib scripts to create packages of Docker for many distributions, but they’re written assuming that you’re running on x86-64. The good news is that all 3 of the distributions mentioned above do ship a version of Docker. For my needs (a well firewalled internal server), the versions they provide are OK.

Testing on CentOS proved interesting. It was easy to get it installed and running from a spare drive on the real hardware. This was just as easy and boring as advertised. After checking basic functionality, I switched to running it in a virtual machine for the rest of my tests. Here’s where things got difficult. First, as of today, CentOS uses 64K pages rather than 4K pages. This in turn disallows the possibility of having 32-bit binary compatibility. One can recompile the CentOS kernel to change the page size and get 32-bit compatibility working. I did this and it proved straightforward yet time consuming. After confirming that this was enough to enable at least basic 32-bit compatibility, I decided to move on.

Debian was the next distribution that I tried out. While the Debian wiki says that you need a newer image to install on AMD Seattle hardware such as the Overdrive 1000, the latest Jessie images are actually new enough to boot and start the installer. At this point in my experiments I didn’t want do another install on the hardware, so the rest of my tests were done on a virtual machine install. This installation didn’t work out well for me and my needs, unfortunately. While there is a docker-engine package available which is used by 96boards.org, I didn’t want to introduce such a large deviation from upstream Debian. The docker.io package exists as a backport to Jessie, but not for arm64. Further, at the time that I was testing things, Sid was in an incomplete state for the docker.io package. While I was able to resolve the dependencies manually, Docker didn’t want to start. I should note that I’m doing my reviews here out of order slightly. At this point I had openSUSE with Docker running, and didn’t want to further dive into this problem, which I’m sure could be resolved. One last note is that I did test 32-bit compatibility, and it works fine on Debian out of the box.

Which brings me back to openSUSE. The main repository has Docker available, and the kernel is already built with 32-bit compatibility enabled. This was the easiest distribution for getting the first order of problems solved on, as everything just worked, and I was also able to easily set up VMs for testing the other distributions. The biggest hurdle I faced on openSUSE is that I didn’t find a lot of documentation about installing Docker, but instead about setting up various flavours of LXC. Fortunately, Docker is not very host distribution specific, so this was not a big problem.

All of the above distributions, and other efforts such as the Linaro distributions for the Dragonboard, suffer from one more issue with respect to running Docker containers consisting of 32-bit binaries. Namely, that we’re not using binaries that we’ve controlled the build process of, we’re just picking up whatever is in various public containers from Docker. Without getting too deep into the technical details of what instructions AArch64 will emulate, not all instructions required to run “armhf” binaries are required. For example, a number of Docker images are built optimized to the ARMv6 architecture as found in some models of the Raspberry Pi, and these instructions require additional Linux kernel support to be enabled. For more details see here.

So, where does that leave us? Well, first of all, I’ve completed the software side of the desired migration. The new server is doing everything the old server was doing before. As far as my end users go, it’s all working just as well as before too. In doing this migration, I’ve been reminded of the phrase “the more things change, the more they stay the same”. Back during the early Linux on PowerPC days, one would often have to deal with the problems induced by “Linux” really meaning “Linux on 32-bit Intel/x86 compatible CPUs” to some project or maintainer. Today one has to deal with the problem of “Linux” often meaning “Linux on 64-bit Intel/AMD x86-64 compatible systems” with an occasional “Linux on the Raspberry Pi” instead. Both of these assumptions make life interesting at times. With the former, for example, it is possible to have Docker images deal with multiple architectures, but that’s just not done today. I suspect that while it will be done for the core Docker images at some point, one is always going to need to take care when using arbitrary community images, setting aside any other concerns one might have in that area. With the latter assumption, we run into the kernel issue mentioned above, as without the Pi, “armhf” would likely mean “ARMv7” in practice. In the end, the cool thing here is that the distributions have achieved pretty close to parity between x86-64 and AArch64 in terms of the offered software. However, once you start to move out of that garden and into the world at large, you’re going to start to find some oddities. Now, if you’re like me and find the challenge fun, or are looking for a reason to dive deeper into how things work, it’s a great time and a great idea. But if you’re looking for a turn-key solution on an AArch64 platform, things aren’t quite there yet.