Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 4

In part three of this series I finished putting together what I wanted to have on my SOHO router and declared it to be done. While I plan to revisit the topic of a SOHO router using the Yocto Project and OpenEmbedded, this is the final part of the series. In this part, I want to focus on some of the things that I learned while doing the project.

The first thing is that I learned a lot about IPv6, specifically how it’s usually implemented within the United States for residential customers, and some of the implications of this implementation. The first thing to note is that I’ve been off-and-on trying to enable IPv6 for general IPv6 connectivity at home for some time now. Long before my ISP offered IPv6 service, I used Hurricane Electric to have a IPv6 tunnel and connectivity. This was great and only sometimes lead to problems, such as when Netflix finally supported IPv6 and began blocking well known tunnels for region-blocking reasons. It wasn’t until I started on this project that I decided to try to make real use of  routable addresses for hosting personal services. My expectations, and that of lots of software designed to manage IPv6 as well, are best described in article from RIPE about understanding IP addressing. In short, my house or “subscriber site” should get at least 256 LAN segments to do with as I want. Docker expects to have it’s own LAN segment to manage as part of configuring network bridges. When you have 256 or more LAN segments to use, that’s not a problem at all.

Unfortunately, my ISP provides only a single LAN segment. This is simultaneously more IPv6 addresses than the whole of IPv4 and something that should not be further subdivided in routing terms. I could subdivide my LAN segment, but this would in turn cause me to have to do a whole lot more work and headaches. That’s because at the routing level IPv6 is designed for my segment to be the smallest unit. Rather than deal with those headaches I switched my plans up from using Docker to using LXC. With LXC it’s easy to dump the container onto my LAN and then it picks up an IPv6 address the same way all of the other machines on my LAN do. This is good enough for my current needs, but it will make other things a lot harder down the line, if I want separation at the routing level for some devices.

But why am I doing that at all? Well, one of the benefits of having a small but still capable router is that I can run my own small services. While I don’t want to get into running my own email I think it makes a whole lot of sense to host my own chat server for example. With closed registration and no (or limited later on perhaps) federation with other servers I don’t need to worry about unauthorized users contacting my family nor do I have to worry about the company deciding it’s time to shutdown the service I use.

Another lesson learned is that while the Yocto Project has great QA, there’s always room to improve things. As part of picking a firewall I found that one of the netfilter logging options had been disabled by accident a while back. As a part of writing this series of articles and testing builds for qemux86-64, I found that one of the sound modules had been disabled. As a result, the instructions I wrote back in part 2 wouldn’t work. Working upstream is always fun and these changes have been merged and will be included in the next release.

I also worked on a few things for this project that I didn’t include directly in the relevant part of the series. For example, while I did include a number of full utilities in the list of packages installed in the router, I didn’t talk about replacing busybox entirely. This is something that OpenEmbedded supports using the PREFERRED_PROVIDERS and VIRTUAL-RUNTIME override mechanisms in the metadata. Prior to this article however, there wasn’t a good example on how to do this in upstream. Furthermore, there wasn’t an easy way to replace all of busybox and instead you had to list a single package and then include the rest of the required packages in your IMAGE_INSTALL or similar mechanism.  I am a fan of using busybox in places where I’m concerned about disk usage. However, on my router I have plenty of disk space so I want to be sure that if I have to go and solve a problem I’m not using my swiss army knife but rather have my full toolbox available. As a result, OpenEmbedded Core Master now has packagegroup-core-base-utils and a documented example of how to use that in local.conf.sample.extended. This means that when I refresh this image to be based on the Warrior branch I can remove a number of things from my IMAGE_INSTALL.

Another lesson is that old habits die hard.  In general, I always try to use the workflow where I make a change outside the device I’m working on, build the change in, and test it, rather than editing things live.  But when it’s “just” a quick one line change I’ll admit I do it live and roll it into my next build sometimes.  And then sometimes I forget to roll all my changes back up.  So while implementing this project I tried even harder than usual to not fall into that “just a quick change” mindset.  For the most part I’ve been successful at sticking to the idea workflow.  I really believe stateless is the right path forward.  And “for the most part” means that, yes, one time I did have to make use of the fact that the old rootfs was still mountable and copied a file over to the new rootfs, and then to the build machine.  I like to think of that as a reminder that A/B updates are more helpful than a “rewrite your disk each time” workflow for those occasional mistakes.

The caveat to the lesson above is because I really did the “git, bitbake, mender” cycle on this project. I didn’t start on it quite as soon as I said in the article, and I spent a lot more time toying with stuff in core-image-minimal instead of following my own advice, too.  I suppose that is the difference between writing a guide on how things should be done compared with how you do things when you just want to test one more thing, then switch over.  I really should have switched earlier however as every time I avoid doing the SD card shuffle it’s a win on a number of levels.

Did I say SD card above?  Yes, I did.  For this project, a 64GB “black box” that’s in the form-factor of a SD card will have as long of a life span as there is in the form-factor of a M.2 SSD or any other common storage format.  While my particular hardware has a SATA port, I don’t want to try to fit the required cabling, let alone the device itself in the case that’s recommended.  I will admit that I’m taking a bit of a risk here, I am putting as much frequent-write files under a ramfs as I can and after all, I did say stateless is a goal.  If everything does really die on me, I can be back up and running fairly quickly.

Last thing I learned is something I knew all along, really. I like the deeper ownership of the router. There’s both the pride and accomplishment in doing it and that “old school” fun of being the admin again, for real.

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 2

In part one of this series I explained some of my motivations for this project. Now it’s time to start on implementing the project itself. At this point I’m going to assume the reader has basic familiarity with using OpenEmbedded.  Otherwise, the Yocto Project (YP) quickstart guide and OpenEmbedded getting started pages are useful to bring yourself up to speed.  I assume that you’ve followed these instructions on how to prepare your build host and gone so far as to have completed a build for some target previous to this.  In this guide, I’m going to do my best to follow common best practices. Whenever I deviate from best practices, I’ll explain why we want something a bit different. After all, best practices are supposed to be taken as guidelines, not absolutes. Finally, I’m going to be working against the Yocto Project thud code-name release as that’s what is current as of this writing.

The first thing I’m going to do in my project is create a build directory now so that I can easily get access to various tools.  The next thing I’m going to do is use those tools to make a layer to store our configuration in.

$ . oe-core/oe-init-build-env
You had no conf/local.conf file. This configuration file has therefore been
created for you with some default values. You may wish to edit it to, for
example, select a different MACHINE (target hardware). See conf/local.conf
for more information as common configuration options are commented.
...
You can also run generated qemu images with a command like 'runqemu qemux86'
$ bitbake-layers create-layer ../meta-local-soho
NOTE: Starting bitbake server...
Add your new layer with 'bitbake-layers add-layer ../meta-local-soho'
$ bitbake-layers add-layer ../meta-local-soho
NOTE: Starting bitbake server...
$

Be sure to change oe-core to wherever the core layer was checked out.  Now that the layer has been created, let’s go ahead and start by putting it into git. Why? I am a firm believer in “commit early and commit often” as well as “cleanup and rebase once you’re done”. To me, one of the big selling points of git is that you can track your work incrementally, and when you notice unexpected breakage later on you can easily go back in time and locate the bugs.

$ cd ../meta-local-soho
$ git init .
$ git add *
$ git commit -s -m "Initial layer creation"
$ git branch -m thud

With the first commit in place, it’s time to start customizing the layer. We don’t need the example recipe, so let’s remove it.

$ git rm -r recipes-example/example
$ git commit -s -m "Remove example recipe"

Next, it’s time for a more substantive set of customizations. I’ll start by editing the README. Why? To start with, I’m going to list all of the layers I know of that are dependencies at this point. The README will also be a handy place to note which physical port is for WAN, which port(s) are for LAN, and the network devices that each port is associated with. For now, I add the following to the README:

  URI: git://git.openembedded.org/meta-openembedded
  branch: thud
  layers: meta-oe, meta-python, meta-networking, meta-filesystems

  URI: git://git.yoctoproject.org/meta-virtualization
  branch: thud

  URI: https://github.com/mendersoftware/meta-mender
  branch: thud

Now that I’ve documented these requirements, I’ll also enforce them in code. I edit conf/layer.conf and document these requirements too. The end of the file should look like:

LAYERDEPENDS_meta-local-soho = "core"
LAYERDEPENDS_meta-local-soho += "openembedded-layer meta-python"
LAYERDEPENDS_meta-local-soho += "networking-layer filesystems-layer"
LAYERDEPENDS_meta-local-soho += "virtualization-layer mender"
LAYERSERIES_COMPAT_meta-local-soho = "thud"

Since there are so many layers in use, I will make use of the TEMPLATECONF functionality so that the build directory will be populated correctly to start with. Next, I copy over meta/conf/bblayers.conf.sample, meta/conf/local.conf.sample and meta/conf/conf-notes.txt from the core layer over to the conf directory and commit them without change. Why? This will isolate my local edits later on and make my life easier next year when I decide it’s time to update to a current release. After I’ve committed those files, I the edit conf/bblayers.conf.sample and change it to look like:

BBLAYERS ?= " \
  ##OEROOT##/meta \
  ##OEROOT##/../meta-openembedded/meta-oe \
  ##OEROOT##/../meta-openembedded/meta-python \
  ##OEROOT##/../meta-openembedded/meta-networking \
  ##OEROOT##/../meta-openembedded/meta-filesystems \
  ##OEROOT##/../meta-virtualization \
  ##OEROOT##/../meta-mender/meta-mender-core \
  ##OEROOT##/../meta-local-soho \

Note that this assumes a directory structure where I’ve put all of the layers I will use in the same base directory. If I didn’t do that then I would need to adjust all of the paths to match how to get from the core layer to where the layers are stored. Now I want to use git add and commit all of these changes, so I can move on to the next step.

I will enable systemd next, and while there’s a number of places I could do this, including going so far as to create my own “distro” policy file, for this article I’ll just use conf/local.conf.sample to store these changes. While the core layer’s conf/local.conf.sample.extended has an example on switching to systemd, I’ll do it slightly differently. At the end of the conf/local.conf.sample insert the following, git add and then commit:

# Switch to systemd
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = ""
VIRTUAL-RUNTIME_syslog = ""
VIRTUAL-RUNTIME_login_manager = "shadow-base"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"

This differs from the core example in a few ways. First, I blank out pulling in an initscripts compat package. While this can be useful, for these images it’s going to end up being redundant. Next, I blank out a syslog provider as I’ll be letting systemd handle all of the logging. Finally, while busybox can be available and provide the login manager, I’ll be using shadow-base instead. This particular change is something now done upstream and will be in later releases, so keep that in mind for the future.

The next functional chunk for conf/local.conf.sample is enabling support for virtualization through meta-virtualization. That layer has its own well documented README file, and taking the time to go over it is a great idea. In my configuration, I’m just going to enable a few things that give me the minimal functionality. So add, git add and git commit:

# Add virtualization support
DISTRO_FEATURES_append = " virtualization aufs kvm"

When talking about system security there are many aspects. One aspect is to harden the system as much as possible by having the compiler apply various build-time safety measures that turn certain classes of attack from “exploit the system” to “Denial of Service by crashing the application”, or even “this is wildly unsafe code, fail to build”. To enable those checks in the build, add, git add and git commit:

# Security flags
require conf/distro/include/security_flags.inc

There’s one last bit of functionality I want in the conf/local.conf.sample file, and that’s support for Mender. How do I go about that? That’s going to depend a bit on what hardware you’re going to do this project on, as it’s a little bit different for something like the APU2 than on an ARM platform. Fortunately, there’s great documentation on how to integrate Mender here. While reading over all of the information there is important, it’s best to focus on the Configuring the build section to understand all of the required variables. In fact, now is a good time to talk more about how I’m going to use Mender in this particular setup. Looking at the overall Mender documentation, it’s very flexible. For this very small deployment scenario, I’ll use standalone mode rather than managed mode. This lets me skip all of the things about setting up a host of other services to make upgrades happen automatically. So, we’ll follow the instructions for configuring Mender in standalone mode. The next thing to touch on is how to handle persistent data. One way to do this would be to make sure that anything which needs to be manually customized or that will be persistently changed at runtime is written somewhere under /data on the device rather than at its normal location. This is because the normal location is going to change when I apply a new update but /data will always be the same. For a lot of deployments, this idea works best, as there are likely many users, and beyond our initial configuration the end user will make changes I know nothing about. For my use case, I am the user, and making the system as stateless as possible will in turn make backing the system up as easy as possible. So instead I will be modifying recipes to include our local configuration when needed. Having an easy to access and modify backup of the various configuration files and so forth will make my life easier in the long run if, for example, something happens to the hardware and it needs to be replaced.

Now that I have everything configured, it’s time to create the build directory. This will let me take advantage of all of the configuration work I just did. At this point, I can go back to the shell, leave meta-local-soho and do this:

$ TEMPLATECONF=../meta-local-soho/conf . oe-core/oe-init-build-env build-router

Again, be sure to change oe-core to wherever the core layer was checked out. Running cat conf/bblayers.conf will show how the changes that were made were now expanded.

What to do next? Well, that depends a little on what I already know about the system. The next step in this project is to configure systemd to take care of creating the networks. If I knew what all of the devices will be named, I could move on to that step. But if I didn’t know, or wasn’t sure, the next step is to build core-image-minimal. That’s as easy as doing:

$ bitbake core-image-minimal

and waiting for the final result, assuming that local.conf.sample file was configured to default to the appropriate target MACHINE. Otherwise I’ll need to pass that in on the command line above. Once that completes, take the appropriate image and boot it. The reward should be a root login prompt.  In this configuration, there’s no root password right now. Login and do:

# ip link

and this will show me what the interface names are. Assuming there is a DHCP server somewhere already, I can then use udhcpc -i NAME after plugging in an Ethernet cable to see which interface is which and note it down for the next step.

Now that I have my network interface names, I can configure systemd to handle them. In my specific case I have 4 Ethernet ports, and coincidentally the one closest to the physical console port (enp1s0) is the one I wanted to call the WAN port. This lets me put the other 3 ports into a bridge for my LAN. Now, to configure these interfaces, I’m going to dump some files under /etc/systemd/network/, and I will use the base-files recipe to own these files, rather than systemd itself. Why? Whenever a change forces a rebuild of systemd that forces a rebuild of a lot of other packages, so this allows me to isolate that kind of build churn. Now I go to meta-local-soho and enter the following:

$ mkdir -p recipes-core/base-files/base-files
$ cd recipes-core/base-files/base-files

I’m going to create four different files. Note that all of the filenames are arbitrary and are meant to help poor humans figure out what file does what. If another naming scheme is helpful, it can be used just as easily. First I’ll take care of the WAN port by creating wan-ethernet.network with the following content:

# Take the eth port closest to the console port for WAN
[Match]
Name=enp1s0

[Network]
DHCP=yes

Next, I’ll create our bridge for the other ports and this is done with two files. First I need a lan-bridge.network with:

# Bridge the other 3 remaining ports into one.
[Match]
Name=enp2s0 enp3s0 enp4s0

[Network]
Bridge=br0

Second I create br0.netdev with:

[NetDev]
Name=br0
Kind=bridge

And now I have a bridge device that I can use to manage all of our LAN ethernet ports. Later I’ll even put the AP on this bridge for simplicity. Now that I have a bridge, I need to configure it, so create bridge-ethernet.network and add:

[Match]
Name=br0

[Network]
Address=192.168.0.1
IPForward=yes
IPMasquerade=yes
IPv6AcceptRA=false
IPv6PrefixDelegation=dhcpv6

[IPv6PrefixDelegation]
RouterLifetimeSec=1800

There’s a lot in there, but it can be broken down pretty easily. Working my way up from the bottom, the first portion is needed to have the IPv6 address on the WAN port pass along what’s needed to the LAN to have devices configure themselves. Since we still live in a world with IPv4, I need to enable masquerade and forwarding. Finally, I configure a static IP for the interface that matches to br0. I haven’t talked about configuring the LAN for IPv4 yet. This is going to be a bit more complex and while systemd supports a trivial DHCP server, I want to do more complex things like assign specific addresses, so that will come later.

Finally, it’s time to make use of these new files. I’ll head up a level and back to recipes-core/base-files and create base-file_%.bbappend with the following:

FILESEXTRAPATHS_prepend := ":${THISDIR}/${PN}"

SRC_URI += "file://wan-ethernet.network \
            file://lan-bridge.network \
            file://bridge-ethernet.network \
            file://br0.netdev \
            "

do_install_append() {
    # Add custom systemd conf network files
    install -d ${D}${sysconfdir}/systemd/network

    # Add custom systemd conf network files
    install -m 0644 ${WORKDIR}/*.network ${D}${sysconfdir}/systemd/network/
    install -m 0644 ${WORKDIR}/*.netdev ${D}${sysconfdir}/systemd/network/
}

What this does is to look in that directory where I created those 4 files, add them to the recipe and then finally install them on the target where systemd will want them. At this point, I can git add the whole of recipes-core/base-files, and git commit.

There’s one last thing I want to configure right now. I’m going to have systemd handle things like turning on NAT, and doing some other basic iptables work. As a result, I need to enable that part of systemd. Another thing I’ll want to do here is work around a current systemd issue with respect to bridges. To do so, I need to make the directory recipes-core/systemd/ and create the file systemd_%.bbappend in that directory with the following:

do_install_append() {
    # There are problems with bridges and this service, see
    # https://github.com/systemd/systemd/issues/2154
    rm -f ${D}${sysconfdir}/systemd/system/network-online.target.wants/systemd-networkd-wait-online.service
}

PACKAGECONFIG_append = " iptc"

The first part of these changes is to work around the github issue mentioned in the comment. While it’s quite frustrating that the issue in question has been open since the end of 2015, I can easily work around it by just deleting the service for my use case. The second part is to add iptc to the PACKAGECONFIG options, and in turn that part of systemd will be enabled and it can support iptables so the IPMasquerade flag above will work.

At this point, I can now build core-image-minimal again and have a system that will automatically bring up the network. However, it is not a router yet. In the next part of the series I will create a new image that is intended to be a router and customize that.

[Go to Part Three of the series.]