Building a DIY SOHO router, 18 months later

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, 18 months later

It’s been around a year since my last post on this project, and 18 months since I posted the article series itself. In the world of home networking equipment that’s practically a lifetime. So it’s worth doing a check-in on this project, I think.

My little router is still going just fine. In terms of things I had talked about earlier in the series, I’m still using the ability to easily roll back a test image to try out backports. Most recently, I tried to add IPv6 Prefix Delegation support to the DHCPv6 client in an older version of systemd. It didn’t work, but with just a reboot I was back to my normal image. As times change, devices at home change, and I’m using the dnsmasq configuration file to document what devices are in the house. Performance? Still doing just great. The next time I do one of these series, my household might have crossed the threshold where newer WiFi standards are in enough devices we own that I want something that handles them, but we aren’t there yet. By then I might also just have a new PCIe card to drop in.

In terms of the article series itself, it’s a testament to the stability of the Yocto Project and everyone who works on it. Since the original posts, Mender has internally upgraded a significant amount, and in turn some of the examples there aren’t quite correct for a modern build. Mender has excellent documentation, however, and it’s been easy to update my build to work all along the way and upgrade from the older version to the new. Everything else? Yes, that’s all still correct. What prompted this particular post is that the Yocto Project has announced another milestone release, and that’s always a good time to make sure my device is up to date. My latest changes were quite literally just renaming the Linux kernel bbappend file and saying the layer is compatible with the new release. That was less work than updating the firewall rules to take in to account both Daylight Savings Time changes and that my kids are a bit older and realistically want to be online a little later.

Using Rust with Yocto Project

by Paul Barker, Konsulko Group

Introduction

Rust is a modern programming language which emerged from Mozilla Research around 2010 and has been stable since 2015 (see the history section of the Wikipedia article). Rust is an excellent fit for embedded software development due to its focus on performance and safety without the overhead of garbage collection or a large language runtime. The zero-cost abstractions provided by Rust support developer productivity, allowing the use of traits, generics, functional programming styles and other high-level approaches in the knowledge that the resulting assembly code will perform just as well as more verbose, hand written lower level code.

Now is a great time to pick up Rust if you work in the Embedded space. The Rust Embedded Working Group has produced an Embedded Rust Book for an audience with existing embedded development knowledge and the Discovery Book for an audience with no embedded experience but a desire to learn. The Linux kernel community has began discussing the use of Rust within the kernel itself and the OpenEmbedded/Yocto Project community is planning to move Rust support into the openembedded-core layer during the next release cycle.

The benefits of Rust largely stem from the language’s memory model, the borrow checker and other features which will be new to most developers learning Rust. The above resources from the Rust Embedded WG along with several published books and online tutorials cover this material well and so it will not be repeated in detail here. However, if you wish to develop software in Rust for production deployment you should expect to invest at least a few weeks in learning the language, assuming existing familiarity with other programming languages like C, Python or Java.

This article will introduce a couple of simple example applications written in Rust, discuss how to compile and run these applications natively on a Linux system using the Cargo build system and show how to create Yocto Project recipes for these applications using the cargo-bitbake tool. Some basic knowledge of the Yocto Project is assumed but no previous experience of Rust should be required.

Getting Started with Rust

In this section we will install Rust on our Linux system and create two simple applications.

Note that it is not necessary to have Rust installed natively in order to build applications and libraries written in Rust under bitbake. However, installing Rust natively does allow us to develop, test and debug applications on the host before we try to cross-compile them and it is an important step to take when learning Rust.

Installing Rust

The recommended way of installing Rust on a Linux system is to download and run the official installation script. When prompted with the installation options it is usually sufficient to select option 1 to use the defaults. This will install Rust in the current user’s home directory and modify the profile files to set necessary environment variables at login.

$ curl https://sh.rustup.rs -sSf | sh

    <output snipped>

Current installation options:


   default host triple: x86_64-unknown-linux-gnu
     default toolchain: stable (default)
               profile: default
  modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>1

    <output snipped>

Rust is installed now. Great!

To get started you need Cargo's bin directory ($HOME/.cargo/bin) in your PATH
environment variable. Next time you log in this will be done
automatically.

To configure your current shell run source $HOME/.cargo/env

After installing Rust you should log out and log back in to ensure that environment variables are set correctly.

Hello, World

As is traditional, we will begin by looking at a “Hello, World” application.

Each Rust application or library should have its own project directory. Within this directory, the source code is usually placed in a src directory and the top level directory contains the Cargo config file for the project
and any other collateral files such as a README file, LICENSE file, etc.

To create our project directory and populate it with initial files we can use the cargo init command:

$ cargo init hello-rs
     Created binary (application) package

Within our new hello-rs directory, this command creates an initial source file in src/main.rs with the following contents:

fn main() {
    println!("Hello, world!");
}

This makes our job of writing a Hello World application very simple – no code modifications are needed!

The cargo init command also creates a Cargo.toml file in our project directory to store project configuration and metadata:

[package]
name = "hello-rs"
version = "0.1.0"
authors = ["Paul Barker <[email protected]>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

We should test our new Hello World application natively before we try to cross-compile it under bitbake. To do this we can use the cargo build and cargo run commands:

$ cargo build
   Compiling hello v0.1.0 (/home/pbarker/Projects/Rust/rust/hello-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 0.29s
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/hello-rs`
Hello, world!

The last stage for our hello world application is to publish it as a git repository so that it can be fetched and built in our Yocto Project environment. We should create a LICENSE file and a README file in the top level of the project for the benefit of anyone who wants to reuse our code. My preference is for the Apache license version 2.0 but feel free to use any appropriate open source license here.

We should also add some extra metadata to the Cargo.toml file to specify the license, briefly describe the project and link to the location where we will be publishing the repository. This metadata will be used by cargo-bitbake when we generate a recipe for our application. In my case the repository will be published to my GitLab account.

--- a/Cargo.toml
+++ b/Cargo.toml
@@ -3,6 +3,9 @@ name = "hello-rs"
 version = "0.1.0"
 authors = ["Paul Barker <[email protected]>"]
 edition = "2018"
+license = "Apache-2.0"
+description = "Hello World application"
+repository = "https://gitlab.com/pbarker.dev/rust/hello-rs.git"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

To finish up, commit the project files to a git repository and push to a location which can be reached when building using Yocto Project. Alternatively, you can make use of the hello-rs repository I published under my GitLab account.

The Hello World application above will demonstrate only part of the capabilities of the cargo-bitbake tool and the Rust support in Yocto Project. To see how Rust dependencies are handled in Yocto Project we’ll need an application which actually depends on a Rust library (known as a crate). The print-rand application will fill this role as it uses the rand crate to generate random numbers.

As before we will start by creating the project using cargo init:

$ cargo init print-rand
     Created binary (application) package

We will now modify the src/main.rs file to generate and print a random number:

use rand::prelude::*;

fn main() {
    let x: u32 = random();
    println!("x = {}", x);
}

We will also modify the Cargo.toml file to list the rand crate as a dependency:

[package]
name = "print-rand"
version = "0.1.0"
authors = ["Paul Barker <[email protected]>"]
edition = "2018"

[dependencies]
rand = "0.7"

We can build and test this application in a similar way to the previous Hello World app. Note that each dependency is downloaded and built along the way:

$ cargo build
   Compiling libc v0.2.77
   Compiling getrandom v0.1.15
   Compiling cfg-if v0.1.10
   Compiling ppv-lite86 v0.2.9
   Compiling rand_core v0.5.1
   Compiling rand_chacha v0.2.2
   Compiling rand v0.7.3
   Compiling print-rand v0.1.0 (/home/pbarker/Projects/Rust/rust/print-rand)
    Finished dev [unoptimized + debuginfo] target(s) in 3.60s
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/print-rand`
x = 643356417

As before, we should add LICENSE and README files at this stage and extend the metadata in the Cargo.toml file:

--- a/Cargo.toml
+++ b/Cargo.toml
@@ -3,6 +3,9 @@ name = "print-rand"
 version = "0.1.0"
 authors = ["Paul Barker <[email protected]>"]
 edition = "2018"
+license = "Apache-2.0"
+description = "Print a random number"
+repository = "https://gitlab.com/pbarker.dev/rust/print-rand.git"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

Lastly, commit the project files to a git repository and push to a public location. As before you may alternatively make use of the print-rand repository I published under my GitLab account.

Using cargo-bitbake

Now that we have created a couple of simple Rust applications and published them we can look at creating bitbake recipes using the cargo-bitbake tool. The first step of this process is to install the tool itself:

$ cargo install cargo-bitbake

    <output snipped>

   Compiling cargo-bitbake v0.3.14
    Finished release [optimized] target(s) in 12m 30s
  Installing /home/debian/.cargo/bin/cargo-bitbake
   Installed package `cargo-bitbake v0.3.14` (executable `cargo-bitbake`)

Before we run cargo bitbake for each our projects we should ensure that all changes are committed and pushed to the remote repository. We should also check the remote URL for each repository as this will be used to set SRC_URI. If the remote URL uses SSH access then you should ensure that this can be reached using an SSH key from the Yocto build environment or switch to HTTPS access.

Once we’re happy with the state of our repository we can run cargo bitbake in the top directory of our hello-rs project (note that we use a space not a dash in the command here):

$ cargo bitbake
No package.homepage set in your Cargo.toml, trying package.repository
Wrote: hello_0.1.0.bb

The message printed by cargo bitbake indicates that the repository entry in our Cargo.toml file was used as the homepage URL since we did not specify a separate homepage.

We can now look at the generated recipe file hello-rs_0.1.0.bb:

# Auto-Generated by cargo-bitbake 0.3.14
#
inherit cargo

# If this is git based prefer versioned ones if they exist
# DEFAULT_PREFERENCE = "-1"

# how to get hello-rs could be as easy as but default to a git checkout:
# SRC_URI += "crate://crates.io/hello-rs/0.1.0"
SRC_URI += "git://gitlab.com/pbarker.dev/rust/hello-rs.git;protocol=https;nobranch=1;branch=dev"
SRCREV = "cec42fb0d147e4a4b271256c475620ff627c8856"
S = "${WORKDIR}/git"
CARGO_SRC_DIR = ""


# please note if you have entries that do not begin with crate://
# you must change them to how that package can be fetched
SRC_URI += " \
"



# FIXME: update generateme with the real MD5 of the license file
LIC_FILES_CHKSUM = " \
    file://LICENSE;md5=86d3f3a95c324c9479bd8986968f4327 \
"

SUMMARY = "Hello World application"
HOMEPAGE = "https://gitlab.com/pbarker.dev/rust/hello-rs.git"
LICENSE = "Apache-2.0"

# includes this file if it exists but does not fail
# this is useful for anything you may want to override from
# what cargo-bitbake generates.
include hello-rs-${PV}.inc
include hello-rs.inc

As we noted above, the SRC_URI and SRCREV entries are based on the remote URI and the currently checked out commit of our local git repository. The SUMMARYHOMEPAGE and LICENSE entries are based on the metadata in our Cargo.toml file. The LIC_FILES_CHKSUM entry is set based on the LICENSE file in our project.

We can also generate a recipe for the print-rand project by running cargo bitbake in the top directory of this project:

$ cargo bitbake
No package.homepage set in your Cargo.toml, trying package.repository
Wrote: print-rand_0.1.0.bb

The print-rand_0.1.0.bb recipe is very similar to the previous recipe for our hello world application. The SRC_URISRCREVSUMMARY and HOMEPAGE are changed to the appropriate values for this project. The SRC_URI variable is extended with a list of the dependency crates required by this project, this includes not just the rand crate that we explicitly listed as a dependency in our Cargo.toml file but also all the recursive dependencies of this crate.

# Auto-Generated by cargo-bitbake 0.3.14
#
inherit cargo

# If this is git based prefer versioned ones if they exist
# DEFAULT_PREFERENCE = "-1"

# how to get print-rand could be as easy as but default to a git checkout:
# SRC_URI += "crate://crates.io/print-rand/0.1.0"
SRC_URI += "git://gitlab.com/pbarker.dev/rust/print-rand.git;protocol=https;nobranch=1;branch=dev"
SRCREV = "3397247f929f28d70adbb65d3990dc72699553bb"
S = "${WORKDIR}/git"
CARGO_SRC_DIR = ""


# please note if you have entries that do not begin with crate://
# you must change them to how that package can be fetched
SRC_URI += " \
    crate://crates.io/cfg-if/0.1.10 \
    crate://crates.io/getrandom/0.1.15 \
    crate://crates.io/libc/0.2.79 \
    crate://crates.io/ppv-lite86/0.2.9 \
    crate://crates.io/rand/0.7.3 \
    crate://crates.io/rand_chacha/0.2.2 \
    crate://crates.io/rand_core/0.5.1 \
    crate://crates.io/rand_hc/0.2.0 \
    crate://crates.io/wasi/0.9.0+wasi-snapshot-preview1 \
"



# FIXME: update generateme with the real MD5 of the license file
LIC_FILES_CHKSUM = " \
    file://LICENSE;md5=86d3f3a95c324c9479bd8986968f4327 \
"

SUMMARY = "Print a random number"
HOMEPAGE = "https://gitlab.com/pbarker.dev/rust/print-rand.git"
LICENSE = "Apache-2.0"

# includes this file if it exists but does not fail
# this is useful for anything you may want to override from
# what cargo-bitbake generates.
include print-rand-${PV}.inc
include print-rand.inc

We can now add these recipes to a Yocto Project layer and build them using bitbake. Support for the Rust toolchain in Yocto Project is provided by the meta-rust layer so this layer must be included before these recipes can be built. As there are many ways to put together a Yocto Project build we won’t cover this in this blog post. However, you can see a walkthrough of this process in the Using Rust with Yocto Project demo presented at theYocto Project Summit Europe 2020.

How Mender works

by Tom Rini, VP Engineering

Software Update solutions are a key part of our services offering. For open source over-the-air updates, we often we recommend and work with mender.io. In fact, Konsulko Group is a Mender Authorized Referral Partner. Recently, a prospective customer expressed an interest in knowing more about how Mender works. Here’s the brief, informal introduction to member.io that I prepared, and now I’m sharing with you.

As a high level starting point, https://mender.io/how-it-works provides a good overview of what’s supported and what it covers. In short, Mender starts off by providing support for a traditional “A/B” approach to system updates, where if the update isn’t marked as valid (and there’s hooks for the application(s) to verify the system before this is done), it’s assumed invalid and the system will roll-back automatically. While “OTA” implies over the network, it can just as easily be done by providing (and validating) a USB key that contains an update.

One of the reasons we recommend Mender is that it has very good in-depth documentation. The starting point for all of that is https://docs.mender.io/2.4/ which covers all of the topic starting from how to implement Mender support in a device and including how to create your own server infrastructure if you don’t want to use their paid service. While there are a number of important pages there, one that I like to highlight is https://docs.mender.io/2.4/artifacts/state-scripts as it shows the state machine for an update and talks about some of the common use cases that come up for user interaction or dealing with failures.

Another place I want to call out is https://docs.mender.io/2.4/devices/update-modules which is also mentioned in the first link. This is how Mender is extended to provide updates for other parts of the system that are not the rootfs itself.  Since updating a Docker container is something that has been mentioned before I want to also note https://hub.mender.io/t/docker/324 as it is a well supported module for this specific case.

I hope you find this information useful. Please contact us if you have specific questions. We’re looking forward to talking to you about your own specific OTA needs.

Getting Started with RAUC on Raspberry Pi

RAUC is a secure, robust and flexible open source software for A/B updates of Embedded Linux devices. It is appropriate for various use cases and it is compatible with all popular build systems: The Yocto Project/ OpenEmbedded, Buildroot and PTXdist.

Konsulko Group engineers have experience with all popular open source solutions for software over the air updates of embedded Linux devices, including Mender, SWUPdate, HERE OTA Connect based on OSTree and Aktualizr. In this article we will discuss the exact steps to integrate RAUC with the Yocto Project (YP) and OpenEmbedded (OE) for Raspberry Pi – the most popular single board computer among students, hobbyists and makers.

For the practical example in this article we will be using the latest and greatest Raspberry Pi as of the moment: Raspberry Pi 4 Model B. Versions with different RAM sizes are available on the market. Any of these Raspberry Pi 4 Model B versions are OK for this RAUC demonstration.

As long time developers and users of the Yocto Project and OpenEmbedded, both have become favorite tools for creating customized distributions for Konsulko engineers. We frequently use and support them commercially. The Yocto Project is a Linux Foundation collaborative open source project for creating custom Linux distributions for embedded devices. It is based on Poky, the reference distribution of the Yocto Project, using the OpenEmbedded build system. The Yocto Project releases on a 6-month cadence. As of the time of this writing, the latest stable release is Dunfell (3.1).

RAUC is a powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image for Raspberry Pi built with the Yocto Project and OpenEmbedded, it requires:

  • U-Boot as a bootloader
  • Enabled SquashFS in the Linux kernel configurations
  • ext4 root file system
  • Specific partitioning of the microSD card that matches the RAUC slots
  • U-Boot environment configurations and a script to properly switch RAUC slots
  • Certificate and a keyring to RAUC’s system.conf

RAUC is capable of covering various use cases and scenarios, including advanced options for single or redundant data partitions. Upgrades are performed through the so called RAUC bundles. It is possible to install them over the air or using the old-fashioned method with a USB stick. For managing updates to a fleet of Internet of Things, it is possible to integrate RAUC with Eclipse hawkBit project that acts as a deployment server with a nice web user interface.

For the sake of simplicity, this article focuses on the most simple and straight-forward use case with 2 identical RAUC slots: A and B. For each slot we will have a separate partition on the microSD card for Raspberry Pi. We have already covered most of the RAUC requirements in an additional Yocto/OE layer called meta-rauc-raspberrypi. We will use it to put the pieces together. First we will build a minimal bootable image for Raspberry Pi 4 with RAUC. We will flash it to both A and B slots. After that we will build a RAUC bundle that adds the text editor nano. Finally we will install this RAUC bundle on the B slot, reboot and verify that nano is present.

Building a Linux Distribution with RAUC

Follow the steps below to build a minimal image for Raspberry Pi with Yocto, OpenEmbedded and RAUC as well as to perform a software update:

  • Download Poky, the reference distribution of the Yocto Project:

git clone -b dunfell git://git.yoctoproject.org/poky poky-rpi-rauc
cd poky-rpi-rauc

  • Download meta-openembedded layer:

git clone -b dunfell git://git.openembedded.org/meta-openembedded

  • Download Yocto/OE BSP layer meta-raspberrypi:

git clone -b dunfell git://git.yoctoproject.org/meta-raspberrypi

  • Download Yocto/OE layers for RAUC:

git clone -b dunfell https://github.com/rauc/meta-rauc.git

git clone -b dunfell https://github.com/leon-anavi/meta-rauc-community.git

  • Initialize the build environment:

source oe-init-build-env

  • Add layers to conf/bblayers.conf:

bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake-layers add-layer ../meta-openembedded/meta-python/
bitbake-layers add-layer ../meta-openembedded/meta-networking/
bitbake-layers add-layer ../meta-openembedded/meta-multimedia/
bitbake-layers add-layer ../meta-raspberrypi/
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-raspberrypi/

  • Adjust conf/local.conf for Raspberry Pi 4 with systemd and RAUC by adding the following configurations to the end of the file:
MACHINE = "raspberrypi4"

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

IMAGE_INSTALL_append = " rauc"

IMAGE_FSTYPES="tar.bz2 ext4 wic.bz2 wic.bmap"
SDIMG_ROOTFS_TYPE="ext4"
ENABLE_UART = "1"
RPI_USE_U_BOOT = "1"
PREFERRED_PROVIDER_virtual/bootloader = "u-boot"

WKS_FILE = "sdimage-dual-raspberrypi.wks"
  • Build a minimal bootable image:

bitbake core-image-minimal

NOTE: Building an image from scratch requires a lot of operations and takes some time so please patiently wait until bitbake completes all tasks.

  • Flash the image to a microSD card and boot it on Raspberry Pi 4:

sudo umount /dev/sdX*
bzcat tmp/deploy/images/raspberrypi4/core-image-minimal-raspberrypi4.wic.bz2 | sudo dd of=/dev/sdX
sync

  • Attach USB to UART debug cable to Raspberry Pi 4, plug ethernet cable and the microSD card. Turn on Raspberry Pi 4. Verify that the system boots successfully.
  • Now, let’s extend the image with the simple text editor nano by adding the following line to the end of conf/local.conf:

IMAGE_INSTALL_append = " nano"

  • Build a RAUC bundle:

bitbake update-bundle

  • Start a web server:

cd tmp/deploy/images/raspberrypi4/
python3 -m http.server

  • On the Raspberry Pi download the RAUC bundle, install it and reboot the board:

wget http://192.168.1.2:8000/update-bundle-raspberrypi4.raucb -P /tmp
rauc install /tmp/update-bundle-raspberrypi4.raucb
reboot

  • After successful upgrade with RAUC reboot the Raspberry Pi and verify that nano is now present:

which nano

  • Check RAUC status to confirm that now the second partition has been booted:

rauc status

For Internet of Things and other real-world products, the whole build procedure with the Yocto Project and OpenEmbedded can be optimized further to just a few commands for easy implementation of continuous integration (CI).

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We have experience with RAUC and various other open source solutions for software updates. Please contact us if you need your “own” rock-solid Linux distro for your own embedded product.

 

Helping Yocto Project work with Python 3

According to the statistics from StackOverflow Python is the fastest-growing major programming language. First released in 1991, Python is nowadays commonly used for various applications in multiple different industries. Python is a first class citizen of many embedded Linux systems.

The Yocto Project, a collaborative project of the Linux Foundation for creating custom Linux distributions for embedded devices, uses the OpenEmbedded build system and relies on layer meta-python from meta-openembedded to deliver Python 3 packages. Until recently, meta-python was providing both python 2 and python 3 versions of each package. The Python community decided that January 1, 2020, was the day to sunset Python 2. Since then Python 2 has been officially deprecated. This triggered major changes related to the support in Yocto and OpenEmbedded. All recipes for version 2 were moved to layer meta-python2 to provide legacy support after the end of life for this Python release. In meta-openembedded/meta-python, the OpenEmbedded community started efforts to remove all recipes for version 2 as well as to consolidate inc and bb files into a single bb file for version 3.

Konsulko Group engineers are regular contributors to various upstream open source projects, including meta-openembedded and more specifically to meta-python. In the past month, Leon Anavi joined the community efforts for consolidating Python 3 recipes in a single file as well as for upgrading various packages. Nowadays, most of the Python 3 recipes are utilizing the pypi bbclass which takes care for downloading and processing packages from pypi.org. This makes most of the upgrades to new releases of a Python package straight-forward. However, it is important to check the list of build and runtime dependencies as well as to ensure that bitbake still works fine with the upgraded recipe version for both x86-64 and ARM architectures prior to submission. 

Let’s have a closer look at the recipe python3-protobuf. It has been recently upgraded from version 3.11.3 to version 3.12.2. Protocol Buffers, also known as protobuf, are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. In the Yocto and OpenEmbedded ecosystem, recipe python3-protobuf depends on recipe protobuf from layer meta-oe. Both meta-oe and meta-python are part of meta-openembedded. So to avoid version mismatch and to ensure that bitbake will be able to successfully build python3-protobuf version 3.12.2 an upgrade of recipe protofobuf to the same version was mandatory. We contributed both upgrades to the master branch of the git repository meta-openembedded. The maintainers took care of cherry-picking them to the dunfell branch which is compatible with the latest stable release of the Yocto Project as of the moment. As a result, if you checkout the latest stable release of Poky, the reference system of the Yocto Project, and meta-openembedded you will be able to quickly build the latest version of protobuf and python3-protobuf out of the box.

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We continue to regularly make upstream contributions to these open source projects. Please contact us if you need your “own” Linux distro for your own embedded product.

Building a DIY SOHO router, 6 months on

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, 6 months on

A little more than six months ago, I posted part 4 of our series on making a SOHO router using the Yocto Project and OpenEmbedded. After 6 months of deployment, this is a good time to follow up on how the router has worked out in residential use.  The zeus code-name Yocto Project release was just announced, and that means that the release we started out with in part 1 is now close to being out of support.  That’s not a problem since we designed in support for moving to new software releases using Mender to deliver software updates.

One of the most important metrics in a project like this is, how does it perform?  From the standpoint of a family of 4 heavy internet users, it’s gone really well.  The WiFi range is only a little better than before, but that’s not really a function of the software.  Making everyone use Pi-hole has only resulted in a small number of places where I needed to override the blacklist and allow something in.  From an end-user point of view, this has worked as well as any off-the-shelf router.  From the administrator point of view, I’ve done scheduled maintenance during the day on a weekend, and it really did take only the 5 minutes I promised everyone rather than turning into one of those worst case scenarios where something broke and it takes an hour to fix it.  In fact, the update portion of the plan has gone exceedingly well.  While I didn’t make a post about moving to warrior from thud, I did that transition a while ago and it went smoothly.  Mender introduced a change that required attention be paid while migrating, but it was documented and went smoothly.  On the metadata side, the upgrade was as easy as one could hope.  A few of the bbappend files needed to be updated for a new version, and some of the changes I had made and pushed upstream as part of the original series were now just included, so they got dropped from my layer.

One of the things I touched on in the series was about using the update functionality to test development changes in a production environment.  The chance to do that came up with a systemd-networkd issue that was a problem in my local setup.  The upstream project requested people verify the problem exists with newer versions of systemd and a new enough version was available in what would become zeus.  So I made a quick weekend project of doing an update of my layers to build with a newer version of all of the metadata, removed the work-around, and flashed the image in place.  A quick reboot confirmed that the issue was indeed fixed, and then rather than commit to running an otherwise in-progress release I simply rebooted and automatically rolled back to my stable release.  With the network back up again, I updated the issue in upstream Bugzilla to let them know the problem was fixed.  After a bit longer, a few other people also confirmed it worked for them and now the issue is resolved.

In terms of the metadata itself, there have been a few clean-ups to what I did in my own layer with each release update I’ve done.  In the series I left out what hardware I was building on, and I also left out talking about using the linux–yocto recipe.  Since I first wrote the series linux-yocto has become easier to use, and I found this as part of reviewing my own changes like they were brand new with each upgrade.  I was setting some variables that initially didn’t have reasonable default values, and now they do and I don’t need to set them myself.  This in fact means that moving forward, rather than a version-specific kernel bbappend file, I can go with an always-used one to enable the additional kernel CONFIG options that I need for a time-based firewall.

I started out by mentioning that zeus has been released, and I’m working on migrating to it as I write this.  In fact, it’s so new that I’m doing my own little port of the meta-mender core layer to zeus for my needs. I expect that by the time I do my first update from one build of zeus to the next there will be an official update I’ll be able to use instead.  Looking forward, this was a great little project that also was a lot of fun.  The goals I set way back at the start have been met, and I’m happier with my network than I have been in a long time.  Of course, the list of features an off-the-shelf system provides is always growing, and there’s now monitoring and display items on my weekend project list now to keep up.  I foresee using and improving this setup for a long time to come.

Custom Linux Distro for NVIDIA CUDA Devices

How to get started and build a minimal custom Linux distribution for embedded NVIDIA CUDA-enabled devices using the Yocto Project (YP) and OpenEmbedded (OE).

Building a DIY SOHO router, Part 4

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 4

In part three of this series I finished putting together what I wanted to have on my SOHO router and declared it to be done. While I plan to revisit the topic of a SOHO router using the Yocto Project and OpenEmbedded, this is the final part of the series. In this part, I want to focus on some of the things that I learned while doing the project.

The first thing is that I learned a lot about IPv6, specifically how it’s usually implemented within the United States for residential customers, and some of the implications of this implementation. The first thing to note is that I’ve been off-and-on trying to enable IPv6 for general IPv6 connectivity at home for some time now. Long before my ISP offered IPv6 service, I used Hurricane Electric to have a IPv6 tunnel and connectivity. This was great and only sometimes lead to problems, such as when Netflix finally supported IPv6 and began blocking well known tunnels for region-blocking reasons. It wasn’t until I started on this project that I decided to try to make real use of  routable addresses for hosting personal services. My expectations, and that of lots of software designed to manage IPv6 as well, are best described in article from RIPE about understanding IP addressing. In short, my house or “subscriber site” should get at least 256 LAN segments to do with as I want. Docker expects to have it’s own LAN segment to manage as part of configuring network bridges. When you have 256 or more LAN segments to use, that’s not a problem at all.

Unfortunately, my ISP provides only a single LAN segment. This is simultaneously more IPv6 addresses than the whole of IPv4 and something that should not be further subdivided in routing terms. I could subdivide my LAN segment, but this would in turn cause me to have to do a whole lot more work and headaches. That’s because at the routing level IPv6 is designed for my segment to be the smallest unit. Rather than deal with those headaches I switched my plans up from using Docker to using LXC. With LXC it’s easy to dump the container onto my LAN and then it picks up an IPv6 address the same way all of the other machines on my LAN do. This is good enough for my current needs, but it will make other things a lot harder down the line, if I want separation at the routing level for some devices.

But why am I doing that at all? Well, one of the benefits of having a small but still capable router is that I can run my own small services. While I don’t want to get into running my own email I think it makes a whole lot of sense to host my own chat server for example. With closed registration and no (or limited later on perhaps) federation with other servers I don’t need to worry about unauthorized users contacting my family nor do I have to worry about the company deciding it’s time to shutdown the service I use.

Another lesson learned is that while the Yocto Project has great QA, there’s always room to improve things. As part of picking a firewall I found that one of the netfilter logging options had been disabled by accident a while back. As a part of writing this series of articles and testing builds for qemux86-64, I found that one of the sound modules had been disabled. As a result, the instructions I wrote back in part 2 wouldn’t work. Working upstream is always fun and these changes have been merged and will be included in the next release.

I also worked on a few things for this project that I didn’t include directly in the relevant part of the series. For example, while I did include a number of full utilities in the list of packages installed in the router, I didn’t talk about replacing busybox entirely. This is something that OpenEmbedded supports using the PREFERRED_PROVIDERS and VIRTUAL-RUNTIME override mechanisms in the metadata. Prior to this article however, there wasn’t a good example on how to do this in upstream. Furthermore, there wasn’t an easy way to replace all of busybox and instead you had to list a single package and then include the rest of the required packages in your IMAGE_INSTALL or similar mechanism.  I am a fan of using busybox in places where I’m concerned about disk usage. However, on my router I have plenty of disk space so I want to be sure that if I have to go and solve a problem I’m not using my swiss army knife but rather have my full toolbox available. As a result, OpenEmbedded Core Master now has packagegroup-core-base-utils and a documented example of how to use that in local.conf.sample.extended. This means that when I refresh this image to be based on the Warrior branch I can remove a number of things from my IMAGE_INSTALL.

Another lesson is that old habits die hard.  In general, I always try to use the workflow where I make a change outside the device I’m working on, build the change in, and test it, rather than editing things live.  But when it’s “just” a quick one line change I’ll admit I do it live and roll it into my next build sometimes.  And then sometimes I forget to roll all my changes back up.  So while implementing this project I tried even harder than usual to not fall into that “just a quick change” mindset.  For the most part I’ve been successful at sticking to the idea workflow.  I really believe stateless is the right path forward.  And “for the most part” means that, yes, one time I did have to make use of the fact that the old rootfs was still mountable and copied a file over to the new rootfs, and then to the build machine.  I like to think of that as a reminder that A/B updates are more helpful than a “rewrite your disk each time” workflow for those occasional mistakes.

The caveat to the lesson above is because I really did the “git, bitbake, mender” cycle on this project. I didn’t start on it quite as soon as I said in the article, and I spent a lot more time toying with stuff in core-image-minimal instead of following my own advice, too.  I suppose that is the difference between writing a guide on how things should be done compared with how you do things when you just want to test one more thing, then switch over.  I really should have switched earlier however as every time I avoid doing the SD card shuffle it’s a win on a number of levels.

Did I say SD card above?  Yes, I did.  For this project, a 64GB “black box” that’s in the form-factor of a SD card will have as long of a life span as there is in the form-factor of a M.2 SSD or any other common storage format.  While my particular hardware has a SATA port, I don’t want to try to fit the required cabling, let alone the device itself in the case that’s recommended.  I will admit that I’m taking a bit of a risk here, I am putting as much frequent-write files under a ramfs as I can and after all, I did say stateless is a goal.  If everything does really die on me, I can be back up and running fairly quickly.

Last thing I learned is something I knew all along, really. I like the deeper ownership of the router. There’s both the pride and accomplishment in doing it and that “old school” fun of being the admin again, for real.

Building a DIY SOHO router, Part 3

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 3

In part two of this series I created a local configuration layer for OpenEmbedded, and had the build target core-image-minimal producing an image. The image that was produced wasn’t really a router, but did let us bring up our board and look around. In this article, I’m going to create a custom image and populate it with additional software packages configured to my  requirements. I’m also going to get started using Over-The-Air (OTA) software updates on the device.

Now that I’ve proven that the image works on the hardware, I can really get down to implementing the project of making a router.  While I could continue to add things to core-image-minimal, it really makes sense at this point to stop and create my own image. Since I want something relatively small, I will still start with core-image-minimal as the base.  Moving back over to meta-local-soho, I’m creating the recipes-core/images directory and then populating core-image-minimal-router.bb with:

require recipes-core/images/core-image-minimal.bb

DESCRIPTION = "Small image for use as a router"

IMAGE_FEATURES += "ssh-server-openssh"
IMAGE_FEATURES += "empty-root-password allow-empty-password allow-root-login"

IMAGE_INSTALL += "\
    "
MENDER_STORAGE_TOTAL_SIZE_MB = "4096"

This tells bitbake that it must have core-image-minimal.bb available and to include it. I then provide a new DESCRIPTION to describe the new image. Next, I include a number of new features in the image. First, I’ll use the normal hook for adding a SSH server. Then I’ll add a line of features for development mode that I’ll remove later.  These features, as their names imply, allow for root to login without a password. This is quite handy for development and quite unwise for production. I’ll circle back and remove these development features later. Next, I give myself an empty list of additional packages to be filled out later. Finally, I tell Mender that it has 4096 megabytes of disk space to work with.  I’m going to hide space from Mender so that I can entirely control that part myself instead. At this point I can build core-image-minimal-router, and it will complete very quickly as I’ve not yet added any packages that have not been previously built. So it’s time to once again git add, git commit, and bitbake these changes.

At this point, I want to flash the new image onto the device and boot it up. The reason for this is that the new image can be used with Mender to test any subsequent image builds. The system is now functional enough to support delivering new image updates via Mender, so it’s good to get into the habit of using the OTA update workflow. It also forces me to treat the device as if it’s really stateless. I’ll talk about how to apply an OTA update when I make the next set of changes.

Now it’s time to begin adding content to the custom image. The first thing I’m going to do is borrow some logic from packagegroup-machine-base. I don’t want to use this packagegroup directly because it will cause bitbake to build a lot of extra stuff that I don’t end up installing. This is due to the fact that it’s part of packagegroup-base.bb (because it’s needed to resolve dependencies of other parts of the packagegroup). Instead, I’m going to add:

    ${MACHINE_EXTRA_RDEPENDS} \
    ${MACHINE_EXTRA_RRECOMMENDS} \

to IMAGE_INSTALL so that any additional machine-specific functionality that’s been specified is installed to the image. Next, I’ll add in kernel-modules to the list so that all of the modules that have been built for the kernel are installed to the image. This will be a lot easier than listing out every module I may need, especially for later on when it comes to various firewall rules I want to use.  On top of all of this, I also want to drop in a bunch of full-versions of common packages I use, and then let busybox fill in the rest.

    bind-utils \
    coreutils \
    findutils \
    iputils-ping \
    iputils-tracepath \
    iputils-traceroute6 \
    iproute2 \
    less \
    ncurses-terminfo \
    net-tools \
    procps \
    util-linux \

Almost everything in this list can be tweaked as desired. There are a couple items that serve a critical purpose and deserve an explanation:

  1. systemd calls out to $PAGER for many functions, including browsing logs with journalctl. If I don’t have the full version of less available, I won’t have a fully functional pager and browsing the output is extremely difficult.
  2. I don’t use xterm for my terminal emulator anymore so I want ncurses-terminfo installed. This ensures that the right terminfo is available and terminal output is correct.

At this point it’s time for a git add, git commit, and then a bitbake of our image too.

Now that I have a new image with additional content to try out, I want to put it on the device and confirm things work. As mentioned before, I’m using Mender in standalone mode since I have a single deployed device.  It’s very simple to serve the new image and then apply it. On the build machine, I do the following (change qemux86-64 to match the machine in use):

$ (cd tmp-glibc/deploy/images/qemux86-64; python3 -m http.server)

And then on the device:

# mender -rootfs http://build-server.local:8000/core-image-minimal-router-qemux86-64.mender
... wait while it downloads and applies ...
# reboot

Once the device comes back up, I’ve logged back in, and confirmed I’m satisfied with my changes, I do:

# mender -commit

This will mark what I am now running as the valid rootfs. However, if the device didn’t boot up or I couldn’t log in, I would simply not commit the changes. To do that I would then just reboot or otherwise power-cycle the device. If I don’t commit the changes to Mender then I get an automatic rollback to the previous install.  Of course, it’s also possible to use any HTTP server on the build machine.

At this point, it’s time to iterate over adding a number of different features that require little more than adding to IMAGE_INSTALL. Since I’ve talked about LXC, I need to add in lxc and gnupg (for verification of containers used from the download template). Once that’s added, I do the git add, git commit, bitbake, and then mender -rootfs cycle again and confirm LXC is working. One thing I noticed when doing this was that containers didn’t autostart because the service isn’t enabled by default.  Since I’m keeping this stateless, I changed that behavior with a bbappend file.  I also ended up installing e2fsprogs-mke2fs to be able to further partition my device to give LXC some room to work with.  This also means that I needed to have base-files provide the fstab that matches my setup, rather than the stock one.  Another small thing to cover is if your hardware does, or does not have a hardware random number genreator available.  If you do have one, you should pull in rng-tools on the image.  If you don’t have one however, you should install haveged to help feed the entropy pool instead.

Now I need to enable a functional access point. This is the first case where it’s really non-trivial to write up the config file to use, so it’s done a little bit differently.  The first step is to install hostapd and iw and boot that.  Now, on the device, edit /etc/hostapd.conf and iterate on editing and testing it on the device until everything is set up as desired. The iw tool can be helpful here to do things like perform a site scan to see what frequencies are already in use.  Once I’m done with the config, I copy the file out from the target and over to my build server with scp as /tmp/hostapd.conf. Then it’s time to make it stateless:

$ mkdir -p recipes-connectivity/hostapd/hostapd
$ cp /tmp/hostapd.conf recipes-connectivity/hostapd/hostapd/

And then I edit recipes-connectivity/hostapd/hostapd_%.bbappend to look like this:

FILESEXTRAPATHS_prepend := ":${THISDIR}/${PN}"

SRC_URI += "file://hostapd.conf"

do_install_append() {
    install -m 0644 ${WORKDIR}/hostapd.conf ${D}${sysconfdir}
}

SYSTEMD_AUTO_ENABLE_${PN} = "enable"

This will do two things. Everything except that last line is to tell bitbake to look in my layer for hostapd.conf and then to install it. The last thing is that now that we have a configured AP we want to start it automatically so have it be an enabled systemd service. Now it’s time once again for the git add, git commit, and so forth cycle.

The next step is to do the same kind of thing to dnsmasq. The good news that this time, the dnsmasq_%.bbappend file only needs one line:

FILESEXTRAPATHS_prepend := ":${THISDIR}/${PN}"

This is because the rest of the recipe already knows to grab dnsmasq.conf from a local file. In the case of my network, I need to pass in a few special options to some DHCP clients and have certain clients be given certain IP addresses, so I’ve gone with dnsmasq as my light-weight, but still fully featured IPv4 configuration server. I could have just as easily gone with ISC DHCPD instead, and it would look much the same as the above.  Conversely, if I didn’t need those few extra rules, I could just let systemd handle DHCP serving.  I left out IPv6 from my statement there as I am letting systemd handle that.

The only thing missing at this point from a router, aside from turning off developer mode features, is to add in a firewall. There are a few ways to go about this.  I already have systemd handling one of the aspects that is often associated with a firewall, setting up IPv4 NAT.  If the only other thing I needed on top of this is to shut the rest of the world out, I can use ufw and potentially even leverage its features that allow for adding iptables commands directly for slight enhancements.  While I have gone that direction for some projects, it’s not a good fit for this one. Instead, I chose to go with arno-iptables-firewall because I’m going to have a more complex setup. The process of customizing the firewall configuration is similar to how I customized hostapd and dnsmasq. That is, I iteratively configure it on the device, test for functionality, and copy the configuration files to my host.  This time, however, the arno-iptables-firewall_%.bbappend will look a little different:

FILESEXTRAPATHS_append := ":${THISDIR}/files"

SRC_URI += "file://firewall.conf \
            file://custom-rules \
"

do_install_append() {
    install -m 0644 ${WORKDIR}/firewall.conf \
    ${D}${sysconfdir}/arno-iptables-firewall/
    install -m 0644 ${WORKDIR}/custom-rules \
    ${D}${sysconfdir}/arno-iptables-firewall/
}

I have two files this time. The first one is the main config file, and the second one is the file that contains my custom rules. This is only necessary because I have a number of custom rules, otherwise it could be omitted.

At this point, looking back at the feature list I laid out in part one, I believe I can check all of my items off now.  I have the following all operational:

  1. access point
  2. firewall
  3. IPv4 and IPv6 network configuration
  4. containers 
  5. OTA software update

I’m building all of my software as hardened as my compiler will allow.  There’s very little state on the router itself to worry about backing up, and everything else is handled by my build server being backed up.  I’m confident in my OTA configuration as I’ve been using it for some time now in the development workflow. I’ve also tweaked the installed package list so that all of my favorite sysadmin tools are available.

At this point, it’s time to lock things down. First up, it’s time to go back to core-image-minimal-router.bb and remove that second line worth of IMAGE_FEATURES. Instead, I’m going to create a new local-user.bb recipe with my own user and SSH key. After listing local-user in IMAGE_INSTALL, I copy meta-skeleton/recipes-skeleton/useradd/useradd-example.bb to somewhere in meta-local-soho, and change it to look like this:

SUMMARY = "SOHO router user"
DESCRIPTION = "Add our own user to the image"
SECTION = "examples"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"

SRC_URI = "file://authorized_keys"

S = "${WORKDIR}"

inherit useradd

# You must set USERADD_PACKAGES when you inherit useradd. This
# lists which output packages will include the user/group
# creation code.
USERADD_PACKAGES = "${PN}"

USERADD_PARAM_${PN} = "-u 1200 -d /data/trini -r -s /bin/bash trini"

do_install () {
install -d -m 0755 ${D}/data/trini
install -d -m 0700 ${D}/data/trini/.ssh

install -m 0600 ${WORKDIR}/authorized_keys ${D}/data/trini/.ssh/

# The new users and groups are created before the do_install
# step, so you are now free to make use of them:
chown -R trini ${D}/data/trini
chgrp -R trini ${D}/data/trini
}

FILES_${PN} = "/data/trini"

# Prevents do_package failures with:
# debugsources.list: No such file or directory:
INHIBIT_PACKAGE_DEBUG_SPLIT = "1"

Now, there’s one slight problem. I added myself with a user under /data which is excluded from Mender updates. The good thing is I get persistent history and so forth.  The bad thing is I’m not installed there yet.  So I either need to re-flash one last time or manually copy the files over from the filesystem image to the device before I reboot.  Finally, I need to enable myself to use sudo. In addition to adding sudo to IMAGE_INSTALL I also need to either tweak the sudo recipe so that /etc/sudoers.d/ is looked under, tweak it so that anyone in the wheel group can use sudo and add a wheel group, or borrow the example from meta/recipes-core/images/build-appliance-image_15.0.0.bb and do the following in core-image-minimal-router.bb:

# Take the example from recipes-core/images/build-appliance-image_15.0.0.bb
# on adding more sudoers
fakeroot do_populate_poky_src () {
    echo "trini ALL=(ALL) NOPASSWD: ALL" >> ${IMAGE_ROOTFS}/etc/sudoers
}
IMAGE_PREPROCESS_COMMAND += "do_populate_poky_src; "

With all of that built, deployed, and unit tested, it’s time to go live.  My SOHO router is done and ready for production.  It’s now on me to make sure this stays up to date, which in some ways is a lot better than the alternative.  With my previous router, I only had an non-volatile RAM dump specific to the model of router as a backup. I now have my complete configuration containing firewall rules, DHCP options, and more saved. Since starting on the project I have even braved a few OTA updates and had minimal downtime.

This concludes the walk through of building a SOHO router with OpenEmbedded. In the final part of this series, I will describe some of the lessons I learned while designing and implementing this project.

[Go to Part Four of the series.]

Building a DIY SOHO router, Part 2

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 2

In part one of this series I explained some of my motivations for this project. Now it’s time to start on implementing the project itself. At this point I’m going to assume the reader has basic familiarity with using OpenEmbedded.  Otherwise, the Yocto Project (YP) quickstart guide and OpenEmbedded getting started pages are useful to bring yourself up to speed.  I assume that you’ve followed these instructions on how to prepare your build host and gone so far as to have completed a build for some target previous to this.  In this guide, I’m going to do my best to follow common best practices. Whenever I deviate from best practices, I’ll explain why we want something a bit different. After all, best practices are supposed to be taken as guidelines, not absolutes. Finally, I’m going to be working against the Yocto Project thud code-name release as that’s what is current as of this writing.

The first thing I’m going to do in my project is create a build directory now so that I can easily get access to various tools.  The next thing I’m going to do is use those tools to make a layer to store our configuration in.

$ . oe-core/oe-init-build-env
You had no conf/local.conf file. This configuration file has therefore been
created for you with some default values. You may wish to edit it to, for
example, select a different MACHINE (target hardware). See conf/local.conf
for more information as common configuration options are commented.
...
You can also run generated qemu images with a command like 'runqemu qemux86'
$ bitbake-layers create-layer ../meta-local-soho NOTE: Starting bitbake server... Add your new layer with 'bitbake-layers add-layer ../meta-local-soho' $ bitbake-layers add-layer ../meta-local-soho NOTE: Starting bitbake server... $

Be sure to change oe-core to wherever the core layer was checked out.  Now that the layer has been created, let’s go ahead and start by putting it into git. Why? I am a firm believer in “commit early and commit often” as well as “cleanup and rebase once you’re done”. To me, one of the big selling points of git is that you can track your work incrementally, and when you notice unexpected breakage later on you can easily go back in time and locate the bugs.

$ cd ../meta-local-soho
$ git init .
$ git add *
$ git commit -s -m "Initial layer creation"
$ git branch -m thud

With the first commit in place, it’s time to start customizing the layer. We don’t need the example recipe, so let’s remove it.

$ git rm -r recipes-example/example
$ git commit -s -m "Remove example recipe"

Next, it’s time for a more substantive set of customizations. I’ll start by editing the README. Why? To start with, I’m going to list all of the layers I know of that are dependencies at this point. The README will also be a handy place to note which physical port is for WAN, which port(s) are for LAN, and the network devices that each port is associated with. For now, I add the following to the README:

  URI: git://git.openembedded.org/meta-openembedded
  branch: thud
  layers: meta-oe, meta-python, meta-networking, meta-filesystems

  URI: git://git.yoctoproject.org/meta-virtualization
  branch: thud

  URI: https://github.com/mendersoftware/meta-mender
  branch: thud

Now that I’ve documented these requirements, I’ll also enforce them in code. I edit conf/layer.conf and document these requirements too. The end of the file should look like:

LAYERDEPENDS_meta-local-soho = "core"
LAYERDEPENDS_meta-local-soho += "openembedded-layer meta-python"
LAYERDEPENDS_meta-local-soho += "networking-layer filesystems-layer"
LAYERDEPENDS_meta-local-soho += "virtualization-layer mender"
LAYERSERIES_COMPAT_meta-local-soho = "thud"

Since there are so many layers in use, I will make use of the TEMPLATECONF functionality so that the build directory will be populated correctly to start with. Next, I copy over meta/conf/bblayers.conf.sample, meta/conf/local.conf.sample and meta/conf/conf-notes.txt from the core layer over to the conf directory and commit them without change. Why? This will isolate my local edits later on and make my life easier next year when I decide it’s time to update to a current release. After I’ve committed those files, I the edit conf/bblayers.conf.sample and change it to look like:

BBLAYERS ?= " \
  ##OEROOT##/meta \
  ##OEROOT##/../meta-openembedded/meta-oe \
  ##OEROOT##/../meta-openembedded/meta-python \
  ##OEROOT##/../meta-openembedded/meta-networking \
  ##OEROOT##/../meta-openembedded/meta-filesystems \
  ##OEROOT##/../meta-virtualization \
  ##OEROOT##/../meta-mender/meta-mender-core \
  ##OEROOT##/../meta-local-soho \

Note that this assumes a directory structure where I’ve put all of the layers I will use in the same base directory. If I didn’t do that then I would need to adjust all of the paths to match how to get from the core layer to where the layers are stored. Now I want to use git add and commit all of these changes, so I can move on to the next step.

I will enable systemd next, and while there’s a number of places I could do this, including going so far as to create my own “distro” policy file, for this article I’ll just use conf/local.conf.sample to store these changes. While the core layer’s conf/local.conf.sample.extended has an example on switching to systemd, I’ll do it slightly differently. At the end of the conf/local.conf.sample insert the following, git add and then commit:

# Switch to systemd
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = ""
VIRTUAL-RUNTIME_syslog = ""
VIRTUAL-RUNTIME_login_manager = "shadow-base"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"

This differs from the core example in a few ways. First, I blank out pulling in an initscripts compat package. While this can be useful, for these images it’s going to end up being redundant. Next, I blank out a syslog provider as I’ll be letting systemd handle all of the logging. Finally, while busybox can be available and provide the login manager, I’ll be using shadow-base instead. This particular change is something now done upstream and will be in later releases, so keep that in mind for the future.

The next functional chunk for conf/local.conf.sample is enabling support for virtualization through meta-virtualization. That layer has its own well documented README file, and taking the time to go over it is a great idea. In my configuration, I’m just going to enable a few things that give me the minimal functionality. So add, git add and git commit:

# Add virtualization support
DISTRO_FEATURES_append = " virtualization aufs kvm"

When talking about system security there are many aspects. One aspect is to harden the system as much as possible by having the compiler apply various build-time safety measures that turn certain classes of attack from “exploit the system” to “Denial of Service by crashing the application”, or even “this is wildly unsafe code, fail to build”. To enable those checks in the build, add, git add and git commit:

# Security flags
require conf/distro/include/security_flags.inc

There’s one last bit of functionality I want in the conf/local.conf.sample file, and that’s support for Mender. How do I go about that? That’s going to depend a bit on what hardware you’re going to do this project on, as it’s a little bit different for something like the APU2 than on an ARM platform. Fortunately, there’s great documentation on how to integrate Mender here. While reading over all of the information there is important, it’s best to focus on the Configuring the build section to understand all of the required variables. In fact, now is a good time to talk more about how I’m going to use Mender in this particular setup. Looking at the overall Mender documentation, it’s very flexible. For this very small deployment scenario, I’ll use standalone mode rather than managed mode. This lets me skip all of the things about setting up a host of other services to make upgrades happen automatically. So, we’ll follow the instructions for configuring Mender in standalone mode. The next thing to touch on is how to handle persistent data. One way to do this would be to make sure that anything which needs to be manually customized or that will be persistently changed at runtime is written somewhere under /data on the device rather than at its normal location. This is because the normal location is going to change when I apply a new update but /data will always be the same. For a lot of deployments, this idea works best, as there are likely many users, and beyond our initial configuration the end user will make changes I know nothing about. For my use case, I am the user, and making the system as stateless as possible will in turn make backing the system up as easy as possible. So instead I will be modifying recipes to include our local configuration when needed. Having an easy to access and modify backup of the various configuration files and so forth will make my life easier in the long run if, for example, something happens to the hardware and it needs to be replaced.

Now that I have everything configured, it’s time to create the build directory. This will let me take advantage of all of the configuration work I just did. At this point, I can go back to the shell, leave meta-local-soho and do this:

$ TEMPLATECONF=../meta-local-soho/conf . oe-core/oe-init-build-env build-router

Again, be sure to change oe-core to wherever the core layer was checked out. Running cat conf/bblayers.conf will show how the changes that were made were now expanded.

What to do next? Well, that depends a little on what I already know about the system. The next step in this project is to configure systemd to take care of creating the networks. If I knew what all of the devices will be named, I could move on to that step. But if I didn’t know, or wasn’t sure, the next step is to build core-image-minimal. That’s as easy as doing:

$ bitbake core-image-minimal

and waiting for the final result, assuming that local.conf.sample file was configured to default to the appropriate target MACHINE. Otherwise I’ll need to pass that in on the command line above. Once that completes, take the appropriate image and boot it. The reward should be a root login prompt.  In this configuration, there’s no root password right now. Login and do:

# ip link

and this will show me what the interface names are. Assuming there is a DHCP server somewhere already, I can then use udhcpc -i NAME after plugging in an Ethernet cable to see which interface is which and note it down for the next step.

Now that I have my network interface names, I can configure systemd to handle them. In my specific case I have 4 Ethernet ports, and coincidentally the one closest to the physical console port (enp1s0) is the one I wanted to call the WAN port. This lets me put the other 3 ports into a bridge for my LAN. Now, to configure these interfaces, I’m going to dump some files under /etc/systemd/network/, and I will use the base-files recipe to own these files, rather than systemd itself. Why? Whenever a change forces a rebuild of systemd that forces a rebuild of a lot of other packages, so this allows me to isolate that kind of build churn. Now I go to meta-local-soho and enter the following:

$ mkdir -p recipes-core/base-files/base-files
$ cd recipes-core/base-files/base-files

I’m going to create four different files. Note that all of the filenames are arbitrary and are meant to help poor humans figure out what file does what. If another naming scheme is helpful, it can be used just as easily. First I’ll take care of the WAN port by creating wan-ethernet.network with the following content:

# Take the eth port closest to the console port for WAN
[Match]
Name=enp1s0

[Network]
DHCP=yes

Next, I’ll create our bridge for the other ports and this is done with two files. First I need a lan-bridge.network with:

# Bridge the other 3 remaining ports into one.
[Match]
Name=enp2s0 enp3s0 enp4s0

[Network]
Bridge=br0

Second I create br0.netdev with:

[NetDev]
Name=br0
Kind=bridge

And now I have a bridge device that I can use to manage all of our LAN ethernet ports. Later I’ll even put the AP on this bridge for simplicity. Now that I have a bridge, I need to configure it, so create bridge-ethernet.network and add:

[Match]
Name=br0

[Network]
Address=192.168.0.1
IPForward=yes
IPMasquerade=yes
IPv6AcceptRA=false
IPv6PrefixDelegation=dhcpv6

[IPv6PrefixDelegation]
RouterLifetimeSec=1800

There’s a lot in there, but it can be broken down pretty easily. Working my way up from the bottom, the first portion is needed to have the IPv6 address on the WAN port pass along what’s needed to the LAN to have devices configure themselves. Since we still live in a world with IPv4, I need to enable masquerade and forwarding. Finally, I configure a static IP for the interface that matches to br0. I haven’t talked about configuring the LAN for IPv4 yet. This is going to be a bit more complex and while systemd supports a trivial DHCP server, I want to do more complex things like assign specific addresses, so that will come later.

Finally, it’s time to make use of these new files. I’ll head up a level and back to recipes-core/base-files and create base-file_%.bbappend with the following:

FILESEXTRAPATHS_prepend := ":${THISDIR}/${PN}"

SRC_URI += "file://wan-ethernet.network \
            file://lan-bridge.network \
            file://bridge-ethernet.network \
            file://br0.netdev \
" do_install_append() { # Add custom systemd conf network files install -d ${D}${sysconfdir}/systemd/network # Add custom systemd conf network files install -m 0644 ${WORKDIR}/*.network ${D}${sysconfdir}/systemd/network/ install -m 0644 ${WORKDIR}/*.netdev ${D}${sysconfdir}/systemd/network/ }

What this does is to look in that directory where I created those 4 files, add them to the recipe and then finally install them on the target where systemd will want them. At this point, I can git add the whole of recipes-core/base-files, and git commit.

There’s one last thing I want to configure right now. I’m going to have systemd handle things like turning on NAT, and doing some other basic iptables work. As a result, I need to enable that part of systemd. Another thing I’ll want to do here is work around a current systemd issue with respect to bridges. To do so, I need to make the directory recipes-core/systemd/ and create the file systemd_%.bbappend in that directory with the following:

do_install_append() {
    # There are problems with bridges and this service, see
    # https://github.com/systemd/systemd/issues/2154
    rm -f ${D}${sysconfdir}/systemd/system/network-online.target.wants/systemd-networkd-wait-online.service
}

PACKAGECONFIG_append = " iptc"

The first part of these changes is to work around the github issue mentioned in the comment. While it’s quite frustrating that the issue in question has been open since the end of 2015, I can easily work around it by just deleting the service for my use case. The second part is to add iptc to the PACKAGECONFIG options, and in turn that part of systemd will be enabled and it can support iptables so the IPMasquerade flag above will work.

At this point, I can now build core-image-minimal again and have a system that will automatically bring up the network. However, it is not a router yet. In the next part of the series I will create a new image that is intended to be a router and customize that.

[Go to Part Three of the series.]