Posts

Building a DIY SOHO router, 18 months later

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, 18 months later

It’s been around a year since my last post on this project, and 18 months since I posted the article series itself. In the world of home networking equipment that’s practically a lifetime. So it’s worth doing a check-in on this project, I think.

My little router is still going just fine. In terms of things I had talked about earlier in the series, I’m still using the ability to easily roll back a test image to try out backports. Most recently, I tried to add IPv6 Prefix Delegation support to the DHCPv6 client in an older version of systemd. It didn’t work, but with just a reboot I was back to my normal image. As times change, devices at home change, and I’m using the dnsmasq configuration file to document what devices are in the house. Performance? Still doing just great. The next time I do one of these series, my household might have crossed the threshold where newer WiFi standards are in enough devices we own that I want something that handles them, but we aren’t there yet. By then I might also just have a new PCIe card to drop in.

In terms of the article series itself, it’s a testament to the stability of the Yocto Project and everyone who works on it. Since the original posts, Mender has internally upgraded a significant amount, and in turn some of the examples there aren’t quite correct for a modern build. Mender has excellent documentation, however, and it’s been easy to update my build to work all along the way and upgrade from the older version to the new. Everything else? Yes, that’s all still correct. What prompted this particular post is that the Yocto Project has announced another milestone release, and that’s always a good time to make sure my device is up to date. My latest changes were quite literally just renaming the Linux kernel bbappend file and saying the layer is compatible with the new release. That was less work than updating the firewall rules to take in to account both Daylight Savings Time changes and that my kids are a bit older and realistically want to be online a little later.

Time to sign up for Virtual ELC Europe 2020

Join us at the Linux Foundation’s Embedded Linux Conference Europe, the premier vendor-neutral technical conference for embedded Linux developers, presented virtually on October 26–29, 2020.

Konsulko Group’s Leon Anavi will give a talk on Software Update Solutions for Yocto Project and OpenEmbedded, exploring the integration in Yocto and OpenEmbedded of A/B and binary delta updates over the air or through a USB stick.

No matter where you are in the world, here’s a good chance to experience ELC Europe for a fraction of the cost of the in-person event.

Getting Started with RAUC on Raspberry Pi

RAUC is a secure, robust and flexible open source software for A/B updates of Embedded Linux devices. It is appropriate for various use cases and it is compatible with all popular build systems: The Yocto Project/ OpenEmbedded, Buildroot and PTXdist.

Konsulko Group engineers have experience with all popular open source solutions for software over the air updates of embedded Linux devices, including Mender, SWUPdate, HERE OTA Connect based on OSTree and Aktualizr. In this article we will discuss the exact steps to integrate RAUC with the Yocto Project (YP) and OpenEmbedded (OE) for Raspberry Pi – the most popular single board computer among students, hobbyists and makers.

For the practical example in this article we will be using the latest and greatest Raspberry Pi as of the moment: Raspberry Pi 4 Model B. Versions with different RAM sizes are available on the market. Any of these Raspberry Pi 4 Model B versions are OK for this RAUC demonstration.

As long time developers and users of the Yocto Project and OpenEmbedded, both have become favorite tools for creating customized distributions for Konsulko engineers. We frequently use and support them commercially. The Yocto Project is a Linux Foundation collaborative open source project for creating custom Linux distributions for embedded devices. It is based on Poky, the reference distribution of the Yocto Project, using the OpenEmbedded build system. The Yocto Project releases on a 6-month cadence. As of the time of this writing, the latest stable release is Dunfell (3.1).

RAUC is a powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image for Raspberry Pi built with the Yocto Project and OpenEmbedded, it requires:

  • U-Boot as a bootloader
  • Enabled SquashFS in the Linux kernel configurations
  • ext4 root file system
  • Specific partitioning of the microSD card that matches the RAUC slots
  • U-Boot environment configurations and a script to properly switch RAUC slots
  • Certificate and a keyring to RAUC’s system.conf

RAUC is capable of covering various use cases and scenarios, including advanced options for single or redundant data partitions. Upgrades are performed through the so called RAUC bundles. It is possible to install them over the air or using the old-fashioned method with a USB stick. For managing updates to a fleet of Internet of Things, it is possible to integrate RAUC with Eclipse hawkBit project that acts as a deployment server with a nice web user interface.

For the sake of simplicity, this article focuses on the most simple and straight-forward use case with 2 identical RAUC slots: A and B. For each slot we will have a separate partition on the microSD card for Raspberry Pi. We have already covered most of the RAUC requirements in an additional Yocto/OE layer called meta-rauc-raspberrypi. We will use it to put the pieces together. First we will build a minimal bootable image for Raspberry Pi 4 with RAUC. We will flash it to both A and B slots. After that we will build a RAUC bundle that adds the text editor nano. Finally we will install this RAUC bundle on the B slot, reboot and verify that nano is present.

Building a Linux Distribution with RAUC

Follow the steps below to build a minimal image for Raspberry Pi with Yocto, OpenEmbedded and RAUC as well as to perform a software update:

  • Download Poky, the reference distribution of the Yocto Project:

git clone -b dunfell git://git.yoctoproject.org/poky poky-rpi-rauc
cd poky-rpi-rauc

  • Download meta-openembedded layer:

git clone -b dunfell git://git.openembedded.org/meta-openembedded

  • Download Yocto/OE BSP layer meta-raspberrypi:

git clone -b dunfell git://git.yoctoproject.org/meta-raspberrypi

  • Download Yocto/OE layers for RAUC:

git clone -b dunfell https://github.com/rauc/meta-rauc.git

git clone -b dunfell https://github.com/leon-anavi/meta-rauc-community.git

  • Initialize the build environment:

source oe-init-build-env

  • Add layers to conf/bblayers.conf:

bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake-layers add-layer ../meta-openembedded/meta-python/
bitbake-layers add-layer ../meta-openembedded/meta-networking/
bitbake-layers add-layer ../meta-openembedded/meta-multimedia/
bitbake-layers add-layer ../meta-raspberrypi/
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-raspberrypi/

  • Adjust conf/local.conf for Raspberry Pi 4 with systemd and RAUC by adding the following configurations to the end of the file:
MACHINE = "raspberrypi4"

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

IMAGE_INSTALL_append = " rauc"

IMAGE_FSTYPES="tar.bz2 ext4 wic.bz2 wic.bmap"
SDIMG_ROOTFS_TYPE="ext4"
ENABLE_UART = "1"
RPI_USE_U_BOOT = "1"
PREFERRED_PROVIDER_virtual/bootloader = "u-boot"

WKS_FILE = "sdimage-dual-raspberrypi.wks"
  • Build a minimal bootable image:

bitbake core-image-minimal

NOTE: Building an image from scratch requires a lot of operations and takes some time so please patiently wait until bitbake completes all tasks.

  • Flash the image to a microSD card and boot it on Raspberry Pi 4:

sudo umount /dev/sdX*
bzcat tmp/deploy/images/raspberrypi4/core-image-minimal-raspberrypi4.wic.bz2 | sudo dd of=/dev/sdX
sync

  • Attach USB to UART debug cable to Raspberry Pi 4, plug ethernet cable and the microSD card. Turn on Raspberry Pi 4. Verify that the system boots successfully.
  • Now, let’s extend the image with the simple text editor nano by adding the following line to the end of conf/local.conf:

IMAGE_INSTALL_append = " nano"

  • Build a RAUC bundle:

bitbake update-bundle

  • Start a web server:

cd tmp/deploy/images/raspberrypi4/
python3 -m http.server

  • On the Raspberry Pi download the RAUC bundle, install it and reboot the board:

wget http://192.168.1.2:8000/update-bundle-raspberrypi4.raucb -P /tmp
rauc install /tmp/update-bundle-raspberrypi4.raucb
reboot

  • After successful upgrade with RAUC reboot the Raspberry Pi and verify that nano is now present:

which nano

  • Check RAUC status to confirm that now the second partition has been booted:

rauc status

For Internet of Things and other real-world products, the whole build procedure with the Yocto Project and OpenEmbedded can be optimized further to just a few commands for easy implementation of continuous integration (CI).

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We have experience with RAUC and various other open source solutions for software updates. Please contact us if you need your “own” rock-solid Linux distro for your own embedded product.

 

Helping Yocto Project work with Python 3

According to the statistics from StackOverflow Python is the fastest-growing major programming language. First released in 1991, Python is nowadays commonly used for various applications in multiple different industries. Python is a first class citizen of many embedded Linux systems.

The Yocto Project, a collaborative project of the Linux Foundation for creating custom Linux distributions for embedded devices, uses the OpenEmbedded build system and relies on layer meta-python from meta-openembedded to deliver Python 3 packages. Until recently, meta-python was providing both python 2 and python 3 versions of each package. The Python community decided that January 1, 2020, was the day to sunset Python 2. Since then Python 2 has been officially deprecated. This triggered major changes related to the support in Yocto and OpenEmbedded. All recipes for version 2 were moved to layer meta-python2 to provide legacy support after the end of life for this Python release. In meta-openembedded/meta-python, the OpenEmbedded community started efforts to remove all recipes for version 2 as well as to consolidate inc and bb files into a single bb file for version 3.

Konsulko Group engineers are regular contributors to various upstream open source projects, including meta-openembedded and more specifically to meta-python. In the past month, Leon Anavi joined the community efforts for consolidating Python 3 recipes in a single file as well as for upgrading various packages. Nowadays, most of the Python 3 recipes are utilizing the pypi bbclass which takes care for downloading and processing packages from pypi.org. This makes most of the upgrades to new releases of a Python package straight-forward. However, it is important to check the list of build and runtime dependencies as well as to ensure that bitbake still works fine with the upgraded recipe version for both x86-64 and ARM architectures prior to submission. 

Let’s have a closer look at the recipe python3-protobuf. It has been recently upgraded from version 3.11.3 to version 3.12.2. Protocol Buffers, also known as protobuf, are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. In the Yocto and OpenEmbedded ecosystem, recipe python3-protobuf depends on recipe protobuf from layer meta-oe. Both meta-oe and meta-python are part of meta-openembedded. So to avoid version mismatch and to ensure that bitbake will be able to successfully build python3-protobuf version 3.12.2 an upgrade of recipe protofobuf to the same version was mandatory. We contributed both upgrades to the master branch of the git repository meta-openembedded. The maintainers took care of cherry-picking them to the dunfell branch which is compatible with the latest stable release of the Yocto Project as of the moment. As a result, if you checkout the latest stable release of Poky, the reference system of the Yocto Project, and meta-openembedded you will be able to quickly build the latest version of protobuf and python3-protobuf out of the box.

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We continue to regularly make upstream contributions to these open source projects. Please contact us if you need your “own” Linux distro for your own embedded product.

Join virtual ELCNA & Yocto Project Dev Day

There’s still time to register for the Linux Foundation’s Embedded Linux Conference North America Virtual Experience and Yocto Project® Dev Day.

Konsulko Group will be giving four presentations. On Tuesday, June 30, Matt Porter will give an in-depth tutorial: Spelunking for Hardware Data and on Wednesday, July 1, Leon Anavi will look at Automotive Grade Linux on Raspberry Pi: How Does It Work?

At Yocto Dev Day on Thursday, July 2, Paul Barker will explore Highly Scalable Build Automation and Scott Murray will discuss Security Hardening.

Please join us for three days of education and virtual collaboration.

Getting some help with build systems

One of the most rewarding parts of working at Konsulko Group has been our returning customers. Months, sometimes years, after we’ve successfully completed an engagement, we’ll hear from the same client (sometimes at the same company, sometimes at a new one) that they would like Konsulko’s help on their latest project.

Konsulko engineers have decades of experience at all levels of open source embedded software, from Linux kernel and low-level subsystems, through middleware and application development, to QA, maintenance and tools, but it is our expertise with the Yocto Project/OpenEmbedded build system that’s usually a part of everything we do.

From the time it first appeared in 2003, the OpenEmbedded build framework revolutionized embedded development, providing a systematic and reusable way to build custom Linux distributions for unique embedded devices. Almost ten years ago, OE became the build system of Yocto Project, and OE’s “recipe” approach was further structured and enhanced by layers.

Konsulko engineers have been there since the earliest days. We’ve seen a lot, learned a lot, and apply our expertise to helping our customers – old and new – build their “own” Linux distro for their own product.

See you at FOSDEM

Konsulko Group engineers will be attending FOSDEM ’20 on February 1 & 2. Look for us during the conference around the Automotive Grade Linux and OpenEmbedded stands, or at Leon Anavi’s presentation on Saturday, Building Homebridge with the Yocto Project. Hope we see you in Brussels!

Building a DIY SOHO router, 6 months on

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, 6 months on

A little more than six months ago, I posted part 4 of our series on making a SOHO router using the Yocto Project and OpenEmbedded. After 6 months of deployment, this is a good time to follow up on how the router has worked out in residential use.  The zeus code-name Yocto Project release was just announced, and that means that the release we started out with in part 1 is now close to being out of support.  That’s not a problem since we designed in support for moving to new software releases using Mender to deliver software updates.

One of the most important metrics in a project like this is, how does it perform?  From the standpoint of a family of 4 heavy internet users, it’s gone really well.  The WiFi range is only a little better than before, but that’s not really a function of the software.  Making everyone use Pi-hole has only resulted in a small number of places where I needed to override the blacklist and allow something in.  From an end-user point of view, this has worked as well as any off-the-shelf router.  From the administrator point of view, I’ve done scheduled maintenance during the day on a weekend, and it really did take only the 5 minutes I promised everyone rather than turning into one of those worst case scenarios where something broke and it takes an hour to fix it.  In fact, the update portion of the plan has gone exceedingly well.  While I didn’t make a post about moving to warrior from thud, I did that transition a while ago and it went smoothly.  Mender introduced a change that required attention be paid while migrating, but it was documented and went smoothly.  On the metadata side, the upgrade was as easy as one could hope.  A few of the bbappend files needed to be updated for a new version, and some of the changes I had made and pushed upstream as part of the original series were now just included, so they got dropped from my layer.

One of the things I touched on in the series was about using the update functionality to test development changes in a production environment.  The chance to do that came up with a systemd-networkd issue that was a problem in my local setup.  The upstream project requested people verify the problem exists with newer versions of systemd and a new enough version was available in what would become zeus.  So I made a quick weekend project of doing an update of my layers to build with a newer version of all of the metadata, removed the work-around, and flashed the image in place.  A quick reboot confirmed that the issue was indeed fixed, and then rather than commit to running an otherwise in-progress release I simply rebooted and automatically rolled back to my stable release.  With the network back up again, I updated the issue in upstream Bugzilla to let them know the problem was fixed.  After a bit longer, a few other people also confirmed it worked for them and now the issue is resolved.

In terms of the metadata itself, there have been a few clean-ups to what I did in my own layer with each release update I’ve done.  In the series I left out what hardware I was building on, and I also left out talking about using the linux–yocto recipe.  Since I first wrote the series linux-yocto has become easier to use, and I found this as part of reviewing my own changes like they were brand new with each upgrade.  I was setting some variables that initially didn’t have reasonable default values, and now they do and I don’t need to set them myself.  This in fact means that moving forward, rather than a version-specific kernel bbappend file, I can go with an always-used one to enable the additional kernel CONFIG options that I need for a time-based firewall.

I started out by mentioning that zeus has been released, and I’m working on migrating to it as I write this.  In fact, it’s so new that I’m doing my own little port of the meta-mender core layer to zeus for my needs. I expect that by the time I do my first update from one build of zeus to the next there will be an official update I’ll be able to use instead.  Looking forward, this was a great little project that also was a lot of fun.  The goals I set way back at the start have been met, and I’m happier with my network than I have been in a long time.  Of course, the list of features an off-the-shelf system provides is always growing, and there’s now monitoring and display items on my weekend project list now to keep up.  I foresee using and improving this setup for a long time to come.

Custom Linux Distro for NVIDIA CUDA Devices

How to get started and build a minimal custom Linux distribution for embedded NVIDIA CUDA-enabled devices using the Yocto Project (YP) and OpenEmbedded (OE).

Building a DIY SOHO router, Part 4

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 4

In part three of this series I finished putting together what I wanted to have on my SOHO router and declared it to be done. While I plan to revisit the topic of a SOHO router using the Yocto Project and OpenEmbedded, this is the final part of the series. In this part, I want to focus on some of the things that I learned while doing the project.

The first thing is that I learned a lot about IPv6, specifically how it’s usually implemented within the United States for residential customers, and some of the implications of this implementation. The first thing to note is that I’ve been off-and-on trying to enable IPv6 for general IPv6 connectivity at home for some time now. Long before my ISP offered IPv6 service, I used Hurricane Electric to have a IPv6 tunnel and connectivity. This was great and only sometimes lead to problems, such as when Netflix finally supported IPv6 and began blocking well known tunnels for region-blocking reasons. It wasn’t until I started on this project that I decided to try to make real use of  routable addresses for hosting personal services. My expectations, and that of lots of software designed to manage IPv6 as well, are best described in article from RIPE about understanding IP addressing. In short, my house or “subscriber site” should get at least 256 LAN segments to do with as I want. Docker expects to have it’s own LAN segment to manage as part of configuring network bridges. When you have 256 or more LAN segments to use, that’s not a problem at all.

Unfortunately, my ISP provides only a single LAN segment. This is simultaneously more IPv6 addresses than the whole of IPv4 and something that should not be further subdivided in routing terms. I could subdivide my LAN segment, but this would in turn cause me to have to do a whole lot more work and headaches. That’s because at the routing level IPv6 is designed for my segment to be the smallest unit. Rather than deal with those headaches I switched my plans up from using Docker to using LXC. With LXC it’s easy to dump the container onto my LAN and then it picks up an IPv6 address the same way all of the other machines on my LAN do. This is good enough for my current needs, but it will make other things a lot harder down the line, if I want separation at the routing level for some devices.

But why am I doing that at all? Well, one of the benefits of having a small but still capable router is that I can run my own small services. While I don’t want to get into running my own email I think it makes a whole lot of sense to host my own chat server for example. With closed registration and no (or limited later on perhaps) federation with other servers I don’t need to worry about unauthorized users contacting my family nor do I have to worry about the company deciding it’s time to shutdown the service I use.

Another lesson learned is that while the Yocto Project has great QA, there’s always room to improve things. As part of picking a firewall I found that one of the netfilter logging options had been disabled by accident a while back. As a part of writing this series of articles and testing builds for qemux86-64, I found that one of the sound modules had been disabled. As a result, the instructions I wrote back in part 2 wouldn’t work. Working upstream is always fun and these changes have been merged and will be included in the next release.

I also worked on a few things for this project that I didn’t include directly in the relevant part of the series. For example, while I did include a number of full utilities in the list of packages installed in the router, I didn’t talk about replacing busybox entirely. This is something that OpenEmbedded supports using the PREFERRED_PROVIDERS and VIRTUAL-RUNTIME override mechanisms in the metadata. Prior to this article however, there wasn’t a good example on how to do this in upstream. Furthermore, there wasn’t an easy way to replace all of busybox and instead you had to list a single package and then include the rest of the required packages in your IMAGE_INSTALL or similar mechanism.  I am a fan of using busybox in places where I’m concerned about disk usage. However, on my router I have plenty of disk space so I want to be sure that if I have to go and solve a problem I’m not using my swiss army knife but rather have my full toolbox available. As a result, OpenEmbedded Core Master now has packagegroup-core-base-utils and a documented example of how to use that in local.conf.sample.extended. This means that when I refresh this image to be based on the Warrior branch I can remove a number of things from my IMAGE_INSTALL.

Another lesson is that old habits die hard.  In general, I always try to use the workflow where I make a change outside the device I’m working on, build the change in, and test it, rather than editing things live.  But when it’s “just” a quick one line change I’ll admit I do it live and roll it into my next build sometimes.  And then sometimes I forget to roll all my changes back up.  So while implementing this project I tried even harder than usual to not fall into that “just a quick change” mindset.  For the most part I’ve been successful at sticking to the idea workflow.  I really believe stateless is the right path forward.  And “for the most part” means that, yes, one time I did have to make use of the fact that the old rootfs was still mountable and copied a file over to the new rootfs, and then to the build machine.  I like to think of that as a reminder that A/B updates are more helpful than a “rewrite your disk each time” workflow for those occasional mistakes.

The caveat to the lesson above is because I really did the “git, bitbake, mender” cycle on this project. I didn’t start on it quite as soon as I said in the article, and I spent a lot more time toying with stuff in core-image-minimal instead of following my own advice, too.  I suppose that is the difference between writing a guide on how things should be done compared with how you do things when you just want to test one more thing, then switch over.  I really should have switched earlier however as every time I avoid doing the SD card shuffle it’s a win on a number of levels.

Did I say SD card above?  Yes, I did.  For this project, a 64GB “black box” that’s in the form-factor of a SD card will have as long of a life span as there is in the form-factor of a M.2 SSD or any other common storage format.  While my particular hardware has a SATA port, I don’t want to try to fit the required cabling, let alone the device itself in the case that’s recommended.  I will admit that I’m taking a bit of a risk here, I am putting as much frequent-write files under a ramfs as I can and after all, I did say stateless is a goal.  If everything does really die on me, I can be back up and running fairly quickly.

Last thing I learned is something I knew all along, really. I like the deeper ownership of the router. There’s both the pride and accomplishment in doing it and that “old school” fun of being the admin again, for real.

Portfolio Items