Porting Mender to Raspberry Pi 5 and Yocto Project Scarthgap

Overview

Mender is an open-source over-the-air (OTA) software update technology for embedded Linux devices and Internet of Things (IoT) ecosystem. It allows developers to remotely manage and update software on embedded Linux devices, ensuring that they remain secure, up-to-date, and functional throughout their lifecycle. Mender simplifies the process of launching a new project by offering official and community-supported board integrations for a wide range of devices and operating systems. These integrations include support for Debian family and the Yocto Project, making it easier for developers to get started with their projects seamlessly.

The Yocto Project is an open-source collaboration project by the Linux Foundation to create custom Linux-based systems for embedded devices. It uses the OpenEmbedded build automation framework with the build tool bitbake and provides Poky as a reference Linux distribution. The Yocto Project follows a regular release cycle, typically with a new version every six months, alongside long-term supported releases available every two years. Currently, the most recent LTS release is version 5.0, codenamed Scarthgap, which became available on April 30, 2024.

Recently, Senior Engineer Leon Anavi from Konsulko Group ported Mender to Raspberry Pi 5. This effort builds upon his previous contributions, where he added Raspberry Pi 5 support to the Yocto Project BSP (Board Support Package) layer meta-raspberrypi. This progress was made possible through sponsorship from the computer emergency response team of the Government of the Grand Duchy of Luxembourg (GOVCERT.LU). If you’re considering using Mender on Raspberry Pi or any other hardware platform for your embedded product needs, feel free to reach out to us to discuss further.

Raspberry Pi 5 introduces significant hardware differences compared to its predecessors. To enable Mender updates on this platform, we require U-Boot version v2024.04 or later. Unfortunately, the Yocto Project release Scarphgap ships with U-Boot version v2024.01, which isn’t compatible. To address this compatibility issue and obtain the necessary U-Boot version, our setup utilizes the scarthgap/u-boot branch from the meta-lts-mixins layer. Tim Orling, Principal Software Engineer at Konsulko Group, contributed the patches in meta-lts-mixins for U-Boot v2024.04.

Raspberry Pi 5 is the first model of the famous single board computers that features a dedicated UART connector is a three-pin header compatible with the Raspberry Pi Debug Connector specification. It can be used with Raspberry Pi Debug Probe, a USB device that provides both a UART serial port and a standard Arm Serial Wire Debug (SWD) interface.

The article offers a practical guide, outlining the exact steps to build a Mender-enabled image for Raspberry Pi 5 and execute an A/B update. To streamline the build setup, we’ll utilize KAS. This Python-based open-source tool effectively handles various Yocto/OpenEmbedded layers. KAS executes builds within a Docker container to ensure consistency and reliable build outcomes, regardless of the primary GNU/Linux distribution on the build machine.

Image for Raspberry Pi 5

Follow the steps below to build core-image-minimal with Mender for Raspberry Pi 5:

  • Install the kas tool (optionally, you can install globally for all users. Run as root, respectively under sudo then):
pip install kas
  • Clone this repository:
git clone -b scarthgap https://github.com/mendersoftware/meta-mender-community
  • Create a build directory and change into it:
mkdir -p meta-mender-community/mender-rpi5 && cd meta-mender-community/mender-rpi5
  • Use kas to build for the Raspberry Pi 5:
kas build ../kas/raspberrypi5.yml
  • Flash tmp/deploy/images/raspberrypi5/core-image-minimal-raspberrypi5.sdimg to a microSD card and boot it on Raspberry Pi 5.

Mender Artifact for Raspberry Pi 5

Follow the steps below to build a Mender Artifact for Raspberry Pi 5 that provides the simple text editor nano:

  • Enter KAS shell:
kas shell ../kas/raspberrypi5.yml
  • Append to the end of conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build both core-image-minimal and a Mender Artifact for it:
bitbake core-image-minimal
  • As a result Mender Artifact containg nano will be generated as file tmp/deploy/images/raspberrypi5/core-image-minimal-raspberrypi5.mender

Update Raspberry Pi 5

As an end to end update solution, Mender provides aserver as the central hub for storing and orchestrating software updates across fleets of devices through over-the-air deployment. Through Mender’s user-friendly web UI or REST APIs, you can easily oversee device management, upload software releases, and create seamless deployments to distribute updates to your devices. However, it is also possible to use Mender in standalone mode without a server.

Follow the steps below to manually perform a standalone deployment with Mender in the terminal. In this scenario, no Mender Server is utilized, and the deployments are triggered directly at the device.

  • Start a simple HTTP server in the directory with the Mender Artifact:
python3 -m http.server
  • Login as root on Raspberry Pi 5 and install the Mender Artifact to perform an upgrade of the device:
mender-update install http://<server>:8000/core-image-minimal-raspberrypi5.mender

NOTE: Replace <server> with the IP address of the machine on which the Python3 HTTP server is running.

  • Reboot Raspberry Pi 5:
reboot
  • Login as root on Raspberry Pi 5 and verify that nano text editor has been installed.
  • Make the deployment permanent:
mender-update commit

This straightforward example showcases the seamless integration of Mender with Raspberry Pi 5 using the Yocto Project release Scarthgap. You can follow up and manage updates of fleets of Raspberry Pi 5 devices through the Mender server. Additionally, Mender offers convenient add-ons for remote troubleshooting, ensuring smooth operations in the field.


Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been active contributors to the community, aiding customers in crafting commercial products using these technologies. Our expertise extends beyond Mender, encompassing various open-source solutions for software updates. Feel free to get in touch to explore how we can assist with your embedded product requirements.

Integrating RAUC with Yocto Project on BeagleBone Black

Konsulko Group has made many upstream contributions to OTA (over-the-air) update solutions for embedded Linux devices. RAUC is a popular open source option as it has been meticulously developed with a keen emphasis on stability, security, and adaptability. Notably, RAUC seamlessly integrates with major build systems such as Yocto Project/OpenEmbedded, Buildroot, and PTXdist.

Functioning across diverse usage scenarios, one of RAUC’s elementary yet impactful functionalities is the A/B update mechanism. In this setup, two identical root filesystems, denoted as A and B, are maintained. The device boots from one of these, while the other serves as the target for updates.

Post-update completion, the bootloader directs the system to boot from the freshly updated partition during the subsequent system startup. RAUC incorporates the ‘verity’ update bundle format. It extends the capabilities of RAUC by introducing built-in support for HTTP(S) network streaming, adaptive delta-like updates, and comprehensive update bundle encryption.

In previous blog posts, Konsulko Group engineers have demonstrated RAUC on Raspberry Pi and NXP devices such as SolidRun Cubox-i and HummingBoard. Recently Leon Anavi, Konsulko Group Senior Engineer and maintainer of meta-rauc-community ported RAUC to BeagleBone Black.

This article provides, as an example, the exact steps how to integrate RAUC with Yocto Project and OpenEmbedded for booting from a microSD card on BeagleBone Black.

Released in 2013, BeagleBone Black is a single-board computer (SBC) developed by the BeagleBoard.org Foundation. It was certified by the Open Source Hardware Association with OSHWA UID US000236. The chipset on BeagleBone Black is Texas Instruments Sitara AM3358 with 1GHz ARM Cortex-A8 CPU and SGX 3D graphics engine. Because of this the demonstrated integration is a suitable reference for other embedded devices equipped Texas Instruments chipsets.

Required Hardware

The hardware used for this step by step tutorial is:

Building a Linux Distribution with RAUC

RAUC, a robust and powerful open-source solution, demands advanced skills for initial integration. In general, to incorporate RAUC in a Yocto Project and OpenEmbedded image for BeagleBone Black the following actions have to be performed:

  • Use U-Boot as a bootloader
  • Enable SquashFS in the Linux kernel configuration
  • Use ext4 root file system
  • Design specific storage partitioning for the certain use case and configure RAUC accordingly
  • Provide a custom U-Boot script to properly switch between RAUC slots
  • Prepare a certificate and keyring to use for signing and verifying RAUC update bundles.

Leon Anavi has already done all these actions for core-image-minimal in Yocto/OpenEmbedded layer meta-rauc-community/meta-rauc-beaglebone. The layer is available at GitHub. Please follow the steps below to build core-image-minimal for BeagleBone Black with it:

  • Download the long term support (LTS) release Kirkstone reference Yocto distribution, Poky:
git clone -b kirkstone https://git.yoctoproject.org/poky poky-rauc-bbb
cd poky-rauc-bbb
  • Download the meta-rauc layer:
git clone -b kirkstone https://github.com/rauc/meta-rauc.git
  • Download meta-rauc-community layers, including meta-rauc-beaglebone:
git clone -b kirkstone https://github.com/rauc/meta-rauc-community.git
  • Download the meta-openembedded layer as it provides a recipe for nano which will be used for the demonstration:
git clone -b kirkstone git://git.openembedded.org/meta-openembedded

Initialize the build environment:

source oe-init-build-env
  • Include all layers in conf/bblayers.conf:
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-beaglebone
  • Adjust conf/local.conf by appending the following configurations to the end of the file:
MACHINE = "beaglebone-yocto"

# Use systemd as init manager
INIT_MANAGER = "systemd"

# Add RAUC to the image
IMAGE_INSTALL:append = " rauc"
DISTRO_FEATURES:append = " rauc"
  • Build a minimal bootable image:
bitbake core-image-minimal

The image creation process from scratch is time-consuming, encompassing various Yocto/OpenEmbedded recipes and configurations. Kindly await completion as bitbake diligently executes each tasks.

  • Flash tmp/deploy/images/beaglebone-yocto/core-image-minimal-beaglebone-yocto.wic.xz to microSD card.
  • Attach the USB-to-UART adapter to BeagleBone Black, plug the ethernet cable and the microSD card.

Press and hold button S2 while plugging in the 5V DC power supply to turn on BeagleBone Black and boot from microSD card.

BeagleBone black board has an onboard button labeled as S2. It is situated near the microSD card slot. Press and hold it while powering the board to boot from microSD card.

  • Verify that the system boots successfully, log in as user root without a password and check RAUC status:
rauc status

On the screenshot BeagleBone Black has been booted from RAUC slot rootfs.0 (A) on the microSD card.

NOTE: The meta-rauc-beaglebone layer includes a core-image-minimal.bbappend file, housing essential configurations for RAUC functionality. Apply these configurations similarly to other images intended for use in your embedded Linux device.

Creating a RAUC Update Bundle

RAUC update bundle comprises the file system image(s) or archive(s) designated for system installation, accompanied by a manifest detailing the images for installation, encompassing options and meta-information. Additionally, it may include scripts designated for execution before, during or after the installation process. To sign and verify the update bundles RAUC uses SSL keys. Layer meta-rauc-beaglebone contains a keyring containing all keys and a recipe for a simple RAUC update bundle for demonstration purposes only.

Follow the steps below to create RAUC update bundle that extends the system by adding the popular text based editor nano:

  • Add to conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build the RAUC update bundle:
bitbake update-bundle

Following a successful execution, bitbake will produce the update-bundle-beaglebone-yocto.raucb file.

Updating BeagleBone Black with RAUC

Follow the steps below to update core-image-minimal running from a microSD card on BeagleBone Black:

  • Start a Python 3 web server on the build machine or another computer where the RAUC update bundle (aka update-bundle-beaglebone-yocto.raucb) is available and within the same network as BeagleBone Black, for example:
cd tmp/deploy/images/beaglebone-yocto/
pip3 install --user rangehttpserver
python3 -m RangeHTTPServer
  • On BeagleBone Black replace <IP> with the actual IP address of the computer on which the HTTP server is running and execute the following command to install the update:
rauc install http://<IP>:8000/update-bundle-beaglebone-yocto.raucb

The screenshot show successful installation of the RAUC updated bundle on BeagleBone Black.

  • Reboot BeagleBone Black to load the updated version:
reboot

NOTE: As alternative, instead of using an HTTP server, you can transfer the update bundle to BeagleBone Black and install it from local storage.

  • Verify that nano was added to the system:
which nano
  • Check RAUC status to confirm the system have booted from the second partition:
rauc status

On the screenshot, after sucessful installation of the RAUC update bundle, BeagleBone Black has been booted from RAUC slot rootfs.1 (B) on the microSD card. This slot contains nano.

In real-world product development, the Yocto Project and OpenEmbedded workflow can be enhanced with a few commands for easy implementation of continuous integration (CI).

From the dawn of OpenEmbedded and the Yocto Project, Konsulko engineers have been community contributors and guides for crafting commercial products. Our expertise spans RAUC, Mender, and various open-source solutions for top-notch software updates. Please get in touch with us to discuss how Konsulko engineers can help your own embedded product development.

Mender Add-ons: Remote Troubleshooting Devices in the Field

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

Konsulko Group often works with Mender.io to provide secure, risk tolerant and efficient Over-The-Air updates for any software on an embedded device. This includes both solutions for robust system updates as well as updates of single applications. Mender is compatible with any Linux based OS, including popular build systems such as Yocto Project/OpenEmbedded, Buildroot and OpenWrt.

Aside from OTA updates Mender also offers several add-ons. These are optional extensions that provide functionality for use cases beyond the core OTA updates features.

One such use case is remote troubleshooting. In practice deployed devices in the field are often hard to reach or retrieve. Troubleshooting individual devices becomes resource inefficient and in some cases they have to be replaced even when it’s just a small software or configuration issue. Our experience has shown in these situations Mender’s Add-ons have proven to be extremely useful.

This article provides as an example, the exact steps how to integrate Mender with The Yocto Project and OpenEmbedded for SolidRun CuBox-I and HummingBoard as well as a demonstration of the Mender Troubleshooting package, more specifically the File Transfer and Remote Terminal Add-ons. With the File Transfer add-on files can be downloaded and uploaded to any accepted device. Remote Terminal add-on allows remote interactive command execution from the Mender UI.

These add-ons are very valuable for system administration and mantainance of Internet of Things or fleets of connected industrial devices. For an example, we will troubleshoot an embedded Linux device remotely with systemd using Mender add-ons. We will upload tools to gather system boot-up performance statistics, run them on the device thanks to the Remote Terminal and download logs for further debugging.

Any Mender enabled device is suitable for this demonstration. Recently the Konsulko Senior Engineer Leon Anavi ported Mender to SolidRun CuBox-I and HummingBoard with NXP i.MX6 SoC so our current setup is based on this hardware platform and Yocto LTS release Kirkstone. Earlier we also ported RAUC, an alternative free and open source OTA update platform to the same hardware. If you are interested in software updates, please contact us to discuss your own embedded product needs in details. The hardware used in this example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • Generic RJ45 network cable with Internet connection
  • Optionally UART to USB adapter for debugging the setup section

Building a Linux Distribution with Yocto/OpenEmbedded

The Mender Community provides a set of examples for integration with various hardware platforms using Yocto/OpenEmbedded in the meta-mender-community repository. Sub-layer meta-mender-nxp in this repository contains the integration for Cubox-I/HummingBoard.

The meta-mender-nxp layer uses Google Repo to provide easy and simple setup and build process for the examples:

$ mkdir mender-nxp && cd mender-nxp
$ repo init -u https://github.com/mendersoftware/meta-mender-community \
            -m meta-mender-nxp/scripts/manifest-nxp.xml \
            -b kirkstone
$ repo sync
$ source setup-environment nxp

Board configuration

To configure the build system we have to append to conf/local.conf inside the build directory.

First we are going to set the build target machine:

MACHINE = "cubox-i"

Then we have to accept the end user agreement required by the BSP layer:

ACCEPT_FSL_EULA = "1"

Note: Usually to enable Mender’s Troubleshooting features we have to add mender-connect to IMAGE_INSTALL. In our case this is already added by the meta-mender-demo layer.

Mender configuration

We have to provide our device with MENDER_SERVER_URL and MENDER_TENANT_TOKEN.
For that reason we have to register at https://mender.io/.

Mender provides a free demo profile with limitation of 1 year and up to 10 devices which can be used to experiment with all of Mender’s features.

In fact, when sourcing the setup-environment script we get most of the mender-specific configuration appended to local.conf. This includes a description of how to get our tenant token:

# Build for Hosted Mender
#
# To get your tenant token:
#    - log in to https://hosted.mender.io
#    - click your email at the top right and then "My organization"
#    - press the "COPY TO CLIPBOARD"
#    - assign content of clipboard to MENDER_TENANT_TOKEN
#
#MENDER_SERVER_URL = "https://hosted.mender.io"
#MENDER_TENANT_TOKEN = ""

Note: If using the European server one has to set MENDER_SERVER_URL = "https://eu.hosted.mender.io"

Once we assign our tenant token and remove the # in front of MENDER_SERVER_URL and MENDER_TENANT_TOKEN we’re ready to build our system.

Building and flashing the system image to a microSD card

Build an example image with Yocto:

$ bitbake core-image-base

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

Once the build is complete flash the image to the microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-base-cubox-i.sdimg.bz2 /dev/sdX
$ sync
$ eject /dev/sdX

Connecting to Mender

Once the board finishes booting it will poll the Mender server. By design the connection has to be established from the board to the server. Mender does not open any ports on the board to provide better security therefore the device has to initiate the connection.

When the connection is established the Mender control panel will indicate one pending device.

To accept the request click on View details under Pending devices.

Select the new device and press accept in the Authorization request section.

Once the connection is accepted head over to the Troubleshooting tab in the Device information section. Here you can launch a remote terminal and transfer files.

The next part of the article will demonstrate preparing, uploading and using systemd-analyze to fetch data about the boot process.

Preparing troubleshooting software

To compile systemd-analyze we have to add it to our image in conf/local.conf:

IMAGE_INSTALL:append = " systemd-analyze"

Rebuild systemd to get the systemd-analyze binary:

$ bitbake systemd -c compile
When using the do_compile command Yocto/OpenEmbedded will preserve the compiled binaries.

Find the systemd-analyze binary and libsystemd-shared-<version>.so shared library:

$ find tmp/work -name "systemd-analyze"
$ find tmp/work -name "libsystemd-shared*.so"

Note: As of the time of writing of this article these files should reside in locations similar to ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/systemd-analyze and ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/src/shared/libsystemd-shared-250.so. These paths depend on the exact version of systemd as well as the build configuration and may not be correct in your case.

Uploading the troubleshooting software

Upload these files to the board using Mender’s File Transfer utility:

  • systemd-analyze into /usr/bin/
  • libsystemd-shared-<version>.so into /usr/lib/

Fetching service initialization logs

Once the troubleshooting software is uploaded we can use the Remote Terminal to execute it.

Permit execution of the systemd-analyze binary:

# chmod +x /usr/bin/systemd-analyze

Check the time it took for the system to initialize:

# systemd-analyze

Export a graphic of all enabled services and the time they took to initialize:

# systemd-analyze plot > init.svg

Download init.svg:

This graphic shows that the device needs around a minute to reach multi-user.target. The longest task is the resizing of the /data partition that runs on first boot and the second longest is the filesystem check for mmcblk1p1 that runs every time the system boots.

Here is another graphic generated after a reboot:

This article demonstrates how to use Mender’s Remote terminal and File Transfer troubleshooting utilities to upload and execute the systemd-analyze binary to profile the initialization process of systemd services. These troubleshooting utilities can be used for variety of different tasks. After debugging a single device and finding an appropriate fix, Mender is capable of performing an A/B or delta software update to all devices or specific group of devices in the field.

Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product needs.

Setting up RAUC on CuBox-I/HummingBoard for Software Updates

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

RAUC is one of the popular solutions that provide OTA (over-the-air) updates for Embedded Linux devices. RAUC is developed with focus on stability, security and flexibility and is compatible with all popular build systems: The Yocto Project/OpenEmbedded, Buildroot and PTXdist.

RAUC is capable of covering various use cases the most simple one being A/B updates.

The A/B updates scenario consists of having 2 identical root filesystems (named A and B), booting from one of them and performing the update on the other. After the update is complete the bootloader will boot from the updated partition on the next system boot. Recently the ‘verity’ update bundle format was introduced in RAUC. This new groundbreaking feature improves the verification process and most importantly allows extending RAUC by built-in HTTP(S) network streaming support, adaptive delta-like updates, and full update bundle encryption.

This article provides an example for setting up RAUC for A/B updates scenario on a HummingBoard board. The hardware used for the example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • UART to USB adapter

RAUC is a robust, powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image built with the Yocto Project and OpenEmbedded for CuBox-I/HummingBoard one needs to:

  • Use U-Boot as a bootloader
  • Enable SquashFS in the Linux kernel configuration
  • Use ext4 root file system
  • Design specific storage partitioning for the certain use case and configure RAUC accordingly
  • Provide a custom U-Boot script to properly switch between RAUC slots
  • Prepare a certificate and keyring to use for signing and verifying RAUC update bundles.

Building a Linux Distribution with RAUC

I’ve recently contributed to meta-rauc-community, a repository containing minimal RAUC example layers for Yocto/OpenEmbedded.

The following steps will show how to use the meta-rauc-nxp layer from that repository to build and update a minimal Linux distribution. The update will install nano (the text editor) to the system.

Download the reference Yocto distribution, Poky.
We’ll use the latest long term support version, kirkstone.

$ git clone -b kirkstone https://git.yoctoproject.org/poky
$ cd poky

Download meta-rauc-community layers (meta-rauc-nxp):

$ git clone https://github.com/rauc/meta-rauc-community.git

Download the meta-rauc layer:

$ git clone -b kirkstone https://github.com/rauc/meta-rauc.git

Download the BSP layers for cubox-i/HumminbBoard boards:

$ git clone -b kirkstone https://git.yoctoproject.org/meta-freescale
$ git clone -b kirkstone https://github.com/Freescale/meta-fsl-arm-extra.git

Download the meta-openembedded layer (provides nano):

$ git clone -b kirkstone git://git.openembedded.org/meta-openembedded

Initialize the build environment:

$ source oe-init-build-env

Add the layers to conf/bblayers.conf:

$ bitbake-layers add-layer ../meta-openembedded/meta-oe
$ bitbake-layers add-layer ../meta-rauc
$ bitbake-layers add-layer ../meta-freescale
$ bitbake-layers add-layer ../meta-fsl-arm-extra
$ bitbake-layers add-layer ../meta-rauc-community/meta-rauc-nxp

Adjust conf/local.conf by adding the following configurations to the end of the file:

# HummingBoard specifications are very similar to Cubox-I
MACHINE = "cubox-i"

# Accept end user agreement required by the BSP layer.
ACCEPT_FSL_EULA = "1"

# Use systemd as init manager
INIT_MANAGER = "systemd"

# Add RAUC to the image
IMAGE_INSTALL:append = " rauc"
DISTRO_FEATURES:append = " rauc"

# Generate ext4 image of the filesystem
IMAGE_FSTYPES:append = " ext4"

# Use the file containing the partition table specification
WKS_FILE = "sdimage-dual-cubox-i.wks.in"
WKS_FILES:prepend = "sdimage-dual-cubox-i.wks.in "

# Add 150 000 KBytes free space to the root filesystem
# (Adding software with updates require space.)
IMAGE_ROOTFS_EXTRA_SPACE:append = " + 150000"

# Add the boot script to the boot partition
IMAGE_BOOT_FILES:append = " boot.scr"

Note that whitespace inside quotes is intentional and important.

To sign and verify the update bundles RAUC uses SSL keys. A keyring containing all keys that will be used for update bundles needs to be installed on the target.

meta-rauc-community provides a script that would generate example keys and configure the current build environment accordingly. (The script has to be called after sourcing oe-init-build-env)

$ ../meta-rauc-community/create-example-keys.sh

Build a minimal bootable image:

$ bitbake core-image-minimal

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

It’s strongly recommended to zero-fill the u-boot environment sectors before flashing the image on the microSD card (replace /dev/sdX with the proper device path):

$ dd if=/dev/zero of=/dev/sdX seek=2032 count=16

After the build is done, flash the image to a microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-minimal-cubox-i.wic.gz /dev/sdX
$ sync
$ eject /dev/sdX

Attach the USB-to-UART adapter to the HummingBoard Pro, plug the ethernet cable and the microSD card. Turn on the board to verify that the system boots successfully.

By default one can login as root without password.

Creating an update bundle for RAUC

After sourcing the oe-init-build-env, append the following line to the build configuration conf/local.conf to add nano to the system:

# Adding nano
IMAGE_INSTALL:append = " nano"

Build the RAUC update bundle:

$ bitbake update-bundle

Start a web server:

$ cd tmp/deploy/images/cubox-i/
$ pip3 install --user rangehttpserver
$ python3 -m RangeHTTPServer

Now you can install the bundle on the board, then reboot:

# rauc install http://192.168.1.2:8000/update-bundle-cubox-i.raucb
# reboot

One of the latest RAUC features is the verity bundle format. This format allows updates to be done without storing the whole bundle on the device in advance, which is useful for devices with limited space. One of the requirements for this feature is hosting the bundle on a server that supports HTTP Range Requests.

As alternative, you can transfer the bundle to the device and install it from local storage.

Verify that nano was added to the system:

# which nano

Check RAUC status to confirm the system have booted from the second partition:

# rauc status

For real-world products, this build procedure with the Yocto Project and OpenEmbedded can be optimized further with just a few commands for easy implementation of continuous integration (CI).

Since the earliest days of the OpenEmbedded build framework and Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product development.

Using kernel config fragments to remove an unwanted feature

Adding a feature to a linux-yocto based kernel is fairly well documented. This makes sense because it is the most common thing you might want to do: “My board needs support for this sensor added to our BSP.”

Konsulko Group recently helped a customer that had exactly the opposite problem, a standard feature in linux-intel (which includes linux-yocto.inc and inherits the kernel-yocto class) needed to be removed. The SoC (in the Intel™ “Bay Trail” family) and the off-the-shelf industrial PC had a problem. When USB 3.0 (xHCI) support is enabled, the default BIOS settings (xHCI Mode = Auto) would cause the system to lock-up upon either warm reboot or shutdown. Given that these systems are deeply embedded in the field—where simply “hooking up a display, keyboard and mouse” is cost prohibitive—we needed to find an option that would not prevent OTA updates.

You might think you could apply a patch via SRC_URI to kmeta (yocto-kernel-cache), but this isn’t supported in the Yocto Project kernel tooling. Instead we can apply kernel config fragments that disable the problematic xHCI feature. The trade off is that USB 3.x devices won’t be able to run at full speed, but the systems in question have no need for USB 3.0 (they are a classic IoT gateway use case).

Determining your existing kernel configuration

The first thing you want to do in this situation is determine what your (default) kernel configuration is. In our case, Konsulko and our customer are using the ‘dunfell’ (3.1.x) branch of meta-intel with MACHINE="intel-corei7-64". Upon building an image or the kernel (e.g. bitbake virtual/kernel or bitbake linux-intel), the kernel configuration can be found at the following path:

<build>tmp/work/corei7-64-intel-common-poky-linux/linux-intel/5.4.170+gitAUTOINC+98cce1c95f_36f93ff941-r0/linux-corei7-64-intel-common-standard-build/.config

Where the kernel version is 5.4.170, the (shortened) git commit hash of the kernel cache is 98cce1c95f and the (shortened) git commit hash of the kernel source is 36f93ff941.

Modifying the kernel configuration with menuconfig

The documented way to modify the kernel configuration is with:

bitbake -c menuconfig linux-intel

This approach works fine, but you must remember to copy the resulting .config to defconfig in your recipe’s SRC_URI.

Alternatively you can create your kernel config fragments in the kernel build directory and then add them to your kernel recipe. The fragments can be created with the help of the diffconfig script in the kernel source tree.

Also note that you must have already run:

$ bitbake -c kernel_configme -f linux-intel

or previously built the kernel in order for the .config to be present.

Modifying the kernel configuration with devtool

It will come as no surprise that my preferred way to modify the kernel configuration is to run:

$ devtool menuconfig linux-intel

But you will get an error:

ERROR: No recipe named 'linux-intel' in your workspace

So first we must get the kernel recipe into our workspace:

$ devtool modify linux-intel

Now we are able to run:

$ devtool menuconfig linux-intel

One benefit of this approach is that devtool will run the required steps that need to happen before menuconfig can be run (most notably the do_kernel_configme task).

Determining the changes needed

Regardless of which method you used to run menuconfig, you will now be presented with the text UI:

Since Konsulko and our customer already knew we needed to change the xHCI enablement, we can quickly </> for Search and then enter xHCI.

This gives us several results, but the ones that we care about have [=y](built-in) next to them:

  1. CONFIG_USB_XHCI_HCD
  2. CONFIG_USB_XHCI_PCI
  3. CONFIG_USB_XHCI_PLATFORM

The top level item that needs to be disabled (set to N) is CONFIG_USB_XHCI_HCD:

After this we can spot check the other values are also disabled (by using </> for Search again):
CONFIG_USB_XHCI_PCI and CONFIG_USB_XHCI_PLATFORM:

Satisfied that we have the needed change, we can save our configuration. Press the <E> key or click on < Exit > to exit the sub-menus until you are at the top of the stack. Press the <S> key or click on < Save > to save the configuration. At they prompt, press enter or click on <Yes>.

Now, the benefit of the devtool workflow comes into play, because we are rewarded with:

INFO: Updating config fragment <build>/workspace/sources/linux-intel/oe-local-files/devtool-fragment.cfg

The contents of this file are what you might expect:

# CONFIG_USB_XHCI_HCD is not set

Applying our configuration change

If we don’t already have one, we need a layer into which to put our changes:

$ bitbake-layers create-layer ~/Projects/meta-awesome-bsp

Add the layer to our active layers:

$ bitbake-layers add-layer ~/Projects/meta-awesome-bsp

Create a directory–following the pattern in openembedded-core–for our kernel changes:

$ mkdir -p ~/Projects/meta-awesome-bsp/recipes-kernel/linux

Finish our recipe:

$ devtool finish linux-intel ~/Projects/meta-awesome-bsp

Examine the resulting directory structure:

$ tree ~/Projects/meta-awesome-bsp
/home/<user>/Projects/meta-awesome-bsp
├── conf
│&nbsp;&nbsp; └── layer.conf
├── COPYING.MIT
├── README
├── recipes-example
│&nbsp;&nbsp; └── example
│&nbsp;&nbsp;     └── example_0.1.bb
└── recipes-kernel
    └── linux
        ├── linux-intel
        │&nbsp;&nbsp; └── devtool-fragment.cfg
        └── linux-intel_%.bbappend

Since this might not be the only change to the kernel we will need to make, let us give the fragment a better name:

pushd ~/Projects/meta-awesome/recipes-kernel/linux/linux-intel
mv devtool-fragment.cfg disable-xhci-hcd.cfg

And create an .scc file to give the Yocto Project kernel tooling better hints of how to apply our change:

cat << EOF >> disable-xhci-hcd.scc
# SPDX-License-Identifier: MIT
define KFEATURE_DESCRIPTION "Disable options for xhci (USB 3.0)"
define KFEATURE_COMPATIBILITY board

kconf hardware disable-xhci-hcd.cfg
EOF

And finally make changes to our linux-intel_%.bbappend to reflect these files:

$ cd ..
$ cat linux-intel_%.bbappend
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

SRC_URI += "\
    file://disable-xhci-hcd.cfg \
    file://disable-xhci-hcd.scc \
"

Now—as long as our layer has high enough priority and no other recipes add kernel config fragments which conflict with our change—we should be able to build the kernel and inspect the resulting .config:

$ popd
$ bitbake linux-intel
$ grep -R XHCI tmp/work/corei7-64-intel-common-poky-linux/linux-intel/5.4.170+gitAUTOINC+98cce1c95f_36f93ff941-r0/linux-corei7-64-intel-common-standard-build/.config
# CONFIG_USB_XHCI_HCD is not set
# CONFIG_USB_ROLES_INTEL_XHCI is not set

We should also be able to run dmesg | grep xhci on the target and we would not expect to see any messages.

Summary

A Konsulko Group customer had a hardware problem that required us to remove a kernel feature. By using tools like devtool, we were able to fairly easily make a change to the kernel configuration and capture those changes in a persistent way with a .cfg fragment and a .bbappend. This approach solved the problem and allowed OTA updates to proceed to these deeply embedded devices in the field. Please contact us to discuss how Konsulko can help you with the unique requirements of your commercial project.

Building Platforms with Secure Over-the-Air Updating

Almost every device in development today requires both over-the-air (OTA) updating capabilities, and up-to-date security with authentication of the device filesystem and encryption of application data storage.

At Konsulko Group, we find that the best approach is to build a secure platform first, starting with the hardware root of trust and establishing a chain of trust by extending the root of trust through each successive component in the system.

  • A first step is to make sure the Universal Boot Loader (U-Boot) is up-to-date. If the device is using an older version, we may port U-Boot support of a more current release. Then we enable signature-based authentication of the device using the U-Boot verified boot feature.
  • To provide authentication of the device filesystem and encryption of application data storage, we often use the Linux kernel Device-Mapper infrastructure to create virtual layers of block devices: dm-verity for root filesystem authentication, dm-crypt for data encryption, and dm-integrity for read/write data volume integrity.
  • Finally, we can integrate Mender I/O support for OTA together with dm-verity and Yocto Project so that device updates can be performed while maintaining the system’s secure chain of trust.

This step-by-step methodology ensures straight-forward and predictable development. Please contact us to discuss how we can help you build a secure platform for OTA updating on your device.

How Mender works

by Tom Rini, VP Engineering

Software Update solutions are a key part of our services offering. For open source over-the-air updates, we often we recommend and work with mender.io. In fact, Konsulko Group is a Mender Authorized Referral Partner. Recently, a prospective customer expressed an interest in knowing more about how Mender works. Here’s the brief, informal introduction to member.io that I prepared, and now I’m sharing with you.

As a high level starting point, https://mender.io/how-it-works provides a good overview of what’s supported and what it covers. In short, Mender starts off by providing support for a traditional “A/B” approach to system updates, where if the update isn’t marked as valid (and there’s hooks for the application(s) to verify the system before this is done), it’s assumed invalid and the system will roll-back automatically. While “OTA” implies over the network, it can just as easily be done by providing (and validating) a USB key that contains an update.

One of the reasons we recommend Mender is that it has very good in-depth documentation. The starting point for all of that is https://docs.mender.io/2.4/ which covers all of the topic starting from how to implement Mender support in a device and including how to create your own server infrastructure if you don’t want to use their paid service. While there are a number of important pages there, one that I like to highlight is https://docs.mender.io/2.4/artifacts/state-scripts as it shows the state machine for an update and talks about some of the common use cases that come up for user interaction or dealing with failures.

Another place I want to call out is https://docs.mender.io/2.4/devices/update-modules which is also mentioned in the first link. This is how Mender is extended to provide updates for other parts of the system that are not the rootfs itself.  Since updating a Docker container is something that has been mentioned before I want to also note https://hub.mender.io/t/docker/324 as it is a well supported module for this specific case.

I hope you find this information useful. Please contact us if you have specific questions. We’re looking forward to talking to you about your own specific OTA needs.

Getting Started with RAUC on Raspberry Pi

RAUC is a secure, robust and flexible open source software for A/B updates of Embedded Linux devices. It is appropriate for various use cases and it is compatible with all popular build systems: The Yocto Project/ OpenEmbedded, Buildroot and PTXdist.

Konsulko Group engineers have experience with all popular open source solutions for software over the air updates of embedded Linux devices, including Mender, SWUPdate, HERE OTA Connect based on OSTree and Aktualizr. In this article we will discuss the exact steps to integrate RAUC with the Yocto Project (YP) and OpenEmbedded (OE) for Raspberry Pi – the most popular single board computer among students, hobbyists and makers.

For the practical example in this article we will be using the latest and greatest Raspberry Pi as of the moment: Raspberry Pi 4 Model B. Versions with different RAM sizes are available on the market. Any of these Raspberry Pi 4 Model B versions are OK for this RAUC demonstration.

As long time developers and users of the Yocto Project and OpenEmbedded, both have become favorite tools for creating customized distributions for Konsulko engineers. We frequently use and support them commercially. The Yocto Project is a Linux Foundation collaborative open source project for creating custom Linux distributions for embedded devices. It is based on Poky, the reference distribution of the Yocto Project, using the OpenEmbedded build system. The Yocto Project releases on a 6-month cadence. As of the time of this writing, the latest stable release is Dunfell (3.1).

RAUC is a powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image for Raspberry Pi built with the Yocto Project and OpenEmbedded, it requires:

  • U-Boot as a bootloader
  • Enabled SquashFS in the Linux kernel configurations
  • ext4 root file system
  • Specific partitioning of the microSD card that matches the RAUC slots
  • U-Boot environment configurations and a script to properly switch RAUC slots
  • Certificate and a keyring to RAUC’s system.conf

RAUC is capable of covering various use cases and scenarios, including advanced options for single or redundant data partitions. Upgrades are performed through the so called RAUC bundles. It is possible to install them over the air or using the old-fashioned method with a USB stick. For managing updates to a fleet of Internet of Things, it is possible to integrate RAUC with Eclipse hawkBit project that acts as a deployment server with a nice web user interface.

For the sake of simplicity, this article focuses on the most simple and straight-forward use case with 2 identical RAUC slots: A and B. For each slot we will have a separate partition on the microSD card for Raspberry Pi. We have already covered most of the RAUC requirements in an additional Yocto/OE layer called meta-rauc-raspberrypi. We will use it to put the pieces together. First we will build a minimal bootable image for Raspberry Pi 4 with RAUC. We will flash it to both A and B slots. After that we will build a RAUC bundle that adds the text editor nano. Finally we will install this RAUC bundle on the B slot, reboot and verify that nano is present.

Building a Linux Distribution with RAUC

Follow the steps below to build a minimal image for Raspberry Pi with Yocto, OpenEmbedded and RAUC as well as to perform a software update:

  • Download Poky, the reference distribution of the Yocto Project:

git clone -b dunfell git://git.yoctoproject.org/poky poky-rpi-rauc
cd poky-rpi-rauc

  • Download meta-openembedded layer:

git clone -b dunfell git://git.openembedded.org/meta-openembedded

  • Download Yocto/OE BSP layer meta-raspberrypi:

git clone -b dunfell git://git.yoctoproject.org/meta-raspberrypi

  • Download Yocto/OE layers for RAUC:

git clone -b dunfell https://github.com/rauc/meta-rauc.git

git clone -b dunfell https://github.com/leon-anavi/meta-rauc-community.git

  • Initialize the build environment:

source oe-init-build-env

  • Add layers to conf/bblayers.conf:

bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake-layers add-layer ../meta-openembedded/meta-python/
bitbake-layers add-layer ../meta-openembedded/meta-networking/
bitbake-layers add-layer ../meta-openembedded/meta-multimedia/
bitbake-layers add-layer ../meta-raspberrypi/
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-raspberrypi/

  • Adjust conf/local.conf for Raspberry Pi 4 with systemd and RAUC by adding the following configurations to the end of the file:
MACHINE = "raspberrypi4"

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

IMAGE_INSTALL_append = " rauc"

IMAGE_FSTYPES="tar.bz2 ext4 wic.bz2 wic.bmap"
SDIMG_ROOTFS_TYPE="ext4"
ENABLE_UART = "1"
RPI_USE_U_BOOT = "1"
PREFERRED_PROVIDER_virtual/bootloader = "u-boot"

WKS_FILE = "sdimage-dual-raspberrypi.wks.in"
  • Build a minimal bootable image:

bitbake core-image-minimal

NOTE: Building an image from scratch requires a lot of operations and takes some time so please patiently wait until bitbake completes all tasks.

  • Flash the image to a microSD card and boot it on Raspberry Pi 4:

sudo umount /dev/sdX*
bzcat tmp/deploy/images/raspberrypi4/core-image-minimal-raspberrypi4.wic.bz2 | sudo dd of=/dev/sdX
sync

  • Attach USB to UART debug cable to Raspberry Pi 4, plug ethernet cable and the microSD card. Turn on Raspberry Pi 4. Verify that the system boots successfully.
  • Now, let’s extend the image with the simple text editor nano by adding the following line to the end of conf/local.conf:

IMAGE_INSTALL_append = " nano"

  • Build a RAUC bundle:

bitbake update-bundle

  • Start a web server:

cd tmp/deploy/images/raspberrypi4/
python3 -m http.server

  • On the Raspberry Pi download the RAUC bundle, install it and reboot the board:

wget http://192.168.1.2:8000/update-bundle-raspberrypi4.raucb -P /tmp
rauc install /tmp/update-bundle-raspberrypi4.raucb
reboot

  • After successful upgrade with RAUC reboot the Raspberry Pi and verify that nano is now present:

which nano

  • Check RAUC status to confirm that now the second partition has been booted:

rauc status

For Internet of Things and other real-world products, the whole build procedure with the Yocto Project and OpenEmbedded can be optimized further to just a few commands for easy implementation of continuous integration (CI).

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We have experience with RAUC and various other open source solutions for software updates. Please contact us if you need your “own” rock-solid Linux distro for your own embedded product.