Tag Archive for: Leon Anavi

IMA-measurement with Yocto Project and OpenEmbedded

Integrity Measurement Architecture (IMA-measurement) is a subsystem in the Linux kernel designed to provide a framework for maintaining the integrity of files on a system. The primary purpose of IMA is to ensure that only trusted code and data are executed on a system and that the integrity of critical system components is maintained.

IMA was merged into the mainline Linux kernel in 2004 with the release of version 2.6.30. It evolved over time, adding features such as TPM integration, extended support, and continued maintenance, becoming a key component for ensuring the integrity of files in Linux-based systems. IMA works by calculating cryptographic hashes of files at various points in their lifecycle, such as when they are accessed, executed, or modified.

This article shares the exacts steps to build a minimal Linux distribution with IMA support for QEMU x86-64 using the Yocto Project and OpenEmbedded. The Yocto Project is an open-source collaboration project that enables developers to create lightweight, optimized, and customizable Linux distributions for embedded devices while maintaining control over the software components and configurations included in the system. To enable IMA, we use Yocto/OpenEmbedded layers meta-security and meta-integrity. These layers offer a comprehensive suite of security tools and hardening utilities designed for Linux kernels, along with libraries that facilitate the implementation of robust security mechanisms.

Building a Linux Distribution with IMA

Recently Leon Anavi, Konsulko Group Senior Engineer, contributed a couple of patches to the upstream of meta-security/meta-integrity to simplify using integrity-image-minimal. This is a small image provided as an example to demonstrate IMA support.

The following steps outline the process of building an image with Integrity Measurement Architecture (IMA) using the Yocto Project and OpenEmbedded. This demonstration uses the default debug keys provided in the “data” directory of layer meta-integrity. Because everyone has access to these private keys, for devices in production you must create your own private keys and use them instead. Enabling the audit kernel subsystem provides additional informational integrity auditing messages useful for debugging any appraisal issues that may arise during the development process.

Kindly be aware that this article utilizes source code derived from the primary branches of associated Yocto/OE layers. Consequently, we are selecting specific git commits that have been confirmed to function reliably. These commits will be part of the next long-term support (LTS) release of the Yocto Project which is version 5.0 with codename Scarthgap. It is scheduled for release in April 2024 and will be supported for 4 years until April 2028.

  • Download the source code:
git clone git://git.yoctoproject.org/poky poky-qemu
cd poky-qemu
git checkout e31be0b0e6ed6855787ebfbacc15bdbf1b9e511c
git clone git://git.yoctoproject.org/meta-security
cd meta-security
git checkout 30e755c59204cbd64c3aa12e64ab33041f6f02c0q
git clone git://git.openembedded.org/meta-openembedded
cd meta-openembedded
git checkout 57db42741df341718b9bb4b1b8f20d2e2fcf7010
  • Initialize the built envieronment:
source oe-init-build-env
  • Include additional layers:
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-security
bitbake-layers add-layer ../meta-security/meta-integrity
  • Append the following configurations to conf/local.conf:
DISTRO_FEATURES:append = " integrity ima"

IMAGE_CLASSES += "ima-evm-rootfs"

IMA_EVM_KEY_DIR = "${INTEGRITY_BASE}/data/debug-keys"
IMA_EVM_PRIVKEY = "${IMA_EVM_KEY_DIR}/privkey_ima.pem"
IMA_EVM_X509 = "${IMA_EVM_KEY_DIR}/x509_ima.der"
IMA_EVM_ROOT_CA = "${IMA_EVM_KEY_DIR}/ima-local-ca.pem"

IMA_EVM_POLICY = "${INTEGRITY_BASE}/recipes-security/ima_policy_hashed/files/ima_policy_hashed"

SRC_URI:append:pn-linux-yocto = " file://audit.cfg"
CORE_IMAGE_EXTRA_INSTALL += "auditd"

QB_KERNEL_CMDLINE_APPEND:remove:pn-integrity-image-minimal = "ima_policy=tcb ima_appraise=fix"
QB_KERNEL_CMDLINE_APPEND:append:pn-integrity-image-minimal = " ima_appraise=log ima_appraise_tcb integrity_audit=1"
  • Built an image with IMA for QEMU x86-64:
bitbake integrity-image-minimal

Testing IMA

After building the image, we can launch it. QEMU, short for Quick Emulator, is an open-source virtualization software that allows users to emulate various hardware platforms and run operating systems on different host systems. We will use it to run and test the image. By utilizing the “nographic” option, QEMU disables the video console, setting the console to “ttys0”. This feature is particularly beneficial when remotely accessing a build server over SSH. To verify the effectiveness of the appraisal process, attempt modifying a file, then confirm that the verification of the altered file subsequently fails.

  • Launch the image in QEMU:
runqemu nographic
  • Login and root and verify the integrity of file /etc/hostname using evmctl:
evmctl ima_verify /etc/hostname

The expected output should resemble:

Poky (Yocto Project Reference Distro) 4.3+snapshot-e31be0b0e6ed6855787ebfbacc15bdbf1b9e511c qemux86-64 /dev/ttyS0

qemux86-64 login: root
root@qemux86-64:~# evmctl ima_verify /etc/hostname
key 1: 6730eefd /etc/keys/x509_evm.der
/etc/hostname: verification is OK
  • Modify /etc/hostname:
echo test > /etc/hostname
  • Verify the integrity of file /etc/hostname again:
evmctl ima_verify /etc/hostname

Now the verification fails because the file has been modified. The anticipated output should be similar to:

root@qemux86-64:~# echo test > /etc/hostname
root@qemux86-64:~# evmctl ima_verify /etc/hostname
key 1: 6730eefd /etc/keys/x509_evm.der
/etc/hostname: verification failed: 0 ((null))

This simple example serves as a demonstration of how Linux IMA operates, using QEMU as a platform. However, to implement Linux IMA on real-world devices, Konsulko Group offers assistance with hardware bring-up and integration of the suitable Yocto/OE BSP (Board Support Package) layers.

Since the inception of OpenEmbedded and the Yocto Project, Konsulko engineers have actively contributed to the community and provided guidance for developing commercial products. We specialize in U-Boot, Linux kernel, RAUC, Mender, and various other open source projects for embedded Linux devices. Contact us to explore how Konsulko engineers can assist with your embedded product development endeavors.

Integrating RAUC with Yocto Project on BeagleBone Black

Konsulko Group has made many upstream contributions to OTA (over-the-air) update solutions for embedded Linux devices. RAUC is a popular open source option as it has been meticulously developed with a keen emphasis on stability, security, and adaptability. Notably, RAUC seamlessly integrates with major build systems such as Yocto Project/OpenEmbedded, Buildroot, and PTXdist.

Functioning across diverse usage scenarios, one of RAUC’s elementary yet impactful functionalities is the A/B update mechanism. In this setup, two identical root filesystems, denoted as A and B, are maintained. The device boots from one of these, while the other serves as the target for updates.

Post-update completion, the bootloader directs the system to boot from the freshly updated partition during the subsequent system startup. RAUC incorporates the ‘verity’ update bundle format. It extends the capabilities of RAUC by introducing built-in support for HTTP(S) network streaming, adaptive delta-like updates, and comprehensive update bundle encryption.

In previous blog posts, Konsulko Group engineers have demonstrated RAUC on Raspberry Pi and NXP devices such as SolidRun Cubox-i and HummingBoard. Recently Leon Anavi, Konsulko Group Senior Engineer and maintainer of meta-rauc-community ported RAUC to BeagleBone Black.

This article provides, as an example, the exact steps how to integrate RAUC with Yocto Project and OpenEmbedded for booting from a microSD card on BeagleBone Black.

Released in 2013, BeagleBone Black is a single-board computer (SBC) developed by the BeagleBoard.org Foundation. It was certified by the Open Source Hardware Association with OSHWA UID US000236. The chipset on BeagleBone Black is Texas Instruments Sitara AM3358 with 1GHz ARM Cortex-A8 CPU and SGX 3D graphics engine. Because of this the demonstrated integration is a suitable reference for other embedded devices equipped Texas Instruments chipsets.

Required Hardware

The hardware used for this step by step tutorial is:

Building a Linux Distribution with RAUC

RAUC, a robust and powerful open-source solution, demands advanced skills for initial integration. In general, to incorporate RAUC in a Yocto Project and OpenEmbedded image for BeagleBone Black the following actions have to be performed:

  • Use U-Boot as a bootloader
  • Enable SquashFS in the Linux kernel configuration
  • Use ext4 root file system
  • Design specific storage partitioning for the certain use case and configure RAUC accordingly
  • Provide a custom U-Boot script to properly switch between RAUC slots
  • Prepare a certificate and keyring to use for signing and verifying RAUC update bundles.

Leon Anavi has already done all these actions for core-image-minimal in Yocto/OpenEmbedded layer meta-rauc-community/meta-rauc-beaglebone. The layer is available at GitHub. Please follow the steps below to build core-image-minimal for BeagleBone Black with it:

  • Download the long term support (LTS) release Kirkstone reference Yocto distribution, Poky:
git clone -b kirkstone https://git.yoctoproject.org/poky poky-rauc-bbb
cd poky-rauc-bbb
  • Download the meta-rauc layer:
git clone -b kirkstone https://github.com/rauc/meta-rauc.git
  • Download meta-rauc-community layers, including meta-rauc-beaglebone:
git clone -b kirkstone https://github.com/rauc/meta-rauc-community.git
  • Download the meta-openembedded layer as it provides a recipe for nano which will be used for the demonstration:
git clone -b kirkstone git://git.openembedded.org/meta-openembedded

Initialize the build environment:

source oe-init-build-env
  • Include all layers in conf/bblayers.conf:
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-beaglebone
  • Adjust conf/local.conf by appending the following configurations to the end of the file:
MACHINE = "beaglebone-yocto"

# Use systemd as init manager
INIT_MANAGER = "systemd"

# Add RAUC to the image
IMAGE_INSTALL:append = " rauc"
DISTRO_FEATURES:append = " rauc"
  • Build a minimal bootable image:
bitbake core-image-minimal

The image creation process from scratch is time-consuming, encompassing various Yocto/OpenEmbedded recipes and configurations. Kindly await completion as bitbake diligently executes each tasks.

  • Flash tmp/deploy/images/beaglebone-yocto/core-image-minimal-beaglebone-yocto.wic.xz to microSD card.
  • Attach the USB-to-UART adapter to BeagleBone Black, plug the ethernet cable and the microSD card.

Press and hold button S2 while plugging in the 5V DC power supply to turn on BeagleBone Black and boot from microSD card.

BeagleBone black board has an onboard button labeled as S2. It is situated near the microSD card slot. Press and hold it while powering the board to boot from microSD card.

  • Verify that the system boots successfully, log in as user root without a password and check RAUC status:
rauc status

On the screenshot BeagleBone Black has been booted from RAUC slot rootfs.0 (A) on the microSD card.

NOTE: The meta-rauc-beaglebone layer includes a core-image-minimal.bbappend file, housing essential configurations for RAUC functionality. Apply these configurations similarly to other images intended for use in your embedded Linux device.

Creating a RAUC Update Bundle

RAUC update bundle comprises the file system image(s) or archive(s) designated for system installation, accompanied by a manifest detailing the images for installation, encompassing options and meta-information. Additionally, it may include scripts designated for execution before, during or after the installation process. To sign and verify the update bundles RAUC uses SSL keys. Layer meta-rauc-beaglebone contains a keyring containing all keys and a recipe for a simple RAUC update bundle for demonstration purposes only.

Follow the steps below to create RAUC update bundle that extends the system by adding the popular text based editor nano:

  • Add to conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build the RAUC update bundle:
bitbake update-bundle

Following a successful execution, bitbake will produce the update-bundle-beaglebone-yocto.raucb file.

Updating BeagleBone Black with RAUC

Follow the steps below to update core-image-minimal running from a microSD card on BeagleBone Black:

  • Start a Python 3 web server on the build machine or another computer where the RAUC update bundle (aka update-bundle-beaglebone-yocto.raucb) is available and within the same network as BeagleBone Black, for example:
cd tmp/deploy/images/beaglebone-yocto/
pip3 install --user rangehttpserver
python3 -m RangeHTTPServer
  • On BeagleBone Black replace <IP> with the actual IP address of the computer on which the HTTP server is running and execute the following command to install the update:
rauc install http://<IP>:8000/update-bundle-beaglebone-yocto.raucb

The screenshot show successful installation of the RAUC updated bundle on BeagleBone Black.

  • Reboot BeagleBone Black to load the updated version:
reboot

NOTE: As alternative, instead of using an HTTP server, you can transfer the update bundle to BeagleBone Black and install it from local storage.

  • Verify that nano was added to the system:
which nano
  • Check RAUC status to confirm the system have booted from the second partition:
rauc status

On the screenshot, after sucessful installation of the RAUC update bundle, BeagleBone Black has been booted from RAUC slot rootfs.1 (B) on the microSD card. This slot contains nano.

In real-world product development, the Yocto Project and OpenEmbedded workflow can be enhanced with a few commands for easy implementation of continuous integration (CI).

From the dawn of OpenEmbedded and the Yocto Project, Konsulko engineers have been community contributors and guides for crafting commercial products. Our expertise spans RAUC, Mender, and various open-source solutions for top-notch software updates. Please get in touch with us to discuss how Konsulko engineers can help your own embedded product development.

Konsulko Group engineer to speak at Yocto Project Summit

Konsulko Group senior engineer Leon Anavi will be presenting at the Yocto Project Virtual Summit, Tuesday, November 28 through Thursday, November 30, 2023.

Formerly known as the Yocto Project Developer Day, the Summit has an expanded scope to include both workshops for engineers building customized Linux distributions and applications, as well as an open forum where maintainers, trainers and users present papers on how the project is evolving and how they are using it.

On Wednesday, November 29, Leon will present Integrating VNC in Weston with the Yocto Project and OpenEmbedded. Weston 12 introduces Virtual Network Computing (VNC) support, allowing remote connection and control through graphical desktop sharing.

Leon will examine the technical aspects — dependance on Neat VNC and AML libraries, TLS support and user authentication — followed by a step by step tutorial to demonstrate remote connection from another computer.

The presentation aims to simplify the process of setting up the VNC backend in Weston and empower developers to utilize this powerful feature seamlessly within their projects.

We hope you will join us for this presentation, and all of this new edition of Yocto Project Virtual Summit.

Mender Add-ons: Remote Troubleshooting Devices in the Field

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

Konsulko Group often works with Mender.io to provide secure, risk tolerant and efficient Over-The-Air updates for any software on an embedded device. This includes both solutions for robust system updates as well as updates of single applications. Mender is compatible with any Linux based OS, including popular build systems such as Yocto Project/OpenEmbedded, Buildroot and OpenWrt.

Aside from OTA updates Mender also offers several add-ons. These are optional extensions that provide functionality for use cases beyond the core OTA updates features.

One such use case is remote troubleshooting. In practice deployed devices in the field are often hard to reach or retrieve. Troubleshooting individual devices becomes resource inefficient and in some cases they have to be replaced even when it’s just a small software or configuration issue. Our experience has shown in these situations Mender’s Add-ons have proven to be extremely useful.

This article provides as an example, the exact steps how to integrate Mender with The Yocto Project and OpenEmbedded for SolidRun CuBox-I and HummingBoard as well as a demonstration of the Mender Troubleshooting package, more specifically the File Transfer and Remote Terminal Add-ons. With the File Transfer add-on files can be downloaded and uploaded to any accepted device. Remote Terminal add-on allows remote interactive command execution from the Mender UI.

These add-ons are very valuable for system administration and mantainance of Internet of Things or fleets of connected industrial devices. For an example, we will troubleshoot an embedded Linux device remotely with systemd using Mender add-ons. We will upload tools to gather system boot-up performance statistics, run them on the device thanks to the Remote Terminal and download logs for further debugging.

Any Mender enabled device is suitable for this demonstration. Recently the Konsulko Senior Engineer Leon Anavi ported Mender to SolidRun CuBox-I and HummingBoard with NXP i.MX6 SoC so our current setup is based on this hardware platform and Yocto LTS release Kirkstone. Earlier we also ported RAUC, an alternative free and open source OTA update platform to the same hardware. If you are interested in software updates, please contact us to discuss your own embedded product needs in details. The hardware used in this example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • Generic RJ45 network cable with Internet connection
  • Optionally UART to USB adapter for debugging the setup section

Building a Linux Distribution with Yocto/OpenEmbedded

The Mender Community provides a set of examples for integration with various hardware platforms using Yocto/OpenEmbedded in the meta-mender-community repository. Sub-layer meta-mender-nxp in this repository contains the integration for Cubox-I/HummingBoard.

The meta-mender-nxp layer uses Google Repo to provide easy and simple setup and build process for the examples:

$ mkdir mender-nxp && cd mender-nxp
$ repo init -u https://github.com/mendersoftware/meta-mender-community \
            -m meta-mender-nxp/scripts/manifest-nxp.xml \
            -b kirkstone
$ repo sync
$ source setup-environment nxp

Board configuration

To configure the build system we have to append to conf/local.conf inside the build directory.

First we are going to set the build target machine:

MACHINE = "cubox-i"

Then we have to accept the end user agreement required by the BSP layer:

ACCEPT_FSL_EULA = "1"

Note: Usually to enable Mender’s Troubleshooting features we have to add mender-connect to IMAGE_INSTALL. In our case this is already added by the meta-mender-demo layer.

Mender configuration

We have to provide our device with MENDER_SERVER_URL and MENDER_TENANT_TOKEN.
For that reason we have to register at https://mender.io/.

Mender provides a free demo profile with limitation of 1 year and up to 10 devices which can be used to experiment with all of Mender’s features.

In fact, when sourcing the setup-environment script we get most of the mender-specific configuration appended to local.conf. This includes a description of how to get our tenant token:

# Build for Hosted Mender
#
# To get your tenant token:
#    - log in to https://hosted.mender.io
#    - click your email at the top right and then "My organization"
#    - press the "COPY TO CLIPBOARD"
#    - assign content of clipboard to MENDER_TENANT_TOKEN
#
#MENDER_SERVER_URL = "https://hosted.mender.io"
#MENDER_TENANT_TOKEN = ""

Note: If using the European server one has to set MENDER_SERVER_URL = "https://eu.hosted.mender.io"

Once we assign our tenant token and remove the # in front of MENDER_SERVER_URL and MENDER_TENANT_TOKEN we’re ready to build our system.

Building and flashing the system image to a microSD card

Build an example image with Yocto:

$ bitbake core-image-base

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

Once the build is complete flash the image to the microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-base-cubox-i.sdimg.bz2 /dev/sdX
$ sync
$ eject /dev/sdX

Connecting to Mender

Once the board finishes booting it will poll the Mender server. By design the connection has to be established from the board to the server. Mender does not open any ports on the board to provide better security therefore the device has to initiate the connection.

When the connection is established the Mender control panel will indicate one pending device.

To accept the request click on View details under Pending devices.

Select the new device and press accept in the Authorization request section.

Once the connection is accepted head over to the Troubleshooting tab in the Device information section. Here you can launch a remote terminal and transfer files.

The next part of the article will demonstrate preparing, uploading and using systemd-analyze to fetch data about the boot process.

Preparing troubleshooting software

To compile systemd-analyze we have to add it to our image in conf/local.conf:

IMAGE_INSTALL:append = " systemd-analyze"

Rebuild systemd to get the systemd-analyze binary:

$ bitbake systemd -c compile
When using the do_compile command Yocto/OpenEmbedded will preserve the compiled binaries.

Find the systemd-analyze binary and libsystemd-shared-<version>.so shared library:

$ find tmp/work -name "systemd-analyze"
$ find tmp/work -name "libsystemd-shared*.so"

Note: As of the time of writing of this article these files should reside in locations similar to ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/systemd-analyze and ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/src/shared/libsystemd-shared-250.so. These paths depend on the exact version of systemd as well as the build configuration and may not be correct in your case.

Uploading the troubleshooting software

Upload these files to the board using Mender’s File Transfer utility:

  • systemd-analyze into /usr/bin/
  • libsystemd-shared-<version>.so into /usr/lib/

Fetching service initialization logs

Once the troubleshooting software is uploaded we can use the Remote Terminal to execute it.

Permit execution of the systemd-analyze binary:

# chmod +x /usr/bin/systemd-analyze

Check the time it took for the system to initialize:

# systemd-analyze

Export a graphic of all enabled services and the time they took to initialize:

# systemd-analyze plot > init.svg

Download init.svg:

This graphic shows that the device needs around a minute to reach multi-user.target. The longest task is the resizing of the /data partition that runs on first boot and the second longest is the filesystem check for mmcblk1p1 that runs every time the system boots.

Here is another graphic generated after a reboot:

This article demonstrates how to use Mender’s Remote terminal and File Transfer troubleshooting utilities to upload and execute the systemd-analyze binary to profile the initialization process of systemd services. These troubleshooting utilities can be used for variety of different tasks. After debugging a single device and finding an appropriate fix, Mender is capable of performing an A/B or delta software update to all devices or specific group of devices in the field.

Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product needs.

Konsulko connects RDP with Wayland, Weston & Yocto at FOSDEM

Taking place February 4 & 5, 2023 in the beautiful city of Brussels (Belgium), FOSDEM is a two-day event organized by volunteers to promote the widespread use of free and open source software.

This year, Konsulko Group senior software engineer Leon Anavi will present Ups and Downs with Remote Desktop Protocol (RDP) on Wayland, Weston and the Yocto Project.

Leon’s tutorial provides the exact steps to build-from-scratch core-image-weston, setup RDP and remotely access the embedded device using wlfreerdp (for Wayland) or xfreerdp (for X11) from a personal computer with Linux in the same network. Examples will be based on the kirkstone long-term support release of Yocto Project.

Leon will also discuss the advantages and the disadvantages of RDP as well as some troubleshooting guidelines.

If you are attending FOSDEM, be sure to to catch this lightning talk at the Embedded, Mobile and Automotive devroom on Saturday.

Konsulko Group: The Year in Review 2022

2022 went by quickly with Konsulko engineers working closely with our customers, our partners and the open source community. For the tenth straight year, Konsulko Group has helped our clients build outstanding commercial products with Embedded Linux, Yocto Project and OpenEmbedded, as well as deploying Over-the-Air (OTA) software updating.

Engaging with our customers

We had particular success with our Konsulko Continuous Time Engagement™ offering, providing dedicated engineering resources for two of the world’s largest semiconductor companies. This model provides guaranteed engineering time for an agreed period. KCTE has allowed our customers to use our engineers on a variety of their requirements, and switch between these tasks as the rest of their project and in-house engineering required.

Of course, many of our clients prefer Konsulko OnDemand Time Engagement™, SOW-based engineering for high level consulting, on-demand support, and specific tasks within a larger project. KOTE is also best for longer term engagements without a hard deadline that can be stopped and started as necessary, and projects with an expected pause (such as bring-up of new hardware which almost always results in a re-spin of the PCB).

Partnerships and Conferences

We continue our strong relationship with the Linux Foundation and Automotive Grade Linux. We worked with mender.io and PHYTEC, providing support and development for their customers. With our friends at ICS, we presented at a joint webinar in August, Software Update Mechanisms: Selecting the Best Solution by Konsulko’s Leon Anavi and ICS’ Jeff Tranter.

Our Konsulko engineers were active participants (often in person, sometimes virtually) in conferences and developer gatherings. Leon Anavi, Vitaly and Maria Wool, and Tim Orling made presentations at the Embedded Linux Conferences (North America and Europe), FOSDEM and Yocto Project Summits:

Tim also posted a technical paper on Using kernel config fragments to remove an unwanted feature.

Two top engineers join the Konsulko team

The bar is pretty high for joining Konsulko Group. Some of our team have thirty years experience with embedded software and Linux. Most have over twenty and all have over fifteen years, working (and playing) with Linux and the open source community. So we were happy to welcome Denys Dmytriyenko and António Oliveira to Konsulko Group this year.

Denys is a long time Open Source Software developer, contributing code to many FOSS projects, like glibc, PostgreSQL, KDE, MythTV, LIRC, OpenEmbedded, and Yocto Project. Before joining Konsulko Group, he worked as Software Developer and Architect for Texas Instruments for almost 20 years on numerous Embedded Linux (MIPS and ARM-based) products. Since 2011 Denys has served on the OpenEmbedded Board of Directors, and since 2019, on the Yocto Project Technical Steering Committee. He is a long time contributor and maintainer of numerous recipes and layers in the OpenEmbedded/Yocto Project universe.

António has worked in development of highly optimized hardware and software ranging from deeply embedded microcontrollers to full-fledged graphical user interfaces. During this time, he has gained extensive experience in build automation tools, analog and digital circuits, powerline communications, and low power and low frequency radio communications. An active member of the open source community, he has contributed to Yocto Project and the Linux kernel. In addition to his work as an embedded engineer, António served eight years on the executive board of his local parish in Portugal, including four years on the municipal council.

We hope you will have the opportunity to work with Denys, António and the rest of Konsulko Group in 2023.

Konsulko Group engineers to speak at ELC Seattle

Embedded Linux Conference, September 27- 30, 2021

Hyatt Regency Seattle | Seattle, Washington + Virtual

For over 15 years, Embedded Linux Conference (ELC) has been the premier, vendor-neutral technical conference for companies and developers using Linux in embedded products. Recent ELCs have expanded the scope to include both the user-space developers building applications on embedded Linux as well as the architects and developers working to deliver smart connected products and industrial IoT solutions.

At the upcoming Embedded Linux Conference, September 27- 30, 2021 at the Hyatt Regency Seattle (with a virtual option available), Konsulko Group engineers will give detailed technical talks on building RISC-V Linux systems, and Linux A/B upgrades.

Building a Low-key XIP-enabled RISC-V Linux System

Tuesday, September 28 • 4:00pm – 4:50pm

Principal Software Engineer (and GM Konsulko AB) Vitaly Vul (aka, Vitaly Wool) will discuss new hardware designs based on RISC-V, an open standard instruction set architecture. In modern times, RISC-V SoCs quite often have QSPI flash onboard which makes them perfect candidates to use XIP (eXecute In Place) technology to execute directly from flash without copying the code to RAM first. That allows to optimize memory footprint very tightly and thus opens up to really low-power IoT Linux appliances. Vitaly will present a demo how to run a mainline kernel configured for XIP on a RISC-V board, and discuss extending XIP support for RISC-V to 32-bit and MMU-less systems for low-key battery-powered RISC-V systems with RAM shortage.

Practical Experience with Linux A/B Upgrades

Wednesday, September 29 • 1:30pm – 2:20pm

Senior Software Engineer Leon Anavi will discuss deploying software upgrades to fleets of embedded Linux IoT devices, using A/B redundant systems with two identical partitions using popular open source solutions (like Mender or RAUC). Although they provide great features out of the box there are still plenty of technical obstacles to overcome for real-world use cases. In this presentation, we will walk through practical experiences using both Mender and RAUC with Yocto Project and OpenEmbedded. Leon will discuss BSP integration and porting efforts for new devices, bootloader requirements and configurations, tips and tricks for ARM and x86-64 systems, read-only file systems, partition layouts, storing persistent data during upgrades, managing single-file artifacts, simultaneous setup with Docker and other containers.

Join us in Seattle or virtually for this always informative, educational and fun conference for commercial and community embedded Linux developers.

Time to sign up for Virtual ELC Europe 2020

Join us at the Linux Foundation’s Embedded Linux Conference Europe, the premier vendor-neutral technical conference for embedded Linux developers, presented virtually on October 26–29, 2020.

Konsulko Group’s Leon Anavi will give a talk on Software Update Solutions for Yocto Project and OpenEmbedded, exploring the integration in Yocto and OpenEmbedded of A/B and binary delta updates over the air or through a USB stick.

No matter where you are in the world, here’s a good chance to experience ELC Europe for a fraction of the cost of the in-person event.

Getting Started with RAUC on Raspberry Pi

RAUC is a secure, robust and flexible open source software for A/B updates of Embedded Linux devices. It is appropriate for various use cases and it is compatible with all popular build systems: The Yocto Project/ OpenEmbedded, Buildroot and PTXdist.

Konsulko Group engineers have experience with all popular open source solutions for software over the air updates of embedded Linux devices, including Mender, SWUPdate, HERE OTA Connect based on OSTree and Aktualizr. In this article we will discuss the exact steps to integrate RAUC with the Yocto Project (YP) and OpenEmbedded (OE) for Raspberry Pi – the most popular single board computer among students, hobbyists and makers.

For the practical example in this article we will be using the latest and greatest Raspberry Pi as of the moment: Raspberry Pi 4 Model B. Versions with different RAM sizes are available on the market. Any of these Raspberry Pi 4 Model B versions are OK for this RAUC demonstration.

As long time developers and users of the Yocto Project and OpenEmbedded, both have become favorite tools for creating customized distributions for Konsulko engineers. We frequently use and support them commercially. The Yocto Project is a Linux Foundation collaborative open source project for creating custom Linux distributions for embedded devices. It is based on Poky, the reference distribution of the Yocto Project, using the OpenEmbedded build system. The Yocto Project releases on a 6-month cadence. As of the time of this writing, the latest stable release is Dunfell (3.1).

RAUC is a powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image for Raspberry Pi built with the Yocto Project and OpenEmbedded, it requires:

  • U-Boot as a bootloader
  • Enabled SquashFS in the Linux kernel configurations
  • ext4 root file system
  • Specific partitioning of the microSD card that matches the RAUC slots
  • U-Boot environment configurations and a script to properly switch RAUC slots
  • Certificate and a keyring to RAUC’s system.conf

RAUC is capable of covering various use cases and scenarios, including advanced options for single or redundant data partitions. Upgrades are performed through the so called RAUC bundles. It is possible to install them over the air or using the old-fashioned method with a USB stick. For managing updates to a fleet of Internet of Things, it is possible to integrate RAUC with Eclipse hawkBit project that acts as a deployment server with a nice web user interface.

For the sake of simplicity, this article focuses on the most simple and straight-forward use case with 2 identical RAUC slots: A and B. For each slot we will have a separate partition on the microSD card for Raspberry Pi. We have already covered most of the RAUC requirements in an additional Yocto/OE layer called meta-rauc-raspberrypi. We will use it to put the pieces together. First we will build a minimal bootable image for Raspberry Pi 4 with RAUC. We will flash it to both A and B slots. After that we will build a RAUC bundle that adds the text editor nano. Finally we will install this RAUC bundle on the B slot, reboot and verify that nano is present.

Building a Linux Distribution with RAUC

Follow the steps below to build a minimal image for Raspberry Pi with Yocto, OpenEmbedded and RAUC as well as to perform a software update:

  • Download Poky, the reference distribution of the Yocto Project:

git clone -b dunfell git://git.yoctoproject.org/poky poky-rpi-rauc
cd poky-rpi-rauc

  • Download meta-openembedded layer:

git clone -b dunfell git://git.openembedded.org/meta-openembedded

  • Download Yocto/OE BSP layer meta-raspberrypi:

git clone -b dunfell git://git.yoctoproject.org/meta-raspberrypi

  • Download Yocto/OE layers for RAUC:

git clone -b dunfell https://github.com/rauc/meta-rauc.git

git clone -b dunfell https://github.com/leon-anavi/meta-rauc-community.git

  • Initialize the build environment:

source oe-init-build-env

  • Add layers to conf/bblayers.conf:

bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake-layers add-layer ../meta-openembedded/meta-python/
bitbake-layers add-layer ../meta-openembedded/meta-networking/
bitbake-layers add-layer ../meta-openembedded/meta-multimedia/
bitbake-layers add-layer ../meta-raspberrypi/
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-raspberrypi/

  • Adjust conf/local.conf for Raspberry Pi 4 with systemd and RAUC by adding the following configurations to the end of the file:
MACHINE = "raspberrypi4"

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

IMAGE_INSTALL_append = " rauc"

IMAGE_FSTYPES="tar.bz2 ext4 wic.bz2 wic.bmap"
SDIMG_ROOTFS_TYPE="ext4"
ENABLE_UART = "1"
RPI_USE_U_BOOT = "1"
PREFERRED_PROVIDER_virtual/bootloader = "u-boot"

WKS_FILE = "sdimage-dual-raspberrypi.wks.in"
  • Build a minimal bootable image:

bitbake core-image-minimal

NOTE: Building an image from scratch requires a lot of operations and takes some time so please patiently wait until bitbake completes all tasks.

  • Flash the image to a microSD card and boot it on Raspberry Pi 4:

sudo umount /dev/sdX*
bzcat tmp/deploy/images/raspberrypi4/core-image-minimal-raspberrypi4.wic.bz2 | sudo dd of=/dev/sdX
sync

  • Attach USB to UART debug cable to Raspberry Pi 4, plug ethernet cable and the microSD card. Turn on Raspberry Pi 4. Verify that the system boots successfully.
  • Now, let’s extend the image with the simple text editor nano by adding the following line to the end of conf/local.conf:

IMAGE_INSTALL_append = " nano"

  • Build a RAUC bundle:

bitbake update-bundle

  • Start a web server:

cd tmp/deploy/images/raspberrypi4/
python3 -m http.server

  • On the Raspberry Pi download the RAUC bundle, install it and reboot the board:

wget http://192.168.1.2:8000/update-bundle-raspberrypi4.raucb -P /tmp
rauc install /tmp/update-bundle-raspberrypi4.raucb
reboot

  • After successful upgrade with RAUC reboot the Raspberry Pi and verify that nano is now present:

which nano

  • Check RAUC status to confirm that now the second partition has been booted:

rauc status

For Internet of Things and other real-world products, the whole build procedure with the Yocto Project and OpenEmbedded can be optimized further to just a few commands for easy implementation of continuous integration (CI).

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We have experience with RAUC and various other open source solutions for software updates. Please contact us if you need your “own” rock-solid Linux distro for your own embedded product.

 

Helping Yocto Project work with Python 3

According to the statistics from StackOverflow Python is the fastest-growing major programming language. First released in 1991, Python is nowadays commonly used for various applications in multiple different industries. Python is a first class citizen of many embedded Linux systems.

The Yocto Project, a collaborative project of the Linux Foundation for creating custom Linux distributions for embedded devices, uses the OpenEmbedded build system and relies on layer meta-python from meta-openembedded to deliver Python 3 packages. Until recently, meta-python was providing both python 2 and python 3 versions of each package. The Python community decided that January 1, 2020, was the day to sunset Python 2. Since then Python 2 has been officially deprecated. This triggered major changes related to the support in Yocto and OpenEmbedded. All recipes for version 2 were moved to layer meta-python2 to provide legacy support after the end of life for this Python release. In meta-openembedded/meta-python, the OpenEmbedded community started efforts to remove all recipes for version 2 as well as to consolidate inc and bb files into a single bb file for version 3.

Konsulko Group engineers are regular contributors to various upstream open source projects, including meta-openembedded and more specifically to meta-python. In the past month, Leon Anavi joined the community efforts for consolidating Python 3 recipes in a single file as well as for upgrading various packages. Nowadays, most of the Python 3 recipes are utilizing the pypi bbclass which takes care for downloading and processing packages from pypi.org. This makes most of the upgrades to new releases of a Python package straight-forward. However, it is important to check the list of build and runtime dependencies as well as to ensure that bitbake still works fine with the upgraded recipe version for both x86-64 and ARM architectures prior to submission. 

Let’s have a closer look at the recipe python3-protobuf. It has been recently upgraded from version 3.11.3 to version 3.12.2. Protocol Buffers, also known as protobuf, are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. In the Yocto and OpenEmbedded ecosystem, recipe python3-protobuf depends on recipe protobuf from layer meta-oe. Both meta-oe and meta-python are part of meta-openembedded. So to avoid version mismatch and to ensure that bitbake will be able to successfully build python3-protobuf version 3.12.2 an upgrade of recipe protofobuf to the same version was mandatory. We contributed both upgrades to the master branch of the git repository meta-openembedded. The maintainers took care of cherry-picking them to the dunfell branch which is compatible with the latest stable release of the Yocto Project as of the moment. As a result, if you checkout the latest stable release of Poky, the reference system of the Yocto Project, and meta-openembedded you will be able to quickly build the latest version of protobuf and python3-protobuf out of the box.

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We continue to regularly make upstream contributions to these open source projects. Please contact us if you need your “own” Linux distro for your own embedded product.

Tag Archive for: Leon Anavi