Tag Archive for: konsulko group

Build an image and perform updates with RAUC on Rockchip

Over the years Konsulko Group engineers made many upstream contributions to various open source OTA (over-the-air) update solutions for embedded Linux devices. Recently Leon Anavi, Konsulko Group Senior Engineer and maintainer of meta-rauc-community ported RAUC to Radxa Rock Pi 4 Model B. This is the first Rockchip device supported in meta-rauc-community.

RAUC is one of the popular solutions that provide OTA updates for Embedded Linux devices. RAUC is developed with focus on stability, security and flexibility and is compatible with all popular build systems.

The Radxa Rock Pi 4 Model B is a single-board computer with a Rockchip RK3399 processor, 4GB RAM, and various storage options including microSD card and NVMe SSD. It features Gigabit Ethernet, dual-band Wi-Fi, Bluetooth 5.0, USB 3.0, 4K video output via HDMI and USB-C, and a 40-pin GPIO header compatible with Raspberry Pi. This article explains how to build an image for the Radxa Rock Pi 4 Model B using the Yocto Project and OpenEmbedded, and how to perform software updates with RAUC.

Building an Image

  • Download the long term support (LTS) release Scarthgap reference Yocto distribution, Poky:
git clone -b scarthgap https://git.yoctoproject.org/poky poky-rauc-rockchip
cd poky-rauc-rockchip
  • Download the meta-arm BSP layer:
git clone -b scarthgap git://git.yoctoproject.org/meta-arm
  • Download the meta-rockchip BSP layer:
git clone -b scarthgap git://git.yoctoproject.org/meta-rockchip
  • Download the meta-rauc layer:
git clone -b scarthgap https://github.com/rauc/meta-rauc.git
  • Download meta-rauc-community layers, including meta-rauc-rockchip:
git clone -b scarthgap https://github.com/rauc/meta-rauc-community.git
  • Download the meta-openembedded layer as it provides a recipe for nano which will be used for the demonstration:
git clone -b scarthgap git://git.openembedded.org/meta-openembedded
  • Initialize the build environment:
source oe-init-build-env
  • Include all layers in conf/bblayers.conf:
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain
bitbake-layers add-layer ../meta-arm/meta-arm
bitbake-layers add-layer ../meta-rockchip
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-rockchip
  • Adjust conf/local.conf by appending the following configurations to the end of the file:
MACHINE = "rock-pi-4b"

SERIAL_CONSOLES="115200;ttyS2"
IMAGE_FSTYPES:append = " ext4"
WKS_FILE = "rockchip-dual.wks.in"
MACHINE_FEATURES:append = " rk-u-boot-env"
UBOOT_EXTLINUX_KERNEL_IMAGE="/${KERNEL_IMAGETYPE}"
UBOOT_EXTLINUX_ROOT="root=PARTLABEL=${bootpart}"
UBOOT_EXTLINUX_KERNEL_ARGS = "rootwait rw rootfstype=ext4 rauc.slot=${raucslot}"
WIC_CREATE_EXTRA_ARGS = "--no-fstab-update"

DISTRO_FEATURES:append = " rauc"

INIT_MANAGER = "systemd"
  • Build the image:
bitbake core-image-base

Building an image from scratch is a lengthy process that involves numerous Yocto/OpenEmbedded recipes and configurations. Please be patient while bitbake methodically handles each task.

  • Flash tmp/deploy/images/rock-pi-4b/core-image-base-rock-pi-4b.rootfs.wic to microSD card.
  • Attach the USB-to-UART adapter to Raxda Rock Pi 4 Model B, plug the ethernet cable and the microSD card.
  • Plug USB-C power supply.
  • Verify that the system boots successfully, log in as user root without a password and check RAUC status:
rauc status

Creating a RAUC Update Bundle

A RAUC update bundle comprises the file system image(s) or archive(s) designated for system installation, accompanied by a manifest detailing the images for installation, encompassing options and meta-information. Additionally, it may include scripts designated for execution before, during or after the installation process. To sign and verify the update bundles RAUC uses SSL keys. Layer meta-rauc-beaglebone contains a keyring containing all keys and a recipe for a simple RAUC update bundle for demonstration purposes only.

Follow the steps below to create RAUC update bundle that extends the system by adding the popular text based editor nano:

  • Add to conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build the RAUC update bundle:
bitbake update-bundle

Updating Radxa Rock with RAUC

Follow the steps below to update core-image-minimal running from a microSD card on Radxa Rock Pi 4 Model B:

  • Start a Python 3 web server on the build machine or another computer where the RAUC update bundle (aka update-bundle-rock-pi-4b.raucb) is available and within the same network as Radxa Rock Pi 4 Model B, for example:
cd tmp/deploy/images/rock-pi-4b/
pip3 install --user rangehttpserver
python3 -m RangeHTTPServer
  • On Radxa Rock Pi 4 Model B replace <IP> with the actual IP address of the computer on which the HTTP server is running and execute the following command to install the update:
rauc install http://<IP>:8000/update-bundle-rock-pi-4b.raucb

NOTE: As alternative, instead of using an HTTP server, you can transfer the update bundle to Rock Pi 4 Model B and install it from local storage.

  • Reboot Radxa Rock Pi 4 Model B Black to load the updated version:
reboot
  • Verify that the system boots and nano was added:
which nano

Check RAUC status to confirm the system have booted from the second partition:

rauc status

How Does It Work?

The default serial baud rate for the Radxa Rock Pi 4’s U-Boot and kernel console is 1500000. However, many USB-to-UART cables, such as those using the popular Silicon Labs CP2102 chip, cannot handle this high speed. To avoid these issues, a patch is applied to rock-pi-4-rk3399_defconfig through u-boot_%.bbappend, setting U-Boot’s baud rate to 115200. Additionally, for the kernel, we set SERIAL_CONSOLES="115200;ttyS2" in conf/local.conf. This ensures that both U-Boot and the Linux kernel operate at a baud rate of 115200 in our demonstration.

The Radxa Rock Pi 4 Model B uses U-Boot with extlinux support to boot. For RAUC integration, it employs a boot.scr script, which handles the A/B system switching and passes environment variables to extlinux/extlinux.conf. The rockchip-dual.wks.in file creates two identical partitions (A and B), a data partition, and a fixed-size 32MB boot partition. The boot.scr and extlinux/extlinux.conf files are stored in the boot partition.

In real-world product development, the Yocto Project and OpenEmbedded workflow can be improved with some simple commands to facilitate continuous integration (CI).

Since OpenEmbedded and the Yocto Project began, Konsulko engineers have been key contributors and mentors in developing commercial products. Our team excels in using RAUC, Mender, and other open-source tools for delivering superior software updates. Reach out to us to see how Konsulko engineers can support your embedded product development efforts.

Two technical talks about the Yocto Project at TuxCon 2024

TuxCon is an annual open-source conference held in Plovdiv, Bulgaria, aimed to promote the adoption of open-source hardware and software, and organized by passionate volunteers. Since the conference’s inception in 2014, Konsulko Group engineers have participated and supported this important local event.

The 2024 edition of TuxCon took place on May 11th and 12th at the Technical University of Sofia, Plovdiv branch, featuring presentations on various interesting topics. This edition featured a couple of talks (in Bulgarian) about the Yocto Project and OpenEmbedded.

Konsulko Group junior engineer Atanas Bunchev spoke at the event, sharing his experience with Balena on Raspberry Pi and PHYTEC phyBOARD-AM62xBalena.io, commonly known as Balena, is a platform that simplifies the deployment and management of embedded Linux devices using images based on the Yocto Project and OpenEmbedded. It leverages Docker containers for streamlined application deployment across various Internet of Things (IoT), offers centralized device management for monitoring and updates, and supports scalability from prototypes to large-scale projects. Atanas further elaborated on the utilization of Balena AutoKit for conducting automated QA testing on embedded Linux devices.

The second talk about the Yocto Project at TuxCon 2024 was delivered by Sadika Hasan, a senior-year student from “Paisii Hilendarski” University of Plovdiv. She demonstrated how to create a custom Linux distribution and Software Development Kit (SDK) by extending Poky, the Yocto Project’s reference distribution. Additionally, she covered the integration of the custom SDK with Visual Studio Code and the remote debugging of a C++ application on a Raspberry Pi Zero W 2.

Sadika’s presentation is part of her bachelor thesis, supervised by Professor Dimitar Tokmakov from ECIT (Electronics and Information and Communication Technology) department of “Paisii Hilendarski” University of Plovdiv and Konsulko engineer Leon Anavi.

Professor Tokmakov, with the assistance of engineer Nikolay Nedelev from Romit LTD, is actively involved in the ECOVEM project (European Center of Vocational Excellence in Microelectronics). As part of their efforts, two students from PU developed diploma theses for their Bachelor of Science degrees utilizing the Yocto Project. These theses were based on real-life problem solving and the practical experience of the Konsulko Group with industrial embedded Linux devices.


From the early days of OpenEmbedded and the Yocto Project, Konsulko engineers have actively contributed upstream and engaged with the community, offering their expertise in developing high-quality commercial products. If you are developing a new product, get in touch to see how Konsulko’s engineering expertise can enhance your project. Additionally, if you are a Linux software developer passionate about open source, we invite you to explore potential opportunities to join the Konsulko team.

Porting Mender to Raspberry Pi 5 and Yocto Project Scarthgap

Overview

Mender is an open-source over-the-air (OTA) software update technology for embedded Linux devices and Internet of Things (IoT) ecosystem. It allows developers to remotely manage and update software on embedded Linux devices, ensuring that they remain secure, up-to-date, and functional throughout their lifecycle. Mender simplifies the process of launching a new project by offering official and community-supported board integrations for a wide range of devices and operating systems. These integrations include support for Debian family and the Yocto Project, making it easier for developers to get started with their projects seamlessly.

The Yocto Project is an open-source collaboration project by the Linux Foundation to create custom Linux-based systems for embedded devices. It uses the OpenEmbedded build automation framework with the build tool bitbake and provides Poky as a reference Linux distribution. The Yocto Project follows a regular release cycle, typically with a new version every six months, alongside long-term supported releases available every two years. Currently, the most recent LTS release is version 5.0, codenamed Scarthgap, which became available on April 30, 2024.

Recently, Senior Engineer Leon Anavi from Konsulko Group ported Mender to Raspberry Pi 5. This effort builds upon his previous contributions, where he added Raspberry Pi 5 support to the Yocto Project BSP (Board Support Package) layer meta-raspberrypi. This progress was made possible through sponsorship from the computer emergency response team of the Government of the Grand Duchy of Luxembourg (GOVCERT.LU). If you’re considering using Mender on Raspberry Pi or any other hardware platform for your embedded product needs, feel free to reach out to us to discuss further.

Raspberry Pi 5 introduces significant hardware differences compared to its predecessors. To enable Mender updates on this platform, we require U-Boot version v2024.04 or later. Unfortunately, the Yocto Project release Scarphgap ships with U-Boot version v2024.01, which isn’t compatible. To address this compatibility issue and obtain the necessary U-Boot version, our setup utilizes the scarthgap/u-boot branch from the meta-lts-mixins layer. Tim Orling, Principal Software Engineer at Konsulko Group, contributed the patches in meta-lts-mixins for U-Boot v2024.04.

Raspberry Pi 5 is the first model of the famous single board computers that features a dedicated UART connector is a three-pin header compatible with the Raspberry Pi Debug Connector specification. It can be used with Raspberry Pi Debug Probe, a USB device that provides both a UART serial port and a standard Arm Serial Wire Debug (SWD) interface.

The article offers a practical guide, outlining the exact steps to build a Mender-enabled image for Raspberry Pi 5 and execute an A/B update. To streamline the build setup, we’ll utilize KAS. This Python-based open-source tool effectively handles various Yocto/OpenEmbedded layers. KAS executes builds within a Docker container to ensure consistency and reliable build outcomes, regardless of the primary GNU/Linux distribution on the build machine.

Image for Raspberry Pi 5

Follow the steps below to build core-image-minimal with Mender for Raspberry Pi 5:

  • Install the kas tool (optionally, you can install globally for all users. Run as root, respectively under sudo then):
pip install kas
  • Clone this repository:
git clone -b scarthgap https://github.com/mendersoftware/meta-mender-community
  • Create a build directory and change into it:
mkdir -p meta-mender-community/mender-rpi5 && cd meta-mender-community/mender-rpi5
  • Use kas to build for the Raspberry Pi 5:
kas build ../kas/raspberrypi5.yml
  • Flash tmp/deploy/images/raspberrypi5/core-image-minimal-raspberrypi5.sdimg to a microSD card and boot it on Raspberry Pi 5.

Mender Artifact for Raspberry Pi 5

Follow the steps below to build a Mender Artifact for Raspberry Pi 5 that provides the simple text editor nano:

  • Enter KAS shell:
kas shell ../kas/raspberrypi5.yml
  • Append to the end of conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build both core-image-minimal and a Mender Artifact for it:
bitbake core-image-minimal
  • As a result Mender Artifact containg nano will be generated as file tmp/deploy/images/raspberrypi5/core-image-minimal-raspberrypi5.mender

Update Raspberry Pi 5

As an end to end update solution, Mender provides aserver as the central hub for storing and orchestrating software updates across fleets of devices through over-the-air deployment. Through Mender’s user-friendly web UI or REST APIs, you can easily oversee device management, upload software releases, and create seamless deployments to distribute updates to your devices. However, it is also possible to use Mender in standalone mode without a server.

Follow the steps below to manually perform a standalone deployment with Mender in the terminal. In this scenario, no Mender Server is utilized, and the deployments are triggered directly at the device.

  • Start a simple HTTP server in the directory with the Mender Artifact:
python3 -m http.server
  • Login as root on Raspberry Pi 5 and install the Mender Artifact to perform an upgrade of the device:
mender-update install http://<server>:8000/core-image-minimal-raspberrypi5.mender

NOTE: Replace <server> with the IP address of the machine on which the Python3 HTTP server is running.

  • Reboot Raspberry Pi 5:
reboot
  • Login as root on Raspberry Pi 5 and verify that nano text editor has been installed.
  • Make the deployment permanent:
mender-update commit

This straightforward example showcases the seamless integration of Mender with Raspberry Pi 5 using the Yocto Project release Scarthgap. You can follow up and manage updates of fleets of Raspberry Pi 5 devices through the Mender server. Additionally, Mender offers convenient add-ons for remote troubleshooting, ensuring smooth operations in the field.


Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been active contributors to the community, aiding customers in crafting commercial products using these technologies. Our expertise extends beyond Mender, encompassing various open-source solutions for software updates. Feel free to get in touch to explore how we can assist with your embedded product requirements.

Konsulko Group speaks at EOSS North America 2024 in Seattle

Vitaly Wool and Tim Orling, both Principal Software Engineers at Konsulko Group are presenting at the Embedded Open Source Summit (EOSS) North America in Seattle Washington, April 15-18, 2024.

EOSS (which incorporates the Embedded Linux Conference) is an umbrella event for open source embedded projects and developer communities to come together under one roof for important collaboration, discussions and education.

Vitaly Wool

Vitaly, who is also general manager of Konsulko AB in Lund, Sweden, will give a technical talk on “Rusty Swapping: Rewriting a zswap Backend in Rust.”

Rust has gained popularity as the “second” Linux kernel high-level language. There’s been discussions about its applicability in various kernel subsystems which yielded tentative conclusions. Engineers have been advised by kernel gurus to use Rust for subsystem implementations rather than for drivers.

Vitaly will explain how he rewrote a zswap backend called zblock in Rust, then compare the performance and complexity of the two implementations.

Tim Orling

Tim, who serves on the board of directors of OpenEmbedded, will speak at the Yocto Project half-day mini-summit “Journey to Scarthgap 5.0” on Monday, April 15. With other YP community leaders, he will deep dive into upcoming features and improvements in the upcoming LTS release.

The rest of the week, you may find Tim in and around the Yocto Project booth in the exhibition hall. Please stop by with all your Yocto questions.

Balena: Running Containerized Applications on phyBOARD-AM62x

This article was prepared by Atanas Bunchev.

Balena.io, commonly known as Balena, is a platform simplifying IoT device deployment and management. It uses Docker containers for easy application deployment across various embedded Linux devices, offers centralized device management for monitoring and updates, and supports scalability from prototypes to large-scale projects. With over-the-air (OTA) updates and monitoring tools, Balena streamlines IoT application development and management. Balena supports over a 100 device types with robust and resilient remote updates, combined with a powerful toolset for monitoring, maintaining and debugging.

Recently PHYTEC took interest into having their AM62x-based development board added to the list of Balena-supported devices and Konsulko was on the task. The process of bringing Balena support for a new board consists of creating a custom Yocto-based Board Support Repository and having it pass an automated testing procedure run on the actual hardware.

At the heart of the phyBOARD-AM62x is the industrial PHYTEC phyCORE-AM62x. This versatile System-on-Module (SoM) is powered by Texas Instruments AM62x Sitara processor and is equipped with Ethernet, CAN, UART, I2C, SPI, dual display, MIPI CSI-2 camera and audio. The phyBOARD-AM62x offers a modern FTDI interface allowing software download and debugging, perfect for development. Paired alongside the M.2 connectorized WiFi and Bluetooth module extensions makes a perfect product for IoT devices.

Konsulko engineers created balena-phytec git repository which is currently publicly available at the BalenaOS GitHub organization. It is based on the Yocto/OpenEmbedded board support package (BSP) layers meta-ti and meta-phytec. The repository provides all necessary modifications to run BalenaOS (Balena’s specific Linux distribution) on the phyBOARD-AM62x, including a new Yocto/OpenEmbedded integration layer meta-balena-phytec that extends the recipes for the U-Boot bootloader and Linux kernel.

Texas Instruments AM62x SoCs come with a RTI/WWDT Windowed Watchdog Timer which would turn off the board when not serviced within a specific time interval. RTI only supports a windowed mode, where the watchdog can only be petted during the open window; not too early or not too late. However, due to the nature of systemd utilized by BalenaOS, it is unable to ping the watchdog within the designated open window period, as it attempts to ping at watchdog_timeout/2 ticks. Therefore the systemd watchdog has been disabled by a BitBake append file for systemd in layer meta-balena-phytec.

To finish the task Konsulko and Balena engineers ran a special test suite on the board using the Balena AutoKit. Shortened from Automation Kit, the AutoKit is a complete hardware solution for automated interaction with embedded Linux device. It features SD card multiplexing, Ethernet and power control, serial communication, HDMI capture and support for other USB peripherals.

The rest of the article covers a step-by-step guide for connecting the phyBOARD-AM62x to BalenaCloud and deploying a simple containerized application on it, as well as demonstration of some of the basic features provided by BalenaCloud’s Dashboard.

Downloading an Image

The first thing you have to do is to register at https://dashboard.balena-cloud.com/signup. Every user is allowed to have up to 10 devices with all features enabled for free. While the software that Balena develops is open source, the usage of their cloud infrastructure with more than 10 connected IoT devices requires a plan subscription, with better plans coming with more devices and better support response time.

Once you register and confirm your e-mail you’ll be welcomed with a screen providing an introduction to Balena and inviting you to create a fleet.

A fleet is a group of devices with identical configuration and with the same applications deployed on them. Create a new fleet and set the device type to phyBOARD-AM62x.

Once your fleet is created click the “Add device” button. Select the Development edition, as it includes a few quality of life modifications for developers, such as passwordless ssh access as root on port 22222.

Once you have picked the configuration for the new image you can download it by selecting the Download balenaOS option from the menu next to the Flash button and pressing on that button after that.

Alternatively if you have balenaEtcher installed you can directly click on the Flash button. That way balenaEtcher will start up with the correct URL for the image already selected, leaving only the output device selection to you.

The third option – downloading a configuration file – is applicable when having a custom BalenaOS image that you want to configure for a given fleet.

Flashing the Installation media

Once the image is downloaded you need to flash it on a microSD card. There are several ways of flashing an image. One way is to use a graphical tool for the task, such as balenaEtcher or Rufus (keep in mind that some of these will expect you to unzip the image in advance).

Another way to do is to use the Linux shell. Replace <your_image>.img.zip with the compressed image you downloaded and <your_image>.img with the extracted file. Replace /dev/sdX with the path to the microSD card.

unzip <your_image>.img.zip
umount /dev/sdX*
dd if=<your_image>.img of=/dev/sdX bs=128k status=progress
sync

Keep in mind that flashing an image on a microSD card will delete all data stored on the microSD card.

Installing Balena on the phyBOARD-AM62x

Note: The following procedure will delete all data on the internal eMMC storage of your phyBOARD-AM62x.

Plug in the newly-flashed microSD card into the phyBOARD-AM62x board. Make sure the boot switches are set to SD card mode. Power on the board and wait for the installation to the internal eMMC to finish.

Once the installation is complete the board will turn off. This is indicated by LEDs D11 and D12 turning off.

As the installation on the internal eMMC has finished, set the boot mode to eMMC. Disconnect the microSD card, connect phyBOARD-AM62x to Ethernet and power it on.

In few moments the board will show up on the fleet page at the Dashboard.

Clicking on phyBOARD-AM62x in the table will open a dedicated page for the board that shows details and current status information.

On the left you can see the fleet containing the phyBOARD-AM62x, its online status, its unique UUID, the Host OS and Balena Supervisor versions, the local and public IP address of the board. You can enable or disable the Public Device URL of the board, as well as use the Actions menu to manage the board.

The Identify action will make a specific LED on phyBOARD-AM62x blink for half a minute to make identifying the exact board easier – a very useful feature when having a bunch of similar boards lying around. On phyBOARD-AM62x this is User LED 1, a bright red LED at the left edge of the carrier board.

At the upper-right corner you can see the current resource usage on the board, as well as the temperature and free persistent storage space.

Below them are the Logs and Terminal sections, which can be used for advanced remote debugging and troubleshooting of the board.

Lastly, at the currently empty space at the bottom-left part of the screen we’ll be able to see the status of the containerized applications currently deployed on the board.

Pushing a hello-world application to phyBOARD-AM62x

The last section of this article contains a quick example of how to push a containerized application to a board that runs Balena. The example application that will be deployed is a Docker container consisting of a simple web page hosted with NodeJS Express.

Download Balena CLI tool, a Command Line Interface for balenaCloud or openBalena. Select the standalone binary for your workstation – the machine you’re going to use to deploy the application, not the board you’re going to deploy to.

As of the time of writing, the most recent version is balena-cli-v18.1.5-linux-x64-standalone.zip. If you are utilizing a newer release or a different operating system, kindly substitute the filename accordingly.

unzip balena-cli-v18.1.5-linux-x64-standalone.zip
cd balena-cli

You need to login in your profile from the CLI before you can push anything to your devices. For that task you can use the ./balena login command:

./balena login
 _            _
| |__   __ _ | |  ____  _ __    __ _
| '_ \ / _` || | / __ \| '_ \  / _` |
| |_) | (_) || ||  ___/| | | || (_) |
|_.__/ \__,_||_| \____/|_| |_| \__,_|


Logging in to balena-cloud.com
? How would you like to login? (Use arrow keys)
❯ Web authorization (recommended)
  Credentials
  Authentication token
  I don't have a balena account!

Press Enter to select Web authorization and confirm the authentication request on the web page that opens:

Alternatively, select the Credentials option and insert your username and password.

Once authentication is successful, clone the Hello World sample from https://github.com/balena-io-examples/balena-nodejs-hello-world and push it to your fleet. Replace <fleet name> with the actual name of the fleet that the phyBOARD-AM62x is part of.

git clone https://github.com/balena-io-examples/balena-nodejs-hello-world.git
cd balena-nodejs-hello-world
../balena push <fleet name>

Once the application is uploaded you can see it on the board’s page at the web dashboard.

The Hello World example is a web server hosting a single web page. To check it out, you can either visit the board’s local IP address if you’re in the same local network as phyBOARD-AM62x, or enable the PUBLIC DEVICE URL and open the hyperlink that appears next to the switch once enabled.

This article demonstrates how to setup and use the PHYTEC phyBOARD-AM62x development board with Balena, a container-based distribution designed for easy and rapid development of embedded applications, packed with useful device management and troubleshooting features as well as sophisticated monitoring and automation-oriented ones. Get in touch with us to discuss additional features. Our team is ready to help in development of new or already existing Linux embedded projects.

About Konsulko Group

From the earliest days of OpenEmbedded and the Yocto Project, Konsulko engineers have actively contributed upstream and participated in the community, offering their expertise and guidance in developing high-quality commercial products. Our proficiency extends to Balena, RAUC, Mender, and other open-source solutions, ensuring seamless and reliable over-the-air updates. Contact us to explore how Konsulko engineers can contribute to the advancement of your embedded Linux product development.

___

Konsulko Group extends its hardware services business

Konsulko Group welcomes three veteran hardware developers to our growing team. Alexandar Kalaydjiev will serve as Konsulko Hardware Director. Joining as Principal Hardware Engineers are Marin Balkandjiev and Tsvetan Mudrov, PhD.

Customers can now leverage their technical expertise and extensive experience to create comprehensive products and solutions tailored to their specific requirements.

Konsulko hardware engineers can manage and facilitate the production of the developed products, ensuring their successful realization. We can provide deployment support as required, ensuring a smooth transition for the end-customer.

Meet the new members of the Konsulko team

Alexandar Kalaydjiev (middle photo) was an integral member of a team that developed a DVD writer in the early 2001, focusing on the control mechanisms for the semiconductor laser during the writing process. His contributions extended to the mechatronic design of both pneumatic and electrical robotic manipulators, specifically tailored for critical procedures in the manufacturing of DVD discs. Over the last 15 years, he has dedicated himself to comprehensive product design encompassing electrical design, PCB layout, CAD modeling, and production management.

Marin Balkandjiev (left photo) has spent over 8 years in the Telecom industry working with access multiplexers, data link converters and other key hardware. He has 10 years experience in the Automotive industry developing automotive interior products, and 10 years building consumer and industrial products, IoT, and sensors. He worked previously for Johnson Controls Automotive Electronics.

Tsvetan Mudrov (right photo) has 20 years of experience in development of Medical Electronics including external defibrillators, biomedical signals acquisition and processing (including ECG, EEG, PPG, bio impedance), medical telemetry and long-term monitoring. His expertise includes development of hardware and software for embedded systems, PCB layout routing, production management and final testing. He is experienced with ISO9001, ISO13485, MDR and other quality management procedures and certification. Tsvetan wrote his PhD dissertation on “HV generators for external defibrillators.”

Hardware Design and Development

Konsulko Group offers complete hardware services including design, integration, validation and certification, rapid prototyping, biometric signals, sensors, wireless communications, low power applications and manufacturing test/support. Please visit our Hardware Design Services page or contact us for more information.

___

IMA-measurement with Yocto Project and OpenEmbedded

Integrity Measurement Architecture (IMA-measurement) is a subsystem in the Linux kernel designed to provide a framework for maintaining the integrity of files on a system. The primary purpose of IMA is to ensure that only trusted code and data are executed on a system and that the integrity of critical system components is maintained.

IMA was merged into the mainline Linux kernel in 2004 with the release of version 2.6.30. It evolved over time, adding features such as TPM integration, extended support, and continued maintenance, becoming a key component for ensuring the integrity of files in Linux-based systems. IMA works by calculating cryptographic hashes of files at various points in their lifecycle, such as when they are accessed, executed, or modified.

This article shares the exacts steps to build a minimal Linux distribution with IMA support for QEMU x86-64 using the Yocto Project and OpenEmbedded. The Yocto Project is an open-source collaboration project that enables developers to create lightweight, optimized, and customizable Linux distributions for embedded devices while maintaining control over the software components and configurations included in the system. To enable IMA, we use Yocto/OpenEmbedded layers meta-security and meta-integrity. These layers offer a comprehensive suite of security tools and hardening utilities designed for Linux kernels, along with libraries that facilitate the implementation of robust security mechanisms.

Building a Linux Distribution with IMA

Recently Leon Anavi, Konsulko Group Senior Engineer, contributed a couple of patches to the upstream of meta-security/meta-integrity to simplify using integrity-image-minimal. This is a small image provided as an example to demonstrate IMA support.

The following steps outline the process of building an image with Integrity Measurement Architecture (IMA) using the Yocto Project and OpenEmbedded. This demonstration uses the default debug keys provided in the “data” directory of layer meta-integrity. Because everyone has access to these private keys, for devices in production you must create your own private keys and use them instead. Enabling the audit kernel subsystem provides additional informational integrity auditing messages useful for debugging any appraisal issues that may arise during the development process.

Kindly be aware that this article utilizes source code derived from the primary branches of associated Yocto/OE layers. Consequently, we are selecting specific git commits that have been confirmed to function reliably. These commits will be part of the next long-term support (LTS) release of the Yocto Project which is version 5.0 with codename Scarthgap. It is scheduled for release in April 2024 and will be supported for 4 years until April 2028.

  • Download the source code:
git clone git://git.yoctoproject.org/poky poky-qemu
cd poky-qemu
git checkout e31be0b0e6ed6855787ebfbacc15bdbf1b9e511c
git clone git://git.yoctoproject.org/meta-security
cd meta-security
git checkout 30e755c59204cbd64c3aa12e64ab33041f6f02c0q
git clone git://git.openembedded.org/meta-openembedded
cd meta-openembedded
git checkout 57db42741df341718b9bb4b1b8f20d2e2fcf7010
  • Initialize the built envieronment:
source oe-init-build-env
  • Include additional layers:
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-security
bitbake-layers add-layer ../meta-security/meta-integrity
  • Append the following configurations to conf/local.conf:
DISTRO_FEATURES:append = " integrity ima"

IMAGE_CLASSES += "ima-evm-rootfs"

IMA_EVM_KEY_DIR = "${INTEGRITY_BASE}/data/debug-keys"
IMA_EVM_PRIVKEY = "${IMA_EVM_KEY_DIR}/privkey_ima.pem"
IMA_EVM_X509 = "${IMA_EVM_KEY_DIR}/x509_ima.der"
IMA_EVM_ROOT_CA = "${IMA_EVM_KEY_DIR}/ima-local-ca.pem"

IMA_EVM_POLICY = "${INTEGRITY_BASE}/recipes-security/ima_policy_hashed/files/ima_policy_hashed"

SRC_URI:append:pn-linux-yocto = " file://audit.cfg"
CORE_IMAGE_EXTRA_INSTALL += "auditd"

QB_KERNEL_CMDLINE_APPEND:remove:pn-integrity-image-minimal = "ima_policy=tcb ima_appraise=fix"
QB_KERNEL_CMDLINE_APPEND:append:pn-integrity-image-minimal = " ima_appraise=log ima_appraise_tcb integrity_audit=1"
  • Built an image with IMA for QEMU x86-64:
bitbake integrity-image-minimal

Testing IMA

After building the image, we can launch it. QEMU, short for Quick Emulator, is an open-source virtualization software that allows users to emulate various hardware platforms and run operating systems on different host systems. We will use it to run and test the image. By utilizing the “nographic” option, QEMU disables the video console, setting the console to “ttys0”. This feature is particularly beneficial when remotely accessing a build server over SSH. To verify the effectiveness of the appraisal process, attempt modifying a file, then confirm that the verification of the altered file subsequently fails.

  • Launch the image in QEMU:
runqemu nographic
  • Login and root and verify the integrity of file /etc/hostname using evmctl:
evmctl ima_verify /etc/hostname

The expected output should resemble:

Poky (Yocto Project Reference Distro) 4.3+snapshot-e31be0b0e6ed6855787ebfbacc15bdbf1b9e511c qemux86-64 /dev/ttyS0

qemux86-64 login: root
root@qemux86-64:~# evmctl ima_verify /etc/hostname
key 1: 6730eefd /etc/keys/x509_evm.der
/etc/hostname: verification is OK
  • Modify /etc/hostname:
echo test > /etc/hostname
  • Verify the integrity of file /etc/hostname again:
evmctl ima_verify /etc/hostname

Now the verification fails because the file has been modified. The anticipated output should be similar to:

root@qemux86-64:~# echo test > /etc/hostname
root@qemux86-64:~# evmctl ima_verify /etc/hostname
key 1: 6730eefd /etc/keys/x509_evm.der
/etc/hostname: verification failed: 0 ((null))

This simple example serves as a demonstration of how Linux IMA operates, using QEMU as a platform. However, to implement Linux IMA on real-world devices, Konsulko Group offers assistance with hardware bring-up and integration of the suitable Yocto/OE BSP (Board Support Package) layers.

Since the inception of OpenEmbedded and the Yocto Project, Konsulko engineers have actively contributed to the community and provided guidance for developing commercial products. We specialize in U-Boot, Linux kernel, RAUC, Mender, and various other open source projects for embedded Linux devices. Contact us to explore how Konsulko engineers can assist with your embedded product development endeavors.

Integrating RAUC with Yocto Project on BeagleBone Black

Konsulko Group has made many upstream contributions to OTA (over-the-air) update solutions for embedded Linux devices. RAUC is a popular open source option as it has been meticulously developed with a keen emphasis on stability, security, and adaptability. Notably, RAUC seamlessly integrates with major build systems such as Yocto Project/OpenEmbedded, Buildroot, and PTXdist.

Functioning across diverse usage scenarios, one of RAUC’s elementary yet impactful functionalities is the A/B update mechanism. In this setup, two identical root filesystems, denoted as A and B, are maintained. The device boots from one of these, while the other serves as the target for updates.

Post-update completion, the bootloader directs the system to boot from the freshly updated partition during the subsequent system startup. RAUC incorporates the ‘verity’ update bundle format. It extends the capabilities of RAUC by introducing built-in support for HTTP(S) network streaming, adaptive delta-like updates, and comprehensive update bundle encryption.

In previous blog posts, Konsulko Group engineers have demonstrated RAUC on Raspberry Pi and NXP devices such as SolidRun Cubox-i and HummingBoard. Recently Leon Anavi, Konsulko Group Senior Engineer and maintainer of meta-rauc-community ported RAUC to BeagleBone Black.

This article provides, as an example, the exact steps how to integrate RAUC with Yocto Project and OpenEmbedded for booting from a microSD card on BeagleBone Black.

Released in 2013, BeagleBone Black is a single-board computer (SBC) developed by the BeagleBoard.org Foundation. It was certified by the Open Source Hardware Association with OSHWA UID US000236. The chipset on BeagleBone Black is Texas Instruments Sitara AM3358 with 1GHz ARM Cortex-A8 CPU and SGX 3D graphics engine. Because of this the demonstrated integration is a suitable reference for other embedded devices equipped Texas Instruments chipsets.

Required Hardware

The hardware used for this step by step tutorial is:

Building a Linux Distribution with RAUC

RAUC, a robust and powerful open-source solution, demands advanced skills for initial integration. In general, to incorporate RAUC in a Yocto Project and OpenEmbedded image for BeagleBone Black the following actions have to be performed:

  • Use U-Boot as a bootloader
  • Enable SquashFS in the Linux kernel configuration
  • Use ext4 root file system
  • Design specific storage partitioning for the certain use case and configure RAUC accordingly
  • Provide a custom U-Boot script to properly switch between RAUC slots
  • Prepare a certificate and keyring to use for signing and verifying RAUC update bundles.

Leon Anavi has already done all these actions for core-image-minimal in Yocto/OpenEmbedded layer meta-rauc-community/meta-rauc-beaglebone. The layer is available at GitHub. Please follow the steps below to build core-image-minimal for BeagleBone Black with it:

  • Download the long term support (LTS) release Kirkstone reference Yocto distribution, Poky:
git clone -b kirkstone https://git.yoctoproject.org/poky poky-rauc-bbb
cd poky-rauc-bbb
  • Download the meta-rauc layer:
git clone -b kirkstone https://github.com/rauc/meta-rauc.git
  • Download meta-rauc-community layers, including meta-rauc-beaglebone:
git clone -b kirkstone https://github.com/rauc/meta-rauc-community.git
  • Download the meta-openembedded layer as it provides a recipe for nano which will be used for the demonstration:
git clone -b kirkstone git://git.openembedded.org/meta-openembedded

Initialize the build environment:

source oe-init-build-env
  • Include all layers in conf/bblayers.conf:
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-beaglebone
  • Adjust conf/local.conf by appending the following configurations to the end of the file:
MACHINE = "beaglebone-yocto"

# Use systemd as init manager
INIT_MANAGER = "systemd"

# Add RAUC to the image
IMAGE_INSTALL:append = " rauc"
DISTRO_FEATURES:append = " rauc"
  • Build a minimal bootable image:
bitbake core-image-minimal

The image creation process from scratch is time-consuming, encompassing various Yocto/OpenEmbedded recipes and configurations. Kindly await completion as bitbake diligently executes each tasks.

  • Flash tmp/deploy/images/beaglebone-yocto/core-image-minimal-beaglebone-yocto.wic.xz to microSD card.
  • Attach the USB-to-UART adapter to BeagleBone Black, plug the ethernet cable and the microSD card.

Press and hold button S2 while plugging in the 5V DC power supply to turn on BeagleBone Black and boot from microSD card.

BeagleBone black board has an onboard button labeled as S2. It is situated near the microSD card slot. Press and hold it while powering the board to boot from microSD card.

  • Verify that the system boots successfully, log in as user root without a password and check RAUC status:
rauc status

On the screenshot BeagleBone Black has been booted from RAUC slot rootfs.0 (A) on the microSD card.

NOTE: The meta-rauc-beaglebone layer includes a core-image-minimal.bbappend file, housing essential configurations for RAUC functionality. Apply these configurations similarly to other images intended for use in your embedded Linux device.

Creating a RAUC Update Bundle

RAUC update bundle comprises the file system image(s) or archive(s) designated for system installation, accompanied by a manifest detailing the images for installation, encompassing options and meta-information. Additionally, it may include scripts designated for execution before, during or after the installation process. To sign and verify the update bundles RAUC uses SSL keys. Layer meta-rauc-beaglebone contains a keyring containing all keys and a recipe for a simple RAUC update bundle for demonstration purposes only.

Follow the steps below to create RAUC update bundle that extends the system by adding the popular text based editor nano:

  • Add to conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build the RAUC update bundle:
bitbake update-bundle

Following a successful execution, bitbake will produce the update-bundle-beaglebone-yocto.raucb file.

Updating BeagleBone Black with RAUC

Follow the steps below to update core-image-minimal running from a microSD card on BeagleBone Black:

  • Start a Python 3 web server on the build machine or another computer where the RAUC update bundle (aka update-bundle-beaglebone-yocto.raucb) is available and within the same network as BeagleBone Black, for example:
cd tmp/deploy/images/beaglebone-yocto/
pip3 install --user rangehttpserver
python3 -m RangeHTTPServer
  • On BeagleBone Black replace <IP> with the actual IP address of the computer on which the HTTP server is running and execute the following command to install the update:
rauc install http://<IP>:8000/update-bundle-beaglebone-yocto.raucb

The screenshot show successful installation of the RAUC updated bundle on BeagleBone Black.

  • Reboot BeagleBone Black to load the updated version:
reboot

NOTE: As alternative, instead of using an HTTP server, you can transfer the update bundle to BeagleBone Black and install it from local storage.

  • Verify that nano was added to the system:
which nano
  • Check RAUC status to confirm the system have booted from the second partition:
rauc status

On the screenshot, after sucessful installation of the RAUC update bundle, BeagleBone Black has been booted from RAUC slot rootfs.1 (B) on the microSD card. This slot contains nano.

In real-world product development, the Yocto Project and OpenEmbedded workflow can be enhanced with a few commands for easy implementation of continuous integration (CI).

From the dawn of OpenEmbedded and the Yocto Project, Konsulko engineers have been community contributors and guides for crafting commercial products. Our expertise spans RAUC, Mender, and various open-source solutions for top-notch software updates. Please get in touch with us to discuss how Konsulko engineers can help your own embedded product development.

What to do when your commercial Linux goes away

How do you move forward with your software development when the commercial embedded Linux you’ve used to build your products is no longer available? Customers often come to Konsulko Group for help, particularly when a commercial Linux is phased-out or end-of-lifed.

Konsulko is in a unique position as many of our engineers have 25+ years experience with embedded Linux, and some were instrumental in building the now-discontinued Linux products we are helping our customers replace. We know well what needs to be done (and what doesn’t) and where to look for potential problems along the way. 

We take a three-step approach for our customers. 

  • First, we rebuild their software stack outside the dependencies of the commercial distribution. 
  • This is not a simple task, but when it is done, we have a “clean” software stack that we can update with the latest open source components. 
  • Finally, we systematically address security, OTA and other customizations required by our customer.

The result is customer-specific, fully maintainable embedded Linux that is free of the dependancies of the marketplace. If you would like some help moving forward with your current and future embedded Linux needs, please contact us.

Konsulko Group: The Year in Review 2023

Konsulko Group has had another great year. We’ve helped our customers build new breakthrough embedded products of all sizes, from semiconductors to medical devices to automotive to heavy equipment.

We continue strong relationships with the Linux Foundation, Yocto Project and Automotive Grade Linux. We work with our partners mender.io and PHYTEC, providing support and development for their customers.

Konsulko is growing

We’ve expanded our footprint across the US and Europe, welcoming three outstanding engineers to the team: George McCollister, Darko Alavanja and Bryan Cisneros.

George McCollister has over 25 years of experience in Embedded Systems development. Starting with 8051 microcontrollers and quickly adopting Linux, he has worked on a wide range of technologies from network switches and storage appliances to automated utility fault restoration and process automation. He was a key designer and architect of an industry leading utility automation platform.

Darko Alavanja was deeply involved with robotics as a student, competing in several teams in the Eurobot contest. He designed mechanical components, PCBs, sensor electronics, actuator systems and software used for creating mobile robots. Darko has developed embedded systems for industrial devices such as FPGA-based hardware-in-the-loop equipment, controllers for industrial machinery and protocols for communication gateways.

Bryan Cisneros has developed embedded software, UIs, and test code across various industries, including medical devices, RF modules, and AI-enabled cameras. Before joining Konsulko, Bryan worked in the defense industry developing networked applications for weapons and information systems, focusing on redesigning outdated UIs and programs, and implementing CI/CD pipelines.

Committed to the Open Source community

In addition to our consulting work for our customers, Konsulko Group continues to actively participate in the Open Source community and its conferences around the world.

Konsulko’s senior leadership have been contributors in the Linux kernel and other OSS communities since the late 1990s. The entire Konsulko team has been involved in a number of Open Source projects including U-Boot, Yocto Project, OpenEmbedded and Automotive Grade Linux (AGL).

Konsulko principal engineer Tim Orling serves on OpenEmbedded Board of Directors. He co-presented Maintaining a Community BSP Layer: Updating Meta-Tegra with Ilies Chergui (Medtronic) at Embedded Open Source Summit in June 2023, and Customize your CROPS containers with crops-generator with Eilís ‘pidge’ Ní Fhlannagáin (BayLibre) at Yocto Project Developer Day.

Principal engineer Denys Dmytriyenko and the Yocto Project Technical Steering Committee were instrumental in helping Yocto Project secure important new funding from the Sovereign Tech Fund. Denys also wrote about some of the technical highlights from the 2023 Linux Plumbers Conference.

Principal engineer Scott Murray presented Vehicle Signaling Specification and KUKSA.val at Automotive Grade Linux All Member Meeting Berlin, a “lightning talk” on VSS Updates in AGL at Automotive Linux Summit, Evolving VSS Usage in AGL at AGL AMM Japan, and Automotive Grade Linux: Status and Roadmap at Embedded Recipes Paris.

Senior engineer Leon Anavi spoke about RDP with Wayland, Weston & Yocto at FOSDEM, and Integrating VNC/Weston with the Yocto Project/OpenEmbedded at Yocto Project Virtual Summit 2023.

Vitaly Wool, principal engineer and General Manager, Konsulko AB presented Implementing secure boot for AOSP running U-Boot at the Lund Linux Conference 2023.

Finally, Konsulko Group intern Atanas Bunchev demonstrated remote updates and troubleshooting of connected embedded Linux devices using Mender.io at TuxCon 2023. The presentation (in Bulgarian) spread the word about the Yocto Project, OpenEmbedded and various Mender features among the local community. Atanas also co-wrote (with Leon Anavi) RAUC on CuBox-I/HummingBoard for Software Updates and Mender Add-ons: Remote Troubleshooting Devices in the Field.

Tag Archive for: konsulko group