Software Updates on i.MX8MP, Part 2: Mender & Yocto Project

Software over-the-air (OTA) updates are essential for any modern embedded Linux device. In part 1, we explored A/B software updates using RAUC and qbee.io. For our demonstrations, we utilized the new open source hardware Olimex’s iMX8MPlus System on Module (SOM) and Evaluation Board (EVB). The NXP i.MX8MP is a robust microprocessor, ideal for industrial-grade applications and widely adopted across various industries. In part 2, we will use Mender to update the same hardware.

About Mender

Mender.io is an open-source platform designed for managing and deploying over-the-air (OTA) updates to embedded devices. It provides a reliable and secure method to keep devices up-to-date, minimizing downtime and reducing the risks of manual updates. Mender supports an open-source A/B update scheme and offers an optional proprietary implementation for delta updates.

As a turnkey solution, Mender features a web interface for comprehensive device management and is available as a Software-as-a-Service (SaaS) for small and medium businesses, as well as hosted or on-premise solutions for large enterprises. It supports robust update strategies, rollback mechanisms, and add-ons for configuring, monitoring, and troubleshooting devices. Mender is a state-of-the-art solution for maintaining and managing fleets of connected devices across various industries.

Building an Image with Mender

Here are the steps to build core-image-base with Mender for Olimex:

  • Install the kas tool (optional: to install it globally for all users, run the installation as root or using sudo):
pip install kas
  • Clone meta-mender-community git repository for Yocto LTS release 5.0 (scarthgap):
git clone -b scarthgap https://github.com/mendersoftware/meta-mender-community
  • Create a build directory and navigate into it:
mkdir -p meta-mender-community/mender-nxp && cd meta-mender-community/mender-nxp
  • Create a kas configuration add-on to enable passwordless root access for development purposes:
cat <<EOF > debug-image.yml
header:
  version: 14

local_conf_header:
  developer-features: |
    EXTRA_IMAGE_FEATURES = "debug-tweaks"
EOF
  • Run the following command to start the build process:
kas build ../kas/olimex-imx8mp-evb.yml:debug-image.yml

Initiating the build process from scratch is a bit of a marathon, as kas and bitbake need to download all the source code and execute a plethora of tasks. Feel free to grab a cup of tea (or maybe a whole teapot) while you wait!

Flashing the Mender Image

Using kas and BitBake will result in the production of an image file. After the build process is complete, you will find the generated image at the following relative path: build/tmp/deploy/images/olimex-imx8mp-evb/core-image-base-olimex-imx8mp-evb.sdimg. This path indicates the location within the build directory where the final image is stored, ready for deployment to Olimex iMX8MP-SOM-4GB-IND and iMX8MP-SOM-EVB-IND.

The core-image-base-olimex-imx8mp-evb.sdimg file must be flashed onto a microSD card to be used with your device. This can be accomplished using various applications such as dd or bmaptool for command-line options. If you prefer a user-friendly application with a graphical interface, you can use Balena Etcher, which simplifies the flashing process and provides a straightforward GUI.

To set up and verify your Olimex iMX8MP-SOM-EVB-IND board, follow these steps:

  • Connect the USB-to-UART adapter to the A53_DBG1 connector on the Olimex iMX8MP-SOM-EVB-IND, and insert the Ethernet cable and the microSD card.

  • Plug the 5V power supply into the power jack of the iMX8MP-SOM-EVB-IND to power up the board.
  • Ensure the system boots successfully, then log in as the root user (no password required) and check the Mender status by executing the following commands:
mender-update show-provides
mender-update show-artifact
mender-update --version
changes

Creating a Mender Artifact

Mender artifact refers to a package format used by the Mender update manager for over-the-air (OTA) software updates. It contains all the necessary components such as firmware, scripts, configuration files, and metadata required to update a device’s software reliably and securely. The layer meta-mender/meta-mender-core is essential for integrating Mender’s functionality into the Yocto Project and OpenEmbedded build system as it provides required classes and scripts that automate the creation of Mender artifacts as part of the build process.

Follow the steps below to build a Mender Artifact for iMX8MP-SOM-EVB-IND that extends the system with the addition of the simple text editor nano:

  • Create a kas configuration add-on to add nano:
cat <<EOF > update-image.yml
header:
  version: 14

local_conf_header:
  update-image: |
    IMAGE_INSTALL:append = " nano"
EOF
  • Build both core-image-base and its corresponding Mender Artifact using:
kas build ../kas/olimex-imx8mp-evb.yml:debug-image.yml:update-image.yml
  • This process will generate a Mender Artifact containing nano at the file path build/tmp/deploy/images/olimex-imx8mp-evb/core-image-base-olimex-imx8mp-evb.mender.

Installing the Mender Artifact

Mender serves as a comprehensive update solution, featuring a central server that acts as a hub for storing and orchestrating software updates across fleets of devices via over-the-air deployment. Using Mender’s intuitive web UI or REST APIs, you can efficiently manage devices, upload software releases, and execute seamless deployments to distribute updates across your devices. Alternatively, Mender can operate in standalone mode, independent of a server.

To perform a manual standalone deployment using Mender in the terminal, follow the steps below. In this setup, no Mender Server is involved, and updates are initiated directly on the device.

  • Start a simple HTTP server in the directory with the Mender Artifact:
python3 -m http.server
mender-update install http://<server>:8000/core-image-base-olimex-imx8mp-evb.mender

NOTE: Replace <server> with the IP address of the machine on which the Python3 HTTP server is running.

changes

  • Reboot the embedded Linux device:
reboot
  • Login as root on the board and verify that nano text editor has been installed.
  • Ensure the new deployment becomes permanent:
mender-update commit
changes

This example illustrates the seamless integration of Mender using the Yocto Project release Scarthgap on an embedded computer powered by the NXP i.MX8MP SoC. It demonstrates how you can effectively manage updates across fleets of devices using the Mender server. Furthermore, Mender also provides additional tools for remote troubleshooting, ensuring smooth operations in the field.

How do Mender.io and RAUC differ?

The Mender client is an application that runs on embedded devices. In a production setup, it connects to the Mender server to perform automatic updates by downloading and installing Mender Artifacts as they become available. Initially, the Mender client was developed in Go. However, a strategic decision was made to rewrite it in C++ to reduce the application’s footprint and support more platforms, including real-time operating systems (RTOS).

In comparison, RAUC, the alternative A/B open-source solution explored in part one, also has an application running on the embedded Linux device, but it is written in C. Unlike Mender, RAUC does not provide a server to manage devices, so a third-party solution such as qbee.io or Eclipse hawkBit is required.

Another notable technical difference is that RAUC integration through meta-rauc-nxp relies on a wks file where the A and B partitions are explicitly specified. In contrast, meta-mender-nxp uses classes and special variables provided by Mender to define those partitions, and BitBake generates a temporary wks file while building Mender-enabled images.

In terms of security, both Mender and RAUC support signing and verification of updates. Mender supports signing artifacts using RSA with a recommended key length of at least 3072 bits or ECDSA with curve P-256. RAUC employs X.509 cryptography for signing and verifying update bundles.

Leveraging the insights and experiences discussed in parts 1 and 2 of this article, here is a side-by-side comparison of the key features of Mender and RAUC:

FeatureMenderRAUC
A/B updatesYesYes
Roll-backYesYes
Configure add-onAvailableNo
Monitor add-onAvailableNo
Troubleshoot add-onAvailableNo
Client implementationC++C
Client licenseApache 2.0LGPL-2.1
Yocto Project integrationYesYes
Management serverYes3rd-party

Mender.io provides a comprehensive, turnkey solution that covers everything from embedded devices to cloud-based software-as-a-service for managing fleets of connected devices. It also offers convenient add-ons and proprietary delta updates. Meanwhile, RAUC reliably integrates seamlessly with both in-house and third-party device management systems. The choice of update technology should be based on your specific requirements and use cases.

About Konsulko Group

Over the years, Konsulko engineers have made significant contributions to the community and crucial embedded Linux open-source projects, including the Yocto Project, OpenEmbedded, the Linux kernel, and U-Boot. We specialize in assisting customers in developing commercial products leveraging these technologies. With expertise in BSP bring-up on diverse hardware platforms for embedded devices, our services encompass a wide range of open-source solutions for software updates. Contact us to discuss the best update strategy and technology for your embedded product requirements.

Software Updates on the i.MX8MP, Part 1: qbee and RAUC

Software over-the-air (OTA) updates are essential for embedded Linux devices as they ensure timely application of security patches, fix vulnerabilities, and enhance security features to protect against threats. They offer a convenient way to deliver new features, performance improvements, and bug fixes without needing physical access to the device, keeping it up-to-date and functional.

Several high-quality open-source OTA solutions are available today, allowing developers to leverage and customize existing systems rather than creating proprietary ones, saving both time and money. In this series of articles, we will explore and compare software update strategies using the A/B scheme, where two identical copies of the root filesystem are maintained, one active and one for the next update. Popular OTA update solutions like MenderRAUC, and Swupdate implement this approach. This article will focus on RAUC using a typical development setup. In part 2 we will implement software updates with Mender. For our demonstration, we will use the i.MX8MP, a versatile microprocessor from NXP Semiconductors known for its industrial-grade reliability and popularity in smart home devices, industrial automation, medical equipment, and multimedia systems.

Recently, Olimex launched an open-source hardware iMX8MPlus System on Module (SOM) and Evaluation Board (EVB) tailored for industrial applications. Leon Anavi, Senior Engineer at Konsulko Group, contributed support for Olimex iMX8MP-SOM-4GB-IND and iMX8MP-SOM-EVB-IND to the community-maintained Yocto and OpenEmbedded BSP layers. This effort encompassed Linux kernel and U-Boot uplift. Subsequently, leveraging his role as founder and maintainer of the meta-rauc-community layer, Leon integrated support for RAUC software updates on these boards and seamlessly integrated them with qbee cloud service.

Qbee.io is a comprehensive cloud platform for managing and maintaining IoT and edge devices. Using the qbee-agent running on the embedded devices, it offers features such as configuration management, remote accesss, monitorning, security and OTA software updates based on RAUC. These capabilities enable businesses to efficiently oversee their distributed technology infrastructure, ensuring devices remain up-to-date, secure, and perform optimally. Earlier in 2024 Tim Orling, Konsulko Group Principal Software Engineer, implemented image update with qbee and RAUC on Raspberry Pi 5.

This technical article will guide you through the exacts steps to build a basic image for Olimex iMX8MP-SOM-4GB-IND and iMX8MP-SOM-EVB-IND using Yocto Project release 5.0 LTS (scarthgap) as well as to perform a software update using qbee and RAUC.

Building an Image

  • Download the long term support (LTS) release Scarthgap reference Yocto distribution, Poky:
git clone -b scarthgap https://git.yoctoproject.org/poky poky-olimex-imx8mp
cd poky-olimex-imx8mp
  • Download BSP layers:
git clone -b scarthgap https://github.com/Freescale/meta-freescale.git
git clone -b scarthgap https://github.com/Freescale/meta-freescale-3rdparty.git
git clone -b scarthgap https://github.com/Freescale/meta-freescale-distro.git
  • Download the meta-rauc layer:
git clone -b scarthgap https://github.com/rauc/meta-rauc.git
  • Download meta-rauc-community layers, including meta-rauc-nxp:
git clone -b scarthgap https://github.com/rauc/meta-rauc-community.git
  • Download layer providing the qbee-agent and qbee.io integration:
git clone -b master https://github.com/qbee-io/meta-qbee.git
  • Download the meta-openembedded layer as it provides a recipe for nano which will be used for the demonstration:
git clone -b scarthgap git://git.openembedded.org/meta-openembedded
  • Initialize the build environment:
source oe-init-build-env
  • Include all layers in conf/bblayers.conf:
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-freescale
bitbake-layers add-layer ../meta-freescale-3rdparty
bitbake-layers add-layer ../meta-freescale-distro
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-nxp
bitbake-layers add-layer ../meta-qbee/meta-qbee
  • Adjust conf/local.conf by appending the following configurations to the end of the file:
MACHINE = "olimex-imx8mp-evb"

INIT_MANAGER = "systemd"

ACCEPT_FSL_EULA = "1"

WKS_FILE = "dual-imx-boot-bootpart.wks.in"
DISTRO_FEATURES:append = " rauc"
IMAGE_FSTYPES:append = " ext4"
IMAGE_BOOT_FILES:append = " boot.scr"

IMAGE_INSTALL:append = " rauc-grow-data-part"
  • Visit qbee.io, register and sign in
  • Click on your profile name at the top right corner and select Bootstrap keys.
  • Copy the key.
  • Replace <bootstrap_key> with the qbee bootstrap key and append to conf/local.conf:
QBEE_BOOTSTRAP_KEY = "<bootstrap_key>"
  • Build an image:
bitbake core-image-base

Creating an image from the ground up is a time-consuming process that requires numerous Yocto/OpenEmbedded recipes and configurations. Please be patient as bitbake systematically manages each step.

  • Flash tmp/deploy/images/olimex-imx8mp-evb/core-image-base-olimex-imx8mp-evb.rootfs.wic.gz to microSD card.
  • Attach the USB-to-UART adapter to connector A53_DBG1 Olimex iMX8MP-SOM-EVB-IND, plug the ethernet cable and the microSD card.
  • Plug 5V power supply to the power jack on iMX8MP-SOM-EVB-IND to turn on the board.
  • Verify that the system boots successfully, log in as user root without a password and check RAUC status:
  • Visit qbee.io, click Devices and verify that olimex-imx8mp-evb has successfully connected:

Creating a RAUC Update Bundle

A RAUC update bundle comprises the file system image(s) or archive(s) designated for system installation, accompanied by a manifest detailing the images for installation, encompassing options and meta-information. Additionally, it may include scripts designated for execution before, during or after the installation process. To sign and verify the update bundles RAUC uses SSL keys. Layer meta-rauc-beaglebone contains a keyring containing all keys and a recipe for a simple RAUC update bundle for demonstration purposes only.

Follow the steps below to create RAUC update bundle that extends the system by adding the popular text based editor nano:

  • Add to conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build the RAUC update bundle:
bitbake update-bundle

Running RAUC Update from qbee.io

Please follow the steps below to upload the RAUC bundle to qbee.io and update the board:

  • Visit qbee.io, click File Manager and upload the RAUC bundle:
  • Select Devices, click on olimex-imx8mp-evb and go to tab Configure. From Settings > OTA enable RAUC image updates:
  • Select the RAUC bundle.
  • Click Save Changes, then click Commit Changes and enter a commit message:
  • Wait for the qbee agent to apply the RAUC update bundle. By default, the agent checks for changes every 5 minutes. To force an immediate check, click the Run agent button in the Device Overview. Allow a few minutes for the update bundle to be transferred and installed on the board.
  • After the update is finished, the board will automatically restart. You can verify that the active RAUC rootfs slot has been updated and nano is present:
  • To optionally verify the update of the embedded Linux device from the cloud service, select Devices, click on olimex-imx8mp-evb, navigate to the Logs tab, and review the logs related to the RAUC update.

With qbee.io, multiple IoT devices can be grouped together and managed as a fleet from the cloud service. In practical product development scenarios, enhancing the Yocto Project and OpenEmbedded workflow can be achieved through a few straightforward commands to streamline continuous integration (CI).

The second part of the article will detail the exact steps to build, boot, and update an image on the Olimex iMX8MP-SOM-4GB-IND and iMX8MP-SOM-EVB-IND hardware, this time using Mender instead of RAUC, with the Yocto Project release 5.0 LTS (scarthgap).

Konsulko engineers have played pivotal roles as contributors and mentors in the commercial product space from the early days of OpenEmbedded and the Yocto Project. Our team excels in utilizing RAUC, Mender, Swupdate and a range of other open-source tools to deliver comprehensive software update solutionsContact us to discuss your specific product requirements and discover how Konsulko engineers can improve your embedded Linux development projects.

Konsulko engineer speaks at AGL All Member Meeting in Berlin

Scott Murray, Principal Software Engineer at Konsulko Group presented at the summer Automotive Grade Linux AMM. Scott shared details around the work done the past few years to prepare AGL before it upgraded to the new Yocto Project long-term support release, Scarthgap, earlier this year.