Tag Archive for: mender.io

Software Updates on i.MX8MP, Part 2: Mender & Yocto Project

/in , , /by

Software over-the-air (OTA) updates are essential for any modern embedded Linux device. In part 1, we explored A/B software updates using RAUC and qbee.io. For our demonstrations, we utilized the new open source hardware Olimex’s iMX8MPlus System on Module (SOM) and Evaluation Board (EVB). The NXP i.MX8MP is a robust microprocessor, ideal for industrial-grade applications and widely adopted across various industries. In part 2, we will use Mender to update the same hardware.

About Mender

Mender.io is an open-source platform designed for managing and deploying over-the-air (OTA) updates to embedded devices. It provides a reliable and secure method to keep devices up-to-date, minimizing downtime and reducing the risks of manual updates. Mender supports an open-source A/B update scheme and offers an optional proprietary implementation for delta updates.

As a turnkey solution, Mender features a web interface for comprehensive device management and is available as a Software-as-a-Service (SaaS) for small and medium businesses, as well as hosted or on-premise solutions for large enterprises. It supports robust update strategies, rollback mechanisms, and add-ons for configuring, monitoring, and troubleshooting devices. Mender is a state-of-the-art solution for maintaining and managing fleets of connected devices across various industries.

Building an Image with Mender

Here are the steps to build core-image-base with Mender for Olimex:

  • Install the kas tool (optional: to install it globally for all users, run the installation as root or using sudo):
pip install kas
  • Clone meta-mender-community git repository for Yocto LTS release 5.0 (scarthgap):
git clone -b scarthgap https://github.com/mendersoftware/meta-mender-community
  • Create a build directory and navigate into it:
mkdir -p meta-mender-community/mender-nxp && cd meta-mender-community/mender-nxp
  • Create a kas configuration add-on to enable passwordless root access for development purposes:
cat <<EOF > debug-image.yml
header:
  version: 14

local_conf_header:
  developer-features: |
    EXTRA_IMAGE_FEATURES = "debug-tweaks"
EOF
  • Run the following command to start the build process:
kas build ../kas/olimex-imx8mp-evb.yml:debug-image.yml

Initiating the build process from scratch is a bit of a marathon, as kas and bitbake need to download all the source code and execute a plethora of tasks. Feel free to grab a cup of tea (or maybe a whole teapot) while you wait!

Flashing the Mender Image

Using kas and BitBake will result in the production of an image file. After the build process is complete, you will find the generated image at the following relative path: build/tmp/deploy/images/olimex-imx8mp-evb/core-image-base-olimex-imx8mp-evb.sdimg. This path indicates the location within the build directory where the final image is stored, ready for deployment to Olimex iMX8MP-SOM-4GB-IND and iMX8MP-SOM-EVB-IND.

The core-image-base-olimex-imx8mp-evb.sdimg file must be flashed onto a microSD card to be used with your device. This can be accomplished using various applications such as dd or bmaptool for command-line options. If you prefer a user-friendly application with a graphical interface, you can use Balena Etcher, which simplifies the flashing process and provides a straightforward GUI.

To set up and verify your Olimex iMX8MP-SOM-EVB-IND board, follow these steps:

  • Connect the USB-to-UART adapter to the A53_DBG1 connector on the Olimex iMX8MP-SOM-EVB-IND, and insert the Ethernet cable and the microSD card.

  • Plug the 5V power supply into the power jack of the iMX8MP-SOM-EVB-IND to power up the board.
  • Ensure the system boots successfully, then log in as the root user (no password required) and check the Mender status by executing the following commands:
mender-update show-provides
mender-update show-artifact
mender-update --version

Creating a Mender Artifact

A Mender artifact refers to a package format used by the Mender update manager for over-the-air (OTA) software updates. It contains all the necessary components such as firmware, scripts, configuration files, and metadata required to update a device’s software reliably and securely. The layer meta-mender/meta-mender-core is essential for integrating Mender’s functionality into the Yocto Project and OpenEmbedded build system as it provides required classes and scripts that automate the creation of Mender artifacts as part of the build process.

Follow the steps below to build a Mender Artifact for iMX8MP-SOM-EVB-IND that extends the system with the addition of the simple text editor nano:

  • Create a kas configuration add-on to add nano:
cat <<EOF > update-image.yml
header:
  version: 14

local_conf_header:
  update-image: |
    IMAGE_INSTALL:append = " nano"
EOF
  • Build both core-image-base and its corresponding Mender Artifact using:
kas build ../kas/olimex-imx8mp-evb.yml:debug-image.yml:update-image.yml
  • This process will generate a Mender Artifact containing nano at the file path build/tmp/deploy/images/olimex-imx8mp-evb/core-image-base-olimex-imx8mp-evb.mender.

Installing the Mender Artifact

Mender serves as a comprehensive update solution, featuring a central server that acts as a hub for storing and orchestrating software updates across fleets of devices via over-the-air deployment. Using Mender’s intuitive web UI or REST APIs, you can efficiently manage devices, upload software releases, and execute seamless deployments to distribute updates across your devices. Alternatively, Mender can operate in standalone mode, independent of a server.

To perform a manual standalone deployment using Mender in the terminal, follow the steps below. In this setup, no Mender Server is involved, and updates are initiated directly on the device.

  • Start a simple HTTP server in the directory with the Mender Artifact:
python3 -m http.server

mender-update install http://<server>:8000/core-image-base-olimex-imx8mp-evb.mender

NOTE: Replace <server> with the IP address of the machine on which the Python3 HTTP server is running.

  • Reboot the embedded Linux device:
reboot
  • Login as root on the board and verify that nano text editor has been installed.
  • Ensure the new deployment becomes permanent:
mender-update commit

This example illustrates the seamless integration of Mender using the Yocto Project release Scarthgap on an embedded computer powered by the NXP i.MX8MP SoC. It demonstrates how you can effectively manage updates across fleets of devices using the Mender server. Furthermore, Mender also provides additional tools for remote troubleshooting, ensuring smooth operations in the field.

How do Mender.io and RAUC differ?

The Mender client is an application that runs on embedded devices. In a production setup, it connects to the Mender server to perform automatic updates by downloading and installing Mender Artifacts as they become available. Initially, the Mender client was developed in Go. However, a strategic decision was made to rewrite it in C++ to reduce the application’s footprint and support more platforms, including real-time operating systems (RTOS).

In comparison, RAUC, the alternative A/B open-source solution explored in part one, also has an application running on the embedded Linux device, but it is written in C. Unlike Mender, RAUC does not provide a server to manage devices, so a third-party solution such as qbee.io or Eclipse hawkBit is required.

Another notable technical difference is that RAUC integration through meta-rauc-nxp relies on a wks file where the A and B partitions are explicitly specified. In contrast, meta-mender-nxp uses classes and special variables provided by Mender to define those partitions, and BitBake generates a temporary wks file while building Mender-enabled images.

In terms of security, both Mender and RAUC support signing and verification of updates. Mender supports signing artifacts using RSA with a recommended key length of at least 3072 bits or ECDSA with curve P-256. RAUC employs X.509 cryptography for signing and verifying update bundles.

Leveraging the insights and experiences discussed in parts 1 and 2 of this article, here is a side-by-side comparison of the key features of Mender and RAUC:

FeatureMenderRAUC
A/B updatesYesYes
Roll-backYesYes
Configure add-onAvailableNo
Monitor add-onAvailableNo
Troubleshoot add-onAvailableNo
Client implementationC++C
Client licenseApache 2.0LGPL-2.1
Yocto Project integrationYesYes
Management serverYes3rd-party

Mender.io provides a comprehensive, turnkey solution that covers everything from embedded devices to cloud-based software-as-a-service for managing fleets of connected devices. It also offers convenient add-ons and proprietary delta updates. Meanwhile, RAUC reliably integrates seamlessly with both in-house and third-party device management systems. The choice of update technology should be based on your specific requirements and use cases.

About Konsulko Group

Over the years, Konsulko engineers have made significant contributions to the community and crucial embedded Linux open-source projects, including the Yocto Project, OpenEmbedded, the Linux kernel, and U-Boot. We specialize in assisting customers in developing commercial products leveraging these technologies. With expertise in BSP bring-up on diverse hardware platforms for embedded devices, our services encompass a wide range of open-source solutions for software updates. Contact us to discuss the best update strategy and technology for your embedded product requirements.

Porting Mender to Raspberry Pi 5 and Yocto Project Scarthgap

/in , , /by

Overview

Mender is an open-source over-the-air (OTA) software update technology for embedded Linux devices and Internet of Things (IoT) ecosystem. It allows developers to remotely manage and update software on embedded Linux devices, ensuring that they remain secure, up-to-date, and functional throughout their lifecycle. Mender simplifies the process of launching a new project by offering official and community-supported board integrations for a wide range of devices and operating systems. These integrations include support for Debian family and the Yocto Project, making it easier for developers to get started with their projects seamlessly.

The Yocto Project is an open-source collaboration project by the Linux Foundation to create custom Linux-based systems for embedded devices. It uses the OpenEmbedded build automation framework with the build tool bitbake and provides Poky as a reference Linux distribution. The Yocto Project follows a regular release cycle, typically with a new version every six months, alongside long-term supported releases available every two years. Currently, the most recent LTS release is version 5.0, codenamed Scarthgap, which became available on April 30, 2024.

Recently, Senior Engineer Leon Anavi from Konsulko Group ported Mender to Raspberry Pi 5. This effort builds upon his previous contributions, where he added Raspberry Pi 5 support to the Yocto Project BSP (Board Support Package) layer meta-raspberrypi. This progress was made possible through sponsorship from the computer emergency response team of the Government of the Grand Duchy of Luxembourg (GOVCERT.LU). If you’re considering using Mender on Raspberry Pi or any other hardware platform for your embedded product needs, feel free to reach out to us to discuss further.

Raspberry Pi 5 introduces significant hardware differences compared to its predecessors. To enable Mender updates on this platform, we require U-Boot version v2024.04 or later. Unfortunately, the Yocto Project release Scarphgap ships with U-Boot version v2024.01, which isn’t compatible. To address this compatibility issue and obtain the necessary U-Boot version, our setup utilizes the scarthgap/u-boot branch from the meta-lts-mixins layer. Tim Orling, Principal Software Engineer at Konsulko Group, contributed the patches in meta-lts-mixins for U-Boot v2024.04.

Raspberry Pi 5 is the first model of the famous single board computers that features a dedicated UART connector is a three-pin header compatible with the Raspberry Pi Debug Connector specification. It can be used with Raspberry Pi Debug Probe, a USB device that provides both a UART serial port and a standard Arm Serial Wire Debug (SWD) interface.

The article offers a practical guide, outlining the exact steps to build a Mender-enabled image for Raspberry Pi 5 and execute an A/B update. To streamline the build setup, we’ll utilize KAS. This Python-based open-source tool effectively handles various Yocto/OpenEmbedded layers. KAS executes builds within a Docker container to ensure consistency and reliable build outcomes, regardless of the primary GNU/Linux distribution on the build machine.

Image for Raspberry Pi 5

Follow the steps below to build core-image-minimal with Mender for Raspberry Pi 5:

  • Install the kas tool (optionally, you can install globally for all users. Run as root, respectively under sudo then):
pip install kas
  • Clone this repository:
git clone -b scarthgap https://github.com/mendersoftware/meta-mender-community
  • Create a build directory and change into it:
mkdir -p meta-mender-community/mender-rpi5 && cd meta-mender-community/mender-rpi5
  • Use kas to build for the Raspberry Pi 5:
kas build ../kas/raspberrypi5.yml
  • Flash tmp/deploy/images/raspberrypi5/core-image-minimal-raspberrypi5.sdimg to a microSD card and boot it on Raspberry Pi 5.

Mender Artifact for Raspberry Pi 5

Follow the steps below to build a Mender Artifact for Raspberry Pi 5 that provides the simple text editor nano:

  • Enter KAS shell:
kas shell ../kas/raspberrypi5.yml
  • Append to the end of conf/local.conf:
IMAGE_INSTALL:append = " nano"
  • Build both core-image-minimal and a Mender Artifact for it:
bitbake core-image-minimal
  • As a result Mender Artifact containg nano will be generated as file tmp/deploy/images/raspberrypi5/core-image-minimal-raspberrypi5.mender

Update Raspberry Pi 5

As an end to end update solution, Mender provides aserver as the central hub for storing and orchestrating software updates across fleets of devices through over-the-air deployment. Through Mender’s user-friendly web UI or REST APIs, you can easily oversee device management, upload software releases, and create seamless deployments to distribute updates to your devices. However, it is also possible to use Mender in standalone mode without a server.

Follow the steps below to manually perform a standalone deployment with Mender in the terminal. In this scenario, no Mender Server is utilized, and the deployments are triggered directly at the device.

  • Start a simple HTTP server in the directory with the Mender Artifact:
python3 -m http.server
  • Login as root on Raspberry Pi 5 and install the Mender Artifact to perform an upgrade of the device:
mender-update install http://<server>:8000/core-image-minimal-raspberrypi5.mender

NOTE: Replace <server> with the IP address of the machine on which the Python3 HTTP server is running.

  • Reboot Raspberry Pi 5:
reboot
  • Login as root on Raspberry Pi 5 and verify that nano text editor has been installed.
  • Make the deployment permanent:
mender-update commit

This straightforward example showcases the seamless integration of Mender with Raspberry Pi 5 using the Yocto Project release Scarthgap. You can follow up and manage updates of fleets of Raspberry Pi 5 devices through the Mender server. Additionally, Mender offers convenient add-ons for remote troubleshooting, ensuring smooth operations in the field.

Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been active contributors to the community, aiding customers in crafting commercial products using these technologies. Our expertise extends beyond Mender, encompassing various open-source solutions for software updates. Feel free to get in touch to explore how we can assist with your embedded product requirements.

Konsulko Group: The Year in Review 2023

/in /by

Konsulko Group has had another great year. We’ve helped our customers build new breakthrough embedded products of all sizes, from semiconductors to medical devices to automotive to heavy equipment.

We continue strong relationships with the Linux Foundation, Yocto Project and Automotive Grade Linux. We work with our partners mender.io and PHYTEC, providing support and development for their customers.

Konsulko is growing

We’ve expanded our footprint across the US and Europe, welcoming three outstanding engineers to the team: George McCollister, Darko Alavanja and Bryan Cisneros.

George McCollister has over 25 years of experience in Embedded Systems development. Starting with 8051 microcontrollers and quickly adopting Linux, he has worked on a wide range of technologies from network switches and storage appliances to automated utility fault restoration and process automation. He was a key designer and architect of an industry leading utility automation platform.

Darko Alavanja was deeply involved with robotics as a student, competing in several teams in the Eurobot contest. He designed mechanical components, PCBs, sensor electronics, actuator systems and software used for creating mobile robots. Darko has developed embedded systems for industrial devices such as FPGA-based hardware-in-the-loop equipment, controllers for industrial machinery and protocols for communication gateways.

Bryan Cisneros has developed embedded software, UIs, and test code across various industries, including medical devices, RF modules, and AI-enabled cameras. Before joining Konsulko, Bryan worked in the defense industry developing networked applications for weapons and information systems, focusing on redesigning outdated UIs and programs, and implementing CI/CD pipelines.

Committed to the Open Source community

In addition to our consulting work for our customers, Konsulko Group continues to actively participate in the Open Source community and its conferences around the world.

Konsulko’s senior leadership have been contributors in the Linux kernel and other OSS communities since the late 1990s. The entire Konsulko team has been involved in a number of Open Source projects including U-Boot, Yocto Project, OpenEmbedded and Automotive Grade Linux (AGL).

Konsulko principal engineer Tim Orling serves on OpenEmbedded Board of Directors. He co-presented Maintaining a Community BSP Layer: Updating Meta-Tegra with Ilies Chergui (Medtronic) at Embedded Open Source Summit in June 2023, and Customize your CROPS containers with crops-generator with Eilís ‘pidge’ Ní Fhlannagáin (BayLibre) at Yocto Project Developer Day.

Principal engineer Denys Dmytriyenko and the Yocto Project Technical Steering Committee were instrumental in helping Yocto Project secure important new funding from the Sovereign Tech Fund. Denys also wrote about some of the technical highlights from the 2023 Linux Plumbers Conference.

Principal engineer Scott Murray presented Vehicle Signaling Specification and KUKSA.val at Automotive Grade Linux All Member Meeting Berlin, a “lightning talk” on VSS Updates in AGL at Automotive Linux Summit, Evolving VSS Usage in AGL at AGL AMM Japan, and Automotive Grade Linux: Status and Roadmap at Embedded Recipes Paris.

Senior engineer Leon Anavi spoke about RDP with Wayland, Weston & Yocto at FOSDEM, and Integrating VNC/Weston with the Yocto Project/OpenEmbedded at Yocto Project Virtual Summit 2023.

Vitaly Wool, principal engineer and General Manager, Konsulko AB presented Implementing secure boot for AOSP running U-Boot at the Lund Linux Conference 2023.

Finally, Konsulko Group intern Atanas Bunchev demonstrated remote updates and troubleshooting of connected embedded Linux devices using Mender.io at TuxCon 2023. The presentation (in Bulgarian) spread the word about the Yocto Project, OpenEmbedded and various Mender features among the local community. Atanas also co-wrote (with Leon Anavi) RAUC on CuBox-I/HummingBoard for Software Updates and Mender Add-ons: Remote Troubleshooting Devices in the Field.

Konsulko Group updates Embedded Linux devices at TuxCon 2023

/in /by

Held on June 10th and 11th, 2023 in the beautiful city of Plovdiv (Bulgaria), TuxCon 2023 marked the 10th anniversary of this annual local two-day conference. Organized by dedicated volunteers, the event aims to advocate the adoption of open source hardware and foster the use of free and open source software.

Konsulko Group intern Atanas Bunchev participated as a speaker, and demonstrated remote updates and troubleshooting of connected embedded Linux devices using Mender.io. The presentation was in Bulgarian and spread the word about the Yocto Project, OpenEmbedded and various Mender features among the local community.

Mender is an open-source, over-the-air (OTA) software update management platform specifically designed for embedded devices. It provides an efficient way for software updates and management of fleets of connected devices in various industries, such as IoT (Internet of Things), automotive, industrial automation, and healthcare. It provides options for ensuring the resilience of system-wide updates and also for updating individual applications. Furthermore debugging of devices is possible with the troubleshooting package, such as the File Transfer and Remote Terminal Add-ons.

Konsulko Group, a partner of Northern.tech, often works with Mender.io. If you are developing a new product, we would be thrilled to discuss how Konsulko’s engineering expertise and experience can contribute to your project. Or if you are a Linux software developer who is passionate about open source, we invite you to reach out to us regarding potential opportunities to join the Konsulko team.

Mender Add-ons: Remote Troubleshooting Devices in the Field

/in , , /by

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

Konsulko Group often works with Mender.io to provide secure, risk tolerant and efficient Over-The-Air updates for any software on an embedded device. This includes both solutions for robust system updates as well as updates of single applications. Mender is compatible with any Linux based OS, including popular build systems such as Yocto Project/OpenEmbedded, Buildroot and OpenWrt.

Aside from OTA updates Mender also offers several add-ons. These are optional extensions that provide functionality for use cases beyond the core OTA updates features.

One such use case is remote troubleshooting. In practice deployed devices in the field are often hard to reach or retrieve. Troubleshooting individual devices becomes resource inefficient and in some cases they have to be replaced even when it’s just a small software or configuration issue. Our experience has shown in these situations Mender’s Add-ons have proven to be extremely useful.

This article provides as an example, the exact steps how to integrate Mender with The Yocto Project and OpenEmbedded for SolidRun CuBox-I and HummingBoard as well as a demonstration of the Mender Troubleshooting package, more specifically the File Transfer and Remote Terminal Add-ons. With the File Transfer add-on files can be downloaded and uploaded to any accepted device. Remote Terminal add-on allows remote interactive command execution from the Mender UI.

These add-ons are very valuable for system administration and mantainance of Internet of Things or fleets of connected industrial devices. For an example, we will troubleshoot an embedded Linux device remotely with systemd using Mender add-ons. We will upload tools to gather system boot-up performance statistics, run them on the device thanks to the Remote Terminal and download logs for further debugging.

Any Mender enabled device is suitable for this demonstration. Recently the Konsulko Senior Engineer Leon Anavi ported Mender to SolidRun CuBox-I and HummingBoard with NXP i.MX6 SoC so our current setup is based on this hardware platform and Yocto LTS release Kirkstone. Earlier we also ported RAUC, an alternative free and open source OTA update platform to the same hardware. If you are interested in software updates, please contact us to discuss your own embedded product needs in details. The hardware used in this example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • Generic RJ45 network cable with Internet connection
  • Optionally UART to USB adapter for debugging the setup section

Building a Linux Distribution with Yocto/OpenEmbedded

The Mender Community provides a set of examples for integration with various hardware platforms using Yocto/OpenEmbedded in the meta-mender-community repository. Sub-layer meta-mender-nxp in this repository contains the integration for Cubox-I/HummingBoard.

The meta-mender-nxp layer uses Google Repo to provide easy and simple setup and build process for the examples:

$ mkdir mender-nxp && cd mender-nxp
$ repo init -u https://github.com/mendersoftware/meta-mender-community \
            -m meta-mender-nxp/scripts/manifest-nxp.xml \
            -b kirkstone
$ repo sync
$ source setup-environment nxp

Board configuration

To configure the build system we have to append to conf/local.conf inside the build directory.

First we are going to set the build target machine:

MACHINE = "cubox-i"

Then we have to accept the end user agreement required by the BSP layer:

ACCEPT_FSL_EULA = "1"

Note: Usually to enable Mender’s Troubleshooting features we have to add mender-connect to IMAGE_INSTALL. In our case this is already added by the meta-mender-demo layer.

Mender configuration

We have to provide our device with MENDER_SERVER_URL and MENDER_TENANT_TOKEN.
For that reason we have to register at https://mender.io/.

Mender provides a free demo profile with limitation of 1 year and up to 10 devices which can be used to experiment with all of Mender’s features.

In fact, when sourcing the setup-environment script we get most of the mender-specific configuration appended to local.conf. This includes a description of how to get our tenant token:

# Build for Hosted Mender
#
# To get your tenant token:
#    - log in to https://hosted.mender.io
#    - click your email at the top right and then "My organization"
#    - press the "COPY TO CLIPBOARD"
#    - assign content of clipboard to MENDER_TENANT_TOKEN
#
#MENDER_SERVER_URL = "https://hosted.mender.io"
#MENDER_TENANT_TOKEN = ""

Note: If using the European server one has to set MENDER_SERVER_URL = "https://eu.hosted.mender.io"

Once we assign our tenant token and remove the # in front of MENDER_SERVER_URL and MENDER_TENANT_TOKEN we’re ready to build our system.

Building and flashing the system image to a microSD card

Build an example image with Yocto:

$ bitbake core-image-base

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

Once the build is complete flash the image to the microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-base-cubox-i.sdimg.bz2 /dev/sdX
$ sync
$ eject /dev/sdX

Connecting to Mender

Once the board finishes booting it will poll the Mender server. By design the connection has to be established from the board to the server. Mender does not open any ports on the board to provide better security therefore the device has to initiate the connection.

When the connection is established the Mender control panel will indicate one pending device.

To accept the request click on View details under Pending devices.

Select the new device and press accept in the Authorization request section.

Once the connection is accepted head over to the Troubleshooting tab in the Device information section. Here you can launch a remote terminal and transfer files.

The next part of the article will demonstrate preparing, uploading and using systemd-analyze to fetch data about the boot process.

Preparing troubleshooting software

To compile systemd-analyze we have to add it to our image in conf/local.conf:

IMAGE_INSTALL:append = " systemd-analyze"

Rebuild systemd to get the systemd-analyze binary:

$ bitbake systemd -c compile
When using the do_compile command Yocto/OpenEmbedded will preserve the compiled binaries.

Find the systemd-analyze binary and libsystemd-shared-<version>.so shared library:

$ find tmp/work -name "systemd-analyze"
$ find tmp/work -name "libsystemd-shared*.so"

Note: As of the time of writing of this article these files should reside in locations similar to ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/systemd-analyze and ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/src/shared/libsystemd-shared-250.so. These paths depend on the exact version of systemd as well as the build configuration and may not be correct in your case.

Uploading the troubleshooting software

Upload these files to the board using Mender’s File Transfer utility:

  • systemd-analyze into /usr/bin/
  • libsystemd-shared-<version>.so into /usr/lib/

Fetching service initialization logs

Once the troubleshooting software is uploaded we can use the Remote Terminal to execute it.

Permit execution of the systemd-analyze binary:

# chmod +x /usr/bin/systemd-analyze

Check the time it took for the system to initialize:

# systemd-analyze

Export a graphic of all enabled services and the time they took to initialize:

# systemd-analyze plot > init.svg

Download init.svg:

This graphic shows that the device needs around a minute to reach multi-user.target. The longest task is the resizing of the /data partition that runs on first boot and the second longest is the filesystem check for mmcblk1p1 that runs every time the system boots.

Here is another graphic generated after a reboot:

This article demonstrates how to use Mender’s Remote terminal and File Transfer troubleshooting utilities to upload and execute the systemd-analyze binary to profile the initialization process of systemd services. These troubleshooting utilities can be used for variety of different tasks. After debugging a single device and finding an appropriate fix, Mender is capable of performing an A/B or delta software update to all devices or specific group of devices in the field.

Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product needs.

Konsulko Group to speak at Yocto Project Virtual Summit

/in /by

Konsulko Group’s Leon Anavi will present Porting Mender to New i.MX 8M Plus Machine and Working with Toradex BSP at the Yocto Project Virtual Summit. Leon will focus on porting Mender, a popular open source over-the-air software update solution, to Toradex Verdin iMX8M Plus SoM. He’ll examine the exact steps to build images for it and the Dahlia carrier board using the long-term support release of Yocto Project and the Toradex BSP.

Tim Orling will present two technical “lightning talks.” Running QEMU Images with virtio, virsh and virt-manager will discuss some experiences building and launching QEMU images with tools that leverage libvirt and virtio on a host. Creating an Installer that Leverages bmaptool will describe how to create an installer image using output from another image and writing all the partitions at once to the target using bmaptool.

The Yocto Project Virtual Summit is a 3-day technical conference for engineers, open source technologists, students and academia in the OSS space. Learn about Yocto Projects’ direction, get training on the next wave of embedded Linux technologies and network with industry peers, Yocto Project maintainers, OpenEmbedded maintainers and other experts.

Registration is $40 for the whole event. We hope you are able to join us.

Konsulko Group speaking at Embedded Linux Conference Austin

/in /by

Konsulko Group engineers will make two presentations at the Embedded Linux Conference North America (ELC) in Austin, Texas (and virtual), June 21-24, 2022.

Software Update Mechanisms

On Wednesday, June 22 at 2:35pm CDT (Texas time), Leon Anavi will present How to Choose a Software Update Mechanism for Embedded Linux Devices.

This talk will look at the advantages and disadvantages of widely-used industry approaches: A/B updates with dual redundant scheme, delta updates, container-based updates and combined strategies. Open source technologies such as Mender, RAUC and libostree-based solutions implement these strategies and provide tools to manage updates of multiple devices. Leon will discuss how to choose an appropriate open source solution to implement for a specific project.

Edge Computing with RISC-V and Linux

On Thursday, June 23 at 2:55pm CDT, Vitaly Vul and Maria Vul will present Edge Computing with RISC-V Platforms Running XIP Linux.

XIP stands for eXecute In Place, allowing code to be executed directly from flash without copying the code to RAM first, making it possible to run Linux on such RISC-V devices as Kendryte K210, which has only 8 MB of SRAM, not only for demonstration purposes but for real applications as well. This talk will cover running edge computing specifically on K210 under Linux with XIP enabled.

Part of Open Source Summit North America

These and many other excellent presentations at ELC are part of the Linux Foundation’s Open Source Summit. We hope you will join us in June.

Building Platforms with Secure Over-the-Air Updating

/in , , /by

Almost every device in development today requires both over-the-air (OTA) updating capabilities, and up-to-date security with authentication of the device filesystem and encryption of application data storage.

At Konsulko Group, we find that the best approach is to build a secure platform first, starting with the hardware root of trust and establishing a chain of trust by extending the root of trust through each successive component in the system.

  • A first step is to make sure the Universal Boot Loader (U-Boot) is up-to-date. If the device is using an older version, we may port U-Boot support of a more current release. Then we enable signature-based authentication of the device using the U-Boot verified boot feature.
  • To provide authentication of the device filesystem and encryption of application data storage, we often use the Linux kernel Device-Mapper infrastructure to create virtual layers of block devices: dm-verity for root filesystem authentication, dm-crypt for data encryption, and dm-integrity for read/write data volume integrity.
  • Finally, we can integrate Mender I/O support for OTA together with dm-verity and Yocto Project so that device updates can be performed while maintaining the system’s secure chain of trust.

This step-by-step methodology ensures straight-forward and predictable development. Please contact us to discuss how we can help you build a secure platform for OTA updating on your device.

CEO Pete Popov looks back on 2020

/in /by

As we approach the end of December, it’s time to review a year that we will certainly remember for the rest of our lives. It hasn’t been easy, but all of us at Konsulko Group are still working hard, supporting the open source community and helping our customers build forward-looking products.

Even at the start of 2020, we knew this year would be different. The coronavirus was the talk of CES in January with some companies pulling out at the last minute, and everyone wondering what the global business climate would be in the months ahead.

By the time FOSDEM rolled around a few weeks later, it was clear the virus would disrupt commerce worldwide, and by the end of February, we had to cut short our presence at Embedded World because of new travel restrictions.

Then the world locked down completely. Since at Konsulko Group we all work remotely by design, we didn’t have to adjust our way of developing software, but as it did for everyone, we had to significantly change our face-to-face participation in embedded Linux, Yocto Project and other community events.

We taught ourselves to use video editing tools, and gave “virtual” presentations from our desks at the Embedded Linux Conference North America and Yocto Project Dev Day at the end of June, the virtual Automotive Grade Linux All Member Meeting in mid July, Linaro Connect in September, Virtual ELCE and Yocto Project Virtual Summit Europe at the end of October, and participated in the virtual Automotive Linux Summit the first of December.

In December, we also presented an AGL Webinar, Getting Started with AGL using Raspberry Pi.

Early in the year, we announced that we had become a Mender Authorized Referral Partner, and that important alliance has provided dividends to both Konsulko Group and our customers as the year progressed.

Konsulko engineers continued our series of technical blogs…

Helping Yocto Project work with Python 3

Getting Started with RAUC on Raspberry Pi

How Mender works

Using Rust with Yocto Project

Building a DIY SOHO router, 18 months later

…and we posted six new videos of our presentations:

Building Containers with OpenEmbedded

Highly Scalable Yocto Project Build Automation

Security Hardening with OpenEmbedded / Yocto Project

Open Source License Compliance with Yocto Project

Demo: Using Rust with Yocto Project

Software Update Solutions for Yocto and OpenEmbedded

Still, we have all been touched by the physical and emotional toll of COVID-19. Two of our engineers have endured a bout with the virus (and thankfully recovered). Some of us have family, friends or acquaintances who have become seriously ill or even passed away. We can only hope the post-pandemic world is now in sight.

In the long run, the challenges that 2020 have brought us closer together in many ways, and hopefully taught us valuable lessons that will make us stronger in a better new year.

How Mender works

/in , /by

by Tom Rini, VP Engineering

Software Update solutions are a key part of our services offering. For open source over-the-air updates, we often we recommend and work with mender.io. In fact, Konsulko Group is a Mender Authorized Referral Partner. Recently, a prospective customer expressed an interest in knowing more about how Mender works. Here’s the brief, informal introduction to member.io that I prepared, and now I’m sharing with you.

—

As a high level starting point, https://mender.io/how-it-works provides a good overview of what’s supported and what it covers. In short, Mender starts off by providing support for a traditional “A/B” approach to system updates, where if the update isn’t marked as valid (and there’s hooks for the application(s) to verify the system before this is done), it’s assumed invalid and the system will roll-back automatically. While “OTA” implies over the network, it can just as easily be done by providing (and validating) a USB key that contains an update.

One of the reasons we recommend Mender is that it has very good in-depth documentation. The starting point for all of that is https://docs.mender.io/2.4/ which covers all of the topic starting from how to implement Mender support in a device and including how to create your own server infrastructure if you don’t want to use their paid service. While there are a number of important pages there, one that I like to highlight is https://docs.mender.io/2.4/artifacts/state-scripts as it shows the state machine for an update and talks about some of the common use cases that come up for user interaction or dealing with failures.

Another place I want to call out is https://docs.mender.io/2.4/devices/update-modules which is also mentioned in the first link. This is how Mender is extended to provide updates for other parts of the system that are not the rootfs itself.  Since updating a Docker container is something that has been mentioned before I want to also note https://hub.mender.io/t/docker/324 as it is a well supported module for this specific case.

I hope you find this information useful. Please contact us if you have specific questions. We’re looking forward to talking to you about your own specific OTA needs.

Tag Archive for: mender.io