Tag Archive for: embedded linux

Konsulko Group a sponsor of Embedded Recipes 2023 in Paris

/in /by

Now in its fifth edition, Embedded Recipes is a small, technically-focused, informal conference about embedded software and hardware, held in Paris, France on September 28 and 29, 2023.

With two days, 15+ presentations and lightning talks, Embedded Recipes takes a deep dive into the developments, contributions, tools and platforms that Open Source developers use to create solutions.

Konsulko Group is proud to be a supporter sponsor of Embedded Recipes 2023. We hope you will join our engineers, together with some of the main contributors of the embedded world from throughout Europe and North America, at the Fondation Biermans Lapotre in Paris for this event.

Konsulko Group updates Embedded Linux devices at TuxCon 2023

/in /by

Held on June 10th and 11th, 2023 in the beautiful city of Plovdiv (Bulgaria), TuxCon 2023 marked the 10th anniversary of this annual local two-day conference. Organized by dedicated volunteers, the event aims to advocate the adoption of open source hardware and foster the use of free and open source software.

Konsulko Group intern Atanas Bunchev participated as a speaker, and demonstrated remote updates and troubleshooting of connected embedded Linux devices using Mender.io. The presentation was in Bulgarian and spread the word about the Yocto Project, OpenEmbedded and various Mender features among the local community.

Mender is an open-source, over-the-air (OTA) software update management platform specifically designed for embedded devices. It provides an efficient way for software updates and management of fleets of connected devices in various industries, such as IoT (Internet of Things), automotive, industrial automation, and healthcare. It provides options for ensuring the resilience of system-wide updates and also for updating individual applications. Furthermore debugging of devices is possible with the troubleshooting package, such as the File Transfer and Remote Terminal Add-ons.

Konsulko Group, a partner of Northern.tech, often works with Mender.io. If you are developing a new product, we would be thrilled to discuss how Konsulko’s engineering expertise and experience can contribute to your project. Or if you are a Linux software developer who is passionate about open source, we invite you to reach out to us regarding potential opportunities to join the Konsulko team.

Konsulko Group to speak at EOSS / ELC / ALS in Prague

/in /by

Konsulko engineers will be giving two talks at Embedded Open Source Summit (EOSS), a new Linux Foundation umbrella event held in Prague, Czech Republic, June 27-30, 2023. Attending virtually is also an option. Incorporated into EOSS are the Embedded Linux Conference (ELC) and the Automotive Linux Summit Europe (ALS).

On Tuesday, June 27, Konsulko Principal Engineer Scott Murray will present a Lightning Talk: Vehicle Signal Specification and KUKSA.val Updates in AGL at ALS.

The Vehicle Signal Specification (VSS) has been included in the AGL UCB by incorporating the KUKSA.val project. Scott will provide an update on the work done for the upcoming Prickly Pike UCB release.

On Thursday, June 29 at ELC, Tim Orling, Konsulko Principal Engineer and Ilies Chergui of Medtronic will speak about Maintaining a Community BSP Layer: Updating Meta-Tegra Through Major Changes.

The meta-tegra BSP layer supports the NVIDIA Jetson family of systems-on-module and has an active OpenEmbedded for Tegra community. Tim and Ilies will discuss changes in the latest SDKs including the bootloader and the trusted OS, and how these affect secure boot and disk encryption implementations. They will also discuss changes needed to support over the air updates and cover how the community keeps up with upstream Open Embedded/Yocto Project changes, as well as the roadmap for support for future Jetson platforms and Yocto Project releases.

Please join Scott, Tim, and a good-sized contingent of our engineers in Prague (or virtually). Register here.

Mender Add-ons: Remote Troubleshooting Devices in the Field

/in , , /by

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

Konsulko Group often works with Mender.io to provide secure, risk tolerant and efficient Over-The-Air updates for any software on an embedded device. This includes both solutions for robust system updates as well as updates of single applications. Mender is compatible with any Linux based OS, including popular build systems such as Yocto Project/OpenEmbedded, Buildroot and OpenWrt.

Aside from OTA updates Mender also offers several add-ons. These are optional extensions that provide functionality for use cases beyond the core OTA updates features.

One such use case is remote troubleshooting. In practice deployed devices in the field are often hard to reach or retrieve. Troubleshooting individual devices becomes resource inefficient and in some cases they have to be replaced even when it’s just a small software or configuration issue. Our experience has shown in these situations Mender’s Add-ons have proven to be extremely useful.

This article provides as an example, the exact steps how to integrate Mender with The Yocto Project and OpenEmbedded for SolidRun CuBox-I and HummingBoard as well as a demonstration of the Mender Troubleshooting package, more specifically the File Transfer and Remote Terminal Add-ons. With the File Transfer add-on files can be downloaded and uploaded to any accepted device. Remote Terminal add-on allows remote interactive command execution from the Mender UI.

These add-ons are very valuable for system administration and mantainance of Internet of Things or fleets of connected industrial devices. For an example, we will troubleshoot an embedded Linux device remotely with systemd using Mender add-ons. We will upload tools to gather system boot-up performance statistics, run them on the device thanks to the Remote Terminal and download logs for further debugging.

Any Mender enabled device is suitable for this demonstration. Recently the Konsulko Senior Engineer Leon Anavi ported Mender to SolidRun CuBox-I and HummingBoard with NXP i.MX6 SoC so our current setup is based on this hardware platform and Yocto LTS release Kirkstone. Earlier we also ported RAUC, an alternative free and open source OTA update platform to the same hardware. If you are interested in software updates, please contact us to discuss your own embedded product needs in details. The hardware used in this example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • Generic RJ45 network cable with Internet connection
  • Optionally UART to USB adapter for debugging the setup section

Building a Linux Distribution with Yocto/OpenEmbedded

The Mender Community provides a set of examples for integration with various hardware platforms using Yocto/OpenEmbedded in the meta-mender-community repository. Sub-layer meta-mender-nxp in this repository contains the integration for Cubox-I/HummingBoard.

The meta-mender-nxp layer uses Google Repo to provide easy and simple setup and build process for the examples:

$ mkdir mender-nxp && cd mender-nxp
$ repo init -u https://github.com/mendersoftware/meta-mender-community \
            -m meta-mender-nxp/scripts/manifest-nxp.xml \
            -b kirkstone
$ repo sync
$ source setup-environment nxp

Board configuration

To configure the build system we have to append to conf/local.conf inside the build directory.

First we are going to set the build target machine:

MACHINE = "cubox-i"

Then we have to accept the end user agreement required by the BSP layer:

ACCEPT_FSL_EULA = "1"

Note: Usually to enable Mender’s Troubleshooting features we have to add mender-connect to IMAGE_INSTALL. In our case this is already added by the meta-mender-demo layer.

Mender configuration

We have to provide our device with MENDER_SERVER_URL and MENDER_TENANT_TOKEN.
For that reason we have to register at https://mender.io/.

Mender provides a free demo profile with limitation of 1 year and up to 10 devices which can be used to experiment with all of Mender’s features.

In fact, when sourcing the setup-environment script we get most of the mender-specific configuration appended to local.conf. This includes a description of how to get our tenant token:

# Build for Hosted Mender
#
# To get your tenant token:
#    - log in to https://hosted.mender.io
#    - click your email at the top right and then "My organization"
#    - press the "COPY TO CLIPBOARD"
#    - assign content of clipboard to MENDER_TENANT_TOKEN
#
#MENDER_SERVER_URL = "https://hosted.mender.io"
#MENDER_TENANT_TOKEN = ""

Note: If using the European server one has to set MENDER_SERVER_URL = "https://eu.hosted.mender.io"

Once we assign our tenant token and remove the # in front of MENDER_SERVER_URL and MENDER_TENANT_TOKEN we’re ready to build our system.

Building and flashing the system image to a microSD card

Build an example image with Yocto:

$ bitbake core-image-base

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

Once the build is complete flash the image to the microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-base-cubox-i.sdimg.bz2 /dev/sdX
$ sync
$ eject /dev/sdX

Connecting to Mender

Once the board finishes booting it will poll the Mender server. By design the connection has to be established from the board to the server. Mender does not open any ports on the board to provide better security therefore the device has to initiate the connection.

When the connection is established the Mender control panel will indicate one pending device.

To accept the request click on View details under Pending devices.

Select the new device and press accept in the Authorization request section.

Once the connection is accepted head over to the Troubleshooting tab in the Device information section. Here you can launch a remote terminal and transfer files.

The next part of the article will demonstrate preparing, uploading and using systemd-analyze to fetch data about the boot process.

Preparing troubleshooting software

To compile systemd-analyze we have to add it to our image in conf/local.conf:

IMAGE_INSTALL:append = " systemd-analyze"

Rebuild systemd to get the systemd-analyze binary:

$ bitbake systemd -c compile
When using the do_compile command Yocto/OpenEmbedded will preserve the compiled binaries.

Find the systemd-analyze binary and libsystemd-shared-<version>.so shared library:

$ find tmp/work -name "systemd-analyze"
$ find tmp/work -name "libsystemd-shared*.so"

Note: As of the time of writing of this article these files should reside in locations similar to ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/systemd-analyze and ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/src/shared/libsystemd-shared-250.so. These paths depend on the exact version of systemd as well as the build configuration and may not be correct in your case.

Uploading the troubleshooting software

Upload these files to the board using Mender’s File Transfer utility:

  • systemd-analyze into /usr/bin/
  • libsystemd-shared-<version>.so into /usr/lib/

Fetching service initialization logs

Once the troubleshooting software is uploaded we can use the Remote Terminal to execute it.

Permit execution of the systemd-analyze binary:

# chmod +x /usr/bin/systemd-analyze

Check the time it took for the system to initialize:

# systemd-analyze

Export a graphic of all enabled services and the time they took to initialize:

# systemd-analyze plot > init.svg

Download init.svg:

This graphic shows that the device needs around a minute to reach multi-user.target. The longest task is the resizing of the /data partition that runs on first boot and the second longest is the filesystem check for mmcblk1p1 that runs every time the system boots.

Here is another graphic generated after a reboot:

This article demonstrates how to use Mender’s Remote terminal and File Transfer troubleshooting utilities to upload and execute the systemd-analyze binary to profile the initialization process of systemd services. These troubleshooting utilities can be used for variety of different tasks. After debugging a single device and finding an appropriate fix, Mender is capable of performing an A/B or delta software update to all devices or specific group of devices in the field.

Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product needs.

Scott Murray on VSS & KUKSA.val at AGL AMM & Embedded World

/in /by

Konsulko Group Principal Software Engineer, Scott Murray will present Vehicle Signaling Specification and KUKSA.val in AGL at the Automotive Grade Linux All Member Meeting in Berlin on Thursday, March 9, 2023.

The Vehicle Signaling Specification (VSS) is an emerging standard to define and categorize vehicle signals. AGL members who attend the session will receive a high-level overview of what VSS and the related Vehicle Information Service Specification (VISS) provide. This will be followed by a discussion of how the KUKSA.val VSS implementation has been integrated into the AGL demonstration stack.

The following week, Scott will be on hand at the AGL stand at Embedded World (Hall 4, Stand 4-171) in Nuremberg, March 14-16. He’ll be happy to talk about VSS, KUKSA.val and many other AGL technical topics. Please stop by if you are visiting the show.

Setting up RAUC on CuBox-I/HummingBoard for Software Updates

/in , , /by

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

RAUC is one of the popular solutions that provide OTA (over-the-air) updates for Embedded Linux devices. RAUC is developed with focus on stability, security and flexibility and is compatible with all popular build systems: The Yocto Project/OpenEmbedded, Buildroot and PTXdist.

RAUC is capable of covering various use cases the most simple one being A/B updates.

The A/B updates scenario consists of having 2 identical root filesystems (named A and B), booting from one of them and performing the update on the other. After the update is complete the bootloader will boot from the updated partition on the next system boot. Recently the ‘verity’ update bundle format was introduced in RAUC. This new groundbreaking feature improves the verification process and most importantly allows extending RAUC by built-in HTTP(S) network streaming support, adaptive delta-like updates, and full update bundle encryption.

This article provides an example for setting up RAUC for A/B updates scenario on a HummingBoard board. The hardware used for the example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • UART to USB adapter

RAUC is a robust, powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image built with the Yocto Project and OpenEmbedded for CuBox-I/HummingBoard one needs to:

  • Use U-Boot as a bootloader
  • Enable SquashFS in the Linux kernel configuration
  • Use ext4 root file system
  • Design specific storage partitioning for the certain use case and configure RAUC accordingly
  • Provide a custom U-Boot script to properly switch between RAUC slots
  • Prepare a certificate and keyring to use for signing and verifying RAUC update bundles.

Building a Linux Distribution with RAUC

I’ve recently contributed to meta-rauc-community, a repository containing minimal RAUC example layers for Yocto/OpenEmbedded.

The following steps will show how to use the meta-rauc-nxp layer from that repository to build and update a minimal Linux distribution. The update will install nano (the text editor) to the system.

Download the reference Yocto distribution, Poky.
We’ll use the latest long term support version, kirkstone.

$ git clone -b kirkstone https://git.yoctoproject.org/poky
$ cd poky

Download meta-rauc-community layers (meta-rauc-nxp):

$ git clone https://github.com/rauc/meta-rauc-community.git

Download the meta-rauc layer:

$ git clone -b kirkstone https://github.com/rauc/meta-rauc.git

Download the BSP layers for cubox-i/HumminbBoard boards:

$ git clone -b kirkstone https://git.yoctoproject.org/meta-freescale
$ git clone -b kirkstone https://github.com/Freescale/meta-fsl-arm-extra.git

Download the meta-openembedded layer (provides nano):

$ git clone -b kirkstone git://git.openembedded.org/meta-openembedded

Initialize the build environment:

$ source oe-init-build-env

Add the layers to conf/bblayers.conf:

$ bitbake-layers add-layer ../meta-openembedded/meta-oe
$ bitbake-layers add-layer ../meta-rauc
$ bitbake-layers add-layer ../meta-freescale
$ bitbake-layers add-layer ../meta-fsl-arm-extra
$ bitbake-layers add-layer ../meta-rauc-community/meta-rauc-nxp

Adjust conf/local.conf by adding the following configurations to the end of the file:

# HummingBoard specifications are very similar to Cubox-I
MACHINE = "cubox-i"

# Accept end user agreement required by the BSP layer.
ACCEPT_FSL_EULA = "1"

# Use systemd as init manager
INIT_MANAGER = "systemd"

# Add RAUC to the image
IMAGE_INSTALL:append = " rauc"
DISTRO_FEATURES:append = " rauc"

# Generate ext4 image of the filesystem
IMAGE_FSTYPES:append = " ext4"

# Use the file containing the partition table specification
WKS_FILE = "sdimage-dual-cubox-i.wks.in"
WKS_FILES:prepend = "sdimage-dual-cubox-i.wks.in "

# Add 150 000 KBytes free space to the root filesystem
# (Adding software with updates require space.)
IMAGE_ROOTFS_EXTRA_SPACE:append = " + 150000"

# Add the boot script to the boot partition
IMAGE_BOOT_FILES:append = " boot.scr"

Note that whitespace inside quotes is intentional and important.

To sign and verify the update bundles RAUC uses SSL keys. A keyring containing all keys that will be used for update bundles needs to be installed on the target.

meta-rauc-community provides a script that would generate example keys and configure the current build environment accordingly. (The script has to be called after sourcing oe-init-build-env)

$ ../meta-rauc-community/create-example-keys.sh

Build a minimal bootable image:

$ bitbake core-image-minimal

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

It’s strongly recommended to zero-fill the u-boot environment sectors before flashing the image on the microSD card (replace /dev/sdX with the proper device path):

$ dd if=/dev/zero of=/dev/sdX seek=2032 count=16

After the build is done, flash the image to a microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-minimal-cubox-i.wic.gz /dev/sdX
$ sync
$ eject /dev/sdX

Attach the USB-to-UART adapter to the HummingBoard Pro, plug the ethernet cable and the microSD card. Turn on the board to verify that the system boots successfully.

By default one can login as root without password.

Creating an update bundle for RAUC

After sourcing the oe-init-build-env, append the following line to the build configuration conf/local.conf to add nano to the system:

# Adding nano
IMAGE_INSTALL:append = " nano"

Build the RAUC update bundle:

$ bitbake update-bundle

Start a web server:

$ cd tmp/deploy/images/cubox-i/
$ pip3 install --user rangehttpserver
$ python3 -m RangeHTTPServer

Now you can install the bundle on the board, then reboot:

# rauc install http://192.168.1.2:8000/update-bundle-cubox-i.raucb
# reboot

One of the latest RAUC features is the verity bundle format. This format allows updates to be done without storing the whole bundle on the device in advance, which is useful for devices with limited space. One of the requirements for this feature is hosting the bundle on a server that supports HTTP Range Requests.

As alternative, you can transfer the bundle to the device and install it from local storage.

Verify that nano was added to the system:

# which nano

Check RAUC status to confirm the system have booted from the second partition:

# rauc status

For real-world products, this build procedure with the Yocto Project and OpenEmbedded can be optimized further with just a few commands for easy implementation of continuous integration (CI).

Since the earliest days of the OpenEmbedded build framework and Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product development.

Konsulko Group: The Year in Review 2022

/in /by

2022 went by quickly with Konsulko engineers working closely with our customers, our partners and the open source community. For the tenth straight year, Konsulko Group has helped our clients build outstanding commercial products with Embedded Linux, Yocto Project and OpenEmbedded, as well as deploying Over-the-Air (OTA) software updating.

Engaging with our customers

We had particular success with our Konsulko Continuous Time Engagement™ offering, providing dedicated engineering resources for two of the world’s largest semiconductor companies. This model provides guaranteed engineering time for an agreed period. KCTE has allowed our customers to use our engineers on a variety of their requirements, and switch between these tasks as the rest of their project and in-house engineering required.

Of course, many of our clients prefer Konsulko OnDemand Time Engagement™, SOW-based engineering for high level consulting, on-demand support, and specific tasks within a larger project. KOTE is also best for longer term engagements without a hard deadline that can be stopped and started as necessary, and projects with an expected pause (such as bring-up of new hardware which almost always results in a re-spin of the PCB).

Partnerships and Conferences

We continue our strong relationship with the Linux Foundation and Automotive Grade Linux. We worked with mender.io and PHYTEC, providing support and development for their customers. With our friends at ICS, we presented at a joint webinar in August, Software Update Mechanisms: Selecting the Best Solution by Konsulko’s Leon Anavi and ICS’ Jeff Tranter.

Our Konsulko engineers were active participants (often in person, sometimes virtually) in conferences and developer gatherings. Leon Anavi, Vitaly and Maria Wool, and Tim Orling made presentations at the Embedded Linux Conferences (North America and Europe), FOSDEM and Yocto Project Summits:

Tim also posted a technical paper on Using kernel config fragments to remove an unwanted feature.

Two top engineers join the Konsulko team

The bar is pretty high for joining Konsulko Group. Some of our team have thirty years experience with embedded software and Linux. Most have over twenty and all have over fifteen years, working (and playing) with Linux and the open source community. So we were happy to welcome Denys Dmytriyenko and António Oliveira to Konsulko Group this year.

Denys is a long time Open Source Software developer, contributing code to many FOSS projects, like glibc, PostgreSQL, KDE, MythTV, LIRC, OpenEmbedded, and Yocto Project. Before joining Konsulko Group, he worked as Software Developer and Architect for Texas Instruments for almost 20 years on numerous Embedded Linux (MIPS and ARM-based) products. Since 2011 Denys has served on the OpenEmbedded Board of Directors, and since 2019, on the Yocto Project Technical Steering Committee. He is a long time contributor and maintainer of numerous recipes and layers in the OpenEmbedded/Yocto Project universe.

António has worked in development of highly optimized hardware and software ranging from deeply embedded microcontrollers to full-fledged graphical user interfaces. During this time, he has gained extensive experience in build automation tools, analog and digital circuits, powerline communications, and low power and low frequency radio communications. An active member of the open source community, he has contributed to Yocto Project and the Linux kernel. In addition to his work as an embedded engineer, António served eight years on the executive board of his local parish in Portugal, including four years on the municipal council.

We hope you will have the opportunity to work with Denys, António and the rest of Konsulko Group in 2023.

Konsulko Group to speak at Yocto Project Virtual Summit

/in /by

Konsulko Group’s Leon Anavi will present Porting Mender to New i.MX 8M Plus Machine and Working with Toradex BSP at the Yocto Project Virtual Summit. Leon will focus on porting Mender, a popular open source over-the-air software update solution, to Toradex Verdin iMX8M Plus SoM. He’ll examine the exact steps to build images for it and the Dahlia carrier board using the long-term support release of Yocto Project and the Toradex BSP.

Tim Orling will present two technical “lightning talks.” Running QEMU Images with virtio, virsh and virt-manager will discuss some experiences building and launching QEMU images with tools that leverage libvirt and virtio on a host. Creating an Installer that Leverages bmaptool will describe how to create an installer image using output from another image and writing all the partitions at once to the target using bmaptool.

The Yocto Project Virtual Summit is a 3-day technical conference for engineers, open source technologists, students and academia in the OSS space. Learn about Yocto Projects’ direction, get training on the next wave of embedded Linux technologies and network with industry peers, Yocto Project maintainers, OpenEmbedded maintainers and other experts.

Registration is $40 for the whole event. We hope you are able to join us.

Pick the Right Software Update Solution for Your Product

/in /by

Live Webinar:

Software Update Mechanisms: Selecting the Best Solution for Your Embedded Linux Device

August 18, 2022 at 1 pm EDT/7pm CET

Get the foundation you need to make an informed decision on the right solution for your software update needs. Leon Anavi of Konsulko Group and Jeff Tranter of ICS will examine several different industry approaches, including A/B updates with a dual-redundant scheme, delta updates, container-based updates and combined strategies, as well as the leading technologies that support these approaches.

You can find complete information and register here for the free webinar on the ICS website. Hope you’ll join us on August 18.

Join us at Embedded World in Nuremberg, June 21-23, 2022

/in /by

Are you coming to Embedded World? Konsulko Group will be there, too, supporting our partners and friends.

Be sure to stop by the Automotive Grade Linux (AGL) stand in Hall 4 to see the AGL ecosystem and to say hello.

Better still, if you’d like to set up an appointment at the show to talk with us about your company’s embedded products and your software development requirements, please contact us to set up a meeting.

Tag Archive for: embedded linux