Held on June 10th and 11th, 2023 in the beautiful city of Plovdiv (Bulgaria), TuxCon 2023 marked the 10th anniversary of this annual local two-day conference. Organized by dedicated volunteers, the event aims to advocate the adoption of open source hardware and foster the use of free and open source software.
Konsulko Group intern Atanas Bunchev participated as a speaker, and demonstrated remote updates and troubleshooting of connected embedded Linux devices using Mender.io. The presentation was in Bulgarian and spread the word about the Yocto Project, OpenEmbedded and various Mender features among the local community.
Mender is an open-source, over-the-air (OTA) software update management platform specifically designed for embedded devices. It provides an efficient way for software updates and management of fleets of connected devices in various industries, such as IoT (Internet of Things), automotive, industrial automation, and healthcare. It provides options for ensuring the resilience of system-wide updates and also for updating individual applications. Furthermore debugging of devices is possible with the troubleshooting package, such as the File Transfer and Remote Terminal Add-ons.
Konsulko Group, a partner of Northern.tech, often works with Mender.io. If you are developing a new product, we would be thrilled to discuss how Konsulko’s engineering expertise and experience can contribute to your project. Or if you are a Linux software developer who is passionate about open source, we invite you to reach out to us regarding potential opportunities to join the Konsulko team.
https://www.konsulko.com/wp-content/uploads/2023/06/TUXCON23-1.png900900Konsulko Grouphttps://www.konsulko.com/wp-content/uploads/2021/03/KonsulkoGroup-fullcolorlogo-340x156-1.pngKonsulko Group2023-06-27 10:43:472023-06-27 10:43:48Konsulko Group updates Embedded Linux devices at TuxCon 2023
(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)
Konsulko Group often works with Mender.io to provide secure, risk tolerant and efficient Over-The-Air updates for any software on an embedded device. This includes both solutions for robust system updates as well as updates of single applications. Mender is compatible with any Linux based OS, including popular build systems such as Yocto Project/OpenEmbedded, Buildroot and OpenWrt.
Aside from OTA updates Mender also offers several add-ons. These are optional extensions that provide functionality for use cases beyond the core OTA updates features.
One such use case is remote troubleshooting. In practice deployed devices in the field are often hard to reach or retrieve. Troubleshooting individual devices becomes resource inefficient and in some cases they have to be replaced even when it’s just a small software or configuration issue. Our experience has shown in these situations Mender’s Add-ons have proven to be extremely useful.
This article provides as an example, the exact steps how to integrate Mender with The Yocto Project and OpenEmbedded for SolidRun CuBox-I and HummingBoard as well as a demonstration of the Mender Troubleshooting package, more specifically the File Transfer and Remote Terminal Add-ons. With the File Transfer add-on files can be downloaded and uploaded to any accepted device. Remote Terminal add-on allows remote interactive command execution from the Mender UI.
These add-ons are very valuable for system administration and mantainance of Internet of Things or fleets of connected industrial devices. For an example, we will troubleshoot an embedded Linux device remotely with systemd using Mender add-ons. We will upload tools to gather system boot-up performance statistics, run them on the device thanks to the Remote Terminal and download logs for further debugging.
Generic RJ45 network cable with Internet connection
Optionally UART to USB adapter for debugging the setup section
Building a Linux Distribution with Yocto/OpenEmbedded
The Mender Community provides a set of examples for integration with various hardware platforms using Yocto/OpenEmbedded in the meta-mender-community repository. Sub-layer meta-mender-nxp in this repository contains the integration for Cubox-I/HummingBoard.
The meta-mender-nxp layer uses Google Repo to provide easy and simple setup and build process for the examples:
To configure the build system we have to append to conf/local.conf inside the build directory.
First we are going to set the build target machine:
MACHINE = "cubox-i"
Then we have to accept the end user agreement required by the BSP layer:
ACCEPT_FSL_EULA = "1"
Note:Usually to enable Mender’s Troubleshooting features we have to add mender-connecttoIMAGE_INSTALL. In our case this is already added by the meta-mender-demo layer.
Mender configuration
We have to provide our device with MENDER_SERVER_URL and MENDER_TENANT_TOKEN. For that reason we have to register at https://mender.io/.
Mender provides a free demo profile with limitation of 1 year and up to 10 devices which can be used to experiment with all of Mender’s features.
In fact, when sourcing the setup-environment script we get most of the mender-specific configuration appended to local.conf. This includes a description of how to get our tenant token:
# Build for Hosted Mender
#
# To get your tenant token:
# - log in to https://hosted.mender.io
# - click your email at the top right and then "My organization"
# - press the "COPY TO CLIPBOARD"
# - assign content of clipboard to MENDER_TENANT_TOKEN
#
#MENDER_SERVER_URL = "https://hosted.mender.io"
#MENDER_TENANT_TOKEN = ""
Note:If using the European server one has to setMENDER_SERVER_URL = "https://eu.hosted.mender.io"
Once we assign our tenant token and remove the # in front of MENDER_SERVER_URL and MENDER_TENANT_TOKEN we’re ready to build our system.
Building and flashing the system image to a microSD card
Build an example image with Yocto:
$ bitbake core-image-base
Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.
Once the build is complete flash the image to the microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:
Once the board finishes booting it will poll the Mender server. By design the connection has to be established from the board to the server. Mender does not open any ports on the board to provide better security therefore the device has to initiate the connection.
When the connection is established the Mender control panel will indicate one pending device.
To accept the request click on View details under Pending devices.
Select the new device and press accept in the Authorization request section.
Once the connection is accepted head over to the Troubleshooting tab in the Device information section. Here you can launch a remote terminal and transfer files.
The next part of the article will demonstrate preparing, uploading and using systemd-analyze to fetch data about the boot process.
Preparing troubleshooting software
To compile systemd-analyze we have to add it to our image in conf/local.conf:
IMAGE_INSTALL:append = " systemd-analyze"
Rebuild systemd to get the systemd-analyze binary:
$ bitbake systemd -c compile
When using the do_compile command Yocto/OpenEmbedded will preserve the compiled binaries.
Find the systemd-analyze binary and libsystemd-shared-<version>.so shared library:
Note:As of the time of writing of this article these files should reside in locations similar to./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/systemd-analyze and ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/src/shared/libsystemd-shared-250.so. These paths depend on the exact version of systemd as well as the build configuration and may not be correct in your case.
Uploading the troubleshooting software
Upload these files to the board using Mender’s File Transfer utility:
systemd-analyze into /usr/bin/
libsystemd-shared-<version>.so into /usr/lib/
Fetching service initialization logs
Once the troubleshooting software is uploaded we can use the Remote Terminal to execute it.
Permit execution of the systemd-analyze binary:
# chmod +x /usr/bin/systemd-analyze
Check the time it took for the system to initialize:
# systemd-analyze
Export a graphic of all enabled services and the time they took to initialize:
# systemd-analyze plot > init.svg
Download init.svg:
This graphic shows that the device needs around a minute to reach multi-user.target. The longest task is the resizing of the /data partition that runs on first boot and the second longest is the filesystem check for mmcblk1p1 that runs every time the system boots.
Here is another graphic generated after a reboot:
This article demonstrates how to use Mender’s Remote terminal and File Transfer troubleshooting utilities to upload and execute the systemd-analyze binary to profile the initialization process of systemd services. These troubleshooting utilities can be used for variety of different tasks. After debugging a single device and finding an appropriate fix, Mender is capable of performing an A/B or delta software update to all devices or specific group of devices in the field.
Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product needs.
https://www.konsulko.com/wp-content/uploads/2023/04/mender-add-ons-graphic.png798798Konsulko Grouphttps://www.konsulko.com/wp-content/uploads/2021/03/KonsulkoGroup-fullcolorlogo-340x156-1.pngKonsulko Group2023-04-03 10:50:122023-05-01 10:51:41Mender Add-ons: Remote Troubleshooting Devices in the Field
Konsulko Group’s Leon Anavi will present Porting Mender to New i.MX 8M Plus Machine and Working with Toradex BSP at the Yocto Project Virtual Summit. Leon will focus on porting Mender, a popular open source over-the-air software update solution, to Toradex Verdin iMX8M Plus SoM. He’ll examine the exact steps to build images for it and the Dahlia carrier board using the long-term support release of Yocto Project and the Toradex BSP.
The Yocto Project Virtual Summit is a 3-day technical conference for engineers, open source technologists, students and academia in the OSS space. Learn about Yocto Projects’ direction, get training on the next wave of embedded Linux technologies and network with industry peers, Yocto Project maintainers, OpenEmbedded maintainers and other experts.
Registration is $40 for the whole event. We hope you are able to join us.
https://www.konsulko.com/wp-content/uploads/2022/11/yocto-project-SUMMIT-11292022-1.png17981798Konsulko Grouphttps://www.konsulko.com/wp-content/uploads/2021/03/KonsulkoGroup-fullcolorlogo-340x156-1.pngKonsulko Group2022-11-03 15:49:102022-12-26 08:08:56Konsulko Group to speak at Yocto Project Virtual Summit
This talk will look at the advantages and disadvantages of widely-used industry approaches: A/B updates with dual redundant scheme, delta updates, container-based updates and combined strategies. Open source technologies such as Mender, RAUC and libostree-based solutions implement these strategies and provide tools to manage updates of multiple devices. Leon will discuss how to choose an appropriate open source solution to implement for a specific project.
XIP stands for eXecute In Place, allowing code to be executed directly from flash without copying the code to RAM first, making it possible to run Linux on such RISC-V devices as Kendryte K210, which has only 8 MB of SRAM, not only for demonstration purposes but for real applications as well. This talk will cover running edge computing specifically on K210 under Linux with XIP enabled.
Part of Open Source Summit North America
These and many other excellent presentations at ELC are part of the Linux Foundation’s Open Source Summit. We hope you will join us in June.
https://www.konsulko.com/wp-content/uploads/2022/05/Austin3-copy.png10861086Konsulko Grouphttps://www.konsulko.com/wp-content/uploads/2021/03/KonsulkoGroup-fullcolorlogo-340x156-1.pngKonsulko Group2022-05-18 10:01:202022-12-26 08:11:14Konsulko Group speaking at Embedded Linux Conference Austin
Almost every device in development today requires both over-the-air (OTA) updating capabilities, and up-to-date security with authentication of the device filesystem and encryption of application data storage.
At Konsulko Group, we find that the best approach is to build a secure platform first, starting with the hardware root of trust and establishing a chain of trust by extending the root of trust through each successive component in the system.
A first step is to make sure the Universal Boot Loader (U-Boot) is up-to-date. If the device is using an older version, we may port U-Boot support of a more current release. Then we enable signature-based authentication of the device using the U-Boot verified boot feature.
To provide authentication of the device filesystem and encryption of application data storage, we often use the Linux kernel Device-Mapper infrastructure to create virtual layers of block devices: dm-verity for root filesystem authentication, dm-crypt for data encryption, and dm-integrity for read/write data volume integrity.
Finally, we can integrate Mender I/O support for OTA together with dm-verity and Yocto Project so that device updates can be performed while maintaining the system’s secure chain of trust.
This step-by-step methodology ensures straight-forward and predictable development. Please contact us to discuss how we can help you build a secure platform for OTA updating on your device.
https://www.konsulko.com/wp-content/uploads/2021/02/Security-graphic-1.png10221022Konsulko Grouphttps://www.konsulko.com/wp-content/uploads/2021/03/KonsulkoGroup-fullcolorlogo-340x156-1.pngKonsulko Group2021-04-13 07:10:102023-05-01 10:54:48Building Platforms with Secure Over-the-Air Updating
As we approach the end of December, it’s time to review a year that we will certainly remember for the rest of our lives. It hasn’t been easy, but all of us at Konsulko Group are still working hard, supporting the open source community and helping our customers build forward-looking products.
Even at the start of 2020, we knew this year would be different. The coronavirus was the talk of CES in January with some companies pulling out at the last minute, and everyone wondering what the global business climate would be in the months ahead.
By the time FOSDEM rolled around a few weeks later, it was clear the virus would disrupt commerce worldwide, and by the end of February, we had to cut short our presence at Embedded World because of new travel restrictions.
Then the world locked down completely. Since at Konsulko Group we all work remotely by design, we didn’t have to adjust our way of developing software, but as it did for everyone, we had to significantly change our face-to-face participation in embedded Linux, Yocto Project and other community events.
We taught ourselves to use video editing tools, and gave “virtual” presentations from our desks at the Embedded Linux Conference North America and Yocto Project Dev Day at the end of June, the virtual Automotive Grade Linux All Member Meeting in mid July, Linaro Connect in September, Virtual ELCE and Yocto Project Virtual Summit Europe at the end of October, and participated in the virtual Automotive Linux Summit the first of December.
Early in the year, we announced that we had become a Mender Authorized Referral Partner, and that important alliance has provided dividends to both Konsulko Group and our customers as the year progressed.
Konsulko engineers continued our series of technical blogs…
Still, we have all been touched by the physical and emotional toll of COVID-19. Two of our engineers have endured a bout with the virus (and thankfully recovered). Some of us have family, friends or acquaintances who have become seriously ill or even passed away. We can only hope the post-pandemic world is now in sight.
In the long run, the challenges that 2020 have brought us closer together in many ways, and hopefully taught us valuable lessons that will make us stronger in a better new year.
https://www.konsulko.com/wp-content/uploads/2020/12/2020-THE-YEAR-IN-REVIEW.png12001200Pete Popovhttps://www.konsulko.com/wp-content/uploads/2021/03/KonsulkoGroup-fullcolorlogo-340x156-1.pngPete Popov2020-12-18 08:39:142023-02-26 11:14:19CEO Pete Popov looks back on 2020
Software Update solutions are a key part of our services offering. For open source over-the-air updates, we often we recommend and work with mender.io. In fact, Konsulko Group is a Mender Authorized Referral Partner. Recently, a prospective customer expressed an interest in knowing more about how Mender works. Here’s the brief, informal introduction to member.io that I prepared, and now I’m sharing with you.
—
As a high level starting point, https://mender.io/how-it-works provides a good overview of what’s supported and what it covers. In short, Mender starts off by providing support for a traditional “A/B” approach to system updates, where if the update isn’t marked as valid (and there’s hooks for the application(s) to verify the system before this is done), it’s assumed invalid and the system will roll-back automatically. While “OTA” implies over the network, it can just as easily be done by providing (and validating) a USB key that contains an update.
One of the reasons we recommend Mender is that it has very good in-depth documentation. The starting point for all of that is https://docs.mender.io/2.4/ which covers all of the topic starting from how to implement Mender support in a device and including how to create your own server infrastructure if you don’t want to use their paid service. While there are a number of important pages there, one that I like to highlight is https://docs.mender.io/2.4/artifacts/state-scripts as it shows the state machine for an update and talks about some of the common use cases that come up for user interaction or dealing with failures.
Another place I want to call out is https://docs.mender.io/2.4/devices/update-modules which is also mentioned in the first link. This is how Mender is extended to provide updates for other parts of the system that are not the rootfs itself. Since updating a Docker container is something that has been mentioned before I want to also note https://hub.mender.io/t/docker/324 as it is a well supported module for this specific case.
I hope you find this information useful. Please contact us if you have specific questions. We’re looking forward to talking to you about your own specific OTA needs.
https://www.konsulko.com/wp-content/uploads/2020/10/How-Mender-works.png922922Tom Rinihttps://www.konsulko.com/wp-content/uploads/2021/03/KonsulkoGroup-fullcolorlogo-340x156-1.pngTom Rini2020-10-23 09:57:372023-05-01 10:56:11How Mender works
As Embedded Linux pioneers, Konsulko Group is excited to work with Mender.io, a leader in open source over-the-air (OTA) software updates. Konsulko has already completed several successful projects for our customers using Mender end-to-end OTA software update manager.
As a Mender Authorized Referral Partner we believe we can give our customers robust and secure open source OTA solutions that can be extended for large scale software deployments.
Please contact Konsulko Group directly to see how we can integrate open source over-the-air software updates into your next product.
