Some of the Highlights from Linux Plumbers Conference 2023

/in /by

In November, Konsulko Group Principal Software Engineer Denys Dmytriyenko attended The Linux Plumbers Conference (LPC), a developer conference for the open source community, held this year in Richmond, Virginia. Here is his report on the highlights of the conference.

LPC brings together the top developers working on the “plumbing” of Linux – kernel subsystems, core libraries, windowing systems, etc. – and gives them three days to work together on core design problems. The conference is divided into several working sessions focusing on different “plumbing” topics, as well as a general paper track.

Usually there are several main tracks and many Microconferences all fit into 3 days. And this year the list included Refereed Track, Kernel Summit, Toolchains Track and BoF. As well as Microconferences on Kernel Testing & Dependability, Real-time and Scheduling, RISC-V, Compute Express Link, Containers and Checkpoint/Restore, Android, Build Systems, Linux Kernel Debugging, KVM, Confidential Computing, Power Management and Thermal Control, Tracing, VFIO/IOMMU/PCI, Live Patching, Rust and Internet of Things.

Due to the number of parallel tracks and Microconferences with a high number of interesting topics, it’s quite impossible to attend all the talks one would like.

Here are just a few that I was able to attend.

Resolve and standardize early access to hardware for automotive industry with Linux

Khasim Syed Mohammed, Texas Instruments

SoC manufacturers are adopting heterogeneous architectures, using MCU cores for safety-critical tasks with RTOS and MPU running Linux for general purpose. Key use cases in the automotive industry, such as quick boot times, early display response, and predictable access to hardware, pose challenges for Linux-based systems. The current solutions of employing heterogeneous processors lack scalability, standardization, and Linux friendliness.

Khasim explored the specific challenges faced by the automotive industry and the existing solutions, addressing the standardization of “Linux late attach” with heterogeneous SoCs. He invited collaboration between automotive OEMs, SoC manufacturers, and Linux kernel/user space maintainers to establish “Linux automotive” standards, enhance the Linux kernel and drivers to meet performance requirements.

Securing build platforms

Joshua Lock, Verizon

Historically, Open Source Software was always built on trust – you trusted the sources, the recipe, the builder to produce the resulting artifacts for a given distro that many use. But with a growing number of attacks on the software supply chain, securing build platforms and providing a verifiable way to link packages back to their sources, as well as specific build instructions used to package and publish them, is becoming very important these days.

Joshua listed several of the big and famous recent supply chain attacks and talked about build provenance being adopted by different players in this field to battle such attacks. He then delved into adoption of the Supply-chain Levels for Software Artifacts, or SLSA (“salsa”) architecture in the Yocto Project and its Autobuilder CI framework – generating provenance during the build as L1 level requirement and signing said provenance at L2 level. While this definitely helps with detecting such attacks, fully securing the software supply chain needs more work.

Improving UAPI Compatibility Review with Automated Tooling

John Moon, Qualcomm

Preserving Linux Userspace API (UAPI) compatibility has been crucial for enabling smooth kernel upgrades without breaking userspace programs. Traditionally, kernel devs relied on code review and testing to catch UAPI-breaking changes.

John showed a new tool they developed at Qualcomm that uses libabigail, letting developers analyze patches for UAPI issues before executing them. This tool could integrate into build systems like OpenEmbedded and the Yocto Project, providing immediate feedback and solidifying the kernel’s UAPI stability policy.

The talk also delved into other interfaces like sysfs and module parameters. However, since libabigail can’t analyze these interfaces directly due to the lack of a clean C API, a couple proposals were presented and sparked a lively discussion with Greg KH and other attendees.

kernel: build system outputs and workflows (and how to balance them)

Bruce Ashfield, AMD

Bruce talked about all the complexities around maintaining and building the Linux kernel in OpenEmbedded and the Yocto Project. All the various inputs, such as vendor BSPs vs. upstream mainline kernel, latest version vs. LTS, different patches, configurations, policies, etc. It gets complicated by additional product requirements, like reproducibility, flexibility, maintenance and support, build speed and so on. And the variety of output end results only adds overhead, when one has to deal with kernel images, modules, DTBs, firmware images, binary packages, debug information, licensing, SBoM manifests, CVE reports, boot components, containers and VMs, SDKs, libc headers, etc. It also differs a bit based on the persona or workflow of one’s building the kernel – developer (kernel vs. userspace), integrator, distributor, contributor or a release engineer.

Trying to keep everyone happy at the same time is almost impossible. Still, linux-yocto, the reference kernel in OpenEmbedded-Core, tries to cover most of the bases and solve as many of the outlined challenges. Bruce talked a bit more about how he maintains linux-yocto, how things are improving over time, where we are heading and opened the floor to discussion.

How big of a problem are Un-upstreamed patches?

Jon Mason, ARM

Build Systems and Distros accumulate local changes against individual components in the form of patches that either fix newly found issues or customize component’s behavior to suit their own requirements. How should those patches be handled and what to do with them? Upstreaming them to be integrated into the future releases of corresponding components is, of course, the best approach. What if it’s not possible or doesn’t happen for other reasons? Is there any structure or process for that? Jon looked at these problems and tried answering the questions from the perspective of the Yocto Project, Buildroot, Debian and other distros.

Building for Heterogeneous Systems

Alejandro Hernandez Samaniego, Microsoft

Heterogeneous systems are widely used in embedded products for better performance, safety, etc. While using different operating systems on various core architectures in a single device is beneficial, each OS has unique dependencies and developer workflows. To optimize development, a single build system should be able to cross-compile different OSs for different architectures simultaneously and Bitbake offers this through its multiconfig feature. Unifying the developer workflow and enhancing the usability of features like multiconfig would significantly improve teams efforts in creating new products.

Alejandro went over examples of setting up multiconfig in the Yocto Project for building bare metal and/or RTOS components along with the general purpose LInux system and integrating final results into a single filesystem image. At the end we discussed some of the remaining challenges and open defects with some of the aspects of multiconfig, like separating temp directories, SDK coverage, etc.

A kernel documentation update

Jonathan Corbet, LWN.net

Jonathan provided an update on the recent changes in the Linux kernel documentation – restructuring of architecture-specific docs, new Rust documentation, added new translations, new theme for HTML-rendered pages, texinfo support, lots of new docs were added, rework of the top-level index.rst, etc. He also commented on some of the future plans, such as reworking of device-related docs, which are now spanned across 31 subdirectories, and adding more structure to the rest of areas in the documentation.

A lively debate was sparked around plain text vs. HTML and format of the docs in general, e.g. plain text is still preferred by some kernel maintainers over HTML, but adding RST markup to the input text files makes them less readable. Sphinx and tooling were then discussed, build-time improvements, support for partial builds to speed things up, etc.

Embedded Linux BoF

Tim Bird, Sony

Tim led an open discussion session covering different aspects of embedded Linux, such as Architectures, Bootloaders, Boot time, Filesystems, Networking, Security, Testing, Tools, Toolchains, Tracing, System Size and Build Tools and Distros. He mentioned latest updates to U-boot, referenced earlier talk by Khasim on reducing boot time by offloading some tasks to MCUs and deferring non-critical device probing to later time in user-space.

For size reduction he listed removal of SLOB and planned removal of SLAB. But as Arnd Bergmann, kernel developer and arm-soc tree maintainer commented, system size may no longer be relevant, as memory becomes cheaper and ARM manufacturers actively switching from 32-bit to 64-bit solutions, while small footprint systems are relegated to Zephyr. On the other hand, nolibc allows building very small statically-linked binaries for Linux.

Finishing up with closing remarks about the Embedded Linux Community with shoutouts for mailing lists, different Organizations and Projects operating in this field, popular upcoming events and the state of eLinux wiki losing its funding soon.

Zephyr Retro-and-Prospective: Project Growth, Long Term Support, and Linux Interoperability

Chris Friedt, Meta

Chris’ lightning talk quickly went over Zephyr’s brief, but very rich and rapidly-changing history. Added new architectures, 64-bit support, new protocols, new boards, Device Tree API, TF-M, LLVM and other toolchains, Power Management, USB-C/PD, DSP subsystem and many more features and improvements. Many bugs were fixed, new maintainers and contributors joined. All in all Zephyr enjoyed tremendous growth in recent years.

Challenges in Device Tree Sync – kernel, Zephyr, U-boot, System DT

Nishanth Menon, Texas Instruments

These days Device Tree is commonly used to describe the hardware in different ecosystems, such as Linux kernel, U-boot and Zephyr. But each ecosystem has its own goals and limitations and along with limited standardization of Device Tree description of hardware that results in very different user/developer experiences and challenges keeping them in sync.

With a BeaglePlay board using TI AM625 heterogeneous SoC, Nishanth was hoping to blink one LED from A53 core running Linux and blink another LED from M4F core running Zephyr, all described by the same Device Tree. But it wasn’t as simple – he showed examples of different Device Trees describing the same hardware from the perspective of Linux kernel, U-boot bootloader and Zephyr RTOS. The Linux kernel has A53-centric view, while U-boot also focuses on R5F core taking care of the initial boot, along with binman and bootph specifics, where Zephyr diverges from both of them due to its focus on performance and resource constraints, utilizing C headers generated from the Device Tree.

There are some other differences between ecosystems using Device Tree, such as licensing, bindings, firmware usage and tooling. But one of the proposed solutions to unify the above inconsistencies was a lopper tool and the use of a common/canonical System Device Tree with personality overlays that produces required device-specific views.

Konsulko Group in Paris at Embedded Recipes 2023

/in /by

Three Konsulko engineers traveled to Paris (left to right, in créatif senior Frank Tizzoni‘s great caricature): António Oliveira came from Portugal, Darko Alavanja from Serbia, Scott Murray from Canada.

Embedded Recipes was held over two days in late September at Fondation Biermans Lapotre. The event had a nice format: single room, single track. That meant everyone stayed together for its duration which made it easier to meet people.

We talked a little about everything: kernel internals, prototyping tools, software updates, Yocto Project, machine learning…

Scott gave a nice overview of the Automotive Grade Linux project, its history, current work and future plans.

Mender Add-ons: Remote Troubleshooting Devices in the Field

/in , , /by

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

Konsulko Group often works with Mender.io to provide secure, risk tolerant and efficient Over-The-Air updates for any software on an embedded device. This includes both solutions for robust system updates as well as updates of single applications. Mender is compatible with any Linux based OS, including popular build systems such as Yocto Project/OpenEmbedded, Buildroot and OpenWrt.

Aside from OTA updates Mender also offers several add-ons. These are optional extensions that provide functionality for use cases beyond the core OTA updates features.

One such use case is remote troubleshooting. In practice deployed devices in the field are often hard to reach or retrieve. Troubleshooting individual devices becomes resource inefficient and in some cases they have to be replaced even when it’s just a small software or configuration issue. Our experience has shown in these situations Mender’s Add-ons have proven to be extremely useful.

This article provides as an example, the exact steps how to integrate Mender with The Yocto Project and OpenEmbedded for SolidRun CuBox-I and HummingBoard as well as a demonstration of the Mender Troubleshooting package, more specifically the File Transfer and Remote Terminal Add-ons. With the File Transfer add-on files can be downloaded and uploaded to any accepted device. Remote Terminal add-on allows remote interactive command execution from the Mender UI.

These add-ons are very valuable for system administration and mantainance of Internet of Things or fleets of connected industrial devices. For an example, we will troubleshoot an embedded Linux device remotely with systemd using Mender add-ons. We will upload tools to gather system boot-up performance statistics, run them on the device thanks to the Remote Terminal and download logs for further debugging.

Any Mender enabled device is suitable for this demonstration. Recently the Konsulko Senior Engineer Leon Anavi ported Mender to SolidRun CuBox-I and HummingBoard with NXP i.MX6 SoC so our current setup is based on this hardware platform and Yocto LTS release Kirkstone. Earlier we also ported RAUC, an alternative free and open source OTA update platform to the same hardware. If you are interested in software updates, please contact us to discuss your own embedded product needs in details. The hardware used in this example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • Generic RJ45 network cable with Internet connection
  • Optionally UART to USB adapter for debugging the setup section

Building a Linux Distribution with Yocto/OpenEmbedded

The Mender Community provides a set of examples for integration with various hardware platforms using Yocto/OpenEmbedded in the meta-mender-community repository. Sub-layer meta-mender-nxp in this repository contains the integration for Cubox-I/HummingBoard.

The meta-mender-nxp layer uses Google Repo to provide easy and simple setup and build process for the examples:

$ mkdir mender-nxp && cd mender-nxp
$ repo init -u https://github.com/mendersoftware/meta-mender-community \
            -m meta-mender-nxp/scripts/manifest-nxp.xml \
            -b kirkstone
$ repo sync
$ source setup-environment nxp

Board configuration

To configure the build system we have to append to conf/local.conf inside the build directory.

First we are going to set the build target machine:

MACHINE = "cubox-i"

Then we have to accept the end user agreement required by the BSP layer:

ACCEPT_FSL_EULA = "1"

Note: Usually to enable Mender’s Troubleshooting features we have to add mender-connect to IMAGE_INSTALL. In our case this is already added by the meta-mender-demo layer.

Mender configuration

We have to provide our device with MENDER_SERVER_URL and MENDER_TENANT_TOKEN.
For that reason we have to register at https://mender.io/.

Mender provides a free demo profile with limitation of 1 year and up to 10 devices which can be used to experiment with all of Mender’s features.

In fact, when sourcing the setup-environment script we get most of the mender-specific configuration appended to local.conf. This includes a description of how to get our tenant token:

# Build for Hosted Mender
#
# To get your tenant token:
#    - log in to https://hosted.mender.io
#    - click your email at the top right and then "My organization"
#    - press the "COPY TO CLIPBOARD"
#    - assign content of clipboard to MENDER_TENANT_TOKEN
#
#MENDER_SERVER_URL = "https://hosted.mender.io"
#MENDER_TENANT_TOKEN = ""

Note: If using the European server one has to set MENDER_SERVER_URL = "https://eu.hosted.mender.io"

Once we assign our tenant token and remove the # in front of MENDER_SERVER_URL and MENDER_TENANT_TOKEN we’re ready to build our system.

Building and flashing the system image to a microSD card

Build an example image with Yocto:

$ bitbake core-image-base

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

Once the build is complete flash the image to the microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-base-cubox-i.sdimg.bz2 /dev/sdX
$ sync
$ eject /dev/sdX

Connecting to Mender

Once the board finishes booting it will poll the Mender server. By design the connection has to be established from the board to the server. Mender does not open any ports on the board to provide better security therefore the device has to initiate the connection.

When the connection is established the Mender control panel will indicate one pending device.

To accept the request click on View details under Pending devices.

Select the new device and press accept in the Authorization request section.

Once the connection is accepted head over to the Troubleshooting tab in the Device information section. Here you can launch a remote terminal and transfer files.

The next part of the article will demonstrate preparing, uploading and using systemd-analyze to fetch data about the boot process.

Preparing troubleshooting software

To compile systemd-analyze we have to add it to our image in conf/local.conf:

IMAGE_INSTALL:append = " systemd-analyze"

Rebuild systemd to get the systemd-analyze binary:

$ bitbake systemd -c compile
When using the do_compile command Yocto/OpenEmbedded will preserve the compiled binaries.

Find the systemd-analyze binary and libsystemd-shared-<version>.so shared library:

$ find tmp/work -name "systemd-analyze"
$ find tmp/work -name "libsystemd-shared*.so"

Note: As of the time of writing of this article these files should reside in locations similar to ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/systemd-analyze and ./tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/systemd/1_250.5-r0/build/src/shared/libsystemd-shared-250.so. These paths depend on the exact version of systemd as well as the build configuration and may not be correct in your case.

Uploading the troubleshooting software

Upload these files to the board using Mender’s File Transfer utility:

  • systemd-analyze into /usr/bin/
  • libsystemd-shared-<version>.so into /usr/lib/

Fetching service initialization logs

Once the troubleshooting software is uploaded we can use the Remote Terminal to execute it.

Permit execution of the systemd-analyze binary:

# chmod +x /usr/bin/systemd-analyze

Check the time it took for the system to initialize:

# systemd-analyze

Export a graphic of all enabled services and the time they took to initialize:

# systemd-analyze plot > init.svg

Download init.svg:

This graphic shows that the device needs around a minute to reach multi-user.target. The longest task is the resizing of the /data partition that runs on first boot and the second longest is the filesystem check for mmcblk1p1 that runs every time the system boots.

Here is another graphic generated after a reboot:

This article demonstrates how to use Mender’s Remote terminal and File Transfer troubleshooting utilities to upload and execute the systemd-analyze binary to profile the initialization process of systemd services. These troubleshooting utilities can be used for variety of different tasks. After debugging a single device and finding an appropriate fix, Mender is capable of performing an A/B or delta software update to all devices or specific group of devices in the field.

Since the earliest days of the OpenEmbedded build framework and the Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product needs.

Setting up RAUC on CuBox-I/HummingBoard for Software Updates

/in , , /by

(This article was written by open source software enthusiast and Konsulko Group intern Atanas Bunchev, working with Konsulko Senior Engineer Leon Anavi.)

RAUC is one of the popular solutions that provide OTA (over-the-air) updates for Embedded Linux devices. RAUC is developed with focus on stability, security and flexibility and is compatible with all popular build systems: The Yocto Project/OpenEmbedded, Buildroot and PTXdist.

RAUC is capable of covering various use cases the most simple one being A/B updates.

The A/B updates scenario consists of having 2 identical root filesystems (named A and B), booting from one of them and performing the update on the other. After the update is complete the bootloader will boot from the updated partition on the next system boot. Recently the ‘verity’ update bundle format was introduced in RAUC. This new groundbreaking feature improves the verification process and most importantly allows extending RAUC by built-in HTTP(S) network streaming support, adaptive delta-like updates, and full update bundle encryption.

This article provides an example for setting up RAUC for A/B updates scenario on a HummingBoard board. The hardware used for the example is:

  • HummingBoard Pro board
  • 32GB microSD card
  • UART to USB adapter

RAUC is a robust, powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image built with the Yocto Project and OpenEmbedded for CuBox-I/HummingBoard one needs to:

  • Use U-Boot as a bootloader
  • Enable SquashFS in the Linux kernel configuration
  • Use ext4 root file system
  • Design specific storage partitioning for the certain use case and configure RAUC accordingly
  • Provide a custom U-Boot script to properly switch between RAUC slots
  • Prepare a certificate and keyring to use for signing and verifying RAUC update bundles.

Building a Linux Distribution with RAUC

I’ve recently contributed to meta-rauc-community, a repository containing minimal RAUC example layers for Yocto/OpenEmbedded.

The following steps will show how to use the meta-rauc-nxp layer from that repository to build and update a minimal Linux distribution. The update will install nano (the text editor) to the system.

Download the reference Yocto distribution, Poky.
We’ll use the latest long term support version, kirkstone.

$ git clone -b kirkstone https://git.yoctoproject.org/poky
$ cd poky

Download meta-rauc-community layers (meta-rauc-nxp):

$ git clone https://github.com/rauc/meta-rauc-community.git

Download the meta-rauc layer:

$ git clone -b kirkstone https://github.com/rauc/meta-rauc.git

Download the BSP layers for cubox-i/HumminbBoard boards:

$ git clone -b kirkstone https://git.yoctoproject.org/meta-freescale
$ git clone -b kirkstone https://github.com/Freescale/meta-fsl-arm-extra.git

Download the meta-openembedded layer (provides nano):

$ git clone -b kirkstone git://git.openembedded.org/meta-openembedded

Initialize the build environment:

$ source oe-init-build-env

Add the layers to conf/bblayers.conf:

$ bitbake-layers add-layer ../meta-openembedded/meta-oe
$ bitbake-layers add-layer ../meta-rauc
$ bitbake-layers add-layer ../meta-freescale
$ bitbake-layers add-layer ../meta-fsl-arm-extra
$ bitbake-layers add-layer ../meta-rauc-community/meta-rauc-nxp

Adjust conf/local.conf by adding the following configurations to the end of the file:

# HummingBoard specifications are very similar to Cubox-I
MACHINE = "cubox-i"

# Accept end user agreement required by the BSP layer.
ACCEPT_FSL_EULA = "1"

# Use systemd as init manager
INIT_MANAGER = "systemd"

# Add RAUC to the image
IMAGE_INSTALL:append = " rauc"
DISTRO_FEATURES:append = " rauc"

# Generate ext4 image of the filesystem
IMAGE_FSTYPES:append = " ext4"

# Use the file containing the partition table specification
WKS_FILE = "sdimage-dual-cubox-i.wks.in"
WKS_FILES:prepend = "sdimage-dual-cubox-i.wks.in "

# Add 150 000 KBytes free space to the root filesystem
# (Adding software with updates require space.)
IMAGE_ROOTFS_EXTRA_SPACE:append = " + 150000"

# Add the boot script to the boot partition
IMAGE_BOOT_FILES:append = " boot.scr"

Note that whitespace inside quotes is intentional and important.

To sign and verify the update bundles RAUC uses SSL keys. A keyring containing all keys that will be used for update bundles needs to be installed on the target.

meta-rauc-community provides a script that would generate example keys and configure the current build environment accordingly. (The script has to be called after sourcing oe-init-build-env)

$ ../meta-rauc-community/create-example-keys.sh

Build a minimal bootable image:

$ bitbake core-image-minimal

Building an image from scratch is a long process involving a lot of tasks. Please patiently wait until bitbake completes all tasks.

It’s strongly recommended to zero-fill the u-boot environment sectors before flashing the image on the microSD card (replace /dev/sdX with the proper device path):

$ dd if=/dev/zero of=/dev/sdX seek=2032 count=16

After the build is done, flash the image to a microSD card (replace /dev/sdX with the proper device path) and boot it on the HummingBoard:

$ bmaptool copy tmp/deploy/images/cubox-i/core-image-minimal-cubox-i.wic.gz /dev/sdX
$ sync
$ eject /dev/sdX

Attach the USB-to-UART adapter to the HummingBoard Pro, plug the ethernet cable and the microSD card. Turn on the board to verify that the system boots successfully.

By default one can login as root without password.

Creating an update bundle for RAUC

After sourcing the oe-init-build-env, append the following line to the build configuration conf/local.conf to add nano to the system:

# Adding nano
IMAGE_INSTALL:append = " nano"

Build the RAUC update bundle:

$ bitbake update-bundle

Start a web server:

$ cd tmp/deploy/images/cubox-i/
$ pip3 install --user rangehttpserver
$ python3 -m RangeHTTPServer

Now you can install the bundle on the board, then reboot:

# rauc install http://192.168.1.2:8000/update-bundle-cubox-i.raucb
# reboot

One of the latest RAUC features is the verity bundle format. This format allows updates to be done without storing the whole bundle on the device in advance, which is useful for devices with limited space. One of the requirements for this feature is hosting the bundle on a server that supports HTTP Range Requests.

As alternative, you can transfer the bundle to the device and install it from local storage.

Verify that nano was added to the system:

# which nano

Check RAUC status to confirm the system have booted from the second partition:

# rauc status

For real-world products, this build procedure with the Yocto Project and OpenEmbedded can be optimized further with just a few commands for easy implementation of continuous integration (CI).

Since the earliest days of the OpenEmbedded build framework and Yocto Project, Konsulko engineers have been contributing to the community and helping customers build commercial products with these technologies. We have experience with RAUC, Mender and other open source solutions for software updates. Please contact us to discuss your own embedded product development.

Konsulko Group: The Year in Review 2022

/in /by

2022 went by quickly with Konsulko engineers working closely with our customers, our partners and the open source community. For the tenth straight year, Konsulko Group has helped our clients build outstanding commercial products with Embedded Linux, Yocto Project and OpenEmbedded, as well as deploying Over-the-Air (OTA) software updating.

Engaging with our customers

We had particular success with our Konsulko Continuous Time Engagement™ offering, providing dedicated engineering resources for two of the world’s largest semiconductor companies. This model provides guaranteed engineering time for an agreed period. KCTE has allowed our customers to use our engineers on a variety of their requirements, and switch between these tasks as the rest of their project and in-house engineering required.

Of course, many of our clients prefer Konsulko OnDemand Time Engagement™, SOW-based engineering for high level consulting, on-demand support, and specific tasks within a larger project. KOTE is also best for longer term engagements without a hard deadline that can be stopped and started as necessary, and projects with an expected pause (such as bring-up of new hardware which almost always results in a re-spin of the PCB).

Partnerships and Conferences

We continue our strong relationship with the Linux Foundation and Automotive Grade Linux. We worked with mender.io and PHYTEC, providing support and development for their customers. With our friends at ICS, we presented at a joint webinar in August, Software Update Mechanisms: Selecting the Best Solution by Konsulko’s Leon Anavi and ICS’ Jeff Tranter.

Our Konsulko engineers were active participants (often in person, sometimes virtually) in conferences and developer gatherings. Leon Anavi, Vitaly and Maria Wool, and Tim Orling made presentations at the Embedded Linux Conferences (North America and Europe), FOSDEM and Yocto Project Summits:

Tim also posted a technical paper on Using kernel config fragments to remove an unwanted feature.

Two top engineers join the Konsulko team

The bar is pretty high for joining Konsulko Group. Some of our team have thirty years experience with embedded software and Linux. Most have over twenty and all have over fifteen years, working (and playing) with Linux and the open source community. So we were happy to welcome Denys Dmytriyenko and António Oliveira to Konsulko Group this year.

Denys is a long time Open Source Software developer, contributing code to many FOSS projects, like glibc, PostgreSQL, KDE, MythTV, LIRC, OpenEmbedded, and Yocto Project. Before joining Konsulko Group, he worked as Software Developer and Architect for Texas Instruments for almost 20 years on numerous Embedded Linux (MIPS and ARM-based) products. Since 2011 Denys has served on the OpenEmbedded Board of Directors, and since 2019, on the Yocto Project Technical Steering Committee. He is a long time contributor and maintainer of numerous recipes and layers in the OpenEmbedded/Yocto Project universe.

António has worked in development of highly optimized hardware and software ranging from deeply embedded microcontrollers to full-fledged graphical user interfaces. During this time, he has gained extensive experience in build automation tools, analog and digital circuits, powerline communications, and low power and low frequency radio communications. An active member of the open source community, he has contributed to Yocto Project and the Linux kernel. In addition to his work as an embedded engineer, António served eight years on the executive board of his local parish in Portugal, including four years on the municipal council.

We hope you will have the opportunity to work with Denys, António and the rest of Konsulko Group in 2023.

Embedded Linux engineers have a choice to make in June

/in /by

At Konsulko Group, we’ve participated in the “virtual” developer conferences created out of necessity over the past couple of years. We eagerly awaited the return of the face-to-face meetings of the embedded community.

Our first opportunity was the Embedded Linux Conference in Seattle last September. Although major components were still virtual, those who where able to attend in person experienced the event “together“ in ways that even the best remote simulation cannot achieve.

So we are very much looking forward to ELC (part of Open Source Summit North America 2022) in Austin, Texas, and the return of Embedded World in Nuremberg, Germany. Unfortunately, the pandemic initially forced rescheduling of both. Now they will occur on exactly the same dates (starting June 21, 2022) half a world apart, making it impossible to attend both onsite.

Which conference should you attend?

Unless you live in Texas or Germany, choosing by geography may not be the best option. The conferences are very different.

ELC is the premier event for companies and developers using Linux in embedded products. It gathers the technical experts working on embedded systems and applications for education and collaboration, historically paving the way for transformation in these far reaching areas.

ELC is also always a good time.

For 20 years, Embedded World has combined

  • an exhibition for engineers and technical management, and
  • a world-leading conference at the intersection of applied research and industrial applications.

The trade show is massive, filling several halls, with many of Konsulko’s friends and partners exhibiting year after year. (If you go, be sure to stop by the Automotive Grade Linux stand in Hall 4.)

We hope to see you at one of these important events in June.

Using kernel config fragments to remove an unwanted feature

/in , , , /by

Adding a feature to a linux-yocto based kernel is fairly well documented. This makes sense because it is the most common thing you might want to do: “My board needs support for this sensor added to our BSP.”

Konsulko Group recently helped a customer that had exactly the opposite problem, a standard feature in linux-intel (which includes linux-yocto.inc and inherits the kernel-yocto class) needed to be removed. The SoC (in the Intel™ “Bay Trail” family) and the off-the-shelf industrial PC had a problem. When USB 3.0 (xHCI) support is enabled, the default BIOS settings (xHCI Mode = Auto) would cause the system to lock-up upon either warm reboot or shutdown. Given that these systems are deeply embedded in the field—where simply “hooking up a display, keyboard and mouse” is cost prohibitive—we needed to find an option that would not prevent OTA updates.

You might think you could apply a patch via SRC_URI to kmeta (yocto-kernel-cache), but this isn’t supported in the Yocto Project kernel tooling. Instead we can apply kernel config fragments that disable the problematic xHCI feature. The trade off is that USB 3.x devices won’t be able to run at full speed, but the systems in question have no need for USB 3.0 (they are a classic IoT gateway use case).

Determining your existing kernel configuration

The first thing you want to do in this situation is determine what your (default) kernel configuration is. In our case, Konsulko and our customer are using the ‘dunfell’ (3.1.x) branch of meta-intel with MACHINE="intel-corei7-64". Upon building an image or the kernel (e.g. bitbake virtual/kernel or bitbake linux-intel), the kernel configuration can be found at the following path:

<build>tmp/work/corei7-64-intel-common-poky-linux/linux-intel/5.4.170+gitAUTOINC+98cce1c95f_36f93ff941-r0/linux-corei7-64-intel-common-standard-build/.config

Where the kernel version is 5.4.170, the (shortened) git commit hash of the kernel cache is 98cce1c95f and the (shortened) git commit hash of the kernel source is 36f93ff941.

Modifying the kernel configuration with menuconfig

The documented way to modify the kernel configuration is with:

bitbake -c menuconfig linux-intel

This approach works fine, but you must remember to copy the resulting .config to defconfig in your recipe’s SRC_URI.

Alternatively you can create your kernel config fragments in the kernel build directory and then add them to your kernel recipe. The fragments can be created with the help of the diffconfig script in the kernel source tree.

Also note that you must have already run:

$ bitbake -c kernel_configme -f linux-intel

or previously built the kernel in order for the .config to be present.

Modifying the kernel configuration with devtool

It will come as no surprise that my preferred way to modify the kernel configuration is to run:

$ devtool menuconfig linux-intel

But you will get an error:

ERROR: No recipe named 'linux-intel' in your workspace

So first we must get the kernel recipe into our workspace:

$ devtool modify linux-intel

Now we are able to run:

$ devtool menuconfig linux-intel

One benefit of this approach is that devtool will run the required steps that need to happen before menuconfig can be run (most notably the do_kernel_configme task).

Determining the changes needed

Regardless of which method you used to run menuconfig, you will now be presented with the text UI:
changes needed

Since Konsulko and our customer already knew we needed to change the xHCI enablement, we can quickly </> for Search and then enter xHCI.
changes

This gives us several results, but the ones that we care about have [=y](built-in) next to them:
changes

  1. CONFIG_USB_XHCI_HCD
  2. CONFIG_USB_XHCI_PCI
  3. CONFIG_USB_XHCI_PLATFORM

The top level item that needs to be disabled (set to N) is CONFIG_USB_XHCI_HCD:
changes

After this we can spot check the other values are also disabled (by using </> for Search again):
CONFIG_USB_XHCI_PCI and CONFIG_USB_XHCI_PLATFORM:
changes

Satisfied that we have the needed change, we can save our configuration. Press the <E> key or click on < Exit > to exit the sub-menus until you are at the top of the stack. Press the <S> key or click on < Save > to save the configuration. At they prompt, press enter or click on <Yes>.
changes

Now, the benefit of the devtool workflow comes into play, because we are rewarded with:

INFO: Updating config fragment <build>/workspace/sources/linux-intel/oe-local-files/devtool-fragment.cfg

The contents of this file are what you might expect:

# CONFIG_USB_XHCI_HCD is not set

Applying our configuration change

If we don’t already have one, we need a layer into which to put our changes:

$ bitbake-layers create-layer ~/Projects/meta-awesome-bsp

Add the layer to our active layers:

$ bitbake-layers add-layer ~/Projects/meta-awesome-bsp

Create a directory–following the pattern in openembedded-core–for our kernel changes:

$ mkdir -p ~/Projects/meta-awesome-bsp/recipes-kernel/linux

Finish our recipe:

$ devtool finish linux-intel ~/Projects/meta-awesome-bsp

Examine the resulting directory structure:

$ tree ~/Projects/meta-awesome-bsp
/home/<user>/Projects/meta-awesome-bsp
├── conf
│&nbsp;&nbsp; └── layer.conf
├── COPYING.MIT
├── README
├── recipes-example
│&nbsp;&nbsp; └── example
│&nbsp;&nbsp;     └── example_0.1.bb
└── recipes-kernel
    └── linux
        ├── linux-intel
        │&nbsp;&nbsp; └── devtool-fragment.cfg
        └── linux-intel_%.bbappend

Since this might not be the only change to the kernel we will need to make, let us give the fragment a better name:

pushd ~/Projects/meta-awesome/recipes-kernel/linux/linux-intel
mv devtool-fragment.cfg disable-xhci-hcd.cfg

And create an .scc file to give the Yocto Project kernel tooling better hints of how to apply our change:

cat << EOF >> disable-xhci-hcd.scc
# SPDX-License-Identifier: MIT
define KFEATURE_DESCRIPTION "Disable options for xhci (USB 3.0)"
define KFEATURE_COMPATIBILITY board

kconf hardware disable-xhci-hcd.cfg
EOF

And finally make changes to our linux-intel_%.bbappend to reflect these files:

$ cd ..
$ cat linux-intel_%.bbappend
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

SRC_URI += "\
    file://disable-xhci-hcd.cfg \
    file://disable-xhci-hcd.scc \
"

Now—as long as our layer has high enough priority and no other recipes add kernel config fragments which conflict with our change—we should be able to build the kernel and inspect the resulting .config:

$ popd
$ bitbake linux-intel
$ grep -R XHCI tmp/work/corei7-64-intel-common-poky-linux/linux-intel/5.4.170+gitAUTOINC+98cce1c95f_36f93ff941-r0/linux-corei7-64-intel-common-standard-build/.config
# CONFIG_USB_XHCI_HCD is not set
# CONFIG_USB_ROLES_INTEL_XHCI is not set

We should also be able to run dmesg | grep xhci on the target and we would not expect to see any messages.

Summary

A Konsulko Group customer had a hardware problem that required us to remove a kernel feature. By using tools like devtool, we were able to fairly easily make a change to the kernel configuration and capture those changes in a persistent way with a .cfg fragment and a .bbappend. This approach solved the problem and allowed OTA updates to proceed to these deeply embedded devices in the field. Please contact us to discuss how Konsulko can help you with the unique requirements of your commercial project.

Konsulko Group: The Year in Review 2021

/in /by

Thanks to our customers, our partners and our dedicated team of engineers, 2021 was a very successful year for Konsulko Group. 

We’ve been chosen to work on important projects in consumer electronics, automotive, medical devices, agriculture, mining, finance, and autonomous vehicles. Our engineers helped our clients, new and old, build outstanding commercial products with Embedded Linux, Yocto Project and OpenEmbedded, as well as deploying Over-the-Air (OTA) software updating.

Partnerships

“Konsulko is a recommended and trusted partner for helping Mender customers and prospects succeed in the Embedded Linux space. They have an excellent team of professionals who will deliver on time and as agreed.”

Thomas Ryd, CEO, mender.io

We continued our strong relationships with mender.io, and with the Linux Foundation and Automotive Grade Linux (AGL). 

We also launched a new partnership with PHYTEC at the Embedded Linux Conference (ELC) in Seattle. Konsulko is providing support and development for PHYTEC customers using their Systems on Modules (SOMs) and Yocto Project Linux distribution.

Conferences

As in previous years, Konsulko engineers were active participants (often virtually) in conferences and developer gatherings, making presentations at ELC and Yocto Project Summits, and writing technical papers on Yocto Project, security, and OTA. We hope for a time in the not-too-distant future when it makes sense for developers from all over the world to gather again face-to-face to share ideas. 

Contributions

As many of our engineers have been working (and playing) with Open Source Software (OSS) for over 20 years, Konsulko Group is proud of our continuing commitment to contribute to the community. Last month we were happy to hear that Konsulko is #3 in contributions to Yocto Project “Non Core,” just behind industry giants ARM and Fujitsu, and ahead of every other software or hardware company.

This was even before we welcomed OSS veteran Tim Orling to the team in December. Tim was recently Core OS Architect and Yocto Project Architect for Intel Corporation, and serves as a maintainer of meta-python and many recipes in openembedded-core. He has authored hundreds of applied patches for openembedded-core and meta-openembedded. 

Doing business with Konsulko Group

As a team we have found that the best way to work with our customers and prospective customers is to provide a high level of transparency about our business processes. This year, we’ve published blogs on scoping the effort required to successfully complete a customer project, and another describing our two models of engagement: 

  • Konsulko Continuous Time Engagement™ for dedicated engineering resources
  • Konsulko OnDemand Time Engagement™ for high level consulting, on-demand support, and specific tasks within a larger project.

We hope you have found these useful, and we look forward to working with you in 2022.

Choosing the best model to engage with Konsulko Group

/in /by

As a consulting and services company, Konsulko Group is committed to acting as our customer’s trusted advisor. That extends across our entire business – from the technologies and services we recommend, to our time tracking and billing practices. We’re committed to running our business with the utmost integrity and transparency, following the best industry practices. 

Over the past couple of years, we have written about Konsulko’s Globally Employable Engineers™ based throughout the world and working together remotely (something we have done successfully long before the pandemic). Members of our team have years of software engineering experience, excellent English language skills, and are comfortable working directly with our customers. 

Konsulko engineers could get good jobs anywhere, but they choose to live in their home country, or elsewhere in North America or Europe. Demand for their skills is strong and tapping their talent does not come at an offshore salary cost, so understanding Konsulko’s engagement models can be crucial to matching resources to the job at hand.

Konsulko Continuous Time Engagement™ 

Under this model, sometimes referred to as “dedicated engineering” or “staff augmentation” in the industry, Konsulko provides dedicated and thus guaranteed engineering time for an agreed period. We will allocate the resource(s) to you, whether it is on a full time basis or some other number of hours per week, and those resources will be dedicated to you. 

As such, we will start invoicing for our time as soon as the resources are allocated to you and there is a signed Master Service Agreement (MSA), Statement of Work (SOW) and purchase order (PO) in place. We will not accrue billable time if any of these resources are out for personal reasons, holidays or vacation.

We will, however, accrue all other time, even if we’re blocked from making progress for reasons outside of our control, such as, hardware problems, VPN or other IT issues, access to your internal infrastructure when necessary, or lack of adequate documentation. This guarantees that our engineering resource stays dedicated to your project, so as soon as we get “unblocked” the work can immediately resume.

The Continuous Time Engagement model works well for larger projects with multiple requirements and deliverables,  projects where the requirements are fluid and you need maximum flexibility, or any project where the schedule calls for dedicated resources for the duration of the work. Customers who are willing to get directly involved in managing the hand-offs and task alignment between their own engineering team and Konsulko engineers often find this an efficient way to work with us.

Konsulko OnDemand Time Engagement™

Under this model of engagement, you will be invoiced only for actual time we spend on the project. If we estimate a job will take six engineer weeks and it only takes five, you pay for five weeks not six. Startup time setting up your hardware or working with your IT department to enable access to your infrastructure is considered part of the project and thus billable hours. 

If we get blocked from making progress for reasons such as those mentioned above, we stop accruing time until we are unblocked, and our resources will likely be reassigned to another project. Although we’ll do our best to switch them back to your project as soon as we’re re-enabled, we cannot guarantee the length of the delay, or the time required to get back up to full speed. 

The OnDemand Time Engagement can be best for a large number of projects:

  • Specific tasks within your larger project 
  • Longer term projects without a hard deadline that can be stopped and started as necessary
  • Projects where you reasonably expect to stop and restart the work (such as bring-up of new hardware which almost always results in a respin of the PCB)
  • On-demand support
  • High level consulting

Which is best for you? When you contact us, we will work with you to determine which model works for your requirements and timeline, and do our best to make your engagement with Konsulko Group efficient and memorable for all the right reasons.

The black magic of scoping the effort in software projects

/in /by

The software industry is notorious for missed schedules and project budgets that end up being double, triple or even higher than original estimates. One of the first managers I worked for, early in my career, joked that when an engineer gives you an estimate, you should double it and then move to the next unit of measure. In his mind, a two weeks estimate thus became four months. This was obviously an exaggeration, but it does point to the magnitude of the problem.

Three reasons for missing an estimate

There are multiple sources for missed estimates. First, there’s the shear complexity of scoping the effort of large projects. 

Second, engineers often think “dev done / it sort of works” as the end point when estimating the time it will take them. 

Lastly, there is the explicit or implicit pressure most engineers feel to come up with an estimate that will be palatable to upper management. The problem is even greater, I believe, if you consider the long hours engineers put in every day, plus many weekends and sometimes even holidays, in order to attempt to make the impossible schedules they’ve signed up for.

Missing schedules and blowing up budgets is not a luxury a professional services company has. Roughly 85%-90% of our projects end up under budget, and even the few that end up over budget are for reasons outside of our control. While there is no precise formula to use to properly scope the effort, I’ll share just a few notes that developers in our space may find useful.

Make sure you make the right assumptions

Your estimates are based on assumptions: 

  • working hardware (no glitches with the components or the PCB itself), 
  • deliverables owed to you by the customer, 
  • documentation that you might not be able to obtain easily and the customer will have to get it for you (due to strict NDAs, for example), 
  • any possible software deliverables by the customer to you, and so on. 

Start by listing your assumptions and make it known that if any of them prove wrong, additional time and budget will likely be required.

Look at specific tasks in the project and lean on past experience when you can

Break down the project into manageable tasks that you think you can estimate but lean back on past experiences with similar projects. One example that developers in the embedded Linux space will recognize is uplifting a BSP from a certain kernel version to the latest stable (same hardware, no new board spins to deal with). It’s easy to look at the list of, say, 10 device drivers that “should just work” and estimate that each one will take a day or two only, and 90% of the time you’ll probably be wrong. 

What we know from experience is that when doing similar work, it took on average, for example, one week per driver. Some may take a few hours; but others will take two weeks, and you don’t know which ones ahead of time. Don’t let the customer pick your estimates apart. Group this effort into a single task, “Uplifting the following device drivers:” and provide only a single estimate for the task.

Take a high level view

If you can’t break down the project into small tasks or perhaps it’s hard or impossible to estimate them, then step back and take a very high level view, or use a mix of approaches. 

We recently completed a project that included, among other tasks, bring-up of brand new hardware and writing some new, rather complex, Linux device drivers from scratch. The drivers we estimated separately, based on prior experience with these types of drivers and the corresponding Linux subsystem. As always, we listed our assumptions. 

The hardware bring-up, on the other hand, is impossible to estimate with precision. Instead, we used completely different high level heuristics once again based on 20+ years of industry experience: looking at past projects of similar complexity, how much time the customer took to go from first board spin to production, how much time we needed to get our work done, how much time the customer required for support and so on. 

Using such heuristics, we came up with an estimate for bringing up the hardware through the first spin (there is always a respin of new boards) and the writing the new drivers. We beat that estimate by a couple of weeks. 

Consider making the estimate Phase 0 of the project

Sometimes you just don’t know until you start working on the project and you’re two or four or more weeks into it. If that’s the case, make that a Phase 0 with the deliverable being a well defined and fleshed out effort estimate and project schedule, one that you can sign up for with a higher degree of confidence. 

These are a few of the ways we look at estimating a software project. As you can see, it’s not “black magic” after all but a multi-faceted approach to accurately scoping the effort. Contact us to learn more and prepare an estimate for your project.