Choosing the best model to engage with Konsulko Group

/by

As a consulting and services company, Konsulko Group is committed to acting as our customer’s trusted advisor. That extends across our entire business – from the technologies and services we recommend, to our time tracking and billing practices. We’re committed to running our business with the utmost integrity and transparency, following the best industry practices. 

Over the past couple of years, we have written about Konsulko’s Globally Employable Engineers™ based throughout the world and working together remotely (something we have done successfully long before the pandemic). Members of our team have years of software engineering experience, excellent English language skills, and are comfortable working directly with our customers. 

Konsulko engineers could get good jobs anywhere, but they choose to live in their home country, or elsewhere in North America or Europe. Demand for their skills is strong and tapping their talent does not come at an offshore salary cost, so understanding Konsulko’s engagement models can be crucial to matching resources to the job at hand.

Konsulko Continuous Time Engagement™ 

Under this model, sometimes referred to as “dedicated engineering” or “staff augmentation” in the industry, Konsulko provides dedicated and thus guaranteed engineering time for an agreed period. We will allocate the resource(s) to you, whether it is on a full time basis or some other number of hours per week, and those resources will be dedicated to you. 

As such, we will start invoicing for our time as soon as the resources are allocated to you and there is a signed Master Service Agreement (MSA), Statement of Work (SOW) and purchase order (PO) in place. We will not accrue billable time if any of these resources are out for personal reasons, holidays or vacation.

We will, however, accrue all other time, even if we’re blocked from making progress for reasons outside of our control, such as, hardware problems, VPN or other IT issues, access to your internal infrastructure when necessary, or lack of adequate documentation. This guarantees that our engineering resource stays dedicated to your project, so as soon as we get “unblocked” the work can immediately resume.

The Continuous Time Engagement model works well for larger projects with multiple requirements and deliverables,  projects where the requirements are fluid and you need maximum flexibility, or any project where the schedule calls for dedicated resources for the duration of the work. Customers who are willing to get directly involved in managing the hand-offs and task alignment between their own engineering team and Konsulko engineers often find this an efficient way to work with us.

Konsulko OnDemand Time Engagement™

Under this model of engagement, you will be invoiced only for actual time we spend on the project. If we estimate a job will take six engineer weeks and it only takes five, you pay for five weeks not six. Startup time setting up your hardware or working with your IT department to enable access to your infrastructure is considered part of the project and thus billable hours. 

If we get blocked from making progress for reasons such as those mentioned above, we stop accruing time until we are unblocked, and our resources will likely be reassigned to another project. Although we’ll do our best to switch them back to your project as soon as we’re re-enabled, we cannot guarantee the length of the delay, or the time required to get back up to full speed. 

The OnDemand Time Engagement can be best for a large number of projects:

  • Specific tasks within your larger project 
  • Longer term projects without a hard deadline that can be stopped and started as necessary
  • Projects where you reasonably expect to stop and restart the work (such as bring-up of new hardware which almost always results in a respin of the PCB)
  • On-demand support
  • High level consulting

Which is best for you? When you contact us, we will work with you to determine which model works for your requirements and timeline, and do our best to make your engagement with Konsulko Group efficient and memorable for all the right reasons.

The black magic of scoping the effort in software projects

/by

The software industry is notorious for missed schedules and project budgets that end up being double, triple or even higher than original estimates. One of the first managers I worked for, early in my career, joked that when an engineer gives you an estimate, you should double it and then move to the next unit of measure. In his mind, a two weeks estimate thus became four months. This was obviously an exaggeration, but it does point to the magnitude of the problem.

Three reasons for missing an estimate

There are multiple sources for missed estimates. First, there’s the shear complexity of scoping the effort of large projects. 

Second, engineers often think “dev done / it sort of works” as the end point when estimating the time it will take them. 

Lastly, there is the explicit or implicit pressure most engineers feel to come up with an estimate that will be palatable to upper management. The problem is even greater, I believe, if you consider the long hours engineers put in every day, plus many weekends and sometimes even holidays, in order to attempt to make the impossible schedules they’ve signed up for.

Missing schedules and blowing up budgets is not a luxury a professional services company has. Roughly 85%-90% of our projects end up under budget, and even the few that end up over budget are for reasons outside of our control. While there is no precise formula to use to properly scope the effort, I’ll share just a few notes that developers in our space may find useful.

Make sure you make the right assumptions

Your estimates are based on assumptions: 

  • working hardware (no glitches with the components or the PCB itself), 
  • deliverables owed to you by the customer, 
  • documentation that you might not be able to obtain easily and the customer will have to get it for you (due to strict NDAs, for example), 
  • any possible software deliverables by the customer to you, and so on. 

Start by listing your assumptions and make it known that if any of them prove wrong, additional time and budget will likely be required.

Look at specific tasks in the project and lean on past experience when you can

Break down the project into manageable tasks that you think you can estimate but lean back on past experiences with similar projects. One example that developers in the embedded Linux space will recognize is uplifting a BSP from a certain kernel version to the latest stable (same hardware, no new board spins to deal with). It’s easy to look at the list of, say, 10 device drivers that “should just work” and estimate that each one will take a day or two only, and 90% of the time you’ll probably be wrong. 

What we know from experience is that when doing similar work, it took on average, for example, one week per driver. Some may take a few hours; but others will take two weeks, and you don’t know which ones ahead of time. Don’t let the customer pick your estimates apart. Group this effort into a single task, “Uplifting the following device drivers:” and provide only a single estimate for the task.

Take a high level view

If you can’t break down the project into small tasks or perhaps it’s hard or impossible to estimate them, then step back and take a very high level view, or use a mix of approaches. 

We recently completed a project that included, among other tasks, bring-up of brand new hardware and writing some new, rather complex, Linux device drivers from scratch. The drivers we estimated separately, based on prior experience with these types of drivers and the corresponding Linux subsystem. As always, we listed our assumptions. 

The hardware bring-up, on the other hand, is impossible to estimate with precision. Instead, we used completely different high level heuristics once again based on 20+ years of industry experience: looking at past projects of similar complexity, how much time the customer took to go from first board spin to production, how much time we needed to get our work done, how much time the customer required for support and so on. 

Using such heuristics, we came up with an estimate for bringing up the hardware through the first spin (there is always a respin of new boards) and the writing the new drivers. We beat that estimate by a couple of weeks. 

Consider making the estimate Phase 0 of the project

Sometimes you just don’t know until you start working on the project and you’re two or four or more weeks into it. If that’s the case, make that a Phase 0 with the deliverable being a well defined and fleshed out effort estimate and project schedule, one that you can sign up for with a higher degree of confidence. 

These are a few of the ways we look at estimating a software project. As you can see, it’s not “black magic” after all but a multi-faceted approach to accurately scoping the effort. Contact us to learn more and prepare an estimate for your project.

Building Platforms with Secure Over-the-Air Updating

/by

Almost every device in development today requires both over-the-air (OTA) updating capabilities, and up-to-date security with authentication of the device filesystem and encryption of application data storage.

At Konsulko Group, we find that the best approach is to build a secure platform first, starting with the hardware root of trust and establishing a chain of trust by extending the root of trust through each successive component in the system.

  • A first step is to make sure the Universal Boot Loader (U-Boot) is up-to-date. If the device is using an older version, we may port U-Boot support of a more current release. Then we enable signature-based authentication of the device using the U-Boot verified boot feature.
  • To provide authentication of the device filesystem and encryption of application data storage, we often use the Linux kernel Device-Mapper infrastructure to create virtual layers of block devices: dm-verity for root filesystem authentication, dm-crypt for data encryption, and dm-integrity for read/write data volume integrity.
  • Finally, we can integrate Mender I/O support for OTA together with dm-verity and Yocto Project so that device updates can be performed while maintaining the system’s secure chain of trust.

This step-by-step methodology ensures straight-forward and predictable development. Please contact us to discuss how we can help you build a secure platform for OTA updating on your device.

CEO Pete Popov looks back on 2020

/by

As we approach the end of December, it’s time to review a year that we will certainly remember for the rest of our lives. It hasn’t been easy, but all of us at Konsulko Group are still working hard, supporting the open source community and helping our customers build forward-looking products.

Even at the start of 2020, we knew this year would be different. The coronavirus was the talk of CES in January with some companies pulling out at the last minute, and everyone wondering what the global business climate would be in the months ahead.

By the time FOSDEM rolled around a few weeks later, it was clear the virus would disrupt commerce worldwide, and by the end of February, we had to cut short our presence at Embedded World because of new travel restrictions.

Then the world locked down completely. Since at Konsulko Group we all work remotely by design, we didn’t have to adjust our way of developing software, but as it did for everyone, we had to significantly change our face-to-face participation in embedded Linux, Yocto Project and other community events.

We taught ourselves to use video editing tools, and gave “virtual” presentations from our desks at the Embedded Linux Conference North America and Yocto Project Dev Day at the end of June, the virtual Automotive Grade Linux All Member Meeting in mid July, Linaro Connect in September, Virtual ELCE and Yocto Project Virtual Summit Europe at the end of October, and participated in the virtual Automotive Linux Summit the first of December.

In December, we also presented an AGL Webinar, Getting Started with AGL using Raspberry Pi.

Early in the year, we announced that we had become a Mender Authorized Referral Partner, and that important alliance has provided dividends to both Konsulko Group and our customers as the year progressed.

Konsulko engineers continued our series of technical blogs…

Helping Yocto Project work with Python 3

Getting Started with RAUC on Raspberry Pi

How Mender works

Using Rust with Yocto Project

Building a DIY SOHO router, 18 months later

…and we posted six new videos of our presentations:

Building Containers with OpenEmbedded

Highly Scalable Yocto Project Build Automation

Security Hardening with OpenEmbedded / Yocto Project

Open Source License Compliance with Yocto Project

Demo: Using Rust with Yocto Project

Software Update Solutions for Yocto and OpenEmbedded

Still, we have all been touched by the physical and emotional toll of COVID-19. Two of our engineers have endured a bout with the virus (and thankfully recovered). Some of us have family, friends or acquaintances who have become seriously ill or even passed away. We can only hope the post-pandemic world is now in sight.

In the long run, the challenges that 2020 have brought us closer together in many ways, and hopefully taught us valuable lessons that will make us stronger in a better new year.

KonsulkoGroup12

Building a DIY SOHO router, 18 months later

/by

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, 18 months later

It’s been around a year since my last post on this project, and 18 months since I posted the article series itself. In the world of home networking equipment that’s practically a lifetime. So it’s worth doing a check-in on this project, I think.

My little router is still going just fine. In terms of things I had talked about earlier in the series, I’m still using the ability to easily roll back a test image to try out backports. Most recently, I tried to add IPv6 Prefix Delegation support to the DHCPv6 client in an older version of systemd. It didn’t work, but with just a reboot I was back to my normal image. As times change, devices at home change, and I’m using the dnsmasq configuration file to document what devices are in the house. Performance? Still doing just great. The next time I do one of these series, my household might have crossed the threshold where newer WiFi standards are in enough devices we own that I want something that handles them, but we aren’t there yet. By then I might also just have a new PCIe card to drop in.

In terms of the article series itself, it’s a testament to the stability of the Yocto Project and everyone who works on it. Since the original posts, Mender has internally upgraded a significant amount, and in turn some of the examples there aren’t quite correct for a modern build. Mender has excellent documentation, however, and it’s been easy to update my build to work all along the way and upgrade from the older version to the new. Everything else? Yes, that’s all still correct. What prompted this particular post is that the Yocto Project has announced another milestone release, and that’s always a good time to make sure my device is up to date. My latest changes were quite literally just renaming the Linux kernel bbappend file and saying the layer is compatible with the new release. That was less work than updating the firewall rules to take in to account both Daylight Savings Time changes and that my kids are a bit older and realistically want to be online a little later.

USING RUST WITH YOCTO PROJECT

Using Rust with Yocto Project

/by
by Paul Barker, Konsulko Group

Introduction

Rust is a modern programming language which emerged from Mozilla Research around 2010 and has been stable since 2015 (see the history section of the Wikipedia article). Rust is an excellent fit for embedded software development due to its focus on performance and safety without the overhead of garbage collection or a large language runtime. The zero-cost abstractions provided by Rust support developer productivity, allowing the use of traits, generics, functional programming styles and other high-level approaches in the knowledge that the resulting assembly code will perform just as well as more verbose, hand written lower level code.

Now is a great time to pick up Rust if you work in the Embedded space. The Rust Embedded Working Group has produced an Embedded Rust Book for an audience with existing embedded development knowledge and the Discovery Book for an audience with no embedded experience but a desire to learn. The Linux kernel community has began discussing the use of Rust within the kernel itself and the OpenEmbedded/Yocto Project community is planning to move Rust support into the openembedded-core layer during the next release cycle.

The benefits of Rust largely stem from the language’s memory model, the borrow checker and other features which will be new to most developers learning Rust. The above resources from the Rust Embedded WG along with several published books and online tutorials cover this material well and so it will not be repeated in detail here. However, if you wish to develop software in Rust for production deployment you should expect to invest at least a few weeks in learning the language, assuming existing familiarity with other programming languages like C, Python or Java.

This article will introduce a couple of simple example applications written in Rust, discuss how to compile and run these applications natively on a Linux system using the Cargo build system and show how to create Yocto Project recipes for these applications using the cargo-bitbake tool. Some basic knowledge of the Yocto Project is assumed but no previous experience of Rust should be required.

Getting Started with Rust

In this section we will install Rust on our Linux system and create two simple applications.

Note that it is not necessary to have Rust installed natively in order to build applications and libraries written in Rust under bitbake. However, installing Rust natively does allow us to develop, test and debug applications on the host before we try to cross-compile them and it is an important step to take when learning Rust.

Installing Rust

The recommended way of installing Rust on a Linux system is to download and run the official installation script. When prompted with the installation options it is usually sufficient to select option 1 to use the defaults. This will install Rust in the current user’s home directory and modify the profile files to set necessary environment variables at login.

$ curl https://sh.rustup.rs -sSf | sh

    <output snipped>

Current installation options:


   default host triple: x86_64-unknown-linux-gnu
     default toolchain: stable (default)
               profile: default
  modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>1

    <output snipped>

Rust is installed now. Great!

To get started you need Cargo's bin directory ($HOME/.cargo/bin) in your PATH
environment variable. Next time you log in this will be done
automatically.

To configure your current shell run source $HOME/.cargo/env

After installing Rust you should log out and log back in to ensure that environment variables are set correctly.

Hello, World

As is traditional, we will begin by looking at a “Hello, World” application.

Each Rust application or library should have its own project directory. Within this directory, the source code is usually placed in a src directory and the top level directory contains the Cargo config file for the project
and any other collateral files such as a README file, LICENSE file, etc.

To create our project directory and populate it with initial files we can use the cargo init command:

$ cargo init hello-rs
     Created binary (application) package

Within our new hello-rs directory, this command creates an initial source file in src/main.rs with the following contents:

fn main() {
    println!("Hello, world!");
}

This makes our job of writing a Hello World application very simple – no code modifications are needed!

The cargo init command also creates a Cargo.toml file in our project directory to store project configuration and metadata:

[package]
name = "hello-rs"
version = "0.1.0"
authors = ["Paul Barker <pbarker@konsulko.com>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

We should test our new Hello World application natively before we try to cross-compile it under bitbake. To do this we can use the cargo build and cargo run commands:

$ cargo build
   Compiling hello v0.1.0 (/home/pbarker/Projects/Rust/rust/hello-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 0.29s
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/hello-rs`
Hello, world!

The last stage for our hello world application is to publish it as a git repository so that it can be fetched and built in our Yocto Project environment. We should create a LICENSE file and a README file in the top level of the project for the benefit of anyone who wants to reuse our code. My preference is for the Apache license version 2.0 but feel free to use any appropriate open source license here.

We should also add some extra metadata to the Cargo.toml file to specify the license, briefly describe the project and link to the location where we will be publishing the repository. This metadata will be used by cargo-bitbake when we generate a recipe for our application. In my case the repository will be published to my GitLab account.

--- a/Cargo.toml
+++ b/Cargo.toml
@@ -3,6 +3,9 @@ name = "hello-rs"
 version = "0.1.0"
 authors = ["Paul Barker <pbarker@konsulko.com>"]
 edition = "2018"
+license = "Apache-2.0"
+description = "Hello World application"
+repository = "https://gitlab.com/pbarker.dev/rust/hello-rs.git"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

To finish up, commit the project files to a git repository and push to a location which can be reached when building using Yocto Project. Alternatively, you can make use of the hello-rs repository I published under my GitLab account.

The Hello World application above will demonstrate only part of the capabilities of the cargo-bitbake tool and the Rust support in Yocto Project. To see how Rust dependencies are handled in Yocto Project we’ll need an application which actually depends on a Rust library (known as a crate). The print-rand application will fill this role as it uses the rand crate to generate random numbers.

As before we will start by creating the project using cargo init:

$ cargo init print-rand
     Created binary (application) package

We will now modify the src/main.rs file to generate and print a random number:

use rand::prelude::*;

fn main() {
    let x: u32 = random();
    println!("x = {}", x);
}

We will also modify the Cargo.toml file to list the rand crate as a dependency:

[package]
name = "print-rand"
version = "0.1.0"
authors = ["Paul Barker <pbarker@konsulko.com>"]
edition = "2018"

[dependencies]
rand = "0.7"

We can build and test this application in a similar way to the previous Hello World app. Note that each dependency is downloaded and built along the way:

$ cargo build
   Compiling libc v0.2.77
   Compiling getrandom v0.1.15
   Compiling cfg-if v0.1.10
   Compiling ppv-lite86 v0.2.9
   Compiling rand_core v0.5.1
   Compiling rand_chacha v0.2.2
   Compiling rand v0.7.3
   Compiling print-rand v0.1.0 (/home/pbarker/Projects/Rust/rust/print-rand)
    Finished dev [unoptimized + debuginfo] target(s) in 3.60s
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/print-rand`
x = 643356417

As before, we should add LICENSE and README files at this stage and extend the metadata in the Cargo.toml file:

--- a/Cargo.toml
+++ b/Cargo.toml
@@ -3,6 +3,9 @@ name = "print-rand"
 version = "0.1.0"
 authors = ["Paul Barker <pbarker@konsulko.com>"]
 edition = "2018"
+license = "Apache-2.0"
+description = "Print a random number"
+repository = "https://gitlab.com/pbarker.dev/rust/print-rand.git"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

Lastly, commit the project files to a git repository and push to a public location. As before you may alternatively make use of the print-rand repository I published under my GitLab account.

Using cargo-bitbake

Now that we have created a couple of simple Rust applications and published them we can look at creating bitbake recipes using the cargo-bitbake tool. The first step of this process is to install the tool itself:

$ cargo install cargo-bitbake

    <output snipped>

   Compiling cargo-bitbake v0.3.14
    Finished release [optimized] target(s) in 12m 30s
  Installing /home/debian/.cargo/bin/cargo-bitbake
   Installed package `cargo-bitbake v0.3.14` (executable `cargo-bitbake`)

Before we run cargo bitbake for each our projects we should ensure that all changes are committed and pushed to the remote repository. We should also check the remote URL for each repository as this will be used to set SRC_URI. If the remote URL uses SSH access then you should ensure that this can be reached using an SSH key from the Yocto build environment or switch to HTTPS access.

Once we’re happy with the state of our repository we can run cargo bitbake in the top directory of our hello-rs project (note that we use a space not a dash in the command here):

$ cargo bitbake
No package.homepage set in your Cargo.toml, trying package.repository
Wrote: hello_0.1.0.bb

The message printed by cargo bitbake indicates that the repository entry in our Cargo.toml file was used as the homepage URL since we did not specify a separate homepage.

We can now look at the generated recipe file hello-rs_0.1.0.bb:

# Auto-Generated by cargo-bitbake 0.3.14
#
inherit cargo

# If this is git based prefer versioned ones if they exist
# DEFAULT_PREFERENCE = "-1"

# how to get hello-rs could be as easy as but default to a git checkout:
# SRC_URI += "crate://crates.io/hello-rs/0.1.0"
SRC_URI += "git://gitlab.com/pbarker.dev/rust/hello-rs.git;protocol=https;nobranch=1;branch=dev"
SRCREV = "cec42fb0d147e4a4b271256c475620ff627c8856"
S = "${WORKDIR}/git"
CARGO_SRC_DIR = ""


# please note if you have entries that do not begin with crate://
# you must change them to how that package can be fetched
SRC_URI += " \
"



# FIXME: update generateme with the real MD5 of the license file
LIC_FILES_CHKSUM = " \
    file://LICENSE;md5=86d3f3a95c324c9479bd8986968f4327 \
"

SUMMARY = "Hello World application"
HOMEPAGE = "https://gitlab.com/pbarker.dev/rust/hello-rs.git"
LICENSE = "Apache-2.0"

# includes this file if it exists but does not fail
# this is useful for anything you may want to override from
# what cargo-bitbake generates.
include hello-rs-${PV}.inc
include hello-rs.inc

As we noted above, the SRC_URI and SRCREV entries are based on the remote URI and the currently checked out commit of our local git repository. The SUMMARY, HOMEPAGE and LICENSE entries are based on the metadata in our Cargo.toml file. The LIC_FILES_CHKSUM entry is set based on the LICENSE file in our project.

We can also generate a recipe for the print-rand project by running cargo bitbake in the top directory of this project:

$ cargo bitbake
No package.homepage set in your Cargo.toml, trying package.repository
Wrote: print-rand_0.1.0.bb

The print-rand_0.1.0.bb recipe is very similar to the previous recipe for our hello world application. The SRC_URI, SRCREV, SUMMARY and HOMEPAGE are changed to the appropriate values for this project. The SRC_URI variable is extended with a list of the dependency crates required by this project, this includes not just the rand crate that we explicitly listed as a dependency in our Cargo.toml file but also all the recursive dependencies of this crate.

# Auto-Generated by cargo-bitbake 0.3.14
#
inherit cargo

# If this is git based prefer versioned ones if they exist
# DEFAULT_PREFERENCE = "-1"

# how to get print-rand could be as easy as but default to a git checkout:
# SRC_URI += "crate://crates.io/print-rand/0.1.0"
SRC_URI += "git://gitlab.com/pbarker.dev/rust/print-rand.git;protocol=https;nobranch=1;branch=dev"
SRCREV = "3397247f929f28d70adbb65d3990dc72699553bb"
S = "${WORKDIR}/git"
CARGO_SRC_DIR = ""


# please note if you have entries that do not begin with crate://
# you must change them to how that package can be fetched
SRC_URI += " \
    crate://crates.io/cfg-if/0.1.10 \
    crate://crates.io/getrandom/0.1.15 \
    crate://crates.io/libc/0.2.79 \
    crate://crates.io/ppv-lite86/0.2.9 \
    crate://crates.io/rand/0.7.3 \
    crate://crates.io/rand_chacha/0.2.2 \
    crate://crates.io/rand_core/0.5.1 \
    crate://crates.io/rand_hc/0.2.0 \
    crate://crates.io/wasi/0.9.0+wasi-snapshot-preview1 \
"



# FIXME: update generateme with the real MD5 of the license file
LIC_FILES_CHKSUM = " \
    file://LICENSE;md5=86d3f3a95c324c9479bd8986968f4327 \
"

SUMMARY = "Print a random number"
HOMEPAGE = "https://gitlab.com/pbarker.dev/rust/print-rand.git"
LICENSE = "Apache-2.0"

# includes this file if it exists but does not fail
# this is useful for anything you may want to override from
# what cargo-bitbake generates.
include print-rand-${PV}.inc
include print-rand.inc

We can now add these recipes to a Yocto Project layer and build them using bitbake. Support for the Rust toolchain in Yocto Project is provided by the meta-rust layer so this layer must be included before these recipes can be built. As there are many ways to put together a Yocto Project build we won’t cover this in this blog post. However, you can see a walkthrough of this process in the Using Rust with Yocto Project demo presented at theYocto Project Summit Europe 2020.

How Mender works

/by

by Tom Rini, VP Engineering

Software Update solutions are a key part of our services offering. For open source over-the-air updates, we often we recommend and work with mender.io. In fact, Konsulko Group is a Mender Authorized Referral Partner. Recently, a prospective customer expressed an interest in knowing more about how Mender works. Here’s the brief, informal introduction to member.io that I prepared, and now I’m sharing with you.

—

As a high level starting point, https://mender.io/how-it-works provides a good overview of what’s supported and what it covers. In short, Mender starts off by providing support for a traditional “A/B” approach to system updates, where if the update isn’t marked as valid (and there’s hooks for the application(s) to verify the system before this is done), it’s assumed invalid and the system will roll-back automatically. While “OTA” implies over the network, it can just as easily be done by providing (and validating) a USB key that contains an update.

One of the reasons we recommend Mender is that it has very good in-depth documentation. The starting point for all of that is https://docs.mender.io/2.4/ which covers all of the topic starting from how to implement Mender support in a device and including how to create your own server infrastructure if you don’t want to use their paid service. While there are a number of important pages there, one that I like to highlight is https://docs.mender.io/2.4/artifacts/state-scripts as it shows the state machine for an update and talks about some of the common use cases that come up for user interaction or dealing with failures.

Another place I want to call out is https://docs.mender.io/2.4/devices/update-modules which is also mentioned in the first link. This is how Mender is extended to provide updates for other parts of the system that are not the rootfs itself.  Since updating a Docker container is something that has been mentioned before I want to also note https://hub.mender.io/t/docker/324 as it is a well supported module for this specific case.

I hope you find this information useful. Please contact us if you have specific questions. We’re looking forward to talking to you about your own specific OTA needs.

Getting Started with RAUC on Raspberry Pi

/2 Comments/by

RAUC is a secure, robust and flexible open source software for A/B updates of Embedded Linux devices. It is appropriate for various use cases and it is compatible with all popular build systems: The Yocto Project/ OpenEmbedded, Buildroot and PTXdist.

Konsulko Group engineers have experience with all popular open source solutions for software over the air updates of embedded Linux devices, including Mender, SWUPdate, HERE OTA Connect based on OSTree and Aktualizr. In this article we will discuss the exact steps to integrate RAUC with the Yocto Project (YP) and OpenEmbedded (OE) for Raspberry Pi – the most popular single board computer among students, hobbyists and makers.

For the practical example in this article we will be using the latest and greatest Raspberry Pi as of the moment: Raspberry Pi 4 Model B. Versions with different RAM sizes are available on the market. Any of these Raspberry Pi 4 Model B versions are OK for this RAUC demonstration.

As long time developers and users of the Yocto Project and OpenEmbedded, both have become favorite tools for creating customized distributions for Konsulko engineers. We frequently use and support them commercially. The Yocto Project is a Linux Foundation collaborative open source project for creating custom Linux distributions for embedded devices. It is based on Poky, the reference distribution of the Yocto Project, using the OpenEmbedded build system. The Yocto Project releases on a 6-month cadence. As of the time of this writing, the latest stable release is Dunfell (3.1).

RAUC is a powerful and flexible open source solution that requires advanced skills for initial integration. To use RAUC in an image for Raspberry Pi built with the Yocto Project and OpenEmbedded, it requires:

  • U-Boot as a bootloader
  • Enabled SquashFS in the Linux kernel configurations
  • ext4 root file system
  • Specific partitioning of the microSD card that matches the RAUC slots
  • U-Boot environment configurations and a script to properly switch RAUC slots
  • Certificate and a keyring to RAUC’s system.conf

RAUC is capable of covering various use cases and scenarios, including advanced options for single or redundant data partitions. Upgrades are performed through the so called RAUC bundles. It is possible to install them over the air or using the old-fashioned method with a USB stick. For managing updates to a fleet of Internet of Things, it is possible to integrate RAUC with Eclipse hawkBit project that acts as a deployment server with a nice web user interface.

For the sake of simplicity, this article focuses on the most simple and straight-forward use case with 2 identical RAUC slots: A and B. For each slot we will have a separate partition on the microSD card for Raspberry Pi. We have already covered most of the RAUC requirements in an additional Yocto/OE layer called meta-rauc-raspberrypi. We will use it to put the pieces together. First we will build a minimal bootable image for Raspberry Pi 4 with RAUC. We will flash it to both A and B slots. After that we will build a RAUC bundle that adds the text editor nano. Finally we will install this RAUC bundle on the B slot, reboot and verify that nano is present.

Building a Linux Distribution with RAUC

Follow the steps below to build a minimal image for Raspberry Pi with Yocto, OpenEmbedded and RAUC as well as to perform a software update:

  • Download Poky, the reference distribution of the Yocto Project:

git clone -b dunfell git://git.yoctoproject.org/poky poky-rpi-rauc
cd poky-rpi-rauc

  • Download meta-openembedded layer:

git clone -b dunfell git://git.openembedded.org/meta-openembedded

  • Download Yocto/OE BSP layer meta-raspberrypi:

git clone -b dunfell git://git.yoctoproject.org/meta-raspberrypi

  • Download Yocto/OE layers for RAUC:

git clone -b dunfell https://github.com/rauc/meta-rauc.git

git clone -b dunfell https://github.com/leon-anavi/meta-rauc-community.git

  • Initialize the build environment:

source oe-init-build-env

  • Add layers to conf/bblayers.conf:

bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake-layers add-layer ../meta-openembedded/meta-python/
bitbake-layers add-layer ../meta-openembedded/meta-networking/
bitbake-layers add-layer ../meta-openembedded/meta-multimedia/
bitbake-layers add-layer ../meta-raspberrypi/
bitbake-layers add-layer ../meta-rauc
bitbake-layers add-layer ../meta-rauc-community/meta-rauc-raspberrypi/

  • Adjust conf/local.conf for Raspberry Pi 4 with systemd and RAUC by adding the following configurations to the end of the file:
MACHINE = "raspberrypi4"

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

IMAGE_INSTALL_append = " rauc"

IMAGE_FSTYPES="tar.bz2 ext4 wic.bz2 wic.bmap"
SDIMG_ROOTFS_TYPE="ext4"
ENABLE_UART = "1"
RPI_USE_U_BOOT = "1"
PREFERRED_PROVIDER_virtual/bootloader = "u-boot"

WKS_FILE = "sdimage-dual-raspberrypi.wks.in"

  • Build a minimal bootable image:

bitbake core-image-minimal

NOTE: Building an image from scratch requires a lot of operations and takes some time so please patiently wait until bitbake completes all tasks.

  • Flash the image to a microSD card and boot it on Raspberry Pi 4:

sudo umount /dev/sdX*
bzcat tmp/deploy/images/raspberrypi4/core-image-minimal-raspberrypi4.wic.bz2 | sudo dd of=/dev/sdX
sync

  • Attach USB to UART debug cable to Raspberry Pi 4, plug ethernet cable and the microSD card. Turn on Raspberry Pi 4. Verify that the system boots successfully.
  • Now, let’s extend the image with the simple text editor nano by adding the following line to the end of conf/local.conf:

IMAGE_INSTALL_append = " nano"

  • Build a RAUC bundle:

bitbake update-bundle

  • Start a web server:

cd tmp/deploy/images/raspberrypi4/
python3 -m http.server

  • On the Raspberry Pi download the RAUC bundle, install it and reboot the board:

wget http://192.168.1.2:8000/update-bundle-raspberrypi4.raucb -P /tmp
rauc install /tmp/update-bundle-raspberrypi4.raucb
reboot

  • After successful upgrade with RAUC reboot the Raspberry Pi and verify that nano is now present:

which nano

  • Check RAUC status to confirm that now the second partition has been booted:

rauc status

For Internet of Things and other real-world products, the whole build procedure with the Yocto Project and OpenEmbedded can be optimized further to just a few commands for easy implementation of continuous integration (CI).

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We have experience with RAUC and various other open source solutions for software updates. Please contact us if you need your “own” rock-solid Linux distro for your own embedded product.

 

Helping Yocto Project work with Python 3

/by

According to the statistics from StackOverflow Python is the fastest-growing major programming language. First released in 1991, Python is nowadays commonly used for various applications in multiple different industries. Python is a first class citizen of many embedded Linux systems.

The Yocto Project, a collaborative project of the Linux Foundation for creating custom Linux distributions for embedded devices, uses the OpenEmbedded build system and relies on layer meta-python from meta-openembedded to deliver Python 3 packages. Until recently, meta-python was providing both python 2 and python 3 versions of each package. The Python community decided that January 1, 2020, was the day to sunset Python 2. Since then Python 2 has been officially deprecated. This triggered major changes related to the support in Yocto and OpenEmbedded. All recipes for version 2 were moved to layer meta-python2 to provide legacy support after the end of life for this Python release. In meta-openembedded/meta-python, the OpenEmbedded community started efforts to remove all recipes for version 2 as well as to consolidate inc and bb files into a single bb file for version 3.

Konsulko Group engineers are regular contributors to various upstream open source projects, including meta-openembedded and more specifically to meta-python. In the past month, Leon Anavi joined the community efforts for consolidating Python 3 recipes in a single file as well as for upgrading various packages. Nowadays, most of the Python 3 recipes are utilizing the pypi bbclass which takes care for downloading and processing packages from pypi.org. This makes most of the upgrades to new releases of a Python package straight-forward. However, it is important to check the list of build and runtime dependencies as well as to ensure that bitbake still works fine with the upgraded recipe version for both x86-64 and ARM architectures prior to submission. 

Let’s have a closer look at the recipe python3-protobuf. It has been recently upgraded from version 3.11.3 to version 3.12.2. Protocol Buffers, also known as protobuf, are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. In the Yocto and OpenEmbedded ecosystem, recipe python3-protobuf depends on recipe protobuf from layer meta-oe. Both meta-oe and meta-python are part of meta-openembedded. So to avoid version mismatch and to ensure that bitbake will be able to successfully build python3-protobuf version 3.12.2 an upgrade of recipe protofobuf to the same version was mandatory. We contributed both upgrades to the master branch of the git repository meta-openembedded. The maintainers took care of cherry-picking them to the dunfell branch which is compatible with the latest stable release of the Yocto Project as of the moment. As a result, if you checkout the latest stable release of Poky, the reference system of the Yocto Project, and meta-openembedded you will be able to quickly build the latest version of protobuf and python3-protobuf out of the box.

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We continue to regularly make upstream contributions to these open source projects. Please contact us if you need your “own” Linux distro for your own embedded product.

OpenEmbedded and Yocto Project

Getting some help with build systems

/by

One of the most rewarding parts of working at Konsulko Group has been our returning customers. Months, sometimes years, after we’ve successfully completed an engagement, we’ll hear from the same client (sometimes at the same company, sometimes at a new one) that they would like Konsulko’s help on their latest project.

Konsulko engineers have decades of experience at all levels of open source embedded software, from Linux kernel and low-level subsystems, through middleware and application development, to QA, maintenance and tools, but it is our expertise with the Yocto Project/OpenEmbedded build system that’s usually a part of everything we do.

From the time it first appeared in 2003, the OpenEmbedded build framework revolutionized embedded development, providing a systematic and reusable way to build custom Linux distributions for unique embedded devices. Almost ten years ago, OE became the build system of Yocto Project, and OE’s “recipe” approach was further structured and enhanced by layers.

Konsulko engineers have been there since the earliest days. We’ve seen a lot, learned a lot, and apply our expertise to helping our customers – old and new – build their “own” Linux distro for their own product.