Helping Yocto Project work with Python 3

According to the statistics from StackOverflow Python is the fastest-growing major programming language. First released in 1991, Python is nowadays commonly used for various applications in multiple different industries. Python is a first class citizen of many embedded Linux systems.

The Yocto Project, a collaborative project of the Linux Foundation for creating custom Linux distributions for embedded devices, uses the OpenEmbedded build system and relies on layer meta-python from meta-openembedded to deliver Python 3 packages. Until recently, meta-python was providing both python 2 and python 3 versions of each package. The Python community decided that January 1, 2020, was the day to sunset Python 2. Since then Python 2 has been officially deprecated. This triggered major changes related to the support in Yocto and OpenEmbedded. All recipes for version 2 were moved to layer meta-python2 to provide legacy support after the end of life for this Python release. In meta-openembedded/meta-python, the OpenEmbedded community started efforts to remove all recipes for version 2 as well as to consolidate inc and bb files into a single bb file for version 3.

Konsulko Group engineers are regular contributors to various upstream open source projects, including meta-openembedded and more specifically to meta-python. In the past month, Leon Anavi joined the community efforts for consolidating Python 3 recipes in a single file as well as for upgrading various packages. Nowadays, most of the Python 3 recipes are utilizing the pypi bbclass which takes care for downloading and processing packages from pypi.org. This makes most of the upgrades to new releases of a Python package straight-forward. However, it is important to check the list of build and runtime dependencies as well as to ensure that bitbake still works fine with the upgraded recipe version for both x86-64 and ARM architectures prior to submission. 

Let’s have a closer look at the recipe python3-protobuf. It has been recently upgraded from version 3.11.3 to version 3.12.2. Protocol Buffers, also known as protobuf, are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. In the Yocto and OpenEmbedded ecosystem, recipe python3-protobuf depends on recipe protobuf from layer meta-oe. Both meta-oe and meta-python are part of meta-openembedded. So to avoid version mismatch and to ensure that bitbake will be able to successfully build python3-protobuf version 3.12.2 an upgrade of recipe protofobuf to the same version was mandatory. We contributed both upgrades to the master branch of the git repository meta-openembedded. The maintainers took care of cherry-picking them to the dunfell branch which is compatible with the latest stable release of the Yocto Project as of the moment. As a result, if you checkout the latest stable release of Poky, the reference system of the Yocto Project, and meta-openembedded you will be able to quickly build the latest version of protobuf and python3-protobuf out of the box.

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We continue to regularly make upstream contributions to these open source projects. Please contact us if you need your “own” Linux distro for your own embedded product.

Getting some help with build systems

One of the most rewarding parts of working at Konsulko Group has been our returning customers. Months, sometimes years, after we’ve successfully completed an engagement, we’ll hear from the same client (sometimes at the same company, sometimes at a new one) that they would like Konsulko’s help on their latest project.

Konsulko engineers have decades of experience at all levels of open source embedded software, from Linux kernel and low-level subsystems, through middleware and application development, to QA, maintenance and tools, but it is our expertise with the Yocto Project/OpenEmbedded build system that’s usually a part of everything we do.

From the time it first appeared in 2003, the OpenEmbedded build framework revolutionized embedded development, providing a systematic and reusable way to build custom Linux distributions for unique embedded devices. Almost ten years ago, OE became the build system of Yocto Project, and OE’s “recipe” approach was further structured and enhanced by layers.

Konsulko engineers have been there since the earliest days. We’ve seen a lot, learned a lot, and apply our expertise to helping our customers – old and new – build their “own” Linux distro for their own product.

Working remotely by design

The coronavirus pandemic has forced all of us to fundamentally change the way we live. Thankfully, at Konsulko Group, it has yet to significantly change the way we work. Working remotely is the norm for Konsulko engineers. In fact, Konsulko was built on the idea of Globally Employable Engineers™ based throughout the world.

We all have permanent personal workspaces that are fully equipped with the best tools for remote development work and collaboration. Members of our team have years of software engineering experience, excellent English language skills, and are comfortable working directly with our customers.

In these times when travel is necessarily limited and in-person meetings can endanger health and safety, we are happy to share our best practices based on our extensive experience as remote workers. Please feel free to contact us.

Meanwhile, we continue our work in the open source community while helping our customers build great products for the better times ahead. Stay healthy.

The Year in Review 2019

Konsulko Group has completed another busy year, working hard to solve our customers’ problems, while building efficient and reliable open source software for their products in development.

We’ve helped our customers launch exciting new products across the spectrum of embedded systems: automotive, medical devices, networking equipment, and cutting-edge consumer electronics.

As Globally Employable Engineers, we are geographically distributed across North America and Europe, but thanks to today’s technology, we have worked together every day as a team, often in real time.

At Konsulko Group we believe production software should be closely aligned with OSS community work upstream, and we work to make that happen. Our community work in key open source projects includes the Linux kernel, Yocto Project, OpenEmbedded and Automotive Grade Linux. We participate in important OSS conferences like FOSDEM and SCaLE, sponsor TuxCon and OpenFest every year, and have published two new “how to” technical papers, useful to developers at all levels (available on this website).

We’ve continued our close relationship with the Linux Foundation and Automotive Grade Linux, supporting AGL development, demos and member meetings. Konsulko Group gave technical talks at ALS Tokyo, ELC San Diego, ELCE and Yocto Project Summit Lyon, and the AGL AMM Monte Carlo, and participated in the OSLS Half Moon Bay.

Through the Embedded Apprentice Linux Engineer program, Konsulko provided well-received training sessions in 2019 at SCaLE 17x Pasadena, ELC San Diego and ELCE Lyon. We’ve given on-site and remote training to major companies in North America, and remain a Linux Foundation Authorized Training Partner.

All in all, 2019 seems like it has gone quickly. We now look forward to working with you in 2020 and beyond.

Building a DIY SOHO router, 6 months on

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, 6 months on

A little more than six months ago, I posted part 4 of our series on making a SOHO router using the Yocto Project and OpenEmbedded. After 6 months of deployment, this is a good time to follow up on how the router has worked out in residential use.  The zeus code-name Yocto Project release was just announced, and that means that the release we started out with in part 1 is now close to being out of support.  That’s not a problem since we designed in support for moving to new software releases using Mender to deliver software updates.

One of the most important metrics in a project like this is, how does it perform?  From the standpoint of a family of 4 heavy internet users, it’s gone really well.  The WiFi range is only a little better than before, but that’s not really a function of the software.  Making everyone use Pi-hole has only resulted in a small number of places where I needed to override the blacklist and allow something in.  From an end-user point of view, this has worked as well as any off-the-shelf router.  From the administrator point of view, I’ve done scheduled maintenance during the day on a weekend, and it really did take only the 5 minutes I promised everyone rather than turning into one of those worst case scenarios where something broke and it takes an hour to fix it.  In fact, the update portion of the plan has gone exceedingly well.  While I didn’t make a post about moving to warrior from thud, I did that transition a while ago and it went smoothly.  Mender introduced a change that required attention be paid while migrating, but it was documented and went smoothly.  On the metadata side, the upgrade was as easy as one could hope.  A few of the bbappend files needed to be updated for a new version, and some of the changes I had made and pushed upstream as part of the original series were now just included, so they got dropped from my layer.

One of the things I touched on in the series was about using the update functionality to test development changes in a production environment.  The chance to do that came up with a systemd-networkd issue that was a problem in my local setup.  The upstream project requested people verify the problem exists with newer versions of systemd and a new enough version was available in what would become zeus.  So I made a quick weekend project of doing an update of my layers to build with a newer version of all of the metadata, removed the work-around, and flashed the image in place.  A quick reboot confirmed that the issue was indeed fixed, and then rather than commit to running an otherwise in-progress release I simply rebooted and automatically rolled back to my stable release.  With the network back up again, I updated the issue in upstream Bugzilla to let them know the problem was fixed.  After a bit longer, a few other people also confirmed it worked for them and now the issue is resolved.

In terms of the metadata itself, there have been a few clean-ups to what I did in my own layer with each release update I’ve done.  In the series I left out what hardware I was building on, and I also left out talking about using the linux–yocto recipe.  Since I first wrote the series linux-yocto has become easier to use, and I found this as part of reviewing my own changes like they were brand new with each upgrade.  I was setting some variables that initially didn’t have reasonable default values, and now they do and I don’t need to set them myself.  This in fact means that moving forward, rather than a version-specific kernel bbappend file, I can go with an always-used one to enable the additional kernel CONFIG options that I need for a time-based firewall.

I started out by mentioning that zeus has been released, and I’m working on migrating to it as I write this.  In fact, it’s so new that I’m doing my own little port of the meta-mender core layer to zeus for my needs. I expect that by the time I do my first update from one build of zeus to the next there will be an official update I’ll be able to use instead.  Looking forward, this was a great little project that also was a lot of fun.  The goals I set way back at the start have been met, and I’m happier with my network than I have been in a long time.  Of course, the list of features an off-the-shelf system provides is always growing, and there’s now monitoring and display items on my weekend project list now to keep up.  I foresee using and improving this setup for a long time to come.

Custom Linux Distro for NVIDIA CUDA Devices

How to get started and build a minimal custom Linux distribution for embedded NVIDIA CUDA-enabled devices using the Yocto Project (YP) and OpenEmbedded (OE).

Globally Employable Engineers

In 2004, we founded Embedded Alley Solutions, and many things we did for the next five years simply felt intuitive. We were, after all, embedded Linux engineers with little business, customer relationship management and human resources experience.

Intuition served us well. We were doing agile-style software development long before it became mainstream, and soon it seemed like everyone knew our name. Our ability to communicate internally was second to none, using nothing more than emails and irc, creating a close knit team that felt as though everyone is in the same office, although the team was highly distributed.

Our recruiting practice was not so much about hiring as it was about building and fostering relationships with top talent around the world until the time was right for both parties to make the move. Outside the US, we tried to find the best open source talent around the world, rather than building an “outsourcing center.” We grew the company with that stellar talent and were acquired in 2009.

In the years that followed, every now and then I would read advice in an article or business book, written by some acknowledged industry expert, and think “That’s right! That’s exactly what we were doing at EA!” It was interesting and amusing to find out from others that what we had done on the business side helped make Embedded Alley so successful. I note only the business aspects here because the engineering talent we had was hard to match.     

At Konsulko Group, I find myself having many of the same discussions with customers as we did back then, and I lean on that experience. One recurring discussion we had at EA was about our “offshore engineers,” as customers would often refer to engineers who resided outside the US. Embedded Alley had a small office in Europe, as well as single employees working from a home office all over the world. We always told customers that, no, we do not have a two-tier outsourcing strategy and these were not “offshore” developers. We simply searched for the best talent, with very specific software development experience, and that talent is not always to be found next door to our Silicon Valley office.

Fast forward 10 years after the EA acquisition. I recently found myself having the same offshore discussion with a customer. A specific principal level engineer was located in an European country not known as a center for high tech. Why was it that we were proposing to invoice him at the same rate as the other US-based principal engineers? This customer had done their homework and evidence showed that the average engineering rates in this European country are significantly lower.  

I paused on the phone and thought back to our Embedded Alley days. What did I tell customers then? I realized that we never had a good, quantifiable way to explain to a customer why our engineering rates for developers outside the US, though lower than market rates in Silicon Valley, were significantly higher than the typical “outsourcing rates” in a particular location. We had talked about our hiring strategy and looking for the best talent wherever we might find it, but there were no metrics I could lean on.

Then I thought about the work this particular engineer had done at Embedded Alley; and then after the acquisition, he continued at Mentor Graphics (always working remotely from Europe). When he left Mentor, he contracted for Texas Instruments, then another US company, and finally, after the founding of Konsulko Group “rejoined” our team. Meanwhile he had had other opportunities, from contracting gigs to full time jobs in the US with H1B visa sponsorship.

And that’s when it struck me. This was not an engineer working in an offshore office at an offshore salary. “This is a globally employable engineer,” I said on the phone, “and we pay him a US level salary in order to retain him.” I continued to recount his work history and track record. The buyer understood and we moved forward with the deal.

What is a globally employable engineer? In my mind, it’s someone that could get a good job anywhere around the world due to demand of their skills. It’s someone with a minimum of ten years of experience, highly talented, with excellent English language skills and some customer-facing experience. The ability to travel when necessary helps a lot, and that means the ability to get a B1/B2 US visa for occasional visits. Such engineers may choose to continue to live in their home country, or elsewhere in the world, but tapping their talent does not come at offshore salary cost.

It is time for the high tech industry to move beyond Outsourcing 1.0, and embrace the Globally Employable model to access the best engineering talent on earth, wherever on the planet they choose to reside.  

A good time to talk with us

Whether you’re in a small start-up, a huge global company, or anything in-between, there are key moments in Linux-based software development when it’s time to decide how much can be handled in-house, and what requires some outside assistance.

Here are a four examples of a good time to talk with us:

  • Your engineers are experts at the top of your software stack, but kernel-level work needs to be done down near the bottom (where you don’t have much experience).
  • You’re dazzled by the power and complexity of the Yocto project and OpenEmbedded build system, and your team needs to get up to speed quickly.
  • You’re building your next generation product on new hardware and experience unforeseen “subtleties” in moving your code to the new platform.
  • You’ve crafted your software architecture from best-of-breed open source projects but you’re finding gaps that still need to be filled.

With 20-plus years of experience in embedded Linux architecture, development, build/CI, QA, maintenance and training, Konsulko Group can help you at every phase of your product cycle.

Any point in your development is a good time to contact Konsulko to discuss how we can work together.

Building a DIY SOHO router, Part 4

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 4

In part three of this series I finished putting together what I wanted to have on my SOHO router and declared it to be done. While I plan to revisit the topic of a SOHO router using the Yocto Project and OpenEmbedded, this is the final part of the series. In this part, I want to focus on some of the things that I learned while doing the project.

The first thing is that I learned a lot about IPv6, specifically how it’s usually implemented within the United States for residential customers, and some of the implications of this implementation. The first thing to note is that I’ve been off-and-on trying to enable IPv6 for general IPv6 connectivity at home for some time now. Long before my ISP offered IPv6 service, I used Hurricane Electric to have a IPv6 tunnel and connectivity. This was great and only sometimes lead to problems, such as when Netflix finally supported IPv6 and began blocking well known tunnels for region-blocking reasons. It wasn’t until I started on this project that I decided to try to make real use of  routable addresses for hosting personal services. My expectations, and that of lots of software designed to manage IPv6 as well, are best described in article from RIPE about understanding IP addressing. In short, my house or “subscriber site” should get at least 256 LAN segments to do with as I want. Docker expects to have it’s own LAN segment to manage as part of configuring network bridges. When you have 256 or more LAN segments to use, that’s not a problem at all.

Unfortunately, my ISP provides only a single LAN segment. This is simultaneously more IPv6 addresses than the whole of IPv4 and something that should not be further subdivided in routing terms. I could subdivide my LAN segment, but this would in turn cause me to have to do a whole lot more work and headaches. That’s because at the routing level IPv6 is designed for my segment to be the smallest unit. Rather than deal with those headaches I switched my plans up from using Docker to using LXC. With LXC it’s easy to dump the container onto my LAN and then it picks up an IPv6 address the same way all of the other machines on my LAN do. This is good enough for my current needs, but it will make other things a lot harder down the line, if I want separation at the routing level for some devices.

But why am I doing that at all? Well, one of the benefits of having a small but still capable router is that I can run my own small services. While I don’t want to get into running my own email I think it makes a whole lot of sense to host my own chat server for example. With closed registration and no (or limited later on perhaps) federation with other servers I don’t need to worry about unauthorized users contacting my family nor do I have to worry about the company deciding it’s time to shutdown the service I use.

Another lesson learned is that while the Yocto Project has great QA, there’s always room to improve things. As part of picking a firewall I found that one of the netfilter logging options had been disabled by accident a while back. As a part of writing this series of articles and testing builds for qemux86-64, I found that one of the sound modules had been disabled. As a result, the instructions I wrote back in part 2 wouldn’t work. Working upstream is always fun and these changes have been merged and will be included in the next release.

I also worked on a few things for this project that I didn’t include directly in the relevant part of the series. For example, while I did include a number of full utilities in the list of packages installed in the router, I didn’t talk about replacing busybox entirely. This is something that OpenEmbedded supports using the PREFERRED_PROVIDERS and VIRTUAL-RUNTIME override mechanisms in the metadata. Prior to this article however, there wasn’t a good example on how to do this in upstream. Furthermore, there wasn’t an easy way to replace all of busybox and instead you had to list a single package and then include the rest of the required packages in your IMAGE_INSTALL or similar mechanism.  I am a fan of using busybox in places where I’m concerned about disk usage. However, on my router I have plenty of disk space so I want to be sure that if I have to go and solve a problem I’m not using my swiss army knife but rather have my full toolbox available. As a result, OpenEmbedded Core Master now has packagegroup-core-base-utils and a documented example of how to use that in local.conf.sample.extended. This means that when I refresh this image to be based on the Warrior branch I can remove a number of things from my IMAGE_INSTALL.

Another lesson is that old habits die hard.  In general, I always try to use the workflow where I make a change outside the device I’m working on, build the change in, and test it, rather than editing things live.  But when it’s “just” a quick one line change I’ll admit I do it live and roll it into my next build sometimes.  And then sometimes I forget to roll all my changes back up.  So while implementing this project I tried even harder than usual to not fall into that “just a quick change” mindset.  For the most part I’ve been successful at sticking to the idea workflow.  I really believe stateless is the right path forward.  And “for the most part” means that, yes, one time I did have to make use of the fact that the old rootfs was still mountable and copied a file over to the new rootfs, and then to the build machine.  I like to think of that as a reminder that A/B updates are more helpful than a “rewrite your disk each time” workflow for those occasional mistakes.

The caveat to the lesson above is because I really did the “git, bitbake, mender” cycle on this project. I didn’t start on it quite as soon as I said in the article, and I spent a lot more time toying with stuff in core-image-minimal instead of following my own advice, too.  I suppose that is the difference between writing a guide on how things should be done compared with how you do things when you just want to test one more thing, then switch over.  I really should have switched earlier however as every time I avoid doing the SD card shuffle it’s a win on a number of levels.

Did I say SD card above?  Yes, I did.  For this project, a 64GB “black box” that’s in the form-factor of a SD card will have as long of a life span as there is in the form-factor of a M.2 SSD or any other common storage format.  While my particular hardware has a SATA port, I don’t want to try to fit the required cabling, let alone the device itself in the case that’s recommended.  I will admit that I’m taking a bit of a risk here, I am putting as much frequent-write files under a ramfs as I can and after all, I did say stateless is a goal.  If everything does really die on me, I can be back up and running fairly quickly.

Last thing I learned is something I knew all along, really. I like the deeper ownership of the router. There’s both the pride and accomplishment in doing it and that “old school” fun of being the admin again, for real.

For training, nothing beats hands-on

There are plenty of YouTube videos (and their open source equivalents) to help budding engineers master the intricacies of development, but often the best way to learn is to get in the same room as the experts, and go step-by-step through the process.

At SCaLE 17x in Pasadena earlier this month, Konsulko Group CTO Matt Porter taught a guided hands-on lab on leveraging IIO and Input kernel subsystems. In real time, Matt went line-by-line through the code, and the students were able to write a new driver and take the results with them on an embedded target board.

In this intimate and interactive setting, apprentice-level engineers could get personal attention if they were stuck or had any question, no matter how basic.

Matt’s session was part of the E-ALE (Embedded Apprentice Linux Engineer) project. At major embedded Linux events, E-ALE provides several days of hands-on tutorials driven by volunteer professional speakers who present apprentice-level material in a way that beginners can understand and use.

We hope to see you during the next set of E-ALE tutorials at the Embedded Linux Conference in San Diego this August.

As always, Konsulko Group can also offer hands-on embedded Linux training at your location for your engineers. Please contact us to discuss your requirements for custom, on-site training.