Helping Yocto Project work with Python 3

According to the statistics from StackOverflow Python is the fastest-growing major programming language. First released in 1991, Python is nowadays commonly used for various applications in multiple different industries. Python is a first class citizen of many embedded Linux systems.

The Yocto Project, a collaborative project of the Linux Foundation for creating custom Linux distributions for embedded devices, uses the OpenEmbedded build system and relies on layer meta-python from meta-openembedded to deliver Python 3 packages. Until recently, meta-python was providing both python 2 and python 3 versions of each package. The Python community decided that January 1, 2020, was the day to sunset Python 2. Since then Python 2 has been officially deprecated. This triggered major changes related to the support in Yocto and OpenEmbedded. All recipes for version 2 were moved to layer meta-python2 to provide legacy support after the end of life for this Python release. In meta-openembedded/meta-python, the OpenEmbedded community started efforts to remove all recipes for version 2 as well as to consolidate inc and bb files into a single bb file for version 3.

Konsulko Group engineers are regular contributors to various upstream open source projects, including meta-openembedded and more specifically to meta-python. In the past month, Leon Anavi joined the community efforts for consolidating Python 3 recipes in a single file as well as for upgrading various packages. Nowadays, most of the Python 3 recipes are utilizing the pypi bbclass which takes care for downloading and processing packages from pypi.org. This makes most of the upgrades to new releases of a Python package straight-forward. However, it is important to check the list of build and runtime dependencies as well as to ensure that bitbake still works fine with the upgraded recipe version for both x86-64 and ARM architectures prior to submission. 

Let’s have a closer look at the recipe python3-protobuf. It has been recently upgraded from version 3.11.3 to version 3.12.2. Protocol Buffers, also known as protobuf, are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. In the Yocto and OpenEmbedded ecosystem, recipe python3-protobuf depends on recipe protobuf from layer meta-oe. Both meta-oe and meta-python are part of meta-openembedded. So to avoid version mismatch and to ensure that bitbake will be able to successfully build python3-protobuf version 3.12.2 an upgrade of recipe protofobuf to the same version was mandatory. We contributed both upgrades to the master branch of the git repository meta-openembedded. The maintainers took care of cherry-picking them to the dunfell branch which is compatible with the latest stable release of the Yocto Project as of the moment. As a result, if you checkout the latest stable release of Poky, the reference system of the Yocto Project, and meta-openembedded you will be able to quickly build the latest version of protobuf and python3-protobuf out of the box.

Konsulko engineers have been there since the earliest days of the OpenEmbedded build framework and the Yocto Project. We continue to regularly make upstream contributions to these open source projects. Please contact us if you need your “own” Linux distro for your own embedded product.

Join virtual ELCNA & Yocto Project Dev Day

There’s still time to register for the Linux Foundation’s Embedded Linux Conference North America Virtual Experience and Yocto Project® Dev Day.

Konsulko Group will be giving four presentations. On Tuesday, June 30, Matt Porter will give an in-depth tutorial: Spelunking for Hardware Data and on Wednesday, July 1, Leon Anavi will look at Automotive Grade Linux on Raspberry Pi: How Does It Work?

At Yocto Dev Day on Thursday, July 2, Paul Barker will explore Highly Scalable Build Automation and Scott Murray will discuss Security Hardening.

Please join us for three days of education and virtual collaboration.

Getting some help with build systems

One of the most rewarding parts of working at Konsulko Group has been our returning customers. Months, sometimes years, after we’ve successfully completed an engagement, we’ll hear from the same client (sometimes at the same company, sometimes at a new one) that they would like Konsulko’s help on their latest project.

Konsulko engineers have decades of experience at all levels of open source embedded software, from Linux kernel and low-level subsystems, through middleware and application development, to QA, maintenance and tools, but it is our expertise with the Yocto Project/OpenEmbedded build system that’s usually a part of everything we do.

From the time it first appeared in 2003, the OpenEmbedded build framework revolutionized embedded development, providing a systematic and reusable way to build custom Linux distributions for unique embedded devices. Almost ten years ago, OE became the build system of Yocto Project, and OE’s “recipe” approach was further structured and enhanced by layers.

Konsulko engineers have been there since the earliest days. We’ve seen a lot, learned a lot, and apply our expertise to helping our customers – old and new – build their “own” Linux distro for their own product.