Speaking at ELCE and Yocto Project Summit

Matt Porter, Konsulko Group CTO and Leon Anavi, Konsulko Senior Software Engineer are presenting at co-located Linux Foundation events in Lyon, France, October 30 – November 1, 2019.

At Embedded Linux Conference Europe, Leon will speak on Home Automation with MQTT, a machine-to-machine real-time communication protocol widely used in the Internet of Things, as part of the Open IOT Summit.

Matt Porter will present his very popular tutorial, Introduction to IIO and Input Drivers. Students will create their own game controller driver and use it to play a game on their devices. The lab will be conducted using the provided hardware kit.

The Yocto Project Summit is a technical conference to learn about Yocto Projects’ direction, get training on the next wave of embedded Linux technologies, and network with Yocto Project maintainers, experts and peers.

In his session on Working with NVIDIA Tegra BSP and Supporting Latest CUDA Versions, Leon Anavi will share his experience in customizing Poky, the reference distribution of the Yocto Project, for embedded devices with NVIDIA Tegra SoCs using OpenEmbedded build system and the BSP meta layer meta-tegra.

We hope to see you there.

Don’t miss the AGL All Member Meeting

The Automotive Grade Linux (AGL) All Member Meeting brings the AGL community together to collaborate, learn about the latest developments and share best practices. What better place to drive rapid innovation than Monte Carlo, the home of the Circuit de Monaco. AGL members who have not already done so can register here. Konsulko Group is looking forward to seeing you there for a great AMM.

Custom Linux Distro for NVIDIA CUDA Devices

How to get started and build a minimal custom Linux distribution for embedded NVIDIA CUDA-enabled devices using the Yocto Project (YP) and OpenEmbedded (OE).

ELCNA technical talks and in-depth training

Embedded Linux Conference (ELC) is the premier vendor-neutral technical conference on embedded Linux and industrial IoT products. This year, ELC North America will be held in San Diego, August 21-23, 2019, at the beautiful Hilton Bayfront, right on the harbor and just over a pedestrian bridge from Petco Park and East Village.

On Thursday, August 22, Leon Anavi, Senior Software Engineer at Konsulko Group will present a Comparison of Open Source Software Home Automation Tools that allow users to customize the setup depending their own specific needs and manage devices manufactured by different vendors in one place. Leon will focus on popular open source tools, Home Assistant, OpenHAB and Domoticz, and explore the supported embedded Linux development boards on which these platforms can be installed, as well as the IoT with which they can interact out of the box.

Also on Thursday, Vitaly Wool, Senior Staff Engineer and General Manager of Konsulko AB, will give a technical talk on Secure Updates for a Memory Constrained XIP (eXecute In Place) System looking at technology that allows code to be executed directly from flash without copying the code to RAM first. The memory footprint can be optimized very tightly and this permits really low-power IoT Linux appliances. However, there is a big obstacle: no standard secure update process for such systems will work due to the very nature of XIP. How can you update the flash when it must always be ready to execute? This talk will provide some real world answers and examples.

The next day, Friday, August 23, Matt Porter, Konsulko Group CTO, will present a tutorial, Introduction to IIO and Input Drivers to briefly look at the Linux IIO and Input subsystems and how to gather information from hardware documentation to assist in software development. In a guided hands-on lab, students will write a new driver that leverages the IIO and Input kernel subsystems, and create their own game controller driver and use it to play a game on their devices.

For the first time in 2019, Embedded Linux Conference North America will co-locate with Open Source Summit North America. We hope you can join us, along with 800+ developers and technical experts from across the globe for education, collaboration, deep-dive learning, and some good times in San Diego.

Konsulko Group sponsors TuxCon conference

On June 8th and 9th, Konsulko Group is proud to again be a Gold Sponsor of the 6th annual TuxCon conference in Plovdiv, Bulgaria, the 2019 European Capital of Culture.

Headquartered in California, Konsulko works with our customers throughout North America, Europe and Asia. Our European subsidiary, Konsulko Ltd is based in Sofia, Bulgaria.

If you are building a new product, we’d love to talk with you about engaging Konsulko’s engineering expertise and experience on your project. Or if you’re a software developer with a passion for Linux, please contact us about joining the Konsulko team.

Globally Employable Engineers

In 2004, we founded Embedded Alley Solutions, and many things we did for the next five years simply felt intuitive. We were, after all, embedded Linux engineers with little business, customer relationship management and human resources experience.

Intuition served us well. We were doing agile-style software development long before it became mainstream, and soon it seemed like everyone knew our name. Our ability to communicate internally was second to none, using nothing more than emails and irc, creating a close knit team that felt as though everyone is in the same office, although the team was highly distributed.

Our recruiting practice was not so much about hiring as it was about building and fostering relationships with top talent around the world until the time was right for both parties to make the move. Outside the US, we tried to find the best open source talent around the world, rather than building an “outsourcing center.” We grew the company with that stellar talent and were acquired in 2009.

In the years that followed, every now and then I would read advice in an article or business book, written by some acknowledged industry expert, and think “That’s right! That’s exactly what we were doing at EA!” It was interesting and amusing to find out from others that what we had done on the business side helped make Embedded Alley so successful. I note only the business aspects here because the engineering talent we had was hard to match.     

At Konsulko Group, I find myself having many of the same discussions with customers as we did back then, and I lean on that experience. One recurring discussion we had at EA was about our “offshore engineers,” as customers would often refer to engineers who resided outside the US. Embedded Alley had a small office in Europe, as well as single employees working from a home office all over the world. We always told customers that, no, we do not have a two-tier outsourcing strategy and these were not “offshore” developers. We simply searched for the best talent, with very specific software development experience, and that talent is not always to be found next door to our Silicon Valley office.

Fast forward 10 years after the EA acquisition. I recently found myself having the same offshore discussion with a customer. A specific principal level engineer was located in an European country not known as a center for high tech. Why was it that we were proposing to invoice him at the same rate as the other US-based principal engineers? This customer had done their homework and evidence showed that the average engineering rates in this European country are significantly lower.  

I paused on the phone and thought back to our Embedded Alley days. What did I tell customers then? I realized that we never had a good, quantifiable way to explain to a customer why our engineering rates for developers outside the US, though lower than market rates in Silicon Valley, were significantly higher than the typical “outsourcing rates” in a particular location. We had talked about our hiring strategy and looking for the best talent wherever we might find it, but there were no metrics I could lean on.

Then I thought about the work this particular engineer had done at Embedded Alley; and then after the acquisition, he continued at Mentor Graphics (always working remotely from Europe). When he left Mentor, he contracted for Texas Instruments, then another US company, and finally, after the founding of Konsulko Group “rejoined” our team. Meanwhile he had had other opportunities, from contracting gigs to full time jobs in the US with H1B visa sponsorship.

And that’s when it struck me. This was not an engineer working in an offshore office at an offshore salary. “This is a globally employable engineer,” I said on the phone, “and we pay him a US level salary in order to retain him.” I continued to recount his work history and track record. The buyer understood and we moved forward with the deal.

What is a globally employable engineer? In my mind, it’s someone that could get a good job anywhere around the world due to demand of their skills. It’s someone with a minimum of ten years of experience, highly talented, with excellent English language skills and some customer-facing experience. The ability to travel when necessary helps a lot, and that means the ability to get a B1/B2 US visa for occasional visits. Such engineers may choose to continue to live in their home country, or elsewhere in the world, but tapping their talent does not come at offshore salary cost.

It is time for the high tech industry to move beyond Outsourcing 1.0, and embrace the Globally Employable model to access the best engineering talent on earth, wherever on the planet they choose to reside.  

Konsulko Group to present at ALS Tokyo

Now in its eighth year, Automotive Linux Summit connects the Linux developer community with vendors and users to drive the future of embedded devices in the automotive arena.

On Wednesday, July 17, Scott Murray and Matt Ranostay of Konsulko Group will present Building an AGL Telematics Profile Demonstration Platform. This profile serves as a base for building headless telematics device images. Scott and Matt will discuss a practical use case, using the profile to build an AGL demonstration platform for a vehicle tracker or an insurance company’s driver data collection device.

Co-located with Open Source Summit Japan, ALS will be held at this year at Toranomon Hills Forum in Tokyo. Registration information can be found here.


A good time to talk with us

Whether you’re in a small start-up, a huge global company, or anything in-between, there are key moments in Linux-based software development when it’s time to decide how much can be handled in-house, and what requires some outside assistance.

Here are a four examples of a good time to talk with us:

  • Your engineers are experts at the top of your software stack, but kernel-level work needs to be done down near the bottom (where you don’t have much experience).
  • You’re dazzled by the power and complexity of the Yocto project and OpenEmbedded build system, and your team needs to get up to speed quickly.
  • You’re building your next generation product on new hardware and experience unforeseen “subtleties” in moving your code to the new platform.
  • You’ve crafted your software architecture from best-of-breed open source projects but you’re finding gaps that still need to be filled.

With 20-plus years of experience in embedded Linux architecture, development, build/CI, QA, maintenance and training, Konsulko Group can help you at every phase of your product cycle.

Any point in your development is a good time to contact Konsulko to discuss how we can work together.

Building a DIY SOHO router, Part 4

Building a DIY SOHO router using the Yocto Project build system OpenEmbedded, Part 4

In part three of this series I finished putting together what I wanted to have on my SOHO router and declared it to be done. While I plan to revisit the topic of a SOHO router using the Yocto Project and OpenEmbedded, this is the final part of the series. In this part, I want to focus on some of the things that I learned while doing the project.

The first thing is that I learned a lot about IPv6, specifically how it’s usually implemented within the United States for residential customers, and some of the implications of this implementation. The first thing to note is that I’ve been off-and-on trying to enable IPv6 for general IPv6 connectivity at home for some time now. Long before my ISP offered IPv6 service, I used Hurricane Electric to have a IPv6 tunnel and connectivity. This was great and only sometimes lead to problems, such as when Netflix finally supported IPv6 and began blocking well known tunnels for region-blocking reasons. It wasn’t until I started on this project that I decided to try to make real use of  routable addresses for hosting personal services. My expectations, and that of lots of software designed to manage IPv6 as well, are best described in article from RIPE about understanding IP addressing. In short, my house or “subscriber site” should get at least 256 LAN segments to do with as I want. Docker expects to have it’s own LAN segment to manage as part of configuring network bridges. When you have 256 or more LAN segments to use, that’s not a problem at all.

Unfortunately, my ISP provides only a single LAN segment. This is simultaneously more IPv6 addresses than the whole of IPv4 and something that should not be further subdivided in routing terms. I could subdivide my LAN segment, but this would in turn cause me to have to do a whole lot more work and headaches. That’s because at the routing level IPv6 is designed for my segment to be the smallest unit. Rather than deal with those headaches I switched my plans up from using Docker to using LXC. With LXC it’s easy to dump the container onto my LAN and then it picks up an IPv6 address the same way all of the other machines on my LAN do. This is good enough for my current needs, but it will make other things a lot harder down the line, if I want separation at the routing level for some devices.

But why am I doing that at all? Well, one of the benefits of having a small but still capable router is that I can run my own small services. While I don’t want to get into running my own email I think it makes a whole lot of sense to host my own chat server for example. With closed registration and no (or limited later on perhaps) federation with other servers I don’t need to worry about unauthorized users contacting my family nor do I have to worry about the company deciding it’s time to shutdown the service I use.

Another lesson learned is that while the Yocto Project has great QA, there’s always room to improve things. As part of picking a firewall I found that one of the netfilter logging options had been disabled by accident a while back. As a part of writing this series of articles and testing builds for qemux86-64, I found that one of the sound modules had been disabled. As a result, the instructions I wrote back in part 2 wouldn’t work. Working upstream is always fun and these changes have been merged and will be included in the next release.

I also worked on a few things for this project that I didn’t include directly in the relevant part of the series. For example, while I did include a number of full utilities in the list of packages installed in the router, I didn’t talk about replacing busybox entirely. This is something that OpenEmbedded supports using the PREFERRED_PROVIDERS and VIRTUAL-RUNTIME override mechanisms in the metadata. Prior to this article however, there wasn’t a good example on how to do this in upstream. Furthermore, there wasn’t an easy way to replace all of busybox and instead you had to list a single package and then include the rest of the required packages in your IMAGE_INSTALL or similar mechanism.  I am a fan of using busybox in places where I’m concerned about disk usage. However, on my router I have plenty of disk space so I want to be sure that if I have to go and solve a problem I’m not using my swiss army knife but rather have my full toolbox available. As a result, OpenEmbedded Core Master now has packagegroup-core-base-utils and a documented example of how to use that in local.conf.sample.extended. This means that when I refresh this image to be based on the Warrior branch I can remove a number of things from my IMAGE_INSTALL.

Another lesson is that old habits die hard.  In general, I always try to use the workflow where I make a change outside the device I’m working on, build the change in, and test it, rather than editing things live.  But when it’s “just” a quick one line change I’ll admit I do it live and roll it into my next build sometimes.  And then sometimes I forget to roll all my changes back up.  So while implementing this project I tried even harder than usual to not fall into that “just a quick change” mindset.  For the most part I’ve been successful at sticking to the idea workflow.  I really believe stateless is the right path forward.  And “for the most part” means that, yes, one time I did have to make use of the fact that the old rootfs was still mountable and copied a file over to the new rootfs, and then to the build machine.  I like to think of that as a reminder that A/B updates are more helpful than a “rewrite your disk each time” workflow for those occasional mistakes.

The caveat to the lesson above is because I really did the “git, bitbake, mender” cycle on this project. I didn’t start on it quite as soon as I said in the article, and I spent a lot more time toying with stuff in core-image-minimal instead of following my own advice, too.  I suppose that is the difference between writing a guide on how things should be done compared with how you do things when you just want to test one more thing, then switch over.  I really should have switched earlier however as every time I avoid doing the SD card shuffle it’s a win on a number of levels.

Did I say SD card above?  Yes, I did.  For this project, a 64GB “black box” that’s in the form-factor of a SD card will have as long of a life span as there is in the form-factor of a M.2 SSD or any other common storage format.  While my particular hardware has a SATA port, I don’t want to try to fit the required cabling, let alone the device itself in the case that’s recommended.  I will admit that I’m taking a bit of a risk here, I am putting as much frequent-write files under a ramfs as I can and after all, I did say stateless is a goal.  If everything does really die on me, I can be back up and running fairly quickly.

Last thing I learned is something I knew all along, really. I like the deeper ownership of the router. There’s both the pride and accomplishment in doing it and that “old school” fun of being the admin again, for real.

For training, nothing beats hands-on

There are plenty of YouTube videos (and their open source equivalents) to help budding engineers master the intricacies of development, but often the best way to learn is to get in the same room as the experts, and go step-by-step through the process.

At SCaLE 17x in Pasadena earlier this month, Konsulko Group CTO Matt Porter taught a guided hands-on lab on leveraging IIO and Input kernel subsystems. In real time, Matt went line-by-line through the code, and the students were able to write a new driver and take the results with them on an embedded target board.

In this intimate and interactive setting, apprentice-level engineers could get personal attention if they were stuck or had any question, no matter how basic.

Matt’s session was part of the E-ALE (Embedded Apprentice Linux Engineer) project. At major embedded Linux events, E-ALE provides several days of hands-on tutorials driven by volunteer professional speakers who present apprentice-level material in a way that beginners can understand and use.

We hope to see you during the next set of E-ALE tutorials at the Embedded Linux Conference in San Diego this August.

As always, Konsulko Group can also offer hands-on embedded Linux training at your location for your engineers. Please contact us to discuss your requirements for custom, on-site training.