Developing on hardware without functional networking

One of the more painful steps in doing development on hardware is when you don’t have any networking that’s functional and reliable yet. So you end up having to shuffle a SD card, USB stick, or similar back and forth. This can be even less fun when you’re working on a prototype, and need to take care to avoid disconnecting a fragile collection of boards, wires, and cables.

While there are a few different ways to get around this problem, depending on what is and isn’t working, my current favorite is wireless enabled SD cards. Why? One reason is that they’re OS-independent. So long as the board can supply power the card will be able to bringup its network. Break booting into Linux with your latest change? Not a problem. Want to integrate with your existing build cycle? There are CLI tools, and at the end of the day you’re sending stuff via HTTP to the card so you can always write something yourself if you don’t like what you find for tooling. The biggest drawback to me is that it only supports exposing the first FAT partition, but on the other hand you can happily partition the card and there is no requirement on where the FAT partition physically resides.

To make the best use of these cards, there are a few tricks you will want to employ. And while everything is documented, I found it handy to make up some templates to work with when I was setting up my lab with a few cards. On each card I would leave the VERSION and CID fields alone as they are pre-populated, and then overwrite the rest of the contents based on my template, fill it in, and go. My template looks like:

[WLANSD]

DHCP_Enabled=NO
IP_Address=192.168.0.XXX
Subnet_Mask=255.255.255.0
Default_Gateway=192.168.0.1
Preferred_DNS_Server=192.168.0.1
Alternate_DNS_Server=8.8.8.8

[Vendor]

CIPATH=/DCIM/100__TSB/FA000001.JPG
PRODUCT=FlashAir
VENDOR=TOSHIBA
WEBDAV=1
TIMEZONE=-28
APPSSID=MY-LOCAL-SSID
APPNETWORKKEY=MY-LOCAL-SSID-PASSPHRASE
APPNAME=flashair-XXX
APPMODE=5
APPAUTOTIME=300000
DNSMODE=0
LOCK=1
UPLOAD=1
WLANAPMODE=0x82

A few of these choices are odd enough that they are worth explaining. First, while the cards can happily do DHCP, I don’t like relying on that in cases like this. I’m much happier to instead put the card in a case with a sticky note on top that notes the IP, for the next project I need it for. Next, I set APPMODE to 5 so that it will join my network rather than start its own. The TIMEZONE key is a little odd. It is UTC based but works in 15 minute increments, which can be useful, but is somewhat unexpected. Note that if you do not spell out UPLOAD=1 uploading of files is disabled. The default upload path is the root, and that is often what we will want. Finally, I have WLANAPMODE set to the value for 802.11n/g rather than the default of 802.11b/g. It is also worth noting that once the card has run with this config file, it will update and asterisk out your network passphrase.

At the end of the day, I like these, and have a number of them because they’re so versatile. They come in sizes up to 32GB, which is enough to have a pretty well featured distribution available; either for the board itself, or to chroot into during development to easily add perf or other tools to a stripped down system. They are a full-size SD card but that’s just an adapter away from fitting into a microSD slot which is how I use about half of mine. And if you don’t have a spare (or functional yet) SD slot a USB card reader works just as well to bring this into the system you’re working on.

Leon Anavi of Konsulko Group Honored by GENIVI Alliance

Leon Anavi of Konsulko Group Honored by GENIVI Alliance

Awarded “Most Valuable Contributor” at All Member Meeting

SAN JOSE, Calif. – Oct. 18, 2016 – Leon Anavi, senior software engineer at Konsulko Group today received a Most Valuable Contributor (MVC) award at the GENIVI All Member Meeting and Open Community Days in Burlingame, Calif.

GENIVI® is a nonprofit industry alliance committed to driving the broad adoption of open source, In-Vehicle Infotainment (IVI) software and providing open technology for the connected car. The alliance provides its members with a global networking community of more than 140 companies, joining connected car stakeholders with world-class developers in a collaborative environment, resulting in free, open source middleware.

The GENIVI MVC award is given to individuals for significant contributions to the work of the alliance. Anavi was nominated by the members of the GENIVI Development Platform (GDP) team for his excellent work on GDP enhancements.

Anavi is an Open Source Software enthusiast and spent more than eight years working on core telephony solutions and mobile applications for mobile network operators before joining Konsulko Group in February 2015. Since then, his focus has been Linux for automotive and upstream contribution to GDP, Automotive Grade Linux (AGL), and the Tizen project.

Learn more at www.konsulko.com.


About Konsulko Group

An experienced leader in community and commercial development, Konsulko Group helps companies around the world build successful products with embedded Linux and open source software, offering consulting, capability, product engineering and support at every stage of every engagement. The Konsulko Group is headquartered in San Jose, Calif.

Media Contact – Konsulko Group:

Peter Popov
Konsulko Group

Supporting Flame Graphs on production kernels

Background

Perf is an amazing tool for observing system performance in Linux. Using perf on production kernels can be filled with pitfalls, due to the rapid pace at which new features are being added. In my case, I support a production kernel team that expects every feature they read about on the web to work on their older production kernel. A good example of a downstream use case of perf is Brendan Gregg’s very nice Flame Graphs tool for visualizing frequently used code paths in a system.

Example mysql Flame Graph

Example mysql Flame Graph

Recording call frame information with perf

Generation of Flame Graphs depends on perf capturing call frames. As documented in the Flame Graph tools, one records perf data on a x86-64 system by enabling DWARF call graph support with a command line like:

$ perf record -F 99 -a --call-graph dwarf -- sleep 60

That, of course, produces the raw perf.data file. The call frames we need are there. However, we need to process this data with a reporting tool.

Problems generating Flame Graphs

Now we start running into the problem with our production kernel. In our case, we are on a 4.1 kernel. Users are happily running perf report, seeing the complete set of call frame information throughout the system components under observation. The interesting thing is that if we generate a Flame Graph using this same data, then the users no longer have visibility into the complete calling tree information. That is, the Flame Graph will simply show time spent in a given library. So what’s wrong? Let’s take a look at how Flame Graphs are generated:

$ perf script > out.perf
$ stackcollapse-perf.pl out.perf > out.folded
$ flamegraph.pl out.folded > out.svg

The key here is that we are no longer parsing the perf data using perf report, but rather using perf script to do the heavy lifting and feeding the result into the Flame Graph generation tools. Doing a bit of git detective work, we can see that perf report added callchain sampling all the way back in 3.18:

$ git describe --contains 0cdccac6fe4b1316f04f0dbfcc4efab51932014a
v3.18-rc1~8^2~2^2~6
$ git log -1 -p 0cdccac6fe4b1316f04f0dbfcc4efab51932014a
commit 0cdccac6fe4b1316f04f0dbfcc4efab51932014a
Author: Namhyung Kim <[email protected]>
Date:   Mon Oct 6 09:45:59 2014 +0900

    perf report: Set callchain_param.record_mode for future use

    Normally the callchain_param.record_mode is used only for record path.
    But as it might need to prepare something for dwarf unwinding, setup
    this info for perf report too.

    Signed-off-by: Namhyung Kim <[email protected]>
    Acked-by: Jiri Olsa <[email protected]>
    Cc: David Ahern <[email protected]>
    Cc: Frederic Weisbecker <[email protected]>
    Cc: Ingo Molnar <[email protected]>
    Cc: Jean Pihet <[email protected]>
    Cc: Jiri Olsa <[email protected]>
    Cc: Namhyung Kim <[email protected]>
    Cc: Paul Mackerras <[email protected]>
    Cc: Peter Zijlstra <[email protected]>
    Link: http://lkml.kernel.org/r/[email protected]
    Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 2cfc4b93..140a6cd 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -257,6 +257,13 @@ static int report__setup_sample_type(struct report *rep)
                }
        }

+       if (symbol_conf.use_callchain || symbol_conf.cumulate_callchain) {
+               if ((sample_type & PERF_SAMPLE_REGS_USER) &&
+                   (sample_type & PERF_SAMPLE_STACK_USER))
+                       callchain_param.record_mode = CALLCHAIN_DWARF;
+               else
+                       callchain_param.record_mode = CALLCHAIN_FP;
+       }
        return 0;
 }

diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 96adb73..fc25e57 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -9,6 +9,7 @@
 #include "perf_regs.h"
 #include "map.h"
 #include "thread.h"
+#include "callchain.h"

 static int mmap_handler(struct perf_tool *tool __maybe_unused,
                        union perf_event *event,
@@ -120,6 +121,8 @@ int test__dwarf_unwind(void)
                return -1;
        }

+       callchain_param.record_mode = CALLCHAIN_DWARF;
+
        if (init_live_machine(machine)) {
                pr_err("Could not init machinen");
                goto out;

Making Flame Graphs work with our kernel

Knowing that this worked on newer versions of perf in at least the 4.6 kernel, we were then able to spot that it wasn’t until 4.3 that perf script gained callchain support. Notice the addition of the analogous code to what was already in perf report:

$ git describe --contains 7322d6c98dd214252bd697f8dde64a3576977fab
v4.3-rc1~138^2~5^2~10
$ git log -1 -p 7322d6c98dd214252bd697f8dde64a3576977fab
commit 7322d6c98dd214252bd697f8dde64a3576977fab
Author: Jiri Olsa <[email protected]>
Date:   Thu Aug 13 09:17:24 2015 +0200

    perf script: Initialize callchain_param.record_mode

    Milian Wolff reported non functional DWARF unwind under perf script. The
    reason is that perf script does not properly configure
    callchain_param.record_mode, which is needed by unwind code.

    Stealing the code from report and leaving the place for more
    initialization code in a hope we could merge it with
    report__setup_sample_type one day.

    Reported-by: Milian Wolff <[email protected]>
    Signed-off-by: Jiri Olsa <[email protected]>
    Tested-by: Milian Wolff <[email protected]>
    Cc: David Ahern <[email protected]>
    Cc: Namhyung Kim <[email protected]>
    Cc: Peter Zijlstra <[email protected]>
    Link: http://lkml.kernel.org/r/[email protected]
    Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 7b376d2..105332e 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -1561,6 +1561,22 @@ static int have_cmd(int argc, const char **argv)
        return 0;
 }

+static void script__setup_sample_type(struct perf_script *script)
+{
+       struct perf_session *session = script->session;
+       u64 sample_type = perf_evlist__combined_sample_type(session->evlist);
+
+       if (symbol_conf.use_callchain || symbol_conf.cumulate_callchain) {
+               if ((sample_type & PERF_SAMPLE_REGS_USER) &&
+                   (sample_type & PERF_SAMPLE_STACK_USER))
+                       callchain_param.record_mode = CALLCHAIN_DWARF;
+               else if (sample_type & PERF_SAMPLE_BRANCH_STACK)
+                       callchain_param.record_mode = CALLCHAIN_LBR;
+               else
+                       callchain_param.record_mode = CALLCHAIN_FP;
+       }
+}
+
 int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
 {
        bool show_full_info = false;
@@ -1849,6 +1865,7 @@ int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
                goto out_delete;

        script.session = session;
+       script__setup_sample_type(&script);

        session->itrace_synth_opts = &itrace_synth_opts;

By backporting this support from the 4.3 version of perf, we were able to support generation of Flame Graphs with our 4.1 production kernel tooling.

Conclusion

The moral of the story is: don’t count on well publicized perf features working on your older kernel. It is just as important to backport updates to the userspace perf tools as it is to backport updates for the production kernel itself.

Konsulko Group becomes Linux Foundation Authorized Training Partner

Konsulko Group becomes Linux Foundation Authorized Training Partner
Passes strict benchmarks for development experience and teaching quality

SAN JOSE, Calif. – Sept. 30, 2016 – Konsulko Group, with over two decades of embedded Linux and open source software development experience, today announced the company has joined the Linux Foundation Authorized Training Partner (ATP) program, providing standard and customized classes for beginning, intermediate and advanced students, using the Linux Foundation’s high quality developer training materials.

“A primary mission of The Linux Foundation is to make quality open source training and certification as accessible as possible,” said Linux Foundation General Manager of Training and Certification, Clyde Seepersad. “Konsulko has proven experience and reach in the IT training space, which will help even more individuals access these offerings, increase their career prospects, and help close the gap for open source talent.”

“Our instructors are both patient teachers and hands-on pioneers in embedded Linux development, which provides a great classroom experience for students of any skill level,” commented Peter Popov, Konsulko Group CEO and president. “We can also develop and teach custom classes on a variety of open source software topics including automotive, security and the Internet of Things.

Learn more at www.konsulko.com.


About Konsulko Group

An experienced leader in community and commercial development, Konsulko Group helps companies around the world build successful products with embedded Linux and open source software, offering consulting, capability, product engineering and support at every stage of every engagement. The Konsulko Group is headquartered in San Jose, Calif.

Media Contact – Konsulko Group:
Peter Popov

Konsulko Group

[email protected]