Profiling in IX

[ash995]

How would we profile applications running on top of IX and the IX code itself using profilers like oprofile and perf? We did try using perf to record events against the IX pid but the recorded samples show no symbols related to our application code or IX code.

[prekageo]

What symbols do you see? What do you see if you run perf script? IX runs in non-root mode. So, you have to profile the guest and pass to perf the ix binary as the guest kernel.

[ash995]

We see find_numasocket.isra.2 at the top of the profile consuming some 30 to 40 percent of the cycles. All the other symbols we see are kernel symbols.

So, are you suggesting we use perf kvm to profile IX and and our application code? If not so, it would be helpful if you could point us to the specific command and also how would we pass the IX binary to the perf profiler in a way such that the IX symbols are exposed to the perf profiler.

[prekageo]

Did you check the output of perf script? Have you read https://man7.org/linux/man-pages/man1/perf-record.1.html and https://man7.org/linux/man-pages/man1/perf-kvm.1.html? Specifically, take a look at the --guestkallsyms option.

[ash995]

We ran the following command

sudo perf kvm --guest --guestkallsyms=/home/fred/ix/dp/ix record -e cycles --pid 169253 sleep 15

Here pid is the IX process id as shown in top.

The output did change but now we see just addresses instead of actual symbols.

Also, we get the following warning when running the profiler.

[prekageo]

You are passing the wrong parameter in guestkallsyms. You should not pass the binary but its symbols in a format compatible to /proc/kallsyms.

[prantosky]

We have generated the symbols for IX, using the following command:
nm -anC ~/ix/dp/ix | c++filt > ix_syms

We think we get the same output as the format of /proc/kallsyms, the symbol file is attached.

We then start the recording using the following command,
sudo perf kvm --guest --guestkallsyms=~/ix_syms record -e cycles --pid <pid>

Later, after sudo perf kvm report, we get the following output,

Is there anything wrong with this procedure?

[prekageo]

It's been many years since I dealt with it but... why do you use --guestkallsyms on record? It makes sense to use it with report.

[prantosky]

We did try --guestkallsyms on record as well as report, still got the same output.