How to make hypercalls from kernel on P4080DS platform

[avasu]

Hello,

I am working on P4080DS plarform. I would like to know the procedure to make hypercalls from the user space in the kernel?

I referred to freescale documents and wrote a simple program that makes a hypercall.

Some of the header files are not present in the root file system. I would like to know if I have to include the hypervisor source in the root file system (my kernel accesses hard disk for RFS).

Thanks,

A Vasu

[scottwood · Accepted solution]

You cannot make hypercalls directly from userspace -- that would be a big security hole.  You need to go through the guest kernel.  Many of the hypercalls are already exposed through kernel interfaces, such as the management driver in drivers/virt/fsl_hypervisor.c.

Which hypercalls are you trying to use?

[avasu]

For example, if I have to start a stopped partition, I can make use of FSL_HV_IOCTL_PARTITION_START. How would I do it in that case? It would be great if I can have an example.

Also, I would like to read certain registers (say, PMR) from user space. How would I do it, if that is possible? I know I can use "mfpmr" instruction and read the registers. But I am not sure what header files must be included and how do I go about executing this instruction from a user space application.

[scottwood]

For partition start, use drivers/virt/fsl_hypervisor.c, either directly or through the "partman" tool.

Userspace PMR access should be no different under a hypervisor than without one.  You should be using a kernel interface such as perf event (which will also give you the option of only having the counters be active during a particular process, which would be hard to do if you're reading them directly).  If you really must read them directly (you will not be able to write directly), it's just an asm instruction.  There is no header.

[avasu]

Ok, I will go through partman for hypercalls.

I used mfpmr instruction in my user application. I got this linker error "undefined reference to mfpmr"

It would be great if you can tell me the exact usage.

Thanks for your help.

[scottwood]

mfpmr is an assembly instruction, not a function call.  I'm not going to give you the exact usage, because I don't think you should be using it that way in a Linux userspace program.  Use perf event.

[avasu]

Yes, I followed your instructions and wrote a kernel module which accesses the performance monitor counters. I had a question. How many events can be monitored at a time? Since we have 4 performance counters, PM0-PM3, I thought only 4 events can be mapped to the counters.

For example, PMC0 - mapped to Cache event, PMC1 - mapped to stash L2 hit event, and so on for PMC 2 and PMC3.

I would like to read or monitor at least 10 events at a time. Is this possible?

[scottwood]

I suggested using perf event, not writing a kernel module...

The hardware only supports 4 counters at a time.  If you use perf event, and create more than 4 counter events, it will timeslice them so that you'll get an approximation of the results you would have gotten with more counters.

[avasu]

I understand that I can use only 4 counters at a time. I tried using perf event for profiling. But, some of the events could not be opened. I get this error message while I try to run perf tool.

  Error: open_counter returned with 95 (Operation not supported). /bin/dmesg may provide additional information.

  Fatal: Not all events could be opened.

If the hardware supports the events that I want to monitor, Can I include those events in perf? Is it possible to do so?

[scottwood]

Which events are you trying to use?  Not all "generic" events map properly to what this hardware provides.  You can use raw events to monitor any hardware counter.

[avasu]

For example, for LLC-load-misses event, I get the error "Not all events could be opened".

[scottwood]

So, what do you even want to measure with "LLC"?  L2?  CPC?  CPC cannot be monitored with core-based performance counters, so let's assume L2.  Which hardware event should correspond to "L2 load misses"?  There's 115 for "L2 cache instruction hits" and 114 for "L2 cache instruction accesses", and you could subtract the two to get misses, but perf doesn't support complex mappings of that sort (as far as I'm aware).

I suggest you use raw events ("rNNN" where "NNN" is the hexadecimal event number) to specify exactly what you want to monitor.

[avasu]

Oh yes, this works perfectly fine. Thank you very much for your great support. One final question. Is it possible to find L3 cache misses or find an estimate of total memory access count? Since the hardware gives events for L1 and L2, we did not find anything relating to L3.

[scottwood]

CPC cannot be measured by the core performance monitor, and thus cannot be measured by perf event.  There are SoC-level counters that can measure this.  I believe the qoriq-dbg module that you referred to in another thread allows access to these counters.  For it to work under the hypervisor you'll need to grant the relevant device tree nodes to the guest (this doesn't require rebuilding the hypervisor, just modifying the configuration tree).

[experimentlearn]

Should it be done in hypervisor configuration tree hv.dts and then the device tree blob built? Can you give an example for this?

[avasu]

Hello,

I used raw events to monitor L1 data cache reloads, L1 data cache castouts, L2 cache data accesses and L2 cache data hits. I used perf API in my c program so that I can reset and enable the counters as and when I need to profile a piece of code.

I get the following values for the events for the piece of code given here:

int a[16];

for(i = 0; i < 16; i++){

     a[i] = i;

}

L1_cache_reloads       1

L1 cache castouts     0

L2 cache data accesses      18

L2 cache data hits                17

How can I interpret this data? The L2 data accesses are way too higher than L1 data cache reloads (L1 data misses).

Can someone explain what is going on?

I reset the counters and enable them before the for loop. The counting is disabled right after the for loop.

Thanks,

A Vasu

[scottwood]

These are counting different things.  You're only dealing with one cache line, so there was only one cache fill (reload), and it was never cast out.  But it was *accessed* numerous times.  I'm guessing this is on a chip that has write shadow mode enabled due to an erratum (i.e. rev2 of p4080, not rev3), so L1 is never dirty and all writes go through to the L2.

[avasu]

Hardware and hardware cache events must be monitored