Get multiple Performance Monitoring Interrupt on qemu-kvv

parfait · **Joined:** Sat Dec 14, 2013 12:33 pm **Posts:** 15

I am not able to get multiple performance monitoring interrupt (PMI - especially instruction counter) in my guest when running on qemu (kvm-enabled, cpu = host) unless I reset the cpu (triple fault).
The code below is intended to reprogram another PMI when it fires. It works fine on real machine but on qemu, after the first received PMI, no other follows. the only way to get another PMI, until now is to reset the CPU.

Code:

// This code handles the PMI and programs another one.
pmi_handler() {
    Console::print("PMI counter %llx perf_reg %x", Msr::read<uint64>(Msr::MSR_PERF_FIXED_CTR0), 
           read(LAPIC_LVT_PERFM));  // PMI counter 10 perf_reg a4
    program_pmi();
    Console::print("perf_reg %x", read(LAPIC_LVT_PERFM)); // perf_reg a4
} 

program_pmi(){
    uint64 msr_glb = Msr::read<uint64>(Msr::MSR_PERF_GLOBAL_CTRL);
    Msr::write(Msr::MSR_PERF_GLOBAL_CTRL, msr_glb | (1ull<<32));
    Msr::write(Msr::MSR_PERF_GLOBAL_OVF_CTRL, Msr::read<uint64>(Msr::MSR_PERF_GLOBAL_OVF_CTRL) & ~(1UL<<32));
    set_perf_lvt(LAPIC_LVT_PERFM, DLV_FIXED, VEC_LVT_PERFM);  // clear the pmc register mask bit
    Msr::write(Msr::MSR_PERF_FIXED_CTR0, 0xffffffff0000);
    Msr::write(Msr::MSR_PERF_FIXED_CTRL, 0xa);
}

However, notice that the pmc register mask bit is cleared (perf_reg[bit 16] = 0) just after the first pmi. This is not normal, according to Intel specifications.
Any Idea?

linuxyne · **Joined:** Sat Jul 02, 2016 7:02 am **Posts:** 210

The ordering in the code is:

Code:

    Msr::write(Msr::MSR_PERF_FIXED_CTRL, 0xa);
    Msr::write(Msr::MSR_PERF_FIXED_CTR0, 0xffffffff0000);

That ordering has been established on the other thread as ineffective (on kvm).

Examination of the source shows that the kernel automatically reprograms the event. What is the behaviour if we do nothing to reprogram?

parfait wrote:

However, notice that the pmc register mask bit is cleared (perf_reg[bit 16] = 0) just after the first pmi. This is not normal, according to Intel specifications.

Most likely reason: incomplete emulation of the APIC model.

parfait · **Joined:** Sat Dec 14, 2013 12:33 pm **Posts:** 15

linuxyne wrote:

That ordering has been established on the other thread as ineffective (on kvm).

That is right, that was an oversight. I would rather update the initial post.
The actual code takes this into account but the problem is the same.

linuxyne wrote:

parfait wrote:

However, notice that the pmc register mask bit is cleared (perf_reg[bit 16] = 0) just after the first pmi. This is not normal, according to Intel specifications.

Most likely reason: incomplete emulation of the APIC model.

Do you think it may be the root cause of the problem, not firing following PMI?

linuxyne · **Joined:** Sat Jul 02, 2016 7:02 am **Posts:** 210

parfait wrote:

The actual code takes this into account but the problem is the same.

The kernel does reprogram the counter, presumably with the intention of raising the next event in time for the counter. Is it possible for you to share the source for others to work with?

parfait wrote:

Do you think it may be the root cause of the problem, not firing following PMI?

The kernel provides partial support for the bit; it names it APIC_LVT_MASKED. It does not set it when raising the PMI. But, even for the emulation, that means that the PMIs are /not/ masked.

parfait · **Joined:** Sat Dec 14, 2013 12:33 pm **Posts:** 15

linuxyne wrote:

parfait wrote:

The actual code takes this into account but the problem is the same.

The kernel does reprogram the counter, presumably with the intention of raising the next event in time for the counter. Is it possible for you to share the source for others to work with?

Here is the link to the project. It is a fork from the Nova hypervisor code.
It is personal work, based on the nova code

linuxyne · **Joined:** Sat Jul 02, 2016 7:02 am **Posts:** 210

I downloaded the 'initial' branch. I was able to build the hypervisor, although I could not get it to work using grub-mkrescue iso cdrom. Grub does load the hypervisor, but it remains stuck somewhere in or after the 'init' function of the hypervisor. The display remains blank.

What steps should one perform to build a working qemu image?

I take it that in your tests, the kvm traces show a single NMI being raised, despite your attempts to reprogram. Is that correct?

The kvm raises PMU and PMI events at the time of overflow. In response to the PMU event, the kvm reprograms the counter the same way it programmed it the first time around. Regardless of the behaviour or its lack within the guest, the kvm traces should at least show multiple PMIs (NMI) being triggered.

parfait · **Joined:** Sat Dec 14, 2013 12:33 pm **Posts:** 15

linuxyne wrote:

I downloaded the 'initial' branch. I was able to build the hypervisor, although I could not get it to work using grub-mkrescue iso cdrom. Grub does load the hypervisor, but it remains stuck somewhere in or after the 'init' function of the hypervisor. The display remains blank.

What steps should one perform to build a working qemu image?

To make it work quickly, I suggest you also download the userland code (Genode) here because Nova is part of a bigger project (Genode OS framework).
You may follow the instructions on the project site, or follow this summary I provide here for the circumstance.
Clone the project into genode directory.

Code:

git clone github.com/genodelabs/genode.git

This folder will be the home directory for the project.
download the latest toolchain and uncompress it to the right location (/usr/local/genode-gcc)
https://sourceforge.net/projects/genode ... ain/17.05/

Code:

sudo tar xPf genode-toolchain-<version>-<arch>.tar.xz

Create the build directory (in this case, for 64 bit binary)

Code:

cd genode 
./tool/create_builddir x86_64

Ajust the build config according to your purpose: Open the file build/x86_64/etc/build.conf and Uncomment lines 78 (libports) 86 (ports) 96 (dde_rump) 103 (gems) 112 (world) 117 (dde_bsd) 122 (dde_ipxe)
Prepare kernel source (and other libs) ports:

Code:

./tool/ports/prepare_port x86emu nova grub2

This should create 3 folders in genode/contrib, one for each source code we import
The Nova code does not contain the Performance Monitoring Interrupt (PMI) code to test PMI. So
Replace the content of genode/contrib/nova-*/src/kernel/nova/ with the Nova code you pulled from my repo (or addapt it to your convenience to test PMI. The objective is to program another PMI each time we have one.
Nova is the kernel.
The file genode/repos/base/src/lib/startup/spec/x86_64/crt0.s contains the first instruction executed in user land. (Ring 3). Add

Code:

jmp start

at line 41 to loop forever just after sysretq from the kernel.
Build the demo scenario. It is just a simple scenario to show some features of Genode.

Code:

    cd build/x86_64/
   make run/demo KERNEL=nova

If all go well, this should produce a working qemu image in genode/build/x86_64/var/run/demo.iso
Now you may test it with

Code:

 qemu-system-x86_64 -enable-kvm -cpu host -m 5120 -net nic,model=e1000 -net user -serial mon:stdio -d int,guest_errors,unimp,pcall,cpu_reset -cdrom var/run/demo.iso

I hope i didn't forget anything. That should work.
If you have any trouble, I 'll be glad to answer

linuxyne wrote:

I take it that in your tests, the kvm traces show a single NMI being raised, despite your attempts to reprogram. Is that correct?

Correct!

Thank you for your help!

linuxyne · **Joined:** Sat Jul 02, 2016 7:02 am **Posts:** 210

Thanks for the steps. I was able to build the iso and capture the PMIs.

Since the kernel triggers the repgramming when we write to msr 0x38d with a value different than that which is already programmed, the below piece of code worked in triggerring multiple PMIs at the default vector 0xa4.

At any place where you need to reprogram the msr 0x38d, place these lines:

Code:

    Msr::write(Msr::MSR_PERF_FIXED_CTR0, 0xfffffffffff0ull);
    Msr::write(Msr::MSR_PERF_FIXED_CTRL, 0x0);
    Msr::write(Msr::MSR_PERF_FIXED_CTR0, 0xfffffffffff0ull);
    Msr::write(Msr::MSR_PERF_FIXED_CTRL, 0xa);

The switching between the values 0x0 and 0xa causes the kernel to reprogram the counter. The kernel does indeed already reprogram it automatically at the time of firing the PMI, but it will take some time debugging to figure out the reason that does not work.

parfait · **Joined:** Sat Dec 14, 2013 12:33 pm **Posts:** 15

Okay, I understand now.
Very glad to be able to use qemu-kvm again in my project! Those kvm behaviours really deserve to be documented.
Thank you Linuxyne, for shedding light on them.

linuxyne · **Joined:** Sat Jul 02, 2016 7:02 am **Posts:** 210

parfait wrote:

Those kvm behaviours really deserve to be documented.

They are (sort of), but in the form of the source code. For details about most behaviours, there might not exist any document other than the source.

OSDev.org

Get multiple Performance Monitoring Interrupt on qemu-kvv

Who is online