OSDev.org

The Place to Start for Operating System Developers
It is currently Wed Nov 25, 2020 10:51 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 14 posts ] 
Author Message
 Post subject: [SOLVED]xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Sun Oct 18, 2020 11:23 pm 
Offline
User avatar

Joined: Mon Feb 24, 2020 10:18 pm
Posts: 18
Location: Japan
Hello, I'm developing my xHCI drivers. I reset xHC by following the instructions of the Intel xHCI manual. An interrupt comes from xHCI on my laptop, but no interruption happens on QEMU. I suspect this is caused by my wrong configuration of the MSI-X table because the laptop has only MSI capability for xHCI, and QEMU has only MSI-X capability for it.

My full code: https://github.com/toku-sa-n/ramen/tree ... _from_qemu

This is the initialization of MSI-X. (kernel/src/device/pci/config/extended_capability/msi_x.rs)
Code:
impl<'a> CapabilitySpec for MsiX<'a> {
    fn init_for_xhci(&self, config_type_spec: &TypeSpec) {
        let base_address = config_type_spec.base_address(self.bir());
        let mut table = self.table(base_address);

        let pending_base = config_type_spec.base_address(self.pending_bir());
        self.pending_bit_table(pending_base)[0] = 1;
        table[0].init_for_xhci();

        self.enable_interrupt();
    }
}



methods
Code:
impl<'a> MsiX<'a> {
    fn enable_interrupt(&self) {
        let val = self.registers.get(self.base) | 0xf000_0000;
        self.registers.set(self.base, val);
    }
}
// kernel/src/device/pci/config/extended_capability/msi_x.rs
bitfield! {
    #[derive(Debug)]
    #[repr(transparent)]
    struct Element(u128);

    u32, from into MessageAddress, message_address,set_message_address: 31, 0;
    u32, from into MessageData, message_data, set_message_data: 95, 64;
    masked, set_mask: 96;
}
impl Element {
    fn init_for_xhci(&mut self) {
        self.message_address().init_for_xhci();
        self.message_data().init_for_xhci();
        self.set_mask(false);
    }
}

// kernel/src/device/pci/config/extended_capability/mod.rs
bitfield! {
    #[repr(transparent)]
    pub struct MessageAddress(u32);

    redirection_hint, set_redirection_hint: 3;
    u8, destination_id, set_destination_id: 19, 12;
    fixed_value, set_fixed_value: 31, 20;
}

impl MessageAddress {
    pub fn init_for_xhci(&mut self) {
        info!("LOCAL APIC ID: {}", Self::get_local_apic_id());
        self.set_destination_id(Self::get_local_apic_id());
        self.set_redirection_hint(false);
        self.set_fixed_value(0xfee);
    }

    fn get_local_apic_id() -> u8 {
        let accessor = single_object::Accessor::<u32>::new(LOCAL_APIC_ID_REGISTER_ADDR, 0);
        u8::try_from(*accessor >> 24).unwrap()
    }
}
bitfield! {
    #[repr(transparent)]
    pub struct MessageData(u32);

    vector, set_vector: 7, 0;
    delivery_mode, set_delivery_mode: 10, 8;
    level, set_level: 14;
    trigger_mode, set_trigger_mode: 15;
}

impl MessageData {
    pub fn init_for_xhci(&mut self) {
        self.set_level_trigger();
        self.set_vector(0x40);
        self.set_delivery_mode(0);
    }

    fn set_level_trigger(&mut self) {
        self.set_trigger_mode(true);
        self.set_level(true);
    }
}


Am I missing something?

_________________
Ramen OS: https://github.com/toku-sa-n/ramen


Last edited by tokusan on Wed Oct 21, 2020 3:06 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Mon Oct 19, 2020 6:35 pm 
Offline
Member
Member
User avatar

Joined: Sat Nov 22, 2014 6:33 pm
Posts: 744
Location: USA
tokusan wrote:
Hello, I'm developing my xHCI drivers. I reset xHC by following the instructions of the Intel xHCI manual. An interrupt comes from xHCI on my laptop, but no interruption happens on QEMU. I suspect this is caused by my wrong configuration of the MSI-X table because the laptop has only MSI capability for xHCI, and QEMU has only MSI-X capability for it.

I don't read RUST that well, so I can't decipher your code.

Have you set bit 15 in the PCIe Configuration-->Extended Capabilities-->MSIX Configuration WORD to enable MSI-X? With this being said, did you disable MSI with bit 0 in the MSI Configuration WORD?

Also, the comment that QEMU "only" supports MSI-X may be wrong, though I don't think you meant to mean it that way. QEMU supports PIN based interrupts as well.

Ben
- http://www.fysnet.net/the_universal_serial_bus.htm


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Mon Oct 19, 2020 7:20 pm 
Offline
User avatar

Joined: Mon Feb 24, 2020 10:18 pm
Posts: 18
Location: Japan
BenLunt wrote:
tokusan wrote:
Hello, I'm developing my xHCI drivers. I reset xHC by following the instructions of the Intel xHCI manual. An interrupt comes from xHCI on my laptop, but no interruption happens on QEMU. I suspect this is caused by my wrong configuration of the MSI-X table because the laptop has only MSI capability for xHCI, and QEMU has only MSI-X capability for it.

I don't read RUST that well, so I can't decipher your code.

Have you set bit 15 in the PCIe Configuration-->Extended Capabilities-->MSIX Configuration WORD to enable MSI-X? With this being said, did you disable MSI with bit 0 in the MSI Configuration WORD?

Also, the comment that QEMU "only" supports MSI-X may be wrong, though I don't think you meant to mean it that way. QEMU supports PIN based interrupts as well.

Ben
- http://www.fysnet.net/the_universal_serial_bus.htm


Thanks for your reply. Yes, I set the enable bit of MSI-X message control register. I can't disable MSI because there is no MSI capability. PCI capability pointer points MSI-X capability and the next pointer of MSI-X cap is 0.

And yes, I meant "only" to say QEMU doesn't have MSI capability. I forgot about PIN based interruption.

_________________
Ramen OS: https://github.com/toku-sa-n/ramen


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Tue Oct 20, 2020 2:18 am 
Offline
Member
Member

Joined: Sat Jul 02, 2016 7:02 am
Posts: 145
From qemu's debugprints and traces:

Code:
17911@1603181440.227296:usb_xhci_runtime_write off 0x0028, val 0x00000100
17911@1603181440.227308:usb_xhci_runtime_write off 0x0030, val 0x10ce9000
17911@1603181440.227311:usb_xhci_runtime_write off 0x0034, val 0x00000000
xhci: invalid value for ERSTSZ: 256
xhci: asserted controller error
...
17911@1603181440.227383:usb_xhci_oper_read off 0x0004, ret 0x00001000  <- USBSTS_HCE


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Tue Oct 20, 2020 2:22 am 
Offline
User avatar

Joined: Mon Feb 24, 2020 10:18 pm
Posts: 18
Location: Japan
linuxyne wrote:
From qemu's debugprints and traces:

Code:
17911@1603181440.227296:usb_xhci_runtime_write off 0x0028, val 0x00000100
17911@1603181440.227308:usb_xhci_runtime_write off 0x0030, val 0x10ce9000
17911@1603181440.227311:usb_xhci_runtime_write off 0x0034, val 0x00000000
xhci: invalid value for ERSTSZ: 256
xhci: asserted controller error
...
17911@1603181440.227383:usb_xhci_oper_read off 0x0004, ret 0x00001000  <- USBSTS_HCE

Thanks! I'll investigate this. BTW how did you print these debug info?

_________________
Ramen OS: https://github.com/toku-sa-n/ramen


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Tue Oct 20, 2020 2:26 am 
Offline
Member
Member

Joined: Sat Jul 02, 2016 7:02 am
Posts: 145
tokusan wrote:
BTW how did you print these debug info?


The lines prefixed with numerical info (most likely PID, etc.) can be enabled at the qemu monitor by the command

Code:
trace-event usb_xhci* on


The other two lines (that begin with xhci:) requires rebuilding qemu with an additional -DDEBUG_XHCI entry into its cflags (or equivalently uncommenting the corresponding #define in the source).


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Tue Oct 20, 2020 6:26 am 
Offline
User avatar

Joined: Mon Feb 24, 2020 10:18 pm
Posts: 18
Location: Japan
Thanks. I fixed the value of ERSTSZ, and it seems event ring TRB is generated (QEMU log says that). Still no interruption happens...

_________________
Ramen OS: https://github.com/toku-sa-n/ramen


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Tue Oct 20, 2020 9:52 am 
Offline
Member
Member

Joined: Sat Jul 02, 2016 7:02 am
Posts: 145
- qemu ignores MMIO writes to PBA.
- The vectors are all masked in the msix table.

Edit:
Code:
    fn enable_interrupt(&self) {
        let val = self.registers.get(self.base) | 0xf000_0000;
        self.registers.set(self.base, val);
    }


It seems the above is trying to enable msi-x, but it also sets a mask:

Code:
#define  PCI_MSIX_FLAGS_MASKALL 0x4000  /* Mask all vectors for this function */
#define  PCI_MSIX_FLAGS_ENABLE  0x8000  /* MSI-X enable */


You may want to OR with 0x8000_0000 alone.


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Tue Oct 20, 2020 10:07 pm 
Offline
User avatar

Joined: Mon Feb 24, 2020 10:18 pm
Posts: 18
Location: Japan
linuxyne wrote:
- qemu ignores MMIO writes to PBA.
- The vectors are all masked in the msix table.

Edit:
Code:
    fn enable_interrupt(&self) {
        let val = self.registers.get(self.base) | 0xf000_0000;
        self.registers.set(self.base, val);
    }


It seems the above is trying to enable msi-x, but it also sets a mask:

Code:
#define  PCI_MSIX_FLAGS_MASKALL 0x4000  /* Mask all vectors for this function */
#define  PCI_MSIX_FLAGS_ENABLE  0x8000  /* MSI-X enable */


You may want to OR with 0x8000_0000 alone.

Thanks. I fixed it. Although no changes occur, I noticed that MSI-X table is in fact not set correctly. I may use bitfield library wrongly. I'll continue investigating.

_________________
Ramen OS: https://github.com/toku-sa-n/ramen


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Tue Oct 20, 2020 10:13 pm 
Offline
Member
Member

Joined: Sat Jul 02, 2016 7:02 am
Posts: 145
The way you are trying to generate the interrupt depends on writing to the PBA, and since qemu ignores such writes, that method will not work.

The other fixes are necessary for proper functioning of the driver, but they alone will not be able to overcome the limitation that qemu ignores PBA writes.


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Wed Oct 21, 2020 3:05 am 
Offline
User avatar

Joined: Mon Feb 24, 2020 10:18 pm
Posts: 18
Location: Japan
linuxyne wrote:
The way you are trying to generate the interrupt depends on writing to the PBA, and since qemu ignores such writes, that method will not work.

The other fixes are necessary for proper functioning of the driver, but they alone will not be able to overcome the limitation that qemu ignores PBA writes.

I confirmed that we can't use PBA on QEMU, but then I can't understand why because there are lines related to PBA in QEMU's source code (inside hw/pci/msix.c). Anyway, I'll write codes which doesn't depend on the interruption from xHCI. Thank you for spending time for me.

_________________
Ramen OS: https://github.com/toku-sa-n/ramen


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Wed Oct 21, 2020 3:46 am 
Offline
Member
Member

Joined: Sat Jul 02, 2016 7:02 am
Posts: 145
tokusan wrote:
I confirmed that we can't use PBA on QEMU, but then I can't understand why because there are lines related to PBA in QEMU's source code (inside hw/pci/msix.c).


Probably because the driver/OS isn't expected to write to PBA; it is managed by the device.
[Edit: PBA is indeed supported by qemu; it just doesn't allow the guest to write to that memory area.]

Here is the patch that added a dummy handler for such writes to PBA.

If you are willing to test, you can change the function msix_pba_mmio_write to call msix_set_pending. That will allow the driver/OS to actually change the bit. [Edit: I tested this change in qemu with your latest fix_no_interrupt_from_qemu branch. The message "INFO: Interrupt from 0x40" is printed.]


Top
 Profile  
 
 Post subject: Re: xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Wed Oct 21, 2020 9:13 am 
Offline
User avatar

Joined: Mon Feb 24, 2020 10:18 pm
Posts: 18
Location: Japan
linuxyne wrote:
If you are willing to test, you can change the function msix_pba_mmio_write to call msix_set_pending. That will allow the driver/OS to actually change the bit. [Edit: I tested this change in qemu with your latest fix_no_interrupt_from_qemu branch. The message "INFO: Interrupt from 0x40" is printed.]

Could you show me the edited msix_pba_mmio_write? I tried several codes, but couldn't get an interruption.

(BTW I said that event TRB was generated, but it turned out that this was not true. I misinterpreted QEMU's output. But at least the problem of the interruptions is solved (the way I want to implement is seemingly not supported by QEMU), so I marked this thread as solved.)

_________________
Ramen OS: https://github.com/toku-sa-n/ramen


Top
 Profile  
 
 Post subject: Re: [SOLVED]xHC interruption with MSI-X on QEMU doesn't work
PostPosted: Wed Oct 21, 2020 9:27 am 
Offline
Member
Member

Joined: Sat Jul 02, 2016 7:02 am
Posts: 145
tokusan wrote:
Could you show me the edited msix_pba_mmio_write? I tried several codes, but couldn't get an interruption.


Below is a hack; it marks the vector# 0 as pending, regardless of the bit the driver/OS wants to
set in PBA. So, for testing, your code must always unmask vector# 0 (which it already does).

Code:
static void msix_pba_mmio_write(void *opaque, hwaddr addr,
                                uint64_t val, unsigned size)
{
   PCIDevice *dev = opaque;
        msix_set_pending(dev, 0);
}



tokusan wrote:
...(the way I want to implement is seemingly not supported by QEMU),...


That's true, but I suppose that that way is only for testing. I do not think that commercial production OS and drivers actually write to PBA. They might read PBA to display interrupt status info. Moreover, if you add a usb-storage device to qemu's xhci bus, and run Linux on the guest, I can see that the xhci driver uses MSI-X, so the MSI-X functionality in qemu does work.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 10 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group