OSDev.org

I couldn't find this in the forum archives or the FAQ ... is there something special you have to do to make PCIe devices (appear to) interrupt the same way that regular PCI devices do?

I got a RTL8169 NIC driver working and wanted to make it work with the RTL8168/RTL8111, which is a PCIe version of the same architecture (with a few piddling differences which I *think* I've addressed in my init code but I accept that that could be the true problem). It sends Ethernet frames fine but never interrupts ... even though the chip's ISR shows the proper int status bits, and its IMR is set to enable those ints.

A little googling tells me that PCIe has no interrupt lines whatever and it's all emulated using MSI. Emulated by whom? Can I count on my ol' buddy the BIOS to have set this up for me? After all, it's the one that told me my device is on IRQ5 -- which incidentally is shared with a couple of other things, which has never been a problem before, but how can you wire-OR the interrupt lines if one of them has no wire? So there's no way that's going to just *happen* to work -- the MSI has to be pointed at some small device that asserts a physical int line, or else the BIOS has to do unspeakable evil to make it seem like that's what's happening (like how USB-to-PS/2 keyboard emulation is done).

Of course maybe none of this has anything to do with PCIe (hardware guys usually do an excellent job of maintaining backwards compatibility -- a lot better than some OSes I could mention, well actually ALL OSes), and it's just a regular interrupt bug, but I'm thrown by the fact that the ISR/IMR regs look like the Ethernet chip *thinks* it's interrupting.

If possible I really want to avoid embracing MSI for real, since even when I enable SMP I'm still programming the IOAPICS in virtual-wire mode so I can keep the 8259As in charge of ints. (I have to do that because another build of my system runs under DOS instead of being its own OS, so there may be externals drivers/TSRs that are catching interrupts, so I have to party like it's 1981 just to be compatible with them.) So I'm hoping I just need to unmask an MSI int bit, or enable a bus bridge (this is the only thing on bus #2), or some other thing.

Thanks!

John Wilson
D Bit

You can't avoid it. You MUST program the ethernet controller to do MSI. I'm not sure of the mechanics of this; but the BIOS will not do it for you. There is no emulation going on.

MSI is always sent directly to the local APIC(s).

By the way: Quite often the PIC is wired to one of the LINT pins on the BSP but not any APs - therefore, virtual wire mode will not distribute interrupts to other CPUs. Avoid using the PIC at all costs; it additionally adds a lot of cycles of bus overhead per interrupt, both reporting it and servicing it.

Groan! I was afraid it'd be something like that. Very bad news. But thanks for the answer.



That is all not only OK, but actually *required* in my config (if I don't find INTR on one of the BSP's LINTs, I look for whatever IOAPIC it turns up on). Since I need to support a DOS build, I handle all interrupts on the BSP anyway (otherwise there'd be no way to chain ints when IRQs are shared with a previously loaded TSR/driver that knows nothing of SMP and may even run in V86 mode). Yes, it's inefficient. Yes, it makes for bedlam on the BSP while the APs are eerily quiet places (SO unfair!). Yes, I also have to make sure I migrate to the BSP when I need to *call* DOS or the BIOS too. It's all horrifying but in practice it works great.

So I'm really hoping I can avoid having to change everything just for this one NIC driver (and the Via Velocity one I'll be debugging next). Now obviously it's no big deal for me to catch interrupts by whatever means and translate them to PUSHF / CLI / CALL FAR on the BSP (for the benefit of external drivers), but it could potentially get yucky if the external ISR tries to talk to the PIC in any serious way. If I assume all anyone ever does is write an EOI to 20h (and maybe 0A0h too) I'll probably get away with it 99% of the time, but I'd hate to have to virtualize the PIC (and add infinitely more complexity to my already too-complicated GPF handler for V86 mode) just for that other 1%. I'm really kind of shocked if there's really no route from PCIe to the PIC at all ... it seems out of character for PCI.

Thanks!!

John Wilson
D Bit

Actually it turns out that once again the HW guys did it right and the SW guys had blown it -- a BIOS update fixed my problem and now my PCIe ints (with RTL8111B and VT6130 Ethernet) work the same as with PCI. A long and boring Google break showed that PCIe has Assert_INTx and Deassert_INTx messages which emulate PCI's INTA::INTD pins (omitted on PCIe) -- including the fact that they're level-triggered (unlike MSI ints which are edge-triggered, presumably because there's no reason for them to be shared). I guess the "root complex" (is that really what the PCIe bridge is called?) takes care of converting the messages to real INTx lines (that can be ORed with other devices). I'm sure that's not what they plan for the far future but for now it works and life is groovy. Of course my code is still riddled with bugs but that's my problem.

John Wilson
D Bit

Hi, Can you elaborate on the emulated messages? Do I need to enable anything in the APIC for them to be able to interpret the messages to INTx signals?

Hi,



That sounds like a major over-simplification to me.

For MSI, as far as PCI is concerned, it's not really any different to any other "write to the physical address space" issued by a device. The trickery is that the address being written to just happens to correspond to the local APIC/s and the local APIC interprets this write as an IRQ occurring.

For legacy PCI IRQs, there's a special "legacy IRQ that is not a write to the physical address space at all" message from the device to the PCI host; where (at least conceptually) the PCI host converts this special message into a signal on a wire that's connected to all the legacy stuff in the chipset (possibly an "IMCR" thing to disconnect/connect wires to IO APIC or PIC, an IO APIC input, a "PCI to ISA IRQ" router, ...).

It is impossible to emulate legacy IRQs (which need to be a signal on a wire connected indirectly to bunch of stuff) with MSI (which completely bypasses the IO APIC and PIC and goes directly to local APIC/s).



Don't mix PIC and IO APIC.

Once upon a time (a long time ago) this was pointless but "supported in theory, sort of, for some chipsets and not others". Then ACPI was invented. ACPI only supports 2 possibilities - using PIC and not using IO APIC (default), or using IO APIC and not using PIC.



Your build system should be irrelevant - all it does is create and process files, possibly resulting in a small number of "final files" (e.g. maybe a single "bootable ISO image for CDs" file). It has nothing to do with what happens when the OS is running (e.g. when someone boots the "bootable CD").

I think what you mean here is that you decided to extend DOS (and use DOS at run-time and not just for a build system), to make sure that you have no hope of ever doing anything correctly due to the severe (and "undesirable for everyone involved") limitations of an OS that sadly became obsolete garbage 20 years ago (note: "sadly" because it should have became obsolete garbage over 30 years ago instead).


Cheers,

Brendan

Given that the conversation dates back over half a decade, and the OP (John Wilson) hasn't visited the forum for two years? Probably not.