OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 3:19 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 23 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Determining valid PCI address ranges
PostPosted: Wed Oct 13, 2021 5:59 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
So, I know that when the firmware initializes the hardware it assigns all the valid PCI device ranges and stuff (though I've no idea how it does this). By that, I mean that when you use the PCI I/O protocol in UEFI for example and you write to offset 0x00 of a devices BAR, the firmware knows if the write is a valid write or not and errors if its invalid. I'm wondering how I can do this within a kernel. Is this even possible or does each driver have to (manually) figure this stuff out on there own?

Another way to ask what I'm trying to say is: how would I go about determining the address range used by a given BAR (e.g.: if a device uses up to register offset 0x600, but only the datasheet says that, how does the firmware know that and is there a way I can tell that from something like ACPI)?


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Wed Oct 13, 2021 6:12 pm 
Offline
Member
Member
User avatar

Joined: Sun Apr 30, 2017 12:16 pm
Posts: 68
Location: Poland
You can check how large the BAR is by checking how many address bits the device decodes from it. To do that, write all 1s into the BAR, then read the value back. Only the uppermost N bits will remain as 1s since they're what the device actually looks at to determine if it belongs to that BAR, and the remaining bits (which are all 0s) form the offset into the BAR, which you can use to calculate it's size. Do note that the BARs have some hardwired type bits which you need to take care of (by masking them off).

_________________
Working on managarm.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Wed Oct 13, 2021 7:17 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
qookie wrote:
You can check how large the BAR is by checking how many address bits the device decodes from it. To do that, write all 1s into the BAR, then read the value back. Only the uppermost N bits will remain as 1s since they're what the device actually looks at to determine if it belongs to that BAR, and the remaining bits (which are all 0s) form the offset into the BAR, which you can use to calculate it's size. Do note that the BARs have some hardwired type bits which you need to take care of (by masking them off).

So if I understand you right, I'd write all ones to the BAR, read back the value and ignore bits 2:1, and then count all leading zeros?


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Wed Oct 13, 2021 8:00 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5100
The number of trailing bits you need to ignore (treat as always 0) depends on the type of BAR. For memory BARs, it's four bits. For I/O BARs, it's two.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Fri Oct 15, 2021 7:13 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
Octocontrabass wrote:
The number of trailing bits you need to ignore (treat as always 0) depends on the type of BAR. For memory BARs, it's four bits. For I/O BARs, it's two.

Thanks. Would that be in Mbytes? Like if I count all the zeros, and there are only 4, would that be 4 MB? (Supposedly - according to the PCIe spec -- I could also check the PCIe capabilities structure for some of this info too. But that means the device has to be PCIe capable.)
Edit: nevermind, figured it out thanks to the OSDev wiki article -- thanks for the help!


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Fri Oct 15, 2021 7:59 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
Okay... Final question and then I'll have figured this out. What do I do about 64-bit BARs? Do both BARs have the same size?


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Fri Oct 15, 2021 8:38 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5100
That question doesn't make sense. 64-bit BARs describe a single memory range, so there's one size. The only difference from a 32-bit BAR is that there are more address bits. The logic to find the size works exactly the same (but with more bits).


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Fri Oct 15, 2021 9:37 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
Octocontrabass wrote:
That question doesn't make sense. 64-bit BARs describe a single memory range, so there's one size. The only difference from a 32-bit BAR is that there are more address bits. The logic to find the size works exactly the same (but with more bits).

I think I got it... My code is below. I only scan BARs that are nonzero,and I don't convert 64-bit base addresses into their full addresses because I was getting ridiculous results (e.g. one device claimed it needed 2 << 37 bytes of address space which is ridiculous). Just want to make sure this is correct:
Code:
    if dev.header_type == 0x00 || dev.header_type == 0x01 {
        dev.bars
            .iter()
            .filter(|(idx, _)| **idx <= 5)
            .for_each(|(idx, bar)| {
                if *bar != 0x00 {
                    let i = idx;
                    let idx = match idx {
                        0 => BAR0,
                        1 => BAR1,
                        2 => BAR2,
                        3 => BAR3,
                        4 => BAR4,
                        5 => BAR5,
                        _ => 0,
                    };
                    let oldbar = read_dword(addr, idx);
                    write_dword(addr, idx, u32::MAX);
                    let bar = read_dword(addr, idx);
                    write_dword(addr, idx, oldbar);
                    let bar = if bar.get_bit(0) {
                        bar.get_bits(2..32)
                    } else {
                        bar.get_bits(4..32)
                    };
                    let bar = !bar;
                    info!("BAR {:X} consumes {} bytes", i, 2 << bar.count_ones());
                }
            });
    }

I ignore header type 2 because those addresses aren't directly mapped to the BAR registers like header types 0 and 1 are (though I should probably do those too...). This code gives me reasonable results:
Quote:
0:0:0:0: Found Bridge (Host bridge) with vendor ID 8086 and device ID 29C0
0:0:1:0: Found Display controller (VGA compatible controller) with vendor ID 1234 and device ID 1111
BAR 0 consumes 33554432 bytes
BAR 2 consumes 8192 bytes
0:0:2:0: Found Network controller (Ethernet controller) with vendor ID 8086 and device ID 10D3
BAR 0 consumes 262144 bytes
BAR 1 consumes 262144 bytes
BAR 2 consumes 64 bytes
BAR 3 consumes 32768 bytes
0:0:3:0: Found Mass storage controller (Non-Volatile memory controller) with vendor ID 1B36 and device ID 10
BAR 0 consumes 32768 bytes
0:0:4:0: Found Mass storage controller (SATA controller) with vendor ID 8086 and device ID 2922
BAR 4 consumes 64 bytes
BAR 5 consumes 8192 bytes
0:0:5:0: Found Multimedia controller (Audio device) with vendor ID 8086 and device ID 293E
BAR 0 consumes 32768 bytes
0:0:1D:0: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 2934
BAR 4 consumes 64 bytes
0:0:1D:1: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 2935
BAR 4 consumes 64 bytes
0:0:1D:2: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 2936
BAR 4 consumes 64 bytes
0:0:1D:7: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 293A
BAR 0 consumes 8192 bytes
0:0:1F:0: Found Bridge (ISA bridge) with vendor ID 8086 and device ID 2918
0:0:1F:2: Found Mass storage controller (SATA controller) with vendor ID 8086 and device ID 2922
BAR 4 consumes 64 bytes
BAR 5 consumes 8192 bytes
0:0:1F:3: Found Serial bus controller (SMBus) with vendor ID 8086 and device ID 2930
BAR 4 consumes 128 bytes
PCIe scan complete; 13 devices found

The display controller has a bit of a large address space but I chalk that up to it being a graphics card. The only flaw with this code is that it scans addresses that are (supposed) to be 64-bits so I need to skip those if I already scan their lower dwords.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Fri Oct 15, 2021 10:24 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5100
Ethin wrote:
I don't convert 64-bit base addresses into their full addresses because I was getting ridiculous results

Sounds like you might have been putting the two dwords the wrong way around when assembling the whole qword.

Code:
                    info!("BAR {:X} consumes {} bytes", i, 2 << bar.count_ones());

That should be a 1 instead of a 2. Right now, you're doubling all of the numbers you print.

Ethin wrote:
The display controller has a bit of a large address space but I chalk that up to it being a graphics card.

It's actually pretty small compared to a typical graphics card.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Fri Oct 15, 2021 10:49 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
Octocontrabass wrote:
Ethin wrote:
I don't convert 64-bit base addresses into their full addresses because I was getting ridiculous results

Sounds like you might have been putting the two dwords the wrong way around when assembling the whole qword.


I don't think so. I read the first BAR, then set bits 32..64 to the next BAR (BAR index + 4). Something like:
Code:
let bar = read_dword(addr, idx) as u64; // idx contains current BAR index
if bar.get_bits(1 ..= 2) == 0x02 {
let _ = bar.set_bits(32 .. 64, read_dword(addr, idx + 4) as u64);
}
// normal size calculation code

(I also wrote all ones to both BARs -- that might've messed everything up too.)

Octocontrabass wrote:
Code:
                    info!("BAR {:X} consumes {} bytes", i, 2 << bar.count_ones());

That should be a 1 instead of a 2. Right now, you're doubling all of the numbers you print.

Whoops?
Octocontrabass wrote:
Ethin wrote:
The display controller has a bit of a large address space but I chalk that up to it being a graphics card.

It's actually pretty small compared to a typical graphics card.

Oh okay, good to know. If I had a secondary GPU I'd test with real-world hardware. Same if I had PCI passthrough. But I don't have any IOMMU groups and I don't really have any HW I could pass through anyway.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Sat Oct 16, 2021 1:16 am 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
For reference, my PCI code is here. The lines concerned are lines 286-341. Building the qwords gives me a shift-by-left overflow panic, so I'm really, really confused.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Sat Oct 16, 2021 4:34 am 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 1593
You define "bar" as a dword by initializing it from "read_dword". I am not an expert on rust, but I'm going to assume variables work kind of like C++11's "auto", right? That would mean bar is a dword and shifting further than 32 bits does not work. But apparently you can specify a type explicitly, or so a short web search informed me.

_________________
Carpe diem!


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Sat Oct 16, 2021 10:56 am 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
nullplan wrote:
You define "bar" as a dword by initializing it from "read_dword". I am not an expert on rust, but I'm going to assume variables work kind of like C++11's "auto", right? That would mean bar is a dword and shifting further than 32 bits does not work. But apparently you can specify a type explicitly, or so a short web search informed me.

I type-cast that to a 64-bit integer, which results in zero-extension. I could just perform a qword read though... I presume PCIe supports that.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Sat Oct 16, 2021 2:47 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5100
Ethin wrote:
I type-cast that to a 64-bit integer, which results in zero-extension.

So, for BARs that only have 32 bits, did you continue to use those zero-filled bits in the later calculations? You're counting the number of bits that read back as zero after trying to set them to one, so that could give you an extra 32 zeroes.


Top
 Profile  
 
 Post subject: Re: Determining valid PCI address ranges
PostPosted: Sat Oct 16, 2021 4:39 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
So I'm (starting) to get (almost right) results with this code:
Code:
        let mut idx = 0;
        loop {
            if !dev.bars.contains_key(&idx) || dev.bars[&idx] == 0x00 {
                break;
            }
            let real_idx = match idx {
                0 => BAR0,
                1 => BAR1,
                2 => BAR2,
                3 => BAR3,
                4 => BAR4,
                5 => BAR5,
                _ => 0,
            };
            let oldbar = read_dword(addr, real_idx);
            let oldbar2 = if oldbar.get_bits(1..=2) == 0x02 {
                read_dword(addr, real_idx + 4)
            } else {
                0
            };
            write_dword(addr, real_idx, u32::MAX);
            if oldbar.get_bits(1..=2) == 0x02 {
                write_dword(addr, real_idx + 4, u32::MAX);
            }
            let mut bar = read_dword(addr, real_idx);
            let bar2 = if oldbar.get_bits(1..=2) == 0x02 {
                read_dword(addr, real_idx + 4)
            } else {
                0
            };
            write_dword(addr, real_idx, oldbar);
            if bar2 == 0x00 {
                write_dword(addr, real_idx + 4, oldbar2);
            }
            let ones = if bar2 == 0x00 {
                let bar = if bar.get_bit(0) {
                    bar.set_bits(0..2, 0)
                } else {
                    bar.set_bits(0..4, 0)
                };
                !bar.count_ones()
            } else {
                let mut bar = (bar2 as u64) << 32 | (bar as u64);
                if bar.get_bit(0) {
                    bar.set_bits(0..2, 0);
                } else {
                    bar.set_bits(0..4, 0);
                }
                !bar.count_ones()
            };
            info!("BAR {:X} consumes {} bytes", idx, 1 << ones.count_ones());
            if oldbar.get_bits(1..=2) == 0x02 {
                idx += 2;
            } else {
                idx += 1;
            }
        }

Doesn't quite work right though. Either skips BARs or tells me that a BAR needs 256M of RAM (which I don't think is right). Or it might be... Qemu args:
Code:
qemu-system-x86_64 -drive format=raw,file=/home/ethin/source/kernel/target/x86_64-kernel-none/debug/boot-uefi-kernel.img -drive if=pflash,format=raw,file=OVMF.fd,readonly=on -drive file=disk-nvme.img,if=none,id=NVME01 -device nvme,drive=NVME01,serial=0001 -drive id=disk,file=disk-sata.img,if=none -device ahci,id=ahci -device ide-hd,drive=disk,bus=ahci.0 -usb -rtc clock=host -cpu host -smp cpus=8 -M q35 -name kernel -nographic -debugcon file:qemu.log -global isa-debugcon.iobase=0x402 -d int -D qemu2.log -device usb-audio,audiodev=alsa -audiodev alsa,id=alsa -device ich9-intel-hda -device hda-duplex,audiodev=alsa -no-shutdown -no-reboot -device qemu-xhci -device virtio-net,netdev=nic -netdev user,hostname=kernel,id=nic -device virtio-balloon -device virtio-serial-pci -device virtio-rng-pci,rng=rng0 -object rng-random,id=rng0,filename=/dev/urandom -device virtio-gpu --enable-kvm


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 23 posts ]  Go to page 1, 2  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: 8infy, Bing [Bot] and 61 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group