OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 4:35 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: MMIO accesses to the same register in parallel?
PostPosted: Tue Jul 12, 2022 1:48 am 
Offline
Member
Member

Joined: Mon Dec 07, 2020 8:09 am
Posts: 212
Tested my AHCI code on virtualbox and the port sometimes hangs. When it hangs, the set bit in port CI register never clears and no error bits are getting set.

Was quite a head scratcher until I took a look at the code on the other side. #-o

In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):

Code:
PortCmdIssue_w(...)
{
...
pAhciPort->regCI &= ~uCIValue;
...
}

PortCmdIssue_r(...)
{
...
pAhciPort->regCI &= ~uCIValue;
...
}


Without otherwise serializing MMIO accesses in virtualbox code, these two read-modify-write accesses to regCI could result in data corruption if invoked from different guest CPUs and the time lines up.

Adding serialization in my code for accesses to this register seems to have resolved the hang, sort of proving that there's no general MMIO serialization in virtualbox protecting the 2 functions above.

Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?


Top
 Profile  
 
 Post subject: Re: MMIO accesses to the same register in parallel?
PostPosted: Tue Jul 12, 2022 8:51 am 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 1593
It is probably about as great an idea as you would expect a data race to be. Two simultaneous accesses must be atomic or serialized. Otherwise you are always going to have a race condition.

_________________
Carpe diem!


Top
 Profile  
 
 Post subject: Re: MMIO accesses to the same register in parallel?
PostPosted: Tue Jul 12, 2022 12:05 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5100
The hardware doesn't know which CPU is accessing MMIO. If the order of access doesn't matter, you should be able to do it in parallel.

There may be architecture-specific limitations, though. For example, the x86 LOCK prefix tends to not actually lock the bus for MMIO, so you should use some other form of synchronization when performing a read-modify-write.


Top
 Profile  
 
 Post subject: Re: MMIO accesses to the same register in parallel?
PostPosted: Sat Jul 16, 2022 1:54 pm 
Offline
Member
Member
User avatar

Joined: Fri Jun 11, 2021 6:02 am
Posts: 96
Location: Belgium
xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?


If you use volatile accesses it should be fine (it's target dependent for LLVM, at least). AFAIK it is fine on all major platforms, especially on x86 where regular load/stores are always atomic.

In short:

xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?


Yes, provided you use volatile accesses (or maybe atomic, though volatile is the defacto way to do it).

(Of course, like Octocontrabass said, if the order matters you should use some synchronization primitive. Some hardware even provide dedicated bits in MMIO registers for that purpose).

xeyes wrote:
In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):


I'm pretty sure that's a bug and potentially a security hole in VirtualBox if there really is a race condition. I assume that VirtualBox does have some protection against that though.

_________________
My OS is Norost B (website, Github, sourcehut)
My filesystem is NRFS (Github, sourcehut)


Top
 Profile  
 
 Post subject: Re: MMIO accesses to the same register in parallel?
PostPosted: Sun Jul 17, 2022 2:41 am 
Offline
Member
Member

Joined: Mon Dec 07, 2020 8:09 am
Posts: 212
Octocontrabass wrote:
The hardware doesn't know which CPU is accessing MMIO. If the order of access doesn't matter, you should be able to do it in parallel.

There may be architecture-specific limitations, though. For example, the x86 LOCK prefix tends to not actually lock the bus for MMIO, so you should use some other form of synchronization when performing a read-modify-write.


Ordering seems okay in my case as data hazards among the ata commands are checked before the threads reach to the point of writing the CI register.

There's no point in RMW-ing CI though? Since the CPU can't clear bits in it.


Top
 Profile  
 
 Post subject: Re: MMIO accesses to the same register in parallel?
PostPosted: Sun Jul 17, 2022 2:45 am 
Offline
Member
Member

Joined: Mon Dec 07, 2020 8:09 am
Posts: 212
Demindiro wrote:
xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?


If you use volatile accesses it should be fine (it's target dependent for LLVM, at least). AFAIK it is fine on all major platforms, especially on x86 where regular load/stores are always atomic.

In short:

xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?


Yes, provided you use volatile accesses (or maybe atomic, though volatile is the defacto way to do it).

(Of course, like Octocontrabass said, if the order matters you should use some synchronization primitive. Some hardware even provide dedicated bits in MMIO registers for that purpose).



Yes I'm using volatile pointers and yes virtualbox is used as a x86 VMM in this case, compiler explorer seems like a really interesting site. :D

Demindiro wrote:

xeyes wrote:
In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):


I'm pretty sure that's a bug and potentially a security hole in VirtualBox if there really is a race condition. I assume that VirtualBox does have some protection against that though.



If parallel accesses are okay to perform, this code does look buggy. I didn't need the lock for other VMMs either. However, mainstream OSes don't seem to have any issue with this though, maybe they don't access this register in parallel?

Hopefully VirtualBox have protections against anyone trying to use this or other race conditions in their device code. That said, virtualbox isn't typically used on servers to run untrusted VMs either.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 71 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group