rdos wrote:
I have a physical page that is mapped at one linear address in kernel space and at another linear address in user space. I designed it like this because I want the pages to have different access rights, with user space only having read access. The kernel linear address also is available in all processes.
However, I'm having transient issues where the kernel space page has been updated (which I can see through logs), but the user space page is not. The update might have happened on a different CPU core than the later read. The next time the page is read a bit later, the contents are updated.
The cache control bits for both pages are set to "normal state".
What could cause this problem? Isn't caching supposed to operate on physical addresses?
Cache protocols operate only at the cache level.
But CPU cores have store buffers, which might result in writes to memory being written to cache (and thus invalidated in remote caches) in a different order to the order of the writes in the program flow.
The order might be different because an earlier write is waiting for exclusive write access to a cache line, but a later write might be to a cache line the core already has write access to, so it can physically be written to the cache before the earlier write.
This will manifest itself in those writes appearing to happen in the reverse order in the remote cache, and so this is probably what you're seeing in your code.
You need a memory fence (lfence, sfence and mfence instructions on x86) to ensure writes are written to memory in a defined order. So, when updating some memory protected by a spin lock, you'll need a store fence between the last write to the protected data and the write to clear the spin lock. That barrier will ensure the protected data update is visible in other CPUs before the spin lock is unlocked by the latter write.