Curious case of memory corruption

[bdsy]

Hello

When running an ordinary Linux distro on an ordinary PC, I've got a file cached in memory corrupted in an unusual way. I'd really want to know the root cause rather than apply some generic advice in hopes the issue disappears. Perhaps you have some ideas.

Diff between the correct file and the corrupted one:

What I see here:

• A chunk of 4 bytes right before the next chunk
• A chunk of 256 bytes, 4096-aligned. There's a pattern of 8-byte groups:
  • Looks like a little-endian number that increases by 13.
  • Fifth byte is not overwritten.

Initially I noticed that something is wrong with the file, calculated its hash, and got a mismatch. I saved the corrupted copy to disk. The file was read multiple times in process from the memory cache so it couldn't be a one time read from RAM error. Also, I ran Memtest86+ for 19.5 hours (14 passes) and it found nothing. It also couldn't be a disk fault because the file was stored on an encrypted partition, so a corrupted block after decryption would've been something random-looking.

What's probably more important, I found the same issue reported online (in a place I'd rather not link to). In their case, the corruption happened in Windows driver code:

windows-chkimg.txt

(4.66 KiB) Downloaded 3037 times

So, the issue is not OS-specific. But the common part seems to be AMD Zen 4 CPU, B650 chipset and Gigabyte motherboard. I wonder if the corruption could've been caused by SMM? Or else, what kind of hardware failure can cause these patterns?

I'm planning to write a program that scans the whole RAM for this pattern to at least know the rate at which this occurs and potentially correlate it with other things happening on the PC.

[Octocontrabass]

I'd really want to know the root cause rather than apply some generic advice in hopes the issue disappears.

Vendors won't admit there's a problem if they can avoid it, but they'll fix problems anyway. That's why the generic advice like "update your BIOS" and "install components according to the latest version of the user manual and compatibility list" so often fixes problems.

Also, I ran Memtest86+ for 19.5 hours (14 passes) and it found nothing.

Memory tests don't always catch faulty RAM. Sometimes the fault only occurs with specific access patterns. This corruption doesn't look like faulty RAM, though.

I wonder if the corruption could've been caused by SMM?

It could be.

Or else, what kind of hardware failure can cause these patterns?

Any hardware component that can perform DMA can corrupt memory. If that's what's happening, Linux might be able to use the IOMMU to catch the offending device.

[bdsy]

Thank you for the reply! I'll try to see what I can do with IOMMU.

I'd really want to know the root cause rather than apply some generic advice in hopes the issue disappears.

Vendors won't admit there's a problem if they can avoid it, but they'll fix problems anyway. That's why the generic advice like "update your BIOS" and "install components according to the latest version of the user manual and compatibility list" so often fixes problems.

Now it looks like that's what I'm going to do eventually.

[mtretu]

Now it looks like that's what I'm going to do eventually.

Did your problem get fixed by a later BIOS update, and what hardware/CPU/GPU were you using, more specifically?

I also experienced the same type of memory corruption on a AMD AM5 Zen 4 CPU, Ryzen 5 7500F, but in my case on Asrock B650M PG Lightning motherboard. I had also a Vega 56 GPU with 8GB of VRAM.
A BIOS update "seems to" have fixed the problem.

• The corruption seemed to occur quite reliably after waking the PC from sleep, or - at least once - after a long period of idle on desktop, so it was perhaps related to either temperature or power states. It did not seem to really occur during active use (or at least I did not happen to notice), and 12+ hours of Memtest86+/Memtest86 did not detect anything wrong.
• Usual stuff like remounting the CPU / DIMMs did not help.
• The issue was OS-independent, on both memory channels, with 3 different RAM DIMMs at 4800/5200 MT/s, independently as well as with two channels filled (in configs of 16GB or 32GB or 48GB).
• I also tried ECC RAM (Kingston Server Premier UDIMM), verified in Linux that ECC is enabled, but EDAC did not detect any corrected or uncorrected errors when the corruption occurred.

With all that, and the corruption not being random, the problem certainly did not lie in the connection between the CPU and RAM, but happened at any point before CPU memory controller calculates ECC for the data. (Well, AFAIK on DDR5 UDIMM, address/command bus/CS are not protected by ECC/parity, but exact same corruption occurring on both channels seemed still quite impossible).

Swapping the CPU from the 2024 model to one manufactured in 2025 did not help, so probably not a single "bad batch".

However, BIOS update from Asrock BIOS 2.02 to 3.50 seems to have fixed the issue, though as manufacturers do not publish detailed changelogs it is impossible to know if the bug just occurs 100 times less often now.
BIOS 2.02 had CPU microcode A601206 and SMU firmware 84.79.223 (Agesa ComboAM5 1.1.0.0 apparently?), BIOS 3.50 has CPU microcode A60120C and SMU firmware 4.84.3.0 (Agesa ComboAM5 1.2.0.3g). I did not try the BIOS versions inbetween.
Still don't know if the problem was related to (firmware of) the CPU, chipset, or some other component on the motherboard though that might be less likely with our different motherboard manufacturers.


Below is a hex editor screenshot of the corruption that I captured by saving a corrupted RAM-cached file (correct data blacked out):

corrupted_bytes.jpg (60.76 KiB) Viewed 28495 times

The starting values are different but the format is same as yours, 4k-aligned main chunk, increment by 13, though if you look at the 8th 64-bit value in the main chunk where the value overflows, it doesn't seem like normal 64-bit little endian int byte order (0xBB->0xBC instead of 0x10->0x11)?
The corrupt values seemed to be always those, with the newer CPU too. I checked it maybe about 10 times.

The 4 bytes before the 256 bytes chunk were usually but not always there, sometimes I couldn't find them in the file, once it was about 600MB earlier; but since I don't know how the OS mapped the file in physical RAM, it could well have been just that the memory pages for the file weren't allocated linearly.