OSDev.org

The Place to Start for Operating System Developers
It is currently Fri Apr 19, 2024 9:05 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 11 posts ] 
Author Message
 Post subject: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 1:57 pm 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1225
I have my 32-bit x86 test code here:
Imagehttp://devel.archefire.org/PRJ/OS/LowEST/LVL1/LowEST-Kernel-LEVEL-1-base--2017-07-14--src.zip


Inside the source directory "dbg/externalProgs/paging/pgtst0/", I have this main function which detects the presence or absence of a 4096-byte physical memory page (with paging disabled, only flat code and data segments):
Code:
;Inputs:
;        AL    -- initial sequential value to write
;       WIDESI -- 4096 bytes page base address
;       WIDEDI -- buffer to preserve original page contents
;
;;
check_page_existence__0000:
owide pushf


;INIT: Cache the original page contents
;INIT: Cache the original page contents
;INIT: Cache the original page contents
;INIT: Cache the original page contents

   push widecx
   push widesi
   push widedi

     xor widecx,widecx
     mov cx,(4096/WIDEWORD_SZ)
     rep movswide

   pop widedi
   pop widesi
   pop widecx

;END:  Cache the original page contents
;END:  Cache the original page contents
;END:  Cache the original page contents
;END:  Cache the original page contents




;INIT: Write the page
;INIT: Write the page
;INIT: Write the page
;INIT: Write the page

   ;NOTE: Since the initial test value will be normally random,
   ;      we could also normally call the function without setting
   ;      AL/WIDEAX and using whatever initial value is there,
   ;      but also saving it for being able to compare it later.
   ;;
    push wideax
    push widecx
    push widesi
    xor widecx,widecx
    mov cx,4096
    .l_alterpage:

     ;Prepare writing 4096 individual bytes
     ;from the original WIDESI, and increase
     ;the byte value and the address on each
     ;iteration to prove whether this physical
     ;page can maintain memory contents. It will
     ;probably be better to always allocate pages
     ;in this way even if it's slower overall for
     ;our system, but at least it will allow us to
     ;never use damaged memory even if it becomes
     ;damaged at run time under our OS, so this test
     ;for every time that we allocate or swap out
     ;a page will give us increased robustness.
     ;;
      mov byte[widesi],al
      inc widesi
      inc al
   
    dec widecx
    jnz .l_alterpage
    pop widesi
    pop widecx
    pop wideax

;END:  Write the page
;END:  Write the page
;END:  Write the page
;END:  Write the page




;INIT: See if it held its contents
;INIT: See if it held its contents
;INIT: See if it held its contents
;INIT: See if it held its contents


    push widecx
    push widesi
    xor widecx,widecx
    mov cx,4096
    .l_checkpage:

     ;Prepare reading 4096 individual bytes
     ;from the original WIDESI, and increase
     ;the byte value and the address on each
     ;iteration to see whether this physical
     ;page can maintain memory contents.
     ;
     ;If the current byte doesn't contain the
     ;value that it should, end indicating that
     ;the page didn't hold its contents; otherwise
     ;continue the test of 4096 bytes:
     ;;
      cmp byte[widesi],al
      jne .l_checkpage_end_notequal
      inc widesi
      inc al
   
    dec widecx
    jnz .l_checkpage
    .l_checkpage_end_equal:
     xor wideax,wideax   ;Indicate that the page held its contents
     inc al     


    jmp .l_checkpage_END
    .l_checkpage_end_notequal:
     xor wideax,wideax   ;Indicate that the page didn't hold its contents



    .l_checkpage_END:
    pop widesi
    pop widecx


;At this point, WIDEAX contains TRUE or FALSE
;for this function:
;;


;END:  See if it held its contents
;END:  See if it held its contents
;END:  See if it held its contents
;END:  See if it held its contents



;INIT: Restore original page contents
;INIT: Restore original page contents
;INIT: Restore original page contents
;INIT: Restore original page contents

   
   push widecx
   push widesi
   push widedi

    ;Make the cached page buffer the source memory
    ;and the original address the destination memory:
    ;;
     xchg widesi,widedi

    ;Copy it to restore:
    ;;
     xor widecx,widecx
     mov cx,(4096/WIDEWORD_SZ)
     rep movswide

   pop widedi
   pop widesi
   pop widecx


;END:  Restore original page contents
;END:  Restore original page contents
;END:  Restore original page contents
;END:  Restore original page contents




owide popf
retwide


;Should implement RETWIDE to automatically select between 16, 32 or 64-bit stack
;return address POPs.






;EOF




Now what I want to know is why some people say that the presence of memory cannot be detected manually. It seems unreasonable to me. That would imply that we wouldn't be able to detect bad memory areas either.

As far as I have tested this really detects memory manually.

Now, in my program I have the measure to start reading from physical address 1048576*2 because I assume that the system will run in a 386+ with at least 2MB of RAM.

This code is perfectly fine as it is, and I know that I will need to read the configuration of PCI devices and other likely known holes to skip like the one around 16MB to simply skip those hardware-specific areas to keep detecting regular memory.

Maybe people who says that it's not possible to detect memory manually haven't considered skipping present hardware-reserved regions or read memory from the first Megabyte, and the result is that they don't detect it correctly.

I assume that when paging is disabled, then the CPU cache memory is disabled too. Is that so?

If so, I don't see how this function could fail to correctly detect memory manually if you help it with detector functions that skip hardware-reserved memory areas dynamically and at least the very first Megabyte which we have to assume that is present in a 386+ at least for a starting workspace-ground-creating code version (and is useful if you just want to allocate new paged memory without detecting how much memory really is at boot time... there could be some machines that could report say 2GB but really have usable 4GB... like the Thinkpad T43 that is supposed to support up to 2GB but I have found some supposed units with 4GB on eBay).

_________________
Live PC 1: Image Live PC 2: Image

YouTube:
http://youtube.com/@AltComp126/streams
http://youtube.com/@proyectos/streams

http://master.dl.sourceforge.net/projec ... 7z?viasf=1


Last edited by ~ on Tue Feb 27, 2018 2:03 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 2:05 pm 
Offline
Member
Member
User avatar

Joined: Sat Dec 27, 2014 9:11 am
Posts: 901
Location: Maadi, Cairo, Egypt
~ wrote:
This code is perfectly fine as it is, and I know that I will need to read the configuration of PCI devices and other likely known holes to skip like the one around 16MB to simply skip those hardware-specific areas to keep detecting regular memory.

Some devices use MMIO but don't use PCI (i.e. HPET, local APIC, I/O APIC, etc.) Maybe you can use the ACPI tables to handle these cases, but what if somebody invented a new MMIO non-PCI device today? That would break your code. Every non-PCI MMIO device that your code is not aware of will break it.

~ wrote:
I assume that when paging is disabled, then the CPU cache memory is disabled too. Is that so?

No. You manually disable the CPU's cache by writing the cache disable bits in CR0, or playing with MTRR registers or PAT, although PAT is irrelevant because you say you are not using paging.

All in all, you shouldn't detect memory manually, even if it works now. First, because you may write to MMIO regions as I have said. Second, because this is very slow on PCs with a lot of RAM. The functions provided by BIOS/EFI are sufficient and decent enough to properly detect memory.

_________________
You know your OS is advanced when you stop using the Intel programming guide as a reference.


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 2:26 pm 
Offline
Member
Member
User avatar

Joined: Fri Feb 17, 2017 4:01 pm
Posts: 642
Location: Ukraine, Bachmut
^ exactly. the memory has been initialized by the FW, so this is why it is "detectable". But if the FW intialized the memory, it definitely knows about its map best. So why one should make such a useless things as above instead of just asking FW for a memory map?

on a non-x86 architectures such an approach 100% would hang the system instantly. there everything is memory mapped and this is done the way the vendor wants and without knowing their exact decisions, you can't step one step into memory reliably. even innocent read will hang the system. or, in the better case, you make blinking some leds. :D

_________________
ANT - NT-like OS for x64 and arm64.
efify - UEFI for a couple of boards (mips and arm). suspended due to lost of all the target park boards (russians destroyed our town).


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 2:27 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

~ wrote:
Now what I want to know is why some people say that the presence of memory cannot be detected manually. It seems unreasonable to me. That would imply that we wouldn't be able to detect bad memory areas either.


I can't remember anyone ever saying that it's not possible (they all just say it's extremely stupid and has many major pitfalls). An example is our own wiki that says it's "discouraged" (and then describes major pitfalls).

~ wrote:
As far as I have tested this really detects memory manually.


Out of the 12345 billion computers that exist (and the 12345 billion computers that don't exist yet but will exist in the next 20 years), what percentage of computers did you test? My guess is, not enough to be statistically relevant.

~ wrote:
This code is perfectly fine as it is, and I know that I will need to read the configuration of PCI devices and other likely known holes to skip like the one around 16MB to simply skip those hardware-specific areas to keep detecting regular memory.


The hole just below 16 MiB is typically used by ISA devices that (unlike PCI devices) have no detection/enumeration mechanism. If you had a few hundred ISA (and "VESA Local Bus") SVGA video card drivers you still wouldn't be able to determine if the area is used by a device that you don't have a driver for.

Also note that it's impossible to tell the difference between "RAM that is free for the OS to use" and "RAM that is reserved because it's used by firmware that the OS should not touch".


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 2:37 pm 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1225
Brendan wrote:
Hi,

~ wrote:
I can't remember anyone ever saying that it's not possible (they all just say it's extremely stupid and has many major pitfalls). An example is our own wiki
that says it's "discouraged" (and then describes major pitfalls).
It could imply that trying to detect bad memory manually could be useless but even then I feel that it's unreasonable to say (what if it fails at run time but you have paging code that checks that the page isn't damaged every time you allocate it or swap it back from disk to avoid using a suddenly damaged area, it could increase reliability in a very critical system).



Brendan wrote:
~ wrote:
As far as I have tested this really detects memory manually.


Out of the 12345 billion computers that exist (and the 12345 billion computers that don't exist yet but will exist in the next 20 years), what percentage of computers did you test? My guess is, not enough to be statistically relevant.
All K8MM-V AMD64 Athlon 3000+ machines with an nVidia 512MB AGP card installed, a standard PCI LifeView FlyTV FlyVideo 2000 from 2004, and 2GB of RAM installed, using PS/2 keyboard and a symmetrical form factor serial/PS/2-capable trackball A4 Tech mouse, booting and executing all from floppy. If you test one specific machine model you can assume that you have tested them all and you are only left with figuring out how to detect hardware errors and tricks for that specific model.

_________________
Live PC 1: Image Live PC 2: Image

YouTube:
http://youtube.com/@AltComp126/streams
http://youtube.com/@proyectos/streams

http://master.dl.sourceforge.net/projec ... 7z?viasf=1


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 2:50 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

~ wrote:
If you test one specific machine model you can assume that you have tested them all and you are only left with figuring out how to detect hardware errors and tricks for that specific model.


Nonsense.

If you test one K8MM-V AMD64 3000+ machine with an nVidia 512MB AGP card and it works; then changing the amount of RAM could make it fail, changing any device (e.g. different video card, new SCSI RAID controller, ...) could make it fail, changing the motherboard could make it fail, and changing the firmware (e.g. "firmware update") could make it fail.

Note: To be completely honest; I've had "very cautious" manual detection code in every single version of my OS for the last 10 years or more. This code is only ever used as a fall-back when every other possible option ("int 0x15, eax=0xE820", "int 0x15, eax=0xE801", ..) has failed.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 5:04 pm 
Offline
Member
Member

Joined: Thu Aug 13, 2015 4:57 pm
Posts: 384
Brendan wrote:
Note: To be completely honest; I've had "very cautious" manual detection code in every single version of my OS for the last 10 years or more. This code is only ever used as a fall-back when every other possible option ("int 0x15, eax=0xE820", "int 0x15, eax=0xE801", ..) has failed.

Why? I would assume any reasonably recent PC would support E820 (or UEFI), if it's not supported I'd just give an error message. I think I've read that you plan to support 386+, are there any cases that E820 (or the others) fail?

Seems a bit pointless to me, to each their own I guess =)

And for ~, nobody said detecting bad memory is a bad idea. They say that manual detection of memory map is bad idea. Get the map from firmware and then test the memory in the map as much as you like, but don't go poking at the physical memory address space, since you don't know what's there and what effect it would have.

Note, if you want to do memory testing because the system is critical, then you probably should continue doing the tests during run time as well, not just during boot. Probing the entire up to 64-bit physical address space would seem very wasteful to me, especially if you keep doing it during run time.

Note also that manual probing will also slow down your boot where as testing memory doesn't have to (to a significant degree). Didn't check your method, but remember that if you only check first byte of 4KiB aligned "segments", there's no guarantee that a part of the 4KiB "segment" isn't used for something else and might even be write protected, so you have to test all 4KiB reliably.


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 5:25 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

LtG wrote:
Brendan wrote:
Note: To be completely honest; I've had "very cautious" manual detection code in every single version of my OS for the last 10 years or more. This code is only ever used as a fall-back when every other possible option ("int 0x15, eax=0xE820", "int 0x15, eax=0xE801", ..) has failed.

Why? I would assume any reasonably recent PC would support E820 (or UEFI), if it's not supported I'd just give an error message. I think I've read that you plan to support 386+, are there any cases that E820 (or the others) fail?


Because I support "80486 and later" and have an old 80486 machine where everything else fails. Note: From memory, I think that specific computer gets down to "CMOS locations 0x70 and 0x71" where my code decides that the value may have been limited to the first 16 MiB and then it probes for memory above 16 MiB; and it doesn't go all the way down to the "probe for memory above 1 MiB" case.

For why I still support 80486, that's more of an "old habits" kind of thing - they were relatively new back when I started OS development and I never really found a good reason to stop supporting them.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 6:47 pm 
Offline
Member
Member

Joined: Thu Aug 13, 2015 4:57 pm
Posts: 384
Brendan wrote:
For why I still support 80486, that's more of an "old habits" kind of thing - they were relatively new back when I started OS development and I never really found a good reason to stop supporting them.

That's something I struggle with too, though more recently I've been thinking of dropping* support for 32-bit as well as UP, so only supporting 64-bit SMP. For me the kernel is tiny, so I decided some time ago that I'm going to treat all four combinations separately:
32-bit UP
32-bit SMP
64-bit UP
64-bit SMP

Given the relatively big differences between 32 vs 64 and UP vs SMP, and that I plan on keeping my kernel tiny which makes the common parts even smaller, so seemed like a good idea to treat all four as separate archs. But I guess it's getting a bit OT at this point..

*) Prioritizing might be more accurate, nothing really prevents me from doing the other three kernels later, but doesn't seem likely I'll ever find the time.


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Wed Jun 14, 2017 7:44 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

LtG wrote:
Brendan wrote:
For why I still support 80486, that's more of an "old habits" kind of thing - they were relatively new back when I started OS development and I never really found a good reason to stop supporting them.

That's something I struggle with too, though more recently I've been thinking of dropping* support for 32-bit as well as UP, so only supporting 64-bit SMP. For me the kernel is tiny, so I decided some time ago that I'm going to treat all four combinations separately:
32-bit UP
32-bit SMP
64-bit UP
64-bit SMP

Given the relatively big differences between 32 vs 64 and UP vs SMP, and that I plan on keeping my kernel tiny which makes the common parts even smaller, so seemed like a good idea to treat all four as separate archs. But I guess it's getting a bit OT at this point..

*) Prioritizing might be more accurate, nothing really prevents me from doing the other three kernels later, but doesn't seem likely I'll ever find the time.


I used to have 6 kernels (like yours, but with "32-bit plain paging" and "32-bit PAE paging" separate); then I shifted to "modular micro-kernel" where pieces are combined at boot (e.g. "several scheduler modules, several virtual memory manager modules, .... = many permutations"); then I got a waylaid by language/compiler/tool design and did some research into that; then I decided to quickly slap together the least possible for "80486 single CPU, plain paging" so I could build tools on top of it and then use my tools to rewrite everything one piece at a time and write all the pieces I skipped.

That "quickly slap together the least possible" decision was about 2 years ago now. So far I've discovered that I'm extremely bad at "quickly slap together the least possible" (for the current version of my OS I still don't have any pieces of any kernel; but I did some useful research into 3D rendering and have far better security now; so I got that going for me, which is nice! :roll:).


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: Detecting 4096-byte Memory Pages Manually
PostPosted: Thu Jun 15, 2017 4:15 pm 
Offline
Member
Member
User avatar

Joined: Fri Oct 27, 2006 9:42 am
Posts: 1925
Location: Athens, GA, USA
Is that what the Lehma told you? :P

note to self - find better way to mimic in text that Brooklyn (or was it Bronx?) accent Murray was using

_________________
Rev. First Speaker Schol-R-LEA;2 LCF ELF JAM POEE KoR KCO PPWMTF
Ordo OS Project
Lisp programmers tend to seem very odd to outsiders, just like anyone else who has had a religious experience they can't quite explain to others.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 82 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group