OSDev.org

The Place to Start for Operating System Developers
It is currently Wed Aug 05, 2020 11:14 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 25 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Oct 29, 2019 9:53 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1224
I created a new Administrator user, so the resource usage is almost nothing (the CPU cores are also running to 3% or 1% and up).

Having the full native power is well worth it (more than 4GB of RAM, 16/32/64-bit emulators, BIOS, DOS, DPMI, VGA, VESA, Sound Blaster, DirectX, Win16, ZSNES, NO$GMB, Z26, Nesticle, Turbo C, Pascal, all of the old demos).

But the question is, even with that much RAM, does Windows let so much fragmentation happen without swapping out as to have to close running windows programs to reserve 40 contiguous Megabytes? It looks like having that amount of tolerance to fragmentation would be enough most of the time for a simple paged OS.

The screenshot is Resource Hacker from SourceForge. 1.6GB is only 23% of usable PAE RAM. If I open 32 YouTube tabs in Firefox, it goes beyond 4GB:


Attachments:
More than 4GB with Win32.png
More than 4GB with Win32.png [ 52.1 KiB | Viewed 5144 times ]

_________________
http://www.ebay.com/usr/udocproject3 (Updated IP)
http://190.53.102.175/api/doc/en/ (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe
Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Oct 29, 2019 10:58 am 
Offline
Member
Member

Joined: Thu Aug 13, 2015 4:57 pm
Posts: 384
According to Wikipedia 32-bit Windows limit RAM to 4GiB, reason was that drivers didn't work well with more, alternatively it was due to licensing.
https://en.wikipedia.org/wiki/Physical_Address_Extension#Microsoft_Windows

You have a 64-bit CPU, why not use 64-bit OS?

As far as the memory allocation in half-life failing, have you disabled swapping? I would imagine that if swapping is enabled, then Windows will swap enough RAM to disk to make room for the 40MiB allocation. If all RAM is used and swapping is disabled, then all allocations will fail.


Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Oct 29, 2019 11:06 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1224
The swap file is 4GB and enabled. Maybe it should be 16GB for 12GB.

The 16-bit and DPMI help me make most of my OS. I can't abandon all the code for learning, 64-bit OSes turn into an unbearable desert when I want to learn more tricks for the lack of PC-specific code and hardware compatibility.

_________________
http://www.ebay.com/usr/udocproject3 (Updated IP)
http://190.53.102.175/api/doc/en/ (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe


Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Oct 29, 2019 11:17 am 
Offline
Member
Member

Joined: Thu Aug 13, 2015 4:57 pm
Posts: 384
~ wrote:
The 16-bit and DPMI help me make most of my OS. I can't abandon all the code for learning, 64-bit OSes turn into an unbearable desert when I want to learn more tricks for the lack of PC-specific code and hardware compatibility.

I don't understand what you are trying to say here.

Personally I prefer for my host OS to have all the resources it can (64-bit OS using all of 64GiB of RAM), and then I start a VM (Virtual Machine, like VMWare, Qemu, etc) that can have as little RAM as I want, and do there whatever I want.

So why don't you use a 64-bit OS, start a VM and do your 16-bit DPMI stuff there?


Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Oct 29, 2019 11:30 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1224
Every emulator layer has all the physical reserved Gigabytes less of RAM, so it's bad just for that when tiny things can be run in the real machine.

I have like 75 old books with floppy/CDs and other PDFs/ZIPs/ISOs that work under DOS or Win16 (VxDs, fonts, SVGA, 3D, file formats, WinAPI, MFC).

I have mirrors from programmersheaven, planetsourcecode and many FTP repositories that have Gigabytes of interesting tricks but in 16-bit mode.

I don't find material that teaches the basics from the mining principles for the components in 64-bit mode or in well-finished platforms like pure Linux or Windows, etc. I can learn in 16/32-bit environments and port to 64-bit or JavaScript. I can't run the code to see how the real hardware behaves with Win64.

I like to handle VGA/VESA graphics. With that, DPMI, my emulation, and Sound Blaster, Bochs, and some old Pentiums, I can implement my OS under Windows 7 without rebooting. What depends on the host OS can be replaced by ring 0 assembly components later, but an OS with GUI, manual VGA and SVGA for the current card, indirect network through special files, and native APIs can be done in Win32. I could add a 64-bit emulator for 32-bit CPUs, and a 16-bit one for 64-bit ones.
----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
Skeleton of Page-Disabled Functions

Code:
OPCODE__CPU_x86_32__page_disable_function_skeleton:
;Save flags and temporary registers,
;disable interrupts and paging,
;add the custom function code:
;;
pushfwide
push wideax
;;
cli
call OPCODE__CPU_Read_Default_Paging
push wideax

call OPCODE__CPU_Disable_Default_Paging
;;











;;
pop wideax
call OPCODE__CPU_Restore_Default_Paging
;;



pop wideax
popfwide
;Enable paging and restore temporary registers
;(restoring FLAGS will restore interrupts
;to their intended state):
;;
retwide





Skeleton of Page-Disabled Functions for cdecl/Local Variables



Skeleton of Page-Disabled Functions for stdcall/Local Variables



Get CR3 Value From CR3 Array Index
Code:
      ;Get the base address of the specified
      ;page directory in WIDEDX:
      ;;
       mov widedx,.CR3_array_index
       mov widedx,[CR3_array+(widedx*wideword_sz)]

_________________
http://www.ebay.com/usr/udocproject3 (Updated IP)
http://190.53.102.175/api/doc/en/ (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe


Last edited by ~ on Thu Nov 07, 2019 10:41 am, edited 5 times in total.

Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Oct 29, 2019 1:24 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 150
Location: North Dakota, United States
I'm quite confused on how an emulator/virtual machine can't satisfy your needs. (Then again, I'm quite confused on why anyone would ever want to do what your doing, too.) A VM can run in 16-bit mode, 32-bit mode, and 64-bit mode. An emulator can, too. Do all your tricks and hacks and other "learning" in that sandboxed environment. So, explain (again) how a VM or emulator can't do what you want to do, and how physical hardware can, somehow, do what a VM can't?
I would understand if your testing your OS on physical hardware to fix bugs and to make workarounds for buggy hardware, but this does not seem like it at all.


Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Sat Dec 07, 2019 11:49 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1224
I can perfectly do the following with this library version (moving big physically fragmented blocks that don't fit for expansion and shrinking is left to do):

Code:
tmprealloc=malloc(655360);
realloc(tmprealloc,1310720);
free(tmprealloc);



https://sourceforge.net/projects/lowest-kernel/files/

I have to test allocating/reallocating up and down/freeing blocks that span several page directory entries, like 40MB blocks.

_________________
http://www.ebay.com/usr/udocproject3 (Updated IP)
http://190.53.102.175/api/doc/en/ (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe


Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Dec 24, 2019 11:10 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1224
What should happen if I allocate, say, a 32768-byte buffer ad get a ptr value (ptr=malloc(32768)) and then I do a free with a pointer N bytes later (ptr+=N; free(N))?

_________________
http://www.ebay.com/usr/udocproject3 (Updated IP)
http://190.53.102.175/api/doc/en/ (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe


Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Dec 24, 2019 12:36 pm 
Offline

Joined: Fri Oct 04, 2019 10:10 am
Posts: 18
if ptr+N happens to end up as another pointer you got from malloc(), it will probably work. If it is anything else, it will probably fail horribly.

The math to do that is technically undefined, given the rules about what you can point to, and the limits of pointer arithmetic, but those don't really apply to actually implementing malloc.

Most malloc implementations store the book keeping and management information in-line with the memory blocks, just before and after the block itself, and use the pointer passed in to find it. If you pass a random pointer in, it will look for that data just before the value you pass in, find who knows what, and then promptly corrupt your heap.

Some paranoid ones could protect from that with some canary values, sanity checks, or explicitly walking the allocation list to find the block. Those are rare in practice, as that can make free() rather slow, or increase the allocation overhead unreasonably.

There are some allocators that keep their bookkeeping data out of line, but they tend to not use quite the same interface as malloc/free, other than, say, page allocators.


Top
 Profile  
 
 Post subject: Re: x86 32-bit paging management
PostPosted: Tue Dec 24, 2019 3:33 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 547
reapersms wrote:
Some paranoid ones could protect from that with some canary values, sanity checks, or explicitly walking the allocation list to find the block. Those are rare in practice, as that can make free() rather slow, or increase the allocation overhead unreasonably.
Well, musl's malloc isn't that slow, and it keeps a tiny bit of safety information: The size of the chunk is saved in a header as well as a footer, and if they don't match in free(), then musl will deliberately crash. This basically just fell out of the design of having all memory chunks in the same block as part of a doubly linked list.

_________________
Life is beautiful. Nobody said easy.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 25 posts ]  Go to page Previous  1, 2

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot] and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group