OSDev.org

The Place to Start for Operating System Developers
It is currently Fri Mar 29, 2024 8:14 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 24 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Mon Apr 29, 2019 3:52 pm 
Offline
Member
Member
User avatar

Joined: Fri Feb 17, 2017 4:01 pm
Posts: 641
Location: Ukraine, Bachmut
Solar, kewl. I definitely consider your C library as CRT for my project when it will be something more than just babbling, untested pieces of mess, unimplemented ideas and jerky loader. :mrgreen: The words "compliance" and "minimalism" are so pleasant to the ears.

_________________
ANT - NT-like OS for x64 and arm64.
efify - UEFI for a couple of boards (mips and arm). suspended due to lost of all the target park boards (russians destroyed our town).


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Tue Apr 30, 2019 9:57 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 999
Time for another update on managarm: a few weeks ago, I implemented write support for ext2fs. With all the page cache and writeback infrastructure in place (which I wrote about in my previous post), this was surprisingly simple. Basically, to create new files, the driver just needs to map block allocation bitmaps, inode tables and directory structures into virtual memory. After that is done, it modifies them in place and triggers a writeback. Modifying file data is even more simple: here, the driver just maps the file and does a memcpy() to modify the mapped pages. Again, a writeback is triggered subsequently. What I find particularly appealing is that no direct writeSectors() call is required in any of the write operations -- those are done transparently by the page cache.

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Fri May 03, 2019 8:04 am 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 812
Location: Hyperspace
Solar wrote:
[*] Integration of dlmalloc; PDCLib now has decent memory management.

Thanks for the link, I've filed it. I may not be working with C, but I will need decent memory management. This article contains some useful guidance.

_________________
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Fri May 03, 2019 9:11 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7612
Location: Germany
eekee wrote:
I may not be working with C, but I will need decent memory management. This article contains some useful guidance.


From an OS designer's perspective, it's an easy trap to fall in: Lumping everything together under the label "memory management".

I found that it's "better" to, in your mind, realize that it's two separate things, really.

On the one hand you have the kernel doing the MMU / paging / address space setup thing. It also needs to provide some way(s) to pass out memory to user space. This is not malloc() / free(), though. It's more in the terms of brk() / sbrk() / mmap() / whatever_your_idea_is(), i.e. the system calls that a malloc() implementation like dlmalloc, or ptmalloc, or Your_Languages_Interpreter interfaces with.

On the other hand you have the user-space support library. malloc() / calloc() / realloc() / free() / etc. for standard C; this gets memory from the kernel in bulk (brk() / sbrk() / mmap() / ...) and does the smart parceling out / reclaiming.

There might be some need for a kernel function that passes out sub-page sized memory bits to the kernel proper, like kmalloc(). (Which might or might not use similar logic to user-space malloc().) But for architecture, I feel it's important to keep the system calls (brk() / sbrk() / mmap() / ...) and the user lib calls (malloc() / ... / free()) separate in your head.

Just sayin'. It took me some confused hours back in the day to figure that one out on my own. ;-)

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Fri May 03, 2019 9:31 am 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 812
Location: Hyperspace
Thanks! :D I was headed for confusion, I think. I'm aware they're normally separate, but with so many architectural decisions still up in the air, I risk conflating different systems all the time.

_________________
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Mon May 27, 2019 6:13 pm 
Offline
Member
Member
User avatar

Joined: Sat Nov 22, 2014 6:33 pm
Posts: 934
Location: USA
Hi everyone,

It has been a while. I have been busy with other things, mainly a new job, but I still get back to this hobby.

After a lot of comments and suggestions, I have started a re-write of my project so that it will boot on a 64-bit UEFI machine.

After a bit of work, of course having to write all the new 64-bit hardware specific tasks, I have got a working project.

Comments:
1) Even though you try your best not to assume a 32-bit machine, it is amazing how many places my code assumed pointers were 32-bits in size.
2) I (re)wrote the interrupt code to be 64-bit, writing a more efficient and easier to use method.
3) I also had a few bugs that I would never have found unless I did a re-write such as this.
4) Many more improvements and additions.
5) With the improvements, came the improved code for the ACPI, IOAPIC, and the HPET. I am still having issues on some machines with the xHCI, but am working on that as well.
6) I (hopefully) have written the new changes so that a couple of #defines will specify the target and the code should compile for a 32-bit Legacy, 32-bit UEFI, or 64-bit UEFI, simply by changing a few #defines.

I still have some work and testing, but it looks like my code will now work on a 64-bit machine just fine, in 64-bit native mode.

Anyway, just an update and a comment to say I am still here and still interested in this hobby of ours. For those interested, yes, I will be updating my books to show the new content and documentation needed to write a 64-bit kernel.

Hope all is well with you and yours,
Ben
- http://www.fysnet.net/osdesign_book_series.htm


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Mon Jul 22, 2019 6:58 am 
Offline

Joined: Wed Jul 17, 2019 2:27 pm
Posts: 18
Solar wrote:
I know there is ptmalloc, and yet other allocators. I will probably look into making them available as options. For now dlmalloc was a) code I had looked at before, b) code that had been "touched" by the maintainer more recently than alternatives I browsed through at a glance, c) code that's also CC0 (hence does not require additional licensing legalese), and d) so much of an improvement over what was previously in PDCLib that merits relative to other available allocator implementations didn't really matter. :-D


I was looking at memory allocator code recently myself. I was considering nedmalloc which is based on dlmalloc since it's fast and works well with threading. I also ran across rpmalloc ( https://github.com/rampantpixels/rpmalloc/releases ) which looked very promising and is also public domain but required too much C11 specific functionality (atomics support) which made it hard to port to platforms that didn't provide that yet. I did end up using dlmalloc in my own project for now because it is public domain and fairly easy to integrate. If you run across other public domain options that are better than dlmalloc, would be interested to hear about them.

Found it interesting that you implemented C11 as a wrapper to pthreads. Most projects seem to do that. I've done it the other way around with my own project. I personally like the C11 threading API much better than the POSIX threads API.


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Wed May 20, 2020 6:38 pm 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 402
Solar wrote:
pat wrote:
Have you recently added a new feature to your OS, or accomplished some milestone?


Over the course of two weeks spent at a small vacation home at the North Sea coast, I spent quite some hours working on PDCLib, my C standard library implementation aimed specifically at compliance, minimalism (i.e. no "extensions"), and adaptability to your OS of choice. The project, for those who don't know, is what's left of my own OS project from the early 2000's, and explicitly geared toward the OSDev crowd (although I learned it has been adopted in a couple other places as well, including an inofficial XBox SDK, which I think is awesome).


Thanks for PDCLib. I didn't fancy writing a libc for my userspace, and found other libc too difficult to bootstrap. This was some time ago, though, and I'm learning all the time, so I may fair better with alternative libc now, but for the time being PDCLib is my go to libc at least for anything in my initial ram disk.

Solar wrote:
...
This adds some requirements on the hosting OS (dlmalloc makes use of a variety of system calls), but I feel it's a massive improvement over my my own stopgap implementation (which used only brk / sbrk, but never gave memory back to the OS until the process terminated, fragmented like crazy and was all kinds of ineffective).


I actually found brk/sbrk tricky to implement, so this was a fantastic test case and very useful. Especially as I was using Linux's system call interface as a model (so I can test binaries on both to check for correct behaviour.)


Top
 Profile  
 
 Post subject: Re: The Terrific Tell and Talk Thread
PostPosted: Wed May 27, 2020 10:54 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 999
Nice that this thread got resurrected. Here is my newest achievement:

Managarm now has a kernel-level profiler that uses performance counters to sample the currently executed function in a regular interval (e.g., every 2 million clock cycles). This is the output produced:
Code:
1.27% (10088 samples) in:
    syscallStub in stubs.S
1.43% (11363 samples) in:
    thor::disableUserAccess() in cpu.cpp
1.55% (12304 samples) in:
    thor::sendPingIpi(unsigned int) in mem_space.hpp
1.57% (12483 samples) in:
    memcpy in libc.cpp
2.04% (16215 samples) in:
    thor::enableInts() in ints.cpp
2.13% (16923 samples) in:
    frg::slab_pool<thor::KernelVirtualAlloc, thor::IrqSpinlock>::deallocate(void*, unsigned long) in slab.hpp
2.65% (21098 samples) in:
    thor::getPlatformCpuData() in cpu.cpp
3.04% (24217 samples) in:
    thor::intsAreEnabled() in ints.cpp
3.22% (25626 samples) in:
    frg::slab_pool<thor::KernelVirtualAlloc, thor::IrqSpinlock>::allocate(unsigned long) in slab.hpp
3.28% (26077 samples) in:
    memset in libc.cpp
4.16% (33059 samples) in:
    thor::LocalApicContext::_updateLocalTimer() in pic.cpp
74.02% of all samples in the kernel


It works as follows:

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 24 posts ]  Go to page Previous  1, 2

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot], Google [Bot], KN9296 and 158 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group