OSDev.org

Hey everybody. So, today i decided to create an os, called OrbOS. I have just started it, and it won't use the grub bootloader, because i want to design how the bootloader works, and the kernel is just a stub pe kernel, but the bootloader isn't finished yet, because i am still trying to figure out how to load pe files and run them, and I need a lot of help with that, please give me some help for that below. For apps i am planning on using my own set of system calls, but still in the pe format(non-windows compatible). I am going to start by putting the shell in the kernel, and later i might make it an app, and add a config file. Thanks for reading, and bye.

edit 1: So, i am also needing help on a vesa driver. Also, i want to know how to program the pic for multitasking.

While this seems a bit... troll-y, this is probably just my paranoia talking, so I will for the moment assume you are sincere. Therefore, let me give you my the latest version of my "standard advice to beginners" speech.

If you haven't already, I would strongly advise you to read the introductory material in the wiki:

• Getting Started
• How To Ask Questions
• FAQ
• Required Knowledge
• Beginner Mistakes

After this, go through the material on the practical aspects of
running an OS-dev project:

• What order should I make things in
• Code Management
• How kernel, compiler, and C library work together
• Using Programming Languages other than C
• Assembly Language

I strongly suggest that you read through these pages in detail, along with the appropriate ones to follow, before doing any actual development. These pages should ensure that you have at least the basic groundwork for learning about OS dev covered.

This brings you to your first big decision: which platform, or platforms, to target. Commonly options include:

• x86 - the CPU architecture of the stock PC desktops and laptops, and the system which remains the 'default' for OS dev on this group. However, it is notoriously quirky, especially regarding Memory Segmentation, and the sharp divisions between 16-bit Real Mode, 16-bit and 32-bit Protected Modes, and 64-bit Long Mode.
• ARM - a RISC architecture widely used on mobile devices and for 'Internet of Things' and 'Maker' equipment, including the popular Raspberry Pi and Beagleboard single board computers. While it is generally seen as easier to work with that x86, most notably in the much less severe differences in between the 32-bit and 64-bit modes and the lack of memory segmentation, the wiki and other resources don't cover it nearly as well (though this is changing over time as it becomes more commonly targeted).
• MIPS, another RISC design which is slightly older than ARM. It is one of the first RISC design to come out, being part of the reason the idea caught on, and is even simpler than ARM in terms of programming, though a bit tedious when it comes to assembly programming. While it was widely used in workstations and game consoles in the 1990s, it has declined significantly due to mismanagement by the owners of the design, and is mostly seen in devices such as routers. There are a handful of System on Chip single-board computers that use it, such as the Creator Board and the Onion Omega2, and manufacturers in both China and Russia have licensed the ISA with the idea of breaking their dependence on Intel. Finding good information on the instruction set is easy, as it is widely used in courses on assembly language and computer architecture and there are several emulators that run MIPS code, but finding usable information on the actual hardware systems using it is often difficult at best.

You then need to decide which language to use for the kernel. I gather you are using C, which is the usual recommendation. However, you then need to choose the compiler, assembler, linker, build tool, and support utilities to use - what is called the 'toolchain' for your OS.

For most platforms, there aren't many to choose from, and the obvious choice would be GCC and the Binutils toolchain due to their ubiquity. However, on the Intel x86 platform, it isn't as simple, as there are several other toolchains which are in widespread use for it, the most notable being the Microsoft one - a very familiar one to Windows programmers, but one which presents problems in OSDev. The biggest issue with Visual Studio, and with proprietary toolchains in general, is that using it rules out the possibility of your OS being "self-hosting" - that is to say, being able to develop your OS in the OS itself, something most OSdevs do want to eventually be able to do. The fact that Porting GCC to your OS is feasible, whereas porting proprietary x86 toolchains isn't, is a big factor in the use Binutils and GCC, as it their deep connection to Linux and other Unix derivatives.

Regardless of the high-level language you use for OS dev (if any), you will still need to use assembly language, which means choosing an assembler. If you are using Binutils and GCC, the obvious choice would be GAS, but for x86 especially, there are other assemblers which many OSdevs prefer, such as Netwide Assembler (NASM) and Flat Assembler (FASM).

The important thing here is that assembly language syntax varies more among the x86 assemblers than it does for most other platforms, with the biggest difference being that between the Intel syntax used in the majority of x86 assemblers, and the AT&T syntax used in GAS. You can see an overview of the differences on the somewhat misnamed wiki page Opcode syntax.

It is still important to understand that the various Intel syntax assemblers - NASM, FASM, and YASM among others - have differences in how they handle indexing, in the directives they use, and in their support for features such as macros and defining data structures. While most of these follow the general syntax of Microsoft Assembler (MASM), they all diverge from it in various ways.

Once you know which platform you are targeting, and the toolchain you want to use, you need to understand them. You should read up on the core technologies for the platform. Assuming that you are targeting the PC architecture, this would include:

• Real Mode, especially the section on memory addressing, and Segmentation
• Protected mode
• Long Mode
• Learning 80x86 Assembly, especially real mode assembly
• Memory Map, Detecting Memory and A20 Line
• Interrupts
• BIOS, and Boot Sequence

This leads to the next big decision: which Bootloader to use. There are a number of different standard bootloaders for x86, with the most prominent being GRUB. We strong recommend against Rolling Your Own Bootloader, but it is an option as well.

You need to consider what kind of File System to use. Common ones used when starting out in OS dev include:

• FAT
• SFS
• Ext2fs

We generally don't recommend designing your own, but as with boot loaders, it is a possibility as well.

While this is a lot of reading, it simply reflects the due diligence that any OS-devver needs to go through in order to get anywhere. OS development, even as a simple project, is not amenable to the Stack Overflow cut-and-paste model of software development; you really need to understand a fair amount of the concepts and principles before writing any code, and the examples given in tutorials and forum posts generally are exactly that. Copying an existing code snippet without at least a basic idea of what it is doing simply won't do. While learning itself is an iterative process - you learn one thing, try it out, see what worked and what didn't, read some more, etc. - in this case a basic foundation is needed at the start. Without a solid understanding of at least some of the core ideas before starting, you simply can't get very far in OS dev.

Hopefully, this won't scare you off; it isn't nearly as bad as it sounds. It just takes a lot of patience and a bit of effort, a little at a time.

Sorry if it sounds a little bit trolly but i must have put a little bit too-much in my todo list for orb os. Here is my official list of project goals:

Fit on one floppy disk
Fat12/16/32 R/W Support
Multi-drive support(not /dev/xxx i want it to be win-like(A:, B:, C;, etc))
I want it to run on the x86 arch(16-bit real mode)
Fancy boot screen(maybe)
Written in asm

Thank you for reading, and Bye!

In my opinion, your aims 4 and 6 are classic beginner's mistakes that limit the ambition and interest of your OS (and make your task unnecessarily difficult). Honestly, the answer to your questions are easily found with a little research and I don't see much point in spoonfeeding you with them.

Just my opinion, and intended as constructive criticism, so forgive me if I sound rude.

Sorry, but i prefer ASM because it is easier for operating system development(just in my opinion), and 16-bit real mode is easiest for me.

There are quite a few posts on this forum and others about someone incarnating a new OS and that's exactly what forums are all about, to either gain insight or share experiences. When posing a questions such as yours, one of the things that would really be helpful is delineating WHY?
I'm all about using ASM and building from the basement so to speak, but if these were of interest to me too, I'd be asking myself, why not just use DOS or Linux.

Let's digest one question at a time

1: Why 16 bit and why do you think it's easier than protected or long modes?

http://sourceforge.net/p/x86winapiasmpedllskeleton32/
I have implemented PE EXEs and DLLs that contain the full standard WinAPI since Windows 9x as NASM skeletons that don't need linkers to build. I've made a few Win32 applications in pure assembly in this way.

I don't know how to boot a PE EXE, but I've pointed you to the easy to use PE skeletons. It seems that the only starting difference between a PE EXE and a PE DLL is that the DLL contains an export table and an entry point like DllMain that might not contain anything, the EXE only contains an import table normally, and the objective of the DLL is being used by other programs as a module.

_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
Could you share what you know in your own clear words/code/pseudocode about how to boot a PE EXE? Obviously you would first need to initialize Protected Mode, you could probably load the PE EXE in a hacky way, make the DOS stub contain code to initialize Protected Mode, and load 32ñbit sections in a proper memory location from your boot manager, so that your DOS stub passes control to the PE .text section once it finishes initializing 32 or 64-bit mode.

What I can tell you is that doing things in that way wouldn't look clean. What I do is using DOS as my bootloader, and from there I have a set of 16-bit programs that set up 32-bit mode and then load my kernel from DOS, so I almost always can exit to DOS from my OS immediately with an "exittodos" console command.

I think what Schol-R-LEA means by troll-y, is that there are some people who come in pretending to be new, and ask vague questions that make the regulars do a ton of work to try to be helpful. the most important thing to do is read the beginner information on the wiki, once you do that you will have some things to work on, and some questions, but the questions will be more specific, so we can help you better. Also people will know that you're for real if you do some research yourself.

Anyway, feedback on these particular goals;
For now, ignore the one floppy disk rule. It will make it so difficult to make progress if you are trying to make everything as small as possible. Make things work, then make them smaller.
R/W FAT*, this is fine. These formats are limited, but they are very easy to learn.
Linux also supports multiple "drives" but they mount them all under one root. There is nothing wrong with the DOS model of alphabetical drive names though. The VFS wiki page will help you to understand how this all fits together.
People will attempt to discourage you from using real mode, though it makes progress faster early on, it severely limits the potential of your OS in the long run. A real mode OS will still take a lot of time to make, be sure you know what you'll be missing out on, so it isn't a rude shock later.
Fancy boots screens are great
Like real mode, ASM seems like a good idea to a lot of people early on. It is easier to set up and get coding when you don't need a cross compiler, or linker scripts, and you don't need to sort out the C runtime stuff. But C is easier, it just is. It will be orders of magnitude easier and faster to develop in C. Additionally, the output binaries will almost certainly run faster and be smaller, because the C compiler is much better at generating machine code than 99.9% of programmers would be. You can use ASM if you want, but realize you are making your task harder, with no objective benefit.

With regards to your question, how far have you come with your bootloader? You say you just started and you're trying to load a PE and run it, that's a big task, break it down into little tasks which you can test as you go. For example: 1. boot into your code 2. print to the screen 3. find a file on the disk 4. load the file into ram 5. parse the PE format in ram 6. map the PE sections into ram 7. run your PE.

Here is my operating system, consisting in a set of programs that work as bootloader from DOS, and my kernel:
http://sourceforge.net/projects/lowest-kernel/files/
http://forum.osdev.org/viewtopic.php?t=32121

As you can see, it's a more gradual process than trying to set up protected mode from the DOS stub of the PE, and then running the PE .text section.

All I need to learn now is knowing how to boot a PE EXE, but I don't need it at least as far as I can see in time because it really is enough to use a raw binary kernel image and COM programs to set up the system.

Hi and Welcome!

I just want to pick up on a point:



From your OP, you are obviously quite new to OS Development. By what measure do you then determine ASM to be the easiest choice? I'm not going to make language recommendations due to the possible ensuing flamefest, but would advise that when you get to the kernel you may like to see if the initial cost of setting up a runtime is outweighed by the speed of development with a slightly higher level language.

Also, if others manage to persuade you that real mode is not the only option, a different language choice may help portability-wise.

Cheers,
Adam

[EDIT: PS - if this is an experiment for your own learning, destined to only ever run on emulators, I can see the case for RM. If you ever want to run on real hardware, please make use of the features that the other operating modes have to offer!]

I think that people that begin to learn genuinely cannot express effectively why they feel that assembly is the easiest language, but I've stayed in that environment since I started, so I know why he says that.

We think and know that if we have 100% of implementable information about the assembly for every CPU architecture, if we have 100% of the information for programming all of the hardware devices in a given machine, if we have 100% of information on algorithms, file formats and implementation algorithms for all existing programming tasks, then assembly is effectively easier because we only need to write code that looks clean, well commented, portable assembly. If we have the information on how to build executable file formats and include information about external system/library functions, then we don't even need a linker to include the things necessary to import functions and the like to our program. We can do with raw binaries and we can build our own executables by hand reliably if we are presented with 100% of the information we will ever need, just like happens with the people that programs in Visual Basic 6, PHP and JavaScript/HTML5. Programming in assembly makes it too easy to define our variables EXACTLY as we like, without any padding or unexpected elements at all. If we want to optimize code we can design our own opcode mnemonic names for optimizing sequences of instructions, and we can use an infinite number of tricks besides that.

On the other hand, programming with C upwards is too tricky, you cannot create raw binaries with ease, there aren't enough pragmas or preprocessor directives to make the language simpler, more portable, more capable; all existing command line compiler flags from all compilers should actually be preprocessor directives and pragmas to strengthen the language, to make it self-contained and easier to implement and use. You cannot easily and directly inspect or ensure that your C code will look in a specific way when compiled, but you should be able to tell easily. You will run in so many differences among compilers for the same C or C++ language that the best choice is to use only the standard features of the language present in all versions of the language to ensure that you will be able to have modern applications even in the oldest (DOS, Win9x, BeOS, Win16, OS/2...) and the simplest hobby OSes. It's also preferable to rewrite all functions on our own as long as they don't rely on critical unavoidable features of the OS that we need to use, instead of using different C libraries that might prevent us from compiling our program in any environment, in which case we will only need to modify/portabilize a very tiny set of low level functions and preprocessor code instead of making our program inherently dependent to a language and an OS. That's why I'm trying to make my own public domain C compiler and later a C++ one, the intention is to have a compiler that is extremely easy to compile and run in DOS, Windows, Linux, BSD, UNIX and my own OS, in all of their versions, and make it portable to all compiler/language extensions. I can add pragmas instead of command line switches and only use them in C files with special directives to build a skeleton for my compiler but not for other compilers. I like to make things based in raw skeletons, so I could try to provide for example a PE EXE skeleton written in C so that the compiler only outputs raw binary but becomes capable of manually outputting any executable format by using the skeleton of a given file format. I can make that my compiler generates data without padding by default. That's the kind of things that we would want to have to concentrate in making our OS and good programs. We wish that C and C++, specially their compilers and standard libraries, were as easy to use and portable as JavaScript/PHP/HTML as programming environment. We wish that compilers were as coherent and immediately usable, self-containing all APIs to be used without needing SDKs, like JavaScript.

We might as well write a good OS in clean portable assembly in the base for each CPU architecture, and isolate all languages we will use modularly, in C, C++, assembly, JavaScript, HTML/HTML4/HTML5 and other libraries and language environments, but glue everything, and write our main OS code in the language we feel is the easiest for us, which could be assembly if we manage to provide compatibility for other languages through standard libraries and compilers.

That's how we think when we say that assembly is easier, provided that we are given source cod eand information developed sweetly towards 100% completion of 100% of things that are known to programming to learn how to implement them right away in the immediate term. Not copy/paste but a minimal functional ground to reimplement on our own as we please. It's not ridiculous, that was the attitude when people used Visual Basic 6 and Geocities, and achieved in a great part with things like Programmer's Heaven source files, OSDever.net, OSDev wiki in the 90's and 2000's, and some open source programs (the biggest and most used projects like Linux need completion and more concentrated attention per year to study them and making them easier to understand).

Portable assembly is when you can use wideword typed variables and instructions to automatically select the register width with the very same code, which is a feature of modern CPUs that assembly language surprisingly doesn't use but that can be easily added with %ixdefines and macros.

Hi,

Another larely incoherent wall of text and some quasi-religious points that I haven't seen since LoseThos. There seems to be an obsession with mentioning markup and scripting languages in the same breath as compiled languages, and also with doing things over the course of a year.


I wouldn't disagree that assembly is easy. It is. Whether maintaining assembly for a large project is easy is another matter.


??


I suppose that depends how familiar you are with your chosen language...

...and toolchain.

I'm not going to spend my lunchtime taking apart the rest of your post.

There are good reasons for using assembly. There are good reasons for using C et. al. There are good reasons for using HTML5 and JavaScript. I'm not arguing against any language per say. I'm just asking the OP to look at what they can expect to achieve with real mode assembly, look at where they want their OS to be in 5 years and check whether the two are compatible.

Cheers,
Adam

Sorry ~, nothing personal but that post is all rubbish. I'm not going to tell anyone to use or not use ASM but don't make out like there is some special purity to it. Also, stop hanging onto the myth that it's more performant, or creates binaries with smaller footprints than a higher level language. It simply isn't true these days and it's misleading for new programmers to hear it repeated like gospel.

It's not that assembly is easier, it's a prerequisite to learn the C/C++ languages if you are going to deal with heavy machine handling or are to make any serious programming with them. You will really get lost fast with pointers and run-time problems that don't have to do with syntax if you can't recall why those things happen in assembly to correct with an assembler mentality.

Currently it's suitable for making raw binaries and raw executables with pure assembly code, and making programs as good as you can with no additional libraries. Think that you can access any system or C/C++ libraries from it for the OS you use.

For C you need to know many more layers until there are simpler but equally capable compilers. Try to create an executable file with section headers, imports and all in C without a linker.

You first need to learn assembly and some OS development, then try to make your own C compiler. I first made a C-like assembly language called RealC in 2007, most of what I've learned about compilers, then it's only now that I am writing my own C compiler when I'm really learning to use C and being able to see the differences in C headers and linker libraries so finely as to choose only the most stable, simplest and most portable language features and library functions that are truly cross-platform and cross-language-version.

To learn C you need to know assembly and make your own C compiler in short, and the same goes for C++. You need to know OS details and executable file formats for being able to decide what to use from the language and what is too custom, unstable or absent from some compilers to avoid it.

I agree that assembly is the easiest way to create a flat binary without using a linker. What is the point of that though? Linkers exist because decoupling the structure of source code with it's layout in memory is desirable.
Additionally I agree that you need to know some ASM to create an OS, however using it exclusivly will make it more difficult and time consuming to write an OS. I think that the OP deserves to know that.
I disagree however that coding a large and complex application in ASM, or writing your own compiler, gives any special insight into coding which will make you more able to write good C or C++ code.

Remember, the topic is orb's OS and we are giving him advice, and my advice it use C whereever possible in your OS because it will mean you can do more in less time and you may regret using only ASM down the ljne.