OSDev.org

The Place to Start for Operating System Developers
It is currently Wed Mar 03, 2021 9:18 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 85 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Thu Feb 11, 2021 11:00 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
Yeah. Maybe these might shed some light on the topic, both sentences from Wikipedia:
Quote:
The POSIX specifications for Unix-like operating systems originally consisted of a single document for the core programming interface
Quote:
Before 1997, POSIX comprised several standards: POSIX.1: Core Services (incorporates Standard ANSI C) (IEEE Std 1003.1-1988)


Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Tue Feb 16, 2021 7:09 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
Hi All,

I've added UTF-8 support. This means no more fiddling with wchar_t and L"" literals any more:
Code:
#include <uefi.h>

int main (int argc, char **argv)
{
  printf("Hello, world!\n");
  return 0;
}
Just as with any other POSIX systems (except for the includes)! You can still get the old behaviour (main(int, wchar_t) and printf(L"")) if you comment out USE_UTF8 in uefi.h. This will make POSIX-UEFI less POSIXish, but there will be no string conversions between your application and the UEFI interfaces, all strings will be passed as-is. To write code for both modes, char_t is defined (either char or wchar_t), and the CL() macro takes care of the L prefix for constant literals.

I've also made LLVM the default toolchain, from now on you can switch to gcc by using USE_GCC in the Makefile.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Tue Feb 16, 2021 9:19 am 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 855
bzt wrote:
I've added UTF-8 support.
Cool. As a proponent of "UTF-8 everywhere", I think that is a positive development.

_________________
Thou hast outraged, not insulted me, sir; but for that I ask thee not to beware of Starbuck; thou wouldst but laugh; but let Ahab beware of Ahab; beware of thyself, old man.


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Tue Feb 16, 2021 9:25 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 824
Agreed, UTF-16 is quite tedious to work with.

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Tue Feb 16, 2021 10:58 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
Thanks! I'm an UTF-8 fan too. Plus this allowed me to use "char" in the API, so now POSIX-UEFI really looks like the usual ANSI C libc interface. Except getenv and setenv, I could not help those because UEFI environment variables are binary blobs.

The other difference I'm thinking what to do with is fread. If you open a standard file with fopen, then you can use fseek and fread as usual. However if you open "/dev/diskX", then (because UEFI doesn't manage offsets for Block IO internally), fseek is useless and fread looks like "fread(buffer, size, lba, stream)", so instead of the number of blocks argument you pass the required lba address. I'm not sure what to do with this, as it is simple enough, and I'm not sure it worth implementing fseek for Block IO protocol. What is your opinion? BTW, I've added typedef structs for the GPT too.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Tue Feb 16, 2021 12:46 pm 
Offline
Member
Member
User avatar

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 639
bzt wrote:
I've added UTF-8 support.

That was my biggest bone with UEFI was that using a common codebase for legacy and EFI was tedious, as I had to have Unicode and ANSI functions separate throughout. I created some ugly macros that helped somewhat, but that's a great improvement, just using UTF-8 from the start!

_________________
Currently working on the Nexware project, an attempt to make a less bloated version of GNU. All repos for it can be found at https://github.com/Nexware-Project.


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Wed Feb 17, 2021 10:08 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
Okay, I've added more examples:
  • the mandatory Hello World of course
  • getting the command line arguments
  • dumping memory (feature of POSIX-UEFI's printf %D)
  • listing directories (using opendir / readdir / closedir)
  • Reading files (with fopen / fread / fclose)
  • Dumping sectors from block devices (with fopen / fread / fclose), oh I've implemented the offsets for block devices btw
  • Using serial port (with fopen / fwrite / fprintf / fread / fclose)
  • Using GOP to query and set video modes (using UEFI API)
  • Printing UTF-8 string with bitmap as well as vector fonts (with and without the SSFN library)

There are screenshots too, for example
Image

I had to add that last UTF-8 printing part because I've tried to do
Code:
ST->ConOut->OutputString(L"Здравствуйте\r\n");
and it doesn't work with TianoCore! WTF? Why the f*ck does UEFI mandate wchar_t then if it can't support non-Latin alphabets at all? Yet another indicator of M$ / Intel complete incompetence. Let's waste double space for every string without providing actual support for UNICODE BMP! Yeah, what could possibly go wrong? UEFI Consortium don't pay for it, so let's just waste the user's money, storage, RAM and network bandwidth! Grrr. Anyway, using my own fonts I was able to display UTF-8 text properly, so that's the preferred (and only) way of printing multi-lingual strings with POSIX-UEFI.

Image

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Wed Feb 17, 2021 3:15 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 2815
bzt wrote:
I had to add that last UTF-8 printing part because I've tried to do
Code:
ST->ConOut->OutputString(L"Здравствуйте\r\n");
and it doesn't work with TianoCore! WTF? Why the f*ck does UEFI mandate wchar_t then if it can't support non-Latin alphabets at all?

It does support non-Latin alphabets, you just need to include your own font and register it with the HII database.


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Wed Feb 17, 2021 4:27 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
Octocontrabass wrote:
It does support non-Latin alphabets, you just need to include your own font and register it with the HII database.
Only in theory. UEFI Driver Writer's Guide says about HII fonts
Quote:
The system carries the Latin-1 (Western European) character set. Other characters must be provided if they are to be displayed.
The same document also classifies "HII Font" a rarely used protocol, and does not care about providing a detailed description or an example on how actually do that, eg. how to "provide the characters to be displayed". Plus including the glyphs for the entire CJK area in each and every driver which needs Chinese translation as UEFI blogspot suggests is ridiculous.
Quote:
Let's say I'm designing a plug-in card and I want to support Chinese. But I don't know whether or not the system firmware in the platform where my card is installed supports all the Chinese characters (glyphs) that I need. What can I do? Well, with UEFI 2.10, your driver can just carry the characters you need that are in addition to those provided by the ISO Latin-1 characters that the platform is required to carry in its firmware.
That's the way to do it! Yeah, let's duplicate 65536 - 256 glyphs in each and every driver! The firmware isn't bloated enough already! (sarcasm)


That's about theory. Now in practice, taking a deep dive in TianoCore's source, GetGlyphBuffer only supports SimpleFont with EFI_NARROW_GLYPH and hardcoded EFI_GLYPH_WIDTH. Forget that you set the font width from the font. Again, in theory GetGlyphBuffer consults GlobalFont->FontPackage, but there's no way to register sane fonts with it, as GraphicsConsole.c itself has hardcoded EFI_NARROW_GLYPH, and "This->Mode->CursorColumn * EFI_GLYPH_WIDTH" is all over the place. In RegisterFontPackage at line 2064 it only sets up package size for narrow fonts, does not check its input (it is a notification callback) which makes you wonder what a nice buffer overflow could happen if you actually try to register a wide font....

So the statement that UEFI supports proportional fonts is just yet another big lie. It doesn't. Even if you figure out how to register HII Font Packages, you cannot display glyphs with different sizes using OutputString, that's for sure.

Now why do both Font.c and GraphicsConsole.c define exactly the same palette over and over again but separately is beyond my comprehention (specially when they should not need it as colours are taken care of in StringToImage() in Image.c). When I first tried to understand TianoCore's source, I was just confused. Then I did so much FacePalms (tm), that my head became red :-) Now I just laugh when I think about how hilariously overengineered and completely useless garbage UEFi is! And the more time I spent with it, the more convinced I became :-D If someone says that UEFI is a progress over BIOS, then a) they are paid to say that b) they are simply fools or idiots.

Do you want to hear another epic fail? UEFI Firmware must keep track of allocated memory pool sizes, for BS->FreePool only receives a pointer. But so must YOUR APPLICATION TOO, if you ever want to reallocate a pool! Even though AllocatePool receives a memory pointer, it uses that for output only, so no reallocation possible, you must do that yourself in your application with memcpy! Which means you must keep track of the allocated memory pointers and sizes. You simply cannot not wonder, what a total idiot designed this interface?

Anyway, POSIX-UEFI hides all this sh*t, so that you can simply use stb_image.h to load PNG images for example:
Image

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Sat Feb 20, 2021 10:20 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Now the examples are complete, they cover all the topics one might need for writing their own boot loader: loading files or sectors, printing UTF-8 strings, sending formatted messages to serial for debugging, getting memory map, setting video modes, displaying PNG images, loading and executing ELFs, passing boot parameters to it, exit Boot Services etc. Everything is self-contained and dependency-free, and I've tried to write all code to be small and easy to follow. Longest being about 130 SLoC.

https://gitlab.com/bztsrc/posix-uefi/-/tree/master/examples

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Sat Feb 20, 2021 11:19 am 
Online
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 155
bzt wrote:
Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Now the examples are complete, they cover all the topics one might need for writing their own boot loader: loading files or sectors, printing UTF-8 strings, sending formatted messages to serial for debugging, getting memory map, setting video modes, displaying PNG images, loading and executing ELFs, passing boot parameters to it, exit Boot Services etc. Everything is self-contained and dependency-free, and I've tried to write all code to be small and easy to follow. Longest being about 130 SLoC.

https://gitlab.com/bztsrc/posix-uefi/-/tree/master/examples

Cheers,
bzt



Bravo, good work!


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Sat Feb 20, 2021 6:08 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 2815
bzt wrote:
Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Do you have any more details about the bug or your workaround? I'd like to know more.


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Sun Feb 21, 2021 12:32 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
Octocontrabass wrote:
bzt wrote:
Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Do you have any more details about the bug or your workaround? I'd like to know more.
Sure.

TL;DR GCC incorrectly inlined memcmp() under some circumstances, which resulted in an infinite loop.

I noticed it in fopen line 215. This line
Code:
if(!memcmp(__filename, CL("/dev/serial"), 11 * sizeof(char_t))) {
  printf("match\n");
  ...
}
printf("no match\n");
froze in an infinite loop with the memcmp call, and nothing was printed. The first argument is checked not to be null, the second argument is a constant string literal (CL() macro does nothing), and the third is again a constant, not zero. There is no UB here, and memcmp() shouldn't freeze for sure. Taking a look at objdump -d stdio.o and no wonder that the code froze, the inlining was completely wrong. Interestingly it worked from the user app (where it was compiled as a call to memcmp@PLT), but not in the other object files of the library, like stdio.o or stdlib.o. It also worked with CLang as it should. I've used a naming trick to prefix these problematic functions with the C-precompiler, so that GCC won't try to replace my memcmp with its own buggy one, and problem solved. Now memcmp is called "memcmp" with Clang, and "__uefi_memcmp" with GCC, in which case a define makes it possible that user app can use "memcmp" as usual.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Sun Feb 21, 2021 1:16 pm 
Offline
Member
Member

Joined: Mon Feb 02, 2015 7:11 pm
Posts: 572
The optimizer simply detected that it could optimize your code by calling... memcpy(), resulting in infinite recursion. The same thing can happen with memset().

You can work around this by disabling the specific optimization causing the problem. Here is what I did for my memset():
Code:
#pragma GCC optimize "no-tree-loop-distribute-patterns"

I have also run into a very similar problem where GCC optimizes malloc() + memset() into calloc(). The problem was that I was providing my own implementation of calloc() that was basically doing malloc() + memset(), resulting in another infinite loop. This was was fixed by:
Code:
#pragma GCC optimize "no-optimize-strlen"

_________________
https://github.com/kiznit/rainbow-os


Top
 Profile  
 
 Post subject: Re: Announcing POSIX-UEFI
PostPosted: Sun Feb 21, 2021 1:31 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1303
kzinti wrote:
The optimizer simply detected that it could optimize your code by calling... memcpy()
Sadly no, that would make some sense at least. There was no call to "memcpy" at all, instead just an incorrect test eax, eax loop.

kzinti wrote:
You can work around this by disabling the specific optimization causing the problem.
Which weren't enabled in the first place :-) Plus I have no control over what optimizations the user wants to use. Using the __uefi_ prefix works, as GCC doesn't think it's memcmp any more :-)

kzinti wrote:
The problem was that I was providing my own implementation of calloc() that was basically doing malloc() + memset(), resulting in another infinite loop. This was was fixed by:
Code:
#pragma GCC optimize "no-optimize-strlen"
Eeeeeek.... What has strlen to do with that? :-D But yeah, I know this bug, I've also run into that. And another, if you have a struct at address 0, then accessing any of its fields (at non-zero address) would generate an UD2 instruction instead of a MOV, how lovely, isn't it?

Cheers,
bzt


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 85 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group