OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 9:26 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: unicode-org icu - Super huge icudt72l_dat!
PostPosted: Thu Oct 06, 2022 3:16 pm 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
Has anyone else dealt with porting https://github.com/unicode-org/icu? The stuff inside of icu4c/source/data builds a super huge icudt72l_dat.S, that when statically linked into my applications, causes the binaries to grow by 43MB! :evil:

It's read inside of source/common/udata.c in doLoadFromCommonData and openCommonData via the macro U_ICUDATA_ENTRY_POINT. I noticed the macro ICU_DATA_DIR_WINDOWS, but I'm going to have to dig into it a little more to see if I can move it to a file. And perhaps implement processes sharing the memory of files.

_________________
My OS is Perception.


Top
 Profile  
 
 Post subject: Re: unicode-org icu - Super huge icudt72l_dat!
PostPosted: Thu Oct 06, 2022 3:19 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Does your OS support shared libs? If so, then ICU can be a shared lib, and that will reduce the amount of memory / disk space used.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: unicode-org icu - Super huge icudt72l_dat!
PostPosted: Thu Oct 06, 2022 3:48 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5099
Run the configure script with "--with-data-packaging=archive" to move the data to a single external file.


Top
 Profile  
 
 Post subject: Re: unicode-org icu - Super huge icudt72l_dat!
PostPosted: Thu Oct 06, 2022 3:54 pm 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
Unfortunately I don't have support for shared libraries. For now, I'd like to load it from a file. I'll change the U_ICUDATA_ENTRY_POINT macro to be a pointer to a function that lazily loads the data file into memory, then returns a pointer to it.

(My microkernel does support shared memory between processes. For large read-only data files, perhaps I could make a service (maybe the VFS) that reads files into shared memory. I'll have to update my shared memory to support one-writer/multiple-readers.)

_________________
My OS is Perception.


Top
 Profile  
 
 Post subject: Re: unicode-org icu - Super huge icudt72l_dat!
PostPosted: Thu Oct 06, 2022 3:55 pm 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
Octocontrabass wrote:


Thanks!

_________________
My OS is Perception.


Top
 Profile  
 
 Post subject: Re: unicode-org icu - Super huge icudt72l_dat!
PostPosted: Fri Oct 07, 2022 6:16 pm 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
I got it to build with the archive file. (I use a custom build system so I'm unravelling what the make rules do and reimplementing it in my system. I had to link in stubdata but then set the macro to the directory where I put the generated .dat file.)

I ported icu because it's a dependency of Skia. I actually have Skia building and drawing my UI widgets now, except text. I have an issue with fontconfig not being able to parse my fonts.conf.

_________________
My OS is Perception.


Top
 Profile  
 
 Post subject: Re: unicode-org icu - Super huge icudt72l_dat!
PostPosted: Mon Oct 10, 2022 4:38 pm 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
I've linked in the stubdata and provided the .dat archive on disk. I'm logging file accesses in my vfs server and noticed it's not being accessed.

My more pressing issue right now is speeding up fontconfig.

_________________
My OS is Perception.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot] and 49 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group