OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Oct 22, 2019 12:43 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 14 posts ] 
Author Message
 Post subject: Ideas for changing how Programs interact with Files
PostPosted: Wed Jul 09, 2014 3:37 am 
Offline
Member
Member

Joined: Sat Mar 15, 2014 3:49 pm
Posts: 96
Back when I was due to start as an architect at Symbian, I published some articles to put out some ideas I had in the public domain.

I think I published them on osnews and a few other places. At the time, they were unflatteringly received. Anyway, for the curious, I think they are as good as ever ideas even today:

Network Transparency and Multi-root Filesystems

Files and Programmatic Streams

Openning Streams by Mime-Type

Feel free to use and abuse these ideas. Hope they give birth to creative useful new ideas too.


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Wed Jul 09, 2014 5:25 am 
willedwards wrote:
I published some articles to put out some ideas I had in the public domain

In my view all file system related ideas lack some generalization. Actually developer need not anything like this:
Code:
File f=new File("http://www.microsoft.com");

Because the developer actually need just some part of the content at "http://www.microsoft.com". And bothering with some "library way" of accessing such content is not very pleasant. Developer needs tooling support for accessing some data and then for extraction of something useful. And this again leads us to the idea of metadata. If we have a world with pervasive metadata then we can ask our tools like this:
Code:
String data[][]=Tooling.find("microsoft.statistics.sales.byRegions");
if (data!=null && data.length>0)
  .....


Top
  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Wed Jul 09, 2014 7:09 am 
Offline
Member
Member

Joined: Sat Mar 15, 2014 3:49 pm
Posts: 96
Embryo, you describe the Wolfram Language. If you make any exciting Wolfram programs, please share.

Of course, you have to use lower level abstractions to implement the Wolfram runtime. Someone somewhere has to actually know how computers work.


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Thu Jul 10, 2014 4:59 am 
willedwards wrote:
Embryo, you describe the Wolfram Language. If you make any exciting Wolfram programs, please share.

Actually the language is Java. And the string with dotted notation is just a widely used method of declaring a path in a tree-like structure. An example of such notaion is - wiki.osdev.org.
willedwards wrote:
Of course, you have to use lower level abstractions to implement the Wolfram runtime. Someone somewhere has to actually know how computers work.

The lower part of the job is done by a disk driver (in case of a disk access, for simplicity) and a piece of software like database server. The server parses the string and looks for mapping from string keys to disk sectors, then the server asks the disk driver to read the sectors. And there is no such entity as file, because it is useless in this case.


Top
  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Thu Jul 10, 2014 10:23 am 
Offline
Member
Member

Joined: Sat Mar 15, 2014 3:49 pm
Posts: 96
embryo wrote:
Code:
String data[][]=Tooling.find("microsoft.statistics.sales.byRegions");
if (data!=null && data.length>0)
  .....


And how would you know how to parse this array of strings? The data has structure more than simply being an array of strings.

If you read my three articles linked in the original, you'll see some thought went into them.


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Fri Jul 11, 2014 4:15 am 
willedwards wrote:
And how would you know how to parse this array of strings? The data has structure more than simply being an array of strings.

How a database knows what to show when parses SQL query?
willedwards wrote:
If you read my three articles linked in the original, you'll see some thought went into them.

If you read some other articles then you'll also see some thought went into them. But if you wand to describe your opinion then the keyboard is near you.


Top
  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Fri Jul 11, 2014 2:29 pm 
Offline
Member
Member
User avatar

Joined: Wed May 21, 2008 4:33 am
Posts: 294
Location: Mars MTC +6:00
willedwards wrote:
FILE *f = fopen("ftp://www.microsoft.com/mission/statement.html","r");


I've wanted to integrate this into an OS for a while. By default all files would be opened with the 'file://' protocol and only extended protcols would need to be specified. This would be supported by drivers and could even be used with hardware like 'audio://0/' for recording from a mic or 'usb://3/' for input from a usb joystick. This would happen at the OS level so if you wanted to use a joystick library the library would detect and open the usb device automatically for you and then you just use the library functions.

_________________
"God! Not Unix" - Richard Stallman

Website: venom Dev
OS project: venom OS
Hexadecimal Editor: hexed


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Fri Jul 11, 2014 9:17 pm 
Offline
Member
Member
User avatar

Joined: Wed Dec 01, 2010 3:41 am
Posts: 1761
Location: Hong Kong
b.zaar wrote:
willedwards wrote:
FILE *f = fopen("ftp://www.microsoft.com/mission/statement.html","r");

I've wanted to integrate this into an OS for a while. By default all files would be opened with the 'file://' protocol and only extended protcols would need to be specified. This would be supported by drivers and could even be used with hardware like 'audio://0/' for recording from a mic or 'usb://3/' for input from a usb joystick. This would happen at the OS level so if you wanted to use a joystick library the library would detect and open the usb device automatically for you and then you just use the library functions.


This cause the same problem with unix as "everything is a file".
How do you handle seeking a joystick, "403 Moved" for a display, "Auth required" on the audio, or changing frequency/presentation format for a normal file?

Do they only exists on the interface and the programmer is told "certain cases is only bound to certain category of devices"?


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Fri Jul 11, 2014 10:11 pm 
Offline
Member
Member
User avatar

Joined: Wed May 21, 2008 4:33 am
Posts: 294
Location: Mars MTC +6:00
I see devices being handled like this more as objects with a simple file name for opening the device initially. Changing any settings on a device like frequency/presentation format would be through I/O control system calls. I'm not sure what problems it would create that's different to other solutions. If a usb joystick is not attached when a program is looking for it on any OS it will appear as 'device not found' in some way. The naming of the devices could be anything such as usb://joystick/0/, usb://joystick/1/ or usb://joystick/logictechAwestick, usb://joystick/MSstuck etc and then every joystick found is listed in the game for a user to select. I haven't looked at other solutions to presenting network files or devices but these problems will still exist just in a different way.

_________________
"God! Not Unix" - Richard Stallman

Website: venom Dev
OS project: venom OS
Hexadecimal Editor: hexed


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Sat Jul 12, 2014 1:01 am 
Offline
Member
Member

Joined: Sat Mar 15, 2014 3:49 pm
Posts: 96
bluemoon wrote:
This cause the same problem with unix as "everything is a file".
How do you handle seeking a joystick, "403 Moved" for a display, "Auth required" on the audio, or changing frequency/presentation format for a normal file?

Do they only exists on the interface and the programmer is told "certain cases is only bound to certain category of devices"?


The mantra "everything is a file" doesn't actually mean everything is seekable. You can't seek in pipes nor sockets, nor even all files. In the old days, languages used to make you specify if you were opening a file for "random access". Unix made it so you just had to check seek() for error.

This is described fleetingly in my three articles, but thanks for bringing it up as it needs more awareness and attention.


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Sat Jul 12, 2014 3:03 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9285
Location: On the balcony, watching the Swedish Chef
I'd personally start with making all storage accesses asynchronous by default (and ditch fopen). It pushes the developer towards making programs more responsive, easily allows multiple accesses to be performed concurrently without multithreading the application, and prevents architecturally-conceived shitstorms like linux kernel samba making it impossible to kill processes in case of an error.

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Sat Jul 12, 2014 3:53 am 
Offline
Member
Member

Joined: Sat Mar 15, 2014 3:49 pm
Posts: 96
Combuster wrote:
I'd personally start with making all storage accesses asynchronous by default (and ditch fopen).


Its an interesting orthogonal problem. I used to do a lot of async io, and have come to prefer lightweight threads.

Blocking io for IPC is the basic building block of L4, and it would be hard to imagine those kind of performance advantages being transferable to async io (for what I called "programmatic streams" in my articles).

On the new Mill CPU, you can actually move between peer protection domains (think process to process) without changing thread. This will lead hopefully to some interesting innovation at the OS level.

Quote:
prevents architecturally-conceived shitstorms like linux kernel samba making it impossible to kill processes in case of an error.


I'm curious about the details of why this is? I recall hearing it before, but googling makes me no wiser.


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Sat Jul 12, 2014 4:20 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9285
Location: On the balcony, watching the Swedish Chef
Try having a stable application when the kernel runs through several minutes of network recovery processes

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
 Post subject: Re: Ideas for changing how Programs interact with Files
PostPosted: Sat Jul 12, 2014 4:31 am 
Offline
Member
Member

Joined: Sat Mar 15, 2014 3:49 pm
Posts: 96
Well I've worked on kernels where disk-wait threads can be aborted, and this is orthogonal to blocking/non-blockingness of the IO calls.

At first glance I can't see why Linux doesn't support it. There may sply be the rationalization that it would affect too many old drivers to change it now, or there may be depths I am unfamiliar with.

If you make a kernel, please make IO tasks orphanable in their own right.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group