OSDev.org

How do you plan your software? On the fly, or in writing, and why?


Personally i structure my software right at my IDE in header files, and imagine its use. I then branch from there (I tend to write a lot of my code to be reusable). I just can't see it on paper, the way i do in my IDE. I am not use to putting ideas on paper, but rather in code.

My question, is this good practice? Is there a "good practice" to planning software?

when i need to plan global thing, i generally use DIA, it's a simple open source diagram software, and i find this really useful if you need to make big diagram

For the API, as i already did plenty of rather low level programming, either under linux or windows, and little bit mac, and dos, and also higher level things, i have good idea already of how i want things to run, at least at high level, so i don't need that much planning.

I go a bit blind on certain things, anyway, it's quite impossible to make a complete design on paper for an OS, specially alone, but it's still useful to have some kind of global overview of how the different part are supposed to interact with each other, dependency chain, and that sort of things, even if it's partially done on the fly.

For now i kept things simple, as i'm mostly writing low level things, drivers have simple api, but i already had an idea of how i wanted to get that done.

The first thing i planned before anything was how to handle asynchronous transfer from pci or interrupt driven device to central memory without block or memory copy.It was my main objective with the design, and i organised pretty much everything for now around concept of bus manager->bus drivers->bus device driver, and how to synchronize data from interrupt driven bus device with the application layer.

The bus manager is a generic api to handle all device request, bus driver can be either pci, usb, for the moment, it's used to scan device, and handle a set of drivers for that particular bus. The device driver is the actual drivers that will be loaded by a bus driver to handle a device on this bus. The device is manipulated from the application or higher level via the bus manager api with a device type, with enumators, and functions to find all device according to various criterias.

For the low level things, i already had pletora of libs to manage whole lot of things like images, events, windows, dialogs, lists, structures, and also had my whole idea inspired from the principle of xslt for how to represent object list or hierarchy in memory.

I already also have plenty of stuff related to rendering, i already had a system of typed variable tree and hierarchy, and i found this very useful for the os. As in 3D rendering, there is often big part of the engine that need to allow for plugins, in order to program certain effect that can use their own set of variable, but that need to be managed from a global scene descriptor to store store and manipulate parameters of effects. Or even any specific class that the engine doesn't know any about, as class can also be derived from a variable list, and this variable list can be parsed and manipulated asynchronously by the engine and the effect or the other class in order to change parameters in real time from input or other animator class, without the main engine know anything about the class.

The other thing i planned is the system of memory with reference counter, and pointer reference, that i use to store any pointer allocated by the system. With a pointer to a function acting like a destructor in C, each allocated memory zone can be attributed a delete function by the allocator. As all pointer are manipulated as references, it can be easy to deal with memory leaks, even by releasing a reference to opaque type, the memory system can still know how to delete all the internal structure of the object pointed by the reference when it's freed. And it can be also easy to synchronize access to all lists and information structure in memory.

The only real things i planned are how to handle bus and bus device with asynchronous interrupt driven data transfer, and this system of memory.

Because i really wanted to have something 100% safe regarding memory handling, as i also planned to avoid to abuse memory protection mechanism, i wanted to have something where memory access can be checked at any point, without pointer ownership, where memory reference to complex type can be passed around from a point to another, without having to worry about memory allocation/deallocation, of list and object hierarchy

like this i can just add some typed property on the fly in any object without having to declare it in any header at all, so i have a very flexible way to allocate/free data and passing complex types around functions without having to declare anything in any header, or to rely on how the compiler will pack structures, and without having to worry on when and how they should be freed, or their size, or anything at all.

I manage all informations that is not an integer id with these tree things, so at least i have a good way to manage complex list, allocate memory for new properties, that can easily be ported from cpu architecture to another, as all memory access are explicitly typed with the type of the node, so i can share typed memory pointer around and change the type without having to change any header at all, and add property to it 100% dynamically, and without risking any problem of bad memory access or leak =) i can realloc any pointer on the fly without having to worry about other module that have a reference to the data being reallocated etc.

It's a thing i still took good deal of time to come up with, specially it has to be made in C, and work at the lowest level of the system, because everything else rely on it.

It is very flexible for on the fly design of data structures used in API, and i don't have to care much about dependencies etc, as if a property in the structure is not set, it will just skip it, and not crash, so i can add or remove or change structure anywhere, and at worst i will have an error message in a module if the object member is not present, so the whole api can be very dynamic without risking to have much crash or memory leaks, or having structure members that are not packed or ordered as they should be.

For now i just try to have one of each type of device, audio, video, usb, hard drive, network to be working together in multi tasking, which i'm almost getting at, to then sort of 'seal' and document the api, at least of all the base hardware things.

But in fact as most of the data structures and types are manipulated using a dynamic tree system, it's not relying that much on C types, the functions are just declared with a node pointer as parameter, and the actual information or data to be passed is all dynamic, and the type of the pointer can be figured out dynamically as well at any point, so it can reduce greatly the trouble that can come from lack of global design. It's more akin to defining structure as with a json kind of format, but in binary form to have fast search between nodes.

I generally just code when I have a good idea of how to structure things ahead of time (or make it up as I go). I usually don't create drawings or write down anything, instead I tend to research existing practice online and perhaps go for a long walk while I think things over. If I run into a design problem I tend to spend a while thinking how to resolve them gracefully; there usually is a good way to design things such that they aren't fragile and do things properly.

@h0bby1: Please use proper capitalization and grammar. :) I didn't see a single period in your post and I find it very hard to read such long sentences. You clearly invested a lot of time writing such a long post, it just feels so wrong to dismiss it because of such a silly reason.

That all depends upon how large or complicated it is. If it is just a utility, needed to do something straightforward, like produce a hex dump, then I just code it. But the larger a project is, the more likely it is to come to grief without some advance planning. Data structures are a good place to start, but even they presuppose some conception of why the data they will be holding is needed, and how it will be used.

At the moment I am planning a file system. I thought I had it fairly well thought out, and I had the data structures drawn up, but then it occurred to me that a part of it was somewhat less than optimal. So now I am thinking again.

Perfection must be reached by degrees; she requires the slow hand of time. --- Voltaire.

I'm in the middle of a potentially long development (5-10 years) so I've started small and I'm growing it bigger. I built a prototype and assumed it would be thrown away so I didn't spend too much effort on making it robust or fully featured. Once this proof of concept was working on very simple test cases, I implemented it properly. I now plan on incremental improvements, adding features or making optimisations, over and over until it's where it needs to be. This isn't easy to split into a planning phase and an implementation phase --- the end goal is just too big to bite off in one chunk.

The Voltaire quote was taken from from Butler Lampson's Hints for Computer System Design: http://research.microsoft.com/en-us/um/people/blampson/33-hints/WebPage.html

Oki, sorry, i tried to improve it a bit

I guess I plan my software both on the fly and in writing. I try to write/draw out the large things and then when it comes to writing something I code it on the fly. I try as much as possible to not be caught off guard by corner cases and limitations that I did not see ahead of time. There has been a many times I spend a week writing some huge experimental system only to end up having to go back to the drawing board and essentially rewrite everything except the support code!

And, pen and paper is my choice for drawing things. I like to draw flow charts, diagrams, and depictions of execution flow. I prefer to write code not on paper but in an editor even if it is pseudo code because with paper I have to do too much erasing, but I may do just a tad of it on paper but not too much..

I also begin with structures and their fields before I do any coding, but as I progress I may alternative between coding some and adding more fields or structures, but I try to get the majority of the structure done before coding. However, at least getting the important fields in place helps. I have been known to sometimes prototype the methods/functions/interface before or at the same time as the structures, but for sure most of the time coding comes toward the end or at least after the major framework has been put into place.

I also break the project into major parts. You know like if it has a server and client components or even many servers and clients. I look at how they are goi ng to communicate and what kind of bandwidth and latency requirements there are, and what technologies can provide this.

From here I look at how each part of the project is going to be driven and how often. For example on demand (command line, web script..), continuously, or in increments such as a video game or simulation. This effects the requirements of the system and such.

Then I basically start implementing and breaking the major parts into sub-parts or services that it provides. This could be a service which runs at interval to provide locations of entities in a virtual world to client. It could be a service that gets a request from another service internally or external (remote client). It could be a service the kernel provides to give memory to an application when it requests it. Maybe a service to manage the usage of a resource. I am also looking at synchronous and asynchronous situations and how they are going to be handled. You know some requests need to block and others can be non-blocking or even need to support both.

Maybe service is not the best word in many cases, but there is always a request for something that must be satisfied. You know even at the basic level you write a function as a provider of a request. You call the function to do something or get something. Really, even if it is doing something your still getting something. So you got all your functions which then get grouped together to provide a service which could be static (only one exists) or it could be multiple instances.

I think understanding what you need to do is a major part, and starting work on something and having no idea what needs to be done can be scary and cause a lot of rewrites. So reading or researching is the best way to reduce the number of major changes you have to make, but being the first time doing something such as writing a kernel, game, or server is going to be tough. You are likely going to make mistakes in the design and the best you can hope for is that the mistakes are easy to fix or correct. You should notice as time goes on that you begin to get better at designing certain types of systems and that is because you end up learning the cons and pros of doing something a certain way.

You know memory, processing, storage, bandwidth, and latency are all factors that have to be considered.

But, the one of the best concepts to get down is the ability to group things together and make your design modular with good interfaces so that making a major rewrite only involves one group and not every group that depends on it.

As a matter of personal opinion I see, what I feel like, is people making things too modular where everything becomes an object/group/service which can be annoying when trying to read code so maybe starting with large groups and then moving down to smaller things will help. You know having too much syntactic sugar or boiler plate code can make reading code difficult and even managing it, but I guess that depends on the language you use to write something too. Of, course this is pure opinion! So do not let me scare you into doing too little or too much. It is just that a fine line exists and I find myself falling to either side sometimes, but I suppose if I had to choose I would rather do a bit too little with making everything an object than too much, but I have no proof about this being correct. This is just a matter of personal taste I guess I could say.

That could be the only situation where i use paper, in fact i have done this not to long ago planning for one of my projects. I have also done this for large if statements.

Thanks again for the responses. Surely this will help others adapt a preferred way of planning for development teams, and not to be afraid of their own techniques.

On a whiteboard.