Brendan wrote:
If a group of one or more malicious processes are able to cause "denial of service" by flooding the VFS with requests; then they are probably also able to cause "denial of service" by flooding the network stack, or the GUI, or many other processes/services. You'd want to solve the problem for all cases (and not just for the VFS case); so the solution should probably should be built into the messaging system itself.
My main defence is prioritisation (both thread priorities and IO priorities). The idea is that normal processes aren't able to use higher priorities; so more important things (GUI, system services, drivers, etc) still function under flood conditions because their requests are higher priority and take precedence. A flood from lower priority (normal) processes only starves lower priority/normal processes of IO and/or CPU time.
Korona suggested that the OS shouldn't allow "arbitrary" numbers of requests, but I'm not really sure what alternatives there are.
If you only allow some relatively low fixed number of requests then isn't it easier to cause DoS? Further, it would have to be dynamically selected number per system, since a 10x system should allow for more requests, how would such an algorithm be devised and how accurate would it be?
I think prioritization is the best solution here, most importantly making sure the user is and remains in control, which means the user can shut down offending processes.
Brendan wrote:
For my OS (because of asynchronous messaging) each message costs RAM (the messages sit in message queues in kernel space until the receiver asks to receive them), and if the receiver can't handle the requests fast enough its message queue continues to grow until the kernel runs out of RAM. To guard against this I'm considering "thread penalties", where (if a thread sends a message to a receiver that already has a "too full" message queue) the scheduler is told to reduce the priority of the sending thread for a while, and these penalties would be cumulative; and I'm also considering "priority boosts" (if a thread's message queue is "too full", tell the scheduler to give the thread a priority boost for a while). This is mostly just basic flow control (if receiver can't keep up; drop speed of sender/s and boost speed of receiver).
Have you considered keeping the message in the senders address space and not in kernel? That way the "cost" is being associated with the problem process and for example a low priority process hogging all RAM can easily be identified and killed. Also helps with accounting, nothing is more annoying than being out of resources (RAM or otherwise) and not being able to tell why.
Brendan wrote:
The other thing I'm considering (because I have an "anyone can send any message to anyone else" model) is some kind of system to report and deal with "spam messages";
I'm still undecided on this, basically I've been thinking of two different options:
- Each process is responsible for their own message filtering. This has the benefit of allowing each process to customize the filtering code for best performance, for example PMM can reject all messages that are not from VMM. Whereas something like an online banking service needs much more complex system.
- Provide filtering in the messaging system. This has the benefit of being able to ensure, to a reasonable degree, that the filtering code itself is flawless, instead of each app rolling their own.
In practice the correct solution is probably something in between, for example a library that has a couple of different filters from simple to complex.