Ethin wrote:
1. Why don't we put all integer arguments in RAX, RBX, RCX, RDX, R8, R9, R10, ..., or use R8-R15 for arguments and RAX/RBX/RCX/RDX/RSI/... for return values?
Having non-volatile registers (registers the callee needs to keep the same) is good for calling functions in a loop. Having all registers as arguments would simultaneously make them all volatile, so before you can call a function, you have to save everything you currently have in registers. And if you need to call functions repeatedly, you have to do that every time. Add to that the fact that most leaf functions don't require all registers, and it becomes obvious that such a calling convention does not use the registers very well, and would over-use the stack.
Ethin wrote:
2. For system calls, I've noticed that -- for example -- Linux uses EAX, EBX, ECX, and EDX fro arguments, but this is very different on x64. Why is this the case? Shouldn't it use the 64-bit RAX/RBX/... registers on x64 like it does on x86?
When Linus designed the system call mechanism for x86, it was early in the development, and he just sorta winged it. By the time AMD64 had come around, Linus was no longer the primary developer, having instead transitioned to a managerial position (he's the guy that approves the patches other people write). The system call mechanism on AMD64 was written by other people, and they thought it would be best to just stick to the calling convention they were going to use. Shuffling arguments around is a major part of calling syscalls on x86, and that is just not useful to anyone. So instead, they kept to the user space argument order, except for rcx, which will be clobbered by the syscall instruction.
Ethin wrote:
3. For vector arguments, vectorcall uses only XMM0-XMM5. So what happens to XMM6-XMM31, and why are they unused? Similarly, what about YMM/ZMM0-31?
Yeah, this one doesn't really make sense to me, either. But then, functions taking more than six floating-point arguments are rare, so who cares? The XMM registers are all marked as volatile in the ABI, so this is not to allow the caller some space to save their FP variables.
Ethin wrote:
4. Would it be okay for my OS to use the above (logical) calling convention, or would it break a lot, and how much effort would it require for me to add that to compilers/interpreters?
Going against the grain on calling conventions requires you to patch at least the compiler (and good luck with that). Unless you have an extremely compelling reason to do so, I don't think it is worth the effort.