x86-64 and calling conventions

Well, I have my own compiler, however, I don't as of yet target x86-64.


one of the main reasons:
the calling conventions used in Linux and friends are, scary...




Short Description:
arguments are passed in registers, meanwhile, others may be left on the
stack, and there is no particular relation between the registers and the
stack layout.


Beyond Just Putting A Big Damper On The Compiler Machinery, It Leaves Many
Ugly Questions:
how does one effectively deal with varargs? (of all things, the convention
still uses regs even for passing varargs...).


it seems almost like one will have to dump all of the registers into a kind
of a context, and keep note of particular counts (GPR pos, XMM pos, stack
pos).


how does one do things like hidden argument injection? (useful for many
tasks, such as closures).
....




Or, in general, there are all sorts of ugly questions wrt the topic of
actually working with the args list in a first-class manner (beyond just the
task of having to get the compiler to work with all this to begin with).




I am left considering the idea of abandoning this convention for internal
usage, and instead using a different default calling convention. probably,
name mangling would be used to distinguish them, and interface stubs would
be generated to handle external calls.


sadly, this would lead to a big issue, which is that there would be a kind
of a "wall" between the two worlds. firstly, interface stubs would be
needed, and secondly, that function pointers could not be safely passed
across this wall (limiting the usage of callbacks, ...).


more-so, these stubs could potentially be limited in terms of what they can
pass along (functions passing raw structs or similar stack-disordering
features may not be wrapable).


so, such a wall could be very problematic (especially in a mixed-compiler
codebase).




now, what kind of convention would I be imagining:
well, basically, it would be a hybrid of the existing conventions, and the
good old 32-bit x86 convention.


in particular, arguments would be passed on the stack (in good old linear
right-to-left ordering).




now, beyond this wall, there is also a possible risk of a performance impact
(both with calls, and in general).


it is also possible I could just "bear with it", and possibly use
stack-flattening/unflattening hacks mostly as a means of dealing with edge
cases.




alternatively, I could use a variant of the MS convention by default (even
on Linux), which is basically about the same (it being fairly simple to dump
arguments to the shadow space, or retrieve them from there).


the task of defeating the "wall" would then be a task for later...




any suggestions?
[My suggestion is that if you want people to use the compiler, use the same
calling sequence as everyone else. The code to deal with varargs is ugly,
but I would be surprised if it were very long. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.