calling convention

A calling convention describes how function arguments are passed to a function call (e.g., use stack or registers; if stack, push in what order), how the callee returns a value to the caller, and who cleans up the stack after a call, e.g., function call arguments are pushed onto the stack right to left (reverse parameter list order), return value is passed on XXX register, caller cleans up the stack, etc.

Some commonly seen calling conventions include:

• __cdecl is the default calling convention for C/C++ on 32-bit systems.
  • The caller cleans up the stack.
  • Function arguments are pushed onto stack in reverse order, i.e., the last argument is pushed first, so that the first argument is the closest to the callee stack frame (and thus requires the least offset from ebp on x86).
  • See x86 calling convention
• __stdcall is an MSVC/Windows-specific calling convention on 32-bit systems; also used for WinAPI and callbacks passed into WinAPI.
  • The callee cleans up the stack.
  • Function arguments are handled in the same way as __cdecl.
• SysV ABI dictates a calling convention Linux x86_64 systems
  • The caller cleans up the stack.
  • First 6 integer/pointer arguments are passed via rdi, rsi, rdx, rcx, r8, r9.
  • The rest are passed on the stack like cdecl.
  • Other arguments are passed via stack or SIMD registers.
• __fastcall is the default calling convention on Windows x64 systems.
  • The callee cleans up the stack.
  • First four arguments are passed by registers on x86_64 (rcx, rdx, r8, r9), and the remaining are passed on stack.
  • This is a non-standard calling convention that differs from compiler to compiler.
• __thiscall is a MSVC-specific extension that passes the this pointer via ecx.
• __clrcall is used for CLR managed code, which is a part of .NET.

To override the calling convention of a C function declaration, add one of these keywords after the return type, e.g., int __cdecl func();.

to-do Linux syscalls (Windows don’t expose them directly)