Comprehensive Analysis of System Call and User-Space Function Calling Conventions for UNIX and Linux on i386 and x86-64 Architectures

Overview of Calling Conventions

In x86 architecture operating system development, system call and function calling conventions constitute fundamental knowledge for low-level programming. These conventions define parameter passing, register usage, and return mechanisms, directly impacting the efficiency of interaction between assembly language and the operating system kernel. Based on the System V ABI specification and Linux kernel implementation, this paper systematically analyzes the differences in calling conventions between i386 and x86-64 architectures.

System Call Conventions for i386 Architecture

Linux System Call Mechanism

In 32-bit Linux systems, system calls are implemented through the int $0x80 software interrupt. Parameter passing relies entirely on registers: the system call number is stored in the %eax register, while six parameters are passed sequentially using %ebx, %ecx, %edx, %esi, %edi, and %ebp. Upon return, the system call result is placed in %eax, with other registers (including EFLAGS) preserved. This design simplifies kernel-user space switching but limits the number of parameters.

When more than six parameters are required, the convention dictates that the memory address of the parameter list be stored in %ebx, though actual system calls rarely need this many parameters. The following example demonstrates basic usage of the write system call:

section .data
msg db 'Hello, World!', 0xA
len equ $ - msg

section .text
global _start
_start:
    mov eax, 4        ; syscall number for write
    mov ebx, 1        ; file descriptor (stdout)
    mov ecx, msg      ; pointer to message
    mov edx, len      ; message length
    int 0x80          ; invoke system call

    mov eax, 1        ; syscall number for exit
    xor ebx, ebx      ; exit code 0
    int 0x80

Modern Linux also supports the sysenter instruction for faster system calls. The kernel maps a memory page containing sysenter user-space code via the vDSO (virtual Dynamic Shared Object) for each process. Developers should invoke through the vDSO interface rather than using sysenter directly to ensure compatibility and correctness.

BSD System Call Mechanism

Unlike Linux, BSD-family systems (e.g., FreeBSD, OpenBSD) on i386 architecture use stack-based parameter passing. Parameters are pushed onto the stack from right to left, followed by a 32-bit dummy value (which actually contains additional information), before executing int $0x80. This stack-based calling convention affects performance but maintains consistency with traditional C calling conventions.

System Call Conventions for x86-64 Architecture

Linux System Call Mechanism

64-bit Linux systems employ the syscall instruction for system calls, significantly improving performance. According to the AMD64 supplement of the System V ABI, the system call interface uses specific register sequences: the system call number is placed in %rax, with parameters passed in order using %rdi, %rsi, %rdx, %r10, %r8, and %r9. Note that %rcx is replaced by %r10 because the syscall instruction clobbers %rcx and %r11.

System calls are limited to six parameters, with no direct stack passing. Upon return, %rax contains the result value; if the value falls within the range -4095 to -1, it indicates an error (-errno). Only parameters of INTEGER or MEMORY class can be passed to the kernel. The following example illustrates a 64-bit write system call:

section .data
msg db 'Hello, x86-64!', 0xA
len equ $ - msg

section .text
global _start
_start:
    mov rax, 1        ; syscall number for write
    mov rdi, 1        ; file descriptor (stdout)
    mov rsi, msg      ; pointer to message
    mov rdx, len      ; message length
    syscall           ; invoke system call

    mov rax, 60       ; syscall number for exit
    xor rdi, rdi      ; exit code 0
    syscall

Using the 32-bit int $0x80 ABI in 64-bit code is possible but strongly discouraged. It truncates inputs to 32 bits, making it unsuitable for pointers, and zeroes the r8-r11 registers, potentially causing unpredictable behavior.

BSD and Other Systems

x86-64 BSD system calling conventions differ from Linux, typically referencing their respective ABI documentation. macOS, based on BSD, has its own unique conventions that require separate consultation. Developers should always verify the specific ABI requirements of the target system.

User-Space Function Calling Conventions

i386 Function Calling Convention

In the 32-bit System V ABI, function parameters are passed via the stack. Parameters are pushed from right to left, followed by execution of the call instruction. Modern Linux requires 16-byte alignment of %esp before call to support SSE instructions, though historical versions only required 4-byte alignment. Other 32-bit systems may still maintain 4-byte alignment requirements, affecting cross-platform compatibility.

x86-64 System V Function Calling Convention

The 64-bit System V ABI uses register-based parameter passing for improved performance. Parameters are first classified into categories such as INTEGER or MEMORY. INTEGER-class parameters use registers in the sequence %rdi, %rsi, %rdx, %rcx, %r8, and %r9. The first eight floating-point parameters use %xmm0-%xmm7, with subsequent parameters passed via the stack. The stack pointer (%rsp) must be 16-byte aligned at the time of call.

If there are more than six INTEGER parameters, the seventh and later are passed on the stack (with the caller responsible for cleanup). Variadic functions (e.g., printf) require setting %al to the number of floating-point register arguments. Structures may be returned via registers (e.g., rdx:rax) or memory, with specific rules detailed in the ABI. The following example demonstrates register parameter passing:

; Assuming call to function func(a, b, c, d, e, f, g)
; a-g are integer parameters
mov rdi, a_val  ; first parameter
mov rsi, b_val  ; second parameter
mov rdx, c_val  ; third parameter
mov rcx, d_val  ; fourth parameter
mov r8, e_val   ; fifth parameter
mov r9, f_val   ; sixth parameter
push g_val      ; seventh parameter via stack
call func
add rsp, 8      ; clean up stack parameter

Windows x64 Calling Convention

The Windows x64 calling convention differs significantly from System V: the caller must reserve shadow space instead of using a red-zone; xmm6-xmm15 are call-preserved registers; and the parameter register sequence differs (e.g., rcx, rdx, r8, r9 for the first four parameters). These distinctions must be considered when developing cross-platform code.

Conclusion and Best Practices

Understanding calling conventions is crucial for low-level programming. In x86-64 systems, prefer the syscall instruction over int 0x80. Always reference the latest ABI documentation (e.g., System V psABI) and test compiler output to ensure compatibility. For performance-sensitive applications, leverage register-based parameter passing to reduce memory access. When developing cross-platform, abstract calling interfaces to handle system differences. By mastering these conventions, developers can write efficient, portable assembly code and gain deeper insight into operating system kernel interaction mechanisms.