Analysis and Solutions for "No space left on device" Error in Linux Systems

Problem Phenomenon and Initial Diagnosis

In Linux system administration practice, "No space left on device" is a common error message that typically appears during file operations such as scp transfers. Users report encountering this issue on CentOS machines, with basic command checks revealing contradictory phenomena: df -h shows root partition usage at 100%, while du -sh / reports only 5G of actual usage. This discrepancy indicates the complexity of disk space management and requires deep analysis of file system internal mechanisms.

Root Cause Analysis: Process Holding Deleted Files

The core reason for inconsistent results between df and du commands lies in Linux file system's reference counting mechanism. When a file is deleted, if processes still maintain open handles to that file, the data blocks are not immediately released. From the file system directory structure perspective, the file has been deleted, but actual disk space remains occupied until all related processes close their file handles.

This hypothesis can be verified using the following command:

lsof | grep deleted

This command output shows all deleted files that remain open and their associated processes. In the user's case, there might be large log files or temporary files that were opened by applications and then deleted, preventing disk space recovery.

Solutions and Implementation Steps

After identifying specific processes, the solution involves several steps:

First, use lsof command to precisely identify problematic processes:

lsof +L1 | grep -i deleted

This command specifically looks for deleted files with link count of 1 (+L1 parameter), providing more accurate identification of space occupation sources. The output will display process ID (PID), command name, and file size information.

Second, take appropriate measures based on process importance:

# For non-critical processes, terminate directly
kill -9 <PID>

# For important service processes, restart gracefully
systemctl restart <service-name>

After restarting processes, the operating system will clean up all closed file handles, releasing the occupied disk space. Running df -h again should then show normal available space.

Other Potential Causes and Investigation Methods

Beyond deleted file occupation, inode exhaustion is another common cause of "No space left on device" errors. Even with sufficient disk space, if the file system's inode table is full, new files cannot be created.

Check inode usage with the following command:

df -i /

If IUse% approaches 100%, you need to clean up numerous small files or adjust file system parameters. Find directories with highest inode usage:

find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

Preventive Measures and Best Practices

To prevent similar issues from recurring, implement the following preventive measures:

Establish regular disk space monitoring mechanisms with warning thresholds. Configure log rotation policies to prevent unlimited log file growth. For critical applications, ensure proper file closure handling to avoid file handle leaks.

In the referenced article about Safari browser cases, although the environment differs (macOS vs Linux), the principles are similar—applications may maintain references to deleted resources for various reasons, causing the system to report insufficient space. This emphasizes the importance of universal principles in cross-platform system management.

Technical Deep Dive

From the operating system kernel perspective, Linux uses reference counting to manage file lifecycles. Each file corresponds to an inode containing a reference count field. When a process opens a file, the reference count increases; when closed, it decreases. A file is only truly deleted when the reference count drops to zero and no directory entries link to it.

This design ensures files aren't accidentally deleted during use but also causes the discrepancy between df and du commands. df reports file system block allocation, while du calculates file sizes in directory trees, excluding deleted but still referenced files.

Understanding this mechanism is crucial for advanced system troubleshooting, especially when dealing with disk space issues in long-running services like databases and web servers.