Technical Analysis and Practice for Fixing systemd Service 203/EXEC Failure (No Such File or Directory)

Problem Background and Error Phenomenon

When configuring scheduled tasks with systemd, users often encounter service startup failures, with the 203/EXEC error being particularly common. This error typically manifests as:

backup.service: Failed at step EXEC spawning /home/user/.scripts/backup.sh: No such file or directory.
backup.service: Main process exited, code=exited, status=203/EXEC
Failed to start backup.
backup.service: Unit entered failed state.
backup.service: Failed with result 'exit-code'.

In-depth Analysis of Error Causes

The root cause of the 203/EXEC error lies in systemd's inability to properly execute the specified program or script. Although the surface error message indicates "No such file or directory," the actual situation may be more complex:

Script Interpreter Issues: When a script path is directly specified in the .service file, systemd attempts to execute the file directly. If the script lacks a proper shebang line, or if the system cannot recognize the script's interpreter, execution will fail.

Consider the following script example:

#!/usr/env/bin bash
rsync -a --delete --quiet /home/user/directory/ /mnt/drive/directory-backup/

The shebang line in this script is problematic; the correct path should be #!/usr/bin/env bash. Such subtle path errors might be overlooked during manual execution but can cause fatal errors in the systemd environment.

Solutions and Practical Verification

Method 1: Explicitly Specify the Interpreter

Explicitly specifying the script interpreter in the .service file is the most reliable solution:

[Service]
Type=oneshot
ExecStart=/bin/bash /home/user/.scripts/backup.sh

This method completely bypasses the script's own shebang line, ensuring that the correct interpreter is used to execute the script. Practical testing confirms that the modified configuration successfully starts the service and executes the scheduled task.

Method 2: Fix the Script Shebang

Another approach is to ensure the script's shebang line is correct:

#!/bin/bash
rsync -a --delete --quiet /home/user/directory/ /mnt/drive/directory-backup/

Or use a more generic approach:

#!/usr/bin/env bash
rsync -a --delete --quiet /home/user/directory/ /mnt/drive/directory-backup/

Related Cases and Extended Analysis

Referencing other similar cases, such as configuration issues with nodepool-launcher services, reveals similar error patterns. When multiple ExecStart directives are incorrectly specified in the .service file, the following error occurs:

nodepool-launcher.service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

This reminds us to pay attention to the following when configuring systemd services:

1. For Type=oneshot services, multiple ExecStart directives can be specified
2. Other types of services can only have one ExecStart directive
3. Ensure the specified executable file actually exists and has execution permissions

Complete Troubleshooting Process

When encountering a 203/EXEC error, it is recommended to follow these troubleshooting steps:

Step 1: Verify File Existence

ls -la /home/user/.scripts/backup.sh

Step 2: Check File Permissions

chmod +x /home/user/.scripts/backup.sh

Step 3: Verify Script Executability

/home/user/.scripts/backup.sh

Step 4: Check Shebang Line

head -1 /home/user/.scripts/backup.sh

Step 5: Use Absolute Path to Specify Interpreter

ExecStart=/bin/bash /home/user/.scripts/backup.sh

Best Practice Recommendations

Based on practical experience, we summarize the following best practices:

1. Always Use Absolute Paths

In .service files, all paths should use absolute paths, including script paths and interpreter paths.

2. Explicitly Specify the Interpreter

Even if the script contains a correct shebang line, explicitly specifying the interpreter in the .service file provides better reliability.

3. Thorough Testing

Before deploying to production, ensure thorough testing:

systemctl --user daemon-reload
systemctl --user start backup.service
systemctl --user status backup.service

4. Log Monitoring

Use journalctl to monitor service logs and promptly identify and resolve issues:

journalctl --user -u backup.service -f

In-depth Discussion of Technical Principles

When executing a service, systemd strictly follows the specified path to find the executable file. When directly specifying a script path, systemd will:

1. Check if the file exists
2. Check if the file has execution permissions
3. Read the file's magic number or shebang line to determine how to execute it
4. If the execution method cannot be determined, return a 203/EXEC error

By explicitly specifying the interpreter, we skip step 3, directly telling systemd to use a specific interpreter to execute the script, thus avoiding execution failures due to shebang issues.

Conclusion

Although the 203/EXEC error appears simple on the surface, it involves multiple aspects of systemd service execution. By deeply understanding the error mechanism, adopting correct configuration methods, and combining systematic troubleshooting processes, we can effectively resolve such issues. The method of explicitly specifying the interpreter not only solves the current problem but also improves the reliability and maintainability of service configuration.