Elegant Redirection of systemd Service Output to Files Using rsyslog

Problem Background and Challenges

In Linux system administration, systemd serves as the modern init system responsible for managing the lifecycle of various system services. However, many administrators encounter difficulties when attempting to redirect service standard output (stdout) and standard error (stderr) to files. Traditional redirection methods often fail to work properly in the systemd environment due to its special handling of process output.

Limitations of Direct File Redirection

From the Q&A data, we can see that users initially attempted configurations like StandardOutput=/var/log1.log and StandardError=/var/log2.log, but this approach presents several critical issues. First, this simple path specification method is not supported in older systemd versions. Second, even when supported in newer versions, direct file redirection lacks flexibility and cannot handle complex scenarios like log rotation and permission management.

The example from the reference article further confirms this problem: even with proper configuration of StandardOutput=file:/home/root/log1.log in the service file, log files might remain empty. This indicates the need for a more systematic solution.

Elegant Solution Based on syslog

Best practices show that combining systemd with rsyslog provides a more flexible and reliable log management solution. The specific implementation steps are as follows:

Modifying systemd Service Configuration

First, configure output redirection to syslog in the systemd service unit file:

[Unit]
Description=customprocess
After=network.target

[Service]
Type=forking
ExecStart=/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=customprocess
Restart=always

[Install]
WantedBy=multi-user.target

The key configurations include:

• StandardOutput=syslog: Redirects standard output to system log
• StandardError=syslog: Redirects standard error to system log
• SyslogIdentifier=customprocess: Sets a unique program identifier

Configuring rsyslog Filtering Rules

Next, create a custom configuration file in rsyslog, typically located in the /etc/rsyslog.d/ directory:

if $programname == 'customprocess' then /var/log/customprocess.log
& stop

This configuration means: if the program name matches 'customprocess', write log messages to the /var/log/customprocess.log file, then stop further processing (to avoid duplicate records).

Permission Management and Service Restart

Ensure the log file has correct permissions:

sudo chown syslog:adm /var/log/customprocess.log
sudo chmod 640 /var/log/customprocess.log

Then restart the rsyslog service to apply configuration changes:

sudo systemctl restart rsyslog

Solution Advantages Analysis

This syslog-based approach offers several significant advantages:

Flexible Log Management

Through rsyslog's filtering capabilities, log routing based on various conditions such as program name, priority, and time can be implemented. This makes log management more flexible, enabling easy log classification, archiving, and rotation.

Dual Access Channels

Log information exists in two locations simultaneously: the specified log file and the systemd journal system. Administrators can view service logs using journalctl -u customprocess or access log files directly through the filesystem.

Better Permission Control

rsyslog runs with specific user privileges, allowing better control over log file access permissions and enhancing system security.

Version Compatibility

This method doesn't depend on specific systemd versions and works reliably on most modern Linux distributions, including CentOS 7 and RHEL 7.

Alternative Solutions Comparison

While direct file redirection may be suitable in some scenarios, it has obvious limitations:

Version Dependency of File Redirection

As mentioned in the Q&A data, the StandardOutput=file:/path/to/file format requires systemd version 236 or newer. In older systems like CentOS 7, this functionality might be unavailable or require additional configuration.

Append Mode vs Overwrite Mode

Direct file redirection supports two modes: file: (overwrite) and append: (append). However, in production environments, the lack of automatic log rotation functionality can become problematic.

Directory Creation Limitations

The file redirection functionality in systemd typically doesn't automatically create non-existent directories, requiring pre-existing directory structures before service configuration.

Practical Recommendations and Best Practices

Based on actual deployment experience, we recommend the following best practices:

Log File Naming Conventions

Use meaningful log file names, such as /var/log/<service-name>.log, for easy identification and management.

Log Rotation Configuration

Configure automatic log rotation using the logrotate tool to prevent unlimited log file growth:

/var/log/customprocess.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 640 syslog adm
}

Monitoring and Alerting

Establish log monitoring mechanisms with real-time alerts for critical error messages to ensure timely problem detection and resolution.

Conclusion

By combining systemd's syslog redirection functionality with rsyslog's flexible filtering mechanisms, we can build a robust, maintainable service log management system. This approach not only addresses basic log redirection requirements but also provides a solid infrastructure foundation for subsequent log analysis, monitoring, and troubleshooting. In practical production environments, this solution has proven to be one of the best practices for managing systemd service logs.