Windows OpenSSH Public Key Authentication Failure: Service Account Permission Analysis and Solutions

Problem Phenomenon and Background

When configuring OpenSSH public key authentication in Windows environments, users frequently encounter sudden connection termination issues. From the provided debug logs, it's evident that the client normally sends public key authentication requests, but the server immediately closes the connection after receiving the public key packet, with error messages showing “Connection closed by 127.0.0.1”. This problem occurs in both local testing and remote connections, and when both password and public key authentication are enabled, the server only attempts public key authentication and doesn't fall back to password authentication upon failure, further complicating troubleshooting.

Root Cause Analysis

The core issue lies in the account permissions under which the OpenSSH service (sshd) runs. In Windows systems, the OpenSSH service may default to running under the Local System account. While this account has high system privileges, it has limitations when accessing user-specific file system resources. Specifically, when the sshd process attempts to read the authorized_keys file in the user's .ssh directory, the Local System account may fail Windows file system permission checks.

Windows security model requires processes to have appropriate permissions to access files in user directories. Even if users have correctly set permissions for the authorized_keys file, if the sshd service runs in the wrong account context, it still cannot read these files. This leads to silent failures in the authentication process—the server cannot verify the public key and therefore terminates the connection directly without providing detailed error information.

Solution Implementation

To resolve this issue, ensure the OpenSSH service runs under an account that can access the target user's files. Here are specific implementation steps:

1. Stop Current Service: First, stop the running OpenSSH service with administrator privileges. This can be done via command prompt with net stop sshd or using PowerShell's Stop-Service sshd command.
2. Configure Service Account: Open Windows Service Manager (services.msc), find the OpenSSH SSH Server service. Right-click for properties, switch to the “Log On” tab. Here, change the service running account from Local System to the target user account (the account you want to connect via SSH).
3. Set Account Permissions: Ensure the selected account has permission to access the C:\Users\[username]\.ssh directory and its contents. Typically, users have full control over their home directories, but verify that permissions for the .ssh directory and authorized_keys file are correctly set.
4. Start Service: After saving configuration changes, start the OpenSSH service. Use net start sshd or Start-Service sshd command.
5. Verify Configuration: Attempt to connect to the server using an SSH client. If configured correctly, public key authentication should work properly.

Additional Configuration Recommendations

Beyond service account configuration, pay attention to these key configuration points:

• File Permission Settings: authorized_keys file permissions must be strictly restricted. Only allow SYSTEM and the corresponding user account access, remove all other permission entries. Inherited permissions should be disabled and converted to explicit permissions.
• sshd_config Configuration: Check relevant settings in the sshd_config file. Ensure PubkeyAuthentication is set to yes, and AuthorizedKeysFile points to the correct location. For Windows 1809 and later versions, you may need to comment out specific key file configurations for administrator groups.
• Service Dependencies: OpenSSH service may depend on other services (such as SSH agent service). Ensure these related services also run under correct accounts and have proper startup order.

Troubleshooting Techniques

If problems persist after implementing the above solution, try these troubleshooting methods:

• Enable Detailed Logging: Add LogLevel DEBUG3 setting in sshd_config, then restart the service. Check Windows Event Viewer or SSH log files for more detailed error information.
• Manual File Access Testing: Use PowerShell or command prompt to attempt reading the authorized_keys file as the service account. This helps confirm whether permission issues have been resolved.
• Network Policy Check: Ensure no firewall or network security policies block SSH connections. Particularly in enterprise environments, group policies may restrict how services run.

Conclusion and Best Practices

The fundamental cause of Windows OpenSSH public key authentication failure is typically insufficient permissions of the service running account. By configuring the service to run under the target user account, you ensure the sshd process can access necessary authentication files. This solution applies not only to the described specific scenario but also to other similar permission-related issues.

In actual deployments, follow these best practices:

1. Create dedicated service accounts for SSH services rather than directly using user accounts.
2. Regularly audit file system permissions to ensure only authorized accounts can access sensitive files.
3. Back up original settings before changing service configurations for quick recovery if problems occur.
4. Thoroughly validate configurations in test environments before deploying to production.

By properly understanding Windows permission models and OpenSSH working principles, you can effectively avoid public key authentication failures and ensure SSH service reliability and security.