Managing SSH Keys in Jenkins: Resolving Host Key Verification Issues for Git Repository Connections

Problem Context and Error Analysis

In continuous integration environments, Jenkins' integration with Git version control systems forms the backbone of automated build pipelines. When connecting to GitHub-hosted repositories via SSH protocol, developers frequently encounter the "Host key verification failed" error, indicating that the SSH client cannot verify the remote server's identity.

The specific error message appears as:

Command "git ls-remote -h git@github.com:***/***.git HEAD" returned status code 128:
stdout: 
stderr: Host key verification failed. 
fatal: The remote end hung up unexpectedly

This issue stems from SSH protocol's security mechanism: during initial connection to a remote host, the client must add the server's host key to the local known_hosts file to establish trust. Jenkins typically runs as the jenkins system user, therefore maintaining an independent $HOME/.ssh directory structure.

SSH Key Verification Mechanism Explained

The SSH connection establishment process involves dual verification: user authentication (via SSH key pairs) and server authentication (via host keys). When Jenkins attempts to execute git ls-remote commands, the Git plugin invokes the system SSH client, which checks whether github.com's host key exists in the Jenkins user's known_hosts file.

In typical Linux distributions, the Jenkins user configuration follows this pattern:

# Jenkins user home directory is usually /var/lib/jenkins
# SSH configuration path is /var/lib/jenkins/.ssh/
# Contains these critical files:
# - id_rsa / id_rsa.pub (private/public key pair)
# - known_hosts (list of verified host keys)
# - config (SSH client configuration)

When the GitHub server entry is missing from the known_hosts file, the SSH client rejects the connection to prevent man-in-the-middle attacks.

Solution Implementation Steps

The most straightforward solution involves manually triggering the initial SSH connection to automatically add the host key. Since the Jenkins user's default shell is typically set to /bin/false (for security reasons), you must explicitly specify a shell for interactive command execution.

The operational workflow proceeds as follows:

# Switch to jenkins user with explicit bash shell
sudo su jenkins -s /bin/bash

# Navigate to temporary working directory
cd /tmp

# Execute git clone operation (or any git command triggering SSH connection)
# System will prompt to add host key to known_hosts
# Confirm by typing "yes"

# Exit jenkins user session
exit

The core principle of this process is: by manually executing one successful SSH connection, the system automatically adds github.com's host key to the /var/lib/jenkins/.ssh/known_hosts file. Subsequently, all Git operations from Jenkins will properly verify server identity.

Configuration Optimization and Extended Discussion

For projects containing multiple Git submodules, key management strategies require special consideration. While separate deploy keys could be configured for each sub-repository, this increases management complexity. Using a personal GitHub account's SSH key as a unified authentication credential provides a cleaner solution, provided the account has access to all relevant repositories.

The SSH configuration file (~/.ssh/config) can further optimize connection behavior:

Host github.com
    User git
    IdentityFile /var/lib/jenkins/.ssh/id_rsa
    StrictHostKeyChecking yes
    UserKnownHostsFile /var/lib/jenkins/.ssh/known_hosts

This configuration explicitly specifies key file paths and host key verification policies, ensuring Jenkins consistently uses the correct identity for authentication.

Security Considerations and Best Practices

When handling SSH keys in automated environments, balancing convenience with security is essential:

1. Key Permission Management: Ensure Jenkins user's private key files (e.g., id_rsa) have permissions set to 600 to prevent unauthorized access.
2. known_hosts Integrity: Regularly verify entries in the known_hosts file, avoiding security-reducing options like StrictHostKeyChecking no.
3. Key Rotation Strategy: Establish mechanisms for periodic SSH key updates, particularly after team member changes or security incidents.
4. Audit Log Monitoring: Monitor Jenkins' Git operation logs to promptly detect anomalous connection attempts.

For legacy systems like CentOS, compatibility issues with SSH client versions require attention. Older OpenSSH versions might not support the latest key algorithms, necessitating adjustments to GitHub's SSH configuration.

Troubleshooting and Verification

After implementing the solution, verify configuration effectiveness through these steps:

# Test SSH connection as jenkins user
sudo -u jenkins ssh -T git@github.com

# Expected output: "Hi username! You've successfully authenticated..."
# Or at least no host key verification errors

# Check known_hosts file content
sudo cat /var/lib/jenkins/.ssh/known_hosts | grep github.com

# Test complete Git operations in Jenkins jobs
# Including cloning, pulling, and submodule initialization

If issues persist, examine network proxy settings, firewall rules, and Git plugin version compatibility. In some enterprise environments, configuring SSH through HTTP proxies or using custom CA certificates may be necessary.

By systematically understanding SSH verification mechanisms and Jenkins' runtime environment, developers can establish stable and reliable Git integration workflows, laying a solid foundation for continuous integration and deployment pipelines.