Resolving Git SSH Host Key Verification Failed in Jenkins

Problem Background and Phenomenon Analysis

In continuous integration environments, integrating Jenkins with Git version control systems is a common configuration scenario. However, when using SSH protocol to connect to Git repositories, "Host key verification failed" errors frequently occur. The core of this issue lies in the security verification mechanism of the SSH protocol.

From the provided Q&A data, we can see that users encounter two different authentication problems when configuring Git in Jenkins:

// HTTPS authentication failure example
Failed to connect to repository : Command "git ls-remote -h https://person@bitbucket.org/person/projectmarket.git HEAD" returned status code 128:
stdout:
stderr: fatal: Authentication failed

// SSH host key verification failure example  
Failed to connect to repository : Command "git ls-remote -h git@bitbucket.org:person/projectmarket.git HEAD" returned status code 128:
stdout:
stderr: Host key verification failed.
fatal: The remote end hung up unexpectedly

Jenkins Service Account Characteristics Analysis

Jenkins runs as a service account, typically without an interactive shell environment. While this design enhances security, it also presents challenges for SSH host key verification. When the Jenkins service account connects to a remote Git server via SSH for the first time, it cannot interactively accept host keys like regular users.

The key point mentioned in the reference article is: "Jenkins is a service account, it doesn't have a shell by design. It is generally accepted that service accounts shouldn't be able to log in interactively." This characteristic directly leads to host key verification failures.

SSH Host Key Verification Mechanism

The SSH protocol uses host key verification to prevent man-in-the-middle attacks. When a client connects to a server for the first time, the server sends its public key fingerprint, and the client needs to confirm and accept this key. Accepted keys are stored in the ~/.ssh/known_hosts file and automatically verified in subsequent connections.

In the Jenkins environment, this verification process cannot proceed interactively, requiring pre-configuration of the known_hosts file or adoption of alternative verification strategies.

Solution: Manual Host Key Verification

Based on the best answer recommendation, the most direct solution is to manually complete the initial host key verification:

// Switch to jenkins user
sudo su -s /bin/bash jenkins

// Manually execute git command to trigger host key verification
git ls-remote -h git@bitbucket.org:person/projectmarket.git HEAD

After executing the above command, the system will display a prompt similar to:

The authenticity of host 'bitbucket.org (207.223.240.181)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?

After typing yes, Bitbucket.org's host key will be added to the /var/lib/jenkins/.ssh/known_hosts file. This process only needs to be performed once, and subsequent Jenkins jobs will be able to connect to the Git repository normally.

SSH Key Configuration Details

In addition to host key verification, proper SSH key configuration is equally important. Here is the complete SSH key configuration process:

// Switch to jenkins user and start bash shell
sudo su -s /bin/bash jenkins

// Generate SSH key pair (if not already exists)
ssh-keygen -t rsa -b 4096 -C "jenkins@yourcompany.com" -f /var/lib/jenkins/.ssh/id_rsa

// View public key content
cat /var/lib/jenkins/.ssh/id_rsa.pub

// Add public key content to Git service provider's deployment keys

During key generation, it's recommended not to set a passphrase so Jenkins can use the key without interaction. If a passphrase is necessary, ssh-agent should be configured to manage the key.

Git Host Key Verification Configuration Strategies

The reference article discusses Jenkins' Git host key verification configuration options in detail. In "Manage Jenkins" → "Configure Global Security" → "Git Host Key Verification Configuration", three strategies are provided:

1. Known hosts file strategy: Uses pre-configured known_hosts files for verification
2. Accept first connection strategy: Automatically accepts host keys for first connections
3. Manually provided keys strategy: Manually specifies trusted host keys

For most scenarios, the "Accept first connection" strategy is the simplest and most effective choice, especially in environments where direct control over agent file systems is not available.

known_hosts File Management

If choosing the known_hosts file strategy, ensure the file exists in the correct location. For Jenkins controllers, the known_hosts file should be located at:

/var/lib/jenkins/.ssh/known_hosts

Host keys can be manually added using the following commands:

// Get Bitbucket's host key and add to known_hosts
ssh-keyscan bitbucket.org >> /var/lib/jenkins/.ssh/known_hosts

// Verify key format and content
ssh-keygen -lf /var/lib/jenkins/.ssh/known_hosts

Permissions and Ownership Configuration

Correct file permissions are crucial for SSH connections:

// Set correct directory permissions
chmod 700 /var/lib/jenkins/.ssh

// Set private key file permissions  
chmod 600 /var/lib/jenkins/.ssh/id_rsa

// Set public key file permissions
chmod 644 /var/lib/jenkins/.ssh/id_rsa.pub

// Set known_hosts file permissions
chmod 644 /var/lib/jenkins/.ssh/known_hosts

// Ensure correct file ownership
chown -R jenkins:jenkins /var/lib/jenkins/.ssh

Troubleshooting and Verification

After completing configuration, comprehensive connection testing is recommended:

// Switch to jenkins user for connection testing
sudo su -s /bin/bash jenkins

// Test SSH connection
ssh -T git@bitbucket.org

// Test specific git operations
git ls-remote git@bitbucket.org:person/projectmarket.git

// Check detailed debug information
GIT_SSH_COMMAND="ssh -v" git ls-remote git@bitbucket.org:person/projectmarket.git

If problems persist, check the following aspects:

• Whether SSH keys are correctly configured with the Git service provider
• Whether known_hosts file content is correct
• File permissions and ownership settings
• Network connectivity and firewall configuration
• Jenkins global security configuration

Best Practices Summary

Based on analysis of Q&A data and reference articles, we summarize the following best practices:

1. Use "Accept first connection" strategy: This is the simplest and most effective solution for most scenarios
2. Unified key management: Create dedicated SSH key pairs for Jenkins, avoiding personal keys
3. Complete testing process: Always perform end-to-end connection testing after configuration
4. Documented configuration: Record all configuration steps for future maintenance and troubleshooting
5. Security considerations: Follow the principle of least privilege while ensuring normal functionality

By systematically understanding and implementing the above solutions, you can completely resolve Git SSH host key verification failures in Jenkins and establish stable and reliable continuous integration pipelines.