Jenkins Job Execution Issues: Comprehensive Analysis and Solutions from Disk Space to Executor Configuration

Problem Phenomenon and Background Analysis

In continuous integration environments, Jenkins jobs remaining in "pending - waiting for next executor" state represent a common operational challenge. Users report that even after disabling all slave nodes, simple jobs fail to execute on the master node. This phenomenon typically indicates fundamental issues with system resources or configuration.

Core Diagnostic Procedure

For jobs waiting for executors, we recommend following this systematic diagnostic process:

First, navigate to the Jenkins -> Manage Jenkins -> Manage Nodes interface to check the master node's online status. If the master node shows as offline, it's likely that disk space exhaustion has triggered automatic shutdown mechanisms.

Disk space verification can be performed using the following commands on the Jenkins server:

df -h /var/lib/jenkins  # Check disk usage for Jenkins workspace
du -sh /var/lib/jenkins/jobs/*  # Analyze space consumption per job directory

Executor Configuration Validation

Beyond disk space issues, improper executor count configuration represents another common cause. Verify executor settings through these steps:

Navigate to Manage Jenkins -> Configure System and locate the "# of executors" parameter in global configuration. By default, Jenkins configures 2 executors, but this might be accidentally set to 0 in some scenarios.

Sample code for verifying executor configuration:

// Check executor configuration via Jenkins API
import jenkins.model.Jenkins

def jenkins = Jenkins.instance
def computer = jenkins.computers[0]
println "Available executors: ${computer.numExecutors}"
println "Busy executors: ${computer.countBusy()}"

Node Label Matching Issues

In Pipeline jobs, incorrect node label configuration prevents jobs from finding suitable executors. For example, specifying non-existent node labels in Jenkinsfile:

Incorrect configuration example:

node('node') {
    // Job steps
}

Correct configuration should specify actual existing nodes:

node('master') {
    // Job steps
}

System Resource Monitoring and Optimization

Establishing continuous resource monitoring mechanisms is crucial. We recommend implementing automated disk space checking scripts:

#!/bin/bash
JENKINS_HOME="/var/lib/jenkins"
THRESHOLD=90  # Disk usage threshold

current_usage=$(df "$JENKINS_HOME" | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$current_usage" -ge "$THRESHOLD" ]; then
    echo "Warning: Jenkins disk usage ${current_usage}% exceeds threshold"
    # Trigger cleanup operations or alerts
fi

Comprehensive Solution Framework

Based on problem analysis, we recommend a layered solution approach:

1. Emergency handling: Immediately check disk space and clean unnecessary build histories and artifacts

2. Configuration validation: Confirm executor count settings and node label matching

3. Preventive measures: Establish regular maintenance procedures and monitoring alert mechanisms

Through systematic diagnosis and optimization, Jenkins job execution issues can be effectively resolved, ensuring stable operation of continuous integration environments.