Analysis of Stuck Jobs in GitLab CI/CD: Runner Tag Configuration and Solutions

Problem Background and Symptom Analysis

In learning and practicing GitLab CI/CD, many developers encounter stuck job issues, especially when Runners are registered and associated with projects but jobs fail to execute. Based on the provided Q&A data, after installing GitLab and GitLab Runner from official sources, the user experiences jobs getting stuck during Maven build pipelines. The error message clearly states: "This job is stuck, because the project doesn't have any runners online assigned to it." This indicates that although Runners are online and available, there is a mismatch in the assignment mechanism between jobs and Runners.

Core Issue: Matching Runner Tags and Job Tags

GitLab CI/CD manages the matching of Runners and jobs through a tagging system. Runners can be configured with one or more tags, and jobs can specify tags in the .gitlab-ci.yml file. A Runner will only execute a job if the job's tags match the Runner's tags. If a Runner has tags configured but a job has no tags specified, or if the job's tags do not match the Runner's tags, the job becomes stuck as the system cannot find a suitable Runner to execute it.

In the user's case, the Runner has tags set (as visible from the images), but the job definitions lack any tags. This prevents jobs from being assigned to the Runner, leading to the stuck state. Below is a simplified example from the user's .gitlab-ci.yml file, showing the absence of tags in job definitions:

image: docker:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay
  SPRING_PROFILES_ACTIVE: gitlab-ci

stages:
  - build
  - package
  - deploy

maven-build:
  image: maven:3-jdk-8
  stage: build
  script: "mvn package -B"
  artifacts:
    paths:
      - target/*.jar

docker-build:
  stage: package
  script:
    - docker build -t registry.com/ci-cd-demo .
    - docker push registry.com/ci-cd-demo

k8s-deploy:
  image: google/cloud-sdk
  stage: deploy
  script:
    - echo "$GOOGLE_KEY" > key.json
    - gcloud container clusters get-credentials standard-cluster-demo --zone us-east1-c --project ascendant-study-222206
    - kubectl apply -f deployment.yml

As shown in the code, jobs maven-build, docker-build, and k8s-deploy do not define a tags field, while the Runner might have specific tags configured (e.g., docker or maven), resulting in a mismatch.

Solution 1: Modify Runner Settings to Allow Untagged Jobs

According to the best answer, the first solution is to adjust the Runner configuration to enable it to run jobs without tags. This can be done through the GitLab admin interface:

1. Navigate to the Runner settings page and find the "Runner settings" section.
2. Uncheck or ensure the "Run untagged jobs" option is enabled (if already checked, confirm it allows untagged jobs). In the provided image examples, this typically involves setting the Runner to allow "Run untagged jobs."
3. After saving the settings, the Runner will be able to execute any job, regardless of whether the job has tags, thus resolving the stuck issue.

This approach is suitable for scenarios where simplification is desired, but it may reduce the Runner's specificity, as all jobs could be assigned to it.

Solution 2: Add Matching Tags to Jobs

Another more recommended method is to maintain the Runner's tag configuration and add corresponding tags to jobs in the .gitlab-ci.yml file. This ensures jobs are only executed by Runners with specific capabilities, improving resource utilization and security. Below is how to modify the user's code example to add tags to jobs:

maven-build:
  image: maven:3-jdk-8
  stage: build
  script: "mvn package -B"
  tags:
    - maven  # Assuming Runner tags include maven
  artifacts:
    paths:
      - target/*.jar

docker-build:
  stage: package
  script:
    - docker build -t registry.com/ci-cd-demo .
    - docker push registry.com/ci-cd-demo
  tags:
    - docker  # Assuming Runner tags include docker

k8s-deploy:
  image: google/cloud-sdk
  stage: deploy
  script:
    - echo "$GOOGLE_KEY" > key.json
    - gcloud container clusters get-credentials standard-cluster-demo --zone us-east1-c --project ascendant-study-222206
    - kubectl apply -f deployment.yml
  tags:
    - kubernetes  # Assuming Runner tags include kubernetes

By adding the tags field, jobs will match with Runners having the corresponding tags, preventing stuck states. Developers should configure tags based on the actual Runner tags, e.g., if Runner tags are java and docker, job tags should be set to these values.

In-Depth Analysis and Best Practices

The tagging mechanism plays a crucial role in GitLab CI/CD, not only for matching Runners and jobs but also for environment isolation and resource optimization. For instance, different Runners can be set with tags like production, testing, or development to ensure jobs run in the correct environment. Additionally, tags can be based on hardware features (e.g., gpu) or software versions (e.g., python-3.8).

To avoid common pitfalls, it is recommended that developers follow these steps when configuring CI/CD pipelines:

1. Check Runner tag settings: Verify the list of tags for Runners in the GitLab Runner management page.
2. Add matching tags to jobs in .gitlab-ci.yml: Use the tags field to specify one or more tags, ensuring consistency with Runner tags.
3. Test the configuration: Run the pipeline and monitor job status; if stuck issues persist, check tag spelling or Runner availability.
4. Refer to official documentation: GitLab provides detailed guides on tag configuration to help developers deepen their understanding.

By correctly configuring tags, developers can leverage the flexibility of GitLab CI/CD to enhance the efficiency and reliability of automated workflows.