Practices and Optimization for Checking Out Multiple Git Repositories into Subdirectories in Jenkins Pipeline

Introduction: Migration Challenges from Multiple SCM to Pipeline

In the practice of continuous integration and continuous deployment (CI/CD), Jenkins, as a widely used automation server, had its Multiple SCM plugin as a common tool for multi-repository management. However, with the plugin marked as deprecated, developers are forced to transition to pipeline methods, introducing new technical challenges. This article is based on a typical scenario: a user needs to check out three Git repositories (CalibrationResults, Combination, CombinationBuilder) into three subdirectories of the same Jenkins job to build unified artifacts. The initial attempt used shell commands to switch directories and execute git operations but failed because the git step executes by default at the workspace root, lacking subdirectory targeting functionality.

Core Solution 1: Using the dir Command for Subdirectory Checkout

The dir command is a key instruction in Jenkins pipeline that allows executing a series of steps in a specified subdirectory. This method is concise and intuitive, particularly suitable for quick migration and simple scenarios. Below is an optimized code example:

node('ATLAS && Linux') {
    dir('CalibrationResults') {
        git url: 'https://github.com/AtlasBID/CalibrationResults.git'
    }
    dir('Combination') {
        git url: 'https://github.com/AtlasBID/Combination.git'
    }
    dir('CombinationBuilder') {
        git url: 'https://github.com/AtlasBID/CombinationBuilder.git'
    }
    sh('ls')
    sh('. CombinationBuilder/build.sh')
}

In this example, the dir command creates three independent subdirectory contexts, each executing a git checkout operation. This avoids the complexity of manual shell commands (e.g., cd) and ensures step atomicity. Advantages include high code readability and ease of debugging, but drawbacks may require additional handling if directories already exist or paths are incorrect.

Core Solution 2: Leveraging the RelativeTargetDirectory Extension of the checkout Step

Another approach is to use the extension functionality of the checkout step, specifying the target directory directly via the RelativeTargetDirectory parameter. This method offers finer-grained control, suitable for complex configuration scenarios. Below is an improved example based on Answer 2:

stage('Checkout') {
    checkout([
        $class: 'GitSCM',
        branches: [[name: 'refs/heads/master']],
        extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'CalibrationResults']],
        userRemoteConfigs: [[url: 'https://github.com/AtlasBID/CalibrationResults.git']]
    ])
    checkout([
        $class: 'GitSCM',
        branches: [[name: 'refs/heads/master']],
        extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'Combination']],
        userRemoteConfigs: [[url: 'https://github.com/AtlasBID/Combination.git']]
    ])
    checkout([
        $class: 'GitSCM',
        branches: [[name: 'refs/heads/master']],
        extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'CombinationBuilder']],
        userRemoteConfigs: [[url: 'https://github.com/AtlasBID/CombinationBuilder.git']]
    ])
}

This method is implemented via Jenkins' GitSCM plugin, where the RelativeTargetDirectory extension allows setting the target path directly during checkout without additional directory switching. It supports more complex configurations such as branch management, credential integration, and submodule handling, but the code is slightly verbose. In practice, it can be optimized with conditional logic or loops to reduce repetition.

Solution Comparison and Best Practice Recommendations

Comparing the two solutions, the dir command is more suitable for simple, quick scenarios, while the checkout step extension method is better for environments requiring advanced configurations (e.g., multi-branch, security credentials). Based on Answer 1's score (10.0) and acceptance, the dir command is the preferred solution due to its ease of use and alignment with pipeline style. However, in large-scale projects, the checkout method may offer better maintainability.

Best practices include: always defining the execution environment within a node block to ensure resource isolation; using stages to organize steps for improved pipeline readability; considering error handling, such as try-catch blocks or retry mechanisms; and for multi-repository scenarios, encapsulating logic into functions or using shared libraries to reduce code duplication. For example, a generic function can be created to handle checkout logic:

def checkoutToDir(repoUrl, dirName) {
    dir(dirName) {
        git url: repoUrl
    }
}
node('ATLAS && Linux') {
    checkoutToDir('https://github.com/AtlasBID/CalibrationResults.git', 'CalibrationResults')
    checkoutToDir('https://github.com/AtlasBID/Combination.git', 'Combination')
    checkoutToDir('https://github.com/AtlasBID/CombinationBuilder.git', 'CombinationBuilder')
    sh('ls')
    sh('. CombinationBuilder/build.sh')
}

Conclusion and Future Outlook

Migrating to Jenkins pipeline brings modernization and flexibility to multi-Git repository management. Through the dir command or checkout extensions, developers can efficiently achieve subdirectory checkouts, overcoming the limitations of the Multiple SCM plugin. The solutions provided in this article are based on actual Q&A data, emphasizing the importance of code refactoring and best practices. As DevOps tools evolve, it is recommended to stay updated with Jenkins official documentation and community updates to adapt to new features like declarative pipelines or cloud-native integrations. Ultimately, the choice of method should be based on project requirements, team skills, and long-term maintenance considerations.