Cross-Repository File Migration in Git: Preserving Complete History

Technical Challenges in Git Cross-Repository Migration

In distributed version control systems, cross-repository file migration represents a common yet complex requirement. Unlike centralized systems such as SVN, Git's distributed nature necessitates special handling for file transfers. When developers need to move specific files or directories from one Git repository to another while preserving complete commit history, the primary challenge lies in Git's commit hash calculation mechanism.

The Necessity of History Rewriting

Git's core design principles dictate that cross-repository file migration must involve history rewriting. Each Git commit generates a unique identifier based on its content and parent commit hashes. When extracting only partial files from a repository, the new commit tree structure differs completely from the original commits, inevitably causing changes to all relevant commit hashes. Consequently, standard Git operations like pull or merge cannot directly accomplish this task since they do not rewrite historical records.

Core Role of the filter-branch Command

The git filter-branch command serves as the crucial tool for cross-repository migration, particularly its --subdirectory-filter parameter. This parameter can rewrite Git history, retaining only files from specified subdirectories and elevating that directory to become the new repository root. This process involves the following key steps:

# Clone source repository
$ git clone project2
$ cd project2

# Rewrite history using subdirectory-filter
$ git filter-branch --subdirectory-filter deeply/buried/java/source/directory/A -- --all

# Clean up original remote references
$ git remote rm origin

After executing these commands, Git traverses all commits, preserving only files from the specified path and regenerating corresponding commit history. This process ensures that only target files and their relevant history are retained, while other unrelated content is completely removed.

Directory Structure Adjustment and File Relocation

Following history rewriting, files typically need relocation to appropriate locations within the target repository. This involves creating new directory structures and moving files using the git mv command:

# Create target directory structure
$ mkdir -p deeply/buried/different/java/source/directory/B

# Batch move files to new directory
$ for f in *.java; do
>   git mv $f deeply/buried/different/java/source/directory/B
> done

# Commit directory structure adjustments
$ git commit -m "moved files to new subdirectory"

This step ensures that file organization within the target repository meets project requirements while maintaining historical continuity.

Target Repository Integration and Merging

Merging processed source repository content into the target repository constitutes the final phase of the migration process. This requires adding remote references and branch merging:

# Clone target repository
$ git clone project1
$ cd project1

# Add processed source repository as remote reference
$ git remote add p2 ../project2
$ git fetch p2

# Create and merge branches
$ git branch p2 remotes/p2/master
$ git merge p2 # --allow-unrelated-histories may be required for Git 2.9+

# Clean up temporary remote references and push changes
$ git remote rm p2
$ git push

The --allow-unrelated-histories parameter becomes necessary because source and target repositories originally lack common ancestor commits. This parameter permits Git to merge two completely independent commit histories.

Process Optimization and Improvements

While the aforementioned method effectively accomplishes the task, practical applications may consider the following optimizations:

First, employing git filter-repo as a modern alternative to filter-branch. Compared to traditional filter-branch, filter-repo offers better performance and more concise syntax:

# Use filter-repo for file filtering
$ git filter-repo --path ansible/k3s-deploy.yml --path ansible/file.yml --refs refs/heads/filter-source

Second, for simple file migrations, consider using the patch approach. Generate patch files via git log, then apply them in the target repository using git am:

# Generate patch file
$ git log --pretty=email --patch-with-stat --reverse --full-index --binary -m --first-parent -- path/to/file_or_folder > patch

# Apply patch in target repository
$ git am --committer-date-is-author-date < patch

Important Considerations and Best Practices

When performing cross-repository migrations, pay special attention to the following aspects:

Operation Safety: All history rewriting operations should occur on repository copies to avoid irreversible impacts on original repositories. Creating complete backups before commencing operations serves as essential precautionary measures.

Team Collaboration Considerations: If migrations involve repositories currently used by teams, notify all relevant members in advance and execute migration operations at appropriate times to avoid conflicts with other developers' work.

Performance Optimization: For large repositories, history rewriting operations may require significant time. Consider executing during non-working hours or using git gc --aggressive and git prune to optimize repository performance.

Conclusion and Future Perspectives

Although Git cross-repository file migration involves complex processes, proper utilization of tools like filter-branch or filter-repo can effectively complete tasks while preserving complete historical records. Understanding Git's underlying mechanisms proves crucial for successfully executing such operations. As Git tools continue evolving, more simplified solutions may emerge in the future, but current methods remain reliable choices for handling such requirements.

In practical projects, recommend selecting the most appropriate migration strategy based on specific needs. For simple file movements, the patch method may prove more efficient; for complex directory structure adjustments, complete history rewriting becomes necessary. Regardless of chosen method, thorough testing and validation constitute key elements for ensuring successful migration.