Resolving GitHub Push Failures: Dealing with Large Files Already Deleted from Git History

Problem Background and Phenomenon Analysis

In the usage of distributed version control system Git, developers frequently encounter a perplexing issue: even after deleting large files from the local repository, they still receive file size limit errors when pushing to GitHub. The fundamental cause of this phenomenon lies in Git's design characteristics—Git not only tracks the current working directory state but also completely preserves the entire project history.

When users execute the git rm --cached <filename> command, this only removes the file from the current staging area and working directory, but the file's historical records in previous commits remain intact. GitHub checks the entire commit history chain, including file contents in all ancestor commits, when receiving pushes. Therefore, even if the large file doesn't exist in the current commit, as long as the historical records contain files exceeding 100MB, the push operation will fail.

Limitations of Traditional Solutions

Many developers first attempt to use the git filter-branch command to clean historical records. The basic syntax of this command is:

git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch <file/dir>' HEAD

This command can indeed rewrite commit history and remove specified files from all relevant commits. However, git filter-branch has several serious drawbacks: First, it generates numerous temporary files that may exhaust disk space when processing large repositories; Second, the rewriting process is extremely slow, particularly for projects with extensive history; Most importantly, this command changes the SHA-1 hash values of commits, which may break branch references, tags, and other developers' local copies.

Git official documentation now explicitly warns developers to avoid using git filter-branch and recommends using more modern alternatives.

Recommended Solution: git filter-repo

git filter-repo is a Python tool specifically designed for safe and efficient Git history rewriting. Compared to git filter-branch, it offers better performance, cleaner interfaces, and more reliable results.

Installing git filter-repo can be achieved through various package managers:

# Using Homebrew (macOS)
brew install git-filter-repo

# Using pip
pip3 install git-filter-repo

# Other package managers
# Refer to specific system installation guides

The basic command format for removing specific file paths using git filter-repo is:

# Remove files with specified path
git filter-repo --path example/path/to/something --invert-paths

# Remove all files except those with specified path
git filter-repo --path example/path/to/something

In practical operation, if the target file is fpss.tar.gz, the corresponding cleanup command should be:

git filter-repo --path fpss.tar.gz --invert-paths

This command scans the entire commit history, permanently removing the specified file from all commits that contain it, while preserving other file contents unchanged. After completing the cleanup, force push to all remote repositories is required:

git push origin --force --all
git push origin --force --tags

Detailed Operational Process

A complete solution should follow this systematic procedure:

1. Preparation: Ensure all important changes are committed, create complete backups of the repository, and notify all collaborators to suspend operations.
2. Problem Diagnosis: Use git log --oneline --name-only to identify which commits contain large files and confirm the specific scope of the issue.
3. Execute Cleanup: Run the git filter-repo command, specifying the file paths to be removed. The tool automatically handles all dependencies and history rewriting.
4. Verify Results: Use git log --oneline to check if the history remains intact and confirm that large files have disappeared from all commits.
5. Update Remote Repository: Since the history has been rewritten, you must use the --force option when pushing changes. This overwrites the existing history in the remote repository.
6. Notify Team: Inform all collaborators that they need to re-clone the repository or reset their local branches, as the original local history is incompatible with the new history.

Alternative Approach Comparison

Besides git filter-repo, developers sometimes consider other methods:

Commit Squashing: This approach uses git reset --soft HEAD~N to revert to an earlier commit, then combines multiple commits into one. While this can eliminate large files from intermediate commits, it cannot completely solve the problem if large files appear in the earliest commits.

Simple Reset: As described in reference articles, using git reset HEAD~2 to revert commits and then recommit. This method only works for simple cases where large files appear in recent commits and has limited effectiveness for complex historical structures.

In comparison, git filter-repo provides the most thorough and reliable solution, capable of handling arbitrarily complex historical structures.

Best Practices and Preventive Measures

To avoid recurrence of similar issues, the following preventive measures are recommended:

• Always check file sizes before using git add, particularly for binary files, compressed archives, and datasets.
• Configure .gitignore files to automatically exclude common large file types such as *.tar.gz, *.zip, *.pdf, etc.
• For large files that must be version-controlled, consider using Git LFS (Large File Storage) extension, which is GitHub's officially recommended solution for large file management.
• Establish team code review processes to check for unnecessary large files before merging requests.
• Regularly use git gc to clean the repository and optimize storage efficiency.

Conclusion

The persistence of large files in Git history stems from Git's complete version tracking mechanism. While traditional git filter-branch can solve the problem, it carries performance and security risks. git filter-repo, as a modern alternative, provides more efficient and secure history rewriting capabilities. Through systematic operational procedures and appropriate preventive measures, developers can thoroughly resolve push failures caused by large files while maintaining healthy code repository status.

It's important to recognize that any history rewriting operation is destructive and should be performed cautiously with full understanding of the consequences. In team collaboration environments, adequate communication and coordination are crucial factors for successfully implementing these operations.