Removing Sensitive Files from GitHub and All Their History

Errors trying to run git filter-repo — invert-paths

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: Git | GitHub Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I’ve been working this series of blog post and noticed that some .swp files got into my GitHub repo. I wanted to remove them. When you simply delete a file and check in your code again, the file is not completely gone. It still exists in the history and can be retrieved.

If you want to see files that have been deleted from git:

git log --diff-filter=D --summary

You can retrieve deleted files (I didn’t try these commands so use at your own risk — make a backup first):

Git: How to find a deleted file in the project commit history?

Use the following command If you know the path the file was at: This command should display a list of commits in all…

intellipaat.com

Let’s say you see a file that you want to completely remove. Github has a page about this:

Removing sensitive data from a repository - GitHub Docs

If you commit sensitive data, such as a password or SSH key into a Git repository, you can remove it from the history…

docs.github.com

I used the filter-repo option. First I got an error that filter repo didn’t exist. I’m running some updates on an Amazon Linux EC2 instance so I had to install filter-repo.

python -m pip install --user git-filter-repo

I got a bunch of errors after trying to run the command to remove the file. I ended up running the commands below but not sure what is actually necessary because worked because I didn’t track what I was doing very well:

#backup local repo
mv [my repo folder] [backup file name]

#start with a clean repo for best results
git clone https://github.com/[your github repo].git

#remove the unwanted file
git filter-repo --invert-paths --path iam/batch_job_role/.deploy.sh.swp

#add git ignore file
echo *.swp >> .gitignore
git commit -m "Add .gitignore"

#force an overwrite of the remote directory
git push origin --force --all

#for some reason my origin disappeared
git remote add origin https://github.com/[repo].git

#I couldn't push without running the following commands first
git add .
git commit -m "add git ignore"
git config --global user.name [your username here]
git push --set-upstream origin main

#at some point I had to run this command:
git config pull.rebase true

Then I removed another file and didn’t seem to get all the errors again. But then I got this error if I simply try to run git pull:

There is no tracking information for the current branch.

Please specify which branch you want to rebase against.

See git-pull(1) for details.

git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

git branch --set-upstream-to=origin/<branch> main

Some combination of the following resolved the problem:

git remote add origin https://github.com/[repo].git

git push origin --force --all

git config pull.rebase true

git remote add origin no@spam.com/[your username here]/[repo].git

git push origin --force --all

git remote add origin https://github.com/[username]/[repo].git

git config --global user.name [username]

git push --set-upstream origin main

Follow for updates.

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab