Test and Production Repositories
ACM.445 Finding injected code more easily by comparing test and production repositories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: AWS Security | Application Security | Batch Jobs
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post I explained how I am going to create credentials that a developer never has access to use.
I mentioned that I was going to rearrange my repository architecture to make it more flexible in a prior post.
The other thing I want to do is set up test and production repositories. I’ve written about this concept before. It’s a way to create a backup of a repository and a clearer picture of what has moved into production. It also provides a backup of your code in case one repository source becomes compromised — presuming you have separate credentials for each repository.
How separate repositories helped me pinpoint a compromise
When teaching a security class, I had a repository students could access and the original repository. During one of the classes an invisible character somehow got inserted into the student repository — but it wasn’t in the other student class repositories or in the original repository.
So I had a pretty good idea that one of the students in the class or someone who got onto the network used by the class may have somehow done that. I don’t think it was very malicious — possibly just to prove they could do it or somehow obstruct one of the labs in the class. I had other experiences demonstrating that someone was obstructing my classes. Perhaps they didn’t like that I was running very successful classes and taking business from other companies? No idea.
No matter. I made sure the students got working materials — one way or another. I dealt with each problem as it arose because I’m only one person, not the largest security training organization in the world with many resources. But I think I was able to lock things down fairly well in the end.
I’m just glad I had separate repositories when the character insertion occurred. That ensured that the blast radius was very limited. I could also see exactly where and when the character got inserted very easily.
Once I knew what happened, I changed my tactics when using repositories in class. I had updated some things in that repository on the fly. The students saw me do this and one made a comment on it. I also got a question about canaries. Hmm.
I imagine that the infiltration of the hidden character was one of the following:
- Leveraging a flaw in BitBucket to access the repo and insert the character.
- Compromising my computer in class when I updated the repository on the fly.
- Compromising the wifi network and inserting the character as the bits and bytes traversed the network.
Not sure exactly because I couldn’t look at all the logs from the conference center — if they even had them. I also use separate computers for public events and private work and wipe my public laptop after each event. Plus who wants to spend time on that.
Instead, I only used my own wifi at future events. However, wifi is never as secure as a hard-wired network no matter how you try to spin it. I stopped updating repositories during class. I added some additional checks which I hope to demonstrate in future posts to validate the integrity of my repositories. I can trigger alerts when things change.
I also stopped using Kali Linux which got somehow updated after I tested my lab at 2 a.m. and students got something different the next day. I started building my own AMI which I’ll show you how to do later hopefully.
These things are all related to supply chain attacks which I’m slated to speak about in Boston at an IANS event on the third week of November. Maybe I’ll be talking about this framework. We’ll see.
By the way, I don’t teach classes anymore but you can ask me questions by scheduling a call through IANS Research. I also help teach teams about security by performing a penetration test on cloud hosted applications (primarily AWS because I am an AWS Security Hero) and then explaining what I find in a report. I can also answer questions on the phone after I complete the report. I feel that is a better use of time than sitting in a class for a week for most people — if you share that report with your team and help them learn from it.
The benefit of separate repositories for different environments
If I had not had separate repositories — it might have been more difficult to prove what happened. If the attacker had access to everything — perhaps they wiped the history showing where they made the change or updated the history somehow. Who knows.
With my separate versions and histories, it was very easy to see the difference when comparing my development repository to the production repository and pinpoint exactly where and when the character was inserted.
My development repositories were locked down and used separate credentials and even a separate source control provider.
A lot of people use complex branching and other mechanisms but I find this separate repository method to be cleaner and it reduces the blast radius. It makes it easier to see production deployments because there’s a lot less noise in the version history in your production repository.
And there should not be any “merges” in your production repository. All merges should be completed before you promote your code to the production repository and in fact should be completed before your code goes to QA. There should only be one set of credentials that ever promotes code to the production repository through an automated process. In other words, merge conflicts should never exist.
Merges are where the evil and mistakes often sneak in. Merges should be not only be completed prior to promoting to production — you should also retest applications after any code merges.
In this post I’m going to explain a simple mechanism to use test and production repositories and to promote code. I’m using GitHub in this example for my test and production repositories.
In the future, I’ll likely use AWS CodeCommit, because unlike GitHub I can restrict it to a private network without running my own servers and enforce MFA on repository actions like get, commit, and push. I can also encrypt my code with a KMS key.
But there is some code that I want to make public so I will push that to GitHub. I will also backup some code to GitHub. That way if something disastrous happens to AWS I have a backup of that code in an alternate location.
I could also use this approach to back up code between AWS CodeCommit repositories in different regions if I didn’t want to use GitHub.
There are other things I will do to protect the code as well which I will write about as time allows and likely be speaking about in Boston.
Create the test and production repositories
First I created the test and production repositories. I used the names in this post for the production repositories:
For the test repositories, I added test- in front of those names.
I added the code in the rearranged format and with the new resources I mentioned in the prior posts and tested all the code.
Create a promote-github-repo job
Next I can create a promote-github-repo job using this framework to promote one repository to another.
This documentation from GitHub explains how to do that:
I can essentially copy the code in that post into the execute.sh file for my job.
I don’t need AWS credentials for this job but I need GitHub credentials. I am going to change a few things in my job execution framework to support GitHub commands.
I am not using GitHub actions for reasons explained in prior posts.
I tried to do the same thing to push from GitHub to CodeCommit in a prior post but had some issues that caused me to alter the code. I can create a separate job for this scenario.
I also want to be able to push code from AWS CodeCommit to GitHub so I’ll add a job for that as well.
All that will be added to the above repositories as I complete it. I just happen to be adding the repositories now so explaining what that is when it shows up for future reference. Also, stay tuned for a new repository location. 😊
Protecting credentials
Note that what I am NOT doing is putting my GitHub or AWS Access Keys into a configuration file or checking them into source control. I’m pulling them from AWS Secrets Manager.
.gitignore
Of course I will add a gitignore file to all these repositories:
gitignore
Of course I will add a .gitignore file to all these repositories.
Enforcing MFA — oh wait you can’t on GitHub
On the AWS side I can enforce MFA to run jobs. For a job purely run using GitHub credentials that is not possible at the time of this writing. Maybe if GitHub had MFA for actions such as git commit that would have helped prevent the rogue character insertion I wrote about above.
I showed how to do that with an AWS CodeCommit git repo here.
Creating a job to promote repositories
The obvious next step would be to create a job to clone repositories. Take the above code and copy it to an execute.sh file and use the job execution framework for that.
We’ll need to add a way to get the git credentials when the job runs if using Git. No problem. Put them in secrets manager and pull them out in the container where the logs should remain in the container and we should limit outbound access to GitHub with a security group that only allows access outbound to GitHub and a NAT to prevent credential leaks.
I explained how to create a prefix list for GitHub for an AWS Security Group here:
Hopefully that still works as it already changed on me once.
I have information on setting up a NAT with a private network in these posts — deployed with this framework moving forward.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2024
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab