Cloning a GitHub Repo to AWS CodeCommit
ACM.322 First Draft — this works but don’t do it — We’ll make adjustments in future posts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: GitHub Security | AWS Security | Appsec | Secure Code
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post I was trying to write this one when a yum issue popped up. I was pondering if there’s a way to get yum updates from private IP addresses among other things.
In this post I will attempt to clone a repo from GitHub to AWS CodeCommit. Mind you, this is not my final solution. This is the initial solution that “works.” In the next post I finally hope to get to something I’ve been planning to write about since almost the beginning of this series but there are so many moving parts — MFA to run a batch job.
Recall that I already covered the following in prior posts:
* A Lambda function that runs a container.
* An ECR repository to store the container used by Lambda.
* An ECR policy to allow Lambda to access ECR.
* A Bash custom Lambda runtime that runs in the container.
* Error handling for our custom runtime.
* Credentials for our Lambda container locally outside of AWS Lambda.
* A private network with access through a NAT.
* A Lambda execution role with appropriate permissions.
* VPC Endpoints to access AWS Services including CodeCommit.
* VPC Endpoints that grant access to our Lambda execution role.
* A Service Control Policy to limit Lambda invocation to a private network.
* A KMS Key for our sandbox environment with appropriate policies for Lambda.
* A GitHub personal access token with limited access.
* Network restrictions to access the GitHub private repositories.
* A security group using a customer managed prefix list to access GitHub.If you missed any of that the posts are listed here:
Now that we have all that it should be easier to create new Lambda functions because we can reuse all those resources and focus on deploying our containers and code. We still need to test and streamline further but we’re getting there.
Securing your home network
I also explained how you can further secure your home network if you are working from home with a personal firewall like pfSense.
And possibly a better wifi product than the consumer grade products such as a Ubiquiti. The Ubiquiti still has some issues but seems to be more robust than the following wifi devices I’ve tested or looked into: Netgear, OPNSense, Google Wifi, or Eero. Still haven’t found what I’m looking for in the words of U2, but this is the best one I’ve tried yet, as long as it’s behind a pfSense. See some of the issues in my posts.
All these layers of defenses help protect your credentials as you work with GitHub and AWS credentials. They work to limit the chances an attacker can get to your credentials in a scenario like the LastPass breach as I wrote about in the prior post.
Initially hardcoded values to simplify testing
To reduce confusion, I’m going to initially hardcode my commands with specific values. Once I know the commands work and all the proper access exists, I will replace the hardcoded credentials and values with the AWS Secrets Manager values.
Mirroring a GitHub repository
I’m going to use a mirror command to mirror one repository to another. When we move them we want to use the -bare flag to get a copy of the repository without the repository-specific files. Then we can push the files up to the new repository which will have its own git-specific files.
To maintain a mirror of a repository without forking it, you can run a special clone command, then mirror-push to the new repository.
We’ll need to make a few changes to the above commands to get this to work in our Lambda container.
Recall that I’m working with this repository for testing, which is a copy of a public website, so nothing sensitive at this point. I wouldn’t do some of the things below if I was using any sensitive code.
dev.rainierrhododendrons.comStep 1: Clone the repository you want to mirror with the bare flag
The first thing we need to do is clone the repository we want to mirror with the bare flag. As mentioned this excludes any repository-specific files.
git clone --bare https://github.com/EXAMPLE-USER/OLD-REPOSITORY.gitWhen I ran the command above as is I got this error:
fatal: could not read Username for ‘https://github.com': No such device or address
That’s expected. We need to set the credentials for this command to the personal access token we stored in AWS Secrets Manager. For my initial cut — only because I am not exposing anything that is not already public with this credentials — I am going to show you what *NOT* to do.
Eventually I’m going to use variables for some of the values so for the first test my code looks like this:
The repo_owner is the name of the owner of the GitHub repository.
For example, if you look at my SecurityMetricsAutomation repository, the owner name is tradichel. Look at the URL for whatever repository you are trying to clone to find the owner name as shown below.
I also hardcoded the personal access token (pat) only because there is absolutely nothing an attacker could get with this token that they could not get publicly. I’m showing you what you should *NOT* do. Do not hardcode tokens the way I’m doing it in this initial test. That is how attackers steal credentials. They get access to your code or environment variables or logs and find the credentials and can reuse them.
I’m going to show you how I instead pull the token from AWS Secrets Manager below. We will pull them just as we need them and then set them to empty values when we are done to make sure they are not hanging around in memory longer than needed.
In the case of a Lambda function you need to understand what persists between Lambda invocations. In the case of a container, anything you hardcode into the container may persist after you delete it in your Dockerfile depending on how the container was built. More on that later.
I don’t love that method I’m using for leveraging git credentials in a URL and in most cases I would not recommend it.
git clone '@github.com/'$repo_owner'/'$repo_folder">https://'$pat'@github.com/'$repousername'/'$repo_folderWhy?
Where are all the places that token is going to show up in logs? If I run this in a Lambda function will the token get output into any Lambda console logs or output? We’ll find out. Right now I’m testing locally.
What if we configure the credentials in the container? Do those persist between invocations?
Which method provides more potential access to the credentials and which should we use?
For now I’m using the URL method because of all the steps I’ve taken above to secure the Lambda and container environment, and because I am not accessing anything in GitHub that is not already public. Once I have a chance to review the logs and output I might take a different approach.
Validating the network allows me to access GitHub
At this point I had to make sure my developer EC2 instance has access to GitHub. I have other posts coming on that and some automation for a developer VPC which I’ll add to my network diagrams in the future. I also already showed you how to set this up for your Lambda environment.
But in case you have any issues, remember how I showed you how to troubleshoot network issues, if you have them.
Validating the function can access AWS CodeCommit
Next I have an error related to accessing AWS Code Commit.
Looks like my local environment cannot access CodeCommit. Here’s my code:
I added a VPC Endpoint for my Lambda function in the Lambda networking I provided in an earlier post. I thought I also added one to my development environment.
Ah, yes. I’ve set up my VCP Endpoint to use the FIPS option (I explained what that is previously.) My code above is connecting to the standard AWS CodeCommit URL.
We need to use the AWS CodeCommit FIPS endpoints:
The interesting thing about the above error is that if you are trying to be secure and use FIPS compliant access to GitHub but you use the wrong URL, AND your network is wide open, your commands may work but be traversing the Internet. I haven’t tested it but presume that would be true if the VPC had a public route.
That fixed that issue.
Next error is that I haven’t created the repository in AWS CodeCommit yet.
We can create that respository with CloudFormation:
However, I’ll save that for another post. For this test I manually created a repository named dev.rainierrhododendrons.com in AWS CodeCommit since this is already taking quite some time.
The interesting thing is that the UI doesn’t mention the FIPS compliant option:
The next problem is that I’ve never cloned this repository locally. The folder I’m in is not a git repository folder.
Well, this mirror command isn’t working for me. I don’t know if it is because AWS does not support the mirror command or what. But before I ever knew this mirror command existed I used to clone repositories between different cloud providers so I’ll just revert to that method. It’s not too different.
Here’s where I realize a basic clone command for AWS CodeCommit is not working. My function just hangs eternally which I presume is a network problem. So I’m going to save the long explanation of what happened next for a separate post.
Just understand that at this point I switched back to a non-FIPS AWS CodeCommit VPC Endpoint.
Finally I’m past the issue and getting a username error for AWS CodeCommit.
{“errorMessage” : “An error occurred on line 55. Exit code 128. 8aa1488f-75b8–449d-a21e-e48a07948727 Cloning into ‘dev.rainierrhododendrons.com’… fatal: could not read Username for ‘https://git-codecommit.us-east-2.amazonaws.com': No such device or address”, “error_type” : “invocation/8aa1488f-75b8–449d-a21e-e48a07948727”}
Ok, I went over how to handle credentials for AWS CodeCommit in another post. You have options. Where was that…searching google for topics plus my last name. Ah yes.
First I showed you how you can use a standard username and password:
Then I showed you how you can enforce MFA using a different method:
Let’s start with the less secure method with user credentials floating around. We can use this method but yuck hardcoded credentials. At the very least we’d want to put them in Secrets Manager. But we’ll do it a better way long term if I can get it to work.
Also, note that when you put the username and password that you cannot generate yourself into the URL you have to encode any special characters. For example, this password had an equal sign at the and I had to encode it for the URL as %3D. If that URL had an = sign in it you’d get an error message about an invalid URL or something to that effect.
You can find information on which characters you need to encode and the corresponding values here:
If you really want to get technical as to what this encoding all means, check out my series on Cybersecurity Math:
In the end I show you how to convert all these values around when evaluating network packets and the same conversions apply to URLs and other places where you have to encode characters.
OK with this method and all the above fixes I was able to get my AWS CodeCommit repository to clone properly.
Let’s try that mirror function again.
And…it works.
Here’s what our very insecure and ugly code looks like — I repeat — do not do this. By the way, I gave my temporary git user created for this post AWS CodeCommit power user access. We can be much more specific and limit to only specific repositories as well if we need to, or simply put different repositories with different trust boundaries in separate accounts.
Once again various access issues and errors slowed me down turning what I thought would be an insanely simple post into something longer. I’ll clean this up in the upcoming posts, and we’ll automate the creation of the CodeCommit repository and user access.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab