Having problems using git-remote-codecommit on Lambda

Seems to be a common problem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Bugs | AWS Security | Secure Code

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

One of my stories on Git Security and AWS CodeCommit Security.

A lot of people are having problems using git-remote-codecommit including me.

The error messages are not very clear. I read some documentation that said that one of the errors could be a path issue or a syntax error. That’s pretty broad.

I wasted too much time just trying to rule out the path issue because I saw it in a bunch of places.

Now my command seems to match the AWS documentation. But perhaps I’ve been looking at this too long and not seeing something obvious.

I installed the tool on my container with a custom bash runtime.

I clearly had this tool working before in this post on requiring MFA for git actions with AWS CodeCommit:

So initially I just installed the tool using the default directory.

I ran my command from the above post got this error.

{ “errorMessage”: “An error occurred on line 55. Exit code 128. xxxx Cloning into ‘/tmp/dev.rainierrhododendrons.com’… fatal: ‘remote-codecommit’ appears to be a git command, but we were not able to execute it. Maybe git-remote-codecommit is broken?”, “error_type”: “invocation/xxxxxxxxxx” }

I thought maybe the problem was that I installed it as root and the user AWS uses for Lambda doesn’t have access to the root user directory.

I tried using the user directive but that doesn’t work because AWS changes the user when you deploy in Lambda.

pip install --user git-remote-codecommit

I tried installing to the tmp directory, changing permissions, and adding the directory to the path, which is git_remote_codecommit.

Now I get this error:

{ “errorMessage”: “An error occurred on line 55. Exit code 128. xxxxxxxx Cloning into ‘/tmp/dev.rainierrhododendrons.com’… git: ‘remote-codecommit’ is not a git command. See ‘git — help’.”, “error_type”: “invocation/xxxxxxxxxxx” }

I’m frustrated with this slowing me down because today’s post was supposed to be quick and simple. I’ll come back later and update this post when I figure it out.

It seems like a pip install in a Docker file based on Amazon Lambda custom runtime container should just work. Not sure what I’m doing wrong, if anything.

#### After walking the dog ####

Read this post:

Installed which in my Dockerfile:

RUN yum install which -y

Figure out where python is installed:

which python

Network errors. Again.

Could not retrieve mirrorlist http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list error was:

12: Timeout on http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list: (28, ‘Connection timeout after 5000 ms’)

But then it worked thankfully.

Ran which python2.7

Check path:

RUN echo $PATH

It’s already in the path. Plus I know that is not the issue because I looked at my local copy of this took and it’s using python 3.8.

I tried running the command locally on the container. It fails on no credentials. That’s expected. It seems like it’s getting past the command issue I get in Lambda.

I figured out that I need to specify no profile and no @ if trying to use the default profile by testing locally and attempting to set credentials. I also don’t need the region.

aws_repo="codecommit://$repo"

I definitely get a different error message when testing on Lambda versus locally so this seems like a Lambda issue, likely to do with the Lambda user and path. Let’s see what the path is in Lambda.

I’m adding the path to my error message in my errors.sh file. (Wrote about custom bash runtime error handling in prior posts.)

I can see the path is the same.

I check where the tool gets installed using which:

That tool is on the path. Let’s check the permissions.

Looks like it’s executable but it’s owned by root and in the root group.

Let’s compare that to the AWS CLI since we know that works in Lambda.

OK, well the permissions are different but AWS makes the Lambda OS read only except for the tmp directory. The location is different. What if we try to deploy into a different directory.

Back to this:

git: 'remote-codecommit' is not a git command. See 'git --help'.

At this point I’m annoyed. I wonder if I can just send commands to my Lambda function like this and execute them:

That didn’t work. It said command not found. Huh? So I print out the command. Looks ok.

I change my code to use the exec function:

That works:

OK, where is my executable.

Well that returns nothing.

So I run this (which I realize later had an extra quote in it but I was testing with the single quote because I was getting back empty strings…and anyway it didn’t work either way):

And it sys the command cannot be found. It doesn’t exist. Or it’s not on the path.

Can I install it (oops quote but it didn’t work anyway without the quote either…)?

I didn’t get an error with the command above initially.

I finally figure out that any commands with spaces don’t work…

One last command. whoami.

I suppose this user is different every time.

OK one more command.

I can run a one-liner env command to get the path:

PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib

Hmm.

Maybe we can push to one of the above folders to get this to work.

And now it’s time for dinner.

Wasting so much time trying to figure this out.

####Time out####

OK I’m back. I was rushing and misread something I think above.

I figured out spaces in commands were breaking due to my function and rather than fight with that I moved the minimal lines of code in line.

Now I can run commands with spaces. But variables don’t resolve.

echo $PATH turns into “$PATH”

Well I really just wanted to test installing to a location in the path.

Now, because I didn’t take a screenshot, I don’t know if something changed or I am delusional. But now. I am seeing two different paths when I run the env command:

LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib

PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin

Hmm.

Trying some other things and getting this error again:

No more mirrors to try again. If I just keep running it, then it eventually works. I don’t have room in my NACLs for more IP ranges. Wish AWS would fix this or maybe I’ll figure out that I’m doing something wrong.

So I reinstalled which again and searched for the git-remote-codecommit package and get this error:

no git-remote-codecommit in (/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin)

OK so next thing I did was go back to my container. I installed git-remote-codecommit to various folders and ran the which command in my container. The only one that worked and produced an executable on the path was the default install with no target directory.

Then the which command returns:

/usr/local/bin/git-remote-codecommit

When I leave it at the default and test that command in Lambda I get the same path. So the path does not appear to be the issue.

I’m back to this error:

remote-codecommit appears to be a git command, but we were not able to execute it. Maybe git-remote-codecommit is broken?

Maybe git-remote-codecommit is broken. I don’t know. I’m not in the mood to reverse engineer it. I really just want it to work.

Now I absolutely know which folder the file is in, that is in the path and that it is in the same folder as other executables the container is running. Everyone should have executable permissions for the file but I go ahead and run the following command to give everyone all permissions to see if that makes a difference (the same permissions the AWS CLI has with one difference):

chmod 777 /usr/local/bin/git-remote-codecommit

That did not make a difference and I don’t see why it would so I will revert it. The Lambda container should have a read-only file system.

Since I am pretty positive the path is not the issue. I start to scrutinize my command. No matter how I test it, I cannot get it to work when I pass it in as a parameter, but maybe the special characters are throwing it off.

I try running it directly in my container. Nope. Same error message. Pretty sure I’m running it how it says to run it in the documentation. Do I really need to reverse-engineer this thing to get it to work?

Later…

Docker:

Directory: rie

entry.sh

errors.sh

See link on error handling above.

rie-bash.sh

Directory: job

handler.sh

init.sh, secrets.sh, and validate.sh do not really matter for the purposes of this problem and were covered in prior posts in this series.

Well, after thinking this over for a bit, I spent way too much time on this already so I’m going to back to read-only git credentials in a secret like I wrote about here:

Update. I figured this out and will publish the solution in my next post and link to it here:

https://readmedium.com/git-and-github-security-8728bef0a057

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab