Client.InternalError: Client.InternalError: Client error on launch

Error messages that don’t tell you what the problem is

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Bugs | AWS Security | Secure Code

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is one of my most dreaded and annoying error messages.

Client.InternalError: Client.InternalError: Client error on launch

I know from past experience what it usually refers to, but the message is not at all helpful. Are you trying to be cryptic, AWS EC2 programmers? People will eventually figure it out via a Google search, but I’ve spent hours of my life on this error message over the years when I forget what it was the last time as I’m always jumping around on projects.

I’m guessing this has to do with the KMS key I passed in to encrypt the AMI. The role that is running my Packer script doesn’t have access to the key to encrypt the Amazon Machine Image I’m trying to build encrypted.

It also occurs when you encrypt and try to share an AMI with another account and that account does not have permissions.

Let’s check that hypothesis.

First of all I need to take a look at the key policy associated with the key and see who is allowed to use it. I presume Packer packer needs to only encrypt on first run, but I build AMIs in stages. The next stage that uses my base AMI will also need to decrypt. I probably don’t need quite this many permissions but I added the following to the end of my KMS key policy (fix the role name to match your own).

, {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::xxxxxxxxxxxx:role/xxxxx-role"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::xxxxxxxxxxxx:role/xxxxx-role"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }

I should probably add an external ID to that role. Additionally, I need to check that the role that is assigned to my ec2-instance has permissions to use KMS as well. I already know it does, so I’m good there.

And…success. Finally, once again, we’re building AMIs with Packer. Some of the other errors in this blog took way too long to resolve as explained — and this is from someone who’s been programming since 1994 (not even counting programming as a kid back in 1980). If it’s hard for me to figure these things out consider how a beginning programmer feels.

This also happens in the EC2 dashboard. An instance shuts down immediately with this cryptic error message. Imagine dealing with 11,000 developers asking you questions like I did at Capital One and they all want to know why their instances won’t run. They all spend time googling or asking the cloud team questions when they try to do things they are not allowed to do. Tons of wasted time over my life due to this cryptic error message.

Fix: Make your error messages better so people can resolve issues in a timely manner!

Fix: Explain WHICH key need access to as well in the error message if it is viewable on the AMI anyway.

You can also get this error when using CloudFormation and it may be a different cause that is very tricky to troubleshoot:

SSO works differently:

Another resource:

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab