Instance i-xxxxxxxxx failed to stabilize. Current state: shutting-down. Reason: Client.InternalError: Client error on launch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics | Code.
🔒 Related Stories: Bugs | AWS Security | Secure Code
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Usually no access to KMS Key causes this error. I already wrote about this error and what to do about it if it’s a KMS Key issue in another post. This is just another variation.
This is one other tricky case which cost me loads and loads of time. I think I am getting this error again so trying to document it a bit better because it took hours of searching to find the answer last time it occurred — maybe days or a week. Here’s what my notes say at the top of a packer build script:
# OMG OMG spent way too long on this: the volume name you add to an
# ami cannot overlap with the root volume name and AWS
# doesn't block from doing that when creating an image.
# UGGGGHHHH. Way too much time.What does that mean exactly? I think it means that you can’t have two block devices added to an AMI with the same name. I *think* that you do want to add a block device with the *same* name as the root volume when you want to encrypt it. See below for details and I hope the documentation all gets updated to make this more clear as right now it’s not. Maybe it will be by the time you read this.
When you create an AMI it has a boot volume (also known as root drive, boot drive, or root volume). It’s a disk that contains the code that starts up your operating system. Then you can attach other volumes to it such as I do in Packer scripts to store data on a separate virtual drive or disk. I assign a KMS key to the block device I’m adding to the volume. I also want to encrypt the root volume.
Here’s some sample code.
"launch_block_device_mappings": [{
"device_name": "/dev/xvda",
"encrypted": "true",
"kms_key_id": "{{ user `kms-key-id` }}",
"volume_size": "{{ user `volume-size` }}",
"delete_on_termination": true
}]The problem occurred last time when I had overlapping block devices where two devices had the same name — I think. I forgot exactly what caused it but I remember I took out or changed those mappings and then it worked. I read that the new nitro instances should not be getting that error.
The way I build AMIs is that one builds a base AMI, then another adds tools, another configures project settings for people who work for me and the last one builds an admin image. That way I have shorter steps to troubleshoot instead of rebuilding everything all the time.
On each Packer build, I had put in the device mapping and the KMS key to encrypt it. In one of the templates I named the block device a different name inadvertently so I ended up with two block devices with two different names and one of them overlaps with the root device as shown here. I need to go back and fix that.
The odd thing is that when I launched the instance using that AMI in the console it worked, as you can see here. Note that in the test above I’m testing the AMI in the account where I built it. I got the CloudFormation error when deploying an instance using the shared AMI from another account to a third account. Not sure if that is relevant.
I also build a Windows instance the same way and did not have this typo and it worked fine so it’s not KMS permissions or some issue with my deployment. It’s specific to the Linux AMI I built with Packer and am trying to deploy with CloudFormation.
The only possible difference from what I am presenting here as a possible cause is that I passed in the incorrect AMI ID which was pulling up and old AMI that doesn’t work right for some reason not clear from this cryptic message. I could re-test this scenario but I don’t have time right now to make sure my drive mapping configuration was the actual problem.
The main frustration of all of this wasted time is that the error message is too cryptic and doesn’t provide enough detail to tell me exactly why this is failing. If I had the AMI ID, drive id, and KMS key ID, I would look and see those were different and instantly know that it didn’t match and that was the cause. But I *think* I was passing in the correct AMI. I can’t go back and test this as I’ve overwritten all my configuration with the new and improved AMI ID that definitely works.The error occurred for me when trying to build a CloudFormation stack with the AMI I built. I presume (after validating KMS permissions are correct) that this is the device mapping issue I got before or some variation. I’ll test it out, and update this blog post. I initially thought it was the other KMS key issue when I hit this, but KMS permissions don’t seem to be the issue, and then I remembered what my notes on this issue where it was somehow related to the drive mappings.
I figured this out last time off one obscure blog post that doesn’t seem to match the error message exactly, but I checked my device names and somehow resolved the problem. But now it’s back. What is going on.
As a test, I just rebuilt the one AMI with the device name that does not match the others to see if the CloudFormation error goes away.
Update ~~~~~~~~
It’s all about the launch_block_device_mappings (and KMS keys?)
It’s working now. I don’t know exactly why but here’s what I did.
I build a base AMI with device mapping like this:
"launch_block_device_mappings": [{
"device_name": "/dev/xvda",
"encrypted": "true",
"kms_key_id": "{{ user `kms-key-id` }}",
"volume_size": "{{ user `volume-size` }}",
"delete_on_termination": true
}]I thought what I did last time was something like this to try to add a second volume and that this caused the error. Note that both the devices have the same device_name.
"launch_block_device_mappings": [{
"device_name": "/dev/xvda",
"encrypted": "true",
"kms_key_id": "{{ user `kms-key-id` }}",
"volume_size": "{{ user `volume-size` }}",
"delete_on_termination": true
},
{
"device_name": "/dev/xvda",
"encrypted": "true",
"kms_key_id": "{{ user `kms-key-id` }}",
"volume_size": "{{ user `volume-size` }}",
"delete_on_termination": true
}]Here’s what I did this time:
I built another AMI using the prior AMI as a base with a device mapping (only one) where I inadvertently changed the name like this:
"launch_block_device_mappings": [{
"device_name": "/dev/[something different here]",
"encrypted": "true",
"kms_key_id": "{{ user `kms-key-id` }}",
"volume_size": "{{ user `volume-size` }}",
"delete_on_termination": true
}]Then I built two more AMIs, each with the prior AMI as a base and I used the original launch_block_device_mappings.
I noticed this discrepancy when reviewing that configuration further. I changed the anomalous launch_block_device_mappings to match the other three AMI packer templates — and then it worked.
As noted above there is one other possible scenario.
First some background: I automate this entire process:
- Build AMI 100% automated using Packer.
- Update an SSM Parameter in the account that builds the AMI with the latest AMI ID.
- Manually add KMS permissions in new account (still need to automate that).
- CloudFormation references the parameter with the latest AMI ID to build an EC2 instance in the new account.
I recently added a new account to build the AMIs so the step to build the AMIs is in one account and the step to run CloudFormation deployments is in a separate account. I also put the KMS keys in a separate account. The CloudFormation builder account still has old AMIs, encryption key, and SSM parameters that store old AMI IDs.
I manually added two new SSM parameters in the CloudFormation builder account (automation change to point to new parameters in AMI account TBD). The CloudFormation script may have been pointing at the old SSM parameters and an old AMI. But even if they were — why did the Windows deployment work and not the Linux deployment? Those old AMI IDs worked fine in the past in another account that had permission to the old key, but the new account would not have permission to the old KMS key used to encrypt the old AMIs. Both the Linux and Windows deployments should have failed if that’s what I did because neither would have access to the old keys. So I think it still may have been the Packer template but I don’t have time to test.
To ensure I don’t use the wrong AMI ID, I need to automate those last few steps. I can also write out a message showing the AMI ID, AMI name, device names and which is the boot volume, and encryption keys for each device prior to deploying the CloudFormation template so I can verify but that’s more work for me.
If the error message was less cryptic, then I would know that instantly. I don’t think using the old AMI ID was the issue but cannot say with 100% certainty. If that was the issue perhaps the old AMI was using a different key and it was the KMS issue all along. In that case here is a more appropriate error message:
IAM User (or role or whatever) does not have access to key ID (the KMS key being accessed) to perform action (encrypt/decrypt/grant, etc.) on the device (device name) attached to ami (ami id).What is very strange:
Why does this only fail in CloudFormation but it works in the console? Is one of those options or the other doing something it should not do? Or did I do something else wrong that caused this anomaly? It would be helpful to output the AMI ID and Key ID or Device Mapping that is causing the problem in the error message. I don't think IDs alone should be a security risk to expose - are they? Then I can validate that I'm spinning up the AMI ID I think I am, the device names I think I am, and that the correct encryption key is being used.Should Packer be failing on the incorrect (is it incorrect?) template configuration? Am I even doing this right to ensure my AMI boot drive and any additional drives are properly encrypted -- because the documentation from Hashicorp Packer is not entirely clear in terms of the Packer configuration and I haven't found an explanation from AWS as to how this works in relation to CloudForamtion and how that maps back to the EBS encryption process details in the AWS documentation.I think I'm doing it correctly but how can I tell? I need to see if the contents of my drive are encrypted or not. How can I do that? I mean, I know there may be a key listed in the configuration (sometimes - it's missing in some places in the console for EC2 where I think it should be listed.) Pondering this question....but right now I've lost so much time I have to get work done...and that's how security problems proliferate.Also, this is not clearly documented and the configuration you have to write seems misleading to me: I think when you add a device block mapping that *matches* the root volume and add a KMS key in packer that it is encrypting the root device. Is that true? I read all this documentation in painstaking detail when creating the templates initially, but forgot since have been doing a lot of other things since. It would be more appropriate to have a configuration item in Packer like a boot disk configuration block and then configuration for additional drives you want to add. Or perhaps you have to indicate with a flag which added volume is the boot volume. How does the image know which is the boot drive? Based on the device name? Maybe that's documented somewhere and I forgot.There's also some flag to encrypt root volume (encrypt_boot) in packer. Do I also have to set that and to what? After reviewing the documentation seems like not:If you have used the launch_block_device_mappings to set an encryption key and that key is the same as the one you want the image encrypted with at the end, then you don't need to set this field; leaving it empty will prevent an unnecessary extra copy step and save you some time.I'm not sure how that device name got altered in the second template. Maybe I didn't check in my code after the last fix.In Packer, for appropriate encryption of the boot drive while Packer is building my AMI, I presume I need to use launch_block_device_mappings not ami_block_device_mappings correct? Because if I do not add the encrypted volume until the end the volume is not encrypted while I'm building the image. Is that the correct interpretation?Fix: Fix ALL the cryptic error messages and state exactly what the problem is.Fix: Make sure all the documentation provides in technical detail what is going on with device mappings, KMS keys, and the root device so people know that their data is encrypted and they have configured things properly. Fix: Add the KMS key used to encrypt each volume on the main EC2 Instance page so it clearly shows which drives are encrypted and with which KMS key, or the default AWS key.When I worked on the original Capital One cloud engineering team this came about as a result of one of our feature requests (a list I had to manage at the time):
~~~~~~~~~~~~
For the KMS issue, follow my instructions in this other post to add credentials.
Note that SSO works differently:
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2022
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
