avatarTeri Radichel

Summary

The author encountered an issue with AWS KMS not working as expected due to a FIPS endpoint configuration, which required specifying the endpoint URL in the CLI call, a step that was not necessary previously.

Abstract

Teri Radichel, a cybersecurity expert, discovered that her AWS KMS (Key Management Service) was not functioning correctly when trying to list KMS aliases. Despite having a VPC endpoint set up, the traffic was not routing to the private network as anticipated. After investigating, it was found that the issue was related to the use of a FIPS (Federal Information Processing Standards) endpoint, which had worked intermittently before but now consistently resulted in network timeouts. The resolution involved explicitly defining the FIPS endpoint in the AWS CLI call, a step that had not been required when the endpoint was initially configured. This experience led Radichel to suggest that AWS could simplify the process by allowing a one-time configuration for all services to use VPC Endpoints on private IP addresses, and to question why the AWS system did not automatically recognize and use the FIPS endpoint when a standard KMS endpoint was not present.

Opinions

  • The author believes that AWS's current setup, which requires individual service configuration for VPC

Connecting to a FIPS endpoint

Getting a timeout with the AWS CLI? Here’s how to fix it — and also — why did it work before it didn’t??

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Bugs | AWS Security | Secure Code

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Not sure if this was in the documentation before but it could be clearer, or it is now.

I set up a FIPS VPC Endpoint for some services and it was merrily sending my traffic off to public IPs instead of the private IPs I expected.

OK so I have KMS set up on a private VPC endpoint and I forgot about it. I’ve been using KMS for a while but suddenly now it’s not working. So I think, “Did I forget to set up a VPC endpoint?” And I go check to see that it’s there.

I mean I’ve been using KMS for a while so it must be there because traffic would be blocked otherwise the way I have my network setup (I think). That’s how I figured out requests weren’t going to private networks in the first place.

So anyway, I’m trying to list KMS aliases tonight and it’s not working and I go check and yes, I have an KMS endpoint. But it’s a FIPS endpoint like the one I set up for CodeCommit above.

Why did it work before but not now?

Also — this worked intermittently just now. Not every call to KMS failed. Odd. Some went through initially. Then they all started failing due to a network timeout.

So I just guess that there must be a different endpoint for FIPs and maybe I need to do something to reach that endpoint (complete guess — because I never did before and it worked). And sure enough now the documentation says I need to specify the FIPS endpoint in my CLI call.

aws kms list-keys — endpoint-url https://vpce-1234abcdf5678c90a-09p7654s-us-east-1a.ec2.us-east-1.vpce.amazonaws.com

Hmm. I set this up a long time ago and never had to do that before and it “seemed” to work. But was it really not going to the VPC endpoint before and I didn’t notice?

AWS could make this a lot easier as I’ve mentioned before.

Set up one configure for all services to go to VPC Endpoints on private IP addresses once. Done.

Why do I have to configure each individual service?

Also, if I set up a FIPS endpoint and no standard KMS you can’t tell that I need the FIPS endpoint and do that automagically for me? Haven’t really thought that through but seems possible, no?

Well anyway, if I use the FIPs endpoint then I have to change all my KMS code in this series to support the FIPS endpoint. Which convinces me every more that something has changed because this has always worked up till now and I set up the FIPS endpoint a long time ago.

Just to see if it works I try it and it works like a charm.

Also, just for kicks, I dig the endpoint and it does return a private IP, which is also reported in the profile.

So that mystery is sort of solved, but it seems like this could all be easier to use.

For now I’m going to switch to a non-fips endpoint so I don’t have to change all my KMS code to use a different endpoint url at this moment. I might change that later.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab
Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation
Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
Fips
Vpc Endpoint
Kms
Timeout
Bug Fix
Recommended from ReadMedium
avatarShuai Li
Most Developers Failed with this Senior-Level Python Interview Question

What is the difference between [0] * 3 and [0, 0, 0]?

8 min read
avatarChristopher Tao
Do Not Use LLM or Generative AI For These Use Cases

Choose correct AI techniques for the right use case families

7 min read
avatarBobby
NiFi 101: Installing and Configuring Apache NiFi Locally with a Container Image

Apache NiFi is a powerful, user-friendly, and scalable data integration tool that supports powerful and scalable directed graphs of data…

3 min read
avatarJames Matson
Building a High Performing Engineering Team When You’re Not Big Tech

Many smaller organisations don’t have the scale, resources or capital of a Meta, Amazon or Netflix in order to implement the ‘big tech’…

24 min read
avatarAbhay Parashar
17 Mindblowing Python Automation Scripts I Use Everyday

Scripts That Increased My Productivity and Performance

14 min read
avatarJakob Pörschmann
Graph RAG — A conceptual introduction

Graph RAG answers the big questions where text embeddings won’t help you.

8 min read