SCP to Allow Closing and Removing AWS Accounts — Part 2

ACM.183a Using CloudTrail to troubleshoot a Service Control Policy and the mysterious GetAccountPreferences action

Part of my series on Automating Cybersecurity Metrics. The Code.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

In the last post I was trying to set up a service control policy architecture that prevents unauthorized removal of an account from an AWS organization, while still allowing for approved changes.

I ran into some issues because I don’t know exactly what actions I need to correctly implement this policy. (And I spent way too long troubleshooting what seems like should be straightforward CloudFormation but clearly I was doing something wrong.)

In this post I’m going to try to figure out what actions I’m missing based on CloudTrail logs and revise my policies to allow the root administrator in the child account to successfully change the payment method. That’s the action that continued to fail in the past post.

Now although AWS is deprecating the old actions, I went through these steps with the old actions and the results were curious.

Reviewing CloudTrail for errors

To troubleshoot this issue, since I’m only in a test account at the moment, I added all CloudTrail actions to the root administrator use in the child account.

Then I took at look at CloudTrail. Now once again you have to know which region to look in to find the logs. For billing, head over to us-east-1.

Next I had to figure out how to query the events. As I explained in a much prior post I added the error column to the logs and started looking for error. I trie to search on a resource type or name like “account” or “payment method” or something to do with “organizations” but that yielded nothing.

I was able to find the actions with errors when looking at the event source which turns out to be:

billing.amazonaws.com

As you can see here, no Resource type or Resource name is listed here. That’s why querying on those options was futile. I guess a payment method is not considered a resource, nor is “billing” a resource type.

I removed some columns so I could more easily see the error column and which action was causing the error. It’s not entirely clear from the details of this action but I believe this may be the action I need to add to allow the payment method update to process. I don’t see any other errors so by process of elimination this must be it?

AWSPaymentPortalService.GetAccountPreferences

Let’s add that to our policy and try adding the payment method again with the root administrator in the child account.

So, where was I? I have to log into the OrgRoot user account and modify the policy to add the new action. It is unfortunate that there are so many different naming conventions on AWS. Consistency would make things easier.

I showed you how to find the prefix for actions in IAM policies in a prior posts. In this case, you can see that the action reported in CloudTrail logs does not match what you need to add to the policy in CloudFormation. I presume this prefix relates to aws-portal so we’ll use that with the action and see what happens.

After updating my SCP, when I login and I can see the payment method has been added to the account. I presume that’s because I added the above permission. Maybe it takes time to show up. In any case the payment method exists in the child account already so I don’t need to re-add it.

Let’s head back over to our OrgRoot user and try to remove the account from the organization.

Um. Hmm. Same error. I’m using the OrgRoot user that has administrator access in the root account of the organization. That means no SCPs apply to this user so looking in CloudTrail logs isn’t going to help us. This user can literally do anything in my organization with no restrictions.

Let’s take a look at the other account one more time. Note that although the account has a payment method — it does not have a default payment method as stated in the error at the top of the screen. Perhaps that didn’t get set due to the earlier permission error. Let’s click the “Set as default” button and see what happens.

Well, this is fun. :) You find all kinds of interesting things when you try to use zero trust policies and networking, in my experience. As mentioned above I noticed that currency was referenced in the CloudFormation actions that caused an error above. I guess that was a good guess.

Click continue.

We’ve hit another error. That’s all we get. So no idea what permission is missing here.

Viewing all the errors in CloudTrail

Back to CloudTrail logs. I usually just remove the entry in Read only to view all events.

I also explained in a prior post how I click the gear on the top right and add the error column. Then I removed the columns taking up space that aren’t helping me solve this problem.

Here we can see a number of failed options, some of which are probably from the home AWS dashboard. And side note, AWS would have less errors and logs in their systems if they would simply not show users things which they are not allowed to access instead of proliferating these things in CloudTrail logs, but I digress.

We are looking for billing and payment method related errors and possibly something related to currency. I see some cost and usage related events here.

Recall that I explained the ce prefix is one of the prefixes AWS is using now for billing related actions in this post on setting up a billing administrator role policy:

How baselines and duplicate error messages help

I’m not sure the ce events are what we are related to this problem. Recall that it takes a minute for logs to show up in CloudTrail. It’s not instantaneous. Perhaps the actions we need aren’t there yet.

However, we do have a baseline of events starting with the signing action for this user. Let’s try the actions we just tried that failed again if we can. Then we can look for duplicate errors in the logs. The duplicate errors (if we do not revisit the main dashboard) should indicate the actions that are preventing us from assigning a default payment method.

Same error.

Now let’s head back to CloudTrail and see if we can figure out which actions are causing this problem. The thing is, I still see the same error in CloudTrail as before.

As I mentioned before these prefixes don’t match. Let’s check out the documentation for aws-portal and verify that this action exists. It seems like it should because we didn’t get any errors when creating our SCP.

Actually, aws-portal:GetAccountPreferences does not exist…hmm.

Well I searched around for “GetAccountPreferences” to see what I could find. Nada. Nothing comes up in Google for “GetAccountPreferences” and AWS.

There is one other odd thing about the error message. It is not an access denied error. It is a “BadRequest” error.

So what’s bad about this request? I’m guessing it has to do with the changes AWS is making to their AWS billing actions for IAM.

Out of curiosity, I wonder what would happen if I try *.GetAccountPreferences. That doesn’t work. CloudFormation gripes at me.

What if I try aws-portal:*.

Aha. When I added that to my policy, the root administrator in the child account could see the screen that was blocked when trying to add the default payment method. Now I can see that the card I added is labeled as the default card.

Now what actions appear in CloudTrail? The events will not show up instantly so we have to wait a minute. Now when I filter on billingconsole.amazonaws.com I see a new action. I’m not sure if I didn’t wait long enough last time but I thought I waited quite a while. Hmm. I don’t see this action in any lists either.

I would prefer to create a very specific SCP in this case. I wonder what happens if I add ConvertCurrencies and GetAccountPreferences to my policy.

No. That does not work either.

Well, this was fun but I don’t have this much time to troubleshoot AWS policies with actions that are being deprecated anyway.

Now let’s return to the OrgRoot user logged into our organization root account and see if we can remove the account.

No. I still cannot remove this account from the organization using the OrgRoot user.

Is it related to that currency conversion?

In summary, it seems like it’s pretty much impossible to do what we are trying to do with the old billing IAM actions. Here are some of the problems:

1. Not all actions in CloudTrail are listed in the AWS IAM actions for the aws portal.
2. I presume all these actions should be part of the AWS portal and not aligned with some other prefix — otherwise that needs to be documented more clearly.
3. It would be nice if the actions in CloudTrail showed the proper IAM Policy prefix that needs to be present in an IAM Policy.
4. BadRequest is generic and seems like an AWS bug they might want to look into.
5. Having worked on financial systems before AWS might want to take a very careful look at the “ConvertCurrency” action to make sure that no funny business is going on that causes loss of funds in any way due to incorrect conversions. That just feels like a red flag to me.
6. In my case, I don’t see any need for that action because AWS is a US company and I am a US customer. Is this function called unnecessarily?

For now, I’m done with this little experiment. Let’s try out the new billing actions AWS created in our SCP and see if they work better.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab