Transform AWS::LanguageExtensions failed with: Could not find a collection or could not be resolved for Fn::ForEach
Troubleshooting the For::Each statement in a CloudFormation template
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics | Code.
🔒 Related Stories: Bugs | AWS Security | Secure Code
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TLDR: You can use ForEach to create a list of similar resources, but not policy statements, because those are not resources. I have a request into AWS to make policy statements a cloudformation resource but who knows if that will happen. Also, you cannot use a ForEach within If but the error message isn’t super clear. It seems to be saying you can’t use some other type of function in a ForEach. And why can’t we have parameters that are not required and default to AWS::NoValue?? Then, if using ForEach, it should simply not do anything if the value of the parameter is AWS::NoValue, right? Too many hijinks to work around that very common use case for conditions and if statements.
I’m trying out the new CloudFormation For::Each function and hit this error. I’m pretty sure it is because I’m passing an empty array to my For::Each statement but let’s see if we change that if the error goes away.
Here are the CommaDelimitedList parameters I’m passing into my CloudFormation template:
As you can see I added Default values so I can get CloudFormation to stop griping if I don’t pass anything into the list.
Now, I’ll admit that right here, I wish AWS would allow a way not require values in AWS templates more easily if that’s what you really want to do. In fact, an option to specify whether a parameter is required or not would be amazing. That way I wouldn’t have to add all these Fn::If statement to templates to say basically, “If a value is set, set it, otherwise set it to AWS::NoValue. I mean, if there’s no value set it to no value? That would be cool. Or, if it’s an empty list, then a For::Each would just do nothing.
Something like this:
The default should be AWS::NoValue as that’s mostly what I have to set property values to in my Fn::If statements when I don’t want to set a property based on a condition. If you’ve been doing a lot of complex CloudFormation templates for a while — how many lines of code would that save you and hours of troubleshooting conditions and fn::If statements? I always seem to have some kind of typo in those and it’s a very common pattern.
Unfortunately we can’t default our parameter to AWS::NoValue. I’d love to do this:
And then just have the For::Each figure out that there’s no list so do nothing. But that yields this error:
An error occurred (ValidationError) when calling the CreateChangeSet operation: Template format error: Every Default member must be a string.
So, setting the default to an empty string again and have to do a bunch of additional lines of code to make this work. First I’m going to create a condition that indicates whether the parameter is set or not by checking to see if it equals an empty string.
Now we have a condition. Let’s see if we can only run the For::Each if the condition is true.
I add my if as follows — here’s that AWS::NoValue I just mentioned:
OK, but now I get a new error:
This isn’t even a standard CloudFormation error. I’m guessing it’s because I’m trying to concatenate a string with a list above and that’s not going to work. AWS has another construct that might help us here. We can get a value from a list using Fn::Select.
I can get the first value in the list and see if it equals an empty string.
But doing that still gave me the error.
To try to resolve the problem I commented out the ForEach loop to make sure the problem was with the condition. This is where line numbers would help. That’s when I got an error message that made more sense. I inadvertently left the Required parameter property I was demonstrating in my code above. I removed it.
Next I get this error:
Reason: Transform AWS::LanguageExtensions failed with: Fn::Not layout is incorrect
Hmm. Fn::Select is supported in Fn::Join.
But I still got that dict error even after I added select. Let’s take that out for a minute and see what happens. Always good to start simple when troubleshooting. This is where AWS handling the no value for a parameter would be very, very nice. I always have typos in this stuff.
Transform AWS::LanguageExtensions failed with: Fn::Not layout is incorrect
For my own sanity, I remove the conditions to make sure that’s the problem. It is. OK what did I do wrong? Sometimes it the most obvious things and I don’t see it, like editing my own typos in my blogs. 😆 Do you see it?
Should be:
This is where an IDE might help me out but they are all so heavy weight. I would have to run a GUI and right now I can do everything from a Linux command line. I also would have to open up outbound ports on my developer instance if I use an online web GUI. Not desirable. Have you ever looked at all the web traffic generated by VSCode? So I wish I could just get better error messages out of CloudFormation. But if you are just starting out perhaps an IDE would help like Cloud9. I’m not sure I haven’t actually used it or any IDE much in years.
But I digress. Now I get this error — indicating my second value in the join is not a string (it’s a CommaDelimitedList parameter, remember?).
Reason: Template error: every Fn::Join object requires two parameters, (1) a string delimiter and (2) a list of strings to be joined or a function that returns a list of strings (such as Fn::GetAZs) to be joined.
So I’m going to add back in the select to see if that fixes the problem. But I had an error there too with my select statement. I forgot to add in the !Ref to my parameter. But with Fn::Select you can’t use !Ref in the short form as noted in the documentation.
But then I get this error:
Fn::Select requires a list argument with two elements: an integer index and a list
So I try making my parameter default a list since my parameter is a list. That makes sense right?
Nope:
Template format error: Every Default member must be a string.
Well what about this?
Nope:
Transform AWS::LanguageExtensions failed with: Intrinsic function input type is invalid
Am I the only one who does all this crazy stuff? Let’s try something else. What if we use Fn::Length?
No, we can’t use that because we can’t set the parameter to an array by default. The value of empty string is still a value. And besides that also gives an error since my default value is a string, not a list.
Transform AWS::LanguageExtensions failed with: The Fn::Length value could not be resolved for HasEncryptServices
What if I just get rid of the not and join and see if the value equals an empty string. I got that join thing from an example. But in this case my value just equals en empty string:
Will this work? No:
Transform AWS::LanguageExtensions failed with: Intrinsic function input type is invalid
What if I just comment out the transform?
A more useful error message:
every Fn::Equals object requires a list of 2 string parameters.
So now why is this happening? Do I need the join?
So I even aligned the format to exactly match the condition above it that is working and that’s when I see it. I seriously have some propensity to skip over letters and even words when reading. It helps me read very fast, but it also causes problems when I miss ONE STINKY LETTER.
One of the problems here is that the Transforms are somehow masking the error messages. But the other thing is, this is such a common pattern that I wish AWS would handle it as noted above.
But I still get an error because equals can’t compare a list and a string so I’m back to this (with the parameter name sans typo):
Cool that works. Now uncomment my Transform and For::Each and see if change the if to work with the new condition logic (NoEncryptServices vs. HasEncryptServices).
Back to this error:
unhashable type: ‘dict’
That was due to a misplaced semicolon:
Next error:
Transform AWS::LanguageExtensions failed with: Fn::If layout is incorrect
Since AWS Transform seems to be rejiggering the errors somehow, let’s remove it and do the if without it to see what error we get.
Without the Transform I get this error:
Template error: Fn::If requires a list argument with three elements
[Shouldn’t the Transform code just pass the error through?]
Anyway, what’s wrong with my list.
Oh duh. Looking at the example in my own code right abive it should look like this:
But that doesn’t work either. Here’s the thing, It’s a statement in a CloudFormation policy. That fist dash indicates we have a statement. And the !If needs to be on the next line. I know. So obvious right? (Not.)
Ok so that works now let’s add back in the Transform and the ForEach.
Yay! That works with no services. What if I pass in services to the template?
I create these variables:
I pass them into a function I wrote so the list of services ultimately gets passed into the CloudFormation template. Luckily my single function for deploying every template can output the command it ran and I see the problem with my parameters:
I am getting the value for the parameter after ServicesCanDecrypt for that parameter so I must have a typo. I’m passing in an invalid value. I’m not sure that is the whole problem but let’s fix that.
But here’s the next problem. I get this error. The problem now is that I cannot see what the parameters were that got passed in since the error occurs before the stack runs. So although it’s nice to get the errors sooner, the errors have to make sense to me and ideally show me the parameters in this case, because I’m guessing that is the problem.
Transform AWS::LanguageExtensions failed with: Mappings not found in template for key /KmsAccount/KeyAdminRoleName on resourceType AWS::KMS::Key
Queen of typos here. That fixed the problem and onto the next.
Template Error: Encountered unsupported function: Fn::ForEach::ServiceCanEncryptLoop Supported functions are: [Fn::AccountIdFromAlias, Fn::Base64, Fn::GetAtt, Fn::GetAZs, Fn::ImportValue, Fn::Join, Fn::Split, Fn::FindInMap, Fn::Select, Ref, Fn::Equals, Fn::If, Fn::Not, Condition, Fn::And, Fn::Or, Fn::Contains, Fn::EachMemberEquals, Fn::EachMemberIn, Fn::ValueOf, Fn::ValueOfAll, Fn::RefAll, Fn::Sub, Fn::Cidr]
So what functions am I using in my loop?
This is where I wish CloudFormation would figure out how to support the short form everywhere. I’m guessing it’s the sub. Changing it to the long form didn’t work.
I found this example so I exactly copied it with the !Ref and the Fn::Sub: and replacement value on one line.
I had quotes around my Sub value. I removed those and did not fix the problem.
Now in the example above they are using a mapping instead of a parameter.
Looking at the example I was referring to before, it is using a map:
Each map has a key, value pair. I’m just passing in a list of values. Does that matter? In another example I see this format:
What if I put brackets around my list?
Nope.
What if I hardcode the services. Is it my parameter?
Nope.
What if I remove the sub? Nope.
Finally, I noticed that all the For::Each examples on the sample page I was looking at have quotes around the For::Each line.
So I added quotes. And finally, a new error appears. Hallelujah.
I’ve faced that message before but for a completely different reason:
That error was due to the fact that I put the last colon inside the quotes so I moved it outside the quotes as shown above.
What is intersting is that when I go to the line number with the error (thank you, thank you, thank you) I see that this blocked worked.
I forgot to move the colon in the second block so I fixed that.
Now I’m back to the same error I had before. So I guess the quotes were not the thing.
What if I just hardcode everything:
Still no dice.
Finally this comes to mind. What if the loop is failing because it’s in an If? That’s not what the error message says but doesn’t look like If supports ForEach.
Finally. Well, that error message seems backwards and misleading to me. Can’t the parser be more specific about which function is in which other function instead of spitting out that whole list?
Next error:
Transform AWS::LanguageExtensions failed with: Duplicate key ‘Sid’ when merging keys to parent object in Fn::ForEach
I tried adding the service name to the Sid. That didn’t work.
I tried messing with the spacing and added another dash (which I didn’t think was correct) and got this:
Transform AWS::LanguageExtensions failed with: The template fragment provided in Fn::ForEach is not valid
And now I finally see that this has all been a royal waste of time.
Here’s why. I didn’t understand this upon first reading the documentation. The output key has to be in the format [UniqueID] : [Output Value]
OutputKey
The key in the transformed template.
${Identifier}must be included within theOutputKeyparameter. For example, ifFn::ForEachis used in the Resources section of the template, this is the logical Id of each resource.
I am trying to create a snippet of a policy which does not meet that criteria. This is where I have a request into AWS to make statements in a policy their own resource. If we could do that, then I could create the statements as resources and append them to the policy no problem. Unfortunately, that’s not possible here.
On an IAM Policy, perhaps I could create a number of policies and attach them to a principle but if my list of statement is too long its going to hit a limit. And, we’re dealing with a KMS key policy here. Each key has one policly.
Game over. As far as I can tell. Thanks for playing.
If you are creating a resource, not a policy like I am, you can simply use your identifier as the logical ID:
Bucket1: [your output]
Bucket2: [your output]Well, I guess we had fun resolving a lot of errors with ForEach anyway. And now I see why I haven’t used Language Transforms as of yet. Back to the drawing board.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
