Troubleshooting the MFA in Lambda— Works Locally, Not In Lambda
ACM.330 Resolved some issues, others could not due to misleading error messages and mismatched environments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics | Code.
🔒 Related Stories: Lambda | GitHub Security | Container Security.
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post I wanted to deploy a pretty cool way to enforce MFA on GitHub actions (like clone, commit, etc.)
Not to be confused with GitHub Actions — which require you to share AWS credentials with GitHub and has proven to be problematic in some cases as I wrote about in prior posts on GitHub Security:
TLDR; Make sure you select the subnet for your VPC endpoint where your resources exist. If you select all three subnets in a VPC with AZ a, b, c and a Lambda function is only in the subnet for AZ a, then your Lambda function may take 10 minutes to complete while it tries all the available networks. See my logs and DNS checks below where I figured that out.
But in the end, the Lambda error message says my role doesn’t have permissions it does. Even though that’s what the error message says, pretty sure that is not the actual problem. I think it may be related to file permissions but did not want to spend more time troubleshooting so moved on. I got this working another way with an EC2 instance and a container here, though I’m not sure this is going to work consistently:
Here’s my attempt to make this work. Maybe I’ll revisit it again later.
Checking the VPC endpoint exists for the correct subnet
Now I know I double checked the networking last time I ran into a problem with STS in a Lambda function. At least I thought I did. But I started to doubt myself. So I double checked and yes, I do have an STS endpoint in my NAT VPC. But to which subnet is it connected?
Yes, it is in the same subnet as my Lambda function.
Making sure the temporary credentials are able to assume a role
But I started to think about a couple of things. First, the last time I tested with STS I was using the AWS Lambda execution role. This page has a list of the different types of STS credentials you can use and their limitations — such as not being allowed to call the STS service.
For example, if we were using GetSessionToken we could not call the STS API.
But we are using AssumeRole with an AWS IAM user so it seems like we should be ok.
I don’t know what operation Lambda is using when calling STS. Perhaps it is blocked — but the timeout feels like a network issue.
Considering the IP addresses accessed by VPC Endpoints
I also started thinking about the problem I had earlier where my endpoint was still using a public IP address.
I figured out I had the above issue after testing STS. Maybe I’m having a similar issue here. Perhaps the last time I was looking for public, not private IP addresses.
There’s no mention of public or private IP addresses here but perhaps I was getting back a public IP address.
I perform a dig command on my EC2 instance to see what I get back — and no that’s not the problem. It’s a private IP address. I would presume if I run a dig command from my Lambda function, which I started to try to do but found complicated so I ditched the idea at the time, it would also return a private IP address.
AWS Health Dashboard
Next I check the AWS Health Dashboard. Perhaps the STS service was having an issue. There are no current issues. I try to look at the history for STS but there is no service specific to STS which stands for Security Token Service.
I guess it falls under IAM. No issues.
Revisiting the logs again
Now, I know I looked at my network logs last time this happened but I’m going to look again in case I missed something.
I look at the logs for my STS VPC Endpoint in the same subnet as my Lambda function. There’s not a single connection attempt.
This leads me back to my Lambda function. Where is it trying to send the data and why is it blocked? The NACL is wide open. That leaves the security group that should be allowing traffic to the STS VPC Endpoint.
Aha. In my test VPC where I have my EC2 instance, I have a single Security Group for all my endpoints. In the NAT VPC where I have my Lambda function I have a separate VPC for each endpoint. I need to add a security group for STS, and CodeCommit while we’re at it.
The thing is, I deploy the security groups automatically with my networking code, I just didn’t apply them to the Lambda function. Oops. I will need to modify my code to do that — however I may simply use one security group in the future to simplify things and prevent errors — like this one. The only reason not to do that would be if you want to limit which AWS services a particular Lambda function can call.
Wouldn’t it be awesome if AWS could just figure out what the problem was and give you an error message telling you how to fix it — or link over to some service that analyzes your account and tells you the possible fixes for this problem? In general, a timeout is a networking issue.
So I add the additional security groups to my Lambda function and retest. The only concern I still have is if the AWS CLI is writing that temp file to a disallowed location. Will it work?
Well, with an invalid code, we now get the expected response. That’s good. My networking fix seems to be working. That could also simplify other code where I was using a workaround because I couldn’t get to STS.
Alright. Now let’s try a valid code and see what happens.
Checking permission to assume the role
While I was out walking the dog I started to wonder if I had granted the user permission to assume the role. I know I updated the trust policy to allow the user to assume the role but did I update the IAM user policy?
The next error indicates lack of permission to assume the role.
So I check the user policy and it does have permission to assume the role:
I double check the role as well.
Hmm. Seems OK.
Also, this all worked locally. So what’s the difference? I’m passing in values from the AWS console.
The next thing I do is tweak my validation functions to remove any quotes that might exist in the numeric token and a few other things to validate the value.
Still didn’t help.
I echoed out the command to the screen to make sure it looked right in the Lambda environment and it does.
Then I add debug to the end to see if that helps.
Here’s what I get back:
{
"errorMessage": "An error occurred on line 55. Exit code 254.
42a93201-0297-4b58-9398-4f3ee5e0bd2e 2023-10-05 23:06:59,577 - MainThread -
awscli.clidriver - DEBUG -
CLI version: aws-cli/2.13.19 Python/3.11.5 Linux/4.14.255-318-256.530.amzn2.aarch64
exec-env/AWS_Lambda_Image exe/aarch64.amzn.2 2023-10-05 23:06:59,577 -
MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts',
'assume-role', '--role-arn', 'arn:aws:iam::464339214996:role/dockertestAssumeRole',
'--role-session', 'dockertest', '--profile', 'dockertest', '--serial-number',
'arn:aws:iam::xxxxxxxxxxxx:mfa/dockertest', '--token-code', 'xxxxxx',
'--profile', 'automation', '--debug'] 2023-10-05 23:06:59,619 - MainThread
- botocore.hooks - DEBUG - Event building-command-table.main: calling handler
<function add_s3 at 0xffffac7fd9e0> 2023-10-05 23:06:59,619 - MainThread -
botocore.hooks - DEBUG - Event building-command-table.main: calling handler
<function add_ddb at 0xffffaca2b2e0> 2023-10-05 23:06:59,619 - MainThread -
botocore.hooks - DEBUG - Event building-command-table.main: calling handler
<bound method BasicCommand.add_command of
<class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-10-05 23:06:59,619 - MainThread - botocore.hooks - DEBUG -
Event building-command-table.main: calling handler <
function change_name at 0xffffacbcd620> 2023-10-05 23:06:59,619 -
MainThread - botocore.hooks - DEBUG - Event building-command-table.main:
calling handler <function change_name at 0xffffacbceca0> 2023-10-05
23:06:59,619 - MainThread - botocore.hooks - DEBUG - Event
building-command-table.main: calling handler <function alias_opsworks_cm
at 0xffffac8104a0> 2023-10-05 23:06:59,619 - MainThread - botocore.hooks -
DEBUG - Event building-command-table.main: calling handler <function
add_history_commands at 0xffffaca8dd00> 2023-10-05 23:06:59,620 -
MainThread - botocore.hooks - DEBUG - Event building-command-table.main:
calling handler <bound method BasicCommand.add_command of <class
'awscli.customizations.devcommands.CLIDevCommand'>> 2023-10-05 23:06:59,620 -
MainThread - botocore.hooks - DEBUG - Event building-command-table.main:
calling handler <function add_waiters at 0xffffac810360>
2023-10-05 23:06:59,620 - MainThread - botocore.hooks - DEBUG -
Event building-command-table.main: calling handler <bound method
AliasSubCommandInjector.on_building_command_table of
<awscli.alias.AliasSubCommandInjector object at 0xffffac8b6590>>
2023-10-05 23:06:59,620 - MainThread - botocore.loaders - DEBUG -
Loading JSON file: /usr/local/aws-cli/v2/2.13.19/dist/awscli/data/cli.json
2023-10-05 23:06:59,622 - MainThread - botocore.hooks - DEBUG -
Event top-level-args-parsed: calling handler <function resolve_types at
0xffffac9376a0> 2023-10-05 23:06:59,622 - MainThread - botocore.hooks -
DEBUG - Event top-level-args-parsed: calling handler
<function no_sign_request at 0xffffac9379c0> 2023-10-05 23:06:59,622 -
MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed:
calling handler <function resolve_verify_ssl at 0xffffac937920>
2023-10-05 23:06:59,622 - MainThread - botocore.hooks - DEBUG -
Event top-level-args-parsed: calling handler <function
resolve_cli_read_timeout at 0xffffac937b00> 2023-10-05 23:06:59,622 -
MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed:
calling handler <function resolve_cli_connect_timeout at 0xffffac937a60>
2023-10-05 23:06:59,622 - MainThread - botocore.hooks - DEBUG -
Event top-level-args-parsed: calling handler <built-in method update of
dict object at 0xffffac8b3d40> 2023-10-05 23:06:59,622 - MainThread -
botocore.session - DEBUG - Setting config variable for profile to
'automation' 2023-10-05 23:06:59,635 - MainThread - awscli.clidriver -
DEBUG - CLI version: aws-cli/2.13.19 Python/3.11.5
Linux/4.14.255-318-256.530.amzn2.aarch64 exec-env/AWS_Lambda_Image
exe/aarch64.amzn.2 prompt/off 2023-10-05 23:06:59,635 - MainThread -
awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'assume-role',
'--role-arn', 'arn:aws:iam::xxxxxxxxxxxx:role/dockertestAssumeRole',
'--role-session', 'dockertest', '--profile', 'dockertest', '--serial-number',
'arn:aws:iam::xxxxxxxxxxxx:mfa/dockertest', '--token-code', 'xxxxxx',
'--profile', 'automation', '--debug'] 2023-10-05 23:06:59,635 - MainThread -
botocore.hooks - DEBUG - Event session-initialized: calling handler
<function add_timestamp_parser at 0xffffac7fe340> 2023-10-05 23:06:59,636 -
MainThread - botocore.hooks - DEBUG - Event session-initialized:
calling handler <function register_uri_param_handler at 0xffffacc4c4a0>
2023-10-05 23:06:59,636 - MainThread - botocore.hooks - DEBUG -
Event session-initialized: calling handler <function add_binary_formatter
at 0xffffac885940> 2023-10-05 23:06:59,636 - MainThread - botocore.hooks -
DEBUG - Event session-initialized: calling handler <function no_pager_handler
at 0xffffad4605e0> 2023-10-05 23:06:59,636 - MainThread - botocore.hooks -
DEBUG - Event session-initialized: calling handler
<function inject_assume_role_provider_cache at 0xffffacc4fd80>
2023-10-05 23:06:59,637 - MainThread - botocore.utils - DEBUG -
IMDS ENDPOINT: http://169.254.169.254/ 2023-10-05 23:06:59,639 - MainThread -
botocore.credentials - DEBUG - Skipping environment variable credential
check because profile name was explicitly set. 2023-10-05 23:06:59,639 -
MainThread - botocore.hooks - DEBUG - Event session-initialized: calling
handler <function attach_history_handler at 0xffffaca6a480> 2023-10-05
23:06:59,639 - MainThread - botocore.hooks - DEBUG - Event
session-initialized: calling handler <function inject_json_file_cache at
0xffffac9ff560> 2023-10-05 23:06:59,696 - MainThread - botocore.loaders -
DEBUG - Loading JSON file:
/usr/local/aws-cli/v2/2.13.19/dist/awscli/botocore/data/sts/2011-06-15/service-2.json
2023-10-05 23:06:59,697 - MainThread - botocore.hooks - DEBUG - Event
building-command-table.sts: calling handler <function add_waiters at
0xffffac810360> 2023-10-05 23:06:59,738 - MainThread - botocore.hooks -
DEBUG - Event building-command-table.sts: calling handler <bound method
AliasSubCommandInjector.on_building_command_table of
<awscli.alias.AliasSubCommandInjector object at 0xffffac8b6590>>
2023-10-05 23:06:59,739 - MainThread - awscli.clidriver - DEBUG -
OrderedDict([('role-arn', <awscli.arguments.CLIArgument object at
0xffffabf33a50>), ('role-session-name', <awscli.arguments.CLIArgument
object at 0xffffabf33b90>), ('policy-arns', <awscli.arguments.ListArgument
object at 0xffffabf33d90>), ('policy', <awscli.arguments.CLIArgument object
at 0xffffabf33f10>), ('duration-seconds', <awscli.arguments.CLIArgument
object at 0xffffabf388d0>), ('tags', <awscli.arguments.ListArgument object at
0xffffabf38990>), ('transitive-tag-keys', <awscli.arguments.ListArgument
object at 0xffffabf38a90>), ('external-id', <awscli.arguments.CLIArgument
object at 0xffffabf38c90>), ('serial-number', <awscli.arguments.CLIArgument
object at 0xffffabf38e10>), ('token-code', <awscli.arguments.CLIArgument
object at 0xffffabf39090>), ('source-identity', <awscli.arguments.CLIArgument
object at 0xffffabf39290>), ('provided-contexts',
<awscli.arguments.ListArgument object at 0xffffabf393d0>)]) 2023-10-05
23:06:59,755 - MainThread - botocore.hooks - DEBUG - Event
building-argument-table.sts.assume-role: calling handler <function
add_streaming_output_arg at 0xffffac7fe840> 2023-10-05 23:06:59,756 -
MainThread - botocore.hooks - DEBUG - Event
building-argument-table.sts.assume-role: calling handler <function
add_cli_input_json at 0xffffacc6c720> 2023-10-05 23:06:59,756 - MainThread -
botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role:
calling handler <function add_cli_input_yaml at 0xffffacc6c7c0> 2023-10-05
23:06:59,756 - MainThread - botocore.hooks - DEBUG - Event
building-argument-table.sts.assume-role: calling handler <function
unify_paging_params at 0xffffaca2b920> 2023-10-05 23:06:59,797 - MainThread -
botocore.loaders - DEBUG - Loading JSON file:
/usr/local/aws-cli/v2/2.13.19/dist/awscli/botocore/data/sts/2011-06-15/paginators-1.json
2023-10-05 23:06:59,797 - MainThread - botocore.hooks - DEBUG -
Event building-argument-table.sts.assume-role: calling handler
<function add_generate_skeleton at 0xffffac935ee0> 2023-10-05 23:06:59,797 -
MainThread - botocore.hooks - DEBUG - Event
before-building-argument-table-parser.sts.assume-role: calling handler
<bound method OverrideRequiredArgsArgument.override_required_args of
<awscli.customizations.cliinput.CliInputJSONArgument object at 0xffffabf39790>>
2023-10-05 23:06:59,797 - MainThread - botocore.hooks - DEBUG - Event
before-building-argument-table-parser.sts.assume-role: calling handler <bound
method OverrideRequiredArgsArgument.override_required_args of
<awscli.customizations.cliinput.CliInputYAMLArgument object at 0xffffabf39e90>>
2023-10-05 23:06:59,797 - MainThread - botocore.hooks - DEBUG - Event
before-building-argument-table-parser.sts.assume-role: calling handler
<bound method GenerateCliSkeletonArgument.override_required_args of
<awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object
at 0xffffabefa390>> 2023-10-05 23:06:59,797 - MainThread - botocore.hooks -
DEBUG - Event building-command-table.sts_assume-role: calling handler
<function add_waiters at 0xffffac810360> 2023-10-05 23:06:59,797 - MainThread
- botocore.hooks - DEBUG - Event building-command-table.sts_assume-role:
calling handler <bound method AliasSubCommandInjector.on_building_command_table
of <awscli.alias.AliasSubCommandInjector object at 0xffffac8b6590>>
2023-10-05 23:06:59,798 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.role-arn: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,798 - MainThread - botocore.hooks - DEBUG -
Event process-cli-arg.sts.assume-role: calling handler
<awscli.argprocess.ParamShorthandParser object at 0xffffad4844d0>
2023-10-05 23:06:59,799 - MainThread - awscli.arguments - DEBUG -
Unpacked value of 'arn:aws:iam::xxxxxxxxxxxx:role/dockertestAssumeRole'
for parameter role_arn: 'arn:aws:iam::xxxxxxxxxxxx:role/dockertestAssumeRole'
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.role-session-name: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0> 2023-10-05
23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event process-cli-arg.sts.assume-role: calling handler
<awscli.argprocess.ParamShorthandParser object at 0xffffad4844d0>
2023-10-05 23:06:59,799 - MainThread - awscli.arguments - DEBUG - Unpacked
value of 'dockertest' for parameter role_session_name: 'dockertest'
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.policy-arns: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0> 2
023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.policy: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.duration-seconds: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.tags: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.transitive-tag-keys: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0> 2023-10-05
23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.external-id: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event load-cli-arg.sts.assume-role.serial-number: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event process-cli-arg.sts.assume-role: calling handler
<awscli.argprocess.ParamShorthandParser object at 0xffffad4844d0>
2023-10-05 23:06:59,799 - MainThread - awscli.arguments - DEBUG -
Unpacked value of 'arn:aws:iam::xxxxxxxxxxxx:mfa/dockertest' for
parameter serial_number: 'arn:aws:iam::xxxxxxxxxxxx:mfa/dockertest'
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG - Event
load-cli-arg.sts.assume-role.token-code: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event process-cli-arg.sts.assume-role: calling handler
<awscli.argprocess.ParamShorthandParser object at 0xffffad4844d0>
2023-10-05 23:06:59,799 - MainThread - awscli.arguments - DEBUG -
Unpacked value of '006754' for parameter token_code: '006754'
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event load-cli-arg.sts.assume-role.source-identity: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event load-cli-arg.sts.assume-role.provided-contexts: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event load-cli-arg.sts.assume-role.cli-input-json: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event load-cli-arg.sts.assume-role.cli-input-yaml: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event load-cli-arg.sts.assume-role.generate-cli-skeleton: calling handler
<awscli.paramfile.URIArgumentHandler object at 0xffffabefadd0>
2023-10-05 23:06:59,799 - MainThread - botocore.hooks - DEBUG -
Event calling-command.sts.assume-role: calling handler <bound method
CliInputArgument.add_to_call_parameters of
<awscli.customizations.cliinput.CliInputJSONArgument object at
0xffffabf39790>> 2023-10-05 23:06:59,800 - MainThread - botocore.hooks -
DEBUG - Event calling-command.sts.assume-role: calling handler
<bound method CliInputArgument.add_to_call_parameters of
<awscli.customizations.cliinput.CliInputYAMLArgument object at 0xffffabf39e90>>
2023-10-05 23:06:59,800 - MainThread - botocore.hooks - DEBUG -
Event calling-command.sts.assume-role: calling handler <bound method
GenerateCliSkeletonArgument.generate_skeleton of
<awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument
object at 0xffffabefa390>> 2023-10-05 23:06:59,800 - MainThread -
botocore.credentials - DEBUG - Looking for credentials via: assume-role
2023-10-05 23:06:59,800 - MainThread - botocore.credentials - DEBUG -
Looking for credentials via: assume-role-with-web-identity 2023-10-05 23:06:59,
800 - MainThread - botocore.credentials - DEBUG - Looking for credentials
via: sso 2023-10-05 23:06:59,800 - MainThread - botocore.credentials -
DEBUG - Looking for credentials via: shared-credentials-file
2023-10-05 23:06:59,800 - MainThread - botocore.credentials - INFO -
Found credentials in shared credentials file: /tmp/.aws/credentials
2023-10-05 23:06:59,801 - MainThread - botocore.loaders -
DEBUG - Loading JSON file:
/usr/local/aws-cli/v2/2.13.19/dist/awscli/botocore/data/endpoints.json
2023-10-05 23:06:59,876 - MainThread - botocore.hooks - DEBUG -
Event choose-service-name: calling handler <function handle_service_name_alias
at 0xffffae1b0ea0> 2023-10-05 23:06:59,917 - MainThread - botocore.loaders -
DEBUG - Loading JSON file:
/usr/local/aws-cli/v2/2.13.19/dist/awscli/botocore/data/sts/2011-06-15/endpoint-rule-set-1.json
2023-10-05 23:06:59,918 - MainThread - botocore.loaders - DEBUG - Loading JSON
file: /usr/local/aws-cli/v2/2.13.19/dist/awscli/botocore/data/partitions.json
2023-10-05 23:06:59,918 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sts:
calling handler <function add_generate_presigned_url at 0xffffae6f13a0>
2023-10-05 23:06:59,919 - MainThread - botocore.configprovider - DEBUG -
Looking for endpoint for sts via: environment_service 2023-10-05 23:06:59,919
- MainThread - botocore.configprovider - DEBUG - Looking for endpoint for sts
via: environment_global 2023-10-05 23:06:59,919 - MainThread -
botocore.configprovider - DEBUG - Looking for endpoint for sts via:
config_service 2023-10-05 23:06:59,919 - MainThread -
botocore.configprovider - DEBUG - Looking for endpoint for sts via:
config_global 2023-10-05 23:06:59,919 - MainThread - botocore.configprovider -
DEBUG - No configured endpoint found. 2023-10-05 23:06:59,937 - MainThread -
botocore.endpoint - DEBUG - Setting sts timeout as (60, 60)
2023-10-05 23:06:59,939 - MainThread - botocore.regions - DEBUG -
Calling endpoint provider with parameters: {'Region': 'us-east-2',
'UseDualStack': False, 'UseFIPS': False, 'UseGlobalEndpoint': False}
2023-10-05 23:06:59,939 - MainThread - botocore.regions - DEBUG -
Endpoint provider result: https://sts.us-east-2.amazonaws.com 2023-10-05
23:06:59,939 - MainThread - botocore.hooks - DEBUG -
Event provide-client-params.sts.AssumeRole: calling handler
<function base64_decode_input_blobs at 0xffffac8859e0>
2023-10-05 23:06:59,939 - MainThread - botocore.hooks - DEBUG -
Event before-parameter-build.sts.AssumeRole: calling handler
<function generate_idempotent_uuid at 0xffffae1b3100> 2023-10-05
23:06:59,939 - MainThread - botocore.hooks - DEBUG -
Event before-call.sts.AssumeRole: calling handler <function
inject_api_version_header_if_needed at 0xffffae1d4c20> 2023-10-05
23:06:59,939 - MainThread - botocore.endpoint - DEBUG - Making request for
OperationModel(name=AssumeRole) with params: {'url_path': '/', 'query_string':
'', 'method': 'POST', 'headers': {'Content-Type':
'application/x-www-form-urlencoded; charset=utf-8',
'User-Agent': 'aws-cli/2.13.19 Python/3.11.5 Linux/4.14.255-318-256.530.amzn2.aarch64
exec-env/AWS_Lambda_Image exe/aarch64.amzn.2 prompt/off
command/sts.assume-role'}, 'body': {'Action': 'AssumeRole',
'Version': '2011-06-15', 'RoleArn':
'arn:aws:iam::xxxxxxxxxxxx:role/dockertestAssumeRole',
'RoleSessionName': 'dockertest', 'SerialNumber':
'arn:aws:iam::xxxxxxxxxxxx:mfa/dockertest', 'TokenCode': 'xxxxxx'},
'url': 'https://sts.us-east-2.amazonaws.com/',
'context': {'client_region': 'us-east-2', 'client_config':
<botocore.config.Config object at 0xffffabf38a10>, 'has_streaming_input':
False, 'auth_type': None}} 2023-10-05 23:06:59,940 - MainThread -
botocore.hooks - DEBUG - Event request-created.sts.AssumeRole:
calling handler <bound method RequestSigner.handler of
<botocore.signers.RequestSigner object at 0xffffaba13010>>
2023-10-05 23:06:59,940 - MainThread - botocore.hooks - DEBUG -
Event choose-signer.sts.AssumeRole: calling handler
<function set_operation_specific_signer at 0xffffae1b2fc0> 2
023-10-05 23:06:59,940 - MainThread - botocore.auth - DEBUG -
Calculating signature using v4 auth. 2023-10-05 23:06:59,940 - MainThread -
botocore.auth - DEBUG - CanonicalRequest: POST /
content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.us-east-2.amazonaws.com x-amz-date:20231005T230659Z
content-type;host;x-amz-date
xxxxxxxxxx
2023-10-05 23:06:59,940 - MainThread - botocore.auth - DEBUG -
StringToSign: AWS4-HMAC-SHA256 20231005T230659Z
20231005/us-east-2/sts/aws4_request
xxxxxxx
2023-10-05 23:06:59,940 - MainThread - botocore.auth - DEBUG - Signature:
xxxxxxxx
2023-10-05 23:06:59,940 - MainThread - botocore.endpoint - DEBUG -
Sending http request: <AWSPreparedRequest stream_output=False, method=POST,
url=https://sts.us-east-2.amazonaws.com/, headers={'Content-Type':
b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent':
b'aws-cli/2.13.19 Python/3.11.5 Linux/4.14.255-318-256.530.amzn2.aarch64
exec-env/AWS_Lambda_Image exe/aarch64.amzn.2 prompt/off
command/sts.assume-role', 'X-Amz-Date': b'20231005T230659Z', 'Authorization':
b'AWS4-HMAC-SHA256
Credential=xxxxxxxxxxxxxxxx/20231005/us-east-2/sts/aws4_request,
SignedHeaders=content-type;host;x-amz-date,
Signature=xxxxxxxxx',
'Content-Length': '218'}> 2023-10-05 23:06:59,941 - MainThread -
botocore.httpsession - DEBUG - Certificate path:
/usr/local/aws-cli/v2/2.13.19/dist/awscli/botocore/cacert.pem
2023-10-05 23:06:59,941 - MainThread - urllib3.connectionpool -
DEBUG - Starting new HTTPS connection (1):
sts.us-east-2.amazonaws.com:443 2023-10-05 23:07:00,070 -
MainThread - urllib3.connectionpool - DEBUG -
https://sts.us-east-2.amazonaws.com:443 POST / H
TTP/1.1 403 401 2023-10-05 23:07:00,071 - MainThread -
botocore.parsers - DEBUG - Response headers:
{'x-amzn-RequestId': '756c81ba-0233-4687-bc21-8aaffbc32503',
'Content-Type': 'text/xml', 'Content-Length': '401', 'Date':
'Thu, 05 Oct 2023 23:06:59 GMT'} 2023-10-05 23:07:00,071 - MainThread -
botocore.parsers - DEBUG - Response body: b'<ErrorResponse
xmlns=https://sts.amazonaws.com/doc/2011-06-15/>\n <Error>\n
<Type>Sender</Type>\n <Code>AccessDenied</Code>\n <Message>User:
arn:aws:iam::xxxxxxxxxxxx:user/dockertest is not authorized to perform:
sts:AssumeRole on resource:
arn:aws:iam::xxxxxxxxxxxx:role/dockertestAssumeRole</Message>\n
</Error>\n <RequestId>756c81ba-0233-4687-bc21-8aaffbc32503</RequestId>\n
</ErrorResponse>\n' 2023-10-05 23:07:00,072 - MainThread - botocore.hooks -
DEBUG - Event needs-retry.sts.AssumeRole: calling handler
<bound method RetryHandler.needs_retry of
<botocore.retries.standard.RetryHandler object at 0xffffabdd89d0>>
2023-10-05 23:07:00,072 - MainThread - botocore.retries.standard -
DEBUG - Not retrying request. 2023-10-05 23:07:00,072 - MainThread -
botocore.hooks - DEBUG - Event after-call.sts.AssumeRole: calling handler
<bound method RetryQuotaChecker.release_retry_quota of
<botocore.retries.standard.RetryQuotaChecker object at 0xffffabdd8410>>
2023-10-05 23:07:00,072 - MainThread - awscli.clidriver - DEBUG -
Exception caught in main() Traceback (most recent call last):
File awscli/clidriver.py, line 460, in main File awscli/clidriver.py,
line 595, in __call__ File awscli/clidriver.py, line 798, in __call__
File awscli/clidriver.py, line 929, in invoke File awscli/clidriver.py,
line 941, in _make_client_call File awscli/botocore/client.py, line 341,
in _api_call File awscli/botocore/client.py, line 697,
in _make_api_call botocore.exceptions.ClientError:
An error occurred (AccessDenied) when calling the AssumeRole operation:
User: arn:aws:iam::xxxxxxxxxxxx:user/dockertest is not authorized to
perform: sts:AssumeRole on resource:
arn:aws:iam::xxxxxxxxxxxx:role/dockertestAssumeRole An error occurred
(AccessDenied) when calling the AssumeRole operation: User:
arn:aws:iam::xxxxxxxxxxxx:user/dockertest is not authorized to perform:
sts:AssumeRole on resource:
arn:aws:iam::xxxxxxxxxxxxx:role/dockertestAssumeRole",
"error_type": "invocation/42a93201-0297-4b58-9398-4f3ee5e0bd2e"
}I can see above that the .aws file is now being written to the tmp directory. I can also see that my MFA code is passed in correctly.
Is it just me or does it seem like there are some repetitive (excessive?) calls in the above stack trace? The Lambda function takes a long time to run also.
Now one odd thing occurred here. My updated role trust policy seemed to revert to the prior version so I added back in the asterisk to allow any user to assume the role. I also removed the requirement for MFA.
I still get the error.
Now here’s the thing. This code works in a local container with the exact same role that is used by Lambda. The credentials are the same. So I do not believe that my code is the problem.
It could be that the temporary file written when the role gets assumed causes an error, but I cannot see any indication that is the problem above?
I ran a whoami in my local container and the local container runs as root. However, when it gets deployed to Lambda, a new and more restricted user gets created and the file system is read only.
It is difficult to understand or troubleshoot what is going on without more information. The error messages do not seem to be complete or precisely accurate. The container provided for local testing should more closely match what is available on Lambda, or provide instructions to create a user and file system with the same permissions once the base container has been configured.
As of now I feel a bit stuck. I’m leaving it at this point and will come back later if I think of something else to test. Maybe some kind person at AWS will DM me on Twitter, LinkedIn, or Mastodon and tell me what the problem is because this doesn’t feel like a support issue. It feels more like a bug, documentation issue, or something that should be addressed with the Lambda RIE. But maybe I’ll think of something I missed later…
I tried to run this one more time and then I got this error after 30 seconds.
Well, I’ll revisit this another time.
Update: Pretty sure this is caused by an internal network issue at AWS but kind of hard to tell from a customer vantage point without an accurate error message.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab