What’s Taking Me So Long To Write This Code?

ACM.421 Slowing down to reflect to speed things up

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: AWS Organizations | IAM | Deploying a Static Website

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I wrote about deploying an AWS Account for a specific project using my container that deploys resources on AWS.

Sometimes, when something is taking longer than it feels like it should (always) I stop to reflect on what is taking me so long and how I can go faster.

Things that worked before no longer work. Well, I tried to manually spin up an Ubuntu instance with instructions I documented that worked before and now they aren’t working. I wasn’t using my automated version of the deployment so maybe that will work better. I may also be using too small of an instance. These things are frustrating — whether it’s caused by a cloud system or an operating system or some other software I’m using but are generally out of my control. I just have to troubleshoot and fix it. If I have to do this too much with a particular software, cloud, or operating system, I’ll likely look for something better that doesn’t require so much time for things that are out of my control.

Update: I started from a copy of an Ubuntu image I created previously, ran all the updates and my steps again on a larger instance size and got it to work. If the instance size was an issue should be reported somewhere so it’s easy for the user to tell that’s the problem.

Here’s another one — KMS just stopped working.

I figured out why but it’s a total distraction and now I can’t create a new KMS endpoint until the other one is completely gone. I’m getting errors about conflicting DNS names even though FIPS and non-FIPS endpoints should have different DNS names, right?

This one just took me half a day to troubleshoot.

These things are distracting and take time.

In which account is my role assumption? I use a global variable for the AWS CLI profile name currently being used by my code that defines the credentials used for an AWS deployment. Those credentials relate to the account where any commands I execute will be run. I commented out my code that creates a new account where I also assume the role in the new account to deploy an IAM account alias.

That design could be better. Proper dependencies will save me time. The role switching should not be dependent on the AWS account creation code. I added the account role assumption call to every account resource but I do not repeat the role assumption if the current profile is the account profile. I added echo commands to indicate each time I switch roles so I can see what profile is currently active and where resources are getting deployed.

>>>> Luckily, I took the time to create separate users and roles and the role I’m using for deployments to accounts does not have permission to deploy things in the root account or I would have a big mess. Instead, I just get a permissions error.

Typos due to manually typed parameters on each run. I get random errors and have to figure out why. In the case of the following I inadvertently created a new account when I did not intend to do so. I need to move my account configuration to the SSM Parameter Store to avoid manually typing parameters and select the type of account I want to deploy from a list or allow the user to create a new one to help prevent this mistake in the long run.

AWS Functions Stop Working: Suddenly cannot switch accounts anymore. Why? Is this because I’m blocking regions where I don’t want to make requests? If I’m switching accounts shouldn’t that be in the global region (but not in Asia if I’m in the US and vice versa?)

Ubuntu on AWS requires IPv6 so I had to rearrange my network to get updates since I don’t use IPv6. Why?

IPv6 is not always “better” or “more secure.”

Companies that require it waste my time. I recently saw a blog by APNIC (explained in my networking posts) suggesting that maybe the smallest size block of IPs reserved by a company should be lowered to help alleviate wasted IPv4 addresses. I have mixed feelings on this but it might help, as long as companies — mostly in Asia and Russia do not request individual IP addresses for 256 addresses in a single company. All the addresses for a single company should be concatenated and consolidated where possible.

Improper use of source control. I had a working networking script to deploy subnets that doesn’t look like it’s going to deploy at all now. Did I do that? Looking at my repository and trying to figure out what’s going on. Need to get a handle on my git check-ins and do it more often. I’ve been holding off because I don’t want to push broken code to GitHub. Need a private and public repo setup which I will do and write about later.

Meanwhile, fixing subnets.

Not taking the time to do it right in the first place. On the topic of subnets I created a way to override the starting avaiability zone when deploying a subnet because the first one may not be avaialable. I made a note in the comments that I really should look up the next available az instead. And as fate would have it, my deployment failed because the first zone was not available.

Doing it right now…

Migrating code: Although I already have all the code written I need to adjust it for the various things I wrote about in this post:

Incorrect values in variables: I keep making stupid typos along the way. I really wish I would stop doing that. One thing that just tripped me up was that a name I was concatenating into a variable was not correct. Need to validate each value along the way.

ImportValue with Sub and Joins in CloudFormation are really hokey looking and anything that looks suspect is generally the source of tricky errors. And it is. I also have to wait for the CloudFormation template to run to diagnose the problem. Removing those almost everywhere in favor of functions to retrieve IDs, written once so accurate, and passing in the IDs instead.

Inconsistencies. Remembering and keeping my naming conventions consistent so I don’t have to keep looking everything up. I deployed a new script in my deploy folder. I forgot to put “role” at the end of the name of the folder the scripts are in, for example.

Positional Parameters as opposed to named parameters are problematic. If a value is missing I get the wrong parameter in the wrong slot and my validation routine reports that the wrong variable is missing. Long term I will change that.

AWS CLI query language is a beast. For me anyway. And I’ve been programming for maybe 30 years now and pick up new languages pretty fast. No matter how long I’ve worked with it I can’t remember how to query simple things like an AZ ID by Index which is tripping me up at the moment. I end up resorting to sed and cut because honestly, I hate it. But I keep trying to ingrain it into my brain. I think the back tics throw me off. Symbols, in general, can cause confusion in my brain so maybe it’s just me.

Also, I’ve never stopped to really grasp the logic of it — which if I did — would probably make it simple. I think I’ve figured it out at times but then forgotten due to infrequency of use. Anyway. I need to figure something out using it right now. Perhaps I should just reverse engineer and write a blog post on it to explain the logic that I myself can reference if I forget.

This page has some nice examples:

So after going to dinner and coming back and looking at the above examples I figured out what I was doing wrong. My memory was not that far off. I had this:

  azid=$(aws ec2 describe-availability-zones \
  --query AvailabilityZones[].ZoneId \
  --profile $profile)

Instead of this:

  azid=$(aws ec2 describe-availability-zones \
  --query 'AvailabilityZones[].ZoneId' \
  --profile $profile)

So first of all — really the AWS CLI can’t figure out I’m missing the quotes and run that command? And second — if you really can’t, you can easily tell me I’m missing a single quote after the query command. That would ahve saved me a lot of time.

Now I have to figure out how to get back only the zone ID I want based on the index. This is the part where I say forget you AWS Query language. I’m just going to use command line tools.

But I figured it out. Here’s another thing I don’t like. Sorry, things that are not intuitive make me grumpy.

I was trying to write the query to use a variable. I jumped through all kinds of hoops trying to make it work in one command. Finally I gave up and just split it two like this. Quotes, in general, seem to be handled inconsistently and be problematic at times both in bash and the CLI. Yes, I should use a real languge — later.

  query='AvailabilityZones[?ends_with(ZoneId,`az'$index'`)].ZoneName'
  azname=$(aws ec2 describe-availability-zones \
  --query $query \
  --profile $profile \
  --output text)
  echo $azname

Now on the other hand, if you try to take the above approach with a filter it won’t work, because the filter implementation is completely different. Why is there not just one query option with the a flag indicating whether it is client side or server side? I don’t know but that would be easier. That said, I found examples find a resource by a tag using a filter but have yet to find that using a query. I’m not sure why but not expending more brain time thinking about it.

Generated code helps, but can’t do everything. Though my code generator writes a lot of the base code, I still have to fill in the parameters that need to be passed from the deployment script to the function and then to the template.

Forgetting the architecture.

In oder to help me focus on the steps I need to take, I need to keep my architecture top of mind. I have a common architecture to keep things simple and remembering this will help me move faster.

Parameters > Deploy > Function > Template

• Recall that I use the deployment scripts as a mechanism to define who owns and can run what deployment deployments. They also contain resource-specific parameters. These scripts increase the amount of code to a degree, but they make it easier to deploy individual resources instead of monolithic CloudFormation stacks.
• The functions scripts are common code used to execute templates. A single function may be able to deploy many different templates. A base function may be called by many functions that generate and modify inputs. The functions help me reduce the amount of code I need to write.
• The templates define the resource that I want to end up with in my Account, once deployed. The templates are really data, or metadata, not executable code. They are a definition of a resource. This is where some people get confused and think why can’t I just write executable statements to deploy everything? Well you can but then you lose these benefits. I explained that in this post, and by the way the CloudFormation team is currently making it faster! 🙏

Validation routines not always working. For some reason my validate_set function just wasn’t working. Validating variables are set when they should be saves me a lot of time. When it doesn’t work, it can waste a lot of time. I need to figure out what caused that and fix it.

Update: Fixed. I forgot that I added in the “caller” to this function and once again positional arguments threw me for a loop. I added additional logic to check for the existence of all variables.

Need more consistency in common function names. I keep revising my code to become more and more consistent. I need to add a get_[resource]_id function to every resource and add the AWS CLI call to look up that resource by the name. I’ve been creating functions for this purpose with all different names and if I name these functions consistently I’ll know what function to call to get a resource ID without having to go back and look at my prior code to figure out the name of the function.

Every function file will have the following:

• deploy_[resourcename]
• get_[resource]_id

Inconsistent AWS outputs: I have to fix the outputs in every new template I write. It would be easier if AWS consistently provided an ID and an ARN for every resource but they don’t. So I add my common code to add both and either I have to check the AWS documentation or run the template and get an error and then fix it. If there’s no ARN I just return the ID for both. If there’s no ID I return the ARN for both. But at least my templates are consistent.

Not testing independent functions before trying to use them. When I create a function to return an ID, for example, I should test in independently from the rest of my code. On that note, I hope everyone is using small functions rather than monolithic code. This aligns with the principle of abstraction I’ve written about many times. But when I try to write a function and use it before I test and make sure it works it’s just taking more time than to slow down and test it first.

So many files, so little time. I’m getting better at searching for files in my history either by grepping parts of the name or looking for recently edited files. Although the naming conventions are consistent and if I think about it I can remember the name, if I just run the command from history that’s generally faster. I was using the up arrow but I’m usually editing a bunch of files at once so I can use history commands instead.

history 50 | grep -v history | grep vi | sort | uniq

Then I can just rerun whatever command by line number.

! [line number]

Yes, an IDE would probably help but most of them are so clunky and no I do not want to use a browser based huge attack vector to edit files. The biggest problem I have with IDEs is the amount of network traffic they send back to vendors. Have you looked at all the network traffic when you open VS Code?

I want my editing to be done inside a VM on AWS, and not have my code stored in an IDE with a backend who knows where or sending a ton of data to the Mother Ship (my pet name for Microsoft as I began my tech career in the shadow of that tech giant in Seattle). I’ve used a lot of Microsoft tools over the years. But I want my development environment to run completely on my VM and not require 100 network connections when I fire it up.

In addition to mostly being a lot of overhead (though yes also very useful), the VMs with UIs cost more. Maybe later. I wish I could use a Mac on AWS but the price point is ridiculous. So is Apple network traffic.

Want to see all the network traffic your latest flashy development tool requires?

Every connection is a potential security problem. Yes, we cannot eliminate all network traffic, obviously, but fewer connections means less your security team has to monitor. And an insecure software update process led to the largest breach of the US government at the time it occurred.

Companies should design with minimization of network connections in mind to help make it easier for customer security teams to monitor for compromises like the above. If I could find an IDE that did that I might use it. In the meantime, I’m gong to check out CodeWhisperer for the AWS CLI and see if that too, requires excessive network connections, or can run on my VM.

I hope it works locally but not sure it can based on the underlying premise.

I’m getting older, I guess. I keep forgetting what I was doing. I was looking up the VPC. Oh yeah. What else does that subnets function need? Now suddenly a file I was editing in the history list seems to be gone. Oh yeah I’m in the wrong folder in that window. I read once that when you’re younger you may be able to work and think faster. But when you’re older your experience helps you avoid known mistakes. You understand what things have and haven’t worked in the past. You learn to leverage resources more efficiently. And that’s what this whole blog series is doing. There are also other reasons that can lead to this sort of spiral that you can get into when you’re in too much of a hurry.

Brain sprain? Can you sprain your brain? I often wonder about that. If your brain is a muscle can you over work it? I’ve been rushing to try to complete all this code all year and reaching the point of something really usable and cool but whenever I’m juggling client projects and trying to write the code and stressing over deadlines I feel like I’m spraining my brain at times — literally. That’s when I know I need to slow down a little bit. I’m prone to headaches more often these days and I wonder if that has something to do with too much time on the computer. Hope to cut that back in the future but I just want to get this code done! Perhaps something I will regret later. I hope not.

Thinking about too many things at once. My brain is an idea generating machine. The ideas and things I want to do come so fast sometimes I have to force myself to focus. I do this by writing down the next thing I need to do and try to stay focused on that. If I have some other thoughts that I’m worried I will forget later, I jot them down on some list somewhere and return to what I am supposed to be working on to meet my current objective. Focus is crucial for success. Read up on Warren Buffet’s recommendations regarding focus.

Stressing about not going fast enough. Sometimes you start spinning your wheels and try to go faster because you’re behind or on a deadline. This rarely helps. Focus is good. Avoid the extraneous. But stressing only leads to more mistakes. Sometimes I just stop to walk the dog when I feel like I’m starting to spin my wheels. Go get a cup of coffee. Give your brain a break. Come back and start over. I find that I get a lot of good ideas with a good night’s sleep or on a walk. You can only do so much and nothing is usually the end of the world. So stop and take a breather and think for a minute.

That’s why I started writing this post last night. I have all this what should be working code that I’m just trying to move over and run but it’s taking longer than it feels like it should. So I stopped to think about why that is and how I can do it faster. Just remembering the structure of what I am trying to write is helpful.

Deployment script > Function > Template

Each deployment script is in the folder of the person who owns it (and could ultimately be in a separate GitHub repository that anyone can read and only the team that owns it can change.)

Plodding through. I’m working through a list of resources I want to configure and deploy in my project account. I already have code for this for the most part. I’m just getting it into the new directory structure and scripts, running it, and fixing any issues. One step at a time. I run it, get an error. Fix. Repeat. I’m a little over halfway through the list but some resources take longer than others to adjust — especially since what was working code doesn’t look right anymore.

That’s what I could think of while working last night and this morning. I might come back and add a few things here if I find any other repeated problems I can somehow fix or feature requests.

Hopefully my next post will contain the complete list of resources and code to set up a new project account.

Taking the time now to save time and money in the long run

I definitely feel like I was faster at some things when I was younger, but overall I feel like I get things done faster by taking time to do the things right up front. This is where a lot of startups and companies fall down. They start fast with some new project or idea but in the long run it is not sustainable without the architecture necessary to support it long term as it grows. A solid deployment system will help you go faster in the long run and avoid mundane errors and mistakes.

That’s similar to what Nubank talked about in Werner Vogels’ recent AWS re:Invent keynote. They had to fix underlying stability problems which led to cost savings. In order to do that they needed visibility into the problems and if you design your architecture up front to provide that, troubleshooting issues will be easier when those problems arise.

You can prevent errors and data breaches by thinking through and developing common templates that help people deploy resources securely.

You cannot scale if you have five people writing the CloudFormation templates for 10,000 developers as I explained when I got to the cloud engineering team at Capital One, but you also cannot have developers deploying things they haven’t been trained to deploy (and number one on that list is usually networking). But you can speed up your network deployments if you automate things as I’ve written about in prior posts.

I worked for a company that wanted me to go “faster, faster.” I kept trying to explain that without the proper underlying architecture and infrastructure we’d be spinning our wheels trying to fix production and security problems later. One team that deployed faster than us had all their systems crash in production. Later, the company had a data breach. I don’t know if it was related to things I tried to explain to them while I was still there. Thinking through your architecture and taking the time to automate deployments can help you avoid those issues.

You can also reduce your costs in the long run by investing some time and money up front, as I kept trying to explain to that company. It takes more time to build the architecture, but you will be going so much faster later. I did some of that by architecting a system that allowed developers to deploy containers, rather than VMs. This was right before Lambda came out, which is essentially the same thing.

We also at one point created a mechanism to measure the cost of part of the architecture my employer wanted us to build so we could analyze the costs. I was trying to explain to them that their desired pricing model was infeasible and it was impossible to hit a fixed price point unless they could tell me exactly how much data the system was going to process — and they couldn’t. Their pricing model did not align with the cost of a cloud model. To understand what I’m talking about, check out Werner Vogels’ recent keynote.

Taking the time to build out automation, governance, and metrics may seem boring — but pay now or pay later.

One of my big motivations for this code, which really I have been working on off and on for a few years now, is to completely automate the deployment of a new penetration test account, environment, and deployment of all the test tools and reporting tools I use on each test. I had a lot of it automated before I started this latest series but now it’s more complete, more solid, and scalable to other types of resources. I have other plans for this code as well. 😉

I still have more to do to shave off some costs, but I can deploy all my resources using an nano arm EC2 instance, which is what I’ve been using in all my recent posts. That’s the cheapest size instance you can deploy. Unfortunately, Burp requires a lot more resources.

What is saving a ton of time

I think AWS added more explicit error messages showing exactly what permissions you are using when trying to take actions in the AWS console. This is saving me tons and tons of time compared to stopping to look up every error in CloudTrail.

Thank you thank you thank you thank you thank you thank you. 🩵

I know someone reading my posts made some requests to AWS on my behalf and provided my blog posts to them. Thank YOU also! 🩵

Back to it.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab