Securing Secrets Used By Your Code

How many chances do you give attackers to steal your secrets?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: AWS Security | Secure Code | Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Perhaps you read the last blog post on configuration and abstraction and you are all set to use configuration files to improve your multi-environment deployments. That’s great! However, a problem occurs when those files contain sensitive data and secrets. A secret is anything that is used to access systems and data. Here are some secrets you should protect from prying eyes:

• Database user names and passwords
• Keys and other credentials that allow you to call APIS
• Credentials for deployment systems
• Symmetric encryption keys used to encrypt and decrypt data
• Asymmetric private encryption keys used for communications
• SSH keys
• Developer, QA, and operation team credentials
• Credentials used by applications
• Basically any kind of user name and password.
• Other secret system configuration data that would aid in an attack.

First, I’ll explain the risks associated with how you may be storing secrets and where they can be accessed in a bit more detail. I’ll include some examples you may or may not have considered. Then I’ll tell you how to fix it.

When this chapter of my book is completed it will include working sample code similar to what I provide in one of my cloud security class labs. For now, I’ll give you an idea of how to proceed with some pseudo-code.

The problem with configuration files

In my talk at RSA with Kolby Allen called Red Team vs. Blue Team on AWS, Kolby used AWS sample code to deploy some resources in an AWS account. I already explained some of the potential security issues when using sample code from a vendor. In that presentation, I show how an attacker might try to get into systems in the cloud environment. While scanning the environment as I would in a penetration test, I end up finding a configuration file for a lambda function. As it turns out, a configuration file exists for the function in an S3 bucket and it contains a database user name and password.

If you don’t know what an S3 bucket or a lambda function is, just think of the function as source code running in some system. The S3 bucket is like a file folder that contains your configuration file, wherever it exists in whatever type of system you run. It could be in an unencrypted file on a server in your data center. It could exist on the machine of a developer or QA person who is testing the code. Perhaps the configuration file exists, unencrypted, in backup systems. Maybe it gets printed out in some log file as the application reads it.

As you can see, using configuration files for sensitive data can be problematic. If you do use one, make sure you use a separate file for your developer, QA, and production environment. Do not check those files into a source control system where many people can access the files.

There’s really no good way to store configuration files on a running system in a secure manner. Even if you encrypt them you have to make the encryption key accessible somehow. Where do you store that key? That’s the catch 22 when using configuration files.

Secrets in public source control systems

I spoke with Tanya Janca (@shehackspurple) at Microsoft Build a few years back before the pandemic brought my travels speaking on cybersecurity across the United States around the world to a halt. One of the presenters from Microsoft got up on stage and said the number one problem they see in terms of cloud security is secrets in code. He proceed to show how they searched GitHub and found many secrets checked into source control. Anyone can search through the code in public GitHub repositories. Anyone could find and use those secrets to access systems, databases, and cloud accounts.

I heard a similar talk on Slack tokens at OWASP AppSec Day in Melbourne, Australia. Someone from Slack explained how developers were exposing their credentials for the Slack API in many different ways. API keys are secrets just like database credentials or your network login for your company laptop. Slack has made some changes to how you need to access their APIs which helps, but you still need to handle any API credentials securely.

When AWS first came out, people would often check the developer credentials used to access cloud accounts into source control systems like GitHub. In some cases, attackers would scan GitHub to find those credentials and create numerous resources in AWS accounts, costing the owners thousands of dollars. Here’s one story by a person who was only using S3 and ended up with a $14,000 bill!

Many similar stories exist.

Since those times, AWS has started scouring GitHub for developer keys and disabling them once discovered. I participated in an event at AWS re:Invent when I started using AWS called Game Day. Everyone got in groups and tried to build infrastructure in AWS accounts. At the end of the competition, people stood up and talked about what they had learned. One individual told us how he had checked his developer credentials into GitHub accidentally during the competition and AWS almost immediately disabled those keys. Unfortunately, his team could not complete the contest and they lost — but he definitely learned something that day!

Secrets in internal systems

Perhaps you think that you are only storing your secrets in systems that are internal to your organization so you are safe. Those systems are not exposed to the entire Internet so it’s ok, right? Think again. Consider the Twitter breach where developers stored secrets in Slack accounts. Attackers got onto internal systems, found those secrets, and used them to steal other people’s accounts in that widely publicized breach.

When I first started working with AWS at Capital One (I left prior to the breach, not that I could have stopped it) I almost immediately noticed that secrets were available after deployments in the AWS console. Developers wrote code to deploy virtual machines using CloudFormation and something called user data to execute scripts when the VM started. That code would log in and execute commands. The logs available in the console would show the scripts that executed which, by initial design, included credentials to the central system that managed AWS deployments.

I wrote a blog about this on the internal Capital One blog platform. I was surprised to learn at a conference a couple of years back that some of my blogs still existed, though heavily edited. Who knows now. I wrote about how anyone who could log into the AWS console and had access to deploy virtual machines (called EC2 instances on AWS) could see the credentials for our deployment system. I wrote this post to inform developers that they should stop doing this and retrieve credentials another way.

Someone posted a comment on my blog post. “This is gold.” I hope it was someone on the Capital One red team not someone with more malicious intent! I explained what a red team is in the presentation at the start of this blog post and in more detail in my book on cybersecurity fundamentals.

Later, Amazon removed the ability to view this log data from the console. That’s great, but I’m going to show you how you should actually fix that problem by altering your code. Credentials you store in source control or embed in your code may show up in many places within systems, backups, application storage, and logs, not just the now-revised AWS console.

Secrets in running applications

When you deploy your applications on a virtual machine, physical server, in a container, or on a developer laptop, secrets may be exposed in many different places. Here are a few examples:

• Configuration files stored on the application
• Environment variables
• Memory on the machine executing the code
• Caches of any kind
• Execution and error logs
• Cloud virtual machine metadata
• S3 bucket metadata (which is not encrypted)
• Cloud shell history
• Bash file history
• Container layers

That’s just a list of places I can think of off the top of my head in a minute. There are many other places where secrets may exist if you don’t handle them with the utmost care.

Retrieve secrets from a secure secrets management system

When you use secrets in your code or other parameters which you want to ensure cannot be inadvertently or maliciously altered, store them in a secrets management system or secure deployment system. The type of system you use for this purpose will depend on the environment where you deploy your code.

Instead of embedding secrets in a configuration file, you can store them in a secure secrets repository. When you need to retrieve the secrets, you can write code to pull the secrets just at the time they are needed by the application, and then eliminate them from access by the application or in memory when no longer required.

When you are storing secrets for multiple environments, you need to ensure your secrets repository only allows access to the people who need a particular secret. Architecting a secrets management solution or secure deployment pipeline is beyond the scope of this book, but whatever system you use should prevent developers from accessing production secrets, for example. Additionally, applications that use secrets should only be able to retrieve their own secrets, not the secrets for some other application.

Secrets in clouds

If you are deploying code in a cloud environment, major cloud providers offer secrets management and configuration management solutions. Configuring these systems properly is beyond the scope of this book. However, I will show you how to obtain secrets from these systems, rather than storing them in a configuration file or your code.

AWS offers two solutions to handle configuration management. The first is called Systems Manager Parameter Store, but the better solution for secrets and sensitive data is called Secrets Manager. The latter service provides better protection for secrets because I noticed that AWS Parameter Store doesn’t hide secrets at all times when using AWS CloudFormation templates. Maybe that will change at some point but if it doesn’t I’ll probably explain it in more detail in my book. Additionally, the service is solely focused on protecting secrets. It’s easier to limit access and to rotate secrets, concepts covered in my book on cybersecurity fundamentals.

You can use Python or any of the other many AWS SDKs to pull secrets out of Secrets Manager when your code executes. Often when taking consulting calls from clients for IANS Research, people will express concern over how difficult it may be to change all their code. Writing the code to obtain a secret from AWS Secrets Manager is so much easier than writing the configuration file code I showed you in my last post on abstraction for configuration.

Are you ready? Here you go:

secret = client.get_secret_value(secret_name)

Phew, that was complicated, right?

Of course, there’s a bit more to it but that is the primary line of code you need to obtain a secret when you need it. You would need to instantiate an AWS client using Python as I explained in a prior post. This same concept may be used with numerous other languages. You can use this same concept in Lambda functions, containers running in Kubernetes or ECS, .NET applications.

You can also create secrets using CloudFormation:

The harder part of using Secrets Manager properly is not the code you write to obtain the secrets. The challenge will be to think through all the ways in which an attacker or malicious insider may obtain access to your secrets. That’s covered in my other security book. You’ll also need a secure deployment pipeline, access management for humans and systems, and ways to govern proper use of secrets through security policies. Those are all concepts beyond the scope of the code itself.

AWS also offers a separate service for managing encryption keys called Key Management Service (KMS). Encryption keys may be retrieved similarly to secrets and used to protect sensitive data. This solves the problem I mentioned earlier about where to put the encryption key for a configuration file in a cloud environment and acts as a secrets vault for encryption keys.

Azure and GCP also offer secrets and key management services:

Azure Key Vault

Google Secret Manager

Google Cloud Key Management

If you are in a multi-cloud environment you may want to consider a secrets management solution that works across all clouds. One popular solution for that is HashiCorp Vault.

Some developers want to use Docker Secrets or Kubernetes Secrets for secrets management. Although there are many ways to separate your secrets from your code, the bigger issue will be your deployment system and configuration management architecture. Without going into too much detail that deviates from the code itself, make sure any solution you choose meets your governance and access requirements. Those two topics were covered in my prior security book.

Secrets in on-premises applications

You could use cloud secrets managers to obtain secrets used by systems that run in your data center or on your development machines. There are a couple of issues with that. First, is where do you store the credentials used to call the cloud service APIs? Secondly, you may have too much latency when trying to obtain secrets for applications. You’re also introducing additional risk by accessing them over disparate networks. The same would be true if you use a secrets manager hosted in your data center from within the cloud.

Some other secret managers exist that you can use on-premises, or in other words in your data center or from systems on your company network.

• HashiCorp Vault works on-premises as well as in the cloud.
• Venafi is a well-known solution for managing TLS certificates. (Please don’t tell me you are managing those in an Excel file as one security vendor I know was doing.)
• Too many other tools to list here want to manage your secrets.

Sometimes tools used to manage application deployments have built-in secrets management. One tool I used in the past did this pretty well but it’s so long ago that other better solutions may now exist. I’m not even sure the product still exists so instead I’ll explain how it worked.

• In this particular product, you could create a deployment pipeline that would deploy the same application to a development environment, QA, or production environment.
• A central team wrote the deployment scripts for the applications used within the company. Other teams used the deployment pipelines to deliver code to the different environments. This provides appropriate separation of duties, a concept explained in my other security book.
• A set of parameters the application required got defined in the system by the developers.
• Each environment (dev, QA, Prod) had its own set of parameters.
• Only developers could read or change the development values, QA could change the QA values, and the operations team could set the production values.

The key point here is that whatever product you choose should allow your code to interact with the system and retrieve secrets in a secure manner. Only the people and applications that need a particular secret should have access to it.

People always want to ask me what product to use on consulting calls or what everyone else is using. I repeatedly tell people that I could not possibly know or use every product in existence, but I can provide recommendations on how to evaluate products. If I have performed a security assessment on a particular product I will be able to provide some advice as to whether or not you should use it. I don’t base that recommendation on how many people use the product. If someone is recommending a product, ask them if they are paid by that company for speaking about the product.

Performing a proper security assessment on products, services, and vendors is crucial as I explain in my Cybersecurity for Executives in the Age of Cloud. I presume someone did a security evaluation of this product prior to selecting it for my company, but I wasn’t involved. There are so many tools that want to manage your secrets and deploy your code. Be very careful when selecting this type of product since it has the keys to your kingdom!

Reduce the risk of secrets exposure in running applications

The one thing you probably cannot eliminate in most cases would be secrets in memory, but you can minimize your exposure. At some point, the application executing the code needs to obtain unencrypted data to understand and act on the values.

Often attackers will try to find a way to access system memory to obtain sensitive data and secrets. They may try to read the memory directly. In some cases, the attacks cause the system to dump system memory into a file or error logs. They could also try to latch onto a service that has access to the memory using various forms of process injection, sometimes called file-less malware attacks.

Advanced forms of protecting system memory exist involving hardware and encryption techniques. Some hardware will store data in a protected location. Advanced encryption research tries to find ways to act on encrypted data but so far that has limited use cases due to the type of data that those systems can process. But this book is focused on writing secure code so we’ll stick with the ways you can protect the data with your code.

Accessing the secrets right when you need them, reduces the risk of exposure. You’ll also want to make sure you limit access to secrets during and after execution of the code that requires them. Avoid printing out secrets where they may end up in system logs. Avoid storing secrets in caches.

Here’s an example:

Secrets in bash history is a MITRE attack vector, a concept I introduced in my prior book so not going to explain it further here. If you are using a system or code that prints out secrets when it executes, make sure your clear your logs or history as soon as the application no longer needs the secret.

Run this command on Linux or a Mac to see your history:

history

Run this command to clear it in any scripts where needed:

history -c

Understand how your system caches data and prevent secrets from ending up in that type of storage. Often attackers leverage attacks that try to obtain data from caches. This attack vector is covered by CAPEC-204 from Mitre. If you must allow secrets to enter a cache, they should be encrypted, but hopefully, you can pull the secret from your secrets vault uncached and remove it from your system prior to caching any data.

The other thing you can do to reduce the risk of stolen secrets is to ensure they are cleared from memory when you are finished using them. Some types of software programming languages and frameworks clear out system memory when you are done using a variable, for example. This may not happen immediately. You can set the value of the variable containing sensitive data to null or None or whatever construct your programming language uses to wipe the data from system memory.

With some programming languages, you may be able to force the system to wipe out the value in memory. However, sometimes you are at the mercy of the application language you chose. For example, Java uses something called a garbage collector. It will periodically clear system memory. You can try to persuade it to clear the values you have in memory

Object obj = new Object();
obj = null; 
System.gc();

I pulled this example from a Stack Overflow thread:

The responses to the person’s question to clear system memory are interesting. One person’s answer: Why would you want to do that? This person may have never considered the fact that a sensitive value may be sitting in system memory, accessible to attackers and the person wants to clear it out as soon as possible.

The useful point of the answer is in this statement:

“Manually” garbage collecting by setting things to null likely slows things down if only because you pay the cost of nulling references pointlessly, or, invoking GC more times than is needed.

It will be important to understand in this case how calling the garbage collector or nullifying the value could impact system performance. Rather than taking someone’s word for it, you could test this theory and work towards a solution that both removes the value from memory as soon as possible within adversely affecting your performance.

It may be that you get a slight performance hit, but if you only use the variable briefly in one place and are concerned about security, the performance hit may be worth it. You may also be able to architect your system in such a way the performance hit does not impact the main application once you have a database connection (an incomplete thought I haven’t really fleshed out).

This is the point where you get into software engineering, versus taking someone’s word for it or simply copying scripts off the Internet. Dig deeper to understand how variables and memory work in your programming language of choice. You may even want to go further to explore what type of attacks could expose system memory by researching MITRE ATT&CK in more depth and designing your defenses accordingly.

If you liked this post, check out this one which shows how to create a generic function for all your Lambda functions. Write your secrets management code once.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab