Concurrency is Everywhere

When the order of things leads to a cybersecurity problem

One of my post that may later become a book on Secure Code. Also one of my posts on Application Security.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

You want your application to run faster. You notice that sometimes it’s waiting around for some action outside the application itself to complete. You decide that you will change your program to send the request to start the external action and while you are waiting for it to complete you’ll use your system processing power to do other things. Then you’ll come back and check on that other item later and continue processing where you left off.

You would be using a concept called concurrency. If two actions are concurrent they are happening at the same time. In software programming, concurrency relates to managing different actions happening at the same time in your application, while still achieving an accurate outcome. That last part, ensuring that you get the correct results, is where things can get tricky. I’ll explain some of the things that can go wrong with concurrency that can lead to a security problem in this post.

Entire books have been written about most of the topics in this book. Concurrency is no different. If you want to know how to implement concurrency with threads, locks, mutexes, and semaphores you’ll need to refer to other sources. One of my favorite books on this topic is Java Concurrency in Practice by Brian Goetz.

If you program in some other language such as .NET or go the concepts will be very similar. However, you may want to pick up a book in that language instead. A book that covers the topic with code samples for your particular language will be very helpful in trying to understand and write concurrent code properly.

Single Versus Multi-Threaded Programs

When a program runs, it can run every line of code for the entire application one after the other in the order they are written. The lines of code execute in what is known as a thread.

When you want to send a request and do something else while waiting for a response you might use more than one thread. Each thread executes a number of lines of code that need to execute in a particular order but they can run at the same time. Now you need to manage when to wait for something to finish and when to switch to another thread and determine when all the lines of code are complete.

Seems simple enough, right? It’s not. Managing the state of a program that includes concurrency can be very difficult to get right. I remember working on a tax application for a large company. One of the developers creating our application was trying to write the code involving concurrency and it was so complex that it was delaying the project. Eventually, the architects got involved. I was able to pick up the code and finish it based on their recommendations, but I can attest that concurrency is very tricky to get right.

How do you troubleshoot an application that is processing data with many different threads? An error occurs. Which thread was it? Have you locked the correct threads so they cannot operate on the same data at the same time? Are the steps in the threads all occurring in the correct order? Do you need transactions to ensure actions that need to complete together do so properly? Are you counting data as it gets processed and how do you ensure your counter is always accurate? Have you correctly instantiated data prior to access? How does your application know when all the threads are done so it doesn’t hang or crash?

Numerous constructs exist within a programming language to help you with all those issues. Some programming languages like go and rust claim to prevent concurrency bugs. I wrote a blog post about go when I was revisiting it for potential use in some pentesting projects.

Let’s dive into the types of problems you might have in a program leveraging concurrency and how that might lead to a security problem.

Race Conditions (Out of order operations)

As I mentioned in the blog post on secure transactions most financial transactions require multiple steps. If these steps occur out of order, you could have a bug that allows attackers to steal money.

One of the classic examples of a problem with concurrency involves an ATM. Someone is withdrawing money from a bank at two different ATMs. First, the program checks the balance to see if there is enough money to withdraw. Then the next step subtracts the money. Another person is using a different ATM to withdraw money at the same time. That transaction checks the balance to verify enough money exists and then withdraws the money. What could happen if these sets of steps occur at the same time on the same bank account and one doesn’t finish before the other?

ATM1 (thread 1)              ATM2 (thread 2)
1.                           withdrawal = 100
2. withdrawal = 100
3. check balance = 200
4.                           check balance = 200
5. if withdrawal < balance
6. withdraw 100
7.                           if withdrawal < balance
8.                           withdraw 100
9. balance -= withdrawal
10.                          balance -= withdrawal
11. update (balance)
12.                          update(balance)                       

What is the new balance when this program runs? The balance starts at $200. After two $100 withdrawals you would expect the new balance to be $0. However, instead, the new balance is $100! That’s because each thread checked the balance before either one of them updated the balance to reflect a withdrawal. They both showed that the balance was $200 and they both subtracted $100 from that to come up with $100. They both updated the new balance to $100.

Although both the above operations checked the balance and it was sufficient to cover the withdrawal, due to the order of operations the balance was checked prior to any withdrawals in both cases. The person is able to obtain more money than should be allowed.

Here’s what should have happened:

ATM1 (thread 1)              ATM2 (thread 2)
1. lock balance              
2.                           withdrawal = 100
3. withdrawal = 100
4. check balance = 200
5.                           can't get lock - have to wait.
6. if withdrawal < balance
7. withdraw 100
8.                           wait some more...
9. withdrawal = 100
10. balance -= withdrawal
11. update (balance)
12. remove lock
13.                          lock balance
14.                          check balance = 100
15.                          if withdrawal < balance
16.                          withdraw 100
17.                          balance -= withdrawal
18.                          update(balance)
19.                          remove lock

The above code uses proper concurrency constructs to lock the balance to prevent any other process from operating on it at the same time. You will probably notice that this is another form of transaction. This block of code needs to perform in the proper order. And in fact, the database programming discussed in the blog post on transactions needs to deal with potentially concurrent updates. That’s why databases and application languages have constructs that help ensure application operations occur in the correct order.

What other types of security problems might occur due to a race condition? Here’s one that includes both concurrency and eventual consistency.

Styra Open Policy Agent (OPA) Gatekeeper through 3.7.0 mishandles concurrency, sometimes resulting in incorrect access control. The data replication mechanism allows policies to access the Kubernetes cluster state. During data replication, OPA/Gatekeeper does not wait for the replication to finish before processing a request, which might cause inconsistencies between the replicated resources in OPA/Gatekeeper and the resources actually present in the cluster. Inconsistency can later be reflected in a policy bypass. NOTE: the vendor disagrees that this is a vulnerability, because Kubernetes states are only eventually consistent.

The vendor disagrees that this is a vulnerability due to the fact that Kubernetes states are eventually consistent. Is that OK? Do you want to be able to immediately block access or will it eventually sync up? If you expect to have the policy immediately applied note that your IAM policies in some cloud providers likely operate in the same fashion. When you apply a policy it may take a few minutes for the policy to block an action due to the nature of how the operations get applied across a complex and distributed cloud environment.

Although the rust programming language advertises safety and protection against concurrency, a CVE exists in a function that deletes files. The CVE states that this issue could trick a program into using privileged permissions to delete files.

Here’s another concerning CVE in a programming language library:

The py-bcrypt module before 0.3 for Python does not properly handle concurrent memory access, which allows attackers to bypass authentication via multiple authentication requests, which trigger the password hash to be overwritten.

OWASP recommends bcrypt for password hashing. I explained why password hashing is important in my last book in conjunction with some major breaches where companies failed to do that. A programmer knows they need to hash passwords so they try to do the right thing. They use this version of the Python bcrypt library they might introduce a vulnerability in the very process they are trying to secure by using that library!

A very significant breach of Microsoft systems involved a race condition that allowed a secret key to end up in a crash dump. That dump got transferred into a less secure debugging environment. Chinese attackers got onto a developer’s machine where the crash dump existed. They found the key and used it to compromise customer mail accounts.

Forcing Order When Concurrency Is Out of Your Control

I experienced a problem with concurrency while using ColdFusion years ago. I was piecing together HTML in my code to display a web page. I gather data and added the data to a buffer of some kind. I created different functions to parse data and add it to my variable instead of writing all the code in one gigantic function. This was likely because I was using the principle of abstraction to minimize my code as I explained in another post. There may have been other reasons as well, which I’ll write about in my chapter on beautiful code.

As I added new lines of code using the ColdFusion application server functionality, it apparently used concurrency under the hood. I would get back my HTML code in random order and my page would look crazy. All the elements were in a mixed-up order all over the page.

I reported the bug to the ColdFusion team, but I realized what was causing the problem. It was due to the fact that my functions all referenced the same buffer variable. The functions calls were not occurring in the order I had written them in my code. Clearly, they were using concurrent programming under the hood. I doubt this was intentional with the particular component I used, but in any case, I didn’t want to wait around for them to decide if it was or was not a bug and fix it.

Although I didn’t control the control of the operations within the application server, I could force the application to act in a way that prevented multiple functions from operating on the value at the same time. To do this you pass in the object you want to change as a parameter of a function. Then you return that object back to the calling code. That essentially locks it in some programming languages.

Here’s the problem code:

buffer='<html><body>'

text1='Title'
text2='Introduction'
text3='Section 1'
text4='Section 2'
text5='Summary'

addText(text)
    buffer = buffer + text

addText(text1)
addText(text2)
addText(text3)
addText(text4)
addText(text5)

print(buffer)

When the above code it would create something like:

Section1
Summary
Introduction
Title
Section2

To force the order I changed the code to look like this:

buffer='<html><body>'

text1='Title'
text2='Introduction'
text3='Section 1'
text4='Section 2'
text5='Summary'

addText(buffer, text)
    buffer = buffer + text
    return buffer

buffer = addText(buffer, text1)
buffer = addText(buffer, text2)
buffer = addText(buffer, text3)
buffer = addText(buffer, text4)
buffer = addText(buffer, text5)

print(buffer)

The code can’t continue until the value of buffer gets set in the current line. That’s what the line is trying to do, after all. That means the program needs to with for the function to complete in order to continue. The output from my second block of code (at least in programming languages in which I’ve used it) would be the text in the order it was passed into the addText function.

One interesting tidbit of information related to this is that when troubleshooting multi-threaded code you can’t just print out lines of code using a logger in some cases. The logging library may be written to ensure log lines are written in the proper order. It may use functions such as the above which hide concurrency issues like the one in my first block of code. In order to troubleshoot concurrency issues, you need an IDE or tool that will show you the operations without trying to force a specific order.

Deadlocks

Have you ever had an application you are working in ‘hang?’ That’s a slang term meaning the application just freezes and stops working. A deadlock occurs when you have two threads have interdependencies that cause an application to wait eternally while trying to get a lock on a value. Neither thread can finish processing because both are waiting on something that the other controls.

Here’s the scenario, borrowed from a StackOverflow thread on deadlocks. I could re-explain this another way but this is a nice, succinct scenario:

X starts to use A.

X and Y try to start using B

Y ‘wins’ and gets B first

now Y needs to use A

A is locked by X, which is waiting for Y

What could go wrong when a deadlock occurs? You could face a denial of service attack (DOS). An attacker knows your application has this bug and repeatedly causes your system to go down. Sometimes DOS attacks are a cover for other malicious actions. While the team is busy fixing the system that’s down they don’t notice some other stealthy nefarious activity.

DOS attacks are problems for logging, monitoring, and security systems. What if your logging system fails just at the point the attacker is taking those nefarious actions in your environment? You won’t be able to figure out later what caused a security incident. What if a DOS attack hits your firewall and it is designed to continue allowing traffic to pass without the rules so as not to cause any downtime? You may allow unwanted traffic into your network.

Here’s an example of a CVE in Mozilla which results in a DOS attack combined with the potential to execute malicious JavaScript:

Serverless functions

Concurrency is not only applicable within applications but between applications and code that operate on the same data. Consider two serverless functions accessing the same file in an S3 bucket. Can you think of an operation that may inadvertently cause a problem?

Let’s say one function tries to access an S3 object. It obtains the name of the object from the database. Then a second function also gets the name of the object from the database. The first function renames the object. Then the second function tries to access it at the old name.

As you can see, problems with concurrency can lead to serious security issues from stolen money to data and memory corruption, system crashes, security bypasses, and code execution. Concurrency is everywhere. Whether processing financial records, files in S3 buckets, system logs, or content management systems, you need to be aware of how concurrent actions may introduce a security problem. I could dive into some other related topics here which I may do in the final book but for now you understand the importance of getting concurrency right, should you chose to implement a multi-threaded application.

You may have also noted that concurrency can be related to transactions, eventual consistency, pointers, and some of the other topics discussed in other posts. Software engineering is more than writing functional code. It involves understanding how your programming choices affect performance and security, stability, and the accuracy of your systems and applications. That requires an understanding of the moving parts that interact in your program and the implications of your implementation choices. All these topics are interrelated.

Next Steps

• Consider whether the benefits to your application warrant the complexity of multi-threaded programming if you choose to use it.
• Dive into the details of how to properly implement concurrency by reading books that cover the topic for your language of choice.
• Make sure you implement multi-threaded programs carefully to maintain the proper order of actions within a program.
• Learn how to use a debugger to see what a multi-threaded application is doing when you run it.
• Beware of functions that may force an order and hide concurrency problems. Use those functions to overcome or prevent concurrency problems if needed.
• Understand how deadlocks can hide or enable other malicious activity. Ensure you do not have deadlocks in your application.
• Consider where concurrency issues may exist besides multi-threaded programming and database transactions.
• Ensure all systems and applications have appropriate logic to prevent out of order concurrent operations, not just multi-threaded code or database operations.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab