SolarWinds Hack: Retrospective 1

Part 1: SolarWinds and the big picture for executives

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: SolarWinds Breach | Data Breaches.

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I had a number of cybersecurity consulting calls over the past week where customers were scrambling to deal with the SolarWinds hack. People have also been asking me questions about it. After letting the dust settle a bit, I took a look at what we know so far as outsiders. Since I am not involved in the incident directly, I can only make an assessment based on public information. I performed a similar analysis and wrote a white paper on the Target Breach, looking into what caused it and what might have helped prevent it.

Security is not easy

Let’s start with the fact that security is not easy. Anyone who scoffs at those who got compromised may be next. People that don’t work in security complain about it and want it to get out of the way. People who work in security are overloaded with logs and information trying to sort it all out and stay on the good side of business people. Often security people have to deal with organizational politics as much as they do malware and network logs, as I wrote about in this two-part series on why patching is hard. Everything is a balancing act, and it’s not an easy feat to get it right. And It’s always easier to attack than to defend.

But we can always do better by learning from the past.

The United States needs more and the right investment in Cybersecurity

We, in the United States, like to assume we are the best. Let’s look at the facts. We’ve been breached here lately — a lot. Recent breaches include top cybersecurity companies, federal agencies with our most sensitive data and cybersecurity tools, the company that maintains all our credit ratings and national IDs, our health care systems, business and government intellectual property, and companies that make security products to defend us. And those are the breaches we know about.

My main thought after the recent FireEye breach, followed by the SolarWinds breach affecting over 18,000 organizations and federal agencies, is that the US needs to step up cybersecurity defenses and invest more heavily in security research, development, and monitoring. But not just any investment. The right investment.

Initially, when I saw the FireEye breach I was extremely concerned about cybersecurity in the United States as a whole. This article sums it up nicely:

If one of the biggest names in cybersecurity can be breached, what chance does an average person stand against hackers?

As additional information came to light, another perspective emerged. Ultimately, I still think we need more investment in cybersecurity. Our national interests and way of life are at stake, not just some computer systems. I provide a tip for end users and small businesses in a link to a prior post at the end of this story.

Security tools alone won’t save you

At first, we only knew about the FireEye breach. I was floored that one of our top cybersecurity firms was affected like this. As I generally recommend, however, wait a few days for the facts to emerge. Although it’s not great news, I was at least happy that FireEye detected the breach when other organizations did not. This further solidifies why you need cybersecurity experts on your team. I’m guessing the attackers sat dormant for a while in the FireEye network and then made their move. Somehow FireEye detected them when many other organizations failed to do so and alerted others to the problem.

This is where my analysis of the breach led to my already held opinion about how we have been approaching cybersecurity for years. The attackers infiltrated some organizations for months, undetected. We are throwing money at security tools, but this breach demonstrates that security tools alone will not save you. We need to be digging into the root problems and using all our resources — including people, improved processes, and new technology — to find new, innovative methods to stop these attacks. However, at the same time we are innovating, we need to understand the fundamentals, which do not change.

I wonder about the architecture of the SolarWinds implementations, and whether organizations employed defense in depth with redundant monitoring for critical systems and zero trust networking as they moved systems to cloud environments. I’m still looking into the details and will write more about that in upcoming posts. As noted, I’m not directly involved and not going to judge from the outside. I can only make recommendations that might help, and each organization can decide for themselves if those recommendations are useful given the circumstances.

We’ve seen this before

This recent breach was the result of the infiltration of a vendor software update process. This same attack vector has been used before in major cybersecurity incidents. The NotPetya malware leveraged the same type of attack, using a Ukrainian vendor software update process to deploy ransomware. That cyberattack took down large companies and hospitals around the world. I warned people about future use of similar attacks in my book, mentioned at the end of this post.

Most have attributed that breach to Russia. Many say Ukraine is Russia’s malware testing ground. That would lead one to presume if attackers are testing out additional types of cyberattacks in other countries they may try them here next. At the very least, different groups of attackers are getting ideas from one another and they may try to pin the blame on the wrong source using false cues in logs.

If it happened to some other country or organization, you need to presume it could happen to you too. Cybersecurity experts study breaches and malware around the world and provide information and warnings to organizations that often go unheeded. Let’s put this information to good use instead of ignoring it or thinking it couldn’t happen to us.

It could get worse

Luckily we have not yet experienced a major successful attack on our power systems or critical infrastructure. Yet. I wrote a book review about Sandworm which explains what that’s like when it happens in other countries. Will it happen here? Refer to the last section.

Other attack vectors are being used to take down critical infrastructure in Eastern Europe. We could see attacks on our electrical grid, banking, and communication systems just as other counties have experienced if we fail to learn from prior breaches and past mistakes. We can be more vigilant. We can invest more in our defenses. We can come up with more innovative mechanisms for defending systems against these attacks.

The same group tentatively attributed to the current attack was also behind an attack on the San Francisco airport earlier this year. What if attackers could affect airport projects, flight schedules, or anything else related to airport security?

Risk-based decisions

I am very apprehensive about flying at the moment for other reasons but recently took a necessary trip — wearing two masks, a face shield, and a hoodie. This is my personal defense in depth. I don’t rely on one flimsy mask. I would rather not fly at all, but sometimes you have to take a risk and get things done.

Each person or organization evaluates the risk of a particular objective and plan of action and then makes risk-based decisions. Organizations determine how they should proceed when making cybersecurity decisions. They decide the amount of risk they will take and the amount of investment they want to make to protect their systems. No right or wrong answer exists. It is an analytical decision (hopefully) and a choice.

On the way back the plane was delayed because it did not have the proper paperwork to take off. The last time this happened I got locked in customs in Toronto for two hours right before the huge Delta system crash. A problem like missing paperwork could be a system glitch. But sometimes system glitches are related to cybersecurity problems.

As a security professional my mind wandered thinking about the possibilities. Luckily we took off shortly after and everything was fine. However, we need to be aware of and defend against future possibilities, not just the threat of the moment. Everyone needs to be vigilant because having attackers steal your money or social security number is bad, but other problems could be worse.

Organizations need to be aware of cybersecurity threats around the world and prepare for them. Make calculated risk-based decisions based on the threats and the criticality of your data. Be aware that your systems may also be used to attack others, so even if you are not worried about your own data, understand that your risky choices could affect others.

We can do better

The point here is not to scare people. The point is that we need to take cybersecurity seriously. We can and should do better. We can find these problems in our networks and the systems that allow users to access our resources. They will be in the logs if we architect systems accordingly and secure logs properly. We can learn from other incidents and come up with better ways to prevent them in the future. It’s in our logs if we are looking. We can create architectures that help us better spot anomalies and block attacks.

We can be more methodical when dealing with cybersecurity anomalies, data, and misconfigured systems. We can obtain more precision in cybersecurity where it really matters. I just watched Werner Vogels’s AWS keynote where he talked about AWS exploring provable security. I love this concept. I have equated cybersecurity systems to financial systems in the past in a blog post on managing your data like you manage your money. These types of new ideas and investments will help us better defend our systems and infrastructure.

What we don’t see

As I mentioned above — this attack is the one we currently know about. The most dangerous attacks may still be coming and we may have adversaries in our networks undetected right now. We need to be looking at our network traffic as well as other system logs more closely. We can’t just rely on a tool to do everything for us. We also need dedicated teams focused on this not just someone doing it once in a while on the side for critical resources. Systems and networks need to be designed in ways that help make these complex attacks more visible.

Note that you’ll have a problem identifying rogue traffic in a cloud environment if you threw network security out the window and are counting on identity. This attack compromised identities and in ways that were challenging for organizations to spot. More to follow as I take a look at the technical details of the attack and how it involved cloud systems.

Anyone with a technical inclination can get involved and investigate their own network traffic. I recently posted an introduction to Wireshark. If more people get curious about what is in their logs and start investigating, we may all collectively be better off. Looking at logs in small networks is much easier than sorting out all the logs in a large environment with massive amounts of traffic. Everyone can scrutinize network logs more closely and help identify malicious network activity. However, understand there is a lot of noise. Often when people first start looking at network traffic it is overwhelming due to the amount of rogue traffic and attacks our systems face every day.

I wrote about the malicious traffic on my network (and yours too most likely) that we seem to accept as normal from sources all over the world. The blog post has a link to a Github repository with the network ranges sending me rogue traffic, along with the originating countries. All this noise makes it harder to spot true threats. I teach people how to weed out the noise to find threats in my cybersecurity classes. I’m hoping to have another one soon for executives but had some changes come up (good changes) that will delay it slightly. I was not involved in this breach and the complications around spotting the rogue traffic but I wish we could somehow come up with a better way to spot it. As I’ve stated repeatedly — the network doesn’t lie.

A tip for home users, schools, and small businesses

For home users and smaller organizations that want to better defend themselves against attacks like this, I’ll refer you to my post about how you can use CloudFlare DNS services that block malicious domain names. As soon as the security community finds out that a domain name is malicious, this service will likely find out about it and block it. I just checked the offending domain in this breach and received the following response (0.0.0.0 is not the actual IP address or a valid IP address):

Cybersecurity directives need to come from the top

Many of the components of this attack are covered in my book Cybersecurity for Executives in the Age of Cloud. I warn organizations about the need to evaluate vendors — even security vendors like the one where I used to work and network monitoring systems like SolarWinds. I explain ports, networking, DNS, and how a security product could be a gateway into your systems. I emphasize the need to carefully architect and secure deployment systems, including third-party software updates. I explain C2 channels and challenges with cloud systems and many other topics that would help anyone consider how to analyze and come up deal with an incident such as the SolarWinds hack.

It also explains why breaches like this one are one reason why executives should care more about cybersecurity. It is not just about your product or systems. It is about national security, protecting the ones you love, and our way of life. Every breached system could affect other unrelated systems. There is likely a long thread of information still not fully known publicly that led up to this attack. I will write more about the involvement of cloud systems in future posts.

At the end of my book, I explain how we could develop better metrics to evaluate cybersecurity posture. Executives can understand enough about cybersecurity to drive these metrics. My book is based on 20 questions to ask your cybersecurity team that executives can use to capture information that helps you better understand your security posture. In my upcoming posts, I will look more at the details of this breach, and some questions people may want to consider to prevent similar attacks in the future.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2020

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab