SolarWinds Hack: Retrospective 2

Part 2: What caused the breach and what does the malware do?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: SolarWinds Breach | Data Breaches.

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What caused the breach?

Attackers inserted malicious software into a SolarWinds software update.

What is SolarWinds?

SolarWinds is a system used by large corporations to monitor any application and any server, anywhere.

If SolarWinds monitors anything, anywhere, why didn’t it spot the malware?

Once the malware embedded itself in the system, I’m guessing it excluded logs related to its own activities from the monitoring system itself. Given the sophisticated nature of this attack, it seems like the obvious thing to do. Once it obtained access to other resources on the network it may have also deleted or altered other system logs.

How could the attackers insert malicious code into the software?

Companies build and package software in many different ways, but typically the code starts in an environment where developers write the code. Then it passes to the Quality Assurance (QA) team that tests the code. Once the QA team approves the code, it moves to production and is publicly available. There are many ways in which code could be inserted into a less-than-secure deployment system.

Insider Threat

In September 2020, a hacker offered a Russian-speaking Tesla employee one million dollars to install malware to execute a ransomware attack against the company. How much do you think someone would pay to insert malicious code into the SolarWinds product? Even if the insider does not receive payment directly, they could sell data or system access or information about the vulnerability so that others could use it later.

At some point in this process, someone may have altered the code. Some companies are not very careful with their development process and allow alteration of code in QA environments by developers, for example. Additionally, the QA team should never change the code, only the development team. I’ve seen deployment processes with manual alteration of code on production night at large corporations. There are a myriad of ways a rogue insider might insert code into the system if the organization does not properly check the integrity of the code as it passes through the software development lifecycle.

In addition to inserting code into the system, an insider could also simply provide access to outsiders to make the change. An insider may set a weak password, for example. According to this Reuters story, earlier this year, a security researcher discovered the password “SolarWinds123” on a SolarWinds update server. Those investigating the breach do not think this was the cause of the SolarWinds breach, but a similar weak setting could allow attackers to access the systems.

Malware compromised systems

Attackers may have also compromised the deployment servers, source control systems, or developer machines to obtain access and insert the rogue code into the system software. Attackers may have leveraged malware at many points along the way as code moves through the software development lifecycle.

• If the code got inserted on a developer machine, one would hope the change would be noticed in a review of source code changes, but some companies don’t have such stringent reviews.
• If the code was changed in the source control system perhaps it went unnoticed as it moved through the systems to production.
• Automated processes may have injected the code if attackers obtained access to the servers that perform the automation.
• Attackers could alter the code as is passed through the network with a man-in-the-middle attack but this would be more complicated to pull off.
• If QA teams have access to change the code, any compromised QA system could have altered the outcome.
• The attackers may have gotten access to the server that hosts the updates and simply replaced a good copy with a malicious copy of the software.

Access to SolarWinds servers sold on the Dark Web

Whether insiders or external criminals initiated the attack, evidence of a prior attempt to sell access to SolarWinds systems exists according to this Reuters article:

“In one previously unreported issue, multiple criminals have offered to sell access to SolarWinds’ computers through underground forums, according to two researchers who separately had access to those forums.”

What did the malware do once it got onto the systems of victims?

FireEye provided a detailed report about what the malware does once it gets on a victim machine:

At a high level, what did the malware do, and why?

• The malware stays dormant for up to two weeks. If someone runs the application and inspects the network traffic, likely they wouldn’t see anything unusual unless they watched it for a very long time.
• The network traffic looks like the Orion Improvement Program (OIP) protocol. The fact that it looks like another allowed network protocol, but one that is proprietary might make it difficult to spot anomalies.
• The malware tries to identify anti-virus and forensic tools. That means the malware likely stops running when it identifies a tool that might tip off someone to the fact it is running on the system.
• The malware makes a request to resolve a subdomain of avsvmcloud[.]com. As I noted in a few Twitter posts previously, the primary domain was hosted on Microsoft IP addresses and the subdomains were hosted on Amazon IP addresses. Hosting the malware on major cloud platforms makes it more difficult for security teams to spot malicious traffic if the organization is using these cloud platforms or if the SolarWinds system contacts systems on AWS for legitimate reasons.

• As shown in the transition above, the hosting of the subdomains shifted to Microsoft after they took over the primary domain. More about the domain takeover in the following article:

• The C2 traffic to the malicious domains is designed to mimic normal SolarWinds API communications. The malware tries to blend into existing traffic.
• The malware uses a Domain Generation Algorithm (DGA) to vary the subdomains with which it communicates. By changing the subdomain each time the malware sends traffic it will be harder to identify the traffic in network logs. It may also be harder to block the traffic.
• A minimum one-minute timeout exists between each communication with the C2 server. Delaying the requests again will make it harder to spot the traffic because the logs won’t show a series of contiguous connections to the same IP or subdomains of the same domain in the logs.
• The FireEye report includes many other details the malware uses to obfuscate its presence. Obfuscation means the malware tries to make it hard for security professionals to identify the malware, making it look like benign files. The authors of the malware also made it hard to see the commands the malware executed by writing the code in very tricky ways. Reversing Labs also analyzed the Sunburst malware:

• Once the malware had access to SolarWinds, the attackers leveraged Azure Active Directory to obtain additional credentials and permissions. Most attackers aim to steal credentials and escalate privileges during an attack. If they can obtain the highest level of system permissions they can do anything they want in your network.

HelpNet Security Writes:

According to Microsoft, they used administrative permissions acquired through an on-premises compromise to gain access to an organization’s trusted SAML token-signing certificate and they forged SAML tokens that impersonate any of the organization’s existing users and accounts (which allowed them to access to on-premises and cloud resources). They also made changes to the organizations’ Azure Active Directory settings to facilitate long term access.

Microsoft provided blog posts on the topic which go into more detail:

How did the malware obtain credentials?

Microsoft explains in the article above:

In actions observed at the Microsoft cloud, attackers have either gained administrative access using compromised privileged account credentials (e.g. stolen passwords) or by forging SAML tokens using compromised SAML token signing certificates.

In cases where we see SAML token signing certificate compromise, there are cases where the specific mechanism by which the actor gains access to the certificate has not been determined.

Another important point:

Anomalous logins using the SAML tokens created by the compromised token signing certificate can then be made against any on-premises resources (regardless of identity system or vendor) as well as to any cloud environment (regardless of vendor) because they have been configured to trust the certificate. Because the SAML tokens are signed with their own trusted certificate, the anomalies might be missed by the organization.

How might the signing certificate have been compromised?

Microsoft writes:

In the cases we have determined that the SAML token signing certificate was compromised, common tools were used to access the database that supports the SAML federation server using administrative access and remote execution capabilities.

The attackers got access to the user accounts by way of the compromised the SolarWinds system. They may have stolen the tokens out of memory or used some other related function with a tool such as Mimikatz. You can read more about the use of Mimikatz to hijack Azure accounts in this blog post:

Perhaps the compromised users had access to create new SSO enabled applications via signing certificates. CyberArk writes about this type of abuse of privileges in this article dubbed Golden SAML:

This same attack vector likely facilitated access via a VMWare vulnerability on-premises and this may be the second attack vector attackers used to access signing certificates Microsoft could not determine above.

If you follow the links in the above article you can find advisories from US government organizations.

What is a SAML token signing certificate?

More about SAML signing certificates on Microsoft Azure:

Auto-generated certificate for gallery and non-gallery applications:

Automate SAML-based SSO app configuration with Microsoft Graph API

How is the above issue with Azure AD related to SolarWinds?

SolarWinds can use SAML with Azure AD to authenticate users to the system via single sign-on (SSO).

Perhaps users on the SolarWinds system were IT professionals with broad access to the cloud environment. Perhaps users on the system had permission to set up SSO enabled applications that generated valid tokens to use throughout the network.

Here are more detailed instructions fro the SolarWinds documentation:

This step obtains the public key related to the federated Azure AD access:

Followed by pasting the certificate back into Azure:

Related capabilities on Azure allowed attackers to use the Golden SAML attack or similar.

Is it the cloud’s fault?

The malware made use of systems on Azure and AWS. Those who are always looking for a problem with cloud-hosted systems will be quick to point to this fact and claim the cloud is obviously not secure. However, at this point, it appears that the attackers signed up for cloud accounts and leveraged cloud platform resources just like any other customer.

The cloud providers are not responsible for monitoring customer systems, and some customers would not want them to do that. They will, however, shut an offending system down. You can see that, for example, in the AWS Acceptable Use Policy:

If you violate the Policy or authorize or help others to do so, we may suspend or terminate your use of the Services.

Besides termination, there is a benefit to attackers hosting malware on large cloud companies. They may be able to seize the hosts involved in this attack and analyze them depending on the terms of service and whether law enforcement is involved. I’m not a lawyer and not going to dig into the terms to try to sort this out. However, these large cloud companies have many more cybersecurity resources at their disposal than most other organizations if they can access and analyze the systems.

Interestingly, the attackers ran the primary domain on Azure and the subdomains on AWS. Perhaps the attackers were trying to make it harder to shut down the malware since shutting down the hosts running subdomains on AWS would not terminate the primary domain or host. Perhaps the systems hosting the primary domain required access to something on Microsoft Azure. We will need to wait and see what additional information becomes available as the story further unfolds.

How can we determine if your organization has been compromised?

TrustedSec provided some indicators of compromise here:

How can we stop breaches like this in the future?

I explore that topic in part three of this blog series on the SolarWinds hack. Hint: It’s all pretty much the same as what I already wrote in my book on Cybersecurity for Executives in the Age of Cloud, but applied to this specific breach.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2020

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab