Efficacy of Security Products and Services

Are you getting a return on your security investment?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: Cybersecurity for Executives | Supply Chain Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Continuing this series of blog posts on Cybersecurity for Executives, we come to concerns the efficacy of security products and services. In other words, how well they work? Once you have evaluated your security vendors and products to ensure they are implementing security controls according to your requirements, you need to analyze the effectiveness of the security product itself. When you purchase a security product or service, how do you know if the product is helping?

Just like anything else, you can gather metrics from and about security products. You can do this in advance of purchase and after. The quality of the data depends on the product. The quality of the analysis of the results depends on the knowledge of your security team and the amount of time they can spend on analysis. If they are always fighting fires related to security incidents, likely they are not spending time analyzing security product effectiveness. If they feel that their analysis of security problems doesn’t lead to any improvements, they will probably not want to invest the time.

Researching product effectiveness

As mentioned in my previous blog post, I am on the IANS faculty, where you can call in and ask security questions. Some faculty tell you which products are best in different categories, in their opinion. I don’t like to sell products. I would instead tell you how to evaluate them for yourself. You can get some idea which products work best from existing customers, but each product may work differently depending on the environment in which it runs. Also, it is nearly impossible for one person to try every security product unless they specialize in that area. For example, I specialize in cloud security. My friend George Gerchow, CISO at Sumo Logic, is also a faculty member. He evaluates a lot of products for their company and likes to answer product-related questions.

Some companies specialize in rating and evaluating products in different categories. Gartner and Forrester provide research services on products, industries, and companies. Sometimes you can find their reports online for free. These may be helpful but do not go into much detail on exactly how security products work and if they are effective. IANS also has research reports for customers in their portal, some of which I write.

NSS Labs explicitly tests security products. In advance of purchase, you may find these reports more helpful. One caveat with these reports, which I learned working at a company that went through the tests. It seems to be a pay to play model. Additionally, the companies go through a series of tests, and then they have a chance to fix the problems and rerun the tests. It seemed a bit like a game to me to try to get into the top ranking. Additionally, they test for specific categories so that companies are conforming products to ace those tests. Those tests may or may not be in alignment with your environment. However, I still think the work NSS is trying to do is great because it’s hard for every individual company to run these tests for all categories of products.

Testing products

A myriad of security products exists for different purposes. It’s hard to know in advance if a security product solves your security problems. My biggest recommendation is to try out the product if possible. Otherwise, at a minimum, get a demo of the product in your environment. Every environment is different. Some environments have more data or less data and different types of network traffic or storage, all of which could affect performance. If you have a large organization, perhaps you can afford to purchase a single piece of hardware or the smallest license option of different products to test them.

You should see if the product finds any existing issues in your environment. It is essential to understand how the product works when formulating your trial or test. Sometimes a product needs to make a baseline of normal traffic and then tell you if there are any anomalies so you won’t see anything immediately. A test like this requires running the product over time.

For example, when we initially installed StealthWatch Cloud (formerly Observable Networks) in an AWS account, it didn’t find anything initially. Later it alerted us to unwanted traffic going to Brazil. In another test account, someone left a Jenkins server running that got compromised at some point after we installed the product in that account. These are the type of tests I hope all vendors would let you run with their products to see if they work and find problems.

Be careful when you deploy a security product in your environment. Security products collect and can see a great deal of data, so I recommend you do a security evaluation of the vendor first, and that you understand what data is collected, where they store it, and how it is protected. You may want to run the product in a lab environment, where you can evaluate which network ports the product requires you to open and the network traffic produced by the product. If products require too much network access, they may create additional risk at the same time you are trying to get additional protections.

Some SAAS (Security as a Service) vendors send all your data to the cloud. Depending on your results of the vendor assessment, you may or may not be OK with this. You should ensure the proper security is in place to make sure other customers and the vendor staff cannot see your data without your permission. If a product stores vulnerabilities associated with your environment, that data could be valuable to an attacker. They would know what to attack. If a product stores network traffic logs, those logs may include sensitive data as people send passwords over the network to log into websites or purchase things with credit cards.

Software Scanning Tools

While completing a research project on code scanning and software vulnerability management, I found the software scanning vendors to be unwilling to give out a free trial for the most part. If a company doesn’t give out a free trial so you can compare it to other software vendors, it is hard to analyze which one is better for your codebase. Of course, they don’t want you to scan the code and then not purchase the product, but I found that they wouldn’t even let me do a free trial on some open source code or small project. I find this interesting because if you can’t test the product and you have to pay thousands of dollars to buy it, you have no way to know in advance if it works for you.

If I could have tested those products, I would have produced a code base with common sample vulnerabilities. Then I would have scanned that code base with each product to see what vulnerabilities they found. It would be easy to determine which one was best suited for your environment if you have an accurate and deep enough sample of vulnerabilities.

When choosing software scanners, you’ll need to understand what languages they work with and what type of vulnerabilities they find. Some scanners are linters, which are focused on evaluating coding best practices for a particular language. They are not focused on security flaws. Some scanners only find CVEs in software dependencies. Others try to alert you to vulnerabilities like the OWASP top 10. Some try to spot malware embedded in your code. Others look for unsafe sources and sinks. The source of a vulnerability is the injection point. The sink refers to data modification to change the behavior of an application. Check to see that the scanner works with the specific programming languages you want to scan.

Networking products

Network security products are interesting because they sit at the edge of your network in some cases. The traffic leaves your network, and then you don’t see it after that point. You don’t know how that device may have altered the traffic as it sent it out. There are a few ways you can test network products. You can set up rules to allow and block traffic and then validate those rules worked. Some network products try to identify traffic. You can send various traffic through the device and see if it was identified correctly. For example, I’ve seen misidentified traffic attributed to the wrong company or cloud service after digging into the details of the traffic.

When the device is at the edge of your network, you could put another network device in front of the one you are testing to inspect the traffic. You can validate that the logs of both report the same results. For a cloud network appliance, you can compare the logs of the appliance to the cloud-native network logs. AWS, Azure, and Google Cloud Platform all have the rough equivalent of NetFlow logs (AWS VPC Flow Logs in AWS or similar names in other clouds). Run the device inside a virtual private network in the cloud provider and compare the cloud-native logs to the appliance logs. Set up rules in both and see if any inappropriate traffic slips through the network appliance. You can also do the reverse to test your cloud provider logging and rules.

Note that just because a vendor sells a product with the same name as the one you get on-premises, they are not the same codebase. Certain pieces of functionality need to be altered to work on cloud platforms. Test the on-premises version and the version for each cloud provider independently.

Some network appliances have an intrusion detection or intrusion prevention system function that alerts on or blocks network security problems, respectively. You can create various types of network attacks to see whether the tool catches the problems or not. Some network appliances require a great deal of tuning and configuration to turn on and off all the appropriate features. Running these tests ensures that you have correctly configured the appliance. Some people buy a network or security appliance and turn it on, leaving all the defaults in place. They may not be getting the full value of the tool they purchased.

I mentioned the tests by NSS Labs at the beginning of this post. One of the challenges in those tests is the ability to identify all the malware and still achieve the desired performance. As more and more features of a product get turned on, it may limit the amount of data the device can process in a given timeframe. Some products advertise performance results with only a small portion of their complete suite of services enabled. Make sure you test the product with the desired features under the expected load.

Also, consider what type of network protocols you use in your environment. A particular wireless product I evaluated advertised a way to stop malicious behavior in the network. After hearing the details of how it worked, it was clear that all the advertised features would not work on networks running a newer version of the IP protocol used on almost every network. The older version is IPV4 and much more common. The newer version is IPV6 and though less common, many networks don’t block it and it can be leveraged in attacks.

Products that identify malware

When evaluating products that identify malware, understand what type of malware they catch and how they do it. Some products are signature-based. Some work based on behavioral analysis. Some try to stop a specific type of malware like ransomware or help you reduce phishing in your environment.

Products that perform these functions are in a never-ending competition with attackers. The attackers create a new type of attack. The vendors analyze and figure out how to block it. Then the attackers figure out how to bypass the protections. The vendors then figure out how to block the new bypass methods, and so on. I don’t know if this is true, but I was watching a presentation by MalwareBytes at an IANS conference. They were talking about the GrandCrab ransomware. They said at one point, the attackers put a comment in the code that said, “Hi MalwareBytes.” Even if it’s not true, I’m sure that’s how the vendors and attackers feel about each other at some point.

The effectiveness of your security products depends in part on the strength of the research team. The research team needs to be able to analyze new malware and determine how it works and how to block it. Different types of malware require different types of skills to reverse engineer it. Some are simple web code. Some require a disassembler to reverse engineer assembly, which is very low-level machine code and not very human-readable (unless you’ve been doing it a long time)! The team needs to quickly be able to parse what the malware is doing and provide information to the team that updates the products. The process for getting updates to customers needs to be efficient so you can get the latest updates to protect your organization as quickly as possible.

Have you ever seen those research reports that say something like a 500-billion percent increase in XYZ malware? Take those with a grain of salt. In light of what I just told you, you can see that the vendors are always playing catchup with the attackers. When you see a significant increase in a particular type of malware, it is likely that the product just started identifying that malware. It probably ramped up at a much slower pace, and it just took the vendors a long time to identify it.

True and false positives and the need for tuning

Most security products try to identify threats. Often the criteria they have to use has some variance. They produce something called true positives and false positives as a result of the lack of concrete decision factors.

True positive: The identified threat or attack was indeed a security problem, and the security appliance or software successfully alerted on or stopped the problem.

False-positive: The security product identified something as a problem, but on further investigation, it turned out to be a non-issue.

As you can imagine, many false positives create a lot of noise and waste time. You can evaluate security products to see how many true and false positives they produce. Also, evaluate how the products can be tuned to reduce false positives without missing true positives. It’s worse to miss an actual attack than to have many false positives. At the same time, if you have so many false positives that people stop responding, then the true positive is missed anyway. The attackers are always changing their tactics, your environment is always changing, and your security vendor adds new alerts from time to time. Make sure you give your security team time to evaluate and tune these products to make sure they are providing value.

Determining return on investment

After you have purchased a product or service, determine if it is helping you. Once you installed the product, how many attacks did it block? These metrics give you some idea of the efficacy of the product and if it was worth the money you paid for it. On the other hand, make sure the things it is blocking are things that matter. Your security team can evaluate whether or not the things that were blocked would have led to a breach that could have resulted in significant financial loss.

You can also see what things your security product missed. Report those deficiencies back to the vendor so they can improve the product. Some vendors design products to prevent a specific type of malware. Then the malware gets smarter, and the product does not adapt. Evaluating whether or not the security appliances are catching the latest malware helps you determine if you should continue to invest in that product or service.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2019

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab