Getting Value From Security Testing

What did you do with that penetration test, audit, assessment, or bug bounty report?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: Cybersecurity for Executives | Penetration Testing

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This post is part of a series of posts I’m writing on Cybersecurity for Executives. This post is my third on the topic of security testing. First, I described cybersecurity audits, assessments, penetration tests, and bug bounties. Next, I explained some strategies for preparing for and improving the effectiveness of your security tests. Now let’s talk about what to do with the results of the test.

The assessment, audit, bug bounty, or pentest report

At the end of a security evaluation, or when a security researcher finds a bug in a bug bounty program, the security tester, auditor, or assessor gives you a report. Mission accomplished. You’re done, right? Not quite!

Now you have some work to do. Undoubtedly, your report will have at least a few findings. What can you expect? Likely you will have an overview of the discoveries, calling out high-risk issues you should look into right away. Next, you will probably have a summary of all the findings followed by a complete list, details, and mitigations. The report will end with a summary and possibly an appendix with additional information.

The test should describe the scope, but also what was and was not tested. During the course of some engagements, I discovered domain names I suspected should be in scope even though they were not. I contacted the customer to confirm these should be added. I noted that in the report. In other cases, bugs or other issues prevented testing certain aspects of the system. I also explain what I could and could not test in my reports if this occurs.

How long will the report be?

I can’t speak for other penetration testing companies, but so far in every pentesting engagement I’ve done, I’ve come up with 40+ pages of issues, explanations, and mitigations. One report was over 80 pages because the customer wanted every website request and response related to an error I discovered. I included multiple request and responses for the same error to make sure they I delivered what they wanted. Before I started pentesting, I was on the other side, building and defending the systems. I have some insight into how they might be attacked for that reason — especially banking systems, cloud systems, and web applications. I also ran web servers, email servers, database servers, and implemented things like firewalls and homegrown WAFs (Web Application Firewalls) before that term existed as I mentioned in other posts. This background helps me when evaluating systems. I know what went into developing and configuring them and how they can be attacked.

The length of a report will depend on many factors, including the types of evaluations, the number of systems, the development team security skills as well as evaluator background and experience. I tend to focus on cloud and web application penetration testing for small to medium-sized companies with teams that are newer to security. I include detailed instructions on how to reproduce the problem, along with resources for further reading, which increases the length of the report. I would love to write a shorter document for the same amount of money if I could produce no findings! However, that was not what I experienced during tests and is not in the best interest of my customers.

Just because a document is long, doesn’t mean it is good. I have to weed through numerous false positives before I deliver a report and only include the things that are a problem. A false positive occurs when an automated tool misreports a vulnerability. Upon further inspection, an experienced penetration tester can see that the finding doesn’t pose a risk to the customer. These should be classified appropriately or weeded out. Often what I report is a small percentage of the output the tools generated. Hopefully, your penetration tester or person using automated tools to assess your systems has done the same.

With some types of audits, like SOC 2 audits, I have been told there is a great deal of boilerplate text that goes into a report. The part the company delivering the document writes from scratch may be a small portion of the overall deliverable. I also read a 500-page book (or shall I say scanned it) on SOC 2 audits. As it turns out, the book was all about how to write a report, not how to find and address security problems. It seemed like some sections were repetitive. If repetitive verbiage appears in the document, it could end up inflating the size of the report and lead to a much longer text than what I produce in a penetration test or assessment. If the findings are not the most critical issues that could cause a breach based on industry knowledge, statistics, and the threats to a particular organization, then the audit report is less valuable than a shorter one focused on what matters.

The findings

Now your job is to evaluate the findings, and for each one, determine your course of action. You have options as to how to address the results in the report. You may not fix every single one. Here are some strategies for dealing with the findings:

Fix: You may opt to take steps to correct the issue. If it is a simple code flaw, for example, you might assign a developer the task of fixing that flaw in the system. Sometimes the fix involves changing a process or training people. It may also include re-architecting systems. Some vulnerabilities may prove to require complex changes that increase the risk of downtime or other errors or will take a long time to implement.

Mitigate: Instead of fixing the finding, you may decide to mitigate it by blocking access to it, for example. In some cases, an organization may have additional information that the pentester did not know that results in a finding becoming a non-issue. Be careful with this one. Sometimes mitigating factors can be flawed as well, so whenever possible fix the discovery. In some cases, it may be too costly or take too much time, so the company chooses a different route.

Accept the risk: An organization may decide the cost to fix is more than the potential loss if an attacker exploits the vulnerability. The other reason a company may accept the risk is that the vulnerability is complicated to exploit and is unlikely an attacker would do it. Additionally, the vulnerability might only expose data that is already public.

Evaluate the risk and assign the work

As mentioned from the start, security is more about risk management than eliminating risk. Executives need to have the right information to make the right decisions. Your penetration test, assessment, and audit reports, along with a basic understanding of cybersecurity, can help! When you look at the findings on the document, determine how much risk they pose to your business and prioritize accordingly.

If you get a report from a security test or audit and talk about it, but then never actually fix the problems, you are increasing your risk. Now multiple people know the vulnerability exists. Who read and had access to the report? Was it transmitted securely in every case? Where is it backed up? Make sure that you have stored the document securely, but more importantly — make sure you mitigate any high-risk issues as quickly as possible.

When you request a penetration test, audit, or assessment, the report may produce work items unaccounted for in existing schedules. Be prepared to adjust schedules. Developers may need to set aside an existing project to have the time to fix the security vulnerabilities. The security team or IT department may require a reprieve from other duties to focus on an important security finding.

Make sure the person assigned to fix the problem has adequate security training to understand it fully and address it appropriately. Too many times, I have seen managers who don’t understand the implications of security (or other) challenges assign work to someone who did not have the background to implement an accurate or sufficient solution. If you are not sure, have someone with more experience review their work before it is released. Hopefully, you have steps to reproduce the issue in your penetration test report for most of the findings so you can verify that the problem is fixed by following the steps in the issue and validating the problem no longer exists.

Be warned, however, that sometimes fixing the exact finding does not fix the entire problem. For example, on a penetration test, I reported a flaw in an XSS filter that was actually causing an XSS vulnerability in a web site. (XSS is one of the OWASP Top 10 web site flaws — and I’m not going to explain it in detail here because I hope you can see my point regardless.) I included the links to the web pages on which I found the vulnerability. If the company only fixed those pages, instead of the underlying problem (the XSS filter itself which is supposed to prevent the type of vulnerability it caused), any new pages that got added to the site after the test would have the same vulnerability. When I retested the site, I found the same problems. I don’t know why it still existed. The second time around, I tried to explain the issue with the underlying software library in more detail.

Address the root cause

Be prepared to provide funding or resources for other types of problems as well. While assisting with an audit, one of the results was that the security team was too small to handle the additional work to use a security tool the company purchased. The result could be that the company needs to hire more people so the security team can effectively use that tool to mitigate threats. In another case, a company had software developers that were unfamiliar with the OWASP top 10, cloud, or security best practices. Security training may be required to resolve these issues.

What else might findings uncover? An audit or assessment may expose that people have too much system access, and this creates a risk for the company. An organization may need to determine if they want to accept that risk or restructure teams, permissions, or architecture of deployment systems to resolve this issue. This change will not be easy because people generally do not like it when you remove their access to things they could do before. Include appropriate communication and explanations as to why the organization is implementing the changes in the roll-out plan. Again, security training so people understand why this is a risk may help.

Finally, executives should be prepared to hear, and better yet, ask what is blocking people from implementing cybersecurity best practices. The answer may be a lack of executive support or political issues within the company that makes it hard for them to do their jobs. The organization may not be incentivizing, motivating, or training people adequately. It could be that the organization processes, workflows, or managers do not prioritize their assigned tasks in alignment with the organization’s security objectives. Sometimes too many competing objectives may exist. Fixing these issues is the responsibility of the executives who have the authority to restructure organizations, workflows, and set priorities. Often executives think that everyone is implementing systems securely, but the reality (speaking from experience!) is far from it for the reasons I just mentioned. Sometimes employees don’t tell executives things they think their leaders don’t want to hear.

Measure the results

Don’t wait for your next audit, assessment, or pentest report to find out if the vulnerabilities still exist. Track findings and measure whether the risk is going up or down within your organization. That is what the 20 questions executives can ask their security team I wrote about in a previous post are all about — finding ways to quantify that you are taking action on and reducing cybersecurity risk within your organization.

I have heard many penetration testers talk about how they come back year after year and perform a test — and the same problems appear on the report. Why is this? It could be that no one ever addressed the prior findings. After you read a security test report, how do you track that the specific vulnerabilities exist or not, month after month, year after year? Do you know if and when they got fixed, and by whom? Waiting a whole year to find out a vulnerability was not fixed is a long time for an attacker to find it!

It could be that someone addressed the original findings on the report, but the same problem has manifested itself again in different ways. Why would that be the case? Perhaps because the people involved who fixed the initial finding are not the ones who created the new finding. New people repeated the same mistakes. Maybe the people who addressed the first issue copied code they received to fix an issue but didn’t understand what caused the vulnerability. This problem, once again, boils down, at least in part, to a lack of cybersecurity training.

Organizations complain that they cannot hire cybersecurity professionals. However, you have numerous talented people in your organization who can help with security. Not only that, your security team can’t do everything themselves. There are too many places where a security problem can insert itself into a system. To effectively improve security in your organization track who is making security mistakes, and send them to training. Better yet, send your whole team to a class where they can discuss together how to make cybersecurity improvements to prevent repeat findings on security reports.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2019

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab