Better testing for better outcomes

Infrastructure, disaster recovery, product, and penetration testing

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The value of testing software are abundantly clear to me coming from a software engineering background that includes e-commerce, financial, tax, and large scale banking application systems. Proper testing ensures applications work as promised and as designed. The amount and level to of testing depends on several factors. Here are a few: the time allotted for testing, which correlates with the budget allocated to the project, the training and experience of the quality assurance professionals, their attention to detail, the number of test cases, and the stability of the products and systems they are testing.

Amount of testing influences the quality of deliverables

Higher amounts of testing produce higher quality products. That should be obvious, but often testing is an afterthought when it comes to cloud infrastructure and security. However, these aspects of your cloud deployments are just as important as your application functionality to ensure your systems work correctly and securely with the desired level of availability.

The reason for lack of testing might be because the people implementing the systems do not come from a software background and are not familiar with a full software development lifecycle (SDLC). Additionally, companies may not have the resources allocated to perform proper testing. People who are not software engineers by trade but IT professionals may not be used to writing and rigorously testing code. Software engineers and QA professionals may be more focused on application functionality and not trained or directed to test security and infrastructure. Some companies may skimp on testing to lower the cost of their products and increase profit margins.

Testing needs to be a priority with adequate time allocated to it. When is the last time the money in your bank account went to the wrong account? That is simply not acceptable. Banks need to ensure their customers trust them by being certain that the transactions you initiated occur according to your instructions in the course of regular transactions. I’m not talking about cases where someone socially engineered your employee to send the money to the wrong account or a data breach. I’m talking about the typical, everyday transactions. When you initiate or receive a payment, it had better end up in the right place. I used to work on these banking systems, and we performed an extensive amount of testing on the back end systems to make sure the money went to the correct account.

The same type of testing applies to your security, your cloud environment, and your products. Before every cloud security class, the 2nd Sight Lab team attempts to run through and retest all our labs used in class. At this point the size of our company allows us to do things that might be harder for a large company. We add new “beta” content in each class where we are researching and working on new developments — either tools we are building for our internal use or new tools from the cloud providers. We can fix issues and re-publish on the fly due to an agile process. In my last class, when something failed, I offered to have a follow-up call with the client to ensure the students could get the lab working as expected. Other than that the students were able to complete all the labs successfully.

Testing requires sufficient time and investment

During the beta class, it was another story. We were practically still writing the class while I was teaching it. Labs were changing on the fly. Many, many things broke. I remember someone helping me thought the class went pretty well, but in my opinion, it was a disaster. I’m a fan of high-quality deliverables. The students were gracious, and it was a free beta class. The bottom line is that we committed to a date, and the deliverables did not all appear in time for proper testing. This lack of testing affected the quality of the class (the product).

In later classes, we had things further developed but were still fixing a few nagging issues that repeatedly occurred in some labs. I would often get deliverables the day of or during class because we had a small team running on a tight budget. I would be up all night trying to test and ensure everything was working. Periodically things fail that worked before because the cloud providers seem to change daily. In December, I took a break from teaching at other work to do two things: 1. Finish my book, and 2. Test every lab in the class and make sure it worked flawlessly. I was able to do that and at the time for that iteration of the material, everything worked.

However, that is not the end of the story. Testing is an on-going process. As we prepare for another class, we are retesting the labs and finding new issues for reasons I mention below. Things change over time. Additionally, due to time constraints, we find things occasionally that we may have missed during prior testing. Every time we deliver, we improve the content. Testing and quality control is a continuous and iterative process. We plan for it and allot time to it.

Testing does not mean “getting something working”

Testing means going in and repeating a process, software installation, or class lab over and over until it works end to end without issues. When you get something working, you may have many stops and starts along the way. You might try one thing and then another, and forget that you ran a different command here or there at the point you got it working. You may think it works, and so you’re done, but unless you go back to the beginning and repeat the whole process step by step using a pristine environment, you might not notice an issue in one of the steps or software components.

Testing should not only include application functionality, but your infrastructure, and deployments. Often with infrastructure, testing is completely overlooked. Do you test that your network rules block what they should and only allow required traffic to pass through? Do you test that your logs and alerts work properly? People also forget to test their deployment systems and processes. When the deployment process fails, security issues can sneak in. Test the scripts and systems used to deploy to production in a prod-like environment prior to deployment. Testing should also include your backup and disaster recovery systems as I wrote about in a separate blog post.

Any change requires retesting whatever that change affects

When I worked on software integration at a bank, things were much different. The vendors we worked with would send pre-releases of software they planned to push into SAAS applications and APIs. We would review the change release notes, and our QA team would deploy and test our applications in a test environment prior to the date the vendor software went live. We would work with the vendor to resolve any issues in advance.

One of the things that makes cloud testing especially complicated is the constant change in the underlying cloud infrastructure and third-party tools. Now cloud and SAAS providers push out changes without any notification or ability to test in advance in most cases, so you need a method for monitoring for issues and a way to test and resolve problems that arise from unannounced updates quickly.

Repeatable results

When testing, you need some standardization on the platform and environment to ensure repeatable results. For example, 2nd Sight Lab used a free virtual machine from the AWS Marketplace for one of the labs. However, the code was maintained by a third-party. Invariably something that worked the day before would fail during class. One night I tested something at 2 a.m. that failed in class the next day!

For this reason, we now use a virtual machine image we designed for the labs, including our penetration testing lab. Even with that change, we use many different software components that change over time. As we rebuild our AMI with the latest source code, things break. Retesting is necessary. We also still have to deal with the underlying changes in the base cloud provider images and any software updates that occur. These changes have proven challenging, so we try to retest everything often.

Test coverage

Another mistake those not trained as professional testers may make is only to test what we like to call the “happy path” in software engineering and software quality assurance. That is the typical path people take to use a piece of software or a product. We intentionally have some people test our class labs who are not cloud, software, or security professionals to find out what things are confusing and to make sure the instructions are unambiguous.

Here’s an example: My nephew tested the labs and told me his screen was freezing when trying to enter his Linux password. I couldn’t understand what would cause something like that. I got on the phone and asked him a few more questions. As it turns out, the issue was that when you type a Linux password, it doesn’t appear on the screen. Anyone working in computers for years might not think twice about that. However, that was confusing to a new person. I added a comment in the labs, which we try to design for any skill levels with both introductory and more advanced sections.

When performing penetration tests, we often use a concept called fuzzing, which tries as many combinations of inputs and paths with different values. We also have people go through and manually examine every piece of functionality in any application we can find and leverage that output to perform even more testing. Our goal is to exercise as much of an application and cloud environment as possible to find errors caused by different values and paths. If time allows on a test, we also perform more reverse engineering to try to understand and exploit application-specific logic, infrastructure, and architecture.

Penetration testing

Most of these same concepts apply to penetration testing. Any change to the system may introduce a new security bug. Frequent and on-going security scans, assessments, and penetration tests help you determine if you have a new issue. Establishing a relationship with a partner who can perform repeatable testing on an on-going basis at reasonable rates helps ensure your systems remain secure over time, not just for the moment. Training your developers to write secure code and providing your QA team with the skills required for basic security testing allows them to assist in finding security problems before they reach production. Integrating security testing into your automated deployment process and monitoring helps ensure your systems are secure after deployment.

In the cloud, penetration testing should include testing infrastructure and deployment systems, as well as cloud applications. Make sure that all aspects of your cloud accounts and software are secure. Ensure that your tests are more than basic software scans but rather tests that aim for more coverage if your goal is to find as many vulnerabilities as possible and reduce that type of risk. In some cases, organizations may have another goal, which is to test their incident response teams to see if they can spot an intrusion. Understand your objective in advance to obtain the desired results.

Automated testing

To improve test results, automate what is repetitive and possible to automate. Not everything is. Some penetration testing work requires more in-depth system analysis and reverse engineering. However, automating what you can helps you work faster and eliminate errors. 2nd Sight Lab continually strives to automate the things we do to provide more value to customers further. If we can get a task done in 30 minutes that used to take 8 hours, then we have 7.5 more hours on a project to dive deeper into security testing. If we can automate testing of class labs, we can know that certain parts of our labs are functional and focus on testing the rest and building new material for classes. Automation takes time and requires investment. However, automating repeated steps where possible to eliminate mistakes, deliver a higher quality deliverable, and provides time for creating more value.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2020

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab