How network traffic got me into cybersecurity

Also — being paid by a large hosting company to go away after reporting a security incident, and other strange events.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: Cybersecurity for Executives | Network Security | Data Breaches

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I used to run a software engineering business for over ten years, creating web applications and e-commerce sites for companies among other things. For a while, I co-located my servers at companies that provided network connectivity and server racks. I would bring in the servers, configure them with various parts bought from a variety of vendors, stack them, and maintain everything end to end. It wasn’t my favorite thing. I burned up multiple hard drives by installing them incorrectly. Someone told me this was no longer possible, but I don’t care to find out. I like virtual machines and some cloud providers for a number of reasons, this being one.

To be honest, the first place I hosted web servers was a guy’s basement with a T1 line, and the mail server was in my condo. That was around 1999, and yes, people used to run small businesses that way back in the day. I wasn’t the only one. Large companies, on the other hand, would host their servers in their corporate offices and buildings and connect everything with private leased lines. I know this because I helped manage telecommunications bills for a large oil company just after college. Many felt the Internet was not secure enough to use for highly sensitive business transactions and data. A lot of small companies were setting up servers and websites any way they could to get things running. I ended up hosting an e-commerce web site in that basement for a business that later became a multi-million dollar company.

Technically my first security incident involved an FTP folder I had an IT person help me set up on one of those servers, and he configured it to allow both upload and download. No one would find it, right? The guys with the T1 line called me up one day to ask why I was using up all their bandwidth. I said I wasn’t. We looked into it and figured out that someone had borrowed that FTP folder to serve up a Warez site (illegal, unlicensed software downloads). Oops. I learned the importance of never allowing both upload and download on an FTP folder. This incident was the early precursor to the S3 bucket problems people are having on AWS. I also learned that if you leave something open on the Internet, people will find it even if you don’t tell them it’s there.

I was hosting systems at a colocation facility provided by Internap in Seattle for a while. I had no problems with Internap. I thought it was cool to scan my hand in the biometric reader like they do in the movies where people get into high-security buildings…at first. But at some point, after dropping my palm on the reader to reboot systems in the middle of the night one too many times, and trying to remotely configure a load balancer over the phone with a Cisco-certified engineer, I decided to move to a managed hosting provider. For me, dealing with the servers wasn’t as fun as writing software. In fact, it was a royal pain. Everything would go wrong at the worst possible time, like when I was on a plane or in that part of the highway on the coast of California where you get no cell reception.

The managed hosting provider would set up the servers and operating systems, networking equipment, firewalls, and load balancers. I just installed my software on top of that. One reason I decided to make this change was that I thought they would know more about security than me. That was an incorrect assumption. Please note that I moved my servers to a large hosting company that some of you out there reading this are still likely using. But they are not the only one out there with similar issues with — so don’t assume that my story applies only to this company.

Although I had a master of software engineering at the time of this incident, no one ever taught me security. I didn’t know anyone who worked in security. I did my best to try to follow the published best practices (which were few and far between at the time). I had SSL certificates. I reduced my liability by never storing credit cards. That was all handled by the processor. I thought no one would bother with me since there was nothing to steal. Many small businesses believe attackers won’t bother with them because they are small. I can tell you from a first-hand experience I had in the early 2000s and the security research I still do that this is not true.

Looking at network traffic is how I discovered and proved my first data breach was happening. I presented the information to my hosting provider — whom I expected to help me. They didn’t. They told me they scanned my system with a virus checker and told me it didn’t find anything, so everything was fine. I knew it wasn’t. I saw strange network traffic every time I restarted my little customized open source web server. It would be hit with five random requests from computers around the world and then stop. I wasn’t getting as many orders suddenly, and I knew something was wrong. When I explained this my hosting provider, they told me not to worry about it. But the thing is, I was losing money, and I knew it. I didn’t like that so much. So no, I’m wasn’t about to not worry about it.

I wasn’t sure what to do, but I knew it had something to do with that network traffic. That was just odd. I went back to support and asked to turn on all the firewall logs. They were only showing me denied traffic. I thought maybe something in the allowed traffic would give me a clue. I wanted to see active connections to my computer. It was just a guess, and I had no idea what I was doing at the time. The first support person I asked could tell. He refused to turn on the logs and told me it would be too much traffic. His words did not deter me. I hung up and waited for the night shift. Now I knew what was possible and what to ask. I also was aware that the night shift is typically a little more lenient, shall we say. I called up and confidently asked the support person to turn on all my firewall logs. He did it happily and without any questions.

At this point, I discovered something even more intriguing. There was a large amount of traffic going out of my server that was indicative of email messages. I didn’t know how to look at network packets to see what was in them as I do now, but I knew that email traffic ran on a specific port. I’ll explain ports in another blog post but it was traffic on port 25 for those who know about SMTP and email. I knew it was too much traffic for what I was running on that server. I sent about eight email messages per day.

As a side note, I was sending very little email from this server because I had outsourced all my e-mail hosting in search of a provider that could stop spam and fix email problems. I was frustrated because I was getting 900 spam messages per day. I started investigating all the email traffic and discovered that the same spam message would arrive five times and the originating computer address showed the messages came from companies like Microsoft, HP, along with other small businesses. It seemed like somehow attackers had compromised these companies in some way and they didn’t know it!

I tried to contact the companies sending the spam, but all the means of contacting them had no effect. I don’t know if they were inundated and not reading the emails or the spammers had infected those mailboxes as well, so no one ever got the messages. All the spam-fighting services out there couldn’t make a significant impact until Postini came along (purchased by Google and now part of Gmail). That was the first product I found that could make a dent in the spam problem. If you want to know the whole incredible story about spam back then, learn about organized crime rings in Russia sending it, and criminals threatening business owners and their families who tried to stop the spammers, read a book by Brian Krebs called Spam Nation.

When I saw the mysterious traffic in the logs, I turned off the email application on my server. Doing so should have stopped any emails from being sent, but when I checked the logs the rogue traffic was still there! I wasn’t running anything on my server that would be creating that network traffic. I thought that now my hosting company would surely believe me. I took the logs back to them and showed them and explained what I did to uncover it. Once again, they said it was no problem. Don’t worry about it. Really?

But it gets even weirder…I have often wondered if there was an insider at this company working with foreign spammers and they just thought I wouldn’t figure it out. Or maybe they just thought I was a dumb girl and what I said didn’t matter and they had no clue what was going on. I can make up all sorts of outlandish theories and a good movie plot to go along with it, but the fact is I was just trying to run my business and get by. I wasn’t concerned with any of that at the time. I just wanted to pay my bills. I don’t know for sure what the story was internally, but here’s what happened next.

I had to figure out through trial and error how to get the malware off my system. I gave up on talking to support because that was clearly useless for this particular matter. I had no idea what to do about malware, but I figured something had to be running in on my system to create this traffic. I looked at every application running on my server to see if something was there that wasn’t supposed to be. Eventually, I started turning services off, one by one. I proceeded to crash the system a few times and did have to call support to reboot it, but other than that, I was on my own. I wouldn’t even know who to call to help me with this. Eventually I turned off one of the services and the traffic stopped! I felt triumphant.

I didn’t know how to find the malware or figure out how it got on the machine but this — and 900 spam messages per day — are the reasons why I’m writing to you today, speaking at conferences like RSA, and providing cloud security training and consulting services to companies. I have since learned how to use sniffers, handle security incidents, and reverse engineer malware among other things on my journey to obtain a masters degree in cybersecurity. I now hold the GSE certification, which some claim is one of the hardest to achieve in the industry.

But as you can see from my story, even if you don’t have people on staff who know how to do all that or hold cybersecurity certifications— you hopefully have someone on your staff who can look at basic network traffic logs. These logs can tell you a lot. They can probably tell you if you have a security breach right now.

My story does not end there. After I figured out how to turn off the malware, I went back to my hosting company. I didn’t bother to tell them what I discovered because at this point it seemed futile. I just asked them to deny outbound network access I didn’t need. My computer didn’t initiate outbound connections to anything. It only replied to requests it received. I only wanted to allow the most minimal traffic required for my applications to run correctly. They argued with me! They said no one checks outbound traffic. I said I didn’t care what everyone else does. I wanted to block it. I figured out how to do it myself. I had very restrictive network rules in my firewall settings that only allowed what the applications on my server required.

And magically, after doing all that and stopping the malware — the orders on my website jumped back up. The malware did not come back. What I didn’t know then that I know now is that I had made it nearly impossible for C2 traffic to function on my server. I will explain this further in an upcoming blog post. I was happily getting some nice side money from my hostel booking venture for about a month, and then even stranger things happened.

First I need to tell you the backstory. I had met a hostel owner while visiting Australia. I stayed in at Manly Beach near Sydney for three months. Why? Because I needed an escape. The business I helped grow to a multi-million dollar company had shafted me. I just wanted to get away and see the other side of the world and start over. I thought about moving to Australia but that didn’t happen.

While I was there, I started talking to this hostel owner. He was complaining about the cost of hostel booking web sites he was using to advertise. When I got back to the US, I created a booking web site for him and just took a cut of the bookings instead of charging him for it. I was pretty good at SEO (getting web pages high up in Google rankings), and my idea was that I was going to create an online hostel booking web site and make a zillion dollars so it would be worth it.

At some point prior to the security incident, the owner of the hostel in Australia contacted me and said there was a new owner. OK fine as long as I’m making money, right? How was I supposed to know what was going on at Manly Beach? Well…after I found and stopped the malware on my server, this new owner, if that’s who he really was and all this was legitimate — contacted me and complained that he was getting too much business. Who complains about making too much money? He also complained the prices of the hostel bookings were too high. I didn’t set those prices — he had control over that. And people were paying them! None of this made sense but I did what he requested.

I had shut down one function of the site I suspected to be involved in the attack, and he complained about that as well. I turned it back on, but as mentioned had made changes to better protect the system like blocking the ports I didn’t need to be open. I also created my very own web application firewall (WAF) even though that term didn’t exist at the time. I was inspecting every single web request and kicking things back that looked suspicious. I learned about website attacks like the OWASP top 10. I tried to subvert anything that would get into my web site and do something evil. The malware did not come back even after restoring the suspect functionality.

By the way — I tried to pitch that WAF to investors and sell it to other customers, but no one understood what I was talking about or believed that these problems I explained to them existed, so I gave up. This seems to be a repeating theme in my life…when I explained to an investor all the things I was seeing back then and recommended that he invest in security — he told me to write a blog. That old security blog is still out there, and some of it is based on the traffic from that WAF. I know a lot more now than I did then.

As I continued to research and talk about security, boyfriends and co-workers told me I was paranoid. When I tried to suggest to a company looking to invest in new security technology that they should buy a company called Observable Networks, certain people never seemed to care what I had to say. Now Cisco has purchased that company and turned it into Stealthwatch Cloud, a service that can monitor your network traffic across all the major cloud providers. Well, you don’t have to listen to me either, but if you do I might be able to help you out with your cyber and cloud security strategies, and that’s why I’m writing my online book about Cybersecurity for Executives and teaching cloud security classes.

We are still not quite at the end of the story. Another strange thing happened. After about a month after I got the malware off my server, my hosting company contacted me and offered to pay me two months of hosting fees to go away. What? Sure. I had not been asking them for any help at this point, and I didn’t tell them, but I was not impressed with their service (understatement) and was planning to leave anyway. I took the money and moved to another hosting provider. They were great, but eventually, I moved all my systems to AWS. Although initially there was no way I was going to move my e-commerce servers to a cloud provider, over time the service improved. I will be explaining some of the security benefits of AWS and certain other cloud providers in future blog posts.

A few weeks after I moved my servers, a story hit the news — and I was following any cybersecurity or IT-related news sources I could find by this time. The headline was something like “Most servers ever were hacked” in data centers and were spewing out stock spam. The malware had an embedded Kaspersky virus checker. I was able to find a reference to this malware on the SecureWorks site in 2006: “The SpamThru bot has the capability to scan the system for other malware using a pirated copy of Kaspersky Anti-Virus.”

The malware wanted to make sure no other sloppy malware co-existed that might cause someone to discover their spam generating software. Genius. The hosting company scans the system. No viruses. All is well. I wonder what virus checker that big hosting company was using at the time. Kaspersky, perhaps? I was pretty sure that this was the malware on my system, but I wasn’t at that company anymore. Even if I was I didn’t have the skills to prove it at the time. Today, it would be a different story.

And that, my friends, is how I got into security. Network traffic can be one of the most powerful indicators of malware on your network. Architecting system communication with malware in mind will make it easier to spot. You can design your network to make it harder for compromises to spread. Don’t assume that the company hosting your systems knows more about security than you do, or is paying attention to your network traffic. That is typically your responsibility unless you have a contract with someone that explicitly states they are doing it for you. Looking at basic network logs is a great place to start to for those who want to get into cybersecurity.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2019

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab