The attackers are in your network — now what?

How will you know and what will you do about it?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: Cybersecurity for Executives | Data Breaches

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

My previous posts in this series on Cybersecurity for Executives covered many precautions you can take to defend systems. No matter how much defense you employ, at some point, you have a security incident. That brings us to the question of how will you know when an attacker has breached your defenses and what will you do about it?

This post covers two topics at a very high level:

Security monitoring: Ensuring all systems have logging enabled, alerts set up for suspicious behavior, and someone is assigned to watch the logs and alerts.

Incident handling: When something in the logs indicates a security breach, a team of professionals trained to investigate the breach takes action to resolve the breach in the appropriate way.

Incident handling and monitoring teams

The most critical point for executives is to ensure you have an incident handling team that is separate from other teams building, testing, and monitoring systems, if at all possible. That team may be monitoring the logs, or you may have a separate team monitoring logs. At some companies, the team that monitors logs for security incidents is called a Security Operations Center, which is more commonly called a SOC (pronounced “Sock”).

Multiple reasons exist for having separate a separate team or teams for this security function. The first reason would be that those monitoring logs have training specifically for incident handling, so they know what to look for in the logs and how to create alerts. This training includes information about how to use security tools like an Intrusion Detection System (IDS) or Intrusion prevention system (IDS), tools to perform disk and memory capture, memory analysis, and reverse-engineering malware. This team should be aware of top threats and how to correctly determine if a suspicious activity in the logs is an event or an incident.

What’s the difference between a security event and a security incident?

Security Event: Something suspicious occurs in the logs.

Security Incident: The suspicious event is an actual security problem.

Although I highly recommend training developers, QA, IT professionals, and DevOps engineers to learn how to spot security problems in logs, these teams are not focused on this task. They generally focus on building things, and that is a different mindset. Hire people specifically focused on finding security threats and resolving security incidents for the best results.

If you are an executive or business owner, you can probably relate to the fact that context-switching is distracting and can waste time. One minute you are negotiating a contract. Then you are scheduling meetings. Next, you are looking over financials. Then you are preparing a presentation for an upcoming speaking engagement. Someone calls and asks if you can participate in their podcast. You are already overwhelmed. If you also had to watch all your logs for security problems, that would be hard, right? Larger organizations can afford to have people focused on monitoring security logs, and they usually have a separate team for this purpose. Smaller companies can outsource this activity to another company.

The SOC

I don’t know what type of security operations center all large companies have, but at one I worked for, it looked like something out of the movies. You walked into a big room with rows of desks pointed towards some large TV screens. The screens were full of news reports, Twitter feeds such as what the hacker group Anonymous was up to, and lots of logs. There was a separate room with no windows or cameras allowed in it. The people that worked in this room performed for need-to-know investigations and employee monitoring.

This team would try to identify threats in the environment and contact the appropriate group to resolve the problem. For example, if a particular IP address appeared to be infiltrating the network or a new strain of malware was taking advantage of a particular open port, the people in the SOC could contact the firewall team to create a new rule and block the attack.

Some organizations do not have money for such an extensive facility, so they make do with people in ordinary desks doing what they can to stay on top of the logs and alerts. Others outsource the SOC function to another company. If monitoring security is not the company’s expertise, this could be a viable option. Another reason companies may outsource the SOC function would be to get 24x7 monitoring around the clock by hiring a company in a different time zone. Be aware that opening your network and sending logs to third parties brings with it the addition risks I mentioned in other posts, so make this vendor selection with care.

It is still a good idea to get security training for your internal team. When the outsourced SOC contacts your organization to tell you an incident has or is occurring, make sure you have people prepared to deal with it appropriately.

Before the Target breach, the company outsourced some of their security monitoring functions to a company in India. The company in India reported the incident to Target in Minnesota, but whoever received that notification did not respond to it. Several possible explanations exist. For example, perhaps the message was not clear, or the person who got the message was distracted by another pressing matter.

Logs — All of them

To determine if your organization is under attack or an attacker has breached any systems, the company needs to collect logs — the more logs, the better. If a security incident occurs and no logs exist, it is impossible to tell what happened. This scenario is problematic for several reasons.

The organization may have been breached but doesn’t know it. As I already explained in prior posts, some companies hat attackers inside their network for months or years. If no logs exist, there is nothing that shows what the attackers are doing.

Some breaches incur fines if a breach of certain types of data occurs. If no logs exist, the people investigating the breach may not be able to tell how many records the attackers accessed. In that case, the organization must provide the worst-case scenario number of records. That amount may be much higher than the actual number of records obtained, and as a result, the company has to pay more than it should have.

If the company wants to take legal action related to a breach, they need proof. The logs provide that proof, showing what actions took place and who did it. If no logs exist, there is almost always no way to know what happened.

There are many types of logs. Different logs exist depending on what type of systems you are running and in what environment. All the different infrastructure in an on-premises office, data center, or cloud environment has system logs. Hopefully, every Internet-connected device has traffic, network, or access logs of some kind.

Overview of common types of logs:

Operating system logs: your laptops, servers, and desktops produce different types of logs for different system functionality.

Application logs: Your developers and vendors build logging into the applications the build. Make sure systems have logging enabled to the appropriate level. Usually, organizations turn off debug logs with detailed error information and possible sensitive data in production.

Server software logs: Turn on logs for web servers, mail servers, DNS servers, NTP servers, and any other type of server software running on your network.

IOT device logs: You may have logs for cameras, printers, automated coffee makers, and even fish tanks. If these devices have access over the network to other systems, make sure the logs are enabled, or at least log network traffic to and from these devices.

Network logs: As data flows through your network and to and from the Internet, it flows through devices that route the packets to the correct destination. These network logs are vital because even if an attacker gets into an individual device and turns off the logs on that device, he or she will generally not have access to turn off network logs. I explained previously how network logs helped me identify my first breach, even though I knew little about cybersecurity.

Cloud logs: Every cloud service you use should produce logs that tell you who logged in when, what they did, and what data they accessed. Ensuring could services can provide the logs you need should be part of your vendor assessment.

All the other logs: Any other device, application, or system connected to your network should have logs, including load balancers, firewalls, security appliances, HVAC systems, voice-enabled devices, alarm systems, mobile devices, and thermostats.

Collecting Logs

To get logs from all these devices, you need to make sure the logging functionality is on. The logs need to be secure, so someone cannot change them or delete them. You can replicate live logs to an alternate location. You may want to encrypt them.

You need to understand how the logs work. Is it a daily log file that overwrites itself each day? Is there a file for each day? Does the log file overwrite itself after a certain amount of time? If an attacker cannot delete or write into the files directly, he or she can write lots of log entries to leverage the way the system works to overwrite the entries that show the actions during the attack. Make sure you structure and back up your logs, so this doesn’t happen. Just I explained in my post on security disaster recovery, make sure your backups are sound.

Logs can take up a great deal of space. Organizations need to determine how long they keep the logs. Use different types of storage for faster or slower access, which can reduce long term archival costs when logs are required for sometimes up to a year for compliance in some industries. Organizations need to make sure they have enough physical hardware to store the logs. Alternatively, use a scalable cloud service.

Make sure all the systems have timestamps in sync. Often an attacker pivots through an environment, and the people investigating the logs need to compare logs from different sources to piece together what happened. If the timestamps are in sync, it is easier to correlate the data in the logs.

Monitoring logs

Now that you have all the logs, someone needs to monitor the logs. They can also create alerts for suspicious activities and use security tools that help analyze and alert people to suspicious events. I wrote about some of these types of security products in my last post.

Another type of security product companies use to help them monitor logs is called a SIEM (Security Incident and Event Monitor) — where companies capture as many of the logs as possible from different sources. This tool helps both track events, helps security teams find threats and respond to events. A SIEM may be used for monitoring by a SOC and help companies with logging requirements for compliance.

Looking for threats in logs is sometimes called threat hunting. Ideally, your security products and service find every attack in your environment, but I already explained why that is not always the case. At this point, training people to look for suspicious activity in your logs helps. You may have a SOC or an incident response team, but training all your IT and operations team members to be aware of threats helps your organization find a breach faster and shut it down.

Logs can contain any number of suspicious patterns such as numerous failed logins, excessively large data transfers, unexpected long connections, and memory dumps in logs indicative of malware attempting buffer overflows or other types of attacks. Some malware embeds itself deeply into an operating system so you won’t see it in any of the management tools in the operating system itself, but you see it on the network as it communicates with other devices if you know what is expected and unexpected and your logs are clear.

There are so many things in logs that can indicate a problem. Those are just a few to consider. Sometimes, you don’t or can’t see anything in the logs, but you know something went wrong for some other reason. Perhaps the FBI (if you are in the United States) came knocking on your door to tell you that someone is selling your data on the Dark Web, the place where criminals and people who want to hide transactions do business. Perhaps money is missing from your bank account. Maybe a user is telling you that their machine is “acting funny,” or someone thinks someone is reading their emails based on some external events.

Incident Response

Once an organization determines that a security incident has occurred, the process of incident response kicks in. There are different versions of an incident response process, and you want people who have received proper incident response training to handle the incident. Some people train employees to handle incidents, and others hire companies that specialize in this area of security.

I’ve already explained why logs are essential in a security incident. The other important factor is that all logs and data must be accessed, stored, and appropriately transferred, otherwise known as chain of custody, to be admissible in a court case. If the security incident involves a physical server, an exact copy of the disk is required and a hash to prove no one altered it at any point after the point of capture.

Unfortunately, some malware hides traffic in expected traffic or encrypts traffic, in which case an incident handler needs to look at the memory on the system. Some incident responders receive training to capture and analyze system memory to determine if malware exists in the system and what it is doing. The memory is lost as soon as someone shuts down or reboots the system. If the people handling the incident don’t know better, they may unplug, reboot, or shut down the system before the incident responses can complete their analysis.

Depending on the size and type of security incident, an organization may need to disclose the breach or call in law enforcement. Organizations need to determine how they communicate the breach and who needs to be involved. One of the reasons for having a separate team handle the incident is in case you have an insider threat. Additionally, incident handlers might not use the organization’s standard means of communication in case the attackers compromised those channels.

Preparation

Incident response involves a lot more than what I have covered here, but as you have probably figured out, you should prepare in advance. Avoid having someone destroy evidence or hinder an investigation by training them upfront.

Ensure that people know what to do and whom to contact when they suspect a security incident. A policy should state the actions people should take if they find or suspect a security problem. Communicate the policy to the entire organization — and don’t expect them to remember it if they watched a video or signed a document. Display the information in places where people will see it. You may have heard the announcement at the airport in the U.S. if you travel: “If you see something? Say something.” This a campaign by the Department of Homeland Security. If you remember this phrase, you likely travel a lot as I do (I’m sitting in an airport as I write this), and the announcements reinforce the message over time. Do this with your security policies, and especially this one.

I worked on a cloud audit at one company where many different departments managed the cloud systems they use. I prepared a list of questions for them regarding their security controls. I also went through the policies the company had created with the help of the auditors. One of the policies was whom to contact in the case of a security incident. When I asked five different teams, none of them gave the correct answer. Additionally, most of them were not looking at logs, or if they were, they didn’t know what to monitor to see if a security problem existed. The security team did not have access to any of the logs.

Most security professionals suggest obtaining law enforcement connections in advance. Additionally, if you think you may need outside assistance establish a relationship with that company and have them on call for the point when an incident occurs.

When a significant incident occurs, the organization may need to disclose information. Someone needs to talk to the press when they call. The organization may need to notify customers through emails, letters, and a statement on the website. Who is responsible for this in your organization, and are they prepared?

You may not want your employees speaking to the press. When I contacted people I knew from Capital One, some of them inside the company (not all) said they couldn’t speak to me because they were “being monitored.” It is probably a good policy to inform people that they should not speak to third-parties about an on-going incident or investigation. Though some people did talk to me who may not have received those directives and some people who already left the company, I did not publish everything they told me as I’m not a dirt-digging reporter. Some news outlets may jump to conclusions and write malicious articles that hurt the organization. Communicate to employees what can and cannot be discussed with external sources.

Practice in advance. Gather teams to do tabletop exercises. Practice capturing evidence such as disk or memory capture. Make sure you have the proper skills and tools to do this effectively without destroying or losing evidence. Executives should participate in incident response preparation. Executives should be involved in decisions and communication throughout the process.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2019

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab