The dotted lines between Threat Hunting and Detection Engineering

There's no way out, the practices of Detection Engineering and Threat Hunting are becoming utterly important within a Cyber Security Program.

How to define boundaries and establish ownership of the processes involved? Where's the overlap? Read along for some insights from the field.

Give me a hunt, I give you a detection

I have been writing about Detection Engineering for some years but never had the need to formally define it as the name implies it all: it's about engineering (cyber threat) detections.

Is someone considered a Detection Engineer only if writing detections for a SIEM? What about other event or log-based platforms (ex.: Spark)? What about Yara, EDR, NIDS, WAF? The concept applies to many technologies.

What's common from all of the above? There's some sort of automated built-in process. There's a rule or analytics engine loaded with some code (process) inspecting data streams (input) and generating events (output).

We give the engine some instructions (logic) and it starts spiting out indicators, alerts, signals or whatever you want to call it — automatically. Does this concept applies to Threat Hunting? Maybe, to some degree.

I believe there's some sort of interactive component to hunting as it does to detection engineering prototyping. The hunter has an idea and starts playing with it. Once successful, a security incident or even a ticket towards Incident Responders or Forensics Analysts might be filed.

Same goes for a detection engineer — when prototyping.

It's fairly common to spot actual incidents when prototyping a SIEM rule, especially when performing historical analysis trying to assess if the new detection is going to generate acceptable alert volume.

When does a hunting lead starts to exist? Is it at the moment a hunter has an idea (hypothesis) and starts digging though logs to validate it? Or is a hunting lead simply a breadcrumb coming in form of an alert?

Yes, it can get a bit blurred with more questions than answer. If you want to read more about the origin of the Hunting term, please check here, it's really worth the read and perhaps that can change your view about it.

But let's exercise that further.

Where does each practice start and end?

Threat Hunting commonly gets input from Threat Intel. So, is it about IOC scanning? Definitely not! Is it about looking for threat actor behaviors? Seems about right. But isn't that a detection motto too?

Try to follow this thought process:

• Threat Hunting works based on hypothesis? Yes.
• Can we automate that process once we find a solid one? Yes.
• So is a hunter also designing detection to a certain degree?
• Isn't that a valid input for the detection engineering process?

So for me, successful hunts have a play on Detection Engineering. And that's pretty much why I like to sometimes answers those discussions with this provocative meme:

Hunting also overlaps with other areas of SecOps such as security analysis and investigation. Here's another exercise:

• What's the output of an automated, successful hypothesis check?
• If I follow up on that output, am I a hunter or a security analyst?
• Is a hunter actually a senior security analyst then?
• Isn't following up on that automated output about triage and investigation?

Can we call a Threat Hunter a 'Jack of all trades'? 🃏 Well, I am 100% sure many small cyber teams are playing that role for a long time anyways, the hunters are no exception.

Regardless, at some point you really need a process with distinct roles and responsibilities (ownership), otherwise there’s no focus. No clear boundaries and overlaps might introduce inefficiencies and even re-work.

The Path: Analyst > Hunter > Engineer > Data Ninja

Also, we might consider that overlapping as a natural step in career development. How can a Detection Engineer write good detections without any security analysis or hunting experience?

While some hunters will specialize in Digital Forensics or Incident Response (DFIR), many will eventually evolve to Data Engineering related positions. Besides security analysis, Offensive Security, Data Engineering and Statistics (Machine Learning) seem to be a good path forward.

It's all about Data

Whether Threat hunting is about finding threats or intruders, there's one thing we should agree here: it's about data analysis. Nobody is pasting notes on a cork board or using nigh-vision binoculars to hunt.

Someone seats in front of a computer and starts questioning data — as boring as it may sound (I don't think it is at all!).

Having said that, I believe there’s one skill both roles should have in common: fluency in data query mechanisms. For example, if the main platform is a SIEM, mastering the query language is a must.

The dotted lines

Assuming a successful hunt leads to a security incident, why wouldn't you be willing to avoid or spot the issue in case it happens again? That's usually done via preventive and detective controls.

I keep repeating this in every project I start: a great detection idea comes from your own incident database. If you are not documenting incidents, how can you learn lessons and track progress to start with?

Now assuming you are tracking incidents, why wouldn't you focus on implementing controls around a threat that already impacted your organization (incident)?

Known incidents will not only help address any log visibility gaps but will also drive good detections when the telemetry is available.

Is "DETH" the answer?

Some organizations are proposing to merge both Detection Engineering (DE) and Threat Hunting (TH) teams as part of one single practice. Is that the way to go? Maybe.

The fact is humans don’t like to be labeled as it somehow seems to limit their abilities to their audience, which is another problem.

As a tech leader, I would ponder what makes more sense within the organization, what solid skills are already in the team and how the prominent members would be able to help you layout processes around it with automation and scalability in mind.

Try to pick one practice and embrace it! Whether you call yourself a Detection Engineer or a Threat Hunter, consider following this blog for more insights on both practices!

The next blog post will be an updated version of the following article:

https://ateixei.medium.com/jira-workflow-for-detection-engineering-teams-a7433f4c2a9f