A few months ago I had the opportunity to speak at Security BSides Boston about a topic near and dear to my heart – how to make sense of threat intelligence data sets that are huge and continue to grow. Here are some of the highlights of the presentation that you can watch in its entirety here.

Threat data is growing
Research shows that the number of connected devices will grow from 28 billion today to an expected 50 billion by 2020. Meanwhile the number of connected people will increase from 2 billion today to around 6 billion by 2020. This means that both the network attack surface and the human attack surface will expand.

On top of that there is a cybersecurity hiring crisis. It is expected that there will be 1.5 million people missing from the security workplace by 2019. So with fewer defenders and more opportunities for adversaries, incidents and the associated data will also continue to rise.

Based on the average dwell time before a breach is discovered and the average number of reported incidents, at a bare minimum we’re talking about 328 concurrent ongoing breaches at this very moment. While most adversaries are financially motivated (stealing information such as credit card numbers, medical records and proprietary information that can be sold on the dark web),  other motivation includes hacktivism, cyber warfare and cyber espionage.

So how can we effectively look into large datasets from security incidents and help detect and prevent the next attack?

The need for data mining
Each incident report includes hundreds of IOCs (indicators of compromise). IOCs can be related to the victim’s host evidence (such as malware type, file name, hash file and registry keys). Additionally, IOCs can be related to the communication lines to the malicious link (such as IP address, domain name, URL and port numbers). Both host-based and network-based IOCs indicate a potential intrusion in your network.

There are many threat intelligence providers out there – from open source to commercial to industry-specific feeds – that continuously update a list of IOCs. Each provider adds a little piece to the puzzle in order to help illustrate as best as possible the current threat landscape. A threat intelligence platform helps bring all the pieces together into one virtual threat intelligence library.

The challenge is that the data isn’t correlated and it’s a massive amount to sift through. Each source can provide anywhere from 300 to 10,000 indicators a day. If you have 4 threat intelligence sources, that provide even just 300 indicators a day, that means you’re getting at least 500,000 indicators a year! This is a completely unmanageable situation. You don’t have time to investigate them all and pushing that information to your sensor grid (IPS, firewalls, etc.) isn’t practical. You end up with tons of false positives and poor performance.

Data mining techniques
The ThreatQ threat intelligence platform helps you more effectively use these big data sets through classification and scoring and allows you to make sense of threat intelligence in a single view.

Classification helps reduce the noise. You can classify by indicator type, including IP address, domain name, URL. You can classify using attributes, for example malware family, geography, language, etc. Classifying by adversary allows you to look at attacks focused on your industry and infrastructure. Classifying by incidents or events (age, owner, day of week, user ID) allows you to connect the dots across the kill chain. And you can classify by relevance, sorting by CVE, OS, user, brand of device – so you get an even more focused look at what you should care about.

Once the threat data is classified, you also need to score it because not all data is equally relevant to every organization. The ThreatQ threat intelligence platform can help you score and automatically prioritize threats based on parameters you set including indicator types, for example by IP address, malware type, host-based vs network-based. You can score based on the indicator source – open source, commercial, industry based, as well as internal sources like your SIEM and ticketing systems. You can score based on indicator attributes or context such as CVE, mobile OS, malware family and process, language, and geography. And, finally, you can score based on adversary attributes such as motivation, attack vectors and TTP (tactics, techniques and procedures).

Threat intelligence investigations
Context is very important in understanding threats, but unfortunately we don’t always get all the context we need so we need to use enrichment tools.  

Some of the methods we use are similar to the process in the TV show, “Who Wants to be a Millionaire?” where you use your lifelines to get help. For example, asking the audience a question is, in effect, crowdsourcing and using VirusTotal is a good example of this in our world. Or you might phone a friend when you have a specific question you know they can answer. This correlates to contacting a specific vendor that specializes in that type of attack for assistance.

Another method used in threat investigations is link analysis which involves identifying the relationships between bad actors, transactions, objects, servers, IP addresses, and specific malware families.

Whatever method you use, the human element is very important. You need a person with intuition working in concert with the technology and tools. The ThreatQ threat intelligence platform is designed to facilitate this with centralized intelligence sharing, analysis and investigation.

Threat intelligence effectiveness
So how can we measure the effectiveness of threat intel? It comes down to reducing time to detection and time to respond.  If we can reduce these numbers, then we’ve done our job.

I believe that the role of threat intelligence is to support threat defense effectiveness. The effectiveness of security tools and technologies erodes over time as adversaries discover vulnerabilities. You can strengthen defenses by finding the overlap between the available threat intelligence and what’s relevant to your organization, and then operationalizing it. This involves sharing curated threat intelligence with security teams and incident responders in order to make it harder and less profitable for adversaries to attack.  

Watch the full presentation.