The use of cyber threat intelligence (CTI) within the security industry is widespread and increasing over time. Attackers change their methods frequently to avoid detection by defenders who use CTI to keep informed of attacker behaviours and infrastructure. As a result, many security solutions use CTI to block or detect attacks.

CTI is used pervasively in anti-virus, firewalling, intrusion detection, web / email content scanning, and just about every other cyber defence domain, which means almost every organisation in the world benefits from CTI through the cyber security infrastructure they use. Mostly, this happens in the background with little user awareness of how critical threat research is to the effectiveness of these solutions. To provide signatures and updates to their solutions, vendor threat teams both conduct their own original research and consume CTI from other originators to improve their coverage and confidence. The need for timely intelligence and the sheer volume of information, has necessitated that CTI become cloud-delivered and real-time.

However, CTI in traditional security solutions has some limitations: 

1. Volume: The sheer number of indicators makes it difficult to keep devices well-informed about all the attacks going on in the wild. For example, there are only so many blocking rules a firewall can realistically implement at any given time.
2. Performance: Vendors focus on the most widespread threats to limit how many signatures they need to deploy because performance degrades as the signature set grows. Less prevalent attacks or threat actors may not make the cut.
3. Timing and timeliness: The speed of adversary activity creates gaps before updates are released and deployed. It also takes time for vendors to validate threat information – especially for blocking actions. Moreover, to block / detect an attack the signature needs to be on the security device before the attack happens.
4. Customised attacks: Adversaries make changes to malware, or how their attacks are delivered, to avoid detection by pre-existing rules and signatures.
5. Closed intel loops: Most vendors collect intelligence, formulate signatures and push them to their products via closed-loop systems. They do not share their CTI or the way their signatures function, nor do they provide the ability to add CTI without going via their analyst team. There is no way to verify if a particular attack will be blocked without asking the vendor or testing it. Note, this is changing and some vendors have APIs that at the very least, allow you to push your own CTI into their system.

The impact of limitations

As a result of these limitations, compromises are successful because the intelligence required to block them is not in place at the time of the attack, and therefore organisations need a reliable and consistent way to find compromises after the fact. Without this, attackers have practically unlimited time to escalate from compromise to breach. The statistics are compelling when it comes to the number of breaches and the mean-time-to-detect them, which get worse every year.

In an effort to compensate for these limitations, organisations are investing in a range of related capabilities within their environments, in other words on the client-side. These include:

• Behavioural and / or TTP detection that does not rely on traditional static indicators and signatures;
• Sandboxing and dynamic analysis to detect targeted attacks where no signatures exist;
• More SOC staff, increased specialisation within the SOC, and tools that improve SOC efficiency, to deal with the volume of  alerts;
• More focus on incident response, threat hunting and alert triage, which are increasingly separate functions with the security group;
• An ability to collect and use CTI outside of the signature systems used by most security technologies.

Furthermore, the number and type of CTI providers has increased rapidly over the last few years as nation states and criminal threat actors have become increasingly organised, mature and industrialised. The threat research groups that track them have become highly specialized, focusing on certain classes of threat actors, or even specific threat groups. They are also using new terminology like surface web, deep and dark web (DDW), advanced persistent threats (APTs), and organised crime groups (OCGs). Much of this analyst research produces finished intelligence reports together with real-time CTI “feeds” at speed and scale for system-to-system use.

Industry groups like ISACs, government agencies, regulatory bodies, vendors, news stories, friendly organisations, and peers all contribute to this growing volume of specialized threat research, adding to the information provided by paid intelligence services, and internal intelligence generated from incident response, threat hunting and the SOC itself. 

Addressing the data overload challenge

SOCs are expected to have a process for dealing with these multiple sources and the data provided in various formats. Traditionally, organisations have used spreadsheets, word documents, small databases, open source tools and the like. But none of these approaches are scalable or automated.

As a consequence, the threat intelligence platform (TIP) market has grown to help organisations manage their CTI.

The first function of a TIP is to store and manage threat information no matter where it comes from. The second function is to contextualise all the data by prioritising the small fraction of relevant information and thereby turning it into useful intelligence. The third function is to share  now actionable intelligence with downstream systems that use it for post-compromise detection, preemptive blocking, patch prioritisation, incident response, threat hunting, and many other use cases.

Returning to our five limitations from legacy security solutions discussed earlier, here is a summary of how a threat intelligence platform helps overcome them:

1. Volume: A TIP can store tens of millions of IOC’s and its performance doesn’t degrade with scale.
2.  Relevance: A TIP can apply a far more granular set of selection criteria to CTI versus one-size-fits all solutions that send the same intelligence to all products equally.
3. Timing and timeliness: A TIP removes the need for the intelligence to be deployed before the attack happens – integration with log sources (e.g., SIEM) provides retrospective detection no matter when the attack happened or when the intelligence arrived in the TIP. While deploying blocking signatures will always require a confidence check, a TIP can automate this, or provide analysts with information to speed the conviction process.
4. Customised attacks: A TIP can collect IOCs and indicators of attack (IoAs) from sandboxes, phishing emails, honeypots, and a range of other internal systems, and automate checking for successful prior attacks, or blocking similar future attacks.
5. Closed intel loops: many vendors have opened their ecosystems with APIs that allow a TIP to push new intelligence to a range of systems like AV, EDR, IPS, DNS, web, and email.

There’s no doubt that the security industry is better armed to deal with cyberthreats thanks to CTI. But CTI in and of itself is not a silver bullet solution to deal with cyber threats. Organizations need a way to store and manage all the data, contextualize and prioritize it for better decision making, and make it actionable. That’s why we developed the ThreatQ platform — so you can do all that and more to get greater value than you ever imagined from your CTI investments.