Rapid Response, Integration, and Customization of New Threat Data Sources

Just like my previous blog post in this short series, it was instigated because of COVID-19, but it’s not really about it. It does however provide a solid real world example to introduce some customer challenges of dealing with new sources of threat data that appear, and the technology components we built into ThreatQ platform to enable us to rapidly respond to them. Our agility in the creation of these types of integrations is critical for our success, since customer intelligence requirements can change rapidly. After all the effort we’ve put into it, It’s turned out to be a capability we’ve ended up doing really well.

Crises and outbreaks require rapid response. One common thread during these challenging times is that from a data and product perspective, there is a strong uptick in the new, disparate sources of threat information our customers are consuming. This has happened before when other specific threats have gained global attention (e.g. Mirai, Wannacry, and Petya) but obviously not to such a scale. We also assume that it will continue in the future, but we just don’t know yet what the trigger will be. 

Many commercial threat intelligence providers kindly provide freely available packages of threat data to help the wider community outside of their existing customer base. Governments at all levels are sharing threat and outbreak-specific data. Data sharing via open source feeds expand greatly. This is a great thing to see to happen, because when we all work together and collaborate, we can all better defend ourselves. It’s these new sources that are driving many inbound integration requests. Becoming aware of these new sources is one thing, but enabling users to consume them, understand the data in aggregate, and enable it as part of their infrastructure and operations is a more interesting challenge. Especially since they look very different:

• Government provided advice and data
• Lists of new ‘potential’ domains that could be used for malicious activity (but are yet to be observed to be malicious)
• Known good and clean sources of COVID-19 data, nobody wants to block access to something clean
• Observed malicious content and infrastructure used in actual campaigns
• Aggregated and interpolated datasets

From our customer’s perspective, it results in a simple and well understood business need:

There is new data available that may help me in my mission, I want access to it now to assess how and if it can help to defend my organization. 

So as a product vendor providing a Threat Intelligence Management function, we need to be able to respond quickly. It does however place interesting engineering challenges on any vendor in this space:

• Agility: How can a platform be made to reliably consume and use these new sources of threat data as quickly as possible: This is actually one key area where customers actually measure products and vendors
• Sustainability: How can these integrations be made in such a way that they are robust and stand up to long term wide scale use 
• Accessibility: How can we empower non expert developers to create integrations like these that are robust, with integrated services for handling common external API error conditions, safe authentication, health alerts, detailed data logging, etc 

We decided to make an investment to build our own advanced framework to make this situation more manageable. It’s this specific engineering investment that has enabled us to now rapidly create new high quality integrations faster than ever before, and with more quality than ever before. Our goal was the ability to create feed integrations measured in minutes or hours, not days or weeks. And those feeds can be created by ThreatQuotient, our partners or our customers.

In the next blog post in this series I’ll take a dive into what one of these integration feeds actually look like from a ‘code’ perspective along with more details on our motivations, but for now let’s focus on the customer problem the framework has enabled us to address.

First the easy stuff: Many of our current integrations with commercial intelligence providers ‘just work’ as they publish new COVID-19 related information. Other formats are naturally compatible because of data packaging, for example, for sources that provide data in STIX 1, STIX 2, or from MISP connections. But what about the others that are provided in their own native formats?    

This is where rapid response and agility comes in. Our partners, customer success teams, and other groups are able to respond to these new requests quickly.

Here is a sample listing of COVID-19 threat data sources we’ve been directly asked about and now have native integration with. But what others are there? If you know of more, please let us know so we can understand them and perhaps build integrations. All of these integrations are currently in the publishing queue for our marketplace so you’ll see them appear publicly over the coming days. If you need early access to them, just let us know.

Cyber Threat Coalition: COVID-19

• Cyber Threat Coalition
• Cyber Threat Coalition in ThreatQ Marketplace

DomainTools COVID-19 Domain List

• DomainTools Free COVID-19 Threat List
• DomainTools CVOID-19 integration in ThreatQ Marketplace

Kaspersky OpenTIP (Threat Intelligence Portal)

• Kaspersky COVID-19 intel feed (zip file)

Malware Bazaar (including a COVID-19)

• Malware Bazaar Database

MISP Warning lists (COVID-19 specific)
There are actually three specific lists that are COVID specific listing benign domain names. Collected from various sources including Cyber Threat Coalition, Krassimir’s Covid-19 whitelist, & Jaime Blasco)

• MISP Warning List

Sophos Labs

• Sophos Labs COVID IOCs

Next time we’ll look at one of the more simple examples from above and look at some of the infrastructure behind it that makes it function.