In mid-2010 I was running a large Defense contractor SOC (35+ analysts) and was forced to build what’s currently known as a threat intelligence platform (TIP) – to manage indicators, adversary profiles, spearphish trends, and team commentary/annotations.  I didn’t have a buy vs. build option because back then there wasn’t an open source or commercial technology dedicated to organizing or streamlining day-to-day workflows around hunting adversaries.  Let me summarize my “build” lessons learned and share why a “buy” option is a better path (IMHO).

At the time, threat intelligence was just getting started (…in the mainstream sense) and “information sharing” was in the industry cross-hairs.  However, my team was losing a ton of efficiency between grabbing indicators/intelligence and storing it in siloed repositories analysts kept to themselves. There was a lot of duplication of effort and wasted time as analysts were stepping on each other’s toes, chasing the same indicators – a relentless open wound many organizations are still facing.

This red flag needed to get resolved immediately both from an effectiveness standpoint but more so because the analysts were reaching a tipping point level of frustration as they were spending a majority of their day performing due diligence on indicator extraction vs. true analysis.  The best way to do this was with a threat intelligence platform but, unfortunately, in mid-2010 the “buy” component wasn’t available so we were forced to build our own.  Here’s an account of our trials and tribulations…

Working with the analysts and management – we defined the “what” through technical requirement and business requirement gathering.  The analysts, largely comprised of security analysts, incident response and threat intelligence folks, along with peripheral input from malware and signature engineers, focused on automation and analysis capability – trends basically.  Whereas, management insisted on quantitative key performance metrics to measure the project against.   

The first problem was trying to justify a true web developer to be the cornerstone of the project.  When I originally approached my boss with the ask I first got a shoulder shrug because a developer in a shop of security analysts caused some head scratching.  After sitting him down and slapping him around a bit [read: chipping away over several weeks] he finally handed me a list of engineers sitting on the “corporate bench” waiting to be picked up by a U.S. Government contract.  I knew how the story would play out – poorly – but had to prove it.  So after 3 months working with someone off the bench we had a rudimentary prototype the analysts were kicking around.  Then it happened…the guy got his ‘white badge’ and got pulled up to the majors and had to report to a government contract the following Monday.  Where did that leave me?!  S.O.L. – documentation was lacking and the next resource would have to reverse engineer his code. And as any developer will tell you it’s easier to start over than to reverse engineer code and/or logic.  But I had no choice. After repeating that same process 2 more times I finally managed to prove my case to my boss that we needed a dedicated developer – NOT someone on the bench.

Job req posted…job req filled…progress gained!!  Now at this point, I’m 8 months into my initial effort to build it due to several false starts by management but it was starting to take shape.  It took another 6 months to build an alpha prototype where analysts were seeing value.  Up until this point, the effort’s focus was indicator management – primarily the ingest of indicators through crawling text with regex (I know…so 1990 from where TIPs are at right now).  Keep in mind, at the time, most sharing revolved around copy and paste from emails or web portals because vendor APIs lacked capability and consistency.

The next milestone of indicator management was integration with existing security infrastructure/applications.  Unfortunately since our developer wasn’t intimate with security technologies we needed to marry a security engineer to him.  It wasn’t an FTE effort but stole probably 35% of the security engineer’s already precious time – he was able to wear dual hats for a couple months while the core tools were integrated.  As a result of bi-directional integrations and the exponential uptick of OSINT blacklists we really needed a dedicated database/big data engineer.  Up to that point we had been able to get by with a relatively simplistic database architecture. But because of the volume of data and the growing complexity of union statements and typical database edge case efforts the effort warranted a FTE.

As more and more analysts across the team began to leverage the TIP and actually rely heavily on it…the code grew and “quick code fixes” morphed into significant efforts in logic contortionist stealing time from adding new features.  A second developer would provide redundancy as now the original developer has security/intelligence development under his belt and that skill-set is silly rare!  …basically you can expect to lose him within the next 10-12 weeks to greener pastures and a compensation bump of 20-30%…one that you can’t reciprocate under the handcuffs of corporate policy.  

We’ve now reached a point of no return where this fun homegrown app is turning into a Corporate-supported initiative with line items in budgets (most SOC Managers can hide 1-2 FTEs in the numbers) but now we need a second developer (or script ninja) and soon a QA engineer.  I anticipate I lost you at QA!!  QA in a SOC?!  You could probably get away with the developers performing peer reviews of code before committing it to the trunk code base but then they are moving at half the pace.  And let’s face it – the analysts’ feature requests are piling up at triple the rate!!  Now is NOT the time to take your foot off the gas!!  This also transforms a portion of the team into a software development effort – a 180 degree pivot from most SOC teams.

So back to the juxtaposition of buy vs. build. I realize I’m now in a vendor role and my answer is bias, but in my experience running several SOC teams …90% of the time the answer is simple – BUY!  My operational ‘gunshot wounds’ described in my blurb above have taught me several lessons about building solutions including:

• Resource challenges: Finding someone (let alone a small development team) with the skillset to build a core software component of a SOC team is rare – and keeping said person beyond the honeymoon is even harder!  Especially in an environment with compensation policies.
• Financial challenges: Building an internal team to support even a small homegrown application consists of 4 FTEs at the very least!  That loosely consists of 2 developers, a database ninja, and QA at likely $100K + 25% fringe pushes an annual price tag of $500K in headcount.  On paper that doesn’t seem too bad…but ALL SOC Managers will tell you getting additional headcount vs. a tool is exponentially more difficult.  Especially when that tool has a lower price tag and shoulders 85-90% of the load.
• Operational disconnects: Software development and security operations are very, very different beasts!!
• Vendor management: As a SOC Manager – it’s always easier to reach out and strangle a vendor vs. firing and re-hiring an employee.  Most SOC Managers are 20% technical chops, 60% human referee, 20% Executive translator so the less FTE juggling that needs to occur the better.
• Innovation: Vendors, especially annual subscriptions, provide for purchasing leverage to ensure the product roadmap still aligns with buyer needs/expectation.  The vendor landscape is robust enough that competition displacement is accompanied by pain points but not insurmountable.

In conclusion, buy vs. build isn’t a slam dunk case… even though I’ve experienced both and am a bit biased.  Do the homework – run the cost projections of maintaining FTEs against a software buy. But regardless which direction you go…build a maturing strategy and be ready to shift accordingly!!