Why does software security keep falling off your budget?

[#1]

Today I'd like to pose a very simple question that's been troubling me for a while now - Why do efforts to build and maintain software security programs keep falling off the priority table in the budgeting cycles at even the "big enough to know better" sized organizations?  It's a question I've been wrestling with for some time now and a few conversations with some very intelligent colleagues from companies you would definitely know at InfoSec World and other venues over the past 3 months have got me perplexed.  Perhaps the answer to this question comes down to corporate culture, enterprise priorities, or maybe it's something else entirely.  Without pointing fingers, this post is dedicated to those who continue to struggle with software security long after we all think everyone should "get it" by now...

Addressing the problem strictly off statistics, we know for a fact that approximately 3 out of 4 modern attacks against your enterprise or organization come at your applications.  Whether it's at your website, at the mobile app you've deployed, or your enterprise API - you're being attacked through the place where the lowest defenses are - the application.  So why is it that we keep spending 3 out of 4 budgetary dollars on network security?  I've heard some of the more smart people in our industry retort with various answers, various ideas, and all sorts of fingers pointed but there hasn't been a compelling way to resolve the situation.  Perhaps we should be asking a more fundamental question - is there even anything to resolve?

Organizations spend money on network-based security because we can understand it, at least that's what I think.  It's easier to understand packets coming down the wire and putting something between you and the "bad guys" to stop those bad packets from hitting you.  Firewalls, IDS/IPS, and other still-critical devices continue to garner a lion's share of our budgets when they're doing little to protect our applications - with notable exception to devices that actually understand application traffic and are able to react accordingly.  Still ...the proportion seems incredibly strange.  Perhaps it's because we feel like if we buy one IPS we can protect a million devices on our network, whereas software security is so nebulous, so difficult and so case-by-case that it's difficult to put a solid risk-reduction metric on.  I think this is more to the point of why we fail.

Firewalls stopping packets is measurable... and as much as it makes most of us cringe (yours truly included) saying things like "our firewall stopped a million attacks last quarter" when in reality you mean a million potentially malicious scripts, bots, scans and other attempted intrusions is convenient and can show perceived value to the stake-holders.  I say perceived value because to us that really understand attacks, attack surface, and the difference between a Nessus scan and a carefully crafted exploit - that metric means less than zero - but to a senior manager looking for an ROI on the spend on budget ...genius.  Perhaps this is what is at the heart of it all - poor metrics.  Now I seem to have uncovered two problems - ones I know I'm not the first to point out...

First, we have been measuring the wrong things, as in the example above, and it's come to bite us in the rear.  We've been trying to measure that which is convenient to us and our cause, and to show the business how much they need us - often sensationalizing the truth (heck the DOD does it!) to suit our own ends.  Now those chickens have come to roost, and we're in deep, deep trouble.  When we come back with metrics we can't easily sensationalize - for example it's much harder to 'tweak' an ROI out of software security assurance programs than a network security approach, every time - we are told that clearly what we're trying to point out is of little significance compared to those massive numbers we've been showing for years on the network side ...and boom.

The other thing is this perceived value problem.  As I said earlier, when you can say (and prove, to some extend) that there were a million malicious packets, or "attacks", in the logs the firewall stopped, or thousands of port scans or worm attempts the IPS stopped ... it's easy to get a cheer from the executives.  But when you say you've got to educate developers, lengthen delivery time, and otherwise spend more money to attempt to merely reduce risk on an application deployment - well ... wouldn't you laugh that one off too?

Alright, having the issues corners, I think, means we need some resolution.  Here it is - better measurements.  We need more measurements that are closer to business value, and we need to start phasing out those silly "stopped a million attacks" metrics we've been yammering about for the last decade ... it's really not helping anyone right now.  So who's in?  There are groups out there that already do this - and getting away from fear-based software security is paramount - but can we push ourselves to do it rather than complain and whine for another 10 years?  I sure hope so...

One note though ... there will always be organizations that refuse to understand, and shift budgets to meet real-world risks.  Those are the jobs you don't want to stick around for in all likelihood because when things fail - and they will - you'll end up the witch being burned at the stake.  For the rest of us - starting right now let's say no to bunk metrics.  Business-relevant value and risk metrics aggregated into KPIs or bust!

[#2]

I definitely think that metrics are the problem and the answer as you stated. We've always been told we can't manage what we can't measure, but at the same time, metrics say what you skew them to say. If you want to show that you're shiny new firewall is paying for itself, you dig down to number of packets, If you pull back and ask how many IP's generated those packets, you'd probably have a much smaller number.

I'm still a newcommer to the app-sec area, but I believe that the lack of a metric like the number of packets blocked is what is slowing down the budget/buy-in currently. I know that a comment heard around our office is that until there is a "b-word", no one cares. I think until there is a way to get a "whoa" response from those with the checkbooks, it will continue to be that way.

[#3]

There are several reasons for this.

1. The people that run the networks and systems administrators have always had a more security focus within corporations - Firewall; AV deployments, etc. 
2. It is easier to think of adding bigger and thicker walls to protect you.
3. Vendors sell to the Sys admins since they are traditionally in charge of "security" budgets.
4. Most corporations seek the cheapest outsourcing solution to their development woes - security/privacy are no where on their radar.
5. Most developers don't consider security to come under their purview unless it is clearly stipulated in non-functional requirements - and even then, they aren't qualified to deliver it.

Software security takes effort and thought. Defining access policies for users; defining access policies between devices; validating inputs on your front end; validating inputs on the business; validating outputs; using parameterized Stored Procedures; hardening the OS/Database/App Server etc; changing default usernames/passwords; TESTING; TESTING; TESTING.

Buying a new firewall? Not so much.

Personally, I've always considered security as part of the quality process - It doesn't cost as much as many orgs think. Getting people to initially change-up and improve their coding skills, testing and attention details, does take a little effort; but once the process is in place and part of the culture, the costs come down and the quality and reuseability of the code improves massively.

[#4]

[to add to the above]

My own analogy is: "Barring the doors (firewalls) but leaving the thousands of windows (applications) wide open".

[#5]

A newcomer to this arena myself, I agree with the above comments and this article.

Budgets are based on tangible results in almost every arena, so these metrics are a huge part of the cause and effect of said budget results. It is easy to pinpoint physical archetypes or fortress type security, throw up a power-point, deliver some metrics and get a bunch of oohs and ahhs on how secure your castle is. Meanwhile "Hacker A" just pulled up your front-facing client's webpage injected a line of code and compromised all of your client's account numbers based  on a new vulnerability.

Relying on Network Metrics will not work but rather perpetuate the fallacy of "Security Theatre" Sure, you can and should use WAF, scanning tools, watch traffic patterns for anomalies, black-list suspect IP’s to protect yourself and all of those things. But the fact remains that the reason you have to continuously spend large amounts of money on these things is because you don’t want to allow for the source of these metrics to be fixed.  

Not to mention that the attack surface in application security is so varying that at any given time a new attack could appear that the current security does not take into account. It will take time and money to review and fix current code and even going forward to produce new code.  But trying to explain that, you're betting on the come so to speak, and hopefully it never does. So that is a real catch 22.

Application breaches/risks/vulnerabilities/statistics are in the news every day, but they aren't necessarily the news that developers or CEO's read. 

We read it, we concern ourselves with it and then we tell each other about it.  (echoes of one of Rafal's previous articles regarding education and information sharing)  Security vendors read it, and produce products to mitigate the Appsec risks. (Not the actual source of those risks)

Executives or even most developer’s primary concern or area of expertise is not security, it will take education to explain to them why AppSec is important and that securing software application begins with coding practices, not treating symptoms after the fact.

So to answer the question asked, what do I do about it? My answer is coming from a slightly different angle, since I am not in the upper ranks and am not an expert by any means and we don't have a CSO or CISO.  So it may not be of any use here.  However, I feel the theory still applies and doesn't discriminate on company size, (only the approach may differ) so I will share it.

I practice Awareness, Education and Persistence, with a dash of FUD , (Financial rather than fear, Uncertainty and Doubt.) Money is the bottom line so exploit that point.  For example, I forward information and articles, like this one http://threatpost.com/en_us/blogs/cisos-guide-application-security-part-2-growing-threat-application...  regarding application security to the people who make the budget/implementation decisions and I ask the question, "DO we address this?" Even if they don't answer me, the seed is planted and I don't stop. Trust me, they would much rather be the one to address it before it happens than to have me say “I told so and so about this” after the cows escaped the barn. (or horses? Or white rabbits? )

 Is it effective? Yes.  I see the issues being raised and changes being implemented. Not overnight, but if they aren't aware or educated about the real risks or the source of them, they may not realize how it will affect their bottom dollar. Network Sec isn’t writing the code, so they won’t speak up for it.

In situations where the CISO is experiencing roadblocks, the same tactic could be used with a variation. "We NEED to address this”   Do they know what the risks are? An average breach cost (5.5 million) regardless of origin still costs the same?  Do they want to be in the headlines? Do they want to watch their stocks plummet 13%? Isn’t the extra time to deliver the product a bigger pay off? Do they understand that this will reduce the attack surface of the company in the long run? One of the statistics that blows my mind is only 6% of breaches are actually internally discovered. The rest are reported by outsiders. How’s that for security metrics?

Basically you are using the results of other organization's misfortunes as your metrics.  The network admins have  visible results, and they have an easier sell. Theirs is a retroactive rather than a proactive approach.  And traditionally that has always been an easier sell in any industry. But does the decision maker understand why the WAFs and scanners are doing what they do? That these are not fixing anything, they are simply stopping access to these holes, while it is you who are doing the fixing.  It’s like going to the doctor and getting a pill to stop the symptoms, meanwhile the actual cause of the symptom is the illness and eventually another symptom will emerge and another pill will be prescribed. Why not allow the doctor to treat the source of the symptoms? It will be cheaper in the end.

So my answer is not a blanket answer, it works where the decision makers are easier to reach and pockets are harder hit, and the hunger to stay competitive is rife. But I don't have a panacea for this situation.   I hate to say it, but it seems that within some organizations, unless a shift in thinking occurs, the only thing that will make a difference is a breach, and even then they will use a security product to address it rather than understand that it can be avoided in the first place, but it won’t happen overnight. Nothing happens overnight.  

[#6]

I like the discussion very much - and agree with the feedback above although mostly technical in nature. Let's take a step back regarding Budgets and Spend (which is different than "cost"...of a breach which we don't naturally budget for, right?) - who the heck is even in charge? The CISO, CFO, CIO or application LOB's? or all of the above (problem #1) which is why an overall security budget is so difficult to wrap nicely? Is there really a separate line item in the P&L for security budgets or just wrapped into the application budget, hidden where it goes undetected? Which is problem #2.   We can't figure out the metrics (technical and business) as rightfully mentioned above until we really know who the audience is.  50% of Global 1000 accounts have a CISO according to GArtner and most of them report into the CIO - although this is changing.  Google's CISO is now reporting to the Chief Legal Counsel .....not sure he owns the entire security budget ......The dynamics of the C-Suite ownership in the budget discussion adds layers of complexity on top of an already complex discussion for those chaps to follow