Logging: Opening Pandora's Box - Part 1 (Anxiety)

This post kicks off a series of posts titled "Opening Pandora's Box", that will cover the untapped wealth that is your corporate logs.  After talking about logging with some people in the customer space, and our engineering and research groups back here at HQ, it's clear to me that logging is more than just something that everyone should be doing ... it's like a Pandora's Box that many organizations are almost afraid to tap into.  I thought it would be a good idea to explore this more, so kicking off this series is phase 1 you'll probably go through when you start thinking about logging - anxiety.

 

Right around now you may be asking yourself ...why would anxiety be the first feeling you get when you start to think about logging in your organization?  Elementary my dear Watson - it's because of what you're probably going to find, and you're afraid of it.  Think about how well you log right now, and think about how much it tells you about your environment.  Logging can tell you everything from how efficient your servers are, to low often specific IP address ranges belonging to certain countries visit your websites, and even whether you've provisioned enough capacity for your applications in that cloud.  Now let's imagine for a minute that you're getting about 1/10th the value from your logs that you can possibly get.  That's right, somewhere around 1/10th the value of what you could be getting - that's around what you're getting today.  The other reason that you're probably starting to feel some anxiety is also the white elephant in the room - logs can give you lots and lots of hints on whether you've been hacked at some point, and how bad it is.

 

Logs are a powerful thing, but most places I've had the pleasure of working with leave the vast majority of their logging treasure untapped.  Let me give you an example from real-life, from my past life.  In an organization I used to work at we generated around a quarter-terabyte of logs a day... from just the security devices.  I'm not kidding.  This only includes things like our firewalls, IPS network, and other security-based devices ... I can't even tell you how many lines of logs that was.  The maddening thing was that there was absolutely zero chance that we were ever going to look at any of that in anything other than historical search... if that.  The Perl scripts one of my colleagues had written would chug through a days' worth of data on our fastest server with the fastest disk array in just under 5 hours searching for pre-defined patterns in the normalized logs.  Again, maddening.

 

The truly insane part of this whole story is that while we were generating just under 250Gb of log data from our security devices per day - this was only from the security devices and didn't really include anything else.  The anything else should have included applications servers, applications, systems, workstations, authentication mechanisms, databases, physical access and so much more.  Here's why I am confident in saying that my team would have gotten anxious any time someone started talking about digging into the logging capabilities and data in our organization - we had no idea what sorts of information was in there.

 

An interesting story comes from a security engineer I recently talked to at a conference who didn't tell me where he worked, but relayed this story.  I think it perfectly illustrates how anxiety is the perfect place to start when you start thinking about logging, and analyzing all those logs you have access to.

 

I'm paraphrasing now...

"We started by realizing that our syslogd server wasn't enough anymore and that even if we stared at it all day the screen scrolled by way too fast for us to see any of the stuff in red that meant that we were supposed to go investigate it.  We upgraded our platform with some trepidation and bought a much bigger machine, adding storage - then we started pushing all of our security systems' logs to the device.  We noticed weird and abnormal stuff, but none of it was really indicative of anything because we didn't have time to investigate since we were all too busy trying to keep the company from drowning itself in bad IT decisions.  When someone suggested we start logging application data per some requirement somewhere in a compliance report - we were all a bit nervous.  What would advanced logging capabilities coupled with analytics find that we had missed all this time?  The answer quickly smacked us in the face.  Once we started getting visibility across many different business units, and many different non-security systems we realized we had so many compromised machines, so many trojaned or hacked systems that we were going to be cleaning up malware and hacks until we were retired... that was a terrifying realization."

 

Look, logging is simple.  You tell the system or application to send log data to device X on port Y, and done.  The challenge is when you look deeper and realize you can code applications to tell you all kinds of cool things like every login event whether it was successful or failed, or every attempt to input the < or > character into any of the user-allowed input fields since there was client-side JavaScript that prohibited this... a clear sign you're under attack, or every CPU spike, every SQL query and time to execute it plus the IP address and tables that were pulled from... loads of data that can then be analyzed.

 

Wait ...you're worried about the analysis aren't you.  You're worried that once someone, or some system points out where the issues are and where there are wonky things going on that you're going to have to do something to figure out what the issue really is, and then go on to remediate the situation.  You'll then be challenged to continue tweaking, refining queries and search parameters, and gather more and more data to get better and better analytical capabilities.  You're probably worried that this is going to consume all of your time, or that you'll find out that your organization has been pwn3d for years and all your data has already been exfiltrated in real-time, and that as you read this your latest database is being copied off to China ...aren't you?

 

Are you anxious yet?  Tomorrow we'll talk about the next stage so that you can get over this anxiety...

Comments
reswob10(anon) | ‎05-02-2012 05:20 PM

Actually, I think the CONCEPT of (centralized) logging is simple.  The actual IMPLEMENTATION is where the anxiety first kicks in.  For example, I know infosec folks who are anxious as to whether or not they can obtain enough capacity and capability to collect and analyze all those logs.  There are many organizations that balk at or outright reject requests to buy and/or build a centralized logging system of size enough capture and process all the logs that you would like.  They will (grudgingly) give enough to meet compliance metrics, but not one megabit more. Therefore, these infosec folks have the ability and know-how to do analytics, but the org has not given them capabilty.  
The first battle then, is winning the political and business battle to convince the front office to fund for that additional capacity in a way that doesn't look like you are failing at your job.  As has been said many times before; this is done by tying the increased security posture (bigger, better log server) to increased business productivity or profit or cost savings or mission accomplishment.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
About the Author


Follow Us
Community Announcements
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation