Sunday, October 11, 2015

Technology Misstep Doesn't Mean Failure

     As I start to write this post I'm reminded of an experience with a previous employer.  I walked into the data-center a few weeks after starting work and noticed a huge stack of Cisco boxes hidden in a corner.  I assumed these were new C-series UCS servers, however, discovered that the boxes belonged to a number of NAC appliances.  I was responsible for networking and security and didn't recall anyone telling me about NAC when I was hired.  I logged into a couple switches and didn't see anything indicating NAC was in the network.  Asking around I finally found someone who knew something but would only smile and say, "follow me".
     We walked down the hall and found, pushed to the back of the top shelf of a rack in a storage closet, were six NAC appliances in all their glory.  They obviously weren't being used.  A little more digging and I came to understand that the project had been mothballed.  Why?  Why spend tens of thousands of dollars on a solution just to stuff it into a storage closet?  The answer is that it didn't fit within the strategic vision of the company and the initial implementation plan didn't fit within the current security policies.  Could it be resurrected?  No.  Could we get our money back?  No.  Why was it still here?  Depreciation.
     Sometimes we make a misstep in selecting a piece of technology.  Sometimes we make a misstep in how we initiate technology project (and consequently they fail).  And sometimes the vendor makes a misstep when releasing technology.  A university I work with recently bought Cisco's new ASA CX Next-Gen firewall module.  If you followed the link you saw that it is already End of Sale.  It seems like it was on the market for, oh, about 3 weeks when they bought Sourcefire and put CX in their taillights.  It happens.  As IT professionals we are attempting to find a technology based solution to meet business needs with the ultimate goal of saving or earning more money for our business.  Sometimes they don't mesh as well as we'd hoped.  But what do you do to turn a failure into merely a misstep?

  1. Can the project, or the technology, be modified for little cost to meet the original goals?
    • Possibly you bought, or were recommended, the wrong model switch or access points.  Possibly you misunderstood a particular feature.  Either way, you may be able to simply add a license, change out a piece of hardware, or enhance it in some way to meet most of your original goal.
  2. Can the technology purchased be used for some other purpose that still benefits the business or organization?
    • Buying the wrong solution isn't always a complete loss.  Few of us work in environments where everything is as new and shiny as we'd like it.  If you bought the wrong model server with not enough RAM or too few processors there is no doubt there is another place in your environment where it would be welcomed.  Planning a large switch roll-out for your primary site and then finding out it can have 0 downtime means those switches get moved out to remote sites and they get upgraded a year or two sooner than planned.
  3. Can the technology be used in a way other than you initially intended?
    • You were planning on using your new Nexus 7000 switches to upgrade your core and replace your EOL FCoE switch but discovered during implementation that the storage team actually meant pure FC (Fiber Channel) and not the FCoE (Fiber Channel over Ethernet) they previously stated.  What now?  Still replace the core and add a small Nexus 5000 switch for the FC access and uplink to the Nexus 7000  with vPCs.  You'll add more redundancy and higher throughput.
      Rarely is a technology misstep a complete loss.  The university's primary goal was to block gaming traffic and filter advertisements.  They're successfully doing that, and plan on moving to Cisco's FirePOWER on ASA in the next budget year or two to take advantage of additional features.  Even those NAC appliances didn't spend the rest of their life in that storage closet.  We pulled them out, wiped their hard drives and re-purposed them as logging servers for our network gear.  Yeah, they were expensive logging servers, but they weren't a complete loss.  Get creative and be flexible and your technology misstep won't become the project failure you're afraid it might.

Monday, June 8, 2015

Sheet Rock Dust

I spent a couple days last week working with a customer who abruptly lost their CIO and is looking for direction.  When that happens one of the things I like to look at, which gives me a good idea of what kind of IT department he ran and what I can expect going forward,
is the data center. It doesn't happen often, but I was taken aback by what I encountered.  I don't dare show all the pics I took, but the two in this post are of hardware covered in sheet rock dust.

There was construction in the building over the past several months and nobody bothered to seal off the drop ceiling or door to the data center.  As you might imagine all that dust was sucked in thru the servers, thru the network gear, and thru the AC unit.  The AC unit got so bad that at one point it failed and shut down due to back pressure from clogged filters.  The temperature spike may have caused a number of servers to fail as well.  When I saw it several weeks had passed since the AC unit had been restored and yet no effort had been made to clean the data center.

To prevent the coating of dust from overheating the systems, every piece of equipment must be powered down, un-racked, opened, vacuumed with a very soft bristle brush, and then put back.  The AC filters have already been replaced and now every surface has to be cleaned.  It will probably take a couple weeks work for two of their people to complete.

More sheet rock dust
A few other things I noticed aside from the general clutter all over the floor (to the point that you cannot walk in there).  First were open jugs of water on the floor; nobody knows why those are there.  The grounding strap for each rack isn't connected to anything.  And, no surprise, hundreds of cables are slung all over the place despite having some very nice cable management and ladder racking.  In one spot a couple fiber runs were resting on the hinges of the server rack door.  If the door were closed it would break the fiber, but don't worry, those were only their primary and secondary WAN connections <sigh>.

Its funny how much the condition of a data center can tell you about the rest of an IT department, how its run, and what kind of detail has been paid to the little things.  These details are what make an IT department run smoothly.  Neglect the little things and everything falls apart.

Just to end on a good note, this last pic is of a data center belonging to a smaller customer but which is very well maintained.  Just like their entire IT organization everything is in its place and working as it should be.

Thursday, May 14, 2015

Wreck A Dirt Bike and Secure A Network

By all accounts it was quite literally the perfect day to get out into the forest and ride.  77 degrees for the high, 4 mph winds, and sunny couldn't be more ideal.  And even better was that I had less than a hundred miles on my brand new Honda CRF250L.  I couldn't wait to get on her and run her down the trails, thru the creeks, and up the hills.
This wasn't my first dirt bike, I'd been riding off road for 17 years and on road even longer.  This was just my first NEW dirt bike and every time I get on her it excites me.  On this trip my 11 year old son was coming along.  He has a passion for dirt bikes and any reason to get on one is a good reason, even if its just to ride around the back yard.

We pulled into the parking area, unloaded the bikes, paid our fee, strapped on our helmets and off we went.  My son hadn't ridden at this park yet so we started off with some easier trails to get him used to the terrain.  An hour and a half into our ride that Saturday morning we decided to take on the longest trail at the park, 6 miles. 

I learned to ride off road in the mountains of Utah.  Six miles up there was not a big deal, you might cover 20-30 miles in a day of riding those old jeep trails or following trails trails over the peaks and into serene hidden valleys.  Riding in the southeast is different.  It is all narrow groomed trails, dense forest, small hills and a lot of water and mud.  It requires a slightly different and more demanding riding style. Six miles thru a national forest in the southeast was going to require some work but we were having a great time and didn't even think twice.

I'm not an aggressive rider.  I'm a cautious and casual trail rider.  I get passed all the time by more aggressive riders.  I'm not into jumps or tricks or high speed maneuvers.  I enjoy being in nature, the pleasure of riding the dirt bike, and the occasional thrill of going over hills or maneuvering thru a tough piece of terrain.  I like to just explore whats out there and that day started out just that way.

Somewhere along the way I think I must have gotten bored creeping along behind my 11yo son thru miles of trails in first gear.  I like riding behind him so I can be sure he's safe and not doing anything dangerous or reckless (he'll have his teenage years for that).  But after two hours I needed to stretch a little, I needed to open the throttle, I needed just a little thrill.  So on a wide bend in the trail I jumped out ahead of him.  Two small hills were just up the trail so I shifted into 2nd gear and went for it. 

The first hill was perfect.  I went over it smoothly, even caught a couple inches of air, and was coming down just right.  I felt the thrill of that little jump fill me and I was looking forward to the next one before I even hit the ground.  When I did hit the ground something went wrong.  The bike lurched forward, the throttle open wide, and shot me toward the second small hill far faster than I should have been going.

It is natural when riding a dirt bike to grip your handlebars tight when you get scared or caught off guard by the bike's behavior.  The problem is that usually you rotate your wrists down and that revs the engine.  I teach my boys to watch for this, be aware of it and how to react to it when it happens.  It is exactly what happened to me.  When I hit the ground after the first hill I was holding the handle bars too tight and the down force caused the engine to rev and launched me over the second hill.  If the second hill hadn't been there I'd have been fine but that's not what happened.

I remember the feeling of being out of control.  I remember sensing that I was going over the handle bars.  I remember wondering if the bike was going to hit me in the back or land on me.  And then 15 yards from the top of that little hill I hit the ground; head first, then left shoulder, then left ribs, and finally left hip.  And I hit it hard.  Wearing the helmet certainly saved me from serious head injury but nothing else I was wearing was going to save me from other injuries.

I broke my collarbone, likely cracked a couple rips, bruised the hip socket, and tore innumerable muscles in my shoulder and down my left side.  I was lucky, it could have been worse.  I walked out of the forest and made my way to the hospital without the help of an ambulance.  I was in a lot of pain, but worse, I was ashamed.

I was ashamed that I had allowed myself to lose control.  I was ashamed that my son had to see it happen.  I was ashamed that I'd miss two family events later in the day and possibly disrupt a vacation.  Mostly, I was ashamed that I hadn't been vigilant.

I had done this so many times.  I had so much experience.  I knew what to look for, I knew what the risks were, I knew how to avoid them.  I knew I wasn't like other riders who risked life and limb every time they got on a bike.  But I also knew it was my fault.  I had lost focus, got distracted or frustrated, took my eye off the ball.  I had stopped being vigilant about my own safety.

It only takes a moment.  I learned the lesson while doing something I love with my son, but it applies to other areas of our lives.  It applies, as security folks, to our careers.  If we lose focus, or get distracted, or drop the ball it may be our companies or our customers who pay the price in stead of us.  We must be vigilant at all times.  One firewall rule change while testing, one unenforced policy, one new vulnerability notification that we didn't bother to read, one malware patch we didn't feel like applying at midnight on Sunday could be all it takes to open the door to attackers.
Damage to the rear

We must be vigilant.  We must also be prepared for what happens when were not.  In seventeen years of riding dirt bikes I never thought about what steps I would take WHEN I wrecked.  Do you think about what steps you will take WHEN you finally get hacked?  Have you written down your plan?  You will, to one degree or another at some point have a breach.  You probably already have and just don't know it yet.  It took me fifteen minutes after the wreck to realize my shoulder was where the real injury was.  Make sure you have a plan when you discover a real injury in your environment.  And be vigilant.

Monday, April 27, 2015

Video Calls For Everyone!?

I spent a good portion of this morning working on an enterprise wide video design for a regional hospital system in the southeast.  The design for the project has been evolving for a number of weeks as we receive new/updated information from the customer.  The scope has been to allow select internal users to have video calls between each other, allow video consultations between doctors at different (non-affiliated) hospitals, webex integration, and scheduling and ad-hoc video conferences with multiple conference room systems thru-out the enterprise.  

Its a healthy project, and it has been fun to work on, but I received an email late last week that made me laugh.  The account manager working on this account forwarded the following question/request from the CEO of the hospital system, "when will they get video on every desktop".  

The idea of every employee using video in place of voice reminded me of an old "The Jetsons" cartoon I saw when I was a kid, set in a world where video calls have replaced voice calls.

Episode 4: "The Space Car"

Video is all the rage, and I understand future looking CEOs wanting to ride that wave, but even working in the industry I wonder whether video is really a necessity for "every employee", especially in a hospital.  What do you think?  Is video on every device for every call really where we're going?

Monday, March 16, 2015

Security101 : What is Malware?

Security101 is a series of short, direct, no fluff articles about security basics.  To see the entire series, click here.

What is Malware? The word Malware is a combination of the words Malicious and Software and in short describes any piece of software that does bad stuff your computer, or uses your computer to do bad stuff.  Over the years the security world has developed/identified numerous categories of malware such as viruses, trojans, rootkits, worms, adware, botnets, bugs, ransomware, spyware, and spam to name a few.  Rather than run down that list each time we have a general conversation about malicious software, the term Malware is used in stead.  In July of 1990 Yisrael Radai, a computer scientist, was the first person to use the word, however Chris Klaus is credited with making the phrase popular by regularly using it in presentations and keynote speeches.  No matter who started using it, the word malware is part of any IT professional's lexicon these days.

Monday, March 9, 2015

Security101 : What is Data Loss Prevention?

Security101 is a series of short, direct, no fluff articles about security basics.  To see the entire series, click here.

What is Data Loss Prevention (DLP)? For most companies, even ones that manufacture products, their most valuable assets are digital and DLP software works to protect those assets.

Much like locks are installed on doors and cameras are mounted thru out a warehouse, digital assets must be protected.  There are a number of reasons companies begin working to protect their data.  The top 3 are typically:
  1. Forced by an industry security policy
  2. Desire to maintain competitive advantage
  3. The need to maintain your customer's confidence 

Data can leave your environment in numerous ways.  DLP software focuses on two areas, first monitoring your data-at-rest and then watching it as it goes into motion.  What that means is you point the software at specific storage areas (SANs, server hard drives, NASs) to be monitored and the software will then report on who accesses that data, how much data they access, and where that data gets copied to.  Some DLP software can also block or throttle that traffic to prevent or limit data theft.  Other types of DLP software monitor outbound email for keywords, file types, file sizes, and file contents.

For a longer discussion of DLP check out this post

Monday, March 2, 2015

Security101 : What is a Next Generation Firewall?

Security101 is a series of short, direct, no fluff articles about security basics.  To see the entire series, click here.

What is a Next Generation Firewall (NGFW)? A Next Generation Firewall is a traditional firewall with some additional functionality added on.  Most NGFWs include, at a minimum, a web filter, basic IDS/IPS functionality, Layer 7 or application visibility, and possibly a botnet filter.

What this means is that in addition to allowing or denying traffic based on source, destination, port, and protocol you can now inspect and control traffic based on a number of additional criteria.  For the privilege, you'll usually be charged more than a traditional firewall and have to pay an annual subscription fee to keep the signatures, lists, and definitions current.

Each of these additional function can be a robust standalone product, but since they all primarily watch traffic at the network's ingress and egress points it was only natural to include them with the firewall since that is where it lives as well.  If you want my Security101 explanation of what each of these are, just click the link below.

Security101: What is a Web Filter?
Security101: What is an IPS or IDS?
Security101: What is Application Visibility?
Security101: What is a Botnet Filter?