Red alerts: Inside Cisco's incident response best practices

The incident response process is so fraught with peril that it's rarely done at a satisfactory level, but there can be valuable lessons in failures, according to Cisco's director of incident response, Sean Mason.

At the Talos Threat Research Summit on Sunday during Cisco Live 2019 in San Diego, incident response challenges and best practices were heavily discussed topics. "No two organizations are the same, which makes [incident response] complicated," Mason said. "People, processes, risk profiles -- whatever it might be -- they're all different. If they were similar, there might be a playbook to follow."

Nevertheless, Mason said there are some general dos and don'ts that enterprises need to keep in mind for incident response planning. In our interview with Mason, he spoke about his recent experiences with client engagements, what some mature organizations do successfully and what many organizations often get wrong.

This interview has been edited for clarity and length.

What makes incident response so hard, and what are the challenges that you see companies struggling with today?

Sean Mason: If you look at a typical incident, there are a lot of moving pieces that are required to actually do IR effectively. If you think about an organization that hasn't had the dollars and the time to invest and execute on an entire security strategy, let alone the bits and pieces that go into IR, then it's going to have a much more difficult time responding -- not only internally with whomever you may or may not have on staff, but [with] outside consultants like our organization. We may find it difficult to do incident response in those situations too. Challenges that come to mind are a lack of proper tools, a lack of properly configured or installed tools, a lack of tuning and a lack of lack of dated logs. Those things all go together, so at the end of day, we can tell the story of what happened with an incident.

And that's without getting into the soft skills side of it. If you look at the maturity model for IR, organizations tend to start with hiring people, installing some tools they can afford and maybe do some logging. But the proverbial cans that are usually kicked down the road include things like figuring who your legal counsel is and establishing a relationship, figuring out who your chief risk officer is, and knowing who [you] call, like a Cisco, in a time of crisis. These types of softer skills tend to get put off, and more mature organizations these days tend to have already tackled these things or are at least still in the process of tackling them.

You mentioned logs. Are most organizations you've seen in incident response engagements struggling with keeping logs?

Mason: When it comes to keeping logs, the worst I've seen was just three days, which was in a somewhat recent IR engagement we had. If you think [about] the aspect of uncovering something that's going on, let alone getting people in to help, three days isn't a lot of time.

Sean Mason

It's not uncommon for us to see three days, seven days or 30 days. It's the rare organization these days that's mature and has the ability to the keep those logs around [longer] and spend that money, because it's not cheap to keep that data. They know they need to keep the logs for a historical record in the event that something happens.

And if you look at any of the breach reports that have come out, there is usually a very long dwell time. The time between the attacker's initial actions and when we notice is still a very high number -- we're talking around a hundred days at this point. That dwell time number in theory gives you an idea of how long you should be keeping your logs. And I think most places aren't even close to that. There are a lot of factors as to why that is, but one of them is definitely cost. I think that as solutions, especially in the cloud, get better, the price will continue to drop, and that will allow folks to keep their logs longer. That remains to be seen.

What new challenges are Cisco customers facing when it comes to IR?

Mason: The one thing we've been seeing more and more lately, and in many ways throws a wrench into incident response, is around cyberinsurance. As we go into emergency engagements with different clients, one of the questions we always ask is, 'Who is your cyberinsurance carrier, and what is your policy?' And the number of times we hear crickets in response or 'I don't know' is very telling.

These are important questions. Did your organization even purchase cyberinsurance? Do you know what your deductible is? Do you know what it says you can and cannot do? I think there's an educational process that needs to happen with security teams around cyberinsurance. This goes back to my point about the soft skills and knowing who your chief risk officer is, because they're most likely the ones buying the cyberinsurance policy.

We've had recent cases where we've had some organizations contact us because they need help, we ask the cyberinsurance question, and they say 'We're not sure; we'll get back to you.' That's actually slowing down the response time, and that's the last thing you want to do.

What are some of the positive things you've seen with IR engagements? What are some things successful companies are doing?

Once you do the IR readiness assessment and you start working on those items, you can couple it with testing, like tabletop exercises.

Mason: We're not always fighting fires every single day, although it seems like it. We're actually doing a lot of proactive work for our clients. Some of the more mature clients that have their ducks in a row are the ones that have gone through what's called an incident response readiness assessment. Essentially, you're not looking at the whole security picture of the organization. You're just focusing on IR. If you were hit tomorrow, would you be able to respond? What are the missing pieces, what are you good at, what are you bad at and where are the opportunities to get better? It's good to have an external party come in and objectively rate you. Let the experts come in and take a very objective look at the organization and make recommendations.

We're seeing a lot of cases where organizations say 'Hey, thanks for the recommendations, but we don't have time to follow through on them, so we'll file them away for later.' We should be taking those recommendations and following through on them as best we can as opposed to just ignoring things.

What other proactive steps can companies take?

Mason: Once you do the IR readiness assessment and you start working on those items, you can couple it with testing, like tabletop exercises. There are a variety of these exercises out there for IR, and one of the ones we've gotten traction with lately is based on different scenarios and has modifiers. We actually had special dice made up for it, and that makes it a little more fun that just blandly walking through [the exercise] and going from slide to slide.

But we've seen some customers also take those tabletop exercises and do it not only at the analyst level where you mock up attack data -- 'How are your SOC [security operations center] and IR folks finding things?' -- but also take it from the CISO up to the C-suite and the CEO, CIO and legal counsel. Doing that kind of exercise at least once a year is invaluable.