NSS Labs CTO discusses advanced endpoint protection testing, challenges

NSS Labs this month unveiled its first major endpoint protection group test since filing an antitrust lawsuit last September, and the tests mark significant changes in methodology and displaying the results.

The security technology testing firm, based in Austin, Texas, released the results of its 2019 Advanced Endpoint Protection Group Test, which featured a whopping 19 different vendor products, during RSA Conference earlier this month. Fourteen of the products were rated "recommended" based on NSS Labs' security effectiveness and metrics for total cost of ownership per agent, with Sophos Intercept X Advanced v2.0.10 achieving the highest overall marks.

The Advanced Endpoint Protection Group Test follows NSS Labs' antitrust suit against the Anti-Malware Testing Standards Organization, as well as CrowdStrike, Symantec and ESET for allegedly conspiring to prevent NSS Labs from testing the companies' products. The three vendors and AMTSO deny the allegations. Despite the lawsuit, Symantec participated in the group test, and its Endpoint Protection and Advanced Threat Protection v14.2.1023.0100 received a "recommended" rating.

NSS Labs CTO Jason Brvenik said his company had to take a different approach to testing this time around because of how advanced endpoint protection products have evolved. In addition, NSS Labs made a small, but significant, change with its final report by redacting the names of three vendors that received a "caution" rating.

Brvenik spoke with us at RSA Conference about how the endpoint security market has changed, the challenges posed to testing firms and why NSS Labs removed the names of lower-scoring products. Here is part one of the conversation with Brvenik.

Editor's note: This interview was edited for clarity and length.

How is this advanced endpoint protection test different than what NSS Labs has done in the past?

Jason Brvenik: The endpoint space specifically has gotten really good at sharing information with each other -- at least sharing intelligence and knowledge. And so, in this round, we found that we actually had to do testing in parallel in under a minute and a half before results got tainted.

That fast?

Brvenik: Yes. We had to actually get that fast before other vendors were able to piggyback somebody else's work.

Are vendors going through platforms like VirusTotal and others?

Brvenik: Yes, it's through things like VirusTotal and side channels and all the other services. And you can't tell the vendors to turn that functionality off, because that's a benefit to the market in general. Everybody benefits the faster you get to those kinds of outcomes.

What else was different about this series of tests?

Brvenik: The speed and parallel testing is one element. Another element is the depth we've gone to in a number of other cases looking at exploitation and lateral movement and evasion techniques.

The fascinating thing for me is there's a continual improvement in endpoint. The bar has clearly been rising, and the market is responding and working to do better. There's no real clear line to me on whether or not one approach over the other has a significant advantage.

It seems like the products that are best suited [for enterprises] take each of the approaches and use them accordingly. Every approach has strengths and weaknesses, and the products that are best integrated across them seem to be the ones who are doing really well.

What are some of the things the leaders in this test did well?

Brvenik: Overall, it was better, faster outcomes. Good blocking, good detection, great timeliness, good analytics -- all of those things come together in producing an output.

Were there some endpoint security products where you saw the opposite, and they just weren't getting to those outputs fast enough?

Brvenik: Yes. There are certainly some legacy vendors and products that have a good opportunity to engage the market differently. But there have been improvements, too.

The big improvement we saw this year is [the] handling of ransomware and the different ransomware detection methods and capabilities in those products. Naturally, you would expect -- with all the pain that's been caused in years past -- the market to produce a solution that's much more effective there.

Because there's so much data being shared so quickly among the vendors, does that produce similar results when you test for specific threats?

Brvenik: No, we get very different results. And that's why the time window matters so much. It actually lets you assess the technology's own merits, not the industry's collective capability. We do see good variation in there, especially when we start looking at how much information these products are gleaning and capable of delivering in those briefs windows.

We assessed a number of elements when we look at the forensic capability, for example, when we look at the ability of a product to prevent an attack and when we look at the ability of a product to identify different techniques. Some are stronger than others in reputation versus machine learning, for example. It's interesting -- there's no one technology that is resistant itself to direct attack. All of them have weaknesses you can find, but their general capability and ability to respond is continuously improving.

How much does signature-based detection matter now that so many vendors have moved toward advanced analytics and machine learning? Is that something that still distinguishes the vendors in the endpoint protection space?

Brvenik: It's still very much relevant. And it does distinguish in some respects. In one of our test cases, it's all about offline functionality where these systems do not have internet connectivity. And [some] use cases there would be you're on an airplane, you're on vacation or you're at a coffee shop where an attacker controls things, and you don't necessarily get the benefit of cloud-based analytics and that other functionality.

There are some interesting data sets you see here. Signatures are still very relevant in much the same way reputation is still very relevant. Rapid response effectively gets very specific things, but it's not durable against mutation and attack. And so, I don't [think] signatures are going away. Certainly, there's value to them in the rapid response; you don't have to do the heavy analytics. But it really is ultimately affording you the opportunity to have the analytics on the remainder, where signatures kind of are taking care of that.

Looking at advanced endpoint protection today, do you feel like the products in this space are getting better at identifying new threats?

Brvenik: I feel like they're getting better at rapidly concluding new things are threatening. I don't know that I would say they're getting better at stopping threats outright, but the ability of these products to conclude in one place and share [the data] in all other places at the same time is a pretty powerful thing.

The endpoint security market seems like it's gotten very crowded and competitive, and some vendors have said there's too much noise and too much infighting in the space right now.

Brvenik: That's actually one of the reasons I looked at the A, B and C vendors and the unvalidated vendors and [removed their names]. I wanted to level-up that conversation, because the market does have this issue where they'll take shots at each other behind the scenes, and I just don't want to [contribute to that]. I want talk about the people that are doing well, that care about their customers, care about their products, and really want to understand what the opportunities are and then go after them. I don't want talk about all this other stuff. And I don't think a vendor that wants to do well but isn't quite there yet should necessarily be penalized. There shouldn't be a ton of sharks waiting to jump on them.

We're trying a different approach of focusing more on the vendors that are doing well and having the right conversations about the opportunities there. If vendors are not in the recommended [quadrant], then they have some room to grow. But if you're looking to buy more products, you should be looking up in that top quadrant right now.

Is that why you didn't say who those vendors were that fell outside the recommended quadrant?

Brvenik: Yes. I prefer a 'promote, not demote' perspective, rather than having somebody participate in a test because they want to do well, but they're not quite there. I don't want to discourage people from getting that experience and getting that exposure through testing and being able to go address [shortcomings] and ultimately help their customers in the market.

At the end of the day, what I care about is getting the best security in the hands of the enterprise. The data from our testing is available; you can see everything there. We're not just throwing that name up on the board so that it makes it possible for everybody to start taking shots at one another.