NSS Labs CTO Jason Brvenik talks security testing challenges

Despite recent challenges with security testing in the endpoint protection market, NSS Labs has no plans to change course.

NSS Labs filed an antitrust lawsuit last September against the Anti-Malware Testing Standards Organization (AMTSO), as well as CrowdStrike, Symantec and ESET for allegedly conspiring to prevent NSS Labs from testing the companies' products. The three vendors and AMTSO deny the allegations.

While the lawsuit is still pending, the security product testing firm, based in Austin, Texas, has pressed ahead with new group tests. For example, NSS Labs last month released the results of its 2019 Advanced Endpoint Protection Group Test, which featured 19 different vendor products.

NSS Labs CTO Jason Brvenik spoke with us at RSA Conference about the security testing challenges his company faces, particularly in the endpoint protection market. In part one of the discussion, he discussed the results of the advanced endpoint protection tests and how NSS Labs' approach to testing changed. Here, Brvenik discusses the difficulties that come with simulating attacks and playing the role of an adversary when testing products.

Editor's note: This interview was edited for clarity and length.

In light of your antitrust suit, were you concerned about participation with this round of tests? Did you feel the vendors that were included wanted to have their products tested?

Jason Brvenik: There are, as you can imagine, various perspectives. The vast majority of vendors in our tests want to be tested and stand on their [products'] strengths; some don't. Each one of them has very different perspectives about how things should be with the tests. I'll give you a common refrain you'll hear from the marketplace and the vendor side when talking about testing: 'There's not enough of those [malware samples] to be statistically relevant.'

Here's my retort to that. An attacker isn't going to use a 'statistically relevant sample.' It's going to use one that works. And so we very much take the same approach. It's not about the quantity. It's not about putting so many things on the board that you can find something. It's about replicating attack scenarios that have the potential for success and measuring how well a product responds to them.

Security testing seems like it's much more challenging than most technology product testing.

Brvenik: Way harder. It's much harder.

Then why do it, especially after the difficulties you've had with some of the endpoint security vendors?

Brvenik: Because it's needed. There is so little transparency between what the user expects and what the product delivers, and the only way to know if something's being effective is to actually try it and the only people trying to beat defenses are the attackers right now. It's about transparency and accountability, allowing the enterprise to at least know the bounds of how much trust they should put in the capabilities being fielded, and how much opportunity they have to close that gap, and to protect their users, to protect their employees, and protect their shareholders. That's a key element -- it's necessary in the industry. It's nontrivial.

It's somewhat sobering that I have a very small team that I call the 'Offensive Research' team that does the net new security testing capabilities, and we've yet to meet a product that we couldn't get past. What does that tell you? Of course, no product is perfect. We can't solve all problems in the industry. We can certainly try to make it much more difficult for somebody to steal from you and take your data.

On the point of the transparency and explaining how security products in general work and how effective they are, do you feel like it's going the right direction or is it going in the wrong direction as we've gotten more into machine learning and AI and so-called 'black box' technology?

Brvenik: To be determined. At least the focus is being put on solving the problems at scale for the broadest flaws with an affordable operational model. If you just look at the dollars being spent versus the dollars being lost, we're not hitting the mark yet. The World Economic Forum estimated that global losses [from cybercrime] are around $500 billion a year. That's bad, obviously.

On that end, one of the bigger issues for security vendors and enterprises is that we're producing and receiving all of this data and it's not being incorporated or used in the right way.

Brvenik: And that's where machine learning has the most opportunity to help. And the big risk is everybody's talking about machine learning and AI as if it's going solve the security problem. I don't think it will. I think what it will do is allow us to efficiently reduce that unconstrained problem space so that [human researchers and analysts] can solve the harder problems more quickly. I don't think it's going to magically make the security thing better.

Five years ago it felt like no one really cared about the endpoint, and now the endpoint security market is booming. Where do you see the market going?

Brvenik: I don't think it's that nobody cared about the endpoint. I think the technology was ripe for disruption. Five or six years ago, AV [antivirus] was ripe for disruption, and it's been disrupted. There's some new players, new thinking, new capabilities. And let's be candid about AV and endpoint protection in the first place -- even bad quote-unquote protection technologies, relative to the market, are better than no protection technologies.

It's still a net value to an operational model because it's some segment of response you have to deal with. The question is: For the same investment, what's the better solution? It's not, 'Do I need one?'

Endpoint protection is key, especially in the modern age where we no longer have networks that are all contained with all of our users. We have very mobile workforces, very different device profiles accessing things, and very different data risk issues than what used to be. So let's just get some good endpoint technologies out there that can work in conjunction with the other pieces and make them all work together. That was the disruption that was ripe for happening.