Eset on the Radio

Subtitle

Does Your Anti-Threat Software Actually Protect - Bogus Testing Methods

Complete Transcript of Interview – Randy Abrams–ESET
Let’s Talk Computers Radio Talk Show
Host Alan Ashendorf
August 25 2007


Alan: You buy an Anti-Threat Software package. How do you really know that it stops threats? Some companies even suggest that you use something like the EICAR Test Virus, but does this really test for viruses? Our guest today is Randy Abrams, Director of Technical Education with ESET. And welcome back to Let’s Talk Computers, Randy.

Randy: Well, thank you very much, Alan. It’s great to be here.

Alan: Randy, first of all, what is the EICAR test virus and why doesn’t it really do what people think that it does?

Randy: The EICAR test file is not a virus, at all. It’s simply a harmless file; the anti-virus companies that detect it have agreed to detect it so that a consumer can determine whether or not their software is actually installed and functioning. It’s a lot like holding a cigarette lighter up to a smoke detector. You might be able to tell if heat will set it off, but you don’t know if it’s going to respond before the house is on fire and it’s burnt to the point where the flames are right next to the detector.

That’s what the EICAR test file is for. It doesn’t tell you how many threats the product can detect; it doesn’t tell you anything about performance or quality. It just says that it’s on or it’s off.

Alan: Using that analogy, if you hold a cigarette lighter up to a fire detector, you are causing the same symptom that a fire is going to cause. It’s going to cause heat and it’s going to cause a flame, so you are, indeed, testing out the fire detector. But, with the EICAR virus, as such, it is just a string of characters that says, “Look at me – detect me.”

Randy: Right. That’s where the lighter analogy falls down, because you’re not actually detecting behavior. The EICAR test file actually is an old DOS program. And, if you run it, all it does is display a message saying, “EICAR test file”, it’s not a virus; it doesn’t replicate; it’s not a Trojan. It doesn’t do anything bad. You’re not testing anything any behavioral component and you’re not testing to see if your anti-virus could actually detect a single “virus in the wild”. I can actually write a “batch file” that will detect the EICAR test virus. Does that mean it’s as good as another product?

Alan: Well, talking about testing – I’ve seen a number of anti-threat software companies that will go out and certify their program as being “tested”, but they won’t test against a large number of viruses. I mean, they will look at maybe twenty different test results. That’s not enough to even get in the ball game, is it?

Randy: A sample set of twenty or even 2,000 samples is statistically insignificant. There’s hundreds of thousands of malicious programs out there. When you have a sample set that small, not only do you not get a good big picture, but also you then run into problems with how was the set created – how was it collected – and has it been tested? You have to actually test your samples to make sure that they are legitimate samples.

If someone sends you a sample and it gets collected while it’s getting emailed, the sample you have may not actually function. If it cannot function, then it’s perfectly legitimate for products not to detect it.

Alan: Well, then you’ve got companies that go out and they will test what they call, “in the wild” viruses against their signature definitions. They say, “Well, okay, we caught them all”. That is a test, but that is not a true test, is it?

Randy: It actually a type of a “certification”–when they say “in the wild” we’re not talking about all of the malicious software that’s out there. For one, this is “only viruses.” It doesn’t include things like back door trojans, keystroke loggers, stuff like that.

And then, of the viruses, it’s only a small subset, because when they say in the wild, they are talking about samples that are on something called “the wild list”. And the wild list is relatively small. And yes, every product should detect every sample on the wild list. But, that’s a certification. The difference between a certification and “a comparative” is that a certification just says, “We meet a base line”. It doesn’t say that’s it’s better or worse, it just says–We meet a base line.

The example I use is that for a car to be sold in the United States, it has to be certified by the Department of Transportation as being, “safe”. Does this mean that a Ford Pinto is as safe as Mercedes S Class?

Alan: I don’t think so?

Randy: No, it’s not that the wild list testing is completely irrelevant, but that doesn’t give you any real good quality information. If a product can’t detect all files on the wild list, there’s a quality problem, there. If it can, all you know is it’s met a base line standard – and you need more information.

Alan: Nowadays, because of the types of threats that are being produced – and I have seen some websites that have a “drive-by install”, meaning they change the program every time a new person goes to the website, so that there is never the same virus or threat that is being produced, ever. It’s like snow flakes; there’s no two alike. How do you trap for that?

Randy: That is a very interesting problem for testers, because for one, you can’t just use signature-based detection to detect all these samples. For another, let’s say a product has added detection for 20,000 of these unique samples. There are more than 20,000 unique samples from just one given type of virus on a website, because they’re being morphed so frequently.

What is important, is that as each of these samples is generated, it is detected – because only one person ever sees that sample until they send it off to the anti-virus company – so if you did not have detection for that when it came out – adding detection for it later is meaningless. It doesn’t detect any one.

Yet, these samples get into “test sets.” So, a product that detects all these samples – if it didn’t detect them when they came out, it didn’t offer any better protection, although the results of the test will make it look like it’s doing really well.

Alan: But, is this like false advertising to see the bogus test results that give you a false sense of security that says, “I’m going to be protected from viruses.” In the old days, remember we had double locks on our front doors, but we had on the back doors skeleton keys so that could get at the five and dime store? Is this the same thing that we are looking at, that gives us a false sense of security?

Randy: It definitely can produce a false sense of security. One of the worst violations of this is when a magazine runs some sort of test and tell you, “Okay, this is the best anti-virus product”–you can’t tell that, based on one test. You need a track record. You need a history of test results to determine what’s best. And basing best on one test is like taking a look at the stock market one day and determining which stock did the best and then saying, “This is the best stock for you to have.”

Alan: Well, I subscribe to the Virus Bulletin magazines. You have a track record, where you have never missed an in the wild virus – ever. You’ve got the 100% Award from them, always.

Randy: ESET Nod32 always gets the 100% Award and that actually is a certification that is based on the wild list. When you subscribe to the magazine, you get additional information about a much larger collection of viruses.

But the important thing about never having missed an in the wild virus in Virus Bulletin testing, even though that’s the limited set of the wild list, what you’re seeing a track record of quality. As I said, every product should detect all the viruses on the wild list. And Nod32 is the only one that has consistently done that in every test.

Alan: Well, what about companies like AV-Comparatives? How do they test for viruses?

Randy: AV-Comparatives does a number of different kinds of tests. They have a very extensive collection – even with the hundreds of thousands samples that they use, there is a margin of error that is probably somewhere between 5-10%. That might sound like a lot, but that’s actually a pretty darn good test, relative to what’s available.

They also do what they call “retrospective testing” and this gets really interesting, because retrospective testing is designed to tell you how good a product’s ability to protect you from unknown threats is. Consumer Reports has done some reports that they claim, will tell you how good a product’s heuristics are, (their ability to proactively detect new viruses). But Consumer Reports gets it all wrong. They write their own viruses for this test, so you don’t have a real-world situation. You’ve got a sample set that is biased by whoever is generating these samples and the samples aren’t ever in the wild.

The retrospective testing gives you a track record over time and what it does is to collect new samples over a period of say, three months and then without updating the scanners for three months, it scans against these new threats. So, the signatures are taken out of play, they don’t do anything; it’s pure heuristics, “pure, proactive protection”.

Alan: So, where does Nod32 fall in that range? How do you compare?

Randy: In almost every retrospective test that AV-Comparatives has done, Nod32 has come out on top. There was a recent test where we came in third and the two products that detected more, also had pretty significant false-positive problems. One of the products had over a thousand false-positives, which means when it says there is a virus, you don’t know if it’s actually a virus or you’ve got a clean file that’s just being called a virus.

The other products still had 9 times as many false-positives. Yes, it’s important to detect new threats, but it’s useless if you just detect everything and say it’s a threat.

Alan: A false-positives can cause more damage to you since it’s trying to fix something that isn’t broken than it is to miss something that is malware or is a threat.

Randy: False-positives can be a huge problem; not only do they scare consumers, but for businesses, it can be very expensive.

Alan: You actually set up like a virtual machine that says, “Okay, virus or threat – try to attack us and then I’m going to see what you do. I’m just going to sit back and watch and if you act like a virus and act like you are going to be dangerous, then I’m going to put you away somewhere.”

Randy: Exactly. We call it our Advanced Heuristics or Active Heuristics and we have set up an emulator, if you will in memory and protect its storage. It’s like setting up fake airport and you are expecting a terrorist to come in. You set up this fake airport and the plane lands you watch what the terrorist does, but none of the passengers any other people are risk, because they are all virtual people. So, you let the terrorists do whatever they want and when they do something bad, now you can catch them, without risking the real population.

Alan: The only drawback to that approach that I can see because it actually emulates a complete computer system and watches what goes on with almost every file that comes down your system is speed. If the program is taking over your computer and slowing everything down, the first thing that someone is going to do is to disable it.

Randy: Exactly. That’s why it’s important to create these things with a lot of intelligence. Nod32 consistently comes in among the fastest anti-virus products on the market. And the way we are able to do that is in knowing when to emulate and how to emulate for.

So, if you see something, suspicious and you are able to see pretty quickly that there’s a legitimate reason for this suspicious behavior and that it makes sense in context, you don’t have to go through the rest of that person’s day, if you will. You can stop–you don’t have to watch them all day long.

So we combine smart “algorhythms” or programming models with a program that’s written primarily in assembly language, and that’s how we are able to get really good throughput, rally good high speed out of the Product.

Alan: Because these people that are writing new-generation threats in this malware – they are very sophisticated and it’s changing every moment of every day. And you also have to change your Software.

Randy: Nod32 by default will check for updates very hour. We don’t have to update every hour; because of our heuristics, we are able to catch a bunch of the new threats without an update. But, when we see new techniques being used and we develop new heuristics, the heuristics are automatically updated with their signatures.

Alan: And not only that, you Software, itself is updated every time you come out with a new version, and we have a license with ESET, we get the newest and the latest Product. We don’t have to worry about being 2 years old.

Randy: Whenever we come out with a new version, the licensed consumer is eligible for that new version. We don’t want our customers to have our second-best technology.

Alan: If someone would like to find more information about ESET Software, viruses, and also your trial software, (which is a full-featured, 30-day trial version)–It’s not stripped down in any manner – where would they go?

Randy: They can go to http://www.eset.com.

Alan: Randy, it’s our pleasure to have you again as our guest on Let’s Talk Computers, talking about how we need to test anti-threat and anti-malware software and we look forward to talking to you again, real soon.

Randy: Thank you, Alan.