Thursday, November 13, 2008

How Outpost Firewall "Lost" a Firewall Challenge, or a Few Words About PR Techniques

Many of you are likely frequent readers of Matousec Transparent Security Lab’s white papers and tests, including the Firewall Challenge. The Firewall Challenge is intended to show firewalls’ and security suites’ capabilities in resisting “leaktests” (tests that simulate data leakage) and other lesser-known ways of bypassing protection.

As can be seen from the list of solutions tested, both standalone firewalls and security suites are submitted for this report. One of those security suites is Kaspersky Internet Security 2009, which holds 7th place in the ratings with a score of 87% across 73 tests, a “very good” protection level, and a complexity level of “10+”.

Agnitum’s products have traditionally enjoyed well-earned leadership in these tests for several years. Currently Outpost Firewall Pro 2009 holds 1st place with 99% (the build tested was an initial one, released July 1, 2008), “excellent” protection and complexity level “10” (minor errors revealed by the test were fixed in subsequent builds, which enables us to claim a 100% result on all suggested levels if the product were retested in its current condition).

So here is our question. Why did a product with a lower total score receive a higher complexity level– 10+? The answer is actually quite simple. To reach a higher level, a product must get no less than 50% at the current level. The products that managed to pass two rather esoteric tests at the last – 10th level – received a 10+ mark according to this criterion. However, this showing doesn’t have a crucial impact on the final results, as a product can marginally pass all previous levels with just 50% and still show a higher result on the last level.

As for Outpost solutions, it’s no secret that Agnitum offers an integrated solution as well: Outpost Security Suite Pro, based on the standalone Outpost Firewall Pro and incorporating the same arsenal of proactive measures. As a suite, Outpost Security Suite Pro also includes antivirus and antispam capabilities, but these tools don’t play any significant role in the Matousec tests, nor do they interfere with the test procedures (even in the performance segment).

Now, let’s take a look at this article (one of many published world-wide on the Internet). Now take another look at a new so-called “test leader” - Kaspersky Internet Security 2009 (the 7th in the overall rating, as noted above)! Surprised? It’s worth noting that Kaspersky Lab doesn’t take Outpost’s results into account because the most recent build of Agnitum’s security suite “hasn’t been tested by Matousec” and, therefore does not even merit a mention as far as Kapersky is concerned.

However, both Agnitum’s products are so close in their proactive protection and anti-leak capabilities that it seems to us rather strange to separate their achievements and give the winner’s crown to another vendor’s product – a security suite! – in the Firewall Challenge.
We’re confident that those who are interested in this topic are smart enough to read between the lines and make their own independent analysis of the test results. Here’s the link once again:

While we fully support Kaspersky’s efforts to promote the Matousec tests as a reliable source of information about the robustness of security products, we believe that the tests and their results should be weighted to reflect accurately the likelihood of users encountering each situation in the real world.

Below is the data derived from the Matousec tests (courtesy of Kaspersky Lab which created the initial chart, to which we have added the Outpost results).

By the way, Outpost demonstrated a 100% result on all levels but two while Kaspersky Internet Security scored 100% just on 4 levels out of 10. We leave you to draw your own conclusions …

Vitaly Yanko
Director of Marketing and Sales, Agnitum


Anonymous said...

Sorry, would you mind telling me what does "PR Techniques mean" here?

Agnitum BLOG said...

When a security vendor offers an interpretation of test results that obscures the real winner's position and highlights its own doubtful achievements, that is, well, PR "techniques".

Pavel Goryakin

Anonymous said...

Парни, а на русском блога нет у вас? Хороший продукт, но с настройками лучше по-русски воевать...

Vladimir Jovanović said...

I don't need to be persuaded that Outpost is the best firewall around. I have tried almost all and this one really stands out.

Don't be discouraged by this foul play, just keep up the great work ;)

All best,
Vladimir Jovanović
web designer

Agnitum BLOG said...

Thank you, Vladimir! We'll do our best.

Pavel Goryakin

Adel said...

I think instead of complaining about the extra "+" that KL got, you should rather work on making your suite more user friendly.

I have use all suites, KIS 2009 seems to be something that would satisfy most uses, if not all.

I see OSS or your firewall product as something for advance users. Being an average user, or lets say, a below average user, i found your firewall product to be very confusing. I do agree you might have the best firewall, but the benefits of that firewall can only be fully enjoyed to the maximum by advanced users.

The firewall product, im sure, is for end users and im sure you dont make the firewall just to satisfy leak tests.

But since you already have the best firewall technology, its better for you guys to work on developing your product further instead of saying why KL got that extra "+" ;)

Dont take me in the wrong way, Agnitum is one company which i always look forward to. And for me a good product it that which is a good recommendation to others, sadly right now.. OSS cant be recommended to many of my friends.

Jorge Z said...

Outpost is the best! There is no doubt about that! keep up the good work!

Anonymous said...

I have used several different Firewalls over the last 10 years or so. The one I used and loved was discontinued and I tried several demos before I purchased OutPost Pro. I like to test everything before I use them, And found Outpost Pro to be very user friendly.I have had my systems tested by many friends who are techs and they haven't gotten in to my system as of yet, so with that said, I feel Outpost Pro will be on my system for many years too come!

Anonymous said...

I am currently on Outpost Security Suite Pro 2009 ver. 6.5.2358.316.0607. Did not get any BSOD yet with this version :) - seems you finally got it fixed. Very user friendly, very adjustable and customizable. Suggesting OSS to my friends.
Спасибо за отличный продукт.

Santa said...

Happy Holidays, Agnitum!!

For those of us who don't easily understand, what does the 72% Performance rating mean? And why does AGN show the lowest percentage among seven companies listed?
Me, the Mrs. (and the elves) are VERY happy with your product/suite. It's good having increased confidence that "critters" are not wandering throughout this "North Pole" system.

Anonymous said...

YOur firewall is the best, Tested numerous ones and your solution is easy to use and I would give it 100% for everything, Great job guys!

Anonymous said...

Outpost Firewall IS the best firewall, end of story. I have used many others but none have the transparent, live connections GUI that OP has. Another crucial thing that other vendors do not have is IMPROVE NET. IMPROVE NET is definitely a plus above all other firewalls. The only reason I do not use the OSS is it's antivirus. Don't get me wrong but there are other top vendors of av-technology.

Anonymous said...

Yeh! In the new Matousec Firewall Challenge Outpost lost without ""!

Anonymous said...

NB! Agnitum has Russian blog - please check

REZ said...

I actually use outpost firewall pro and kaspersky AV, the combination of the two is unbeatable when the setting for both are configured correctly.