Accuracy and Time Costs of Web Application Security Scanner Report

Wednesday, February 03, 2010

Cross Posted from Robert "RSnake" Hansen's blog at

Larry Suto is back with another report outlining the differences between some of the top web application scanners on the market. Before you get all uptight and start flaming me, I in NO WAY sponsored, encouraged or had anything to do with this test in any way. In fact, I only found out about it a few days ago. Not that I think that’ll stop the flame wars, but just direct your ire appropriately, please! Anyway, he took a different approach this time, and instead of running the scanners against something he had devised up to be used only in his own lab, he turned all the scanners on each other’s public test sites. You might think they should all fair fairly well since they’re all public and there’s nothing stopping them from testing to their heart’s content. But that’s anything but what he found. You can download Larry’s report here.

Some of the more interesting findings were that Burp Suite Pro (an extremely cheap product built by Portswigger - and a damned fine manual testing tool, I might add) fared better than Qualys or WebInspect when trained. I always loved Burp Suite! Larry’s commentary is particularly amusing as you go through it, with choice quotes, like:

Accuntix missed 31% of the vulnerabilities after training and 37% without training. This is a significant cause for concern as they should be aware of the links vulnerabilities on their own site and be able to crawl and attack them.


WebInspect missed 66% of the vulnerabilities, even after being trained to know all of the pages. They missed 42% of the vulnerabilities on their own test site after being trained and 55% before training.

NTOSpider by NT Objectives came out in the lead with the best overall score of the application scanners tested (which included Accunetix, Appscan, Burp Suite Pro, Hailstorm, WebInspect, and NTOSpider). He also measured things like how long the various scanners take to configure, support and so on - all important things for companies about to make the big investment. This isn’t all scanners everywhere (notably WhiteHat is missing as is the newest player to the field, NetSparker, and other free web assessment tools, like Nikto etc…), but it’s a great start to a long future of heavily debated research, I’m sure. Love him, or hate him, Larry’s always got interesting research to share!

Possibly Related Articles:
Vulnerabilities Webappsec->General
Pen Testing Web Application Security Scanners
Post Rating I Like this!
Anthony M. Freed Great article!