Another analysis of Larry Suto's comparative review

IBM/Watchfire released their analysis of Larry Suto's web scanner comparative review, which was released in October.  If you recall, we wrote one as well.  IBM/Watchfire questioned Suto's methodology just like we did; they also found discrepancies between their own testing and the scan files Suto provided them (yes, that's right--Suto's reported results apparently don't match Suto's own scan files).  Interesting is their discussion on how vulnerabilities are counted (issues vs. instances), and their knowledge of how NTOSpider apparently counts its findings (it counts instances), causing a higher/inflated vulnerability finding count.

Overall, Suto's analysis illustrates an important concept: testing and product comparisons are not trivial to perform.  You need a sound methodology, and you need to make sure your numbers and math make sense.

| ‎12-06-2007 03:48 PM
Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Showing results for 
Search instead for 
Do you mean 
About the Author

Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.