HP Security Products Blog
From applications to infrastructure, enterprises and governments alike face a constant barrage of digital attacks designed to steal data, cripple networks, damage brands, and perform a host of other malicious intents. HP Enterprise Security Products offers products and services that help organizations meet the security demands of a rapidly changing and more dangerous world. HP ESP enables businesses and institutions to take a proactive approach to security that integrates information correlation, deep application analysis and network-level defense mechanisms—unifying the components of a complete security program and reducing risk across your enterprise. In this blog, we will announce the latest offerings from HP ESP, discuss current trends in vulnerability research and technology, reveal new HP ESP security initiatives and promote our upcoming appearances and speaking engagements.

Another analysis of Larry Suto's comparative review

IBM/Watchfire released their analysis of Larry Suto's web scanner comparative review, which was released in October.  If you recall, we wrote one as well.  IBM/Watchfire questioned Suto's methodology just like we did; they also found discrepancies between their own testing and the scan files Suto provided them (yes, that's right--Suto's reported results apparently don't match Suto's own scan files).  Interesting is their discussion on how vulnerabilities are counted (issues vs. instances), and their knowledge of how NTOSpider apparently counts its findings (it counts instances), causing a higher/inflated vulnerability finding count.

Overall, Suto's analysis illustrates an important concept: testing and product comparisons are not trivial to perform.  You need a sound methodology, and you need to make sure your numbers and math make sense.

Analysis of Larry Suto's comparative case study


In October 2007, Larry Suto released a case study entitled “Analyzing the Effectiveness and Coverage of Web Application Security Scanners,” available for reading at http://www.stratdat.com/webscan.pdf.  The study compared the results of three commercial web application security scanners, including WebInspect.  There has been much discussion in the industry about this study (for a good example, see the “Coverage and a recent paper by L. Suto” thread at http://lists.immunitysec.com/pipermail/dailydave/2007-October/thread.html).  Part of the discussion focuses on Suto’s questionable methodology & conclusions relating to application coverage, and the vagueness of his results.

Since any solid science experiment should be repeatable, SPI Labs set out to re-create Suto’s study to reasonably verify his conclusions and methodology.  In doing so we discovered significant discrepancies between our results and the results reported by Suto.  Attached is our final report (Suto_review_FINAL.pdf), where we indicate the results we received when we tested the same applications.

Showing results for 
Search instead for 
Do you mean 
About the Author(s)
HP Blog

HP Software Solutions Blog


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.