Updated January 18, 2006: Interesting comment posted

Apex Assurance Group was at ShmooCon to talk about Security Testing and Certification. The panelists were as follows:

• Al Potter - ICSA Labs
• Bruce Potter - Booz Allen Hamilton (and Shmoo Organizer)
• Ray Potter - Apex Assurance Group

Note: there is no relation between the panelists. And why get that many Potters in one room? Because we could.

The discussion was in BoF format, and it was certainly one of the more useful, enlightening, and entertaining discussions around security testing we’ve seen recently. First of all, *everyone* in the room was familiar with the FIPS 140, Common Criteria, and ICSA programs. And judging by the insightful comments, it was clear that the audience read the standards, understood their purpose, and implemented/configured certified products.

Must of the discussion centered around Common Criteria, where the group discussed the following:

• Certified products are not necessarily secure

  There is no conclusive research that certified products are more/less vulnerable than uncertified products. FIPS 140 and Common Criteria are about assurance, not about security. One BoF participant made the comment that the programs are for point products, and hooking certified products together will not make the system secure. This is a very true and valid point; Apex Assurance Group would like to remind readers that FIPS and Common Criteria are only a part of systems security design and engineering.

  One suggested resolution was to have a group of independent, certified, firewalled (”ed” - not “firewall”) experts to conduct a series of vulnerability assessments and penetration testing on the product under test in paralle to the work done by the Common Criteria evaluation and testing team. This could provide a more “security” meat behind the assurance certification. Not a bad thought, actually. The devil is in the details and the execution, of course. But it’s a valid concept.

• The value of certification is over-substantiated (or just not understood)

  Surprisingly, FIPS 140 and Common Criteria weren’t faulted in their delivery of security assurance. The audience widely agreed that FIPS 140 and Common Criteria do not meet the security needs of end users. The perception and marketing of these programs took the most criticism.

From our point of view, the main take-away from the discussion is that security assurance and security testing are complex topics. There are misperceptions about the purpose and value of FIPS 140 and Common Criteria, and while they are valuable, there is vast potential for improvement in how these programs can meet the security needs of end users.

Finally, hats off to Bruce Potter and the Shmoo Group for putting this conference together. The con was well organized, well attended, and very well managed.