The Flaw Counting Game Redux

Since releasing Firefox 1.5 in November 2005, Mozilla has patched 59 security vulnerabilities in the browser, more than half rated by the company as "critical."

Over that same period, according to Microsoft's Jeff Jones, the IE patch count stands at 35.

There seems to be a subtle suggestion that flaw counting is a sign that one browser is more secure -- or has less bugs -- than the other. This is a shaky stance for a very simple reason: Microsoft does not publicly disclose everything that's been patched. Or those that they know about for months and months and just doesn't fix. If you don't believe me, just check with Aviv Raff.

A few days ago, I had this discussion with Mozilla's new security chief Window Snyder and she raised another interesting point. Mozilla may prioritize flaws but, when updates are shipped, the company will fix *every* bug reported. By comparison, there are hundreds (if not thousands) of bugs that Microsoft won't fix until a future service pack. "Vulnerabilities that are found internally are fixed in service packs or major revisions and are also not counted," Snyder told me. As a former Microsoft security strategist, she should know.

Borrowing from ideas presented by Dan Geer, Allen Jones and others, Snyder said Mozilla is working on a new proposal to accurately evaluate/compare product security, using things like:

Days of Risk -- How long is the customer at risk?
Transparency -- How can the industry validate the vendor's metrics?
Complexity -- How does the architecture of the product support security?
Scope of Fixes -- Which bugs get fixed?

In the final analysis, it's all for marketing/PR purposes. But, it would sure help stop the silly counting game.