November 30, 2007
Apples, Oranges, and the truth
The IE Blog today linked to a report that "showed that IE7 had both fewer fixed and unfixed vulnerabilities in the first year than the other browsers we compared." Paul has already pointed out that this report was generated by a Microsoft employee, but not explicitly disclosed as such.
Wanting to verify the data I wandered over to the public IE bug database that Microsoft launched to great fanfare and I encountered this:
A vivid reminder that there is no way for anyone outside of Microsoft to confirm how many vulnerabilities ever existed in Internet Explorer. In an earlier post the author of the study touts the benefits of the Software Developement Lifecycle (SDL) at Microsoft as a reason Vista is more secure. Surely one of the goals of this process is to identity and fix security bugs right? How many bugs were identified and fixed using the SDL during development? Your guess is as good as mine.
Bug counts are meaningless, what matters is whether you are at risk or not. Symantec looked at this problem before as has Brian Krebs of the Washington Post. I recently found this up-to-date analysis of data on Secunia which paints the same picture. Firefox is safer than IE:
On a related note - remember the URI vulnerability from July? When we first encountered it we, along with others, were pretty sure it was a flaw in Windows or IE. Many folks attacked us for this stance. Embarrassingly, we were vulnerable to the same issue, and we fixed it one week later.
Microsoft maintained that it was not their issue, even after I sent them this spreadsheet developed by our QA team over a weekend in July which clearly showed a change in behavior for all applications after IE7 was installed.
Three months later, when Microsoft's own Outlook and Outlook Express joined the ranks of affected applications Microsoft finally admitted it was their problem. It took another month before they fixed it. It took them three months to admit the problem and another month to fix it.
Does this look to you like the behavior of vendor trying to be open, transparent, and honest about security issues?
I expect more out of software vendors, and so should you.
Posted by schrep at November 30, 2007 5:42 PM
It's hard to improve one's reputation however "don't be evil" is a must. Will MS get it?
Posted by: funTomas at November 30, 2007 9:50 PM
You are misleading your readers.
There were ~two~ URI vulnerabilities, one in Firefox (expecting quotes), and one in Windows (ShellExec failed to check failures in URI parsing). There was no relationship between the two.
Posted by: Eric at December 1, 2007 11:37 AM
There were two issues:
In which we were not escaping quotes (there seems to be some disagreement as to how the RFC's state whether this is dangerous).
Is changes in the behavior to ShellExec with IE7 installed. The fix for 389106 migated the known attacks for 389580. My point in the post that 389580 was believed to be a windows problem but wasn't fixed for some time.†
Posted by: Schrep at December 1, 2007 2:52 PM
Good to see this. I myself had done an analysis of the days unfixed numbers a few months back and had found IE to be more insecure and posted it on my blog, and was going to reanalyse but I suppose this page makes that unnecessary. And I appreciate your honesty in including Opera numbers in your analysis, reinforcing my belief in the transparency and security of Firefox. I would also like to point out that the whole point of vulnerability patching is preventive, it prevents exploits. These low numbers for IE have not actually prevented zero day exploits, have they? How about starting an "actual days of risk" study where user exposure to exploits inspite of using a fully patched product is used!
Posted by: Nilotpal at December 2, 2007 10:43 AM
In your analysis of the operation of the Microsoft bug database you of course neglect to mention how Mozilla declares vulnerabilities "private" until they can fix them, thus denying the average user the ability to figure out if they are at risk - which I assume is the angle from where you were criticizing Microsoft.
Posted by: ktk at December 2, 2007 12:40 PM
Wow! One more great Microsoft study.
Posted by: CableGuy at December 3, 2007 4:38 AM
OBS: this blog doesn't handle new lines correctly. Anyway, you said: "Bug counts are meaningless, what matters is whether you are at risk or not."
That phrase remembers me of Ranum's 3rd Dumbest Ideas in Computer Security.
It reads: "The premise of the vendors is that they are doing the right thing by pushing out patches to fix the bugs before the hackers and worm-writers can act upon them. Both parties, in this scenario, are being dumb because if the vendors were writing code that had been designed to be secure and reliable then vulnerability discovery would be a tedious and unrewarding game, indeed!...
One clear symptom that you've got a case of "Penetrate and Patch" is when you find that your system is always vulnerable to the 'bug of the week'...
Your software and systems should be secure by design and should have been designed with flaw-handling in mind."
Basically what he is saying is that risk management is for hairy pointy bosses and that programmers should focus on security engineering.
I bet you agree that wu-ftpd is NOT more secure than djb's "publicfile" or Postfix. Now substitute wu-ftp in that phrase by Firefox and publicfile by IE. Yes, what I am meaning is that maybe IE is getting as secure has DJB's softwares and Firefox is getting has insecure as wu-ftp.
And independent of the number of vulnerabilities that IE has, what matters and what I want you to explain me is why Firefox has too many? Is it bad programming pratices or what?
Because while a low number of vulnerabilities is not a symptom of security, a high number is a symptom of insecurity. Now, tell again, how many vulnerabilities Firefox has? That's penetrate and patch.
Why aren't you writing code like Wietse Venema or Dan Bernstein?
PS: And I am actually an Firefox user, but I am just not happy patching my browser once a week.
Posted by: Paul at December 3, 2007 10:48 AM
Nilotpal - we definitely believe in "responsible disclosure" which means the vulnerability should be disclosed first to the vendor to give them a chance to fix it before it is made public. However, we open up these bug reports once the fix is out, so that users, partners, and researchers can understand and verify the fix. We also open it if the issue becomes public before it is fixed. In addition, access to these bugs is provided the security group at Mozilla which includes trusted individuals and organizations outside of Mozilla who can help analyze, verify, and audit our work.
This means if we find and fix a security issue internally we still report it and open the bug and MSFT does not. This makes counting vulnerabilities meaningless since you are missing every internally found and fixed bug in MSFT's case but not ours.
Posted by: Schrep at December 3, 2007 10:53 AM
I understand your concern for security, but it seems to me that:
1. the two bugs you mention are not related to each other (besides the fact that are related to URI handling). Even in absence of the second bug (for example on Vista) the first bug would have been still exploitable and as such "The fix for 389106 migated the known attacks for 389580" is a blatantly inaccurate statement. Am I right?
2. in the last few days Firefox was the only browsare exploitable for the quicktime bug (source: http://it.slashdot.org/article.pl?sid=07/11/27/1851212&from=rss ). Even the worst browser around, IE6, is not affected. This leads me to a question: why system wide security measures like Vista protected mode are not a priority at Mozilla? I am still trying to make sense of what you said in an interview:
"The animated cursor attack would still allow for reading of any files on the local system - so protected mode is no panacea. We believe pro-active and rapid patching of security vulnerabilities is still the best defense."
Pardon the metaphore but if a new door lock is available you don't take it in consideration because that leaves still a way to break in through a window at the fifth floor, so sending the police patrol after every break in is still the best possibile option.
Am I reading this right?
Posted by: Donald at December 4, 2007 9:02 AM
I just thought I'd point out the quote says "..the other browsers we compared.", which may mean they did not include Firefox. Just as the information was not shared that the report was generated by an MS employee, it also doesn't disclose what the "other" browsers were.
Posted by: Tim at December 7, 2007 11:46 AM