Practical Technology

for practical people.

Open-Source Security Idiots

May 15th, 2008 · 13 Comments

Sometimes, people do such stupid things that words almost fail me. That’s the case with a Debian ‘improvement’ to OpenSSL that rendered this network security program next to useless in Debian, Ubuntu and other related Linux distributions.

OpenSSL is used to enable SSL (Secure Socket Layer) and TLS (Transport Layer Security) in Linux, Unix, Windows and many other operating systems. It also includes a general purpose cryptography library. OpenSSL is used not only in operating systems, but in numerous vital applications such as security for Apache Web servers, OpenVPN for virtual private networks, and in security appliances from companies like Check Point and Cisco.

Get the picture? OpenSSL isn’t just important, it’s vital, in network security. It’s quite possible that you’re running OpenSSL even if you don’t have a single Linux server within a mile of your company. It’s that widely used.

Now, OpenSSL itself is still fine. What’s anything but fine is any Linux, or Linux-powered device, that’s based on Debian Linux OpenSSL code from September 17th, 2006 until May 13, 2008.

What happened? This is where the idiot part comes in. Some so-called Debian developer decided to ‘fix’ OpenSSL because it was causing the Valgrind code analysis tool and IBM’s Rational Purify runtime debugging tool to produce warnings about uninitialized data in any code that was linked to OpenSSL. This ‘problem’ and its fix have been known for years. That didn’t stop our moronic developer from fixing it on his own by removing the code that enabled OpenSSL to generate truly random numbers..

After this ‘fix,’ OpenSSL on Debian systems could only use one of a range from 1 to 32,768—the number of possible Linux process identification numbers—as the ‘random’ number for its PRNG (Pseudo Random Number Generator). For cryptography purposes, a range of number like that is a bad joke. Anyone who knows anything about cracking can work up a routine to automatically bust it within a few hours.

Why didn’t the OpenSSL team catch this problem? They didn’t spot it because they didn’t see it. You see Debian developers have this cute habit of keeping their changes to themselves rather than passing them upstream to any program’s actual maintainers. Essentially, what Debian ends up doing is forking programs. There’s the Debian version and then there’s the real version.

Usually, it’s a difference that makes no difference. Sometimes, it just shows how pig-headed Debian developers can be. My favorite case of this is when they decided that rather than allow Mozilla to have control of the logo in the Firefox browser, because that wasn’t open enough according to the Debian Social Contract, they forked Firefox into their own version: Iceweasel.

That was just stupid. This is stupid and it’s put untold numbers of users at risk for security attacks.

First, the mistake itself was something that only a programming newbie would have made and I have no idea how this ever got passed by the Debian code maintainers. This is first-year programming assignment. “What is a random number generator and how do you make one?”

Then, insult to injury, because Debian never passed its ‘fix’ on to OpenSSL, the people who would have caught the problem at a glance, this sloppy, insecure mess has now been used on hundreds of thousands, if not millions, of servers, PCs, and appliances.

This isn’t just bad. This is Microsoft security bad.

Now, there’s a fix for Debian 4.0 Etch and its development builds. Ubuntu, which is based on Debian,, also have fixes for it. In Ubuntu, the versions that need patches are Ubuntu 7.04, Feisty; Ubuntu 7.10, Gutsy; the just released Ubuntu 8.04 LTS Hardy, and the developer builds of Ubuntu Intrepid Ibex.

Debian has also opened a site on how to rollover your insecure security keys to the better ones once you’ve installed the corrected software.. For more on how to fix your system, see Fixing Debian OpenSSL on my ComputerWorld blog, Cyber Cynic.

Tags: Linux · Network Services · Open Source · Security

13 responses so far ↓

  • 1 Apologize SJVN // May 16, 2008 at 5:47 pm

    Wow, Steven you really messed up here. I know it is fashionable to hate the Debian folks, but honestly you should have at least done half a minute of research before posting this crap. You are accusing the Debian developers of keeping this patch to themselves, when they did NO SUCH THING. The proposed patch was posted to the OpenSSL developer mailing list and was given an EXPLICIT “GO AHEAD” by the OpenSSL
    team.

    I think you owe the Debian developers an apology here.

    http://marc.info/?l=openssl-dev&m=114651085826293&w=2

  • 2 sjvn // May 16, 2008 at 7:26 pm

    Ah, but. I do check. I didn’t want to put the blame on one person, but since you brought the matter up. the person cited is Kurt Roeckx: the Debian OpenSSL maintainer.

    http://qa.debian.org/developer.php?login=Roeckx&comaint=yes

    He’s the one who made the mistake. If you read on in the thread. you quote, you’d see that someone gave him the flag to make the debugger code work properly with OpenSSL

    http://marc.info/?l=openssl-dev&m=114654760312453&w=2

    As I mentioned in the story, people had been running into this problem for years and assuming, incorrectly, that the problem was with OpenSSL and being corrected. The difference here is that he was in a position to mangle OpenSSL to ‘fix’ what he thought was a problem and he then didn’t pass it upstream. So, it’s still a Debian problem.

    Then, someone at Debian, presumably Roeckex since it’s his project, decided to fix the problem a few days before it went public without telling anyone.

    See Ben Laurie’s blog (http://www.links.org/?p=327) for details and pointers to the patches.

    For more on what actually happened, see: Gregely Risko’s Debian’s OpenSSL maintainer (Kurt Roeckx) should be changed (http://www.gergely.risko.hu/debian-dsa1571.en.html)

    Sorry, this problem is squarely in Debian’s court.

    Steven

  • 3 timstarling // May 16, 2008 at 7:37 pm

    There was no explicit “go ahead”, the OpenSSL developers just said “compile it with -DPURIFY”. None of them bothered to properly review the proposed patch.

    I think upstream developers generally need to realise that the quality of the Debian packages is crucial to the user experience. They should maintain the packages directly, not leave it to a random self-selected volunteer who will inevitably have to operate with no review process and no expert assistance.

  • 4 sjvn // May 16, 2008 at 7:52 pm

    The problem is that Debian makes a point of maintaining their own and doing it themselves. Other distributions do the same thing, but Debian makes the biggest point about doing it their own way. Since it’s open source you can;t stop them. Perhaps in the future, if Debian continues in this way they should start renaming their programs. After all they did it with Firefox, were I OpenSSL’s developers, I wouldn’t want my name attached to the Debian version of their program.

    Steven

  • 5 gfolkert // May 16, 2008 at 8:18 pm

    Hmmm. So are we reminded yet about the days when RedHat made fixes to packages with “proper” review… and then failed to realize the entire scope of that fix.

    You see RedHat at one time had the Big Ole’ Hat of being the “Largest Bug Fixer that introduced HYARGER holes than they patched” patch producer.

    In fact, there was a point where a certain round of released update RPMs made things *SO TRIVIAL* to remotely exploit a RedHat boxen, that the *ONLY* safe things was to literally *UNPLUG* the network connection until you could either re-install the OS and then patch it up via CDs after the fix for the fix (that broke everything) for the fix was released. Imagine if everyone would have said something along the lines of:

    Wow, how can we trust this company called RedHat and its Linux distribution anymore. Lets all switch to MACOS 8.1 or this NEW to be released RSN, Windows NT5 thing.

    This *WAS* before the day of the 2.4 Kernels… but just. So why aren’t you jumping up and down over those things that happened as accidents then… just like you are on this one that more than likely happened as an accident now.

    Oh… what about the fact that OpenBSD now has had *2* security problems on default installs. Why, why, why even use OpenBSD with its BRAZEN disregard for Security.

    Sorry, Vaughn-Nichols, its an over reaction on your part. Nice job getting click through, though. I like your style most of the time, but … we all have off days.

  • 6 timstarling // May 16, 2008 at 9:04 pm

    If Debian refuses to sponsor requests by upstream developers to maintain their own packages, then yes, I would say we have that kind of problem. The only solution then would be to make fun of Debian and hope that everyone stops using it, as you seem to be suggesting.

    But I don’t think that solution is particularly likely to work. So I’m hoping that the problem is something else: that is, an unwillingness by upstream developers to engage with Debian’s bureaucracy.

  • 7 segedunum // May 17, 2008 at 8:38 am

    Oh, come on people. Let’s treat this with the seriousness it deserves.

    I see that OpenSSL list conversation bandied around as some sort of ‘proof’ that approval was sought and gained for this patch. It wasn’t in any way shape or form. There was *no* explicit ‘GO AHEAD’ anywhere in there at all. All that happened was a short discussion on Valgrind throwing up bogus problems and a discussion of PURIFY. There was no attempt to get this patch upstream where it could be properly reviewed by people who really know the code either, which is where any real ‘GO AHEAD’ would have been given.

    This is a *real* problem with Debian. While you can understand backporting some security fixes (and sometimes this is a problem as upstream moves their focus on to new versions), or tweaking configure so that you can fit the package in as all distributions do, arbitrarily patching an application because you feel like it and because Valgrind throws out stuff you don’t like is absolute amateur hour. There’s even an upstream bug for the issue, and the sensible thing for any distributor to do is wait for it to be concluded.

    When I looked at distributions that I was going to use a few years ago I dismissed Debian for this very reason. The place for development and getting patches approved is upstream. When a new upstream version is released then you release and update for your distribution, or you backport a (security) patch if you really can’t do that. I was really alarmed at their willingness to create patches to upstream code themselves, that weren’t submitted back upstream, because they didn’t like the way it worked or it didn’t fit in with Debian’s policy etc. There is also a worrying review system where things get thrown into testing and unstable, and if no one complains then it gets thrown into a new release. Of course no one complains because no one is properly reviewing the code changes made or their effects. It flies totally in the face of the ‘many eyes’ concept of open source development. As it is, I mainly use CentOS with some OpenSuse thrown in, and CentOS is fastidious about leaving the upstream package alone unless there is some real trademark issue that warrants it being changed.

    I have a great many OpenVPN and SSH keys for all sorts of purposes, and if I had to regenerate all of them I would be seriously annoyed. If it was a genuine security problem upstream, and was unforeseen, then I could just about understand. This was stupid and totally foreseeable.

    I always laugh at people who describe Debian as ‘stable’ and ‘rock solid’ on various forums because of the long testing cycle to get packages into stable. It is anything but. All it means is that Debian gets totally out of date with what upstream are doing, they have to backport a great number of fixes, and on top of that, they hack on the code themselves.

    All you end up having in the end is a seriously out of date distribution, with bugs and security issues no one else has. Great!

  • 8 sjvn // May 17, 2008 at 8:40 am

    I toast everyone who makes mistakes that should have been avoided. So, back in the day, I did blast Red Hat. I blast Microsoft on what feels like a weekly basis, but when someone else blows it, I’ll call them as it well. I hope Linux to the same standards I hold Windows. It’s just that Linux usually gets it right.

    As For Debian, the common practice is for downstream developers, of any project, to push their changes back upstream. It’s the way most open-source projects works.

    I am not saying don’t use Debian. While I rarely use pure Debian I use Ubuntu, Mint and MEPIS, all of which owe their existence to Debian, on a daily basis. What I’m saying is Debian must do a better job of avoiding this kind of whopper of a mistake. It makes Debian look horrible and, since most people can’t tell one distribution from another, it gives all of Linux a black-eye.

    Steven

  • 9 gfolkert // May 17, 2008 at 10:41 am

    Quoting: “What I’m saying is Debian must do a better job of avoiding this kind of whopper of a mistake. It makes Debian look horrible and, since most people can’t tell one distribution from another, it gives all of Linux a black-eye.”

    Yes, I’ll agree and even to the point that Debian was in the wrong and they have responded like this:

    It was a mistake, let’s fix it, move on and be more careful about these things.

    In General (about 99.9%+) Debian *DOES* contribute its patches upstream. In general Debian patches that get refused are refused as “trivial” or “Won’t Fix” as they are deemed Debian Specific. Well, trivial mainly because they are changes to follow the DFSG, the packaging requirements for multi-platforms (suchs as making a docs package -noarch, or config -noarch vs specific binary like sparc or alpha or IA64 etc), and proper file-system heir.

    And to be honest, Debian has as many patches submitted upstream to the OpenSSL project as any other project… actually I believe more. Especially in the day when it (straight from the OpenSSL site) wouldn’t work properly on half the platforms Debian Supported.

    I have one package you might follow-up on in regards to Debian showing the light as to how to do things:

    Exim4 with many small config files.

    Exim4 (as written by Philip Hazel and his trusted team) thinks its an abortion. But I have to disagree. It is by far a MUCH more sensible method of configuration. Rather than having to parse out multi MB text file for the proper location, its a matter of looking at the right smaller related and aptly named config file. I can administrate a huge amount of vhosting and special configs. I can just add well formed files and have support for new features in minutes rather than having to hope I put it in the right order in the config file and hoping it doesn’t bomb out.

    This Modular approach to config files has made administration of things easier BY FAR, plus look at the other Debian Teams picking up this idea and running with it… and Ubuntu seems to have also embraced it. Even some upstream sources have adopted this way.

    Bleah, nuff blather from me.

    Also, one thing: You Holding Linux to the same Standards as Windows… Wow. Just wow.

    Yes, I knew what you meant.

  • 10 linsec.ca blog » Blog Archive » The path of least-patching // May 17, 2008 at 10:59 am

    […] to the OpenSSL issue, with some interesting comments, is on the Practical Technology blog, entitled Open-Source Security Idiots. There are probably others, but I think two is […]

  • 11 daniel.haxx.se » Blaming Debian packaging // May 17, 2008 at 4:05 pm

    […] happened to read the blog post called Open-Source Security Idiots which really is having a go at the poor Debian maintainer of OpenSSL for causing the recent much […]

  • 12 FreeSoftNews » Blog Archive » Open-Source Security Idiots // May 18, 2008 at 1:47 pm

    […] Read more at Practical Technology […]

  • 13 Debian — troubling signs; can Slackware teach us anything? | The Sillican Files // Jun 11, 2008 at 5:23 pm

    […] which is #12. The issue arose because once again, one of the Debian package maintainers decided to go their own way without including upstream developers in the process. The short of it is that Valgrind, a well […]