OSTG | ThinkGeekSlashdotITMJLinux.comSourceForge.netfreshmeatNewslettersPriceGrabberTechJobsBroadband
-advertisement-
NewsForge
April 14, 2005

  Special Offers      
Get news and special offers on:
Networking
Internet Security
E-commerce
Linux
Email:

Operating Systems

Top Story

BitKeeper and Linux: The end of the road?

Monday April 11, 2005 (12:00 PM GMT)
By: Joe Barr

  Printer-friendly   Email story  

BitKeeper, the proprietary source management tool used by Linus Torvalds and other Linux kernel hackers to apply patches to their versions of the kernel, is once again at the center of controversy. This time it looks as if the relationship between BitKeeper and the poster-child project for free software is going to come to an end as the result of irreconcilable differences. We spoke with the three primary figures involved in the dispute -- Larry McVoy, Linus Torvalds, and Andrew "Tridge" Tridgell -- to learn what happened and whether it could have been avoided.

We asked Larry McVoy, BitKeeper's primary author, to tell us what happened to cause him to end the relationship by giving Torvalds and the Linux kernel developers using BitKeeper three months to move to another tool for their source code management. He explained:

Back on February 23 I learned from Linus that Tridge was reverse-engineering BK so that he could pull stuff out of BK trees without agreeing to the BK license. As you might expect, we were less than thrilled and began having talks with Linus, Tridge, and Stuart Cohen, the CEO of OSDL. These talks didn't go anywhere. Tridge believes strongly enough in free software that he thinks anyone using non-free software is living in sin.

Linus worked very hard to get Tridge to stop. He and I spent a significant amount of time on this issue and Linus understands my position very well He summarized it nicely:

Larry is perfectly fine with somebody writing a free replacement. He's told me so, and I believe him, because I actually do believe that he has a strong moral back-bone.

What Larry is _not_ fine with, is somebody writing a free replacement by just reverse-engineering what _he_ did.

Larry has a very clear moral standpoint: "You can compete with me, but you can't do so by riding on my coat-tails. Solve the problems on your own, and compete _honestly_. Don't compete by looking at my solution."

And that is what the BK license boils down to. It says: "Get off my coat-tails, you free-loader". And I can't really argue against that.

This position seemed to be lost on Tridge.

Concurrently we were working with OSDL's management. In this area I pulled in calmer heads than mine and our VP of sales got involved. He negotiates all of our enterprise level agreements, (his strength is finding common ground) so you can imagine he's a pretty reasonable guy. He was unsuccessful in getting anywhere with OSDL's CEO. Stuart's position was that this was not their problem and this is the sort of activity you expect in the open source world. We did get a verbal promise from OSDL that Tridge had discontinued his work and would not begin again as long as we were trying to work things out. We believed we had an uneasy truce, but it ends up Tridge was still working.

We ended up in a no-win situation. OSDL didn't appear to care and we couldn't trust what we were being told. With that we were fairly confident that Tridge was going to release his code. That was a problem for us for two reasons:

a) Corruption. BK is a complicated system, there are >10,000 replicas of the BK database holding Linux floating around. If a problem starts moving through those there is no way to fix them all by hand. This happened once before, a user tweaked the ChangeSet file, and it costs $35,000 plus a custom release to fix it.

If Tridge's tool is out there we are now supporting our code and his code. We couldn't do that.

b) IP loss. If we sat back and did nothing about Tridge then we are implicitly condoning reverse engineering.

Internally, we were looking at ways to best handle this. One option was to have two versions of BK, one that we gave away and another that was commercial only. This had been our course for some time and it wasn't working out. The difficulty with that solution is we couldn't just stop all work on the free version because of future compatibility issues. Trying to maintain compatibility between a free product and commercial version was grinding our development to a halt. Everyone was losing. In order for this to work we had to continuing throwing resources at the problem. We're already up to about $500K/year for the free version and continuing to ratchet that up wouldn't be prudent.

At that point we started looking at what it would be like to discontinue the free BK. Linus strongly encouraged us to do this once he came to the conclusion that the costs of the free version to BitMover outweighed the benefits to BitMover.

OSDL's management was kept informed of what we were thinking and again they seemed rather apathetic about it. Their position was that it was BitMover's problem and we needed to figure out how to fix it. That is until we set the wheels in motion to discontinue the free product. They did make motions very recently that we should work together on this, but it was too little too late.

We finalized our discussions with Linus last weekend and he began the process of migrating off of BK. Linus is a very ethical guy. His feeling was that we were getting a bad deal and he didn't want to be part of that. So off he went. We spent the next couple of days scrambling to figure out how we were going to handle this, announcements, migrations, programs moving forward, etc. We're still working on the details moving forward with some of these issues, but our hope is to make this as smooth as could be expected for this sort of transition.

Torvalds' take on the situation

After reading McVoy's response (he cc:ed Torvalds on the email he sent to us so that he would not misquote or take out of context anything he quoted from their correspondence), we asked Torvalds three questions.

NewsForge: What will you use to replace BitKeeper?

Torvalds: I don't even know yet. I'm playing around with my own scripts and tools right now, and talking to various open-source source code management (SCM) people. In fact, I'm trying to get as many people as are interested in the problem to just explore the options.

NewsForge: How will this impact your workload short or long term?

Torvalds: Short-term we'll merge patches. Right now the -mm tree should work like usual, so people can go on developing. On the other hand, especially people that used BK will just slow down, take a breather, and look around at the alternatives.

And some people will just continue to use BK. It didn't go away, and it's still the best SCM out there, it just got harder to merge with me (and some of the people I work with). So you export BK changes by patches instead, but some people largely worked like that _anyway_ before (i.e. they used BK for its merge capabilities).

So we'll definitely have a slowdown in the short run, but it's not likely to be a huge deal. The biggest worry is developer frustration about the uncertainty, so I'm certainly trying to get to a decision, but on the other hand I don't want to hurry it _too_ much either.

(Ironically, many users and distributions are likely to actually not mind slightly slower development for a while. One of the most common worries for users is just the fact that 2.6.x has continued to be developed at a very high rate thanks to just how smoothly it's been working, so I bet some people are both upset and gratified by this all. ;)

In the long run, we'll just have to see. I think in the _medium_ run the problem is going to be just having to live with less capable tools, and having to possibly teach old developers (me included) new tricks.

And I don't mean the various syntactic differences between different SCMs, I mean new ways of working and adapting to new constraints (and likely new freedoms too -- BK had its own set of constraints simply due to the model of development that _it_ imposed -- every SCM tool to some degree has a "world view," and it takes time to get comfortable with that world view...

NewsForge: Was this split inevitable, or could it have been avoided?

Torvalds: I think everybody saw the split as inevitable _eventually_. I don't think anybody believes that the open-source SCMs wouldn't grow up, and when they would, there would have been obvious reasons to switch over eventually.

But I think it could have been a lot less painful if it happened a year or two down the road, and that's my only real regret. That said, we did get three very productive years out of it, and we not only learnt how SCMs can work, we also taught a lot of people what to expect of a _good_ SCM, so anybody who claims that it was a waste of time to go with BK obviously doesn't have his head screwed on right. BK did good.

Tridge offers his side

There is no doubt Tridge is being cast as the villain in this piece. Here's what he had to tell us when we asked him for his side of the tale:

I expect that in the future I will be able to give a more detailed response, but for now I can only tell you the following:

- In late February I wrote a tool that is interoperable with BitKeeper. The aim was to provide export to other source code management tools and provide a useful tool to the community.

- I did not use BitKeeper at all in writing this tool and thus was never subject to the BitKeeper license. I developed the tool in a completely ethical and legal manner.

At the end

In spite of the end of the relationship, McVoy and Torvalds seem to have lost no respect for each other's integrity or professionalism. Torvalds still admires BitKeeper, and still feels it to be the best tool for the job. Whether this outcome was inevitable or not, it's a little bit sad to see this marriage of proprietary and free software come to an unhappy end. Not to mention a little unsettling, due to the uncertain handling of patches in the future.

  Printer-friendly   Email story  

  Comments      

Top  |  122 comments  |  Search Discussion  |  

The message I'm getting is (Score:0)
By Anonymous Reader on 2005.04.11 8:56 (#110179)
that reverse engineering is evil, and is stealing. And Linus agrees.

Interesting.
[ Reply to This ]
great reporting, Joe (Score:0)
By Anonymous Reader on 2005.04.11 9:00 (#110180)
Finally we get the whole story. Obviously, this has to be taken in context of Linus the pragmatist straddling the free software and commercial software worlds. He's done a brilliant job at it so far, but it remains to be seen if he'll get derailed by these religious skirmishes - GPL v3 could be another one if the FSF goes forward with the SleepyCat CEO floated in the trial balloon.

My quick take is that OSDL management should've read Tridge the riot act, and canned him if necessary. They were the biggest villains here. But that's just me.
[ Reply to This ]
Good Job! (Score:0)
By Anonymous Reader on 2005.04.11 9:39 (#110188)
I for one will sleep better knowing that
McEvoy's "IP" is safe from the unfair and evil competition of reverse engineering!

Damn you Tridge, Damn you (all) to hell! Why don't you go join the Samba Team and you and Samba (and any other open source reverse engineering thugs) and Satan can have a nice little tea party together?

Did I mention that reverse engineering is not only unpatriotic but also communistic AND can give you herpes?

Sorry about the sarcasm the meds haven't kicked in fully yet.

Cheers,

E.

[ Reply to This ]
McEvoy Says more of what Linus says than Linus?!? (Score:0)
By Anonymous Reader on 2005.04.11 10:03 (#110193)
Seems to me like most of what Linus had to say was under what McEvoy said Linus said ("yes, I totally agree with McEvoy, and Tridge was bad bad bad..."). Would have been nice to have heard it come directly from him instead.

From his comments, it looks more like he's not really taking sides on this. More of a, "Well that sucked, but that's life attitude...". Guess that's some of the qualities that make him such a good benevolent Linux dictator.
[ Reply to This ]
Reverse engineering (Score:0)
By Anonymous Reader on 2005.04.11 11:00 (#110195)
One thing that is not clear to me is to what extent Tridge should have been restrained by the terms of the BitKeeper license, and whether it was the free license or a paid-for license.

1. Tridge says he didn't use BK at all in the development of his software. If that is true, then McVoy and BitMover are being unfair (with the caveat that the free use of BK was a gift in the first place, and that no can complain if the gift is withdrawn for any reason, even a whim).

2. If Tridge was using a free copy of BitKeeper to reverse-engineer it, then McVoy can legitimately be upset. However, Tridge says he didn't do that.

3. If Tridge had a paid-for copy of BitKeeper, then I believe most legal opinion would hold that he is legitimately entitled to reverse-engineer it for the purposes of creating interoperable software, even if McVoy doesn't like it.

4. If the whole thing is based upon Tridge being an employee of OSDL and doing things McVoy doesn't like, and McVoy subsequently calling OSDL to task for it, then I don't think McVoy is being fair (subject to the "whim" caveat listed above).

At any rate, I don't see how BitMover benefits at all from these recent events. Keeping Linux kernel developers from using BitKeeper for free isn't going to prevent reverse engineering in any way. If anything, the withdrawal of free BK by BitMover has probably hastened the development of a free (in the FSF sense) BK replacement.
[ Reply to This ]
Oh, Ok.... (Score:0)
By Anonymous Reader on 2005.04.11 12:11 (#110204)
So I guess everyone who agrees with Linus and BK doesn't use some tool to read word docs without using word? I'm glad Tridge did what he did. Or would you guys all rather go DMCA on him.

Long live GNU Arch.

Oh, and don't use samba anymore either.

[ Reply to This ]
what if? (Score:0)
By Anonymous Reader on 2005.04.11 12:54 (#110208)
What if there was a price on open source software (fair market value)? It would be interesting to see how closed source competes. I don't care if open source charges for the source code in camels and chickens.

BK is better than anything out there and Linus is a technical guy - so he chose the best. I now believe that people running Linux aren't doing it because of any technical advantages of having access to source - it's because you guys are too FUCKING CHEAP.

BitKeeper was a "linux" company - they developed and sold software designed for LINUX and now you've found a way to drive them out of business.

Jeez Tridge, there's a million other things to do for Linux than hacking BK. Why the fuck couldn't you let LM/BK live (after all it worked great for 3 years!). How would you like it when someone fucks with your livelihood?


[ Reply to This ]
Is BK asserting privilege over their protocol? (Score:0)
By Anonymous Reader on 2005.04.11 13:31 (#110214)
I confess some ignorance regarding exactly how BK works. However, I don't understand how Tridge could make a tool to work with BitKeeper without reverse engineering. McEvoy says he has no problems with making tools that interoperate with BK, but it does not sound like BitMover makes proper documentation available to do it WITHOUT reverse engineering. So, is BK actually just asserting some kind of trade secret status to their protocol? If so, that's just silly.
[ Reply to This ]
McVoy is the culprit. (Score:1)
By r55 (156761) on 2005.04.11 13:36 (#110218)
He has been very narrow minded, let us see how will his product progress without active usage*
I just hope McVoy will get mad being overjoyed with the result of his dumb tactics.

* I believe Linus Torvalds was the only sane guy using it.
PS: Reverse engineering is not stealing one's intellectual property, it is actually complementing the code's creator of, the intellect that they lack.
[ Reply to This ]
    Talk to IBM (Score:0)
    By Anonymous Reader on 2005.04.11 13:59 (#110221)
    They've had some really good source control tools over the years. They might be in a mood to open source something that could fit the bill nicely. The worst thing that could happen is that nothing they have is suitable, or they aren't prepared to release it yet as open source.
    [ Reply to This ]
    No surprise in this news (Score:0)
    By Anonymous Reader on 2005.04.11 14:42 (#110238)

    People told Linus a long time ago that going with proprietary software for something as important as source control was a bad idea. He is not infallible, you know. I believe he got this decision wrong.

    Slightly more surprising is the range of reactions to the news. Some people seem to think that Tridge did something wrong. Perhaps they will stop using OpenOffice and go back to using MS Office? (OO.o reverse-engineered .doc and .xls files). Or stop using cheap printers, which work under Linux solely because somebody deciphered their data protocols without permission? In view of their limited capacity for logical thought, I think that trying to explain to them the difference between reverse engineering and what Tridge did is a waste of time.


    [ Reply to This ]
    Reverse engineering, BIOS and IBM (Score:0)
    By Anonymous Reader on 2005.04.11 14:44 (#110240)
    For those young guys in the computer world and cannot remember the PC history... If the IBM PC had not been reverse-engineered (a case that IBM lost because it was a cleanroom implementation), then we would not have this discussion at all. Please remember that.

    There is nothing wrong with reverse-engineering. However, a proprietary SW firm wants to make money from software. The trick is to have good marketing (MS has had the best for a long time) and a product noone else has (to some extend). As long as you can prevent the commodity side of your product, then you are ruler when you know how to market your stuff. Once a the product becomes a commodity (think samba), then marketing does little help because cheaper than cheap is not possible.

    The reaction in this case IMO shows that everything is done to prevent commoditizing BK. Linus himself says that a few years might pass before a sufficient replacement would/could be available (on technical grounds!). This time can make Bitmover a lot of money, as long as BK is not commoditized.

    In short, it is all about money.

    [ Reply to This ]
    It costs money to create new products (Score:0)
    By Anonymous Reader on 2005.04.11 15:54 (#110256)
    The reverse-engineering issue is not a simple good vs evil argument. There are valid pros & cons to both sides of the argument.

    I think Linus is handling it in the best possible manner given the unfortunate circumstances.

    Whether your are pro or anti reverse-engineering, I urge you to use your brain and objectively THINK about all the pros & cons from both positions.

    A good clue that you are a dogma-recycling zombie is when you utterly fail to come up with at least one valid & important argument in favor of the other side's position.

    If you are pro reverse-engineering and cannot think of any valid, important benefits to reverse-engineering, then please wake up and educate yourself by learning how reverse-engineering has or can benefit the world (think about important products that are no longer supported, etc).

    If you are anti reverse-engineering and cannot think of any valid, important benefits to prohibiting reverse-engineering, then please wake up and educate yourself by learning how reverse-engineering has or can harm the world (think about reduced R&D investment for medicine, startup funding, etc).

    If we get enough people to stop being mindless zombies, we might be able to come up with pragmatic solutions that give the the most benefits while avoiding the most disadvantages.

    We should incent inventors and their investors in some manner so they expend the time, energy and money to create new, non-trivial inventions. And at the same time, allow others the freedom to compete. These two goals sometimes conflict, but we should be smart enough to find a solution.

    If inventors & investors lose the (financial/recognition) incentive to expend non-trivial resources, then there will be nothing worthwhile to reverse-engineer in the future! And that would be tragic.

    This is the problem the patent system attempts to solve, but is failing miserably at executing. For the patent system to work:
    1. the duration needs to be shortened to 10 years max (which means inventor gets 6-7 years of monopoly in exchange for FULLY & ACCURATELY disclosing to the public how to create his invention)
    2. a LOT more resources should be given to patent examiners so that:
        2a. bullshit patents with prior art do not get issued by mistake
        2b. bullshit patents that fail to describe the invention in ENOUGH DETAIL so that others can build it without UNDUE experimentation do not get issued by mistake (this is an actual criteria most people don't know about)
        2c. valid patents should be issued within 1 year of application, rather than 3-4 years it takes now.
        2d. there should be a 6 month 'inventor cannot sue anyone' period in which the public can submit evidence of prior-art to invalidate the patent after it has been issued and before anyone gets threatened by it. it is better for the public to review ISSUED patents rather than APPLICATIONS due to massive volume--this no-sue period of public review/comment is a perfect solution.

    Again, if the patent doesn't disclose the new invention in enough detail for others of average skill to make it (which makes reverse-engineering unecessary) then the patent SHOULD NOT GET ISSUED (according to current USA patent regulations).

    Everybody wins. Inventors get short-term monopoly in exchange for creating and fully disclosing their inventions.
    [ Reply to This ]
    McVoy is telling the actual reason too: too expens (Score:1)
    By JelleB (190404) on 2005.04.11 16:02 (#110257)
    ...ive This thing is not about tridge reverse engineering BK at all. That code has not been released yet. They state the real reason themselves: it was too expensive to maintain a free and a non-free version. BK had their free publicity/endorsement en sold a lot of licences because of that. Nowhere has McVoy showed gratitude for that. But somehow he has the right to complain when somebody does something that they are legally entitled to. He just needed a scapegoat to be able to pull the free bk version.
    [ Reply to This ]
    Advertising (Score:0)
    By Anonymous Reader on 2005.04.11 18:46 (#110275)
    There goes the best free advertising that BitMover could buy!

    I'm sure quite a few people are aware of BitKeeper only because of Linus & Linux.
    [ Reply to This ]
    Go Tridge! (Score:0)
    By Anonymous Reader on 2005.04.12 1:05 (#110297)
    Tridge, don't listen to Larry and Linus - there is nothing wrong with reverse engineering. Larry is full of shit, as usual. Let him take his precious BK and shove it up his arse.

    This serves Linux development process and Linus quite right. Using a piece of proprietary software to develop a GPL-ed kernel. Nonsense! Hopefully Linus can learn a thing or two about freedom from this (not holding my breath).
    [ Reply to This ]
      Isn't it ironic?... (Score:0)
      By Anonymous Reader on 2005.04.12 3:36 (#110304)
      "What Larry is _not_ fine with, is somebody writing a free replacement by just reverse-engineering what _he_ did."

      That is just pure hypocracy. No one does more reverse engineering, than those who restrict it with their licenses. Real networks did it on Apples DRM, and Microsoft even has howtos on their site!. So to you Larry; Stop crying and wake up and smell the coffee... ;)

      "Larry has a very clear moral standpoint: "You can compete with me, but you can't do so by riding on my coat-tails. Solve the problems on your own, and compete _honestly_. Don't compete by looking at my solution.""

      Lets just call it a standpoint. Calling it a *moral* standpoind, is maybe taking it a little too far... ;) There is absolutely nothing dishonest about reverse engineering. Its a perfectly acceptable, and sometimes even nececary method for interoperability for instance. But before this nonsence goes too fare, maybe we should remember that if he didn't use BK, the license ofcourse doesn't have a rats a** effect on what he can and can't do!... ;)

      "And that is what the BK license boils down to. It says: "Get off my coat-tails, you free-loader". And I can't really argue against that."

      What the license says is more like, stop trying to compete with us. The BK license is among the most absurd, in the unfree software paradigm. And thats even with some tough competition... ;)
      [ Reply to This ]
        not a tragedy (Score:0)
        By Anonymous Reader on 2005.04.12 5:40 (#110312)
        There are many usefull proprietary tools not available to Open Source community. BK is just another example. It is something one learns to live with, eventualy.

        DG
         
        [ Reply to This ]
          Its about time. (Score:0)
          By Anonymous Reader on 2005.04.12 9:02 (#110323)
          Tridge was absolutly within his right to reverse engineer the format. (Not that its a very complex format). This problem has been fortold from day 1. At some point people want control over their data. BK was trying to keep it locked away. Sorry, but that kind of bussiness is going to die in the long run. The software was never a gift. It was a hight interest loan and Tridge got tired of paying it.
          [ Reply to This ]
          Did Torvalds really try to stop Tridge? (Score:0)
          By Anonymous Reader on 2005.04.13 1:45 (#110389)
          McVoy claims Torvalds tried to stop Tridge from developing his free software BitKeeper replacement.

          If that is true, it is shameful. Yet another reason why Torvalds should not be looked upon as a leader in the free software movement. Go to him for technical information and guidance on the inner workings of the Linux kernel, but not on the (these days, frankly, more important) political matters in how a social movement ought to be run.
          [ Reply to This ]
            I was wondering... (Score:0)
            By Anonymous Reader on 2005.04.13 6:53 (#110403)
            whether Larry had kidnapped Linus' wife and children in order to make him say those things?
            [ Reply to This ]
              Took ball home (Score:0)
              By Anonymous Reader on 2005.04.13 15:26 (#110430)
              McVoy simply took his ball home.

              Just as Charlie Brown keeps repeating his mistake with Lucy taking away the football at the last minute and therefore Charlie Brown doesn't learn his lesson, Linus had enough sense to use Richard Stallman's GPL along with the GNU toolset, and enabled GNU/Linux to survive Microsoft's onslaught.

              Yet, Linus in choosing BK, didn't learn his lesson just like Charlie Brown. The football has been pulled away because someone owns the football and permission is needed to use it, even though the rest of the gang had a large hand in helping to assemble and improve the football.

              Had Charlie Brown used a football which the gang had created under Richard Stallman's license, Charlie Brown & gang would be able to continue using that football, with someone else besides McVoy holding it.

              Some people get it the first time, some people get it after you explain it a few times to them in simple terms, some people can't get it, and some people refuse to get it because it may show that Richard Stallman is correct.

              Any guess which applies in this case?
              [ Reply to This ]
              Advertisement

              © Copyright 2005 - OSTG, Inc., All Rights Reserved
              About NewsForge  •  Privacy Statement  •  Terms of Use  •  Advertise  •  Contact Us
              IT Product Guide --Enterprise IT Product Reviews.
              Add our feed to your site