Ask the Advogatos: why do Free Software projects fail?

Too many OSS hackers have no formal training/background in *engineering*. Many hate to do boring but necessary stuff like documentation, design, and regression testing. Also, they are too green (as a rule) to know how to *invent* things; they can only copy what they've seen done a thousand times before -- witness the legions of FTP, IRC, and news clients. Add to that the dangerous ego-bloat prevalent in the Linux community, and you've got real problems.

This is the true "dark side" of Open Source: not all hackers are created equal. Many programmers are simply *bad*, and need much more training before they become adequate. Yet much lousy code makes it into Linux because no one feels empowered to "just say no".

IMO, the Linux Standard Base would be a good place to implement refereeing: vetting code for correctness and cleanness before it is allowed into the "standard" distro. (This would also eliminate a lot of stupid security exploits.) But you can bet that a lot of hackers would protest when their crapware is kicked out of the distros and say that the initiative is "fascist".

It's a hard thing, but it badly needs doing.

I think something like FreeBSD's revision tracking system might help some projects, large ones especially. People can see when changes were made, what was changed, and why. So if "lousy code" does make it in to a project, it would be a little easier to see what was intended and hopefully fix things.

This might not be as useful for a project with just a handful or developers, but if lots of developers contribute, then it might make things easier. Also, if the project takes a turn that a lot of people don't like, they can go and get the source of whatever version they want and work from there. This would prevent people from having to start from scratch if the goals of a project they were contributing to changed radically.

It's difficult and time consuming to bring a piece of software to a "finished" state. It's a lot more work than most anyone is willing to admit.

Seems to me it is pretty obvious why Free Software or any software projects fail. It's hard work. Or am I just kooky?

And if the sheer workload isnt enough, though in personal problems, political problems, leadership problems, and the mother and root of them all: financial, corporate, and monetary issues. fun aint it?

If you want long endless eye glazing semantic jousting about licensing, development methodolgy, personailty traits of developors, paranoid delusional conspiracy theories about any given vendor, or the moral superiority of one or more user communities, I'm sure someone else on advogato can meet those needs.

What does failure mean in this context? I'm tempted to take the view that failure means a project that benefitted no one. I don't think that happens very often. Even the ghost towns are there, if nothing else, to teach the danger of false expectations.

I consider my mpEDIT project to be dead, but it got to be one chapter in somebody's book, was used in a college course, and was picked up briefly by the NCSA. Along the way it helped people learn programming. Even if it never got to be a widely used tool, I count it as a measured success.

Two anecdotes:

1. There was a GPL project once to create a Mathematica style numerical programme - It never even got beyond the design stage. When I looked through the mailing lists there were tons of people all giving their opinions on how the system should be designed, but no-one ever took the inititative to start writing something. As a result everyone eventually lost interest in a project that was clearly going nowhere.
2. I have often considered contributing to a project, but the code was so poorly written (Read: poorly documented/commented, excessively long functions, meaningless or unintelligible variable names, and so on ad nauseum) that it just wasn't worth my time to decipher someone else's crappy code to make the small change I wanted to.

1. When starting a project you need to actually write something before announcing it to discourage the hangers on who have no intention/ability to contribute from flooding your mailing list with indecisiveness.
2. Everyone should be using Knuth's Web system of literate programming to lower the barier for those who want to contribute to your project.

Rob Pike, in his presentation, "Systems Software Research is Irrelevant," encourages work on component-based applications, but I don't think that he would agree at all with Miguel's fawning over MS COM. Pike writes:

There has been much talk about component architectures but only one true success: UNIX pipes. It should be possibile to build interactive and distributed applications from pice parts.

(I think he considers the "plumbing" architecture in Plan 9 to be something like the next stage of UNIX pipes.)

Wrong article... (How did I do that?)

As a practical issue, beyond my what is failure folderol, I agree with mettw that code is the bottom line. I've seen projects get off to the wrong foot, with an empasis on who is managing what, rather than who is coding what. A tool that emphasizes the code base, and spotlights code contributions, might help things stay productive.

I feel that a great fault with many Open Source projects is that they aren't managed well. The initiator of a project doesn't need to necessarily know how to code him or herself, but he or she does need know how to gather, and organize people to accomplish a common goal.

Fortunately, SourceForge is helping out a lot by having hosting for Open Source projects. This gives developers the resources to create a community forum for discussing the project, and keeping it's development under scrutiny.

Personally, the projects I've been in have succeeded, I feel, because of two reasons:

1. First, I am the sole developer; this means I organize, code, and document the project; unfortunately, if I tried to delegate out the responsibility, I would more likely fail, because I may not have the greatest management skills.
2. The second reasons my projects have succeeded is likely that they are small, easily-graspable projects. The goals I set are accomplishable, and I can forsee from the beginning how the project could come together. Helpful in this is the fact that I use a high-level language, Perl; it allows me to develop rapidly, keeping everything in my head at once.

I don't know ftobin, maybe you can show some examples of sites that got into trouble having code but no management. I had the opposite experience. People would email me and offer to suck my 40 thousand lines of code into their project so they could manage me. At one point I joked that I could lay those guys from here (California) to New York. I'm tempted to point to one such site, still on the web ... but I'll leave it anonymous. From that page:

TODO

Establish the feature set for the editor. We need your input, so please feel free to view the features we have considered and send me your comments. We also need to recruit programmers, create a design, write code.

IIRC, it has been that way since sometime in late 1998.

Concerning the quote that higb gave us, I would say that there was definite lack of management going on in such a scenario. It appears that the project in question was poorly managed in the sense that they tried to create a project without any codebase; or they bit off more than they could chew. This is what a good manager needs to make sure doesn't happen.

Managing projects means thinking things through, and making sure the work can and does get done. It is not necessarily in conflict with or exclusive from creating code; rather, making sure the code is created is part of being a good manager.

mettw has the right of this. Most projects fail on startup or near enough to that time through marketing style dross. This app will be incredible, super this, super that, a powerful burst of synergestic energy will sweep away the shadows and we will march into the new era. Seeing as we've got 4 lines of code somewhere along the line someone must write the rest, I know, the other free software hackers will do it.

You got to be focused, and you got to take things one very small step at a time, grandious goals are doomed to failure, did linus hop out and say "ta-da heres a plan for a fully featured kernel with diagrams and neat annotated graphs of future progress, I havn't written a thing yet, but hey lets do it", far too many of these aspirational projects seem to have taken the plot of some sort of movies where the main character calls his friends or sends out a call and this vast army of misfits/ordinary folk/ewoks and succeed in overthrowing the system/baddies/empire because their hearts are pure and they have a half baked plan.

No code, no success. Keep it simple, create informal milestones, "I got this vague semiworking kernel, neat eh check it out", the next small achievable thing to do would be to ...

One achievable step at a time and some serious motivation by the person who started it, you has to want this thing. You have to write it, not your army of willing minions, sure you might get a few patches here and there, and even accrete a group of hackers whose contribution outnumbers your code, but going into the project with that attitude is inane. The Gimp authors did not head into their project with "We're going to write a supreme photoshop competitor and along the while write a Motif killer as a sideeffect", you increment towards it. Proving that your tiny group has the wherewithall to actually achieve something constructive, and then you go on from there.

Firstly, a definition. I believe that a project has failed if it never garners any users or a useable version of the project is never released. A project that is finished but could be improved and has more than one user is successful in this context.

I believe that projects like Berlin or GGI failed because of a lack of political nouse on how to handle the interaction between the established power base - and before they had a single line of code. A project I'm working on, reiserfs, may fail for this reason as well. You must exist with your neighbours and bring them along for the ride - both Sun Tzu and Machiavelli both agreed on this point. Berlin, GGI and reiserfs all fill a void that must be filled, but maybe not by these particular projects because they offended the sensibilities of the narrow-minded establishment.

Some projects never even get off the drawing board simply because it is too hard to write them without either a dedicated team of clever people with some form of an idea of what they are going to achieve. This is why the Gimp succeeded (and Koffice will) and there's still no open source Visio or project management software out there, even though I and lots of others need both programs. These sort of projects need either lots of people or they need a few core employed people with innate or learnt project management and software engineering skills.

Some projects choose inadvisable or just plain wrong tools. Some projects are impossible to maintain, some projects fill voids required a few years ago and are filled in by technology advancement (gopher servers anyone?). Some projects lose interest by their members once they become stable (pnm2ppa is one of these - it's worked good enough for me for the last six months, which is why there's been no patches since then by me, even though it's been finished by others in the meantime).

Good or heavily used projects reach critical mass and are nearly self- perputated - see NetBSD or OpenBSD for a project or two of the living dead. The people I know who are involved in NetBSD (including one core member, who's my best friend) are really really good. Great software engineers, but their visibility in the open source world is near zero. Is this a failed project? I think "maybe" is the answer. There's plenty of NetBSD users, but they have no real visibility in the open source world, particularly at a commercial level. NetBSD are about to get their first commercial distribution (Wasabi), some 8 or so years after Linux. I think that this may have been a contributing factor.

But basically it comes down to a lack of time (most of us are volunteers), a lack of interest, and lastly a lack of skills - including political skills which causes project failure.

NetBSD is already a complete operating system. It doesn't have that huge hole called "everything but the kernel" that's missing in Linux.

Maybe that's something a successful software project needs. Clearly defined areas of doubt and uncertainty.

Or on the other hand, NetBSD is part of the BSD development world. The BSD "commercial distributions" are FreeBSD and BSDI.

Like the Apache "commercial distributions" are things like Netscape's server.

This brings me to what I think is one of the biggest problem for many free software projects that has the potential for greatness, it is not lack of management or coders, but obscurity. The 'obscurity/lack of fame' many good free software apps suffer from leads to few new users -> few new developers -> disencouraged origianal author.

Another problem is lack of knowledge by the coders on a program other systems that perfom the same task etc, which means that lot of work goes into reinventing the wheel for each project. The lead developer or project manager probably should spend a little more time before starting to code, looking around to see what is already available.

I will agree that obscurity is a major problem. One of the major causes of obscurity is that there is so much crappy software that people are unwilling or unable to seperate the wheat from the chaff. Something needs to be done to make good free software available to people.

In theory freshmeat.net helps make free software available. In reality it makes it available for mockery. By ephasizing recentness it creates a push for people to get every minor release out on freshmeat. Maybe we need a goodmeat.net, where only good pieces of free software are listed, categorized and with much comentary.

The gnome.org software map could be this, but it is not. It, like many linux distributions, suffers from including everything submitted so as not to offend. And it still doesn't have a search button.

Point: If such a site does not already exist, there needs to be someplace where my mother can go and get the software she needs, without having to worry that she will get crappy software.

Not that I am going to do anything about it. I have too many projects of my own.

A `goodmeat.net' system could be set up on advogato. All you would need is to allow the projects to be rated like people are.