Extremely Successful Software

Some software projects are orders of magnitude more successful than others. Here is the story of one such project and why things went so well.

by Kenneth Dickey

Imagine this. You need to build a complex, highly reliable software product that works at high speed on multiple networked server platforms and integrates with many kinds of server environments and databases. You have a working prototype but no production code and want product install CD-ROMs in under a year. The product is not totally defined, but you know that your business will change and you will have to feature evolve very fast in a highly competitive market. You are selling to Fortune 1000 companies, which demand 24/7 reliability.

That is just what we did this last year at a Sydney startup called Memetrics. The core software was developed on time, under budget, with a small, 6 person, development team. The software runs on Windows NT, Solaris, and Linux platforms under Apache or IIS servers, makes use of Oracle, SQL Server, or PostgressSQL, has a web based administrative interface, integrates with J2EE, COM and other technologies.

The code is very fast, highly reliable, very maintainable, and extremely flexible. The cost of change is low. The risk of change is low. The system was built for rapid feature evolution.

This was the best code quality and the lowest development cost I have seen in 20 years of commercial software development.

Where did we go right?

Best People

No matter what business you are in good people make a difference. We went after and hired world class people.

We hired from all over the world. Hiring from Australia, we typically brought someone in on a 3 month contract. It's like dating before marriage. It worked well for us. Not many people want to commit to a job half way around the world, but would you turn down a chance to visit Sydney on a 3 month contract? Once you are there, the job is interesting, the people are great... That's how they got me interested!

We felt it is important to go beyond "hiring a skill set" to "inviting someone to be a member of a community." Communities can hold together in times of rapid change. Working with the right people matters.

Best Practices

One advantage small companies have is that they have to think lean. Being lean means that you don't have fat to slow you down. The fat is called muda, , which is Japanese for wasted effort, uselessness, futility. The key idea is to remove everything that is not required, every effort that does not contribute to your ultimate goals.

We did not have funding for a large team, so we used a small team. That meant less chance of miscommunication, and it meant that we had to be hyper productive.

We used the XP (eXtreme Programming) process. XP uses the "voice of the customer" to define a set of use cases or "user stories" that define what the software must do. Programmers estimate the implementation costs; the customer sets the order in which stories are implemented; the programmers implement the stories in that order. The customer can add or remove stories and change priorities at any time.

XP is a very lean process. It removes several classes of muda.

XP removes miscommunication. In developing the user stories (use cases), a common vocabulary is developed that let the Research, Management, Marketing, Sales, and Engineering groups talk about the product in a common language. Priorities are explicit and visible. We created a dictionary. Understanding standard terms means less time is wasted in communication.

We used good User Interface principles to give visibility and control--what is happening and what can I do about it?. Given that we had people in the US, UK and Australia, we put everything on-line. The major milestones, the stories and priorities, the dictionary, weekly engineering status, interactive issues management and so on was all put on secure web pages with sorting and searching facilities. This allows field sales and consulting teams to see that features the real customers are asking for will be available at a particular date, raise issues, request new features and communicate their urgency. Because of the common user stories language, everyone can ask for and discuss things in ways that everyone understands.

XP removes project inertia. Because stories and priorities can change at any time, engineering is set up as a service organization to Business Development. As business conditions change, development can turn on a dime to meet new conditions. There is no "grand plan" to be updated and communicated. Update the stories, update the priorities, and everything required has been communicated. Engineering always implements the highest priority stories.

XP removes integration risk. Extreme Programming requires that all non-trivial functions have automated test cases and that all tests pass 100% of the time. The basic strategy is to always have a running system. One does "the simplest thing that will work" and grows a system rather than having a bunch of lines on a project chart all converge into a single high cost, high risk bubble labeled integration.

We released a new system version in 3 week cycles, each time with more stories implemented and each passing 100% of the tests. The advantage of frequent releases is that it gives a continuum of development with many points where the business can decide, "We have enough value here; we can release this version." This reduces management risk.

XP lowers the risk and cost of change. In XP, everyone owns the code and is empowered to change anything in the system to make it better. Programmers work in pairs, so all work is "code reviewed" by construction. Because there is a continuous testing process, any breakage is detected and fixed early. Because programmers re-pair frequently, people are cross trained and have general knowledge of the entire code base. This knowledge is retained by the development group even as people cycle through a project.

XP reduces "startup time" for new team members. In our world, a new team member starts writing code to implement function of the highest strategic value to the company the first week on the job. Contrast this with typical development processes where people spend the first week getting their accounts set up and reading the design documentation.

Pair Programming increases productivity. It may seem contradictory that two people together can write more and better code than two people writing independently--but it works!

Because of the pair programming process, good implementation technique is communicated. The weak get strong and the strong get stronger. Our experience was that even the best programming gurus write even better code in pairs. Because there are two sets of eyes and minds and because people have to explain themselves to one another, there is less wasted work--fewer defects (a.k.a. bugs) and fewer "wrong turn" implementation choices.

XP removes wasted time. Extreme programming is intense and focused. Our developers typically got to work at around 8 AM, read email and web news for about 10 minutes, then started writing code. Aside from a brief lunch, code was written all day. Typically, code was checked in at the end of the day--which of course passed 100% of the tests--and people left at 5 PM.

What is wrong with this picture? People write software that has tight deadlines, deliver on time, and go home at 5 o'clock?!? Well, that's right. That's just what we did. If you don't waste effort, you can go home at 5 o'clock. As a matter of fact, you cannot work at the focused, sustained pace we did if you don't.

XP removes meetings. People are continuously communicating to produce code that implements the most important user stories. Because of the XP and stories process, there is less misunderstanding and less need for "administrative meetings." Programmers spend virtually their entire time designing and getting code written. When issues arise, people communicate to make decisions, then act. Most traditional development process meetings don't contribute to getting code written. They are muda.

We didn't have time for many meetings. We had to get the code written.

Best Technologies

When choosing implementation technologies, don't go for parity, go for strategic advantage. [See sidebar Steel and Toothpicks].

The goals and constraints of a project define the shape of the design space. We took care to match implementation technologies to the shape that needed to be filled. This was done using engineering cost/benefit analysis to determine which implementation technologies gave the most leverage and slowed us down the least.

What gives a best match implementation technology? In our case it was not being up to date. Using the latest technology typically means that engineering effort is spent on tracking technology as it evolves. It also implies a higher learning cost than picking up training materials, well developed practices, and people who already know the technology. It makes sense to do this in areas where the newest technologies give competitive advantage in your company's value added area. In our case, we did not see value compelling enough to overcome the costs.

So, what technologies give great results, execute fast (compile to native machine code), and run cross platform with the minimum of platform specific code? Read on. There was an additional requirement--high reliability. We were dealing with customers who typically expect processes to run for 2 or 3 years.

You can't do that! In the XP process you write the tests before writing the code that implements the function. In this case a test was written on the client, which queried the server. The test, of course, failed as the corresponding function had not yet been written. Then the server code was written, which implemented the function. The client test was run again and this time passed the tests. Let me paraphrase an interaction between a member of the development group who was new to the programming language technology we chose and a more senior member

NEWBIE: You can't do that!

GURU: Why not?

NEWBIE: You didn't stop the server. You didn't rebuild and relink the code!

GURU: Too right! You have to be able to update the code on a running server. If you can afford to fail, you can implement the server with any technology. We can't afford to fail.

Some systems think failure is ok. The analogy here is visiting your doctor with a hangnail. The doctor says, "Yes, you have a hangnail." He takes out a gun, shoots you, and sends a note to your mother to have another child!

We needed to build a system that allowed us to say, "Hey, wait a bit! Let's consider some alternatives."

OK, enough hints. Doing fundamental analysis can lead to non-standard choices. We used ANSI Common Lisp. Lisp has a stable language standard, good rapid development tools and compilers, adaptors for common databases, and people who know the technology at a high level. We used CORBA network pipes for similar reasons. COM, J2EE, and every major development language has a CORBA interface. We needed to plug into virtually any client's eBusiness technology, and CORBA let us do that. We spent zero engineering time chasing developing technologies, 100% of our engineering resource in our value-added space.

Interestingly, when we checked we found that a number of Fortune 1000 companies were using Lisp in critical applications such as on-line banking and airline reservation systems. So it seems that others in our target market had independently come to the same conclusion. It is good to be able to say that "we use the same technology our customers use."

We chose a lean company, Xanalys, as our Lisp vendor. Xanalys gave us stellar support. When we turned in a bug report we typically (90+%) had a patch the next business day that fixed the problem. We wrote a set of tests, and they gave us bindings for an object/relational layer, which let us keep all persistent state in whatever relational database we were using. Rapid turn support allowed us to write Lisp code rather than dealing with the different variants of the SQL database language--which in turn made us more productive.

Other cost reductions. While we tested and delivered on NT, Solaris, and Linux, we did our development on Linux. As a lean startup, we were very sensitive to licensing costs. We found that, in addition to lower costs, Linux has excellent development tools, was a stable development platform, and was more internet-secure than our second choice platform.

We removed everything that got in the way of being fast.

What did I learn from this project?

Will Rogers said: "Its not what we don't know that hurts, its what we know that ain't so."

I learned to hire good people and trust them. I learned to trust the fundamentals. And I learned that a lot of what conventional thinking deems work is really muda, waiting to be removed.

Kenneth Dickey (kend@memetrics.com) is a Memetrics alumnus based in Portland, Oregon.

Sidebar: Steel and Toothpicks

Use standard engineering cost/benefit analysis and choose the right technologies.

The analogy is that you are a civil engineer and a client asks you to build a major bridge. After discussing traffic capacity and such things, the client says, "Oh, by the way, I'd like to constrain the implementation technology to steel and toothpicks."

ENGINEER: Toothpicks. That's an interesting choice.

CLIENT: Yes, we have these engineers working for us that know all about toothpicks and bonding agents and build amazing chairs and tables out of toothpicks. And they are already trained in toothpick technology!

ENGINEER: That's something to consider. Given the scale of this bridge, I think that I might be able to recommend some alternate technologies, like steel and concrete, which might scale better. If you let me give you a cost/benefit analysis, which includes retraining costs, we can talk about what technologies give the best overall benefits within the total economic context. It is always good to consider alternatives.

CLIENT: Well, OK, but the toothpicks are the latest technology. I'm not sure I can get engineers to want to retrain.

ENGINEER: If they are professionals, they will learn the appropriate technologies to use. If not, perhaps they should go back to building chairs.


eXtreme Programming: www.eXtremeProgramming.org
Memetrics: www.Memetrics.com
Xanalys: www.Xanalys.com