Postmortem:

Sierra Studios’ Gabriel Knight 3:
Blood of the Sacred, Blood of the Damned

 

By Scott Bilas

 

http://www.drizzle.com/~scottb/publish

 

Background

Gabriel Knight 3 is a traditional “Sierra style” murder-mystery adventure game, telling its story through a complex nonlinear mix of dialogue trees, scripted sequences, movies, and puzzles. Most of us should be familiar with this kind of game: it’s paced at the speed of the player, and involves a lot of “inspect the monkey” and “use the banana on the monkey” type of interaction to move the story forward. GK3 differs from its predecessors technologically in a variety of ways, but most important is that it moves the genre to full 3D.

GK3 offers a freely roaming camera that lets the player go where they please, and zoom in on whatever they like. This isn’t just a gimmick – this single feature radically changes the game, making it more like an interactive movie, and less like an interactive comic book. As we found out, moving from 2D to 3D is not just 1/3 more work, it’s more like three times as much work. It affects nearly all aspects of the game, including both the design and the engine. It’s not as simple as just drawing your actors and environments with polygons instead of sprites. Suddenly you have to start worrying about camera angles and dramatic effects that never were possible or necessary in 2D, at least not without resorting to a prerendered movie.

Part of GK3’s design was also to give the player the option to turn off cinematic camera cuts during dialogue sequences. The idea was that the player could be the director and choose their own camera as the action was unfolding. This had a serious and very expensive effect on the art: it meant that artists could take no shortcuts with their animations. In a prerendered movie, an animator has full control over the camera, and can avoid bothering with anything that’s outside of its view. This saves a lot of time. In GK3, because the player may, at almost any time, decide to take control of the camera, they would be able to see the action from any angle they pleased, and go “behind the curtain” so to speak. Because of this, the animators would need to make sure that the entire scene could be viewable and look good from any angle. This increased their workload by an order of magnitude.

Previous Sierra adventure games, including GK1 and GK2 as well as the Space Quest and Leisure Suit Larry series, had been built using the “SCI” game engine. SCI was developed and maintained by Sierra Oakhurst and, for a variety of reasons, Sierra Northwest (the division I worked for) had decided to stop using it. This single decision probably affected the project more than anything else – it meant that GK3 was to be the first adventure game Sierra had built completely from scratch in a very long time. There was a huge amount of work to do!

The project had some important things going in its favor early on. Sierra hired an experienced engineer, Jim Napier, who got the game started by developing its “G‑Engine”: a 3D rendering, sound, and animation toolkit that provided a low-level foundation to build the game upon. After its completion, Jim unfortunately had to leave the project to start on the fledgling SWAT 3 as its lead. The G-Engine was generally a successful part of GK3, providing a stable base for the game and being easy to use and understand. The team was also able to reuse some of the tools and concepts that SCI had provided, for example the content database and lip-synching tools.

Despite these initial advantages, the project ran into problems almost from the start, and I’ll get to those in detail in the “what went wrong” section. Overall, however, I believe that the majority of the problems on GK3 stemmed from two things. First, that the team was building a new game engine and most of its related tools and content development processes from scratch. And second, that they severely underestimated the time, cost, effort, and experience base required to do this. The starting team was not at all equipped for this task.

In the early days of the project, engineering was living in a magical dream world. I really can’t find any other way to explain it, because when I joined the project in early 1998 (GK3 had been in development for over a year and a half), it was scheduled to ship at the end of the summer. This was absolutely and obviously impossible because the code was a big old mess. It was a hacked-up version of a sample app that Jim had written to demonstrate the G‑Engine. This should send shudders up and down the spines of any experienced engineer: sample code was being used in production code! Malignant growths were added haphazardly whenever a new feature was required, making the game completely unstable and difficult to maintain. This problem fed on itself, growing worse over time. One example of this sort of problem was the game’s horrendous startup time. The file and resource system, while sufficient for a sample app’s limited resource set, completely fell over when faced with GK3’s tens of thousands of files and hundreds of directories. The game would take over a full minute just to start up and go to the title screen!

Most of the game’s non-art content was being hard coded – any story sequences (even a simple dialogue exchange between two characters!) were actually written as C++ code. This was a nightmare for a couple of reasons. First, engineers were creating content instead of working on the engine. Engineers generally suck at creating good content, and tend to be very slow at it besides. GK3 was no exception. And second, it required a recompile to change the tiniest detail, such as which line of dialogue was played or what animation to play to open a window. This made the content development process unbelievably inefficient. Artists would potentially have to wait weeks in order to see their work integrated into the game. This resulted in engineering resenting artists “chucking art over the fence” and probably inspired similar resentment on the art side.

If GK3 was to ship at all, all of this had to change. And so it did.

What went right

1. The engineering rearchitecture

A month after I joined, we decided to rebuild the game engine, and a little while later I took over as its architect. We spent a couple of weeks in roundtable design sessions with engineering advisors from other projects (including Jim from SWAT 3) and used an old-fashioned “CRC card” design technique to hash out the systems we would be needing and how they would go together. I thought all of this went pretty well, though slower than most of us liked. Once we started coding, things really got moving. The application core was rewritten in a weekend, and then individual systems (user interface, scene abstraction and configuration, font rendering, the console, etc.) were developed and integrated as fast as we could manage.

In originally proposing the rearchitecture, we had gained the full support of upper management, specifically Mark Hood, the general manager of Sierra. They really had no choice, considering that the only other option was to cancel the project, but I think it’s important to recognize what a risk they were investing in and give them credit for believing in us enough to give it a shot. Throughout development, Mark was always 100% behind us, and never wavered in his desire for us to ship an AAA title of the highest quality. Despite our lack of experience, we mostly delivered what we set out to accomplish.

Unfortunately, I don’t think a lot of the team members outside of engineering ever really understood why we rebuilt the game. Looking back, I wish we had spent a little more time explaining the dire situation the engine was in to them. Though it took months to redesign and rebuild the game engine, a time that understandably confused and frustrated everybody, it improved productivity a couple orders of magnitude and made shipping the game possible. The new engine was stable, flexible, and, though it still had architectural problems (due to our inexperience), it finally worked properly and had good performance.

2. Data-driven engine

GK3 is a very content-heavy game, shipping on 3 CD’s and including over 800 megs of compressed non-movie data, which was mostly texture maps, MP3 dialogue and music, and animations. There are thousands of lines of dialogue and almost as many logic rules tying it all together. During the rearchitecture we went with a data-driven approach, putting as much as we possibly could into text files so that non-engineers could create and maintain content. We were very happy with the results.

One of the project’s major successes in this area was its flexible scripting system called “Sheep”, which used a C-like language and was implemented using a simple compiler and p-code interpreter. The compiler was built with our old friends lex and yacc from the Unix world. Originally designed just for the game’s animated sequences, we also ended up using the Sheep engine for custom rules processing, event handling, resource packaging, scene configuration, debugging, the development console, and even for a little bit of testing automation.

I can’t stress this enough to developers: build a scripting engine, even if you don’t think you need one. Make it generic enough that it can be reused in as many places as possible. I think many engineers are scared off from building one because they think it will take too long to implement or that it will execute scripts too slowly. This was certainly the case with the original GK3 engineering team. I’ve found these fears to be completely unfounded – a scripting engine will pay for itself many times over, and can be easily optimized if necessary to approach the speed of C++ code. Also, don’t invent a new language. Pick a programming language that one of those Teach Yourself Something Useful in 21 Days books exists for, and you can buy copies for your scripters if they don’t already know the language (though this wasn’t necessary for GK3). This will save you some documentation and training time, which is a significant advantage over a completely proprietary system.

Another benefit of Sheep was that, when combined with our redesigned file and resource manager, we were able to dramatically cut down the lead time of art integration. Consider the old process that used hard-coded C++: an artist would need to wait for the next build (at best) before they could see their work integrated into the game. We reduced the days or weeks of waiting that used to be necessary down to minutes or hours, and almost completely removed engineers from the picture. Under the new system, an artist could check in their work and have one of the scripters integrate and demonstrate it immediately on the existing build.

We added clipboard support to GK3’s console to support pasting Sheep code directly into the game. The scripters could then alt-tab away from the game, grab a section of code to test from their text editors, alt-tab back into the game, and paste it into the console and see the results right away. This small feature was an example of the lengths we went to in order to support the developers. With these kinds of features in place, content got poured into the game at a blinding pace.

3. The design

The design was a major success of the project and deserves special mention. GK3 would have simply fallen over and died if we had had a less experienced and professional designer than Jane Jensen to create it. Throughout the entire development process the one thing that we could count on was the design. It was well thought-out and researched, and had a good, entertaining, engrossing story. Best of all, Jane got it right well in advance – aside from some of the puzzles, nothing really needed to be reworked during development. She delivered the design on time and meticulously maintained it as the project went on.

4. Audio

Audio designers and engineers rarely get the credit they deserve, and often end up taking second place to the people drawing the pretty triangles. Fortunately, GK3 is an adventure game, and as such it lives and breathes on the ability of its dialogue and supporting audio to bring the player into the story. Many reviewers picked up on the great audio they found in GK3, and often rated it as one of the best parts of the game (at least, the ones that didn’t have a silly personal issue with Tim Curry cast as Gabriel Knight). From a development point of view, audio content was something we could always rely on. David Henry, GK3’s composer and lead sound designer, had it all done long before we actually needed it, and as such was able to spend more time on polish, adding a ton of small details to the game. And in stark contrast to the other parts of the game, integration and maintenance of the audio content went as smooth as glass.

5. A talented, dedicated core

GK3 never would have shipped without the heroic efforts of critical developers in key places across the board – art, scripting and logic, engineering, design, and testing. These people took over various parts of the project on their own initiative and kept pushing until things were done, and done right. The loss of any one of these superhuman individuals would have severely crippled the game’s chances of making it out in 1999, if at all. Among the crowd, two names are especially worth mentioning. Halfway through the project, we picked up Jessica Tams, who took over the content and put it in order. She wrote nearly all the scripts and logic for the entire game, getting it done on schedule and somehow making it work despite all the problems with the engine (more on these problems in the next section). Ray Bornstein came onto the project with a year to go, and put the animations in order, making a realistic schedule and having the animators actually meet it.

What went wrong

1. Team casting problems

When someone is placed in a role in which they don’t belong, I call this being “badly cast”. Many of the problems with GK3 resulted from developers being badly cast in their roles, usually because the project requirements were so heavily underestimated. To get an idea of the casting problems we had, we went through a total of two producers, three art directors (spending the last year of the project without any), and three project leads, with the producer being forced to take over the project lead role towards the end.

This was an ambitious, massive project that required experienced engineers to develop, and the original team was simply not up to this task. GK3 was initially built from members of the Shivers 2 team (one of the last games built with SCI), and had practically no 3D experience in any department. Engineers under the venerable SCI engine were basically scripters – putting them in charge of building a game engine from scratch was like feeding them into a furnace. To make things worse, developers that were in over their heads didn’t ask for help, which gave management a false sense of progress.

2. Severe morale problems

Hundreds of books and articles have been written about this, and here we have yet another postmortem listing it under “what went wrong”. Soapbox time. To managers everywhere: morale is one of those icky personal political things that many of you like to avoid dealing with. You need to understand that a development team is not a factory churning out content and code. As Peter Sellers might have said, the team is a garden of creativity that requires regular watering and sunshine in order to build strong roots. Loyalty is not something that is easy to come by. The job market is very competitive – your best developers will simply leave and go work for somebody else if they aren’t properly maintained. On GK3, there was a serious lack of love and appreciation that sustained throughout the project. Recognition of work other than relief at its completion was very rare, lacked sincerity, and was always too little, too late.

Internally a lot of the team believed that the game was of poor quality. And of course, all of the web sites and magazines predicting that “adventure games are dead” only made things worse. Tim Schaefer’s brilliant Grim Fandango, though a fabulous game and critically acclaimed, was supposedly (we heard) performing poorly in the marketplace. The rumor mill was running overtime generating a lot of talk about GK3 losing money, especially considering our high burn rate. I attribute this to poor communications. What we heard directly from upper management was always upbeat and positive about GK3; they were behind it 100%. Yet communications like this rarely filtered down to us unless it was bad, and generally it was of the form “we’re late and running out of money”.

The low morale resulted in a lot of sendoff lunches for developers seeking greener pastures. GK3 had a ridiculous amount of turnover which, though it generally turned out for the better, never would have been necessary had these people been properly cast or well treated in the first place. Over 45 total developers were churned through GK3 (with an average standing team size of 15-20), and just a few months after shipping, only seven still remain at Sierra. Strangely, the opposite also happened – we lost some developers to Sierra’s mid-1999 housecleaning layoffs, yet they stayed on until the end of the project, which was a prolonged drain on morale.

After a certain amount of time on a project like this, morale may sink so low that the team will develop an incredible amount of passive resistance to change of any kind. They can get so tired of the project, develop such a hatred for it, that they will avoid doing anything that could possibly make it ship later. This was a terrible problem for the last half of GK3, inspiring one former GK3 developer to call parts of the team “cancerous” in this respect. It resulted in a lot of work that we weren’t very proud of being shipped in the final product, work that should have been fixed but nobody wanted to take the time to do correctly because we were so focused on trying to get the game out. I don’t think anyone on the team is directly at fault for this, and honestly don’t know what we could have done to correct the problem.

3. Schedule problems

Engineering never had an accurate schedule. The ones we had were so obviously wrong that everybody on the team knew there would be no way to meet them. Our leads often flat-out lied to management about progress, tasks, and estimates, and I believe this was because they were in over their heads and simply reacted badly. As a result, upper management thought the project was going to be stable and ready to ship long before it actually was, which precipitated prolonged crunch times. More frequent and honest communication within the team would have avoided a lot of this.

GK3 had no real milestones, which undermined our ability to track progress. A milestone is a critical procedure in any iterative incremental development process. It’s like the pounding drum on a dragon boat – every two months or so, the entire team should sync up and see if they’re on track, and adjust future milestone deliverables and estimates accordingly. GK3 had some early milestones, which were improperly applied, and ended up becoming smoke-and-mirrors demos instead. These eventually went away and were replaced with two simple and unofficial milestones – beta and release to manufacturing. In the push to ship the game, we simply forgot about milestones (because “we’re almost there!”), put the blinders on, and worked like mad.

The crunch mode is probably a reality of any project, but you should not go into one unless the light at the end of the tunnel really is in sight. The GK3 team was pushed into crunch mode three separate times, each time thinking that we were almost ready to ship! Most of the last year of the project we spent in constant ship mode, meaning that even small breaks like vacations, conferences (or even taking off nights and weekends!) were seriously looked down upon as time that the team could not spare to lose. The overtime didn’t help anyway – the project didn’t move any faster or go out any sooner, mostly due to the resultant drain on morale killing productivity.

Because of the high turnover, GK3 always had a high percentage of developers new to the team. Faced with this mandatory overtime, they understandably felt it was unfair to be “punished” to pay for problems caused by the original team or things that they felt management had brought upon itself. GK3 became a black hole that sucked in many developers from other projects, often at the serious expense of those projects. Artists were shifted off the team to cut the burn rate, and then pulled back on later because there was so much work left to do. Art finally got on track in the last year of the project (thanks to Ray), but engineering never got a real schedule together, and as a result we were nearly always late in our feature delivery.

4. Engineering problems

When we rebuilt the game engine, we wanted to retain as much of the original code as we could to get the game up and running again as soon as possible. With the exception of the G-Engine, this was a big mistake. Nearly every one of the systems that we kept caused us no end of problems – they were badly designed, buggy, and inflexible, and should have been redesigned as well. Some of these systems never once worked correctly throughout the lifetime of the project, and had to be hacked around by the content developers to get the game to ship.

Specifically, we had serious problems with the “fidget” system (used by actors when idle or when involved in dialogue), the actor’s walker, the vertex animation system, and the conversation and dialogue systems. All of these regularly failed and were regularly “fixed”, with each bug fix introducing new bugs, usually in the form of hidden time bombs. The engineers responsible for these systems became very defensive about the problems with them, and usually ended up blaming artists and scripters or even other engineers for the cause. Management, thinking that it would save time, would often encourage content developers to hack and work around the problems rather than have them fixed properly. We should have ripped these parts of the game out and rebuilt them, rather than continually attempting to hack at legacy flawed design. It would have saved a lot of time and hard feelings.

We also had a lot of other problems that were mostly out of our control. Most notable were the technical difficulties with the hardware Direct3D drivers provided by hardware vendors for their 3D graphics cards and sound cards, but this probably won’t be news to any 3D game developers. These problems were generally of the form of features implemented improperly or inconsistently, or just outright bugs that caused system crashes or hangs. We also wasted a few weeks trying to add copy protection. During the final push to ship, we repeatedly attempted to make Macrovision’s SafeDisc product work with GK3. SafeDisc has a set of special (and we felt completely unnecessary) anti-hacking measures that got in the way of the game’s execution. It heavily affected performance, taking the frame rate to a third and adding strange intermittent freezes of several seconds while moving the camera. After getting nowhere with Macrovision’s engineering department, we decided to ditch SafeDisc and roll our own (which took less than a day to do). This entire process wasted several weeks of our time, and frustrated all of us all the more because we were ready to ship except for this one issue. Lesson learned: if you are required to use copy protection, don’t put it off until the last month, especially if it’s SafeDisc. We weren’t the first game to have severe problems working with SafeDisc and probably won’t be the last, so if you’re using this product, be sure to do your homework and try it out far in advance of shipping.

5. Art and (more) engineering problems

One of the most expensive mistakes a team can make is ramping up art before the engine is ready. This happens often at larger game companies because developers need a place to go after shipping their most recent game. Unfortunately, this was a serious problem with GK3 – artists were brought onto the game while the (original) engine was in development, long before a stable engine was available. They were creating content for an animation engine design that was untested, building up enough inertia that we ended up keeping the design, which we later discovered to be seriously flawed.

GK3’s animation system is vertex-based, meaning that a model’s individual vertices are animated. Even using some creative compression methods, this is very expensive in terms of memory usage. Contrast this method with a typical skinned skeletal animation system, which only requires that the bones be animated. The worst thing about this system was not the memory usage, however. It was the impact on content creation, and the repercussions of this requirement were not fully realized until the art team was ramped up and churning out models and animations.

GK3 animations are completely coupled to the meshes of the models that they affect. Once an animation is exported from Max, the models it involves cannot be changed in any way, otherwise existing animations created from the old versions of those models would break. Vertices can’t be added or removed, and texture maps can only be tweaked, not remapped. Changing a model would require re-exporting every single animation that it’s involved in. This is very expensive in terms of an animator’s time spent in tedious and repetitive time-consuming labor, but also was often not possible. The source assets (original Max files) had a way of getting lost, and usually ended up stored on backup discs as artists left the project.

The end result of all this was that once a model was created, it could never be changed. GK3 shipped with a lot of bad art that the entire team was dissatisfied with, yet had no choice but to use. An example of this was the Mosely character (sometimes not-so-fondly called “T-Rex man” internally), whose arms were about a foot too short. This really affected morale, and had a lot of us thinking the game was of poor quality: “the art is bad and there’s nothing we can do about it” was the bottom line. Nearly every attempt to change a model was met with the response “well that would require re-exporting all its animations which would take too long”.

What we should have done was to just stop the presses and reboot the art. When we first recognized the severe problems with the vertex-based animation system, we should have rebuilt that part of the G-Engine and replaced it with a very simple skeletal animation system (as the SWAT 3 project later did). We should have also thrown out all existing art and rebuilt it. In retrospect, that would have saved us a lot of time, given us better performance, and made development proceed more smoothly.

Out the door

Gabriel Knight 3 was an extremely ambitious project combined with an extremely inexperienced team. With all that went wrong, you would have every reason to believe it would never be finished. Despite all that was messed up with our development process, we somehow managed to get GK3 out the door and ship it in time for Christmas 1999. Best of all, the game works, and it works well, against all odds. This is a tribute to the testing skills of our QA lead and team – though severely understaffed, they did a great job ensuring that the game that we shipped was high quality. There are no crashes and no hangs in GK3 – Sierra will not be issuing a patch.

This isn’t to say that GK3 has no problems, just no problems that we had any real control over as developers. To the best of our knowledge, every single critical problem with the game is caused by either bad hardware drivers (sound or 3D card), which can always be fixed by upgrading drivers, or CD-ROM problems due to our oversized CD copy protection. Also, the first disc that shipped in some retail boxes was difficult to read on some CD-ROMS, apparently due to duplication problems. This can also be worked around, as the same file exists on the second disc and can be copied to the hard drive.

GK3 is certainly not the game it could have been had we been able to knock down a couple of those “what went wrong” issues early on, but it is a high quality, entertaining game that has been well received by adventure gamers. And that’s something we can all be very proud of.

Data box

Full-time developers: over 45 total, averaged 20 at a time.

Contractors: 3 total.

Budget: originally well below $2 million, final budget over $4.5 million.

Length of development: almost 3 years.

Release date: November 1999 (over a year late).

Platforms: Windows 95, 98, NT, 2000.

Hardware used: lots of expensive stuff that quickly became obsolete.

Software used: 3D Studio Max, Character Studio, Visual C++, SourceSafe, CodeWright, VTune, MKS Lex & Yacc, Photoshop, special Sound Guy magic.

Technologies: DirectX, Bink Video, Standard Template Library, LZO and zlib (free, open source compression libraries).

Bio

Scott Bilas was a senior engineer at Sierra Studios and the technical lead on Gabriel Knight 3. He is currently working at Gas Powered Games, where he spends his time inserting needles into haystacks. Scott lives in Seattle and thinks everyone should get out of their cars and onto their feet. He can be reached at .