Introduction.

BIAMAX S.A is a national dealer of passenger cars, trucks and special vehicles in Greece. Its main businesses of vehicle importation and distribution, spare parts management and distribution and service-warranty management as well as its human resource, financial and accounting management require the processing and storage of a large number of date information.

The first applications were first built in the early 80's, on an IBM midrange system that is still being used today. All software since then has been produced in-house. The early applications, some of which survive today, suffered from bad design, lack of documentation and inadequate testing. These early mistakes and the involvement of people with organizational skills in the MIS team resulted in the formulation of a software engineering methodology that is being used since for the development of all software. This methodology has been critical in the success of the Y2k project.

One central point to the above methodology is the detailed documentation of each program. The documentation is produced at the design stage and updated during the life of the program. The above documentation describes the fields/files used and updated, the screens, the validations, the program logic and anything else relevant to the program. It is very close to the programming language but in a way independent of it. Although the above documentation is handwritten and not produced and maintained by any automated software as a CASE tool it has proved simple and very successful in quickly identifying software problems and impact points.

Another point in our methodology calls for keeping the features used in our programs standardized. Also all applications have been written in the same programming language (RPG III) and there has been an effort to use certain standards during coding. These last two facts vastly simplified the work of the programmers and allowed them to adopt a methodology of working with all programs in a standard way that speeded their work considerably.

The Y2k problem was first identified in 1989, during the development of a major application (module). Because of the high level of integration among applications it was decided that design and implementation would become too complicated in order for the new module to use 4-digit year fields and the rest of the modules to use 2-digit year fields. Only files used for storing statistical data were built with 4-digit year fields. It was decided to go ahead and build the critical applications that were needed by the company at that time and leave the Y2k project for a later date. Finally in 1995, during the redesign of our legacy payroll application into a human-resource management module it was decided that it was the time to go ahead with the Y2k project.

Along with fixing the Y2k problem we decided to fix another problem, similar in scope, that we had identified in earlier years. We have always built our applications to be able to handle multiple companies on the same computer system which is a feature not yet used and never really tested. During the study of the y2k problem we discovered that the multiple company feature did not work properly in a large number of programs. This only came to prove that it is a safe bet to assume that any untested code does not work. Fixing this "bug" would be a second y2k project so we planned to fix it now along with the date problem.

Personnel involved.

The people involved in the project comprise the entire MIS department of the company:

a) Two consultants:
They are responsible for designing and applying the software engineering methodology used and have designed, managed and tested all the programs built with the above methodology.

b) Three programmers:
One of the programmers has been with the company since the early 90's and is the most experienced. The other two programmers were hired one year ago with very little experience and were trained in-house.

c) Two general support personnel:
They were partly involved in the project while attending to their usual everyday duties.

First Stage - preparation.

During the summer of 1995 one of the consultants spent 2 Months planning the project. The documentation of the database and all programs was gathered and studied. The work started with the database and all date fields where identified and recorded by database file. These notes would be used later to guide the programmers in changing the database. Then all programs where studied, module by module, for impact points. For each program needing changes notes where written outlining the impact points and the way to fix them. Handwritten lists where produced indicating what programs needed changes. These lists would be used as one of the checklists used later in the project. They also gave us a first glimpse of the size of the task that lay ahead. It consisted of 830 database files (171 physical and 659 logical files) and 5 modules consisting of 3,773 programs (1,201 RPG programs, 1,276 display files and 381 CL programs). It was a huge task and it was clear from the beginning that it would be the quality of the project management that would be the critical factor in the outcome of the project.

Second Stage - Inventorying - System cleaning.

The second stage of the project involved inventorying and cleaning of our computer system of old and unused software so as to start our main Y2k project with a clean, tidy system. This was necessary so as to have better control and make sure we would not leave out anything important later in the project. Like many other companies around the world we were faced with the unpleasant surprise of finding out that our system was not as tidy as we thought it was. Our software libraries where badly organized and a large number of forgotten files (old versions of source, object and data files) just lay there tying up a lot of disk space. At that point our disk usage had reached 92% and we where considering buying more disk space. Now, after the project has been completed, the disk usage has fallen to 56% (!) even though we are now using 4 digits, instead of 2, for the storage of the year throughout our database. We also found out that a few programs where not fully documented (usually old programs) and had to take time to document them.

The process of cleaning up our system took 3 months during which one programmer and one consultant worked on the task exclusively. Apart from cleaning our system we introduced new guidelines so as to prevent such a situation to emerge again in the future. So the cleaning part of the project was a prerequisite for starting the main y2k project but the results will also have a positive impact in building and maintaining future applications.

During the cleanup stage we produced lists of all the remaining software in our system. Using these lists we would later check the progress of the main y2k project (assignment of programs, compilation, testing). These lists also gave us an even better estimation of the project size. The database accounted of 350,000 LOC and the programs accounted for 1,350,000 LOC, totaling 1,700,000 LOC.

Main Project.

We decided that we would build and test the new software in a parallel system of software libraries on the same computer. When all data files and programs where ready we would replace our existing software with the new software. Then we would restore the company data in the new database files and the whole system would be ready to work. The problem was that restoring the company data in the new database files would have to be done by special programs that would also translate the 2-digit year to a 4-digit year (by adding "19" or "20" in the first two digits). Our first estimates told us that these translation-restoring programs would have to run for 5 days nonstop in order to move the 3 Gbytes of company data into the new database file. Because of the high integration among our modules it was crucial that the whole roll out would be made in one stroke. Initially it looked impossible to fit the whole task in one weekend but during the end of the project we finally found a software solution that speeded our system enough to complete the data transfer in 48 hours.

We also decided to freeze all other development work. So we "solved" the parallel change management problem by trying to avoid it as much as possible. We decided to commit the whole MIS team in the project in order to minimize the total risk. Before starting we made an effort to complete all the changes that were requested at the time but after that our sole consideration became the y2k project. On this we had to get the full commitment of the management first. Without it we could have been pressured to interrupt our work for something that the management might consider more urgent and this would be disastrous for our project. Nevertheless a few bugs that needed immediate fix did surface and we did have to do a little parallel change management. We tracked the programs involved closely and did extra testing on the features fixed so as to insure that the y2k compliant releases of our programs included all the changes. I believe that parallel change management cannot be avoided completely during projects that drag for months.

Having built the parallel system of software libraries we then transferred all the source code from the old software libraries to the new ones and where ready to start. First we fixed our reference files (database field directories used in IBM midrange systems) and then worked on the database files themselves. We compiled them with the new date fields and also took the chance to standardize any nonstandard source code. The above changes where checked by both the programmers and the consultants through printouts produced by the system. This was the easy part. Next we worked on the CL code (Control Language of the IBM operating system). We use CL code mainly for building our application menus. This was also fairly easy since our menus rarely contain date fields. We took the chance to check them for other problems identified in earlier years (such as the proper use of commitment control commands during the call of programs). After finishing with the CL programs we where ready to move into the RPG programs.

We started by fixing all the routines, and rewrote all the date routines. Then we started with the programs themselves. We worked by module (vehicles, spare parts, service-warranties, e.t.a.), completing one module before starting another one. The first few programs that we attempted to fix gave as a clue as to the kind of work that was needed. The programmers made a list of standard things to watch for and fix. Using this list as a checklist, the program documentation and the notes written for each program during the summer of 1995 the programmers worked their way through each program they where assigned.

Testing.

Normally when we build or change a program the programmer has to test the program first. Then one of the consultants has to do a second complete test on the program before it is released. This two tier testing method has proved to be very reliable because it is made by two people that have a different set of mind. However the situation now was different. The programmers had to make few changes in a large number of programs rather than build a whole new program. The time needed to fix each program was too little to allow for the programmer to comprehend each program in the context of the application. As a result it would take the programmer too much time to build the correct test data for testing the program. On the other hand the consultants who have designed all the programs have a much better understanding of each program and where able to test the programs almost as fast as the programmers got them ready. It was decided that testing would not be conducted by the programmers at all. All programs would be tested once by the consultants. A key to quick testing was the sequence by which the programs where tested. First we worked by module and secondly we prepared the programs in the sequence by which information is produced in the company. For example in the vehicle module we first prepared and tested the programs that create the vehicle orders. By running and testing these programs we created a number of orders in our database that would be used as test data for the next bunch of programs that process orders into arrivals. By running and testing the programs that create the arrivals we created test data for the next programs to be tested and so on. By that way we avoided to manually enter test data for every program and improved both the reliability and time of testing.

The fact that the programmers did not test the programs resulted in more bugs being found during testing and programs being sent back more often than usual. It also resulted in the three programmers producing more programs than the two consultants could test. As a result the programmers worked 8-10 hour days while the consultants worked 14 to even 18 hour days in order to complete testing in time and attend to the general organizational tasks of the project.

The part of fixing and testing the software took 4 months. During the last weekend of May 1996 the software was ready and the personnel involved in the project worked around the clock to replace the old software with the new and convert-restore the company data into the new database files. It was a well-prepared task that took a lot of discipline but updated the whole system in one stroke rather than piece by piece that would be much more complicated and prone to problems.

Aftermath.

During the next two weeks of operation we watched for any bugs that would be missed during testing. The whole company was notified to look for any irregularities and report them. A few bugs did surface in the first few days and where fixed.

We changed all the date fields to a 8-digit format in all our display and print files as well as in our database files. Of course there were cases (certain subfiles) where there was not enough room to accommodate a 4-digit year at which cases we used the 2-digit year format. But all input dates are now 8-digit. This approach looked simpler with regard to programming and testing. From the point of view of the users it took them a maximum of two days to get used to inputting the 4-digit year. In all date input fields we use a common validation that does not allow the input of a date before year 1900. Of course most of the people did show some dissatisfaction at the beginning but it did not last long. On the contrary I now believe that just looking at the 8-digit dates has provided enough proof to all non-technical people (from secretaries to upper management) that our y2k project has been successfully completed. I found no other proof as convincing as this. Maybe because one believes what one sees with his own eyes.

We also have external interfaces with supplier factories abroad. We import their EDI files in our midrange and special programs update our database files. All our EDI partners use the 2-digit year format so we use a certain routine to translate it into a 4-digit format by using the threshold of 85 to distinguish between the centuries. This threshold will not cause any problems for the kind of data we exchange. But this is not the real problem. What if one of our EDI partner does not become y2k compliant. Then there is no way that we can predict how his systems will react. The dates in his EDI files could be erroneous and our systems will have accepted a large number of wrong dates before anyone notices. And even if we notice in time we will have to interrupt EDI communications until the problem is solved. We can not afford that to happen. Therefore we are now in the process of getting in touch with our EDI partners on the y2k compliance issue.

This project has covered 171 physical files, 659 logical (index) files, 1,201 RPG programs, 1,276 display files and 381 CL programs, accounting for 1,700,000 LOC but it did not include certain legacy applications built in the early 80's that do not comply with the software engineering methodology presently used. These legacy applications are the payroll, bills receivable and fixed assets and fortunately consist of a few programs each. From these the accounts receivable are being fixed now by the programmer that coded it in the first place. The other two applications are due for replacement before they start having any y2k problems.

I have estimated the cost of our y2k project by using the current Greek market prices asked for the programming and consulting work-hours consumed. In dollar terms it turns out to be $0.30/LOC. But since the per capita income of the US is 3.5 to 4 times that of Greece the same project in the US would cost $1.05 to $1.20/LOC. From this cost up to 30% accounted for testing of changes unrelated to dates. Only 20% of the total cost accounted for actual programming work. The largest cost item was testing time. Of course the above costs must be judged in the light of the method used to complete the project.

Conclusions.

It seems that the level of organization in a information technology department is very critical in the success of a project entailing small changes in a huge number of programs and database files, such as the year 2000 project. In trying to make estimations of the resources needed for such a project (in $/LOC or people terms) the organizational foundation is the prime factor.

Christos J. Andritsoyiannis (xandri@ath.forthnet.gr) business consultant MBA, Walter A. Haas School of Business (UC Berkeley) BA, Athens Economics University

26 August, 1996

http://www.year2000.com/