Faults & Failures - October 2000 IEEE Spectrum

Does anybody really know what time it is?

For much of the past year, self-setting clocks on some videocassette recorders (VCRs) around the United States were showing the wrong time. In portions of the San Francisco Bay area, the clocks were 24 minutes fast. In other parts of the country, they were 1, 2, or 3 hours slow.

Affected VCRs could not be programmed to record television shows when the codes published in program guides were used, and users trying to program their VCRs manually had to recalculate program times by taking the time errors into consideration.

The hunt for the source of the errors is a quirky tale that shows that for even simple systems to work correctly, every link in the chain that connects them to the user must be checked and double-checked. The story also is ironic, in that the "flashing 12:00" of VCRs came, over the years, to be used as a symbol of technologies made too complex. But the autoclock feature that fixed the VCR's flashing 12:00 added problems of its own.

The problems started a year or more ago, but took a long time to be discovered because, it turns out, while thousands of users were affected by the errors, few noticed or cared. (Perhaps the rest were accustomed to VCR-clocks that blink 12:00.)

"If we get an inordinate amount of calls to our customer service center about a specific problem, we are flagged that something is wrong," said Tom Hantson, Panasonic Corp.'s national product manager for VCRs, in Secaucus, N.J. "Even though this had been going on for as much as a year, that didn't happen. Perhaps that means that not as many people are using their timers as we had thought."

It took an enterprising newspaper columnist, David L. Wilson, of the San Jose Mercury News, along with an annoyed consumer complaining about his VCR problems, to bring these clock faults out of the dark and identify, eventually, one source of the failures.

In July, Wilson heard from a reader who had returned two VCRs because of autoclock problems--the clocks on both VCRs ran 24 minutes fast. Did Wilson know what was wrong?

Wilson called the manufacturer's technology support line, and was told that the problem must be a bad timing signal from a Public Broadcasting Service (PBS) station (which turned out to be the right answer). He then called a few certified repair shops. Their response was it must be a chip problem. He then talked to executives at the VCR manufacturer's headquarters.

"They were totally freaked out," Wilson told IEEE Spectrum. "They were thinking they had a chip manufacturing problem, and they wanted to get their hands on the VCR. So they got the customer to swap it for their top-of-the-line model--which also displayed the time 24 minutes fast.

After Wilson published the story of the unsolved VCR clock mystery, he quickly heard from hundreds of readers who were having the same problem with just about every brand of VCR manufactured. It was clearly no single manufacturer's fault.

So whose fault was it?

To trace the source of the error requires first an understanding of the workings of the now ubiquitous autoclock (not many VCR manuals tell you this). Autoclock was developed by PBS, Alexandria, Va., under a grant from Sony Corp., with the goal of making VCRs easier for users--no more blinking "12:00." Whenever an autoclock VCR is turned off, it scans the TV channels, starting with channel 2 and going up. It is looking for a digital time stamp, sent by most PBS stations during the vertical blanking interval, the time during which the electron beam tracing the screen rasters is turned off, having reached the end of the screen and having to return to the beginning of the screen.

This time stamp is coded as part of the extended data service (XDS), a standard used in Europe to send teletext information. Once autoclock finds a time stamp, it uses the data to set or correct its clock. (Some VCRs give users the option of choosing a local PBS channel from which the time signal should be extracted, but most users never figure this out.)

Wilson, meanwhile, was calling a number of the PBS affiliates in the Bay Area. When he contacted Channel 54, San Jose's KTEH, which is the primary PBS broadcaster for San Jose and much of the surrounding area, he found out that the station had not had a chief engineer on duty for more than a year. During that time, it appears, something happened to the XDS clock. The newly hired chief engineer, Carl Rieg, tested the clock signal and confirmed that it was 24 minutes fast. But not knowing what piece of equipment generated the signal, he could not immediately adjust it.

While Rieg was trying to track down documentation that would enable him to fix the problem, Wilson was still gathering information. He called PBS's central engineering facility in Alexandria, Va., and talked to Andy Butler, chief engineer. Butler thought that he ought to talk to Rieg himself.

"Rieg," Butler told Spectrum, "had come from commercial television. He didn't even know that the XDS clock existed, much less that it needed to be checked periodically. And there were no manuals at the station documenting the device." (In some stations, XDS maintenance has become computerized, with a computer hooked up to the XDS box and to the Internet. The computer regularly checks the Internet for the correct time and updates the XDS box accordingly.)

So in a series of phone conversations, PBS engineers walked Rieg through the fix. "Most of the conversations were us on this end trying to describe the box to him as he walked around looking for it," Butler said. "Once he found it, it took 10 minutes to fix." The XDS unit is a black, single-rack box located somewhere in a television studio or among the associated transmitter gear. The fix involved attaching a computer through the serial port of the XDS and resetting the time. (Pretty much any computer will do, since "talking" to the clock requires just using a terminal emulation program.)

Meanwhile, the PBS engineering staff in Virginia was working on another autoclock-related problem. Local PBS stations around the country were getting occasional calls about clock errors from savvy VCR users who knew that PBS was responsible for sending out the signals that set autoclocks. In some places the clocks were consistently 1 hour off, in other places 2, in other places 3. And the problems, which PBS began hearing about in 1999, seemed to be increasing.

"It took almost a year from the first time we heard about it until we figured out what was going on," Butler told Spectrum. These kinds of things are difficult to sort out, he said. "For example, if your TV picture looks funny, is it because of a problem with your cable system, the broadcast station, or your TV? Even that can be hard to determine."

PBS engineers identified the source of the 1-, 2-, and 3-hour problems by comparing notes with their colleagues at several PBS stations around the country. This identification occurred at about the same time the San Jose problem was being solved. The culprit was Fox Broadcasting Corp., which, over the past year, had been signing new affiliates and rapidly increasing its market penetration.

Fox, it turned out, was including XDS clock signals on its network feeds from Los Angeles. The network had planned to distribute local XDS clock signals, like PBS, and was going to use the Los Angeles signals to synchronize the local clocks. However, some Fox affiliates were neglecting to install XDS equipment locally. At these stations, the Los Angeles time codes were being broadcast unaltered. Since in many television markets the local Fox affiliates are assigned to lower channels (frequently 2 or 5) than the local PBS affiliates, VCRs that started scanning for a clock signal on channel 2 found the Fox signal before the PBS signal and reset their clocks--to the correct time in Los Angeles.

Again, as with the KTEH autoclock, whereas finding the source of the Fox error was time-consuming, solving the problem was not. Fox, in effect, turned off its clock, stopping the transmission of XDS signals altogether.

Telda S.Perry, Editor