X-Message-Number: 10488 Date: Mon, 28 Sep 1998 22:45:31 -0700 From: Peter Merel <> Subject: Y2K: Many Hands Make Lights Work Olaf Henny writes, >>cold fact is that 80% of these projects are going to run late. > >Peter, these technology projects are research projects, which >explore NEW ground. This can have unpredictable quirks and >setbacks. No, that's not correct. Check http://www.standishgroup.com/chaos.html for some typical stats. Less than 20% of regular everyday projects, no research quirks involved, are delivered on time. Folks who wonder why might start with Fred Brooks' "Mythical Man Month". >The correction of the Y2K is a (-n at least to some) >well known procedure, which "only" requires hard work, blood >sweat and tears, a process, much more predictable than the >development of a new software program or an even smaller chip >with even greater capacity. Sorry Olaf, again, this isn't correct. Y2K projects often present unique challenges that are far from IT business as usual: 1) Finding the damned chips. Most manufacturing and distribution facilites have grown over more than a generation and contain bits of equipment that are unknown to present day manufacturers and/or are stuck in inaccessible places like sewers, satellites, gas pipes, in the bowels of oil rigs sunk on the bottom of the sea floor, and so on. Some chips with the same numbers on them actually contain different internals. Some general purpose chips that don't use dates for their external functions still use them internally. Some chips without their own date functions obtain and mishandle dates from external sources. Some chips have "time dilation" malfunctions that don't turn up for weeks after rollover. 2) Coming up with replacements. Some of these dud systems were supplied by long dead businesses, or have no source code or development environment still existing. Some are supplied by businesses that, so far, are either unforthcoming with information or with replacements. How many? The New York MTA - the subway system - reported last week that of 60% of its suppliers refused to respond to requests for compliance information. Of those that did respond, less than 25% could assure compliance or replacement. Often coming up with a replacement for these systems is not a patch but a whole new development with the usual risks of new development. 3) Testing installations. Most successful technology projects alot over 50% of their efforts to testing. It makes no difference whether you're trying to integrate replacements or new technology; things never plug together smoothly or without unanticipated failure modes. Many new bugs will be introduced by remediation efforts quite apart from the ones that were already there. But I agree, the basic work in a Y2K remediation is straightforward. The techniques for doing it are well understood by a great many engineers. The scope and logistics of the job, and the problems above, however, make estimating dates of completion still more speculative than for a regular IT project. >I am quite sure these utility companies will be able to scare up >a couple of old-timers, who still have not forgotten how to throw >a switch by hand and how to start up a starter motor, and with it >the main generator. Oh, it won't be quite a efficient as our >computerized system now is, and most certainly lead to some >brown-outs here or there, but it won't be the catastrophe you are >foretelling. I'm not fortelling; I'm trying to figure out where I'm mistaken. All help in that regard very much appreciated. But as to running on manual ... not only are the old-timer operators not being re-recruited, but the knobs and levers they used to use to do the operation are no longer there. There are no more manual switches or shunting-yards for the trains. All the huge buildings full of manual telephone exchanges manned by well-drilled operators have been torn down. Modern networks, furnaces, generators, pipelines, etc. are run by highly redundant SCADA systems that make decisions with millisecond tolerances - and the hardware is built to require those tolerances. We've vastly increased our productivity through all this automation, but without the automation we're effectively hamstrung - that's why people buy these expensive control systems in the first place. You could no more run modern industrial plants by hand than you could replace your car's engine with a horse. >Who says, that these coal fired plants will be allowed to cool >down completely, before they are fired up again? From what I hear from my power-industry cronies, they cool much faster than they heat. But perhaps you're right, perhaps they'll get an immediate restart. And then go right back down again; these things don't trip because there's nothing wrong with them. They have to be fixed - and the time taken to do this is really the point here. A devastating illustration just occurred back in my homeland. Two blokes were working at an Esso gas refinery down in Victoria. No one's quite certain what they were working on, because whatever it was blew them to bits, and the plant along with them. Now the trouble is that this plant was also a hub for the gas distribution network. Hubbing is profitable in distribution networks of all sorts, electricity included, because it decreases infrastructure costs. You see this whenver you fly in the US - you fly on a little infrequent plane to a hub, then on a big frequent plane to another hub, then back to a little infrequent one again to your destination. So as of Friday the whole state of Victoria is without gas. It's early spring down there - cold at night. They have no hot water for showers. They have no bread because the bakeries are all gas fired. The restaurants too - this at the height of their tourist influx for the football grand final. But heavy industry has been hardest hit - 50,000 workers have been stood down since Friday. Estimates for a fix range from two weeks to over a month, with rationing continuing for six months. All this caused by just one little plant failure. You can't just turn a failed plant back on again - it has to be repaired. Repair always takes time, and in that a whole network of dependent plants just shut down. >Subtract from that the fact, that the rate of obsolescence of the >governing software is extremely rapid, and a huge portion of it >has been replaced since Jan. 1st, 1997 (I cannot imagine, that >any reputable software manufacturer was not aware of the Y2K >problem by then), and you can drop a huge chunk of all industries >from your problem list. No, you're thinking of PCs. Embedded systems aren't replaced unless they fail, and they don't often fail. We're talking about the lights and security and thermostats and industrial controls. 90% of all computer chips go into these embedded systems, not PCs. Not to say the PCs don't have their own problems - they do - but the advantage to them is you can install software fixes. Not so with embedded systems - they're controlled by read-only memory - firmware. You can't patch them - you have to replace them by hand. >Banks and other mortgage providers have long dealt with dates >reaching well into the next century. Defense related industries >as well as manufacturers of durable goods have not only delivery >deadlines, which bridge the turn of the century, but most of them >made or still are making damn sure, that their suppliers and >shippers right down the line are Y2K compliant. Examples Please! I'd love to see such excellent reports! I've heard some very nice sounds coming out of the financial sector, and I have no reason to doubt them - our own Perry Metzger has provided some of these. But Defense and manufacturing? Here I've seen only gloom. More relevant still would be any happy faces you can draw on the utility industries. But I'm not picky so long as there's facts here - or at least URLs. >Sure, they are >not all there yet, and there will be a last minute panic, but you >doom and gloom will not happen, because in the end every business >will be motivated to cover their own but. The market says so The market is a damn fine thing. An informed market is the best way to organize anything, I certainly agree. Unfortunately, many of the utilities I'm worrying about are government run. And many commercial infrastructure businesses seem to either be ill informed or asleep at the wheel. Nor does the market reward honesty: just last week Quaker Fabric, one of the Northeast's largest upholstery suppliers, reported that it expected its revenues to dip 10% below previous estimates due to Y2K efforts. The market responded by dumping stock and slashing its value 42% in a single day. What do you think that motivates other businesses to do? >Peter, I don't know which "source" you are quoting on that one, >but you better throw it into the trash and put a very heavy lid >on it. It tends to take credibility away from all your other >concerns. Olaf, I'd love to trash my spilled-wax-armageddon scenario. I just don't have any idea why it won't happen. Without water, phones, electricity, traffic lights and so on, I can't imagine what's to stop it. If you can, I really want to hear you. >While my comments pertaining to computer correction are based on >no particular subject knowledge other than a life time of >experience of human procrastination and their penchant for >finally pulling through in a crunch to safe their hides >(economically speaking), as a veteran of well over 1000 power >blackouts, from 1945 through 1951 in East Germany, I am a bit of >a expert on that subject. I can assure you, that during all that >time I have not heard of anybody setting his/her house on fire >because of it. Admittedly the news reporting at the time was not >as efficient as it is today, but in my small hometown of 12,000, >the grapevine would have reported any incidents. By the >accounting of your source half of the country should have burned >down. Did you have working fire engines during that time? Water pressure? Did this happen in megalopolises like the big US cities, or just in the villages? What was the maximum length of time of the blackouts? Were people used to handling candles and oil lamps? >Awh, Peter you have seen too many action movies. That's true, and I apologize to the list if I'm a little shrill. I don't relish these prospects - they frighten me. To calm down I need to find some happy facts. So far, I've not hit on them - anyone who has, I'd much appreciate whatever pointers you can provide. Peter Merel. Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=10488