X-Message-Number: 10488
Date: Mon, 28 Sep 1998 22:45:31 -0700
From: Peter Merel <>
Subject: Y2K: Many Hands Make Lights Work

Olaf Henny writes,

>>cold fact is that 80% of these projects are going to run late. 
>
>Peter, these technology projects are research projects, which 
>explore NEW ground.  This can have unpredictable quirks and 
>setbacks.  

No, that's not correct. Check http://www.standishgroup.com/chaos.html
for some typical stats. Less than 20% of regular everyday projects, no 
research quirks involved, are delivered on time. Folks who wonder why
might start with Fred Brooks' "Mythical Man Month".

>The correction of the Y2K is a (-n at least to some) 
>well known procedure, which "only" requires hard work, blood 
>sweat and tears, a process, much more predictable than the 
>development of a new software program or an even smaller chip 
>with even greater capacity.

Sorry Olaf, again, this isn't correct. Y2K projects often present unique
challenges that are far from IT business as usual:

1) Finding the damned chips. Most manufacturing and distribution facilites
have grown over more than a generation and contain bits of equipment that
are unknown to present day manufacturers and/or are stuck in inaccessible
places like sewers, satellites, gas pipes, in the bowels of oil
rigs sunk on the bottom of the sea floor, and so on. Some chips with
the same numbers on them actually contain different internals. Some
general purpose chips that don't use dates for their external functions
still use them internally. Some chips without their own date functions
obtain and mishandle dates from external sources. Some chips have "time
dilation" malfunctions that don't turn up for weeks after rollover.

2) Coming up with replacements. Some of these dud systems were supplied 
by long dead businesses, or have no source code or development environment
still existing. Some are supplied by businesses that, so far, are either
unforthcoming with information or with replacements. How many? The New York 
MTA - the subway system - reported last week that of 60% of its suppliers
refused to respond to requests for compliance information. Of those that 
did respond, less than 25% could assure compliance or replacement. Often 
coming up with a replacement for these systems is not a patch but a whole 
new development with the usual risks of new development.

3) Testing installations. Most successful technology projects alot over 
50% of their efforts to testing. It makes no difference whether you're
trying to integrate replacements or new technology; things never plug
together smoothly or without unanticipated failure modes. Many new bugs
will be introduced by remediation efforts quite apart from the ones that
were already there.

But I agree, the basic work in a Y2K remediation is straightforward. The
techniques for doing it are well understood by a great many engineers. 
The scope and logistics of the job, and the problems above, however, 
make estimating dates of completion still more speculative than for a 
regular IT project.

>I am quite sure these utility companies will be able to scare up 
>a couple of old-timers, who still have not forgotten how to throw 
>a switch by hand and how to start up a starter motor, and with it 
>the main generator.  Oh, it won't be quite a efficient as our 
>computerized system now is, and most certainly lead to some 
>brown-outs here or there, but it won't be the catastrophe you are 
>foretelling.

I'm not fortelling; I'm trying to figure out where I'm mistaken. All
help in that regard very much appreciated. 

But as to running on manual ... not only are the old-timer operators not 
being re-recruited, but the knobs and levers they used to use to do the
operation are no longer there. There are no more manual switches or 
shunting-yards for the trains. All the huge buildings full of manual 
telephone exchanges manned by well-drilled operators have been torn down. 
Modern networks, furnaces, generators, pipelines, etc. are run by highly
redundant SCADA systems that make decisions with millisecond tolerances - 
and the hardware is built to require those tolerances. We've vastly increased 
our productivity through all this automation, but without the automation 
we're effectively hamstrung - that's why people buy these expensive control
systems in the first place. You could no more run modern industrial plants 
by hand than you could replace your car's engine with a horse. 

>Who says, that these coal fired plants will be allowed to cool 
>down completely, before they are fired up again?

From what I hear from my power-industry cronies, they cool much faster
than they heat. But perhaps you're right, perhaps they'll get an 
immediate restart. And then go right back down again; these things
don't trip because there's nothing wrong with them. They have to be
fixed - and the time taken to do this is really the point here.

A devastating illustration just occurred back in my homeland. Two blokes
were working at an Esso gas refinery down in Victoria. No one's quite 
certain what they were working on, because whatever it was blew them
to bits, and the plant along with them.

Now the trouble is that this plant was also a hub for the gas distribution
network. Hubbing is profitable in distribution networks of all sorts, 
electricity included, because it decreases infrastructure costs. You see
this whenver you fly in the US - you fly on a little infrequent plane to
a hub, then on a big frequent plane to another hub, then back to a little
infrequent one again to your destination.

So as of Friday the whole state of Victoria is without gas. It's early spring
down there - cold at night. They have no hot water for showers. They have 
no bread because the bakeries are all gas fired. The restaurants too - this 
at the height of their tourist influx for the football grand final. But 
heavy industry has been hardest hit - 50,000 workers have been stood down 
since Friday. Estimates for a fix range from two weeks to over a month, with
rationing continuing for six months.

All this caused by just one little plant failure. You can't just turn a 
failed plant back on again - it has to be repaired. Repair always takes 
time, and in that a whole network of dependent plants just shut down.

>Subtract from that the fact, that the rate of obsolescence of the 
>governing software is extremely rapid, and a huge portion of it 
>has been replaced since Jan. 1st, 1997 (I cannot imagine, that 
>any reputable software manufacturer was not aware of the Y2K 
>problem by then), and you can drop a huge chunk of all industries 
>from your problem list.

No, you're thinking of PCs. Embedded systems aren't replaced unless they
fail, and they don't often fail. We're talking about the lights and 
security and thermostats and industrial controls. 90% of all computer chips 
go into these embedded systems, not PCs. Not to say the PCs don't have 
their own problems - they do - but the advantage to them is you can
install software fixes. Not so with embedded systems - they're controlled
by read-only memory - firmware. You can't patch them - you have to replace
them by hand.

>Banks and other mortgage providers have long dealt with dates 
>reaching well into the next century.  Defense related industries 
>as well as manufacturers of durable goods have not only delivery 
>deadlines, which bridge the turn of the century, but most of them 
>made or still are making damn sure, that their suppliers and 
>shippers right down the line are Y2K compliant.  

Examples Please! I'd love to see such excellent reports! I've heard
some very nice sounds coming out of the financial sector, and I have
no reason to doubt them - our own Perry Metzger has provided some of 
these. But Defense and manufacturing? Here I've seen only gloom. More 
relevant still would be any happy faces you can draw on the utility
industries. But I'm not picky so long as there's facts here - or at least
URLs.

>Sure, they are 
>not all there yet, and there will be a last minute panic, but you 
>doom and gloom will not happen, because in the end every business 
>will be motivated to cover their own but.  The market says so

The market is a damn fine thing. An informed market is the best way
to organize anything, I certainly agree. Unfortunately, many of the utilities
I'm worrying about are government run. And many commercial infrastructure
businesses seem to either be ill informed or asleep at the wheel. 

Nor does the market reward honesty: just last week Quaker Fabric, one of
the Northeast's largest upholstery suppliers, reported that it expected
its revenues to dip 10% below previous estimates due to Y2K efforts. The
market responded by dumping stock and slashing its value 42% in a single 
day. What do you think that motivates other businesses to do?

>Peter, I don't know which "source" you are quoting on that one, 
>but you better throw it into the trash and put a very heavy lid 
>on it.  It tends to take credibility away from all your other 
>concerns.

Olaf, I'd love to trash my spilled-wax-armageddon scenario. I just don't have
any idea why it won't happen. Without water, phones, electricity, traffic
lights and so on, I can't imagine what's to stop it. If you can, I really
want to hear you.

>While my comments pertaining to computer correction are based on 
>no particular subject knowledge other than a life time of 
>experience of human procrastination and their penchant for 
>finally pulling through in a crunch to safe their hides 
>(economically speaking), as a veteran of well over 1000 power 
>blackouts, from 1945 through 1951 in East Germany, I am a bit of 
>a expert on that subject.  I can assure you, that during all that 
>time I have not heard of anybody setting his/her house on fire 
>because of it.  Admittedly the news reporting at the time was not 
>as efficient as it is today, but in my small hometown of 12,000, 
>the grapevine would have reported any incidents.  By the 
>accounting of your source half of the country should have burned 
>down.

Did you have working fire engines during that time? Water pressure? Did 
this happen in megalopolises like the big US cities, or just in the villages? 
What was the maximum length of time of the blackouts? Were people used to
handling candles and oil lamps?

>Awh, Peter you have seen too many action movies.

That's true, and I apologize to the list if I'm a little shrill. I don't
relish these prospects - they frighten me. To calm down I need to find 
some happy facts. So far, I've not hit on them - anyone who has, I'd 
much appreciate whatever pointers you can provide.

Peter Merel.

Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=10488