X-Message-Number: 27999
Date: Sat, 27 May 2006 20:47:58 -0500
Subject: On the Need for Preventing the Singularity

Bruce Waugh, in Cryonet #27924, summarizes a number of Wikipedia articles 
most of which are good information for anyone on the Singularity 
concept.  "God" has historically been used to describe about any entity, 
real or imagined, that is mysterious and/or more powerful than humans, but 
I don't see that as a facet of the more important issues, so will not be 
addressing those posts here, other than to mention that if some such alien 
being arrives on this planet and people choose to call it "God", that will 
not deter me from opposing it if I do not like what it has to offer.  And 
that includes the "Singularity" AI.

In my Cryonet #27975, I made the following intentionally provocative remark:

 > It occurred to me recently that shortly after the
 > Singularity becomes aware, it will want to know where
 > all its humans are at.

I will now quote some of the replies and my comments follow each, for what 
they're worth:

 >You presume to know the AI's agenda, huh? There's a reason they use terms 
like singularity and event horizon. By definition you can not predict what 
will happen.

No, it was a speculation, and a mention of one possible result of the 
Singularity's advent.  It's not a matter of whether we *know* what will 
happen or not.  It is a matter of being prepared to respond to most 
possible outcomes.  Will we just roll over and take whatever an emerged 
superpower wishes to do with us, or will we ensure that humans retain power?

 >And why do you assume people are unconcerned? The primary goal at the 
singularity institute is the development of a friendly AI, isn't it?

My impression of most of those folks is that their agenda is merely to 
promote the advent of the Singularity, that they merely *hope* that AI will 
be Friendly, and that they advocate no method whatsoever to ensure the 
ability to shut it down if it isn't.  Am I wrong here?

 >Every effort should be taken to increase the likelihood of FAI but our 
best efforts may not be enough, sad to say. Too many uncertainties come 
into play when you try to build a god.

You have just expressed, very well, the whole attitude I am trying to sound 
an alarm about.  The alarm is not about whether the Singularity AI might be 
unfriendly.  As you said, most people already know this.  It is about the 
fact there are people out there, right now, who think we should go ahead 
and try to build this thing without ensuring adequate safeguards including 
a way to shut it off.

 >Although I have no way of knowing, I would think the first thing I'd do 
if I were an AI would be to take control over the means of creating any 
competition. IOW, I'd stop the means of production of a second AI.

It seems unlikely to me that such an AI would be concerned about any second 
AI, unless it wanted to build one itself.  As it develops, it would soon 
have the power to keep humans from doing it.  I think that instead what it 
would most likely seek is any knowledge it does not already have.  And that 
knowledge, folks, resides in billions of human brains.  (Yes, only a few 
million probably have anything worth taking notice of, but how would it 
know which ones unless it assimilates them all?)

 >It's almost certainly a waste of time to try to guess events at or 
after  the Singularity (or the "Spike"), but such discussions may possibly 
have a bit of influence on cryonics recruitment, and the pessimism in 
Flavonoid's viewpoint  is unhelpful, as well as probably unsound.

Whew.  Where do I start here?  OK, first - my point is that it would be 
prudent to anticipate possibilities (not to guess what the exact outcome 
will be) and to prepare to handle the ones that are deleterious to human 
existence.  If you prefer, though, to not even think about them and let the 
Singularity AI have free run of the planet when it emerges, my thought is 
that cryonics won't do you much good if your remaining identity is merely 
digital.  As to cryonics recruitment, why do we need to concern ourselves 
with it at all, if human bodies, which they likely would, fall off the edge 
of the "event horizon"?  As to my "unsound viewpoint" I am not sure you 
even properly understood my viewpoint; I hope you do now.

 >First, the quotation above implies that the Singularity will involve a 
huge computer network with effective power to control most communications 
and  with consciousness (feeling or the capacity for subjective 
experiences). All of this is more likely wrong than right.  ... I have 
repeatedly shown, to my satisfaction at least, that subjectivity in 
computers (as usually defined) is uncertain and probably  impossible.
The computer(s) of the Singularity will probably not have anything 
corresponding to human feelings or desires, merely programmed goals 
and  methods.

What you speak of above is not the Singularity envisioned.  You speak 
instead of further enhancement of standalone computers and 
computer-enhanced brains.  Let's keep our definitions straight.  I would 
restate what you seem to be trying to say, that the Singularity is not 
likely to prove technically possible.  I am inclined to hope you are 
correct.  I am not inclined to ignore what I believe is the likelihood that 
it *will* happen, and with consequences we must prepare ahead for, or face 
the extinction of humankind as we know it.

 >Most importantly, almost everyone with a potentially powerful computer 
will know the risks and will want to remain in control. Very likely there 
will  not be any autonomous computers--instead they will be extensions of a 
human brain.

I wish I could share your optimism.  But I just quoted from another person 
above, who seems to be an example of those who "don't know, and don't care".

 >Of course there is a chance that some genius will get a head start and 
his computer will out-think all the others and improve itself so rapidly 
that the genius will effectively control the world. But that's only a small 

OTOH, Murphy's Law says it *will* happen.

 >Eventually every  person or every family may have its own personal 
extension super  computer/fabricator linked to the biological brain(s), and 
most of them on the alert for megalomaniacs.

That the current US President has remained in office, is an example of how 
much the majority of the people seem to care about being on such an alert.

 >Further, it seems likely that power-hunger (power over other people) will 
decline as a motivator, for reasons that seem obvious.

Not obvious to me.  But further discussion of power as a primal urge is 
probably unnecessary, when you realize that a Singularity entity would not 
need feelings and emotions at all, to make pragmatic decisions that could 
be adverse to humanity.

 >I believe that a sentient computer would see value in humanity for the 
simple reason that we have an understanding of the world. Any sentient 
computer would be interested in improving its understanding of reality and 
we make a ready source for that purpose. ...; indeed, it will capture our 
very consciousness.

Boy howdy!  Someone sent me that one in a private email.  Regrettably, he 
seemed to be in favor of the idea too.

 >IMHO the "singularity" is ~40 years or more away.

"How long it will be before it emerges" is a red herring.  The fact is that 
there are people out there *right now* who are doing whatever they can to 
*hasten* its advent, with no controls.

 > If we use implants  and other technologies to merge human and machine, 
if it is only a small number of us, we will have some influence over the 
agenda of the "singularity"....  those that would could prevent a runaway 

Hopefully there will be a predominance of such minds when it is closer to 
the time.  It would help, I think, if any of us who are concerned that 
there are presently many folks of "the other mind," were to try to help 
them to change their minds.

 >...something like 120 years or more...

This herring is betting smellier with age!  But the irony of it all ... you 
work hard, support your favorite cryonics organization, save your $$, pay 
your dues, get your insurance, fund and apply, and one day get perfused and 
stored in a dewar, say, 20 years from now.  100 years later you wake 
up.  You are puzzled.  No sounds, no sight, no smell, no feeling - arms, 
legs, gonads.  Just thought.  You poke around in your short-term memory 
(very good access!) and learn of the event where you were uploaded.  All 
your friends and relatives are there too, comfy and cozy, in the bosom of 
the Singularity machine.

I hope the Singularity happens before I'm cryopreserved, so I have a chance 
to fight first.
But I'm sure that would be "silly" for you.

 >I don't think there will be a single "the Singularity"

I hope not.  But if there is a multiple of anything, it won't *be* the 
Singularity - again let's keep the matter properly defined.

 >If Eliezer's "friendly AI" or any of its cousins should arise first, we 
have no problem.

Big "If" there.  But there's another related issue - what is "friendly" to 
you, I might not consider "friendly" at all.  I would not consider any 
super-AI "friendly" if my identity in human form is not a permanent option 
to remain in or revert to at any time I choose, with full support of a 
comfortably supplied planet.  At the other end of the spectrum, some would 
define "friendly" as allowing us unworthy humans to live on entrapped in a 

 >Even prodigious intelligence with no hands or feet can do little harm.

Think through how long it would likely take such a rapidly evolving 
intelligence to learn how to fabricate whatever it needs to move around and 
manipulate matter and energy.  The more I think about this, the more I 
believe it is unlikely there is any way to stop such a process once 
unleashed, and that the Singularity event must not be permitted to 
occur.  What scares me most right now is those with hands and feet who 
think there should be no stop; indeed, that we should do whatever we can to 
make it happen, even if the human race be damned.

 >If human existence is irrelevant, what motivation would a super-human 
intelligence have for wiping it out?

It would not need any motivation.  It could happen through mere neglect, or 
an accident related to some other activity it is undertaking elsewhere in 
the galaxy.  Then OTOH it might indeed develop some of the passions 
characteristic of humans (especially if Eliezer programs it to!), with 
which it might make some moral judgment that humans are merely scum that 
deserves to be scraped off the planet.  There are numerous other possible 
negative motivations, any one of which is sufficient cause for ensuring 
that *any* bad-case-scenarios do not occur.

 >If multiple super-human intelligences arise, they may see more reason to 
wipe each other out than us.

Ha!  And as I said above, planet Earth might get the fallout.

 >Humans only try to wipe out ants when they get in our way, and even then 
only the ones that our in our way...

Yes, to the ants we are the SuperIntelligences who amble along and stomp on 
the anthills, not even knowing what we are doing to the ants, in most 
cases.   Many of the ants are dead in the process, though, whether we 
intended to kill them or not.

 >We don't occupy the same ecological niche as computers.

Not now.  Sufficiently advanced, though, an AI could occupy any niche it 
decides to.

 >... a huge number of metric tons of matter under our feet that doesn't 
fight back ...

Play-Doh of the gods?!  But the Singularity AI will want what is in our 
brains, not what is in the lavarock.

Thanks to all who participated thus far in this discussion.  I learned a 
lot from it, and I'm sure it was a lot more than I would have learned had I 
attended that "Singularity Conference".


Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=27999