the prevention of knowledge-enabled massdestruction

The experiences of the atomic scientists clearly show the need to takepersonal responsibility, the danger that things will move too fast, andthe way in which a process can take on a life of its own. We can, as theydid, create insurmountable problems in almost no time flat. We mustdo more thinking up front if we are not to be similarly surprised andshocked by the consequences of our inventions.

Knowing is not a rationale for not acting. Can we doubt that knowl-edge has become a weapon we wield against ourselves?

But many other people who know about the dangers still seemstrangely silent. When pressed, they trot out the “this is nothing new”riposte—as if awareness of what could happen is response enough.They tell me, There are universities filled with bioethicists who studythis stuff all day long. They say, All this has been written about before,and by experts. They complain, Your worries and your arguments arealready old hat.

I havefound the ideas in the book Ethics for the New Millennium, by the DalaiLama, to be very helpful. As is perhaps well known but little heeded, theDalai Lama argues that the most important thing is for us to conduct ourlives with love and compassion for others, and that our societies need todevelop a stronger notion of universal responsibility and of our interde-pendency; he proposes a standard of positive ethical conduct for individ-uals and societies that seems consonant with Attali’s Fraternity utopia.The Dalai Lama further argues that we must understand what it is thatmakes people happy, and acknowledge the strong evidence that neithermaterial progress nor the pursuit of the power of knowledge is the key—that there are limits to what science and the scientific pursuit alone can do.

Our Western notion of happiness seems to come from the Greeks,who defined it as “the exercise of vital powers along lines of excellencein a life affording them scope.”15

At the dawn of societies, men saw their passage on Earth as noth-ing more than a labyrinth of pain, at the end of which stood adoor leading, via their death, to the company of gods and to Eter-nity. With the Hebrews and then the Greeks, some men dared freethemselves from theological demands and dream of an ideal Citywhere Liberty would flourish. Others, noting the evolution of themarket society, understood that the liberty of some would entailthe alienation of others, and they sought Equality.

Thoreau also said that we will be “rich in proportion to the numberof things which we can afford to let alone.”

Verifying compliance will also require that scientists and engineersadopt a strong code of ethical conduct, resembling the Hippocraticoath, and that they have the courage to whistleblow as necessary, evenat high personal cost.

The GNR technologies do not divide clearly into commercial andmilitary uses; given their potential in the market, it’s hard to imaginepursuing them only in national laboratories. With their widespreadcommercial pursuit, enforcing relinquishment will require a verificationregime similar to that for biological weapons, but on an unprecedentedscale. This, inevitably, will raise tensions between our individual pri-vacy and desire for proprietary information, and the need for verifica-tion to protect us all. We will undoubtedly encounter strong resistanceto this loss of privacy and freedom of action.

We have embodied our relinquish-ment of biological and chemical weapons in the 1972 BiologicalWeapons Convention (BWC) and the 1993 Chemical Weapons Con-vention (CWC).

As Thoreau said, “We do not ride on the railroad; it rides upon us”;and this is what we must fight, in our time. The question is, indeed,Which is to be master? Will we survive our technologies?

I believe that we all wish our course could be determined by ourcollective values, ethics, and morals.

It was Nietzsche who warned us, at the end of the 19th century, notonly that God is dead but that “faith in science, which after all existsundeniably, cannot owe its origin to a calculus of utility; it must haveoriginated in spite of the fact that the disutility and dangerousness ofthe ‘will to truth,’ of ‘truth at any price’ is proved to it constantly.”

In recent times, we have come to revere scientific knowledge.

We have, as a bedrock value in our society, long agreed on thevalue of open access to information, and recognize the problems thatarise with attempts to restrict access to and development of knowledge.

Aristotle opened hisMetaphysics with the simple statement: “All men by nature desire toknow.”

The only realistic alternative I see is relinquishment: to limit de-velopment of the technologies that are too dangerous, by limiting ourpursuit of certain kinds of knowledge.

Now, as then, we are creators of new technologies and stars of theimagined future, driven—this time by great financial rewards andglobal competition—despite the clear dangers, hardly evaluating whatit may be like to try to live in a world that is the realistic outcome ofwhat we are creating and imagining.

In November 1945, three months after the atomic bombings,Oppenheimer stood firmly behind the scientific attitude, saying, “It isnot possible to be a scientist unless you believe that the knowledge ofthe world, and the power which this gives, is a thing which is of in-trinsic value to humanity, and that you are using it to help in the spreadof knowledge and are willing to take the consequences.”

But thesewarnings haven’t been widely publicized; the public discussions havebeen clearly inadequate. There is no profit in publicizing the dangers.

Among the cognoscenti of nanotechnology, this threat has becomeknown as the “gray goo problem.” Though masses of uncon-trolled replicators need not be gray or gooey, the term “gray goo”emphasizes that replicators able to obliterate life might be less in-spiring than a single species of crabgrass. They might be superiorin an evolutionary sense, but this need not make them valuable.

Drexler startedthe Foresight Institute in the late 1980s “to help prepare society for an-ticipated advanced technologies”

I remember feeling good about nanotechnology after reading Enginesof Creation.

If, for ex-ample, we were to reengineer ourselves into several separate and unequalspecies using the power of genetic engineering, then we would threatenthe notion of equality that is the very cornerstone of our democracy.

It seems to me farmore likely that a robotic existence would not be like a human one inany sense that we understand, that the robots would in no sense be ourchildren, that on this path our humanity may well be lost.

But if we are downloaded into our technology, what are the chancesthat we will thereafter be ourselves or even human?

How soon could such an intelligent robot be built? The coming ad-vances in computing power seem to make it possible by 2030.

in hishistory of such ideas, Darwin Among the Machines, George Dysonwarns: “In the game of life and evolution there are three players at thetable: human beings, nature, and machines. I am firmly on the side ofnature. But nature, I suspect, is on the side of the machines.”

My personal experience suggests we tendto overestimate our design abilities.

The physicistsStephen Wolfram and Brosl Hasslacher introduced me, in the early1980s, to chaos theory and nonlinear systems. In the 1990s, I learnedabout complex systems from conversations with Danny Hillis, the bi-ologist Stuart Kauffman, the Nobel-laureate physicist Murray Gell-Mann, and others. Most recently, Hasslacher and the electrical engineerand device physicist Mark Reed have been giving me insight into the in-credible possibilities of molecular electronics.

After a few years at Berkeley I started to send out some of the soft-ware I had written—an instructional Pascal system, Unix utilities, anda text editor called vi (which is still, to my surprise, widely used morethan 20 years later)—to others who had similar small PDP-11 and VAXminicomputers

In The Agony and the Ecstasy, Irving Stone’s biographical novel ofMichelangelo, Stone described vividly how Michelangelo released thestatues from the stone, “breaking the marble spell,” carving from theimages in his mind.

Thus we have the possibility not just of weapons of mass destructionbut of knowledge-enabled mass destruction (KMD), this destructive-ness hugely amplified by the power of self-replication.

Uncontrolledself-replication in these newer technologies runs a much greater risk: arisk of substantial damage in the physical world.

the changes would come gradually, and that we would get used to them

Danny is also a highly regarded futurist who thinkslong-term—four years ago he started the Long Now Foundation

Danny Hillis

Moravec’s view is that the robots will eventually suc-ceed us—that humans clearly face extinction.

The systems involvedare complex, involving interaction among and feedback between manyparts. Any changes to such a system will cascade in ways that are diffi-cult to predict; this is especially true when human actions are involved.

Our overuse of antibiotics has led to what may be thebiggest such problem so far: the emergence of antibiotic-resistant andmuch more dangerous bacteria. Similar things happened when attemptsto eliminate malarial mosquitoes using DDT caused them to acquireDDT resistance; malarial parasites likewise acquired multi-drug-resistant genes.

In the book, you don’t discover until you turn the page that the au-thor of this passage is Theodore Kaczynski—the Unabomber