There is a new paper by Nick Bostrom and Eliezer Yudkowsky on the ethics of Artificial Intelligence. It will appear in the Cambridge Handbook of Artificial Intelligence:

The possibility of creating thinking machines raises a host of ethical issues.  These
questions relate both to ensuring that such machines do not harm humans and other
morally relevant beings, and to the moral status of the machines themselves.  The first
section discusses issues that may arise in the near future of AI.  The second section
outlines challenges for ensuring that AI operates safely as it approaches humans in its
intelligence.  The third section outlines how we might assess whether, and in what
circumstances, AIs themselves have moral status.  In the fourth section, we consider
how AIs might differ from humans in certain basic respects relevant to our ethical
assessment of them.  The final section addresses the issues of creating AIs more
intelligent than human, and ensuring that they use their advanced intelligence for
good rather than ill.

This paper serves as a good introduction to the problem of Friendly AI.

Louie Helm and Luke Muehlhauser, Singularity Institute staff, are sharing a draft of their chapter submission to The Singularity Hypothesis edited volume for feedback:

The Singularity and Machine Ethics

Abstract: Many researchers have argued that a self-improving artificial intelligence (AI) could become so vastly more powerful than humans that we would not be able to stop it from achieving its goals. If so, and if the AI’s goals differ from ours, then this could be disastrous for humans. One proposed solution is to program the AI’s goal system to want what we want before the AI self-improves beyond our capacity to control it. Unfortunately, it is difficult to specify what we want. After a brief digression concerning human intuitions about intelligence, we offer a series of “intuition pumps” in moral philosophy for our conclusion that human values are complex and difficult to specify. We then survey the evidence from the psychology of motivation, moral psychology, and neuroeconomics that supports our position. We conclude by recommending ideal preference theories of value as a promising approach for developing a machine ethics suitable for navigating the Singularity.

A new paper by Eliezer Yudkowsky is online at our publications page, “Complex Value Systems are Required to Realize Valuable Futures”. This paper was presented at the recent Fourth Conference on Artificial General Intelligence, held at Google HQ in Mountain View.

Abstract: A common reaction to first encountering the problem statement of Friendly AI (”Ensure that the creation of a generally intelligent, self-improving, eventually superintelligent system realizes a positive outcome”) is to propose a single moral value which allegedly suffices; or to reject the problem by replying that “constraining” our creations is undesirable or unnecessary. This paper makes the case that a criterion for describing a “positive outcome”, despite the shortness of the English phrase, contains considerable complexity hidden from us by our own thought processes, which only search positive-value parts of the action space, and implicitly think as if code is interpreted by an anthropomorphic ghost-in-the-machine. Abandoning inheritance from human value (at least as a basis for renormalizing to reflective equilibria) will yield futures worthless even from the standpoint of AGI researchers who consider themselves to have cosmopolitan values not tied to the exact forms or desires of humanity.

Keywords: Friendly AI, machine ethics, anthropomorphism

For those who missed it, philosopher of mind David Chalmers published “The Singularity: a Philosophical Analysis” last April, and has been giving occasional talks on the subject matter. Chalmers was initially inspired to think about the Singularity from his invitation to our Singularity Summit conference in 2009.

Basic AI Drives and Catastophic Risks (Carl Shulman, 2010)
Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics (Nick Tarleton, 2010)
Economic Implications of Software Minds (S. Kaas, S. Rayhawk, A. Salamon and P. Salamon, 2010)
From mostly harmless to civilization-threatening: pathways to dangerous artificial general intelligences (Kaj Sotala, 2010)
Implications of a software‐limited singularity (Carl Shulman, Anders Sandberg, 2010)
Superintelligence does not imply benevolence (Joshua Fox, Carl Shulman, 2010)
Timeless Decision Theory (Eliezer Yudkowsky, 2010)

The above are papers, below are presentations:

How intelligible is intelligence? (Anna Salamon, Stephen Rayhawk, János Kramár, 2010)
Whole Brain Emulation and the Evolution of Superorganisms (Carl Shulman, 2010)
What can evolution tell us about the feasibility of artificial intelligence? (Carl Shulman, 2010)

Here is a quote from the Chalmers paper that I linked yesterday:

One might think that the singularity would be of great interest to academic philosophers, cognitive scientists, and artificial intelligence researchers. In practice, this has not been the case. Good was an eminent academic, but his article was largely unappreciated at the time. The subsequent discussion of the singularity has largely taken place in nonacademic circles, including Internet forums, popular media and books, and workshops organized by the independent Singularity Institute. Perhaps the highly speculative flavor of the singularity idea has been responsible for academic resistance to the idea.

I think this resistance is a shame, as the singularity idea is clearly an important one. The argument for a singularity is one that we should take seriously. And the questions surrounding the singularity are of enormous practical and philosophical concern.

Practically: If there is a singularity, it will be one of the most important events in the history of the planet. An intelligence explosion has enormous potential benefits: a cure for all known diseases, an end to poverty, extraordinary scientific advances, and much more. It also has enormous potential dangers: an end to the human race, an arms race of warring machines, the power to destroy the planet. So if there is even a small chance that there will be a singularity, we would do well to think about what forms it might take and whether there is anything we can do to influence the outcomes in a positive direction.

Great advice for everyone living in the 21st century!

Recently, David Chalmers announced that he was posting a new paper based on his Singularity Summit 2010 talk: “The Singularity: A Philosophical Analysis”. In his announcement, Chalmers notes, “I’m still an amateur on these topics and any feedback would be appreciated.” You can also watch a video of Chalmers’ Summit talk.

Here’s a paper presented at the 2009 Asia-Pacific Conference on Computing and Philosophy by participants in SIAI’s 2009 Visiting Fellows Program that is making the rounds. The point of the paper, which was written by Carl Shulman, Nick Tarleton, and Henrik Jonsson, is that consequentialism as commonly discussed has a number of “free variables” where intuitions disagree about the right values of these variables. Therefore, machine ethics should draw on the emerging field of moral psychology to figure out how to fill in these free variables. This point is plainly put in the title of one of the last sections, “Current moral theories are inadequate for machine ethics”.

A reply from UK philosopher David Pearce has recently been posted by Roko Mijic at Less Wrong.

Check out my blog post, “Hungry Optimizers with Low-Complexity Values” at Accelerating Future.

Ed Boyden, who leads the Synthetic Neurobiology Group at MIT, is concerned about the Singularity, and will be speaking at our upcoming Singularity Summit conference in New York. He recently published the article “The Singularity and the Fixed Point” on the website of Technology Review, MIT’s magazine, which looks into the challenge of giving Artificial Intelligence proper motivations. Making theoretical progress on the question of, “how do we give Artificial Intelligence proper motivations, such that we can trust those motivations even if the AI becomes smarter and more powerful than humans?” is a primary reason for SIAI’s existence.

A Call of Papers has been issued for a track on “Technological Singularity and Acceleration Studies,” at the 7th European conference on Computing And Philosophy?ECAP 2009, Universitat Autònoma de Barcelona, 2-4 July 2009.

Historical analysis of a broad range of paradigm shifts in science, biology, history, and technology–in particular in computing technology–suggests an accelerating rate of progress. This observation has led the attempted unification of the predictive power of biological evolution, cultural evolution, and technological evolution under a “Law of Accelerating Returns.” As a consequence, John von Neumann forecasted the arrival of an ?essential singularity in the history of the race beyond which human affairs as we know them could not continue.? This notion of Singularity coincides in time and nature with Alan Turing (1950) and Stephen Hawking’s (1998) expectation of machines that exhibit intelligence on par with an average human by 2050. John Irving Good (1965) and Vernor Vinge (1993) expect it to take the form of an ‘intelligence explosion’: the process by which ultraintelligent machines design ever more intelligent machines. Transhumanists suggest a parallel process of explosive progress in human intelligence. Unfortunately, the very term ?Singularity? also suggests the presence of an ?event horizon,? an epistemological barrier on our ability to understand the events that may follow it.

We invite abstracts examining the following issues from a philosophical, computational, mathematical, and scientific points of view:

1. Empirical assessments of the Law of Accelerating Returns
2. Estimating the reliability of a technological forecasts
3. Historical analysis of the Law of Accelerating Returns
4. The impact of acceleration on science and society by 2050
5. Hazards of technological acceleration and preventative measures
6. The nature of the Technological Singularity
7. The nature of an intelligence explosion
8. Beyond the ?event horizon? of the Technological Singularity

Papers submitted to the Technological Singularity and Acceleration Studies track in ECAP 2009 will also be considered for publication in a special issue of Technological Forecasting and Social Change (Elsevier).

Received from the track chair, Amnon H. Eden, School of Computer Science & Electronic Engineering, University of Essex, UK and Center For Inquiry, Amherst NY.