Whose values should advanced AI have?

When we're talking about getting advanced AI to behave according to "human values", some people respond by saying something like "which human values?", "whose values?", or "there's no such thing as 'human values', don't you know about [cultural diversity / relativism]."

I sometimes worry that people raising this question aren't being sincere -- instead, they're trying to paint people worried about AI safety as socially and politically naive techno-futurists ("nerds", which we don't like any more), and they're happy to ask the question, show they're superior, and then leave without actually engaging with the question.

Putting this aside, I think "what exactly should we do with powerful AI, and how should we decide?" is a natural political question. However, I also think it's not too different from most other political questions.

For some advanced AI systems, we should probably should treat them like most other artifacts -- they will be used to pursue the goals of their owners, and so the answer to "which values" for "Private Property AI" will be "some of the values of their owners, or some values that their owners want them to have instrumentally to achieve their owners' ends". Use of Private Property AI should be subject to some laws, which might include what values and capabilities Private Property AI is allowed to have, and we'd be hoping that these laws in combination with economic forces would lead to advanced AI having a positive overall impact on human welfare. (It seems like markets have done OK at this for other technologies.)

The private ownership solution unsatisfying if we think that some kinds of advanced AI distributed through markets is likely to be bad for overall human welfare (the way we think that distributing nuclear or biological weapons through markets would be bad for overall human welfare). If advanced AI is powerful enough, it could create super-huge inequality between owners and non-owners, allow owners to defy laws and regulations that are supposed to protect non-owners, or allow owners to control or overcome governments. Owners might also use advanced AI in a way that exposes humanity to unacceptable risks, or governments with advanced AI might use it to dominate their citizens or attack other countries.

In response to this, we'll probably at least restrict ownership of some advanced AI systems. If we want more powerful AI systems to exist, they should be "Public AI", and have their goals chosen with some kind of public interest in mind.

At this point, it seems like the conversation has usually gone to one of two places:

1. We need to solve all of moral philosophy, and put that in the AI.
2. We need to solve meta-ethics, so that the AI system can solve moral philosophy itself.

This leads to questions like "Should advanced AI be built to follow total hedonic utilitarianism? Christian values? Of which sect, and interpreted how? Which ice cream flavor is best? What makes a happy marriage? How can we possibly figure all of this out in time?"

I don't think it's true that we actually need to solve moral philosophy to figure out what values to give Public AI. Instead, we could do what we do with laws: agree collectively to give Public AI systems a tiny set of agreed-upon values (e.g. freedoms and rights, physical safety, meta-laws about how the laws are to be updated, etc.), leaving most value questions like "which ice cream flavor is best" or "what God wants" to civil society.

Political theory / political philosophy have spent some time thinking about the same basic question: "we have this powerful thing [government], and we disagree about a lot of values -- what do we do?" Some example concepts that seem like they could be ported over:

• Neutrality / pluralism except where necessary: don't have governments make decisions about most value-related questions; instead, just have them make decisions about the few they're really needed for (e.g. people can't murder or steal), and have them remain basically neutral about other things.
• Enumerated powers: "The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people." Contrast with letting governments do whatever isn't explicitly prohibited.
• Rule of law: instead of giving governors full power, make a set of laws that everyone including governors is subject to.
• Separation of powers: make it hard for any set of governors to take full power.
• Constitutionalism: explicit codification of things like the above, along with explicit rules about how they can be updated.
• Democracy: give everyone a voice in the creation and enforcement of law, so that the law can evolve over time in a way that reflects the people.
• "Common-value laws": I'm not sure if there's a real term for this, but a lot of laws codify values that are widely shared, e.g. that people shouldn't be able to kill each other or take each other's stuff at will.
• "Value-independent laws": again not sure if there's a real term, but some laws aren't inherently value-related, but are instead meant to make sure that civil processes that generate value for people (like trade) go smoothly.

I think "constitutional democracy" is the right basic way to think about the "whose values" problem for Public AI, and makes the whole thing look at lot less scary.

Brilliance, blunders, and backfires

In AlphaGo's recent play, there were two kinds of moves that stood out:

• Brilliant moves: moves that accomplish the system's goal – winning the game – in ways that a human wouldn't think of, and that might take a while to even be understandable in retrospect (or might elude our understanding altogether).
• Blunders: moves that humans can identify (though sometimes only in retrospect) as bad for the system's goal.

As AI systems become more capable, it will be harder to tell the difference between brilliant moves and blunders until their effects are felt, and even in retrospect they may be hard to diagnose. If hard-to-understand AI systems are given safety-critical or high-impact tasks, blunders could become a source of significant harm.

However, I think we should be at least as concerned about a third kind of behavior:

• Backfires: moves that accomplish the system's nominal goal, but that don't do what the user actually wanted or that have unintended side-effects, and that might only be identified as backfires in retrospect.

Like blunders, we have the challenge that backfires won't easily be distinguished from brilliant moves. Backfires bring additional challenges; unlike blunders, improving a system's ability to achieve its nominal goals won't fix backfires, and may actually make them worse:

• A backfire might accomplish what we really wanted, but with additional effects that we don't want – getting a ball into a hoop while also smashing every vase in the room, or making a cup of coffee while also lighting the house on fire or breaking the law. As systems become more capable, they will be able to cause broader effects, making this problem worse.
• A backfire might accomplish the nominal goal without accomplishing what we really want, e.g. by manipulating a reward signal directly instead of by winning games of Go or Atari. As systems become more capable, they will find more ways of accomplishing their nominal goals, making this problem worse.

Backfires could happen because it's difficult to specify in full what we want a system to accomplish and what unintended consequences we want it to avoid, difficult to know in advance what means a system might use to accomplish its nominal goal, and difficult to specify goals in a way that can't be "gamed".

Updated version of an AI decision theory problem

Let's say we have a model-based RL system in a peculiar episodic environment: at the start of each episode, the system is copied onto another computer, and the copy's actions matter for what happens in the environment (e.g. they are playing some kind of game).

A "correct" model (state, action) → (new state) seems like it should have two properties:

1. If something causes the copy to behave differently from the system -- e.g., the copy is made incorrectly, the computer the copy runs on malfunctions, or the copy is interfered with during its decision -- the model should predict what the malfunctioning copy will actually do.
2. "Otherwise", the model should predict that the system and the copy perform the same action; that is, when the model predicts the copy's action in order to predict new state, the copy's action should be identical to the model's input action.

There are two kinds of models that I think it is reasonable to imagine our RL system making: abstract-ish models and physical-ish models. It seems like abstract-ish models will do well with property 2, and physical-ish models will do well with property 1. I can't picture the type of model that will get both properties.

• When the system has a fairly abstract and non-physical model (state, action) → (new state), it can simply model the copy's action as if it directly depended on the system's action, and it will correctly predict that the system and copy will always act the same way. However, this kind of model will make wrong predictions if something goes physically wrong with the copy.
• As the system learns a model that correctly handles things going wrong with the copy, it will probably no longer model the copy's actions as directly dependent on the system's action (in part because this is an accurate model of the physical setup). However, this kind of model seems like it will not predict that the system and the copy perform the same action when nothing goes wrong with the copy, since the copy will perform the same physical steps to make its decision regardless of what the input action is.

As the system learns, if things do go wrong with the copy, it will probably get evidence that pushes it toward physical-ish models, so that it can predict those things going wrong. However, it seems like there are common cases where a model fails property 2, but doesn't receive evidence to push it to correct this. For example:

Suppose the system is considering which action to take in state s. It considers actions a1, a2, a3. Using a physical-ish model, it predicts that the copy will take action a1 independent of the system's choice, so that possible (system choice, copy choice) pairs are (a1, a1), (a2, a1), (a3, a1), the second and third of which are mispredictions. If the system prefers the state resulting from (a1, a1) to the states resulting from (a2, a1) or (a3, a1), it will choose a1, and will never receive feedback that if it had chosen a2, the copy would have chosen a2 as well, and ditto for a3.

Intuitively, the model can't learn "what would have happened" correctly, and will need to rely on generalization in order to get this right. I don't know what kind of generalization would produce a model that does this correctly. This kind of problem will be especially bad if there are equilibria that result in the system consistently choosing the same action, as in the Prisoner's Dilemma (where the system never realizes that if it cooperated, the copy would cooperate as well). However, even outside this kind of problem, I still don't know how to make a model that fulfills properties 1 and 2.

This is just a toy case, and I expect analogous problems to come up in analogous situations (e.g. situations where there is not an exact copy, or where the "copy" is another system, human, or market that is learning to predict the system's actions). I don't know how much I should expect this kind of problem to come up in model-free approaches, but it seems worth looking into, and it would be disappointing if this kind of problem blocked us from using model-based approaches at all.

Failing to learn about counterfactuals

Let's say that we have a model-based RL system doing episodic RL in the following environment:

• At the start of each episode, the system is copied onto another computer
• The system and its copy play one round of the Prisoner's Dilemma game
• At the end of the episode, the situation is reset (the copy is erased)

When the system has a fairly abstract and non-physical model that maps (state, action) → (state), it can simply model the copy's action as if it directly depended on the system's action. So, when it predicts what would happen if it cooperated, it will predict that the copy will cooperate as well, and likewise for defection.

However, we might hope that the system will gradually learn a more physically realistic model of the world. This model won't contain any physical pathway linking the system's action to the copy's action (since the copy was made before the Prisoner's Dilemma is played), and will allow for the copy system to be interfered with in a variety of ways, breaking the symmetry of the game.

Clearly, this model can't be perfect -- if the system had to predict what a perfect copy would do before it acted, it would fall into an infinite regress. The system will need some way around this, like a limited model class, a contingency for long-running models, or some way of recognizing these kinds of situations (though halting-like problems seem to keep it from recognizing all situations of a similar type). If it can successfully recognize this situation, it seems like it "should" assume that the copy will take the same action as it does, and the problem is resolved. That would be nice! However, I don't think that models the way we build them currently will do this by default.

Let's assume that the model is not totally accurate, either because of a limited model class or because the system defaults to some estimate of the next state when a model takes too long to run. Now, when the system predicts what the copy will do, this prediction is independent of the system's action. Without loss of generality, let's say that the system predicts the copy will defect.

Now, the system needs to make a decision. It will evaluate cooperation and defection, both under the prediction that the copy will defect, and in this case it will choose defection. After it makes this choice, both it and the copy will in fact defect, reinforcing the system's model. However, the situation that the system predicted would follow from the action it didn't take -- the situation where it cooperates where the other system defects -- wouldn't actually have happened! Since the system and its copy are identical, they will always behave the same way; the model has predicted incorrectly. Furthermore, it can never witness this incorrect prediction, making it very hard to correct (dependent on some kind of generalization or regularization perhaps?).

In the case I've given above, the system and its copy miss out on some reward as a result of this inaccuracy -- they "could have" both cooperated and gotten better rewards. However, the failure could just as well be harmless (though still unsettling) -- if the system's imprecise model predicts the copy will cooperate, it will cooperate as well, and everything will be fine. I think there are stranger situations where the system's predictions will always be wrong (for halting-problem reasons).

I hope I'll be able to post a simpler version of this problem in the future -- this one is a little too long-winded to be useful!

AI and self-modeling

One problem in theoretical AI that sometimes comes up is the problem of finding ways for AI systems to model themselves, or at least to act well as if they had models of themselves. I can see how this is a problem for uncomputable agents like AIXI (though I think this problem is largely solved here), but it doesn't seem to me to be a problem for computable agents -- they can learn models of themselves along with the rest of the world. I'll give an example of trouble that some kinds of systems can run into, then my reasons for not thinking this is a big problem (though I'm by no means sure!).

A problem for model-based RL

Suppose that we're using model-based RL; our system learns a model that maps states of the world and actions the system takes to next states and rewards (see e.g. this talk and the slides). This learned model is used to choose actions by building a tree of possible sequences of actions the system could take and the consequences that the model predicts would result. The situation our system is in will be as follows:

• The system is learning to perform some episodic RL task; at the end of each episode, the environment is reset and another instance is run.
• In this environment, the agent has an action that gives a moderately large reward, but that forces the agent to take a null action for the rest of the episode.

The interesting thing here is that the system's model won't learn anything about the bad side effect of this action, even if it impacts the system's total reward a lot. This is because the model maps (state, action) → (next state); it learns what environmental state the bad action leads to, and after that it learns a lot about the effects of the null action, but it doesn't learn that the bad action leads to the null action. Furthermore, the tree search will continue to assume that the system will be able to choose whatever action it wants, even when the system will be forced to take the null action.

This is concerning, but the fix seems simple: simply have the system learn an additional model that maps states to states, implicitly causing it to model the system's action selection. Then, when the agent selects an action, have it use the (state, action) → (state) model followed by several iterations of the (state) → (state) model to see what effects that action will have. This should allow it to learn that it will be forced to take the null action, so that it can choose that action only when it actually maximises rewards.

In general, this kind of approach seems fine to me; a system can learn a model of the environment including itself, and use this model to figure out the long-term consequences of its actions. I haven't yet found a problem with this, and I might look for some kind of formal guarantee.

It's not obvious to me how this kind of problem could affect model-free systems; my feeling is that they should do fine, but I'd like to know more.

All in all, the theoretical problem involving uncomputable ideals like AIXI seems to be mostly solved, and the practical problem doesn't seem like a big deal. Am I missing something?

Deep learning in simple words

I wanted to try out the Up-Goer Five text editor, which lets you use only the 1,000 most commonly used english words (compiled by Randall Munroe), and I settled on trying to explain deep learning. Here's what I ended up with:

Sometimes, we want a computer to answer some kind of question, like who is in a picture or what the best move is in a game. To do this, we tell the computer to pretend it has a lot of little parts; each of the parts will look at the question, do a very easy bit of work, and then give its work to the next parts of the computer. At first, the computer doesn't know what each part should do, so it just picks any bit of work without thinking about it. The last parts of the computer will give us the answer.

At first, the parts don't know what to do in order to answer the questions. To help the computer learn, we write down many questions and the right answers. Then, we show the questions to the computer one at a time. The computer uses its parts to guess the answer and checks if it got it right.

Then comes the most important step: if it guessed the answer wrong, the computer changes all of its parts just a little so that it gets a little closer to the right answer. If it does this enough times, it will start to guess the right answers to new questions more and more often.

In some old learning computers, not very many parts worked on a question one after the other. Since one part can't do very much, and the parts couldn't share their little bits of work with each other, the old computers couldn't learn to answer very hard questions. Deep learning is when many parts of the computer work on the question one after the other, so that each part can build on the work of the other parts to answer hard questions.

When lots of parts work on questions one after the other, it's harder to figure out how to change the parts to make the guesses better, and sometimes the parts never figure out the right things to do. But now, some people figured out how the computer should change its parts, and computers are much better at learning to do lots of things.

I'm pretty happy with this. An interesting effect of using the text editor was the sense of focus it gave me; it pushed me to prioritize the most critical aspects (in my mind) of deep learning, instead of more tangential connections between deep learning and other things. I think the result accurately represents my understanding, and that someone who understands DL better could use it to diagnose holes in my picture of how DL works, which is nice.

This leaves something to be desired; for example, I love Chris Olah's explanation of DL as differentiable functional programming, and my description didn't really explain anything about backpropagation or the importance of differentiability. Still fun!