Whose values should advanced AI have?

When we're talking about getting advanced AI to behave according to "human values", some people respond by saying something like "which human values?", "whose values?", or "there's no such thing as 'human values', don't you know about [cultural diversity / relativism]."

I sometimes worry that people raising this question aren't being sincere -- instead, they're trying to paint people worried about AI safety as socially and politically naive techno-futurists ("nerds", which we don't like any more), and they're happy to ask the question, show they're superior, and then leave without actually engaging with the question.

Putting this aside, I think "what exactly should we do with powerful AI, and how should we decide?" is a natural political question. However, I also think it's not too different from most other political questions.

For some advanced AI systems, we should probably should treat them like most other artifacts -- they will be used to pursue the goals of their owners, and so the answer to "which values" for "Private Property AI" will be "some of the values of their owners, or some values that their owners want them to have instrumentally to achieve their owners' ends". Use of Private Property AI should be subject to some laws, which might include what values and capabilities Private Property AI is allowed to have, and we'd be hoping that these laws in combination with economic forces would lead to advanced AI having a positive overall impact on human welfare. (It seems like markets have done OK at this for other technologies.)

The private ownership solution unsatisfying if we think that some kinds of advanced AI distributed through markets is likely to be bad for overall human welfare (the way we think that distributing nuclear or biological weapons through markets would be bad for overall human welfare). If advanced AI is powerful enough, it could create super-huge inequality between owners and non-owners, allow owners to defy laws and regulations that are supposed to protect non-owners, or allow owners to control or overcome governments. Owners might also use advanced AI in a way that exposes humanity to unacceptable risks, or governments with advanced AI might use it to dominate their citizens or attack other countries.

In response to this, we'll probably at least restrict ownership of some advanced AI systems. If we want more powerful AI systems to exist, they should be "Public AI", and have their goals chosen with some kind of public interest in mind.

At this point, it seems like the conversation has usually gone to one of two places:

1. We need to solve all of moral philosophy, and put that in the AI.
2. We need to solve meta-ethics, so that the AI system can solve moral philosophy itself.

This leads to questions like "Should advanced AI be built to follow total hedonic utilitarianism? Christian values? Of which sect, and interpreted how? Which ice cream flavor is best? What makes a happy marriage? How can we possibly figure all of this out in time?"

I don't think it's true that we actually need to solve moral philosophy to figure out what values to give Public AI. Instead, we could do what we do with laws: agree collectively to give Public AI systems a tiny set of agreed-upon values (e.g. freedoms and rights, physical safety, meta-laws about how the laws are to be updated, etc.), leaving most value questions like "which ice cream flavor is best" or "what God wants" to civil society.

Political theory / political philosophy have spent some time thinking about the same basic question: "we have this powerful thing [government], and we disagree about a lot of values -- what do we do?" Some example concepts that seem like they could be ported over:

• Neutrality / pluralism except where necessary: don't have governments make decisions about most value-related questions; instead, just have them make decisions about the few they're really needed for (e.g. people can't murder or steal), and have them remain basically neutral about other things.
• Enumerated powers: "The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people." Contrast with letting governments do whatever isn't explicitly prohibited.
• Rule of law: instead of giving governors full power, make a set of laws that everyone including governors is subject to.
• Separation of powers: make it hard for any set of governors to take full power.
• Constitutionalism: explicit codification of things like the above, along with explicit rules about how they can be updated.
• Democracy: give everyone a voice in the creation and enforcement of law, so that the law can evolve over time in a way that reflects the people.
• "Common-value laws": I'm not sure if there's a real term for this, but a lot of laws codify values that are widely shared, e.g. that people shouldn't be able to kill each other or take each other's stuff at will.
• "Value-independent laws": again not sure if there's a real term, but some laws aren't inherently value-related, but are instead meant to make sure that civil processes that generate value for people (like trade) go smoothly.

I think "constitutional democracy" is the right basic way to think about the "whose values" problem for Public AI, and makes the whole thing look at lot less scary.