The Late Sleeper: a problem for ataraxians

Dan just wrote about Benetar's "Better Never to have Been," and this neat thought experiment occurred to me when I was talking to Killian about it.

ETA: here's what I should have written. Some people think that so-called "good experiences" are actually just the cessation of suffering, reprieve from craving, etc. That would mean that the best you can do is break even; that's good, but at best it's only as good as it would have been to not have had those cravings / sufferings in the first places. I'm calling this the ataraxian position; see also antifrustrationism and tranquilism.

Does this fit with our experience? Well, it seems like sleeping people are free from suffering/craving, so sleep is an optimal state (along with nonexistence or complete cessation of suffering) under these theories, and it's less confusing than nonexistence. Now let's compare supposedly "positive" experiences to being asleep. If you think "positive" experiences can't be better than being asleep, you're an ataraxian. If you think "positive" experiences can be better than being asleep, you're not an ataraxian. I'm not an ataraxian.

(Overly complicated original post follows)

This doesn't address Benetar's case directly, but it deals with a related view that I've heard: the view that the "best" outcomes/experiences for people are actually just a minimization of their suffering. For example, eating a tasty meal just amounts to temporarily bringing your suffering from a lack of tasty food down to zero; it's good, but at best it's only as good as it would have been never to have been hungry at all. If you think that the best anyone can do is hit the neutral point (instead of having "positive" experiences), then it's better never to have been born. I'm calling this ataraxian ethics -- I'm sure it has some real name, but I don't know what it is.

Here's the thought experiment that makes me reject ataraxian ethics. In case it gets famous, I'm obligated to give it a name and a little story:

The Late Sleeper
Alice is sleeping on a Saturday morning. When she sleeps, she is truly unconscious, and is not experiencing any kind of suffering. There's some chance that she'll happen to wake up early, and if she does, she'll see a beautiful sunrise and enjoy a cup of coffee with a loved one. Whether she wakes and sees the sunrise or sleeps and doesn't see it won't have any further effects on her life. Would it be good for her to happen to wake up early?

The ataraxian would say that this would not be good; when Alice is asleep, she isn't suffering, and according to the ataraxian it's not possible for her to be in a better state than this. At best, this experience could be as good as remaining asleep, but in order for this to be the case, Alice-at-sunrise would have to be in her literally best possible state, completely free of suffering. If there's any chance of a bad experience during the sunrise (e.g. Alice's coffee is a little too bitter), it would be better for her to remain asleep. In fact, there's no experience at all (!) Alice could have, no matter how "good", that would be better than staying asleep.

This makes me pretty sure I'm not an ataraxian; there are a million things that sound better than being asleep, and I'd be willing to take some suffering (e.g. being sleepy later or risking injury by walking down the stairs) for many of them. I'm pretty sure I'm not just being culturally pressured into saying this, and if I'm fooling myself, I think I've fooled myself into actually enjoying those experiences.

I'm tempted to say that people who might think they're ataraxians should obviously agree with this and stop being ataraxians (probably retreating to saying "OK, there are actual good things beyond cessation of suffering, but the bad outweighs the good in basically every life, so it'd still be better not to exist"), but who knows, people think lots of different things :)

Bostrom's observation equation

Nick Bostrom's book "Anthropic Bias" seems to be the most thorough examination of observation selection effects around. However, I don't really understand the reasoning method he proposes at the end of the book (Chapter 10) and in this paper. So, let's work through it.

(Note: I started this post not understanding the observation equation, but now I feel like I do. +1 for writing to understand!)

First, Bostrom states the Strong Self-Sampling Assumption (SSSA), an informal rule for reasoning that he thinks is correct:

(SSSA) Each observer-moment should reason as if it were randomly selected from the set of all observer-moments in its reference class.

Sounds pretty good to me, but the devil's in the details -- in particular, what is a reference class?

Bostrom offers an "observation equation" formalizing SSSA. Suppose an observer-moment \(m\) has evidence \(e\) and is considering hypothesis \(h\). Bostrom proposes this rule for \(m\)'s belief:
\[P(h|e) = \frac{1}{\gamma}\sum_{o\in O_h\cap O_e}{\frac{P(w_o)}{|O_o\cap O(w_o)|}}\]
Okay, what does this mean? Ignore \(\gamma\) for now; it's a normalizing constant that depends only on \(e\) that makes sure probabilities add up to 1, I think. \(O_h\) is the set of observer-moments that are consistent with hypothesis \(h\), and \(O_e\) is the set of observer-moments that have evidence \(e\). So, what we're doing is looking at each observer-moment \(o\) with evidence \(e\) where hypothesis \(h\) is actually true, and adding up the probabilities of the worlds that those \(o\) live in, divided by the number of observers in that world that are members of \(o\)'s "reference class", which we still haven't defined.

Now let's look at the normalization constant:
\[\gamma = \sum_{o\in O_e}{\frac{P(w_o)}{|O_o \cap O(w_o)|}}\]
This is pretty similar to the above, but it just sums over observer-moments that have evidence \(e\). In fact, the inside of the sum is the same function of \(o\) as the inside of the sum of the first equation. In fact, I think we can sensibly pull this out into its own function, which semantically I think is something like the prior probability of "being" each observer:
\[P(o) = \frac{P(w_o)}{|O_o\cap O(w_o)|}\]
For each observer, the prior probability of "being" that observer is the probability of being in that world, split equally among all observers in that world that are in the same "reference class". This in turn lets us rewrite the observation equation as:
\[P(h|e) = \frac{\sum\limits_{o\in O_h\cap O_e}{P(o)}} {\sum\limits_{o\in O_e}{P(o)}}\]
This is useful, because it makes it clear that this is basically the formula for conditional probability!
\[P(h|e) = \frac{P(\text{observe }e\text{ and }h\text{ is true})}{P(\text{observe }e)}\]
So, now I feel like I understand how Bostrom's observation equation works. I expect that I'll mostly be arguing in the future about whether \(P(o)\) is defined correctly, and I still need to come back to what exactly an observer's "reference class" is. Spoiler: Bostrom doesn't pin down reference classes precisely, and he thinks there are a variety of choices.

Observer selection effects from scratch

Suppose that I have only three theories T0, T1, T2, describing three possible worlds W0, W1, and W2. Now, suppose that I observe X, and suppose that the following is true:

• In W0, there are no observers of X.
• In W1, there is one observer of X.
• In W2, there are 100 observers of X.

What should I now believe about my theories? Should my beliefs be sensitive to how many observers of X there are in each world?

It seems pretty clear to me that I shouldn't believe T0, since it's not compatible with my observation of X; that's a minimal level at which my beliefs should be sensitive to the number of observers of X. A way of justifying this is to cash out "I believe in Tn" to mean "I believe I am in Wn", or "I believe that my future observations will be consistent with Tn". Then "I observe X" and "In W0, there are no observers of X" come together to imply "It's not possible that I'm in W0" and hence "I don't believe T0".

What should I think about T1 and T2, though? It's still possible that I'm in either one of their worlds, so I'll believe both of them to some extent. Should I believe one of them more than the other? (Let's assume that T1 and T2 were equally plausible to me before this whole thing started.)

Pretty solid ground so far; now things get shaky.

Let's think about the 101 possible observers distributed among W1 and W2. I think it's meaningful to ask which of those I believe I am; after all, which one I am could imply differences in my future observations.

Nothing about my observation X favors any of these observers over any other, so I don't see how I can believe I'm more likely to be one of them than another one, i.e. I should have equal credence that I'm any one of those observers.

This implies that I should think it's 100 times more likely that I'm in W2 than in W1, since 100 equally likely observers-of-X live in W2 and only one observer-of-X lives in W1. I should think T2 is much more likely than T1. This answers the original question of this blog post.

However, that means that if I'm considering two cosmological theories, and one of them predicts that there are billions of copies of me having the experience I'm having now, I should believe that it's very likely that that theory is true (all else equal). It's weird that I can have that kind of belief about a scientific theory while I'm just sitting in my armchair. (Nick Bostrom calls this "The Presumptuous Philosopher", and thinks you shouldn't reason this way.)

So, it seems like we have to pick one of these weird things:

1. It's nonsensical to have beliefs about which possible observer I am (even if being different observers implies different future observations).
2. Something besides my observations and my prior belief in theories of the world should affect my beliefs (in theories of the world, or in which observer I am).
3. Just by sitting in my armchair and thinking, I can come to strong, justified beliefs about cosmological theories based solely on how many people-thinking-in-armchairs they contain.
4. I've made some other mistake in my reasoning; like, my account of theories and worlds is wrong, or I'm not thinking carefully enough about what it means to be an observer, or I'm not thinking clearly about normative principles around beliefs, or something else. (Actually, me making a mistake wouldn't be so weird.)

?!

I tend to lean toward 3 (well, if I assume 4 isn't true), but smart people disagree with me, and it's kind of a crazy thing to believe. It could also mean that we're Boltzmann brains, thought I'm not sure. See also this paper.

---

Addendum: consider this similarly plausible-sounding reasoning:

1. "I observe X" just means "there exists an observer of X".
2. "There exists an observer of X" rules out T0, but not T1 or T2.
3. "There exists an observer of X" doesn't favor T1 or T2.
4. All else equal, I should have equal belief in T1 and T2.

I think this reasoning is too weak, and leaves out some implications. "I observe X" implies "there exists an observer of X", but I'd argue that it implies some additional things: it has implications about what I should believe I'll observe in the future (not just what some existing observer will observe), what theories I should believe are true (not just some observer), and which observers I should believe I could possibly be (ditto). Maybe I should redo my earlier reasoning in terms of expected observations and see what happens? 

Animal rights?

I was intrigued by this article; I think it correctly points out that caring about animal welfare is pretty different from caring about animal rights, but I was hoping for more of an argument in favor of animals having rights. So, I decided to think about this a bit myself.

At first, I thought that I wouldn't tend to be in favor of animal rights. I generally think about ethics in terms of welfare instead of in terms of rights or obligations, so why would I think animals should have rights? However, after thinking about it more, I've come down in favor of animal rights, and I feel like I have a better understanding of why human rights matter.

So, let's talk about human rights. I haven't seen a metaphysical argument that humans "naturally" have rights, and I'm not sure I'd be convinced by that kind of argument anyway. However, there area couple of reasons that I think it makes sense to assign humans rights:

• Humans have a strong preference for self-determination (which is partially final and partially instrumental)
• Rule-consequentialist / rule-utilitarian: "human rights" are a good policy to agree on, because that policy helps us maximize welfare

"Should we decide to give animals rights?" is the natural question for me; we could decide on rights pragmatically, and then follow them dogmatically, even when we don't see why they're useful in a particular case. I generally don't think the first reason above applies to animals, but I think the second does. So, I think we should give animals rights.

(A note on why I think it makes sense to consider rights as pragmatic things we decide on: rights are pretty complicated, sometimes seem inconsistent or made-up, and they're constantly up for debate. For example, what's the deal with children's rights? What are all the trade-offs and edge cases around the right to free speech, or the right to refuse service? I think it used to be considered a "right" of soldiers to defend themselves and their fellows on a battlefield, and to be exempt of moral blame when they follow orders, but revisionist just-war theorists are working on overturning that, IIRC. These things are clearly cultural constructs, and we should choose them as we see fit.)

Some rights that I think probably make sense for animals:

• They should be represented in decisions (e.g. political decisions)
• Guardianship should be a matter of political debate (and animals should have representatives in these decisions)
• If rights that work for kids don't work for animals, then animals should have more rights than kids do (e.g. people care about kids intrinsically, usually, and don't benefit financially from having them, whereas animals don't have these natural protections)

On this last point: imagine a world where having children could be very lucrative. In this world, we'd probably have to restrict the right to have children, and give children rights that prevent parents from exploiting them for financial gain. I think we probably will need to extend those kinds of rights to animals.

So, animal rights: yes! I'm just not sure which ones, or how we get there.

Boltzmann brains

What if almost everything we thought we knew about our position in the universe was wrong? What if we actually were not members of a species that arose around 200,000 years ago, among life forms that started evolving around 4 billion years ago on a planet that formed 4.5 billion years ago, in a universe that began with a Big Bang around 13 billion years ago? A single cosmological discovery that changed all of that would be an amazingly big deal (at least in terms of scientific knowledge -- it might not change what we actually do with our lives).

That's roughly what's at stake with the question of Boltzmann brains -- whether instead of the picture above, it's more likely that we came into existence a short time ago via random (quantum or thermal) fluctuations during an extremely long quiet period in one of the last ages of the universe. Not only our ideas about our position in the universe are at stake; it's also possible that only my brain arose this way, perhaps a few minutes or seconds ago, meaning that much of what I think is real (other people, places beyond my immediate reach, all of human history, etc.) is not actually physically real.

Now, this sounds suspiciously similar to many radically skeptical arguments, like the brain in a vat thought experiment -- how do you know you're not just a brain in a vat? These arguments are great for an intro-to-Philosophy class, but once the shine wears off, they seem a little thin -- what does it really offer to say "well, you might be a brain in a vat, there might be a deceiving demon, etc.", and what more can we say about these arguments? They might be useful thought experiments for epistemologists who need corner-cases to test their ideas of what "knowledge" really is and what we can really know, but they don't feel productive as a way to think about the world. I think the typical arc is to be surprised by these arguments, live with them for a while, and then forget about them, and I think that's fine.

However, I think the Boltzmann brains (BB) argument is importantly different. The BB argument isn't "how do you know you're not a BB", it's "according to some cosmological theories, many BBs will exist, and using some kinds of anthropic reasoning, it's likely that you're one of those BBs." It's as if scientists pointed their telescopes at the sky and saw vast arrays of brains-in-vats; we have (as far as I know) real reasons to take the BB scenario seriously.

I haven't been able to find a comprehensive survey of argumentation around BBs, or even a very rigorous paper that attempts to thoroughly examine the question; it's usually treated as an example or interesting implication in cosmology or philosophy books and papers, as far as I can tell. It looks like it's only been seriously considered for about 20 years, like so many of the ideas that I think are most important.

To be totally honest, I expect the BB argument to fail. I also don't think it's likely to be importantly action-informing; how would I really make decisions differently if I were a BB? However, it's one of a few really big questions about what the world is actually like that I'd really love to see answered. In fact, I think I'll post again to talk more about those big questions -- stay tuned.

I'm playing with the idea of writing and thinking more about BB -- it's an appealing hobby project. If I do, you'll see it here first!

Acts, omissions, friends, and enemies

The act-omission distinction plays a role in some ethical theories. It doesn't seem relevant to me, because I'm much more concerned with the things that happen to people than with whether a particular actor's behavior meets some criteria. (Of course, if some consequence is more beneficial or harmful if it is caused by an act or an omission, I'd care about that.)

Supererogatory acts, and the broader concept that some acts are required / forbidden while others are not, also play a role in lots of ethical theories, but don't seem relevant to me. To me, ethics is entirely about figuring out which acts or consequences are better or worse, and this doesn't give an obvious opening for making some acts required or forbidden.

I recently had an idea about why these concepts appear in some ethical theories: acts/omissions and supererogatory acts seem useful for identifying allies and enemies. This is roughly because acts tend to be costly (in terms of attention and other resources), and supererogatory acts tend to be expensive as well. There's not a lot more to say:

• Allies will pay the cost to help you through their acts; supererogatory acts are especially good indicators of allies.
• Enemies will pay the cost to hurt you through their acts.
• Neutral parties may hurt or help you through omissions, but since these aren't costly, they don't carry much information about whether that party is an ally or enemy; they don't seem to be thinking about you much.

From my perspective, this is a tentative debunking of these concepts' role in ethics, since allies and enemies don't belong in ethics as far as I can tell. For others, allies and enemies might be important ethical concepts, and maybe this could help them explain these concepts' role in those ethical theories.

Final note: I remember hearing about supererogatory acts' evil twins, i.e. acts that are not forbidden, but are morally blameworthy; "suberogatory acts" (search for the term on this page). These might be useful for identifying allies, who will avoid suberogatory acts, but they don't seem to play much of a role in any ethical theory.

Do reinforcement learning systems feel pain or pleasure?

Do reinforcement learning systems have valenced subjective experiences -- do they feel pain or pleasure? If so, I'd think they mattered morally.

Let's assume for the time being that they can have subjective experiences at all, that there's something it's like to be them. Maybe I'll come back to that question at some point. For now, I want to present a few ideas that could bear on whether RL systems have valenced experiences. The first is an argument that Tomasik points out in his paper, but that I don't think he gives enough weight to:

Pleasure and pain aren't strongly expectation-relative; learning is not necessary for valenced experience

An argument I see frequently in favor of RL systems having positive or negative experiences relies on an analogy with animal brains: animal brains seem to learn to predict whether a situation is going to be good or bad, and dopamine bursts (known to have something to do with how much a human likes or wants something subjectively) transmit prediction errors around the brain. For example, given an unexpected treat, dopamine bursts cause an animal's brain to update its predictions about when it'll get those treats in the future. Brian Tomasik touches on this argument in his paper. This might lead us to think that (1) noticing errors in predicted reward and transmitting them to the brain via dopamine might indicate valenced experience in humans, and by analogy (2) this same kind of learning might indicate valenced experience in machines.

However, I think there is a serious issue with this reward-prediction-learning story. This is that in humans, how painful or pleasurable an experience is is only loosely related to how painful or pleasurable it was expected to be. If I expect a pain or a pleasure, it might reduce my experience of pain or pleasure, but it doesn't seem to me that my valenced experience is closely tied to my prediction error; a fully predicted valence doesn't go away, and in some cases anticipating pain or pleasure might intensify it.

Biologically, it shouldn't be too surprising if updating our predictions of rewards isn't tightly linked to actually liking an experience. There seems to be a difference between "liking" and "wanting" an experience, and in some extreme cases liking and wanting can come apart altogether. Predicting rewards seems very likely to be closely tied to wanting that thing (because the predictions are used to steer us toward the thing), but seem less likely to be tied closely to liking it. It seems quite possible to enjoy something completely expected, and not learn anything new in the process.

In a nutshell, I'm saying something like:

In humans, the size of error between actual drive satisfaction and predicted drive satisfaction doesn't seem strongly linked to valenced experience. Valenced experience seems more strongly linked to actual drive satisfaction.

This seems to me like evidence against RL systems having valenced experience by virtue of predicting rewards and updating based on errors between predicted and actual rewards. Since this is the main thing that RL systems are doing, maybe they don't have valenced experiences.

If valenced experience doesn't consist of noticing differences between expected and actual rewards and updating on those differences to improve future predictions, what might it consist of? It still seems very linked to something like reward, but not linked to the use of reward in updating predictions. Maybe it's related to the production of reward signals (i.e. figuring out which biological drives aren't well-satisfied and incorporating those into a reward signal; salt tastes better when you're short on it, etc.), or maybe to some other use of rewards. One strong contender is reward's role in attention, and the relationship between attention and valenced experience.

The unnoticed stomachache

Consider the following situation (based on an example told to me by Luke about twisting an ankle but not noticing right away):

A person has a stomachache for one day -- their stomach and the nerves running from their stomach to their brain are in a state normally associated with reported discomfort. However, this person doesn't ever notice that they have a stomachache, and doesn't notice any side-effects of this discomfort (e.g. lower overall mood).

Should we say that this person has had an uncomfortable or negative experience? Does the stomachache matter morally?

My intuitions here are mixed. On the one hand, if the person never notices, then I'm inclined to say that they weren't harmed, that they didn't have a bad experience, and that it doesn't matter morally -- it's as if they were anesthetized, or distracted from a pain in order to reduce it. If I had the choice of giving a Tums to one person who did notice their stomachache or to a large number of people who didn't, I would choose the person who did notice their stomachache.

On the other hand, I'm not totally sure, and enough elements of discomfort are present that I'd be nervous about policies that resulted in a lot of these kinds of stomachaches -- maybe there is a sense in which part of the person's brain and body are having bad experiences, and maybe that matters morally, even though the attending/reporting part of the person never had those experiences. Imagine a human and a dog; the dog is in pain, but the human doesn't notice this. Maybe part of our brain is like the dog, and the attentive part of our brain is like the human, so that part of the brain is suffering even though the rest of the brain doesn't notice. This seems a little far-fetched to me, but not totally implausible.

If the unnoticed stomachache is not a valenced experience, then I'd want to look more at the relationship between reward and attention in RL systems. If not, then I'd want to look at other processes that produce or consume reward signals and see which ones seem to track valenced experience in humans.

Either way, I think the basic argument for RL systems having valenced experience doesn't work very well; none of their uses of reward signals "look like" pleasure or pain to me.

Should we bite Occam's Bullet?

(Close-second title: Does Occam's Razor cut too deep?)

(I told Amanda I'd post some philosophy stuff, but I've spent most of my posts this week on AI because I'm certifiably obsessed. So, here's a philosophy thing, and I'll leave out the application to AI for variety.)

I'm a little perturbed about using Occam's razor as a foundation of epistemology, especially in its computational forms. Here's the kind of reasoning I'm concerned about:

Physics test: If Michael Jordan has a vertical leap of 1.29 m, then what is his takeoff speed and his hang time (total time to move upwards to the peak and then return to the ground)?

Student: According to the simplest explanation, Michael Jordan has formed randomly from thermal or quantum fluctuations, and the small bubble of order he inhabits will collapse back into background heat long before he touches the ground.

You would probably not get extra credit for rigor or consistency in answering this question!

My basic worry is that the simplest explanation for a set of observations may be something that doesn't fit with any of my normal beliefs about my situation. This is because simple explanations can expand into vast universes, and in these universes there could be many instances of my circumstances (or something observer-independent, like the Michael Jordan problem above) that are nothing like what I believe to be my current situation; they could be in simulations, part of programs numerating all possible computations in order, fluctuations of some very long-lasting, near-equilibrium cosmological state, or something stranger.

(I don't think the problem goes away when you consider the set of all explanations compatible with observations, weighted by their simplicity, but I might be wrong.)

Of course, people could just as well have had my complaint when physics was just being discovered; our view of what the universe is and our place in it would probably appear extremely weird to them, violating many of their normal beliefs about their situation. Heck, the implications of quantum physics are weird enough to me now. So maybe I'm just being stubborn, and I should bite Occam's bullet and think that most of my normal beliefs about my situation are wrong.

So why don't I think we should use this kind of reasoning? I could have epistemic or instrumental reasons, I could actually be asking a different question from "what is the most likely explanation", or I could use some kind of anthropic reasoning.

• Epistemic: I don't feel like I really believe that the most likely explanation is that I'm a Boltzmann brain; I feel like I have evidence that says otherwise. However, that evidence could be fabricated, which is a big problem -- I may just have an unjustified belief that I'm not a Boltzmann brain! Should I bite Occam's bullet?
• Instrumental: if I am in a Boltzmann brain, things I do matter only over very small timescales (until the bubble collapses).
• Different question: maybe instead I want to know something like "conditioning on some other assumptions (like that most of my evidence is "real", whatever that means), what is the most likely explanation?" This actually doesn't seem so bad; it's the most appealing answer to me at the moment.
• Anthropic reasoning: I'm not particularly satisfied with this, because I'd like questions about situations without observers -- e.g. physics problems like the Michael Jordan problem above (well, versions without MJ the observer!) -- to have "reasonable" answers, instead of silly ones. In fact, that might be the most interesting part of this post -- that these problems seem like they can't be answered fully by anthropics, if we want to answer observer-free questions "sensibly".

I do like the idea of re-framing the basic epistemic question ("what is the best explanation for x, and what does this imply we should expect in x's future"), but I'm not sure where to go from there. Perhaps in future posts!