Updated version of an AI decision theory problem

Let's say we have a model-based RL system in a peculiar episodic environment: at the start of each episode, the system is copied onto another computer, and the copy's actions matter for what happens in the environment (e.g. they are playing some kind of game).

A "correct" model (state, action) → (new state) seems like it should have two properties:

1. If something causes the copy to behave differently from the system -- e.g., the copy is made incorrectly, the computer the copy runs on malfunctions, or the copy is interfered with during its decision -- the model should predict what the malfunctioning copy will actually do.
2. "Otherwise", the model should predict that the system and the copy perform the same action; that is, when the model predicts the copy's action in order to predict new state, the copy's action should be identical to the model's input action.

There are two kinds of models that I think it is reasonable to imagine our RL system making: abstract-ish models and physical-ish models. It seems like abstract-ish models will do well with property 2, and physical-ish models will do well with property 1. I can't picture the type of model that will get both properties.

• When the system has a fairly abstract and non-physical model (state, action) → (new state), it can simply model the copy's action as if it directly depended on the system's action, and it will correctly predict that the system and copy will always act the same way. However, this kind of model will make wrong predictions if something goes physically wrong with the copy.
• As the system learns a model that correctly handles things going wrong with the copy, it will probably no longer model the copy's actions as directly dependent on the system's action (in part because this is an accurate model of the physical setup). However, this kind of model seems like it will not predict that the system and the copy perform the same action when nothing goes wrong with the copy, since the copy will perform the same physical steps to make its decision regardless of what the input action is.

As the system learns, if things do go wrong with the copy, it will probably get evidence that pushes it toward physical-ish models, so that it can predict those things going wrong. However, it seems like there are common cases where a model fails property 2, but doesn't receive evidence to push it to correct this. For example:

Suppose the system is considering which action to take in state s. It considers actions a1, a2, a3. Using a physical-ish model, it predicts that the copy will take action a1 independent of the system's choice, so that possible (system choice, copy choice) pairs are (a1, a1), (a2, a1), (a3, a1), the second and third of which are mispredictions. If the system prefers the state resulting from (a1, a1) to the states resulting from (a2, a1) or (a3, a1), it will choose a1, and will never receive feedback that if it had chosen a2, the copy would have chosen a2 as well, and ditto for a3.

Intuitively, the model can't learn "what would have happened" correctly, and will need to rely on generalization in order to get this right. I don't know what kind of generalization would produce a model that does this correctly. This kind of problem will be especially bad if there are equilibria that result in the system consistently choosing the same action, as in the Prisoner's Dilemma (where the system never realizes that if it cooperated, the copy would cooperate as well). However, even outside this kind of problem, I still don't know how to make a model that fulfills properties 1 and 2.

This is just a toy case, and I expect analogous problems to come up in analogous situations (e.g. situations where there is not an exact copy, or where the "copy" is another system, human, or market that is learning to predict the system's actions). I don't know how much I should expect this kind of problem to come up in model-free approaches, but it seems worth looking into, and it would be disappointing if this kind of problem blocked us from using model-based approaches at all.

Failing to learn about counterfactuals

Let's say that we have a model-based RL system doing episodic RL in the following environment:

• At the start of each episode, the system is copied onto another computer
• The system and its copy play one round of the Prisoner's Dilemma game
• At the end of the episode, the situation is reset (the copy is erased)

When the system has a fairly abstract and non-physical model that maps (state, action) → (state), it can simply model the copy's action as if it directly depended on the system's action. So, when it predicts what would happen if it cooperated, it will predict that the copy will cooperate as well, and likewise for defection.

However, we might hope that the system will gradually learn a more physically realistic model of the world. This model won't contain any physical pathway linking the system's action to the copy's action (since the copy was made before the Prisoner's Dilemma is played), and will allow for the copy system to be interfered with in a variety of ways, breaking the symmetry of the game.

Clearly, this model can't be perfect -- if the system had to predict what a perfect copy would do before it acted, it would fall into an infinite regress. The system will need some way around this, like a limited model class, a contingency for long-running models, or some way of recognizing these kinds of situations (though halting-like problems seem to keep it from recognizing all situations of a similar type). If it can successfully recognize this situation, it seems like it "should" assume that the copy will take the same action as it does, and the problem is resolved. That would be nice! However, I don't think that models the way we build them currently will do this by default.

Let's assume that the model is not totally accurate, either because of a limited model class or because the system defaults to some estimate of the next state when a model takes too long to run. Now, when the system predicts what the copy will do, this prediction is independent of the system's action. Without loss of generality, let's say that the system predicts the copy will defect.

Now, the system needs to make a decision. It will evaluate cooperation and defection, both under the prediction that the copy will defect, and in this case it will choose defection. After it makes this choice, both it and the copy will in fact defect, reinforcing the system's model. However, the situation that the system predicted would follow from the action it didn't take -- the situation where it cooperates where the other system defects -- wouldn't actually have happened! Since the system and its copy are identical, they will always behave the same way; the model has predicted incorrectly. Furthermore, it can never witness this incorrect prediction, making it very hard to correct (dependent on some kind of generalization or regularization perhaps?).

In the case I've given above, the system and its copy miss out on some reward as a result of this inaccuracy -- they "could have" both cooperated and gotten better rewards. However, the failure could just as well be harmless (though still unsettling) -- if the system's imprecise model predicts the copy will cooperate, it will cooperate as well, and everything will be fine. I think there are stranger situations where the system's predictions will always be wrong (for halting-problem reasons).

I hope I'll be able to post a simpler version of this problem in the future -- this one is a little too long-winded to be useful!

Counterpossibles

I'm casually interested in counterpossible reasoning (or "logical counterfactuals"), asking what would be true if something logically impossible were true. Here's an example due to Carrie Jenkins:

1. If the square root of 2 were rational, it could be expressed as n/m, with n, m integers.
2. If the square root of 2 were rational, it couldn't be so expressed.

It sure seems to me, and to her, that 1 is true and 2 is false. However, most logics don't deal with this sort of thing naturally, i.e. with normal conditionals; if we assume that root 2 is rational, then prove that it is irrational, then we can prove any other statement from this contradiction -- if you build counterpossible worlds this way, they all turn out to be worlds where everything is both true and false.

I'd especially like to know about counterpossible reasoning because it seems like it could be part of a theory of reasonable decision-making. Suppose that a computer program is making a decision; it should evaluate what would happen if it chose option 1, 2, or 3. However, since it is a program, it actually only chooses one of these options, so all but one of these choices are logically false (or maybe they are all false, if the program fails to halt). The program should be using counterpossible reasoning to figure out what would happen if it made each choice.

It seems suggestive to me that the program "doesn't yet know" which choice it will make while it's reasoning, and so it will reason the same way for the choices it will choose and the choices it won't. Maybe this gives us some clue, or limits the ways that it can do counterpossible reasoning? On the other hand, this kind of counterpossible reasoning might be specific to decision-makers, and be unsuitable for e.g. imagining counterpossible mathematical systems like Carrie's root-2-rational world I gave above.

⁂

I have a few desiderata for counterpossible reasoning (which I really haven't done serious reading on, so for all I know these are well-known proposals):

1. There should be exactly one world W defined by counterpossibly assuming S. (A world is a set of assignments of truth values to all statements.)
2. W should assign each statement either "true" or "false", not both or neither.
3. S should be true in W.
4. If S is actually true, then W should just be the normal assignment of truth values to statements.
5. If S is not actually true, then in most cases W should assign other statements S', S'', etc. the opposite of their normal truth values. These are the "counterpossible consequences" of S. All other statements are "counterpossibly independent" of S.

5 is interesting, because it means that we can't go for a W that is minimally different from the actual world in its truth assignments; this would be the world where only S is different, which is not very satisfying. These desiderata clearly aren't enough to pin down a particular way of reasoning; defining 5 better is an obvious next step, but I'm not sure how to do it.

⁂

One possibility that is tempting to me is to work in a proof system P instead of a complete world W (which are different because of incompleteness).  To counterpossibly assume S, add S as an axiom of P, and add a rule to P that says something like "whenever a rule in P says you could conclude not-S, conclude S instead". I think this satisfies 2, since we can never conclude not-S and the proof system is otherwise unchanged. I'm somewhat worried about 4; the mangled proof system seems like it could have trouble with proofs by contradiction, since it can't assume Q and conclude not-S to get a contradiction. This construction would also make statements not provable in P counterpossibly independent of S relative to P, which seems OK to me. Is this construction satisfactory overall? I'm not sure, and I'm out of blogging time for today! Maybe I'll come back to it -- writing this post makes me more excited about the possibility.

I'll leave you with something that seems to me like a nice application of counterpossible mathematical reasoning: