Organize by ensemble strategy - #320

What this PR does: Makes it possible to rearrange the “global” structure of the move decision process. Standard behavior is to select a move type, then a specific move within that. Now we can select an ensemble first, then a move within that. We can also switch between these.

Why we should have this: Ensemble-first organization is a step toward SRTIS. The ability to conserve move probabilities when switching selection order makes comparison between SRTIS and normal RETIS more straightforward, and makes it easy for users to switch from one approach to the other.

What is hard about this: My intuition expects different behaviors between ensemble-first and move group-first organization. Details below, but figuring our that there was a difference, and then figuring our exactly what the difference was took some effort (and a major rewrite after my first draft of this).

Tasks:

• EnsembleDictionaryMover
• OrganizeByEnsembleStrategy
• OrganizeByMoveGroupStrategy (rename of Default)
• Check that we get the correct weights regardless of top-level organization strategy
• Update docstrings

Background

Before getting into the details of what this PR contains, let me refresh a few essential background points. A few of these came from #289. The big thing is that movers can be uniquely defined by (groupname, ensemble_signature), and that the main thing that comes out of the highest-level organization strategy is the choice_probabilities, which is a dictionary of {PathMover : probability} .

Goals

• code should allow different intuitive behavior for move-type organization and for ensemble organization
• scheme.choice_probabilities should be preserved (unless explicitly overridden)
• general schemes should have a reasonable default weights

Intuitive behavior differs by global organization type

This is one of the weirder parts of this PR, but it basically boils down to this: when I set the move-type weights in a move-type-first organization scheme, I think of a dictionary like this:

group_weights = {
    ‘shooting’ : 1.0,
    ‘repex’ : 0.5,
    ‘pathreversal’ : 0.5,
    ‘minus’ : 0.2
}

The tricky bit is that there may be 5 shooting movers for every minus mover. If the weights represented “probability of choosing this move type”, then we’d do 5 shooting moves for every minus move. But since there are 5 different shooting movers, that would mean each individual shooting mover was used as frequently as the minus mover.

That’s not what I want. I want the minus move to be done less frequently than shooting: about 1/5 as often. So rather than make the user do the math to get that to work (as my old code did), I make it so that the code figures it out automatically. This means that (assuming all ensembles have the same weight) each shooting move will be 5 times as likely to occur as a minus move. That’s what we want.

On the other hand, ensemble-first organization does just what you’d expect: first you pick an ensemble with the random probability given by ensemble_weights, then you pick a move which has that ensemble as one of its initial ensembles.

Ensemble-first organization does not have a unique solution

The quantity that is conserved is the total choice probability for each ensemble. However, some ensembles (e.g., replica exchange) have more than one input ensemble. This means that they show up in more than one place in the ensemble-first decision process. There is not a unique way to split these weights, so the approach I’ve used is to say that each ensemble contributes the same to the total probability of choosing that mover. Details on this problem, and the approach I used to solve it, are in this gist.

Reasonable defaults

On one hand, it would make sense to normalize most of these results: so the probabilities should be normalized probabilities. On the other hand, that’s harder as a way to think about it -- and obviously doesn’t work with the group_weights described above.

So the approach I’ve taken is the following:

1. choice_probability really is a normalized probability
2. group_weights is rescaled such that ’shooting’ is 1.0 (we usually think in terms of shooting movers); if there is no group called ’shooting’, then we rescale so that the most common value in the group_weights dictionary is 1.0.
3. mover_weights are rescaled so that the most common number within each second-level decision is 1.0.

These are so that, if you say “I want this twice as much shooting in ensembleA as ensembleB,” you can just change the default 1.0 to 2.0.

Caveats when overriding

There are two levels of weights to override -- the weight that is sorted by (either move group or input ensemble) and the weight within that sorting group.

For the sorting groups, overriding the weights are easy: the input dictionaries take the group name or the ensemble and map it to a weight (group_weight or ensemble_weight).

For the second level, this is a little more complicated. First, note that these weights should only be used by pretty advanced users -- typical needs are met without them. These weights are referred to as mover_weights, and the shape of the dictionary keys is different between the two types of organization strategy. For OrganizeByMoveGroupStrategy, the keys are (groupname, ensemble_signature); for OrganizeByEnsembleStrategy, they are (groupname, ensemble_signature, ensemble).

In both cases, the mover_weights default to 1.0, as does ensemble_weights. However, group_weights defaults to the version shown above. This means that to get the standard default behavior with ensemble-first organization, your scheme needs an OrganizeByMoveGroupStrategy, followed by an OrganizeByEnsembleStrategy. (This will eventually be hidden to SRTIS users, who will just ask for an SRTISScheme that handles all of that automatically.)