Skip to content

Organize by ensemble strategy #320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jhprinz merged 60 commits into from
Oct 14, 2015
Merged

Organize by ensemble strategy #320

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
b8dd8db
Make OrgByEnsStrat subclass of DefaultStrat
dwhswenson Jun 29, 2015
b1af386
Merge branch 'balance_and_choice_prob' into organize_by_ensemble_stra…
dwhswenson Jun 29, 2015
97abb06
Draft of OrgBeEns.make_chooser
dwhswenson Jun 30, 2015
3e4dba9
Tests for OrgByEns.make_chooser
dwhswenson Jul 1, 2015
52ef7e3
Merge remote-tracking branch 'upstream/master' into organize_by_ensem…
dwhswenson Jul 2, 2015
6517311
Starting skeleton of OrgByEns.make_movers
dwhswenson Jul 2, 2015
d07dba4
Merge branch 'move_scheme_cleanup' into organize_by_ensemble_strategy
dwhswenson Jul 6, 2015
29003eb
Fix for tests
dwhswenson Jul 6, 2015
fd8bda1
Early draft of EnsembleDictionaryMover
dwhswenson Jul 7, 2015
72e72b2
EnsDict runs, though tests don't check correctness
dwhswenson Jul 7, 2015
2821064
Working test for EnsDict with single replica
dwhswenson Jul 7, 2015
3db71f6
Tests for EnsDict with multiple replicas
dwhswenson Jul 7, 2015
77a6b87
Skeletons of OrgByEns and tests. Needs some redo.
dwhswenson Jul 7, 2015
0c47813
Cleaned up Default.make_chooser (now reusable?)
dwhswenson Jul 8, 2015
13a7849
Generalized choice_probability
dwhswenson Jul 8, 2015
57a3719
Reorg most of get_weights subsys to be general
dwhswenson Jul 8, 2015
122a9d4
I think that fully generalizes get_weight subsys
dwhswenson Jul 8, 2015
3eda6f0
Generalized mover keys. Ready to build new strat
dwhswenson Jul 8, 2015
960deb3
Draft of OrgByEns.make_movers
dwhswenson Jul 9, 2015
12cc5d2
first test of OrgByEns looks decent
dwhswenson Jul 9, 2015
5f20b9e
Upgraded EnsDictMover to support ensemble prob
dwhswenson Jul 9, 2015
6ab53ae
Test for weights in EnsDict
dwhswenson Jul 9, 2015
1eeca62
Test when we change the ensemble weight
dwhswenson Jul 10, 2015
8612f7e
Serious clean-up of mover weights normalization
dwhswenson Jul 10, 2015
77056fe
Added test for changing mover_weights in OrgByEns
dwhswenson Jul 10, 2015
2c81ab1
Various test fixes
dwhswenson Jul 10, 2015
fd58ca3
Committing screwy version before rewrite
dwhswenson Sep 30, 2015
604c8de
New version passes the first test....
dwhswenson Oct 1, 2015
34d4980
New version of choice_probability w/ tests
dwhswenson Oct 2, 2015
76a1a99
New make_movers in DefaultStrategy
dwhswenson Oct 2, 2015
6178d72
DefaultStrategy.weights_from_choice_prob & tests
dwhswenson Oct 2, 2015
2fb17b6
Fixed setting group weights when no shooting group
dwhswenson Oct 2, 2015
8e0bd42
Down to one test for DefaultStrategy
dwhswenson Oct 3, 2015
6a9d671
All tests for DefaultStrategy pass
dwhswenson Oct 3, 2015
2414a98
Looks like OrgByEns.choice_probability works
dwhswenson Oct 3, 2015
e55c7bf
More untested bits of code for OrgByEns
dwhswenson Oct 3, 2015
0d7c775
Fix for ensemble sig problems in test_choice_prob
dwhswenson Oct 4, 2015
0eca9cc
Test for OrgByEns.default_weights
dwhswenson Oct 4, 2015
cab0b2d
Test for chooser_movers
dwhswenson Oct 5, 2015
43302c4
Most of the OrgByEns.weights_from_choice_prob
dwhswenson Oct 5, 2015
67e861d
Separate out most_common calcs for renorming
dwhswenson Oct 6, 2015
4d97b55
OrgByEns.weights_from_choice_prob passes test
dwhswenson Oct 6, 2015
d9db71e
OrgByEns.make_movers (and test)
dwhswenson Oct 6, 2015
adc3e20
Fix bug in EnsDictMover test
dwhswenson Oct 6, 2015
180ba9c
Removed worthless old crap.
dwhswenson Oct 6, 2015
afa494c
DefaultStrategy => OrganizeByMoveGroupStrategy
dwhswenson Oct 6, 2015
daedfc0
Merge remote-tracking branch 'upstream/master' into organize_by_ensem…
dwhswenson Oct 7, 2015
c403c17
Partial fixes for unique ensemble stuff
dwhswenson Oct 7, 2015
a51df68
Fix the rest of stuff unique ensemble messed up
dwhswenson Oct 7, 2015
3f4676f
Removed EnsembleDictionaryMover
dwhswenson Oct 7, 2015
e75f393
testEnsDictMover => testRandomAllowedChoiceMover
dwhswenson Oct 7, 2015
ec36501
Outline of test for switching org strategies
dwhswenson Oct 8, 2015
75388d8
Can switch between global org strategies
dwhswenson Oct 8, 2015
e497fa9
Docstrings for MoveStrategies (part 1)
dwhswenson Oct 8, 2015
e12c0b9
No wonder it was failing. Too many RETIS steps!
dwhswenson Oct 8, 2015
2409601
Enough docstrings for now
dwhswenson Oct 8, 2015
ea1221d
Docstrings & some PEP8
dwhswenson Oct 12, 2015
d27ab0f
Merge remote-tracking branch 'upstream/master' into organize_by_ensem…
dwhswenson Oct 12, 2015
0cb98d9
Removed unused mover_key
dwhswenson Oct 12, 2015
5948097
First switch to using dict.get(key, default)
dwhswenson Oct 13, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
New version passes the first test....
  • Loading branch information
@dwhswenson
dwhswenson committed Oct 1, 2015
commit 604c8de8f9a57457e5004cb4be528409659075af
72 changes: 30 additions & 42 deletions openpathsampling/analysis/move_strategy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,56 +483,44 @@ def normalize_mover_weights(self, scheme, sorted_movers, mover_weights):
}
return mover_weights

def default_weights(self, scheme):
sortkey_w = {}
movers_w = {}
if scheme.choice_probability != {}:
# extract weights from the choice probability
pass
else:
# generate absolutely generic weights
for skey in scheme.movers.keys():
try:
sortkey_w[skey] = self.default_group_weights[skey]
except KeyError:
sortkey_w[skey] = 1.0

@jhprinz jhprinz Oct 13, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a nice trick to write this in one line.

sortkey_w[skey] = self.default_goup_weights.get(key, 1.0)

use the .get(key, default) method. I think effectively it does a try and uses the default otherwise.

@dwhswenson dwhswenson Oct 13, 2015

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wish I'd known about that sooner! I use that idiom all the time. Added a commit here to switch this one; over time we can replace the many other times I've used that.

for mover in scheme.movers[skey]:
total_sig = (skey, mover.ensemble_signature)
movers_w[total_sig] = 1.0
return (sortkey_w, movers_w)

def get_weights(self, scheme, sorted_movers, preset_sortkey_weights):
def override_weights(self, weights, override_w):
for key in override_w.keys():
weights[key] = override_w[key]
return weights


def get_weights(self, scheme, sorted_movers, sort_weights_override={},
mover_weights_override={}):
"""
Gets sort_weights and mover_weights dictionaries.

Notes
-----
The group_weight gives the relative probability of choosing a group;
the mover_weight gives the relative probability of choosing a mover
*within* its group.

Note that only the variables returned from this tell the full story.
The defaults and the self.* version may not contain the real set of
groups/movers.
"""
choice_prob_set = (scheme.choice_probability != {})
sort_set = (preset_sortkey_weights != {})
mover_set = (self.mover_weights != {})
if (sort_set and mover_set) or not choice_prob_set:
sorted_weights = self.strategy_sortkey_weights(
scheme, sorted_movers, preset_sortkey_weights
)
mover_weights = self.strategy_mover_weights(scheme,
sorted_movers)
elif sort_set: #choice_prob is set; mover is not set
# use sorted_weights & choice_probability to set mover_weights
sorted_weights = self.strategy_sortkey_weights(
scheme, sorted_movers, preset_sortkey_weights
)
mover_weights = self.strategy_mover_weights(scheme,
sorted_movers,
sorted_weights)
elif mover_set: #choice_prob is set; sort is not set
# use mover_weights & choice_probability to set group_weights
mover_weights = self.strategy_mover_weights(scheme,
sorted_movers)
sorted_weights = self.strategy_sortkey_weights(
scheme, sorted_movers, preset_sortkey_weights, mover_weights
)
else: #choice_prob is set, neither group nor mover is set
# use default mover weights to get the group weights, then use
# that to get the actual correct mover weights
m_weights = self.strategy_mover_weights(scheme, sorted_movers)
sorted_weights = self.strategy_sortkey_weights(
scheme, sorted_movers, preset_sortkey_weights, m_weights
)
mover_weights = self.strategy_mover_weights(scheme,
sorted_movers,
sorted_weights)

(sorted_w, mover_w) = self.default_weights(scheme)
sorted_weights = self.override_weights(sorted_w, sort_weights_override)
mover_weights = self.override_weights(mover_w, mover_weights_override)
return (sorted_weights, mover_weights) # error if somehow undefined


Expand Down Expand Up @@ -567,7 +555,7 @@ def make_movers(self, scheme):
(group_weights, mover_weights) = self.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=self.group_weights
sort_weights_override=self.group_weights
)
choosers = []
for group in scheme.movers.keys():
Expand Down Expand Up @@ -634,7 +622,7 @@ def make_movers(self, scheme):
(ensemble_weights, mover_weights) = self.get_weights(
scheme=scheme,
sorted_movers=ensemble_movers,
preset_sortkey_weights=self.ensemble_weights
sort_weights_override=self.ensemble_weights
)
ens_list = mover_weights.keys() # used for canonical ordering
# (otherwise the weight list isn't in the same order as choosers!)
Expand Down
51 changes: 30 additions & 21 deletions openpathsampling/tests/testmovestrategy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -530,14 +530,20 @@ def test_get_weights_scheme_all_unset(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
assert_equal(group_weights, {'shooting' : 1.0, 'repex' : 0.5})
for group in mover_weights:
for sig in mover_weights[group]:
assert_equal(mover_weights[group][sig], 1.0)
assert_in(sig, all_movers_sigs)
assert_equal(len(mover_weights[group]), len(scheme.movers[group]))

# check that the number of sigs in a each group matches the number
# of movers in that group
for group in group_weights.keys():
mover_sigs = [sig for sig in mover_weights.keys()
if sig[0]==group]
assert_equal(len(mover_sigs), len(scheme.movers[group]))

for sig in mover_weights.keys():
assert_equal(mover_weights[sig], 1.0)
assert_in(sig[1], all_movers_sigs)

# check that we can reuse these in a different scheme
scheme2 = MoveScheme(self.network)
Expand All @@ -549,15 +555,18 @@ def test_get_weights_scheme_all_unset(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme2,
sorted_movers=scheme2.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
assert_equal(group_weights, {'shooting' : 1.0})
for group in mover_weights:
for sig in mover_weights[group]:
assert_equal(mover_weights[group][sig], 1.0)
assert_in(sig, [m.ensemble_signature
for m in scheme2.movers[group]])
assert_equal(len(mover_weights[group]), len(scheme2.movers[group]))
for sig in mover_weights:
assert_equal(mover_weights[sig], 1.0)
assert_in(sig[1], [m.ensemble_signature
for m in scheme2.movers[sig[0]]])

for group in scheme2.movers.keys():
mover_sigs = [sig for sig in mover_weights.keys()
if sig[0]==group]
assert_equal(len(mover_sigs), len(scheme2.movers[group]))

def test_get_weights_both_internal_weights_set(self):
strategy = DefaultStrategy()
Expand All @@ -581,7 +590,7 @@ def test_get_weights_both_internal_weights_set(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
root = scheme.move_decision_tree(rebuild=True)

Expand Down Expand Up @@ -641,7 +650,7 @@ def test_get_weights_group_weights_set(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
assert_equal(group_weights, {'shooting' : 2.0, 'repex' : 0.5})
for group in mover_weights:
Expand Down Expand Up @@ -671,7 +680,7 @@ def test_get_weights_mover_weights_set(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
assert_equal(group_weights, {'shooting' : 1.0, 'repex' : 3.0})

Expand All @@ -684,7 +693,7 @@ def test_get_weights_mover_weights_set(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
#root = scheme.move_decision_tree(rebuild=True)

Expand Down Expand Up @@ -716,7 +725,7 @@ def test_get_weights_internal_unset_choice_prob_set(self):
(old_group_weights, old_mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)

old_choice_prob = scheme.choice_probability
Expand All @@ -727,7 +736,7 @@ def test_get_weights_internal_unset_choice_prob_set(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
scheme.choice_probability = {}
new_choice_prob = strategy.choice_probability(
Expand Down Expand Up @@ -775,7 +784,7 @@ def test_get_weights_mover_weights_set_no_shooting(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
assert_equal(group_weights, {'pathreversal' : 1.0, 'repex' : 3.0})
ensA = self.network.sampling_transitions[0].ensembles[0]
Expand All @@ -787,7 +796,7 @@ def test_get_weights_mover_weights_set_no_shooting(self):
(group_weights, mover_weights) = strategy.get_weights(
scheme=scheme,
sorted_movers=scheme.movers,
preset_sortkey_weights=strategy.group_weights
sort_weights_override=strategy.group_weights
)
for sig in mover_weights['pathreversal']:
weight_sigA = mover_weights['pathreversal'][ensA_sig]
Expand Down