Skip to content

SimStore performance improvements and fixes - #1157

Merged
dwhswenson merged 4 commits into
openpathsampling:masterfrom
dwhswenson:uuid-indices
Sep 27, 2024
Merged

dwhswenson merged 4 commits into
openpathsampling:masterfrom
dwhswenson:uuid-indices

Conversation

@dwhswenson

Copy link
Copy Markdown
Member

This includes several performance improvements and minor fixes for SimStore. Quick manifest:

  • Added and started using is_storage_string and has_storage_uuid, which use the fast ClassIsSomething model to cache classes with given attributes. The normal is_string and has_uuid methods were taking a larger fraction of time than they should.
  • Added default support for SQL index on UUID (which should have been there all along).
  • Fix an edge case where old-style CVs were no reloaded correctly in SimStore, because the new JSON serialization was already making a callable and the code to make a callable from the marshal data was no longer needed.

This also helps clear up some of the external testing in ops-storage-notebooks which has been off for a while.

@codecov

codecov Bot commented Aug 30, 2024 •

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.28%. Comparing base (bbfe5d7) to head (0aeb5b1).
Report is 5 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1157      +/-   ##
==========================================
- Coverage   81.29%   81.28%   -0.01%     
==========================================
  Files         142      142              
  Lines       15643    15643              
==========================================
- Hits        12717    12716       -1     
- Misses       2926     2927       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dwhswenson

Copy link
Copy Markdown
Member Author

This is ready for review and comment. Without review, I will merge this after at least 18 hours, no sooner than Sat 31 Aug 11:00 GMT (13:00 my local).

@sroet sroet left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One import can be dropped, feel free to ignore. LGTM otherwise

)
from .proxy import GenericLazyLoader
from .uuids import (
has_uuid, get_uuid, set_uuid, encode_uuid, decode_uuid, encoded_uuid_re,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't has_uuid be dropped from the imports? (only reference is in commented out code)
Bit surprised linting did not catch this

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out this was trickier than it appeared. In an early version of the file, has_uuid was defined in serialization_helpers.py. Then I split it off into uuids.py. But I allowed the transitive import.

Fixed that in 8bf1f31.


universal_sql_meta = {
'uuid': {'uuid': {'primary_key': True}},
'uuid': {'uuid': {'primary_key': True, 'index': True}},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥳

@dwhswenson
dwhswenson merged commit 474e9e6 into openpathsampling:master Sep 27, 2024
@dwhswenson
dwhswenson deleted the uuid-indices branch September 27, 2024 00:51
@dwhswenson dwhswenson mentioned this pull request Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants