Repository navigation
Messagepack breaks db decode logic #299
Description
I tried to get a db using tree-sitter graphs working, but I encountered an issue pretty quickly where it would save successfully but it wouldn't be able to load the database back using load_graph_for_file_or_directory successfully again.
Here's a couple issues I've noticed:
There's a place where the code refers to a json column which doesn't exist anymore, shouldn't this be value instead?
stack-graphs/stack-graphs/src/storage.rs
Line 504 in d15b259
In addition, even after this issue is fixed, the MessagePack serialization is currently using rmp_serde::to_vec, and as a result the deserializer encounters issues due to the use of a lot of #[serde(skip_serializing_if = "Option::is_none")] in the StackGraph structure.
Since its all unlabeled array elements, the MessagePack decoder fails to parse fields in the right order correctly, giving a cryptic error:
invalid type: integer `1`, expected a string
It might be a good idea to use rmp_serde::to_vec_named instead or to move back to JSON again, or alternatively remove the skip serializing option on the fields (but if going with this option it would be good to have a thorough check to ensure it doesn't have any edge cases).
Activity
hendrikvanantwerpen commentedon Jul 27, 2023
Thanks for the report and the pointers to what's going wrong. I have a draft PR open where I'm trying to address this. The main blocker for that is that the msgpack create seems to have quite some open issue and I was considering if another binary is more stable and we should switch.
bluskript commentedon Jul 27, 2023
Hi, it seems like with the newly merged changes deserialization still doesn't work.. Postcard returns a WontImplement error which can only happen in the case of
deserialize_identifier,deserialize_any, anddeserialize_ignored_any... The type is known at compile time so its probably not deserialize_any, could it require deserializing identifiers for enums and such maybe?bluskript commentedon Jul 27, 2023
For simplicity it might be worth considering just using https://github.com/rkyv/rkyv and having to reindex whenever the structure changes.. it would guarantee the fastest write and read speeds although it wouldn't have very good forwards compatibility