You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Allow to save and load all Axis and Group objects of a session in/from HDF, CSV and EXCEL files #578
@gdementen Implementing #81 (LFrame) and #6 (multiversion labels) would greatly help to spread the use of LArray among our (potential) current users but in the meantime it is possible to mix Pandas and LArray. Unfortunately, Session.save and Session.load do not handle Pandas objects.
It is really easy to modify Session.save to be able to also save Pandas objects but it is more complicated to do the same for Session.load since we do not know the type of the loaded objects at loading time (we currently assume they are LArray objects).
I suggest to adapt the argument names of Session.load to accept dict with containing pairs 'name: type'. What do you think?
I do not really like this option (but this might be the best option anyway -- this needs more thoughts). At least I hope it would not be required. Can't we autodetect what we have? At least for .h5, we could store some extra metadata to make this possible. For Excel & .csv, we could return an array when \ is present and a Dataframe when \ is not present. That would be backward incompatible though and would break again when we support LFrame.
Also names are ordered while the dict would not be guaranteed to be so on python < 3.7
The whole point behind this is to be able to save Axis and Group objects. Maybe a special object to simulate #6 also.
For HDF files, we can indeed use metadata.
For CSV and Excel, we need a tag or something to tell what it is at reading time.
changed the title [-]Include Pandas objects in Session.save() and Session.load()[/-][+]Allow to save and load Axis and Group objects in/from external files (HDF, CSV, EXCEL)[/+]on Feb 14, 2018
@gdementen
Why not using group_name@associated_axis_name, list, of, labels and @axis_name, list, of, labels CSV files?
Or using a more explicit notation: @Group, group_name, associated_axis_name, list, of, labels and @Axis, axis_name, list, of, labels?
I don't see any obvious way to guess if we are dealing with a group or an axis or a 1D array when reading data from a CSV file or Excel sheet.
The @ idea is interesting, but I think we are looking at it from the wrong angle. There are potentially several different features involved here.
First, if we are loading one axis (or one group) from a file, the user should know what s/he is loading and could use a specific function to load that. eg read_axis(), read_group(), or similar.
It is probably valuable to have a format to save/load those one at a time from a custom format, and in that case the @ solution you describe seems a good compromise between readabilty, simplicity and functionality.
FWIW, I prefer to see "realistic" examples to gauge syntaxes. Your above proposals would be:
to save and load back an entire session containing arrays and groups in whatever format we like. In this case, I don't think a format for a single Axis or Group would help. We need to define a format to encode multiple axes/groups in a single file. HDF5 seems the easiest to implement, but we could also define a custom format for Excel, something a special sheet containing all axes and a special sheet containing all groups would probably be the most convenient for users. I thought this issue was about this part.
HDF5 seems the easiest to implement, but we could also define a custom format for Excel, something a special sheet containing all axes and a special sheet containing all groups would probably be the most convenient for users
That was my first thought actually but I see users coming. I'm pretty sure some of them will ask to export arrays with associated groups in the same sheets and then being able to reload arrays and groups in one operation. Imagine an array called pop and associated groups like teenagers and pensioners. I'm quite sure some users will not like to have teenagers and pensioners groups and the pop array stored in separated CSV files or Excel sheet.
Another stuff I'm worried about: Do we force Groups and Axis to be stored vertically or horizontally?
changed the title [-]Allow to save and load Axis and Group objects in/from external files (HDF, CSV, EXCEL)[/-][+]Allow to save and load all Axis and Group objects of a session in/from HDF, CSV and EXCEL files[/+]on Mar 22, 2018
The name of the Sheet/CSV file/HDF group for axes and groups could be defined by two additional arguments with default values: pathaxes=__axes__ and pathgroup=__groups__ (something like that).
Activity
alixdamman commentedon Feb 13, 2018
@gdementen Implementing #81 (LFrame) and #6 (multiversion labels) would greatly help to spread the use of LArray among our (potential) current users but in the meantime it is possible to mix Pandas and LArray. Unfortunately,
Session.saveandSession.loaddo not handle Pandas objects.It is really easy to modify
Session.saveto be able to also save Pandas objects but it is more complicated to do the same forSession.loadsince we do not know the type of the loaded objects at loading time (we currently assume they are LArray objects).I suggest to adapt the argument
namesofSession.loadto accept dict with containing pairs 'name: type'. What do you think?gdementen commentedon Feb 13, 2018
I do not really like this option (but this might be the best option anyway -- this needs more thoughts). At least I hope it would not be required. Can't we autodetect what we have? At least for .h5, we could store some extra metadata to make this possible. For Excel & .csv, we could return an array when \ is present and a Dataframe when \ is not present. That would be backward incompatible though and would break again when we support LFrame.
Also names are ordered while the dict would not be guaranteed to be so on python < 3.7
alixdamman commentedon Feb 13, 2018
The whole point behind this is to be able to save Axis and Group objects. Maybe a special object to simulate #6 also.
For HDF files, we can indeed use metadata.
For CSV and Excel, we need a tag or something to tell what it is at reading time.
[-]Include Pandas objects in Session.save() and Session.load()[/-][+]Allow to save and load Axis and Group objects in/from external files (HDF, CSV, EXCEL)[/+]alixdamman commentedon Mar 19, 2018
@gdementen
Why not using
group_name@associated_axis_name, list, of, labelsand@axis_name, list, of, labelsCSV files?Or using a more explicit notation:
@Group, group_name, associated_axis_name, list, of, labelsand@Axis, axis_name, list, of, labels?I don't see any obvious way to guess if we are dealing with a group or an axis or a 1D array when reading data from a CSV file or Excel sheet.
gdementen commentedon Mar 21, 2018
The @ idea is interesting, but I think we are looking at it from the wrong angle. There are potentially several different features involved here.
First, if we are loading one axis (or one group) from a file, the user should know what s/he is loading and could use a specific function to load that. eg read_axis(), read_group(), or similar.
It is probably valuable to have a format to save/load those one at a time from a custom format, and in that case the @ solution you describe seems a good compromise between readabilty, simplicity and functionality.
FWIW, I prefer to see "realistic" examples to gauge syntaxes. Your above proposals would be:
But, what I think users need more is
alixdamman commentedon Mar 22, 2018
That was my first thought actually but I see users coming. I'm pretty sure some of them will ask to export arrays with associated groups in the same sheets and then being able to reload arrays and groups in one operation. Imagine an array called
popand associated groups liketeenagersandpensioners. I'm quite sure some users will not like to haveteenagersandpensionersgroups and thepoparray stored in separated CSV files or Excel sheet.Another stuff I'm worried about: Do we force Groups and Axis to be stored vertically or horizontally?
5 remaining items
gdementen commentedon Mar 22, 2018
__groups__?) for all groups + one special sheet (__axes__?) for all axes + one sheet for each array like we have now[-]Allow to save and load Axis and Group objects in/from external files (HDF, CSV, EXCEL)[/-][+]Allow to save and load all Axis and Group objects of a session in/from HDF, CSV and EXCEL files[/+]alixdamman commentedon Mar 22, 2018
The name of the Sheet/CSV file/HDF group for axes and groups could be defined by two additional arguments with default values:
pathaxes=__axes__andpathgroup=__groups__(something like that).allowed to save and load Axis and Group objects in HDF files (larray-…
allowed to save and load Axis and Group objects in HDF files (larray-…
allowed to save and load Axis and Group objects in HDF files (#578) :
added Axis.to_csv (larray-project#578)
allowed to save and load Axis and Group objects in CSV files (larray-…
fix larray-project#578 : allowed to save and load all Axis and Group …
allowed to save and load Axis and Group objects in HDF files (#578) :
fix #578 : allowed to save and load all Axis and Group objects of a s…