Repository navigation
Download function is too slow #312
Description
Currently the download function is quite slow. This has become more of an issue now that there are ~100 data files to download and it takes a significant amount of time to download them all. It would be nice to multi-thread this function and download the files in parallel.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
sgosline commentedon Apr 24, 2025
Need to look into how this would happen. Are there packages available to facilitate this? How about error checking/file fidelity?
ymahlich commentedon Apr 24, 2025
We should also consider to do checksum verification of the downloaded files (see also here: https://help.figshare.com/article/how-to-use-the-figshare-api#basic-coding)
[-]Download Function in Package Could be Multi-Threaded[/-][+]Download function is too slow[/+]sgosline commentedon Jul 30, 2025
Checksum has been implemented, but another possible fix for this is to use parquet file formats for download/load instead of csvs.