You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Implementation idea: make a custom catalog provider #85
I think the underlying dataset could use a PyFileSystem and a PyFileSystemHandler to adapt Xarray to array buffers in order to ultimately materialize the dataset in the catalog.
The goal of this issue is to treat a single Zarr dataset, which often contains groups of variables with different sets of dimensions, as a catalog of tables. The tables in the catalog would have shared rows with each other insofar as they have shared Dataset dimensions -- this means that they could be joined with each other.
For example: ERA5 contains surface and atmospheric weather variables. These differ in dimension in that surface variables lack a pressure level. If we were to open an ERA5 Zarr dataset as a catalog, it would have at least two tables that would both share lat, lon, and time columns, though only the non-surface table would have a pressure level column. The remaining columns would be the data variables associated with the dimensions.
Cool to see you doing this, as my head has been on these exact same topics recently although from a slightly different angle.
The overlap here is making Zarr<>Tabular queries both efficient and "native" via DataFusion. You might want to see some work being done by DevSeed that is related but in early stages https://github.com/developmentseed/zarr-datafusion-search).
I started just this weekend a mini POC attempt to have a zarr.parquet schema where the chunks are referenced in a tabular format explicitly, but that is somewhat different.
Could be willing to do some work here but busy into the holidays, will track.
And, this project would love to have your contributions! Happy to even just trade ideas.
While I haven't seen the DevSeed prototype until now, I have been chatting a bit with Sean and Kyle. Happy to see convergence around this general idea.
Hey just FYI but I am leaving my full time job early May due to an international move, looking to finish up a basic benchmark PR on arrow-zarr before then, but outside of that I have a lot I am looking to get done at PanoAI before my departure (some exciting integrations with cool data!).
Basically in May I will be pivoting totally to this world, a bit of an intellectual sabbatical...long overdue! Alas but I'm not the best at juggling OS stuff on the side. Looking forward to it and you will be hearing/seeing more from me then 😁 @alxmrs
Activity
alxmrs commentedon Oct 26, 2025
I think the underlying dataset could use a PyFileSystem and a PyFileSystemHandler to adapt Xarray to array buffers in order to ultimately materialize the dataset in the catalog.
https://arrow.apache.org/docs/python/generated/pyarrow.fs.FileSystemHandler.html
alxmrs commentedon Nov 14, 2025
The goal of this issue is to treat a single Zarr dataset, which often contains groups of variables with different sets of dimensions, as a catalog of tables. The tables in the catalog would have shared rows with each other insofar as they have shared Dataset dimensions -- this means that they could be joined with each other.
alxmrs commentedon Nov 14, 2025
For example: ERA5 contains surface and atmospheric weather variables. These differ in dimension in that surface variables lack a pressure level. If we were to open an ERA5 Zarr dataset as a catalog, it would have at least two tables that would both share lat, lon, and time columns, though only the non-surface table would have a pressure level column. The remaining columns would be the data variables associated with the dimensions.
xaviernogueira commentedon Nov 25, 2025
Cool to see you doing this, as my head has been on these exact same topics recently although from a slightly different angle.
The overlap here is making Zarr<>Tabular queries both efficient and "native" via DataFusion. You might want to see some work being done by DevSeed that is related but in early stages https://github.com/developmentseed/zarr-datafusion-search).
I started just this weekend a mini POC attempt to have a
zarr.parquetschema where the chunks are referenced in a tabular format explicitly, but that is somewhat different.Could be willing to do some work here but busy into the holidays, will track.
alxmrs commentedon Nov 25, 2025
@xaviernogueira you might want to take a look at the discussion in #4, I have had a few thoughts along those lines!
alxmrs commentedon Nov 25, 2025
And, this project would love to have your contributions! Happy to even just trade ideas.
While I haven't seen the DevSeed prototype until now, I have been chatting a bit with Sean and Kyle. Happy to see convergence around this general idea.
alxmrs commentedon Jan 6, 2026
The more I think about it, the more that we need to implement this issue as the base API for opening Xarray Datasets.
See also: https://www.sciencedirect.com/science/article/pii/S0098300425000810
alxmrs commentedon Feb 7, 2026
Specifics: Let's add a public API to the Xarray SQL context:
xaviernogueira commentedon Feb 7, 2026
Hey just FYI but I am leaving my full time job early May due to an international move, looking to finish up a basic benchmark PR on
arrow-zarrbefore then, but outside of that I have a lot I am looking to get done at PanoAI before my departure (some exciting integrations with cool data!).Basically in May I will be pivoting totally to this world, a bit of an intellectual sabbatical...long overdue! Alas but I'm not the best at juggling OS stuff on the side. Looking forward to it and you will be hearing/seeing more from me then 😁 @alxmrs