Implementation idea: make a custom catalog provider

[alxmrs]

https://github.com/apache/datafusion-python/blob/main/python/tests/test_catalog.py

https://datafusion.apache.org/python/user-guide/data-sources.html#user-defined-catalog-and-schema

[alxmrs]

I think the underlying dataset could use a PyFileSystem and a PyFileSystemHandler to adapt Xarray to array buffers in order to ultimately materialize the dataset in the catalog.

https://arrow.apache.org/docs/python/generated/pyarrow.fs.FileSystemHandler.html

[alxmrs]

The goal of this issue is to treat a single Zarr dataset, which often contains groups of variables with different sets of dimensions, as a catalog of tables. The tables in the catalog would have shared rows with each other insofar as they have shared Dataset dimensions -- this means that they could be joined with each other.

[alxmrs]

For example: ERA5 contains surface and atmospheric weather variables. These differ in dimension in that surface variables lack a pressure level. If we were to open an ERA5 Zarr dataset as a catalog, it would have at least two tables that would both share lat, lon, and time columns, though only the non-surface table would have a pressure level column. The remaining columns would be the data variables associated with the dimensions.

[xaviernogueira]

Cool to see you doing this, as my head has been on these exact same topics recently although from a slightly different angle.

The overlap here is making Zarr<>Tabular queries both efficient and "native" via DataFusion. You might want to see some work being done by DevSeed that is related but in early stages https://github.com/developmentseed/zarr-datafusion-search).

I started just this weekend a mini POC attempt to have a zarr.parquet schema where the chunks are referenced in a tabular format explicitly, but that is somewhat different.

Could be willing to do some work here but busy into the holidays, will track.

[alxmrs]

@xaviernogueira you might want to take a look at the discussion in #4, I have had a few thoughts along those lines!

[alxmrs]

And, this project would love to have your contributions! Happy to even just trade ideas.

While I haven't seen the DevSeed prototype until now, I have been chatting a bit with Sean and Kyle. Happy to see convergence around this general idea.

[alxmrs]

The more I think about it, the more that we need to implement this issue as the base API for opening Xarray Datasets.

See also: https://www.sciencedirect.com/science/article/pii/S0098300425000810