Skip to content

Skip download step, just create xarray datasets #10

@wmay

Description

@wmay

The goal is to open the full dataset. Storing the individual files locally is unnecessary and is actually the slowest part of the process. (Because we may have many thousands of small files, in terms of storage this is essentially random writes and random reads, which is slow.) It's much more efficient to write the entire dataset from xarray as netcdf or zarr.

Support for downloading the individual files can probably be dropped completely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions