Clean up Data handeling methods in adf_dataset.py and fix AMWG table formatting
#427
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will clean up and organize the calls to read files, datasets, and data arrays depending on the file type, ie time series, climo, regridded climo.
The main changes here are to:
adf_dataset.pyto gather new unit infoupdate_unitand call that inload_da. This could be useful as an external method, similarly toget_value_convertersamwg_table.pyto implement these changesImportant changes:
adf_dataset.py. That was the issue with the tables being improperly formatted, see AMWG tables are not getting proper data cleaning #423Now you can load data sets and arrays generically any where in the ADF by calling
load_dataset(fils)and/orload_da(<list of files>, <variable name>, <case name>)or through the three file types:
load_reference_timeseries_dataset/load_timeseries_datasetload_reference_climo_dataset/load_climo_datasetload_reference_regrid_dataset/load_regrid_datasetload_<file type>_datasetwill inherently callload_datasetandload_<file type>_dawill inherently callload_dacalling
load_datasethas a flag for time series files so that it can check the time bounds to ensure proper time alignment. This is only an issue for using older CAM/CESM files but this won't modify anything if the newer files are used.This change will now force the ADF to figure out the modifications at the loading data array stage
Potential issues:
user must make sure of the scale/factor, units if using any premade file; using files that aren't created via ADF from the history files level.
Remedy: The user just has to make sure there are no values for these arguments in the config yaml file if their datasets have been already modified, or use even more generic xarray load via
load_datasetfromadf_utilsNot likely but a definite issue for edge case scenarios.
Closes #423 and #424