GFDL AM4 Model

This repository includes the public release of the GFDL AM4 model code. The AM4 model is described in the two articles published in the Journal of Advances in Modeling Earth Systems (JAMES). More information on the model and access to the output is available on the AM4 data and code site at the Geophysical Fluid Dynamics Laboratory (GFDL).

The layout of this package includes the following directories:

• src - The source code for the AM4 model
• exec - The build directory with Makefiles for building the model executable
• run - Sample run script
• analysis - Sample analysis scripts

Cloning Instructions

This repository uses git submodules to point to other repositories. Thus, care should be taken when cloning, and updating the source to ensure all source. To obtain all source, use the following git command

git clone --recursive https://github.com/NOAA-GFDL/AM4.git

The --recursive option to git clone instructs git to recursively clone all submodules. In the event the repository was not cloned using the --recursive option, the following step must be taken to obtain all sources:

# From within the AM4 parent directory
git submodule update --init --recursive

Source Code

All model source is contained in the src directory. GFDL tracks code using the git version control system. This package includes a single version of the following GFDL model components. The git hash listed corresponds to the commit hash in the internal GFDL git repository.

Component	Commit Hash
atmos_drivers	5ee95d6abf0879594551dd7e6635dff4004c4010
atmos_param	2e94acfd8621e85216bf822c395a8c3f15a511a5
atmos_shared	a557d4d7bab033ef1ad1d400a62fe07a97ccb477
ice_param	1553c8bc4f9a66791c89367b6f327147523155ed
ice_sis	ccc7328dcd79706dd5c17c8bab660222886fc80b
land_lad2	a220288ecb289bf9d793d051fc5076072874ce07

The following components are available in the NOAA-GFDL github organization:

• MOM6
• coupler
• FMS (as shared)
• GFDL_atmos_cubed_sphere (tag AM4.0) (as atmos_cubed_sphere)

Building AM4

The exec directory contains Makefiles that can be used to build the AM4 executable. These Makefiles were generated using the Make Makefile (mkmf) program. Included in the exec direcgtory is a sample make template file for the Intel compilers (intel.mk). This make template can be used on any system with a relatively recent version of the Intel compilers, the netCDF 4 library and the MPICH2 MPI library. Included in the intel.mk file are additional settings that can be modified during the build.

To run the default build (-O3 -msse2), go to the exec directory and enter the command

make

If you would like to change some of the compiler options, there are several different options to add to the make command. For example

make ISA=-xhost BLD_TYPE=REPRO

will replace -msse with -xhost and -O3 with -O2. The three options for BLD_TYPE are
PROD (-O3)
REPRO (-O2)
DEBUG (-O0 and other traps)
All of the make line options can be found in the intel.mk file.

Obtaining the input data

The input data required for running the AM4 model can be found on GFDL's data portal .

The file AM4.tar.gz contains a configured run directory to run a sample experiment of the AM4 model. Included in the tar file is a README.AM4_run with more instructions on how to configure the AM4 run directory.

On Linux systems, the wget command is usually sufficient to download the data file:

wget ftp://nomads.gfdl.noaa.gov/users/Ming.Zhao/AM4Documentation/GFDL-AM4.0/inputData/AM4_run.tar.gz

To ensure the file downloaded is complete and not corrupted, download one of the two files:

wget ftp://nomads.gfdl.noaa.gov/users/Ming.Zhao/AM4Documentation/GFDL-AM4.0/inputData/AM4_run.tar.gz.sha256
wget ftp://nomads.gfdl.noaa.gov/users/Ming.Zhao/AM4Documentation/GFDL-AM4.0/inputData/AM4_run.tar.gz.sig

and run the following command that corresponds to the signature file downloaded:

sha256sum -c AM4_run.tar.gz.sha256

gpg --verify AM4_run.tar.gz.sig

Running AM4

Included in the run directory is a sample run script for reference. To run the AM4 sample experiment, first download the data file mentioned in Obtaining the Input data section. Modify the variables in the configuration section in the sample run script, and then run the script.

The sample data and run script are configured to run on 216 processors. To run on a different number of processors, or modify the experiment, refer to the README.AM4_run file included in the AM4 data tarball.

Note: The input.nml file (found in the AM4 data tarball) contains Fortran namelists and namelist variables that modify, at run time, the model. To learn more about the settings in the input.nml file, please refer to source code where the namelist/variable are defined.

Running AM4 on your local HPC

Yoder: This file is also included in this repository (possibly here, in the root directory). Likely, the information relatefd to running the model is kept separate because it is likely to chane with revisions, but we will review at least some of the details here.

In particular, it is necessary to modify the input.nml file to reflect your HPC topography. This will include entries based on the numbers of tasks (or processes), OMP threads, and nodes. Note also that these configuration settings imply a minimnum task count to run the model. Note that comma-terminating each variable line appears to be optional. One will note that some lines are comma-terminated; others are not; the simulation appears to brek when the values in those fields (literally) to not add up, and run when they do -- in spite of the mixed comma-termination.

Namelist edits:

&coupler_nml

• Number of months and days to run:

  &coupler_nml
        months = 1,
        days   = 0,
  

• Number of porcessors and openmp threads. Note the following examples are set for 432 threads and N=36 core nodes. There is some inconsistency with respect to references to hyperthreading (aka, is this actually 216 physical cores? There is a reference someplace that alludes to this, then a contradiction, and note that in input.nml hyperthreading is turned to .false.

  &coupler_nml
        ocean_npes = 0,
        atmos_npes = 432
        atmos_nthreads = 2
        ncores_per_node = 36
  

• To forego the use of threading, for example if your OS will not support it or if you cannot acquire, or choose not to acquire in order to simplify the allocation of resources, consistent cpus-per-node allocations (aka, a SLURM request with just --ntasks specified), set nthreads to one,

  atmos_nthreads=1
  

• Set nyblocks \times nxblocks = atmos_nthreads

  nxblocks=1
  nyblocks=1
  

Layouts:

• There are three layout formats:
  • (n1,m1) so that n1 \times m1 = N_{procs}
  • (n2,m2) so that n2 \times m2 = N_{procs}/6, where the factor of 6 represents the six sides of a cube.
  • io_layout (j,k): the y component of the layout, (m1, m2) must be evenly divisible by k, mod(m_j,k)=0
• Layout entries include:

  &fv_core_nml
          layout   = 3,24
          io_layout = 1,4
   &ice_model_nml
          layout = 144,3
          io_layout = 1,3
   &land_model_nml
          layout   = 3,24
          io_layout = 1,4
   &ocean_model_nml
          layout = 144,3
          io_layout = 1,3
  

Note that these variables can (in principle) be defined as environment variables, for example:

export n1=144
export m1=3
export n2=3
export m2=8
export j=1
export k=3

and then,

&fv_core_nml
        layout   = ${n2},${m2}
        io_layout = ${j},${k}
&ice_model_nml
      layout = ${n1},${m1}
      io_layout = $j,$k
&land_model_nml
      layout   = $n2,$m2
      io_layout = $j,$k
&ocean_model_nml
      layout = ${n1},${m1}
      io_layout = $j,$k

Note this example changes the layout configuration for &land_model_nml and &fv_core_nml, but in a permissbile manner. Designing input.nml this way potentially simplifies the runtime experience by permitting the configuraiton to be set in the run script, and potentially to be computed and validated dynamically.

Unfortunatly, I have had, to date, very limited luck getting this environment-variable approach to work. Either I am making a mistake somewhere, or some of the variables are not being read correctly. Either way, a better approach is probably to use a script to write a fresh input.nml.

Analysis Scripts

Some of the climate analysis scripts run at NOAA GFDL and used in the AM4 documentation papers are located in the analysis directory. Within each analysis suite, is a jupyter notebook, both readable and runnable from your local jupyter environment, provided all dependencies are installed.

E.g.

• Radiation processor
• Long-term DJF seasonal mean
• Zonal_mean_zonal_wind_stress
• PCMDI Metrics Portrait Plot

Model output and Other References

Please refer to the AM4 data and code site for details about where to find model and OBS data used in the papers.

For all analysis figures and pertaining data, please use the AM4 documentation papers as the original reference.

Please direct your questions and feedback to gfdl.climate.model.info@noaa.gov

Disclaimer

The United States Department of Commerce (DOC) GitHub project code is provided on an 'as is' basis and the user assumes responsibility for its use. DOC has relinquished control of the information and no longer has responsibility to protect the integrity, confidentiality, or availability of the information. Any claims against the Department of Commerce stemming from the use of its GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.

This project code is made available through GitHub but is managed by NOAA-GFDL at https://gitlab.gfdl.noaa.gov.