Comparison of MPAS-model runtime between `CMake` and `make`

[guoqing-noaa]

MPAS-Model currently has two ways to build executables: CMake and make.
It was hypothesized that CMake-built executables will run slower than the make-built executables since the default CMake options are not optimized as make.
rrfs-workflow has adopted the make method so far. The problem is that different spack-stack versions may provide different combinations of C++/C/Fortran compilers and for each new combinations, we may need to add a new section into the MPAS-Model Makefile, such as PR #170. This may not be a sustainable way.

So we would like to transition to use CMake to compile MPAS-Model as early as possible for rrfs-workflow. This will be more aligned with modern software engineering practices, more consistent with other UFS applications, much easier to maintain.

The following is a comparison between CMake and make using current default settings (i.e. make is optimized while CMake needs adding fine-tuned optimizations). Both ran on gaeac6, 12h cold start forecasts on 10 nodes (ppn=160).
Conclusion: CMake completed in 3097s while make completed in 3024s, a difference of 73 seconds slower. I’d consider this within a reasonable margin of variability.
We will add fine-tuned optimization options to CMake and see how it goes.

CMake

    timer_name                                            total       calls        min            max            avg      pct_tot   pct_par     par_eff
  1 total time                                        3097.11304         1     3097.02368     3097.11304     3097.05566   100.00       0.00       1.00
  2  initialize                                         53.16598         1       53.01339       53.16598       53.03205     1.72       1.72       1.00
  3   read_ICs                                           7.81630         1        7.81389        7.81630        7.81475     0.25      14.70       1.00
  2  diagnostic_fields                                  79.28183      5762        0.00218        2.54850        0.01267     2.56       2.56       0.92
  2  stream_output                                     276.86783      2881        0.00000       25.98127        0.09607     8.94       8.94       1.00
  2  stream_input                                       16.86766         1       16.85745       16.86766       16.85988     0.54       0.54       1.00
  2  time integration                                 2601.04175      2880        0.86035        2.93290        0.90244    83.98      83.98       1.00
  3   physics driver                                   457.91266      2880        0.07227        2.21594        0.11411    14.79      17.60       0.72
  4    cal_mynncld                                       0.02754        48        0.00012        0.00185        0.00033     0.00       0.01       0.57
  4    rrtmg_swrad                                      18.24232        48        0.00001        1.36461        0.19843     0.59       3.98       0.52
  4    rrtmg_lwrad                                      31.55703        48        0.47152        0.70754        0.53042     1.02       6.89       0.81
  4    sf_mynnsfclay                                     3.16672      2880        0.00050        0.01856        0.00069     0.10       0.69       0.63
  4    sf_ruc                                            9.87083      2880        0.00005        0.00533        0.00118     0.32       2.16       0.34
  4    bl_mynnedmf                                     252.37164      2880        0.03262        0.10403        0.04327     8.15      55.11       0.49
  4    bl_ugwp_gwdo                                     64.89875      2880        0.01186        0.03679        0.01697     2.10      14.17       0.75
  3   atm_rk_integration_setup                          23.18732      2880        0.00083        0.00983        0.00387     0.75       0.89       0.48
  3   atm_compute_moist_coefficients                    16.44854      2880        0.00126        0.00838        0.00278     0.53       0.63       0.49
  3   physics_get_tend                                  99.27243      2880        0.00166        1.31308        0.01293     3.21       3.82       0.38
  3   atm_compute_vert_imp_coefs                        16.58818      8640        0.00070        0.00510        0.00119     0.54       0.64       0.62
  3   atm_compute_dyn_tend                             308.98950     25920        0.00414        0.03390        0.00948     9.98      11.88       0.80
  3   small_step_prep                                   31.60984     25920        0.00033        0.00357        0.00068     1.02       1.22       0.56
  3   atm_bdy_adjust_dynamics_relaxzone_tend            39.64612     25920        0.00002        0.00438        0.00010     1.28       1.52       0.07
  3   atm_advance_acoustic_step                        142.87941     34560        0.00103        0.01378        0.00275     4.61       5.49       0.67
  3   atm_divergence_damping_3d                         33.90297     34560        0.00032        0.00333        0.00064     1.09       1.30       0.65
  3   atm_recover_large_step_variables                 179.86221     25920        0.00079        0.01022        0.00373     5.81       6.92       0.54
  3   atm_compute_solve_diagnostics                    157.25038     25920        0.00084        0.01267        0.00359     5.08       6.05       0.59
  3   atm_rk_dynamics_substep_finish                    34.66234      8640        0.00030        0.00642        0.00226     1.12       1.33       0.56
  3   atm_advance_scalars                              145.20296      5760        0.01480        0.04295        0.02192     4.69       5.58       0.87
  3   atm_advance_scalars_mono                         485.40430      2880        0.13404        0.42939        0.16317    15.67      18.66       0.97
  3   microphysics                                     299.64453      2880        0.02061        0.26858        0.06058     9.67      11.52       0.58
  4    mp_tempo                                        235.16112      2880        0.00946        0.22619        0.03726     7.59      78.48       0.46

make

    timer_name                                            total       calls        min            max            avg      pct_tot   pct_par     par_eff
  1 total time                                        3024.70166         1     3024.59546     3024.70166     3024.60938   100.00       0.00       1.00
  2  initialize                                         47.04760         1       46.98198       47.04760       47.00020     1.56       1.56       1.00
  3   read_ICs                                           3.34079         1        3.33413        3.34079        3.33882     0.11       7.10       1.00
  2  diagnostic_fields                                  72.75267      5762        0.00218        2.19514        0.01200     2.41       2.41       0.95
  2  stream_output                                     266.47327      2881        0.00000       21.64403        0.09248     8.81       8.81       1.00
  2  stream_input                                       16.09141         1       16.08182       16.09141       16.08426     0.53       0.53       1.00
  2  time integration                                 2574.39526      2880        0.85349        2.91890        0.89322    85.11      85.11       1.00
  3   physics driver                                   452.84753      2880        0.06559        2.20191        0.10717    14.97      17.59       0.68
  4    cal_mynncld                                       0.03018        48        0.00012        0.00191        0.00031     0.00       0.01       0.50
  4    rrtmg_swrad                                      18.08443        48        0.00001        1.36585        0.19733     0.60       3.99       0.52
  4    rrtmg_lwrad                                      35.15728        48        0.46342        0.77106        0.52213     1.16       7.76       0.71
  4    sf_mynnsfclay                                     3.00286      2880        0.00048        0.00439        0.00066     0.10       0.66       0.63
  4    sf_ruc                                            9.67783      2880        0.00004        0.00521        0.00113     0.32       2.14       0.34
  4    bl_mynnedmf                                     239.33063      2880        0.02623        0.09847        0.03679     7.91      52.85       0.44
  4    bl_ugwp_gwdo                                     66.13377      2880        0.01185        0.44685        0.01687     2.19      14.60       0.73
  3   atm_rk_integration_setup                          23.49383      2880        0.00118        0.35943        0.00416     0.78       0.91       0.51
  3   atm_compute_moist_coefficients                    16.65304      2880        0.00127        0.36100        0.00280     0.55       0.65       0.48
  3   physics_get_tend                                  98.10201      2880        0.00168        1.34139        0.01309     3.24       3.81       0.38
  3   atm_compute_vert_imp_coefs                        16.41025      8640        0.00070        0.31947        0.00119     0.54       0.64       0.62
  3   atm_compute_dyn_tend                             321.95660     25920        0.00417        0.42781        0.00948    10.64      12.51       0.76
  3   small_step_prep                                   30.59839     25920        0.00033        0.31577        0.00069     1.01       1.19       0.59
  3   atm_bdy_adjust_dynamics_relaxzone_tend            38.88117     25920        0.00002        0.03143        0.00010     1.29       1.51       0.06
  3   atm_advance_acoustic_step                        135.33881     34560        0.00104        0.04513        0.00277     4.47       5.26       0.71
  3   atm_divergence_damping_3d                         38.36580     34560        0.00037        0.00373        0.00070     1.27       1.49       0.63
  3   atm_recover_large_step_variables                 177.47621     25920        0.00080        0.01080        0.00369     5.87       6.89       0.54
  3   atm_compute_solve_diagnostics                    160.50436     25920        0.00084        0.01115        0.00358     5.31       6.23       0.58
  3   atm_rk_dynamics_substep_finish                    34.15577      8640        0.00030        0.00642        0.00227     1.13       1.33       0.57
  3   atm_advance_scalars                              146.68997      5760        0.01433        0.04096        0.02192     4.85       5.70       0.86
  3   atm_advance_scalars_mono                         485.76904      2880        0.12722        0.18854        0.16303    16.06      18.87       0.97
  3   microphysics                                     291.05881      2880        0.02013        0.12466        0.05969     9.62      11.31       0.59
  4    mp_tempo                                        229.48993      2880        0.00936        0.10149        0.03712     7.59      78.85       0.47

The run directories:

/gpfs/f6/arfs-gsl/world-shared/gge/rrfs2/regressionTests/3km133tquv/stmp/20240506/rrfs_fcst_00_v2.1.2/det.CMake/fcst_00
and
/gpfs/f6/arfs-gsl/world-shared/gge/rrfs2/regressionTests/3km133tquv/stmp/20240506/rrfs_fcst_00_v2.1.2/det.make/fcst_00

This issue is copied at NOAA-EMC/rrfs-workflow#1094.

[guoqing-noaa]

tag @clark-evans @SamuelTrahanNOAA @barlage for awareness

[guoqing-noaa]

For record, I found some build flags in Makefile may not be consistent, for example,
FFLAGS_DEBUG = -g -convert big_endian -free -CU -CB -check all -fpe0 -traceback, where -CU -CB does not apply to intel compilers and has no impact at all.

Current Makefile has the follow optimization flags relevant to rrfs-workflow:

FFLAGS_PROMOTION = -real-size 64
FFLAGS_OPT = -O3 -convert big_endian -free -align array64byte
CFLAGS_OPT = -O3
CXXFLAGS_OPT = -O3
LDFLAGS_OPT = -O3
FFLAGS_DEBUG = -g -convert big_endian -free -check all -fpe0 -traceback

Their counterparts in CMakeCache.txt are as follows:

CMAKE_Fortran_FLAGS_RELEASE:STRING=
CMAKE_C_FLAGS_RELEASE:STRING=
CMAKE_CXX_FLAGS_RELEASE:STRING=
CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING=   &   CMAKE_SHARED_LINKER_FLAGS_RELEASE:STRING=
CMAKE_Fortran_FLAGS_DEBUG:STRING

[guoqing-noaa]

Here are the current build flags in CMakeCache.txt generated by default from CMake:

CMAKE_Fortran_FLAGS_RELEASE:STRING=-O3
CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING=
CMAKE_SHARED_LINKER_FLAGS_RELEASE:STRING=
CMAKE_Fortran_FLAGS_DEBUG:STRING=-g
CMAKE_C_FLAGS_DEBUG:STRING=-g

It looks like MPAS-Model atmosphere does not use C++ compilers, so C++ build flags are not relevant here.
(Note: core_landice/mode_forward/Interface_velocity_solver.hpp:#include <vector> uses C++ but we don't do landice at the moment)

We will focus on the release version at this moment for rrfs-workflow.

The double precision flags (i.e. -real-size 64 or -r8) are overwritten when we set PRECISION=single at the make command line. It does not take effect here.

So we only miss the following flags in CMAKE_Fortran_FLAGS_RELEASE:
-convert big_endian -free -align array64byte
and -O3 for LINKER_FLAGS

[guoqing-noaa]

second run of the make version

    timer_name                                            total       calls        min            max            avg      pct_tot   pct_par     par_eff
  1 total time                                        3049.32593         1     3049.25000     3049.32593     3049.26538   100.00       0.00       1.00
  2  initialize                                         52.82237         1       52.65583       52.82237       52.67844     1.73       1.73       1.00
  3   read_ICs                                           7.33784         1        7.33453        7.33784        7.33623     0.24      13.89       1.00
  2  diagnostic_fields                                  75.35663      5762        0.00215        2.24269        0.01220     2.47       2.47       0.93
  2  stream_output                                     270.68735      2881        0.00000       21.63789        0.09393     8.88       8.88       1.00
  2  stream_input                                       14.69839         1       14.68746       14.69839       14.68981     0.48       0.48       1.00
  2  time integration                                 2573.91968      2880        0.85227        2.93088        0.89304    84.41      84.41       1.00
  3   physics driver                                   439.68127      2880        0.06492        2.21079        0.10745    14.42      17.08       0.70
  4    cal_mynncld                                       0.02705        48        0.00012        0.00194        0.00031     0.00       0.01       0.55
  4    rrtmg_swrad                                      17.99134        48        0.00001        1.36911        0.19724     0.59       4.09       0.53
  4    rrtmg_lwrad                                      30.39681        48        0.46304        0.70340        0.52090     1.00       6.91       0.82
  4    sf_mynnsfclay                                     2.98233      2880        0.00048        0.00627        0.00066     0.10       0.68       0.64
  4    sf_ruc                                            9.74555      2880        0.00004        0.00532        0.00113     0.32       2.22       0.33
  4    bl_mynnedmf                                     234.50246      2880        0.02643        0.64436        0.03706     7.69      53.33       0.46
  4    bl_ugwp_gwdo                                     64.91067      2880        0.01189        0.30186        0.01686     2.13      14.76       0.75
  3   atm_rk_integration_setup                          22.99932      2880        0.00116        0.00984        0.00403     0.75       0.89       0.50
  3   atm_compute_moist_coefficients                    17.39140      2880        0.00128        0.00828        0.00279     0.57       0.68       0.46
  3   physics_get_tend                                  98.59656      2880        0.00168        1.31318        0.01297     3.23       3.83       0.38
  3   atm_compute_vert_imp_coefs                        17.29456      8640        0.00069        0.00461        0.00118     0.57       0.67       0.59
  3   atm_compute_dyn_tend                             317.76089     25920        0.00416        0.03326        0.00946    10.42      12.35       0.77
  3   small_step_prep                                   31.33876     25920        0.00032        0.00497        0.00069     1.03       1.22       0.57
  3   atm_bdy_adjust_dynamics_relaxzone_tend            35.66057     25920        0.00002        0.00435        0.00010     1.17       1.39       0.07
  3   atm_advance_acoustic_step                        137.04344     34560        0.00103        0.01352        0.00275     4.49       5.32       0.69
  3   atm_divergence_damping_3d                         39.77208     34560        0.00036        0.00363        0.00070     1.30       1.55       0.61
  3   atm_recover_large_step_variables                 184.51486     25920        0.00078        0.01037        0.00365     6.05       7.17       0.51
  3   atm_compute_solve_diagnostics                    158.92584     25920        0.00083        0.01143        0.00355     5.21       6.17       0.58
  3   atm_rk_dynamics_substep_finish                    35.81310      8640        0.00031        0.00894        0.00225     1.17       1.39       0.54
  3   atm_advance_scalars                              145.34187      5760        0.01450        0.04101        0.02194     4.77       5.65       0.87
  3   atm_advance_scalars_mono                         485.11905      2880        0.13652        0.18836        0.16309    15.91      18.85       0.97
  3   microphysics                                     294.56409      2880        0.02145        0.12594        0.05976     9.66      11.44       0.58
  4    mp_tempo                                        231.75131      2880        0.00939        0.09900        0.03708     7.60      78.68       0.46

[guoqing-noaa]

second run of CMake version

    timer_name                                            total       calls        min            max            avg      pct_tot   pct_par     par_eff
  1 total time                                        3083.77100         1     3083.68433     3083.77100     3083.70679   100.00       0.00       1.00
  2  initialize                                         51.11495         1       51.03896       51.11495       51.05666     1.66       1.66       1.00
  3   read_ICs                                           6.85586         1        6.85351        6.85586        6.85455     0.22      13.41       1.00
  2  diagnostic_fields                                  82.01717      5762        0.00214        2.75040        0.01318     2.66       2.66       0.93
  2  stream_output                                     281.39169      2881        0.00000       23.12404        0.09764     9.12       9.12       1.00
  2  stream_input                                       16.32683         1       16.31680       16.32683       16.31911     0.53       0.53       1.00
  2  time integration                                 2596.78247      2880        0.85781        3.24037        0.90100    84.21      84.21       1.00
  3   physics driver                                   459.83994      2880        0.07136        2.53051        0.11394    14.91      17.71       0.71
  4    cal_mynncld                                       0.03093        48        0.00012        0.00162        0.00033     0.00       0.01       0.51
  4    rrtmg_swrad                                      18.08801        48        0.00001        1.73278        0.19798     0.59       3.93       0.53
  4    rrtmg_lwrad                                      33.20181        48        0.46981        0.70317        0.52819     1.08       7.22       0.76
  4    sf_mynnsfclay                                     3.16956      2880        0.00050        0.00240        0.00069     0.10       0.69       0.62
  4    sf_ruc                                           10.01549      2880        0.00005        0.00564        0.00118     0.32       2.18       0.34
  4    bl_mynnedmf                                     253.66275      2880        0.03285        0.29723        0.04329     8.23      55.16       0.49
  4    bl_ugwp_gwdo                                     66.00818      2880        0.01184        0.25154        0.01694     2.14      14.35       0.74
  3   atm_rk_integration_setup                          23.29930      2880        0.00083        0.00991        0.00387     0.76       0.90       0.48
  3   atm_compute_moist_coefficients                    16.82117      2880        0.00124        0.01200        0.00278     0.55       0.65       0.48
  3   physics_get_tend                                  98.93919      2880        0.00166        1.31721        0.01303     3.21       3.81       0.38
  3   atm_compute_vert_imp_coefs                        16.62545      8640        0.00069        0.00409        0.00120     0.54       0.64       0.62
  3   atm_compute_dyn_tend                             314.74387     25920        0.00409        0.03026        0.00946    10.21      12.12       0.78
  3   small_step_prep                                   31.34657     25920        0.00032        0.00332        0.00068     1.02       1.21       0.57
  3   atm_bdy_adjust_dynamics_relaxzone_tend            37.46001     25920        0.00002        0.00423        0.00010     1.21       1.44       0.07
  3   atm_advance_acoustic_step                        141.68556     34560        0.00102        0.06116        0.00274     4.59       5.46       0.67
  3   atm_divergence_damping_3d                         33.69181     34560        0.00032        0.05981        0.00064     1.09       1.30       0.65
  3   atm_recover_large_step_variables                 183.48735     25920        0.00079        0.06097        0.00375     5.95       7.07       0.53
  3   atm_compute_solve_diagnostics                    159.92632     25920        0.00083        0.06233        0.00360     5.19       6.16       0.58
  3   atm_rk_dynamics_substep_finish                    35.64038      8640        0.00030        0.00645        0.00230     1.16       1.37       0.56
  3   atm_advance_scalars                              154.50999      5760        0.01439        0.04296        0.02191     5.01       5.95       0.82
  3   atm_advance_scalars_mono                         484.46942      2880        0.13406        0.18676        0.16314    15.71      18.66       0.97
  3   microphysics                                     294.42456      2880        0.02160        0.12714        0.06035     9.55      11.34       0.59
  4    mp_tempo                                        230.92079      2880        0.00943        0.10471        0.03705     7.49      78.43       0.46

[guoqing-noaa]

I tested cmake and make runs at Ursa. Using <nodes>10:ppn=160</nodes> to make 24 hour forecasts.
make took 8823 seconds
cmake took 8778 seconds (default rrfs-workflow build option: -O3 -convert big_endian -free -align array64byte)
This time cmake is 45 seconds faster than make. I will still consider this within a reasonable margin of variability.

So I would think cmake is equivalent to make and no any meaningful runtime differences.
Also, we can add more aggressive optimization options, such as -xHost, -fast, on lots of supported platforms to get further speed up. But this is out of the scope of this issue. We can revisit that when needed.
(note: Gaea is a Cray system, so its intel optimization is very different from other machines)

the make run

/scratch3/BMC/wrfruc/gge/rrfs2/regressionTests/dev1/stmp/20251119/rrfs_fcst_00_v2.1.2/det.make/fcst_00

    timer_name                                            total       calls        min            max            avg      pct_tot   pct_par     par_eff
  1 total time                                        8823.34668         1     8823.22461     8823.34668     8823.26953   100.00       0.00       1.00
  2  initialize                                         39.66726         1       39.48235       39.66726       39.51536     0.45       0.45       1.00
  3   read_ICs                                           9.06612         1        9.06211        9.06612        9.06386     0.10      22.86       1.00
  2  diagnostic_fields                                 155.88295     11522        0.00239        2.44117        0.01275     1.77       1.77       0.94
  2  stream_output                                    3560.23511      5761        0.00000      166.97169        0.61796    40.35      40.35       1.00
  2  stream_input                                       23.28485         1       23.27587       23.28485       23.27892     0.26       0.26       1.00
  2  time integration                                 4913.56348      5760        0.81174        2.83199        0.85240    55.69      55.69       1.00
  3   physics driver                                   827.55884      5760        0.06413        2.14354        0.11009     9.38      16.84       0.77
  4    cal_mynncld                                       0.03267        96        0.00009        0.00598        0.00019     0.00       0.00       0.56
  4    rrtmg_swrad                                      47.01704        96        0.00001        1.27868        0.40937     0.53       5.68       0.84
  4    rrtmg_lwrad                                      60.29138        96        0.46278        0.71029        0.52885     0.68       7.29       0.84
  4    sf_mynnsfclay                                     6.10980      5760        0.00048        0.01718        0.00068     0.07       0.74       0.65
  4    sf_ruc                                           19.77410      5760        0.00004        0.02182        0.00118     0.22       2.39       0.34
  4    bl_mynnedmf                                     384.86240      5760        0.02611        0.10727        0.03802     4.36      46.51       0.57
  4    bl_ugwp_gwdo                                    115.34598      5760        0.01093        0.03463        0.01495     1.31      13.94       0.75
  3   atm_rk_integration_setup                          33.19683      5760        0.00111        0.01912        0.00308     0.38       0.68       0.53
  3   atm_compute_moist_coefficients                    23.96881      5760        0.00120        0.02740        0.00225     0.27       0.49       0.54
  3   physics_get_tend                                 174.72806      5760        0.00161        1.23512        0.01180     1.98       3.56       0.39
  3   atm_compute_vert_imp_coefs                        24.75484     17280        0.00066        0.02487        0.00105     0.28       0.50       0.73
  3   atm_compute_dyn_tend                             590.84143     51840        0.00438        0.08125        0.00917     6.70      12.02       0.80
  3   small_step_prep                                   50.55602     51840        0.00035        0.34076        0.00064     0.57       1.03       0.66
  3   atm_bdy_adjust_dynamics_relaxzone_tend            62.89602     51840        0.00002        0.34010        0.00009     0.71       1.28       0.08
  3   atm_advance_acoustic_step                        235.52963     69120        0.00103        0.34293        0.00251     2.67       4.79       0.74
  3   atm_divergence_damping_3d                         61.55873     69120        0.00034        0.33999        0.00063     0.70       1.25       0.71
  3   atm_recover_large_step_variables                 287.04810     51840        0.00080        0.26503        0.00313     3.25       5.84       0.56
  3   atm_compute_solve_diagnostics                    243.96811     51840        0.00103        0.26201        0.00325     2.77       4.97       0.69
  3   atm_rk_dynamics_substep_finish                    54.49897     17280        0.00028        0.03683        0.00184     0.62       1.11       0.58
  3   atm_advance_scalars                              286.62711     11520        0.01362        0.05743        0.02135     3.25       5.83       0.86
  3   atm_advance_scalars_mono                         882.35876      5760        0.09149        0.47837        0.14885    10.00      17.96       0.97
  3   microphysics                                     510.82532      5760        0.01946        0.12400        0.05787     5.79      10.40       0.65
  4    mp_tempo                                        382.83340      5760        0.00885        0.09521        0.03370     4.34      74.94       0.51

the cmake run

/scratch3/BMC/wrfruc/gge/rrfs2/regressionTests/dev1/stmp/20251119/rrfs_fcst_00_v2.1.2/det.cmake1/fcst_00

    timer_name                                            total       calls        min            max            avg      pct_tot   pct_par     par_eff
  1 total time                                        8778.50684         1     8778.37207     8778.50684     8778.42480   100.00       0.00       1.00
  2  initialize                                         37.03917         1       36.86018       37.03917       36.87985     0.42       0.42       1.00
  3   read_ICs                                           8.28319         1        8.28171        8.28319        8.28240     0.09      22.36       1.00
  2  diagnostic_fields                                 166.08377     11522        0.00240        2.95871        0.01363     1.89       1.89       0.95
  2  stream_output                                    3420.48779      5761        0.00000      161.22186        0.59370    38.96      38.96       1.00
  2  stream_input                                       23.10707         1       23.09675       23.10707       23.10054     0.26       0.26       1.00
  2  time integration                                 4997.29199      5760        0.81354        2.92227        0.86698    56.93      56.93       1.00
  3   physics driver                                   857.36572      5760        0.07062        2.23644        0.11663     9.77      17.16       0.78
  4    cal_mynncld                                       0.03575        96        0.00012        0.00548        0.00021     0.00       0.00       0.56
  4    rrtmg_swrad                                      55.16948        96        0.00002        1.45406        0.41356     0.63       6.43       0.72
  4    rrtmg_lwrad                                      64.02214        96        0.46440        0.79401        0.53306     0.73       7.47       0.80
  4    sf_mynnsfclay                                     6.52053      5760        0.00050        0.01649        0.00072     0.07       0.76       0.63
  4    sf_ruc                                           20.49914      5760        0.00005        0.02301        0.00124     0.23       2.39       0.35
  4    bl_mynnedmf                                     423.14453      5760        0.03205        0.11933        0.04405     4.82      49.35       0.60
  4    bl_ugwp_gwdo                                    117.62984      5760        0.01094        0.03945        0.01505     1.34      13.72       0.74
  3   atm_rk_integration_setup                          32.18695      5760        0.00078        0.02159        0.00290     0.37       0.64       0.52
  3   atm_compute_moist_coefficients                    24.35666      5760        0.00120        0.01498        0.00225     0.28       0.49       0.53
  3   physics_get_tend                                 192.94768      5760        0.00172        1.25818        0.01195     2.20       3.86       0.36
  3   atm_compute_vert_imp_coefs                        25.28198     17280        0.00066        0.05178        0.00106     0.29       0.51       0.72
  3   atm_compute_dyn_tend                             595.61475     51840        0.00447        0.36563        0.00919     6.78      11.92       0.80
  3   small_step_prep                                   49.54308     51840        0.00034        0.02581        0.00063     0.56       0.99       0.66
  3   atm_bdy_adjust_dynamics_relaxzone_tend            64.53658     51840        0.00002        0.00862        0.00009     0.74       1.29       0.08
  3   atm_advance_acoustic_step                        228.97269     69120        0.00101        0.02568        0.00247     2.61       4.58       0.74
  3   atm_divergence_damping_3d                         53.29390     69120        0.00030        0.02525        0.00057     0.61       1.07       0.74
  3   atm_recover_large_step_variables                 285.84552     51840        0.00078        0.35657        0.00312     3.26       5.72       0.57
  3   atm_compute_solve_diagnostics                    237.82588     51840        0.00104        0.36118        0.00325     2.71       4.76       0.71
  3   atm_rk_dynamics_substep_finish                    52.60516     17280        0.00028        0.05283        0.00182     0.60       1.05       0.60
  3   atm_advance_scalars                              271.49118     11520        0.01352        0.04677        0.02134     3.09       5.43       0.91
  3   atm_advance_scalars_mono                         880.35248      5760        0.11310        0.25992        0.14888    10.03      17.62       0.97
  3   microphysics                                     527.41644      5760        0.02223        0.13371        0.05887     6.01      10.55       0.64
  4    mp_tempo                                        393.90625      5760        0.00891        0.09844        0.03414     4.49      74.69       0.50