Segfault Crash with Intel CI Tests

[jderrico-noaa]

Model builds successfully with the Intel compiler, but new CI tests with the Intel compiler are failing with a segfault for both mesoscale reference and convection permitting settings.

This issue appears to have been introduced with the merge of PR #121, which added smoke capabilities, as tests on earlier commits pass successfully.

Test Setting	Apparent Crash Location
Mesoscale Reference	Line 420 in bl_ysu.F90
Convection Permitting	Line 306 in module_bl_mynn.F

The set of tests linked here shows no .err files in the model directory, and includes the output of the log.atmosphere.0000.out file for further investigation.

[clark-evans]

Adding Haiqin to this issue - Haiqin, would you be able to review and work with Janet on debugging?

[haiqinli]

Sure, I will work with Janet on debugging this issue.

[haiqinli]

The intel/2023.2.0 was used for the Regression Tests of #121.

[clark-evans]

The intel/2023.2.0 was used for the Regression Tests of #121.

Thanks, @haiqinli - and it looks like the Intel toolkit we're using uses 2023.2.1, so very close to what you used for RDHPCS testing. (I'll also note that #121 was merged recently enough to trigger our CI tests, which it passed using 2023.2.1.)

[dustinswales]

@jderrico-noaa @clark-evans I can replicate this error using CI, but not on Derecho/Ursa.

It appears that both the feature (ufs/gsl-develop) and baseline (MAPS-Dev) tests are failing. Which makes me think that the intel_ci is not working correctly? If something was introduced in #121, that would only impact the feature tests, not the baseline tests. No?

In any event, I plan on building containerized CI for Intel sometime soon, so I'm inclined to devote my time towards that effort instead. Building containerized CI will have many cross-project benefits, at least for the ones I added CI testing to :)

[clark-evans]

It appears that both the feature (ufs/gsl-develop) and baseline (MAPS-Dev) tests are failing. Which makes me think that the intel_ci is not working correctly? If something was introduced in #121, that would only impact the feature tests, not the baseline tests. No?

That's my understanding. For what it's worth, it's unclear how or why these tests could fail where they are failing. For instance, line 420 of bl_ysu.F90 is the calculation of thx:

thx(i,k) = tx(i,k)/pi2d(i,k)

and line 306 of module_bl_mynn.F is a statement to define an error message variable:

errmsg = " "

We haven't touched either of those files; bl_ysu.F90 exists outside of our repo, whereas module_bl_mynn.F was last updated with the v8.2.2 merge (and presumably was only updated then with NCAR mods). Why this error would suddenly occur after #121 was merged, though, is very perplexing. That is the last PR to affect things like the PBL and physics drivers, but even the NCAR version of the code seems to be crashing in these tests.