MOM6 Development

Marshall Ward

2022-10-18

Presentation is available at

https://marshallward.org/mom6workshop/develop.html

CI Testing

Test Description
Mem Layout Symmetric vs Nonsymmetric
OpenMP With threads
CPU Layout 1 vs 2 CPUs
Rotation 90° Domain Turn
NaN Init Aggressive initialization
Regression Answer changes

Also tests for documentation, MacOS support, etc

Autoconf

.testing is based on Autoconf; let's try it:

# Build FMS
cd ac/deps
make -j

# Create ./configure
cd ../
autoreconf -if

# Build MOM6
cd ..
mkdir build
cd build
../ac/configure
make -j

Reproducibility

Numerical results must reproduce:

Even floating point!

Verification is based on ocean.stats:

Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean sea level,   Total Mass,    Frac Mass Err
          [days]                 [m2 s-2]           [Nondim]        [m]             [kg]           [Nondim]
  0,       0.000,     0, En 3.3361965169932362E-01, CFL  0.01092, SL -0.0000E+00, Mass 3.95904E+18, Me  0.00E+00
144,       1.000,     0, En 3.3113616621947212E-01, CFL  0.01064, SL -0.0000E+00, Mass 3.95904E+18, Me -4.25E-18

Energy/Mass is machine-precision and reproducible

Numerical "style"

  • No ambiguous ordering

    • Bad: a + b + c
    • OK: (a + b) + c

  • Avoid intrinsics

    • sum() is ambiguous; what is the order?
    • use reproducing_sum()
    • Avoid sin(), exp(), even a**b

See my FortranCon talk for more about this

Bit Repro Debugging

In MOM_input:

DEBUG=True

This enables checksums in the output:

u-point: mean=   4.5198826930130449E-02 min=  -2.2679844164834517E-03 max=   3.6386233175123922E-01 u Beginning of step_MOM  [uv]
u-point: c=    320978 sw=    316992 se=    316992 nw=    316992 ne=    316992 u Beginning of step_MOM  [uv]
v-point: mean=   0.0000000000000000E+00 min=   0.0000000000000000E+00 max=   0.0000000000000000E+00 v Beginning of step_MOM  [uv]
v-point: c=         0 sw=         0 se=         0 nw=         0 ne=         0 v Beginning of step_MOM  [uv]
h-point: mean=   1.0000000000000000E+03 min=   2.0022236368437586E+02 max=   1.7997776363156240E+03 Beginning of step_MOM  h
h-point: c=    272640 sw=    274800 se=    274800 nw=    274740 ne=    274740 Beginning of step_MOM  h

Send to stderr (expect a lot of output!)

(See also DIAG_AS_CHKSUM)

Debug Syntax

h-point:

Diagnostic is on a center point

mean, min, max

Calculated over the compute domain (no halos)

c=

Bit checksum over compute domain

sw=, se=, nw=, ne=

Checksum after domain shifted into halos

Call Tree

In MOM_unit:

VERBOSITY=9

Produces call tree (and other info):

NOTE: callTree:    ---> step_MOM(), MOM.F90
NOTE: callTree:       loop DT cycles (step_MOM) n=       1
NOTE: callTree:          o done with set_viscous_BBL (step_MOM)
NOTE: callTree:          ---> step_MOM_dyn_split_RK2(), MOM_dynamics_split_RK2.F90
NOTE: callTree:             o done with PressureForce (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with vertvisc_coef (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity[BT_cont] (step_MOM_dyn_split_RK2)
NOTE: callTree:             ---> btstep(), MOM_barotropic.F90
NOTE: btstep is using a dynamic barotropic timestep of 7.500000E+01 seconds, max 8.529637E+01.
NOTE: callTree:             <--- btstep()
NOTE: callTree:             o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with btcalc[BT_cont_BT_thick] (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with horizontal_viscosity (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree:             ---> btstep(), MOM_barotropic.F90
NOTE: callTree:             <--- btstep()
NOTE: callTree:             o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree:          <--- step_MOM_dyn_split_RK2()
NOTE: callTree:          o finished step_MOM_dyn_split (step_MOM)
NOTE: callTree:          o finished thickness_diffuse (step_MOM)
NOTE: callTree:       <--- DT cycles (step_MOM)
NOTE: callTree:       loop DT cycles (step_MOM) n=       2
NOTE: callTree:          ---> step_MOM_dyn_split_RK2(), MOM_dynamics_split_RK2.F90
NOTE: callTree:             o done with PressureForce (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with vertvisc_coef (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity[BT_cont] (step_MOM_dyn_split_RK2)
NOTE: callTree:             ---> btstep(), MOM_barotropic.F90
NOTE: callTree:             <--- btstep()
NOTE: callTree:             o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with btcalc[BT_cont_BT_thick] (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with horizontal_viscosity (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree:             ---> btstep(), MOM_barotropic.F90
NOTE: callTree:             <--- btstep()
NOTE: callTree:             o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree:          <--- step_MOM_dyn_split_RK2()
NOTE: callTree:          o finished step_MOM_dyn_split (step_MOM)
NOTE: callTree:          o finished thickness_diffuse (step_MOM)
NOTE: callTree:       <--- DT cycles (step_MOM)
....
NOTE: callTree:       o calling extract_surface_state (step_MOM)
NOTE: callTree:       ---> extract_surface_state(), MOM.F90
NOTE: callTree:       <--- extract_surface_sfc_state()
NOTE: callTree:    <--- step_MOM()
NOTE: callTree: <--- Main loop

Exercise

Check out fork and compile:

cd MOM6-examples/src/MOM6
git fetch
git checkout mysterious-crash
# You remember how to compile, right?

Run Phillips 2-layer:

cd ../../ocean_only/Phillips_2layer
../../build/ocean_only/MOM6

What do you see?

E.1 Output

Segmentation fault, no symbols:

MOM Day       0.000      0: En 3.336197E-01, MaxCFL  0.01092, Mass 3.959040000000E+18
    Total Energy: 43B25478EC07BAB8  1.3208135458636902E+18
    Total Mass:   3.9590400000000000E+18, Change:   0.0000000000000000E+00 Error:  0.00000E+00 ( 0.0E+00)
NOTE: callTree:    <--- finish_MOM_initialization()
NOTE: callTree:    ---> step_MOM(), MOM.F90
NOTE: callTree:       loop DT cycles (step_MOM) n=       1
NOTE: callTree:          o done with set_viscous_BBL (step_MOM)
NOTE: callTree:          ---> step_MOM_dyn_split_RK2(), MOM_dynamics_split_RK2.F90
NOTE: callTree:             o done with PressureForce (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with vertvisc_coef (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity[BT_cont] (step_MOM_dyn_split_RK2)
NOTE: callTree:             ---> btstep(), MOM_barotropic.F90
NOTE: btstep is using a dynamic barotropic timestep of 7.500000E+01 seconds, max 8.529637E+01.
NOTE: callTree:             <--- btstep()
NOTE: callTree:             o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with btcalc[BT_cont_BT_thick] (step_MOM_dyn_split_RK2)
NOTE: callTree:             o done with horizontal_viscosity (step_MOM_dyn_split_RK2)

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fe847a6f9ff in ???
#1  0x55893a0b6dad in ???
#2  0x55893a086ee6 in ???
#3  0x55893a439281 in ???
#4  0x55893a39b4a0 in ???
#5  0x558939e76d0e in ???
#6  0x7fe847a5a28f in ???
#7  0x7fe847a5a349 in ???
#8  0x558939e76d44 in _start
	at ../sysdeps/x86_64/start.S:115
#9  0xffffffffffffffff in ???
Segmentation fault (core dumped)

E.1 Hint

Rebuild with DEBUG:

$ make clean
$ make DEBUG=1 -j

This restores symbols:

-g

Symbol tables

-fbacktrace

Enable traceback (the report at the end)

Now check the symbols

Exercise

Another fork to check out:

cd MOM6-examples/src/MOM6
git fetch
git checkout demo-layout-issue

Run with 1 and 4 CPUs:

cd ../../ocean_only/Phillips_2layer
mpirun -np 1 ../../build/ocean_only/MOM6
cp ocean.stats ocean.stats.p1
mpirun -np 4 ../../build/ocean_only/MOM6
cp ocean.stats ocean.stats.p4
diff ocean.stats.p1 ocean.stats.p4

Do they agree?

Style Matters

From Griffies and Hallberg (2000):

This paper discusses a NUMERICAL CLOSURE,motivated from the ideas of
Smagorinsky, for use with a
bi
   harmonic
      operator.
The result is    a  highly
   scale -selective,state- dependent
friction operator for use in
eddy  -permitting geophysical fluid models.

Would you write like this?

Syntax Style

  • Indent properly!

  • Respect line length (<=120 characters)

    • <80 is even better!

  • snake_case

  • Space around operators: if (a == 0)

  • No space in function args: call f(a, b, c=0)

Array Style

  • This is bad: u = 0.

  • This is OK: u(:,:,:) = 0.

    • But only for assignment and initialization

  • Explicit loops:

    do j = js, je ; do i = is, ie
  u(i,j) = v(i,j) + w(i,j)**2
enddo ; enddo

  • Index convention: ij on centers, IJ on edges

Why Style?

  • Consistent patterns: faster to read, faster to write
  • Reduce the size of git commits

Read the guide!