https://marshallward.org/mom6workshop/build.html

Building MOM6

Marshall Ward

2022-10-17

Presentation is available at

https://marshallward.org/mom6workshop/build.html

Preparation

Fortran compiler
- GCC Fortran (gfortran)
- Intel Fortran (ifort) on HPC
- ftn wrapper (Cray)
MPI library
- Open MPI (usually faster)
- MPICH (usually safer, esp on Macs)
netCDF for I/O

Gaea

HPC systems use environment modules:

# Remove active modules
module unload PrgEnv-gnu PrgEnv-intel PrgEnv-pgi

module load PrgEnv-gnu
module swap gcc gcc/9.3.0
module load cray-netcdf

Stellar

Similar to Gaea:

module purge

module load gcc-toolset/10
module load openmpi/gcc/4.1.0
module load hdf5/gcc-toolset-10/1.10.6
module load netcdf/gcc/hdf5-1.10.6/4.7.4

Ubuntu/Debian

Package installation:

apt install make
apt install gfortran
apt install netcdf-bin
apt install libnetcdf-dev
apt install libnetcdff-dev
apt install openmpi-bin
apt install libopenmpi-dev

MacOS

MacOS is a minefield! But Homebrew has the best chance:

brew install make
brew install gfortran
brew install mpich
brew install netcdf
brew install netcdf-fortran

Software can also be installed with Conda.

Get the Code

Get the MOM6-examples monorepo:

$ git clone https://github.com/NOAA-GFDL/MOM6-examples.git \
   --recursive

Verify submodules:

$ cd MOM6-examples
$ git submodule update --init --recursive

GFDL Tools

list_paths: Generates a list of files to be compiled (usually in path_names)
mkmf: Generates a Makefile from path_names and a platform template (e.g. linux-gnu.mk)

FMS Setup

FMS is GFDL's climate model framework

Create the path_names manifest:

$ mkdir -p build/fms
$ cd build/fms
$ ../../src/mkmf/bin/list_paths -l ../../src/FMS

Sample output:

../../src/FMS/horiz_interp/horiz_interp_bilinear.F90
../../src/FMS/horiz_interp/horiz_interp_type.F90
../../src/FMS/horiz_interp/horiz_interp_bicubic.F90
../../src/FMS/horiz_interp/horiz_interp.F90
(...)

`list_paths`

-l: Follow symbolic links (needed for MOM6)
<path> [<path> ...]: Directories to search for source

Creates a file named path_names

Build FMS

Generate the Makefile:

$ ../../src/mkmf/bin/mkmf \
   -t ../../src/mkmf/templates/linux-gnu.mk \
   -p libfms.a \
   -c "-Duse_libMPI -Duse_netCDF" \
   path_names

and build:

$ make NETCDF=3 REPRO=1 libfms.a -j

`mkmf`

-t template: Select a MKMF template file (linux-gnu.mk)
-p program: Target program (MOM6) or library (libfms.a)
-c CPPDEFS: Preprocessor defines (-Duse_netCDF)

MKMF Template

# Template for the GNU Compiler Collection on Linux systems
#
# Typical use with mkmf
# mkmf -t linux-gnu.mk -c"-Duse_libMPI -Duse_netCDF" path_names /usr/local/include

############
# Commands Macors
FC = gfortran
CC = gcc
CXX = g++
LD = gfortran $(MAIN_PROGRAM)

#######################
# Build target macros
#
# Macros that modify compiler flags used in the build.  Target
# macrose are usually set on the call to make:
#
#    make REPRO=on NETCDF=3
#
# Most target macros are activated when their value is non-blank.
# Some have a single value that is checked.  Others will use the
# value of the macro in the compile command.

DEBUG =              # If non-blank, perform a debug build (Cannot be
                     # mixed with REPRO or TEST)

REPRO =              # If non-blank, erform a build that guarentees
                     # reprodicuibilty from run to run.  Cannot be used
                     # with DEBUG or TEST

TEST  =              # If non-blank, use the compiler options defined in
                     # the FFLAGS_TEST and CFLAGS_TEST macros.  Cannot be
                     # use with REPRO or DEBUG

VERBOSE =            # If non-blank, add additional verbosity compiler
                     # options

OPENMP =             # If non-blank, compile with openmp enabled

NO_OVERRIDE_LIMITS = # If non-blank, do not use the -qoverride-limits
                     # compiler option.  Default behavior is to compile
                     # with -qoverride-limits.

NETCDF =             # If value is '3' and CPPDEFS contains
                     # '-Duse_netCDF', then the additional cpp macro
                     # '-Duse_LARGEFILE' is added to the CPPDEFS macro.

INCLUDES =           # A list of -I Include directories to be added to the
                     # the compile command.

SSE =                # The SSE options to be used to compile.  If blank,
                     # than use the default SSE settings for the host.
                     # Current default is to use SSE2.

COVERAGE =           # Add the code coverage compile options.

# Need to use at least GNU Make version 3.81
need := 3.81
ok := $(filter $(need),$(firstword $(sort $(MAKE_VERSION) $(need))))
ifneq ($(need),$(ok))
$(error Need at least make version $(need).  Load module gmake/3.81)
endif

# REPRO, DEBUG and TEST need to be mutually exclusive of each other.
# Make sure the user hasn't supplied two at the same time
ifdef REPRO
ifneq ($(DEBUG),)
$(error Options REPRO and DEBUG cannot be used together)
else ifneq ($(TEST),)
$(error Options REPRO and TEST cannot be used together)
endif
else ifdef DEBUG
ifneq ($(TEST),)
$(error Options DEBUG and TEST cannot be used together)
endif
endif

MAKEFLAGS += --jobs=$(shell grep '^processor' /proc/cpuinfo | wc -l)

# Required Preprocessor Macros:
CPPDEFS += -Duse_netCDF

# Additional Preprocessor Macros needed due to  Autotools and CMake
CPPDEFS += -DHAVE_SCHED_GETAFFINITY

# Macro for Fortran preprocessor
FPPFLAGS := $(INCLUDES)
# Fortran Compiler flags for the NetCDF library
FPPFLAGS += $(shell nf-config --fflags)
# Fortran Compiler flags for the MPICH MPI library
FPPFLAGS += $(shell pkg-config --cflags-only-I mpich2)

# Base set of Fortran compiler flags
FFLAGS := -fcray-pointer -fdefault-double-8 -fdefault-real-8 -Waliasing -ffree-line-length-none -fno-range-check

# Flags based on perforance target (production (OPT), reproduction (REPRO), or debug (DEBUG)
FFLAGS_OPT = -O3
FFLAGS_REPRO = -O2 -fbounds-check
FFLAGS_DEBUG = -O0 -g -W -fbounds-check -fbacktrace -ffpe-trap=invalid,zero,overflow

# Flags to add additional build options
FFLAGS_OPENMP = -fopenmp
FFLAGS_VERBOSE =
FFLAGS_COVERAGE =

# Macro for C preprocessor
CPPFLAGS := $(INCLUDES)
# C Compiler flags for the NetCDF library
CPPFLAGS += $(shell nc-config --cflags)
# C Compiler flags for the MPICH MPI library
CPPFLAGS += $(shell pkg-config --cflags-only-I mpich2)

# Base set of C compiler flags
CFLAGS := -D__IFC

# Flags based on perforance target (production (OPT), reproduction (REPRO), or debug (DEBUG)
CFLAGS_OPT = -O2
CFLAGS_REPRO = -O2
CFLAGS_DEBUG = -O0 -g

# Flags to add additional build options
CFLAGS_OPENMP = -fopenmp
CFLAGS_VERBOSE =
CFLAGS_COVERAGE =

# Optional Testing compile flags.  Mutually exclusive from DEBUG, REPRO, and OPT
# *_TEST will match the production if no new option(s) is(are) to be tested.
FFLAGS_TEST := $(FFLAGS_OPT)
CFLAGS_TEST := $(CFLAGS_OPT)

# Linking flags
LDFLAGS :=
LDFLAGS_OPENMP := -fopenmp
LDFLAGS_VERBOSE :=
LDFLAGS_COVERAGE :=

# Start with a blank LIBS
LIBS =
# NetCDF library flags
LIBS += $(shell nf-config --flibs)
# MPICH MPI library flags
LIBS += $(shell pkg-config --libs mpich2-f90)

# Get compile flags based on target macros.
ifdef REPRO
CFLAGS += $(CFLAGS_REPRO)
FFLAGS += $(FFLAGS_REPRO)
else ifdef DEBUG
CFLAGS += $(CFLAGS_DEBUG)
FFLAGS += $(FFLAGS_DEBUG)
else ifdef TEST
CFLAGS += $(CFLAGS_TEST)
FFLAGS += $(FFLAGS_TEST)
else
CFLAGS += $(CFLAGS_OPT)
FFLAGS += $(FFLAGS_OPT)
endif

ifdef OPENMP
CFLAGS += $(CFLAGS_OPENMP)
FFLAGS += $(FFLAGS_OPENMP)
LDFLAGS += $(LDFLAGS_OPENMP)
endif

ifdef SSE
CFLAGS += $(SSE)
FFLAGS += $(SSE)
endif

ifdef NO_OVERRIDE_LIMITS
FFLAGS += $(FFLAGS_OVERRIDE_LIMITS)
endif

ifdef VERBOSE
CFLAGS += $(CFLAGS_VERBOSE)
FFLAGS += $(FFLAGS_VERBOSE)
LDFLAGS += $(LDFLAGS_VERBOSE)
endif

ifeq ($(NETCDF),3)
  # add the use_LARGEFILE cppdef
  ifneq ($(findstring -Duse_netCDF,$(CPPDEFS)),)
    CPPDEFS += -Duse_LARGEFILE
  endif
endif

ifdef COVERAGE
ifdef BUILDROOT
PROF_DIR=-prof-dir=$(BUILDROOT)
endif
CFLAGS += $(CFLAGS_COVERAGE) $(PROF_DIR)
FFLAGS += $(FFLAGS_COVERAGE) $(PROF_DIR)
LDFLAGS += $(LDFLAGS_COVERAGE) $(PROF_DIR)
endif

LDFLAGS += $(LIBS)

#---------------------------------------------------------------------------
# you should never need to change any lines below.

# see the MIPSPro F90 manual for more details on some of the file extensions
# discussed here.
# this makefile template recognizes fortran sourcefiles with extensions
# .f, .f90, .F, .F90. Given a sourcefile <file>.<ext>, where <ext> is one of
# the above, this provides a number of default actions:

# make <file>.opt	create an optimization report
# make <file>.o		create an object file
# make <file>.s		create an assembly listing
# make <file>.x		create an executable file, assuming standalone
#			source
# make <file>.i		create a preprocessed file (for .F)
# make <file>.i90	create a preprocessed file (for .F90)

# The macro TMPFILES is provided to slate files like the above for removal.

RM = rm -f
TMPFILES = .*.m *.B *.L *.i *.i90 *.l *.s *.mod *.opt

.SUFFIXES: .F .F90 .H .L .T .f .f90 .h .i .i90 .l .o .s .opt .x

.f.L:
	$(FC) $(FFLAGS) -c -listing $*.f
.f.opt:
	$(FC) $(FFLAGS) -c -opt_report_level max -opt_report_phase all -opt_report_file $*.opt $*.f
.f.l:
	$(FC) $(FFLAGS) -c $(LIST) $*.f
.f.T:
	$(FC) $(FFLAGS) -c -cif $*.f
.f.o:
	$(FC) $(FFLAGS) -c $*.f
.f.s:
	$(FC) $(FFLAGS) -S $*.f
.f.x:
	$(FC) $(FFLAGS) -o $*.x $*.f *.o $(LDFLAGS)
.f90.L:
	$(FC) $(FFLAGS) -c -listing $*.f90
.f90.opt:
	$(FC) $(FFLAGS) -c -opt_report_level max -opt_report_phase all -opt_report_file $*.opt $*.f90
.f90.l:
	$(FC) $(FFLAGS) -c $(LIST) $*.f90
.f90.T:
	$(FC) $(FFLAGS) -c -cif $*.f90
.f90.o:
	$(FC) $(FFLAGS) -c $*.f90
.f90.s:
	$(FC) $(FFLAGS) -c -S $*.f90
.f90.x:
	$(FC) $(FFLAGS) -o $*.x $*.f90 *.o $(LDFLAGS)
.F.L:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -listing $*.F
.F.opt:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -opt_report_level max -opt_report_phase all -opt_report_file $*.opt $*.F
.F.l:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c $(LIST) $*.F
.F.T:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -cif $*.F
.F.f:
	$(FC) $(CPPDEFS) $(FPPFLAGS) -EP $*.F > $*.f
.F.i:
	$(FC) $(CPPDEFS) $(FPPFLAGS) -P $*.F
.F.o:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c $*.F
.F.s:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -S $*.F
.F.x:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -o $*.x $*.F *.o $(LDFLAGS)
.F90.L:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -listing $*.F90
.F90.opt:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -opt_report_level max -opt_report_phase all -opt_report_file $*.opt $*.F90
.F90.l:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c $(LIST) $*.F90
.F90.T:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -cif $*.F90
.F90.f90:
	$(FC) $(CPPDEFS) $(FPPFLAGS) -EP $*.F90 > $*.f90
.F90.i90:
	$(FC) $(CPPDEFS) $(FPPFLAGS) -P $*.F90
.F90.o:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c $*.F90
.F90.s:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -c -S $*.F90
.F90.x:
	$(FC) $(CPPDEFS) $(FPPFLAGS) $(FFLAGS) -o $*.x $*.F90 *.o $(LDFLAGS)

NETCDF=3 and REPRO=1 are defined in templates

Recommended Templates

Platform	Template
Gaea	ncrc-gnu.mk
Stellar	linux-ubuntu-jammy-gnu.mk
Ubuntu	linux-ubuntu-jammy-gnu.mk
MacOS	osx-gcc10.mk

MOM6 Setup

$ cd ../..
$ mkdir -p build/ocean_only
$ cd build/ocean_only

Manifest:

$ ../../src/mkmf/bin/list_paths -l \
    ../../src/MOM6/config_src/infra/FMS1/ \
    ../../src/MOM6/config_src/memory/dynamic_symmetric/ \
    ../../src/MOM6/config_src/drivers/solo_driver/ \
    ../../src/MOM6/config_src/external/ \
    ../../src/MOM6/src/*/ \
    ../../src/MOM6/src/*/*/

MOM6 Build Config

Directory	Component
`config_src/infra/FMS1`	Framework
`config_src/memory/dynamic_symmetric`	Memory layout
`config_src/drivers/solo_driver`	Target program/library
`config_src/external`	External packages

See "MOM6 Internals" for more information

Build MOM6

Create Makefile:

$ ../../src/mkmf/bin/mkmf \
   -t ../../src/mkmf/templates/linux-gnu.mk \
   -o '-I../fms' \
   -p MOM6 \
   -l '-L../fms -lfms' \
   path_names

Make it:

$ make NETCDF=3 REPRO=1 MOM6 -j

mkmf part 2

-o: "Other" flags (-I../../shared/repro)
-l: Linker flags (-L../../shared/repro -lfms)

Final Product

Test the executable:

$ cd ../../ocean_only/double_gyre
$ ../../build/ocean_only/MOM6
$ mkdir RESTART

Expected output:

NOTE: MPP_DOMAINS_SET_STACK_SIZE: stack size set to    32768.
&MPP_IO_NML
 HEADER_BUFFER_VAL=16384      ,
 GLOBAL_FIELD_ON_ROOT_PE=T,
 IO_CLOCKS_ON=F,
 SHUFFLE=0          ,
 DEFLATE_LEVEL=-1         ,
 CF_COMPLIANCE=F,
 /
NOTE: MPP_IO_SET_STACK_SIZE: stack size set to     131072.
NOTE: ======== Model being driven by MOM_driver ========
NOTE: callTree: o Program MOM_main, MOM_driver.F90
NOTE: open_param_file: MOM_input has been opened successfully.
NOTE: open_param_file: MOM_override has been opened successfully.
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   44
  Y-AXIS =   40
 MOM_inc domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   22
  Y-AXIS =   20
NOTE: diag_manager_mod::diag_manager_init: prepend_date only supported when diag_manager_init is called with time_init present.
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   44
  Y-AXIS =   40
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   22
  Y-AXIS =   20
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   44
  Y-AXIS =   40
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   22
  Y-AXIS =   20

WARNING: KVML is a deprecated parameter. Use KV_ML_INVZ2 instead.

 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   44
  Y-AXIS =   40
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   22
  Y-AXIS =   20

WARNING: MPP_OPEN: File ./CPU_stats opened WRONLY already exists!


WARNING: MPP_OPEN: File ./ocean.stats opened WRONLY already exists!

MOM Day       0.000      0: En 1.423750E-13, MaxCFL  0.00000, Mass 5.288178268008E+18
    Total Energy: 4126FA1057A6FFDE  7.5290417119597993E+05
    Total Mass:   5.2881782680077681E+18, Change:   0.0000000000000000E+00 Error:  0.00000E+00 ( 0.0E+00)
MOM Day       1.000     72: En 4.580432E-06, MaxCFL  0.00023, Mass 5.288178268008E+18
    Total Energy: 42B607A80A97FCE0  2.4222139848700875E+13
    Total Mass:   5.2881782680077681E+18, Change:  -7.9338251715126091E+01 Error: -7.93383E+01 (-1.5E-17)
MOM Day       2.000    144: En 8.958206E-06, MaxCFL  0.00040, Mass 5.288178268008E+18
    Total Energy: 42C58AE5334C4E2E  4.7372590028956359E+13
    Total Mass:   5.2881782680077681E+18, Change:   4.8792347397912323E+01 Error:  4.87923E+01 ( 9.2E-18)
MOM Day       3.000    216: En 1.166708E-05, MaxCFL  0.00045, Mass 5.288178268008E+18
    Total Energy: 42CC0E8B5CD9282E  6.1697586475600359E+13
    Total Mass:   5.2881782680077681E+18, Change:  -9.5955835958679359E+00 Error: -9.59558E+00 (-1.8E-18)
MOM Day       4.000    288: En 1.542296E-05, MaxCFL  0.00044, Mass 5.288178268008E+18
    Total Energy: 42D28B60D587BD4C  8.1559337443061188E+13
    Total Mass:   5.2881782680077681E+18, Change:  -4.0827784519121622E+00 Error: -4.08278E+00 (-7.7E-19)
MOM Day       5.000    360: En 1.826022E-05, MaxCFL  0.00053, Mass 5.288178268008E+18
    Total Energy: 42D5F4B97D62074A  9.6563312822301156E+13
    Total Mass:   5.2881782680077681E+18, Change:  -9.9329031525191169E+00 Error: -9.93290E+00 (-1.9E-18)
MOM Day       6.000    432: En 2.397431E-05, MaxCFL  0.00062, Mass 5.288178268008E+18
    Total Energy: 42DCD397A5E918F6  1.2678043163965184E+14
    Total Mass:   5.2881782680077681E+18, Change:   8.5185222625386530E+00 Error:  8.51852E+00 ( 1.6E-18)
MOM Day       7.000    504: En 2.927962E-05, MaxCFL  0.00073, Mass 5.288178268008E+18
    Total Energy: 42E19A5109E6F0B8  1.5483585791373375E+14
    Total Mass:   5.2881782680077681E+18, Change:   2.6509695689069929E+01 Error:  2.65097E+01 ( 5.0E-18)
MOM Day       8.000    576: En 3.486441E-05, MaxCFL  0.00081, Mass 5.288178268008E+18
    Total Energy: 42E4F5D995BC277A  1.8436920009554781E+14
    Total Mass:   5.2881782680077681E+18, Change:  -3.9583308161166826E+01 Error: -3.95833E+01 (-7.5E-18)
MOM Day       9.000    648: En 4.195565E-05, MaxCFL  0.00090, Mass 5.288178268008E+18
    Total Energy: 42E9393C9CAF84FA  2.2186896585424781E+14
    Total Mass:   5.2881782680077681E+18, Change:   4.5657849825742858E+00 Error:  4.56578E+00 ( 8.6E-19)
MOM Day      10.000    720: En 4.604729E-05, MaxCFL  0.00100, Mass 5.288178268008E+18
    Total Energy: 42EBAEF71B9AFB0F  2.4350627232764047E+14
    Total Mass:   5.2881782680077681E+18, Change:  -8.9726024220902332E+00 Error: -8.97260E+00 (-1.7E-18)

WARNING: MPP_OPEN: File RESTART/ocean_solo.res opened WRONLY already exists!


WARNING: MPP_OPEN: File exitcode opened WRONLY already exists!

 MPP_DOMAINS_STACK high water mark=           0

Tabulating mpp_clock statistics across      1 PEs...

                                          tmin          tmax          tavg          tstd  tfrac grain pemin pemax
Total runtime                         2.962801      2.962801      2.962801      0.000000  1.000     0     0     0
Initialization                        0.032274      0.032274      0.032274      0.000000  0.011     0     0     0
Main loop                             2.925114      2.925114      2.925114      0.000000  0.987     0     0     0
Termination                           0.004254      0.004254      0.004254      0.000000  0.001     0     0     0

WARNING: KVML is a deprecated parameter. Use KV_ML_INVZ2 instead.


WARNING: MPP_OPEN: File ./CPU_stats opened WRONLY already exists!


WARNING: MPP_OPEN: File ./ocean.stats opened WRONLY already exists!


WARNING: MPP_OPEN: File RESTART/ocean_solo.res opened WRONLY already exists!


WARNING: MPP_OPEN: File exitcode opened WRONLY already exists!

                              20221016 085821.362: Memuse(MB) at Memory HiWaterMark=  5.875E+01  5.875E+01  0.000E+00  5.875E+01
 MPP_STACK high water mark=           0

Exercises

Build the ice-ocean model using mkmf:

https://github.com/NOAA-GFDL/MOM6-examples/wiki/Getting-started#compiling-mom6-in-mom6-sis2-coupled-mode
Build MOM6 using autoconf and run the test suite:

https://github.com/NOAA-GFDL/MOM6/tree/dev/gfdl/ac

Attempt to run the test suite:

$ cd src/MOM6/.testing
$ make -j
$ make -j test