CMAQ

Legacy documentation This page describes a service provided by a retired ACENET system. Most ACENET services are currently provided by national systems, for which please visit https://docs.computecanada.ca.

Description

CMAQ is an active open-source development project of the U.S. EPA Atmospheric Science Modeling Division that consists of a suite of programs for conducting air quality model simulations.

Modulefile

cmaq

Documentation

CMAQ homepage

Notes

• User's experience working with CMAQ on our clusters might be different from that on a personal computer. The default CMAQ distribution appears to expect its users to have full access to the installation. The problem is that this approach is not compatible with how software gets installed on a shared facility to be accessible by all users, which is read-only. An advanced CMAQ user should be able to adjust to this setup rather quickly though.
• For an easy transition, we have modified the run.* files (henceforth the "run scripts") found in the scripts directory so that $BASE is set relative to the installation path, making the CMAQ run scripts independent of the current working directory. Moreover, the line that forcibly changes the current working directory to $BASE is commented.
• All the run-time relevant settings are provided via a modulefile file, and the CMAQ run scripts do not source config.cmaq, allowing users to overwrite $M3DATA.
• Users should be using C-shell in their submission scripts when utilizing CMAQ run scripts. Please ensure that there is an empty line at the end of your submissions script.
• The unlimit command has been commented in the run scripts, such as run.cctm. Users should be setting memory requirements in their job submission scripts. CCTM likely requires an increased stack size, so try specifying -l h_stack=100M if the code fails shortly after starting (see Memory Management).
• The only component that could be run in parallel appears to be CCTM. In order to execute it in parallel, users are expected to make changes to a run script as described below.

Usage

Using standard run scripts

Here is an example how to reproduce the Run Instructions from the readme file. First, prepare the environment:

$ module purge
$ module load intel openmpi/intel cmaq

Make a copy of the data directory (more than 6GB) in your space, either into /home or /home/$USER/scratch/ directory.

$ cd /home/$USER/scratch
$ mkdir cmaq
$ cd cmaq/
$ cp -a $M3HOME/data/* ./

Create a submission script (see below). The CCTM component will require a greater stack size as well as more than 2G of memory (see Memory Management), so specify them explicitly.

#$ -S /bin/csh
#$ -cwd
#$ -j y
#$ -l h_rt=01:00:00
#$ -l h_vmem=4G,h_stack=100M

module purge
module load intel openmpi/intel cmaq

setenv M3DATA /home/$USER/scratch/cmaq
$M3HOME/scripts/icon/run.icon
$M3HOME/scripts/bcon/run.bcon
$M3HOME/scripts/cctm/run.cctm

# Empty line. Keep it

Submit your script with the qsub command to the scheduler.

Customized run scripts

If you need to make modifications to a CMAQ run script, you can simply copy it into your current directory, make modifications and then call it from your submissions script. Here is an example:

 $ cp $M3HOME/scripts/bcon/run.bcon ./

Edit run.cctm to your liking and then create a submission script:

#$ -S /bin/csh
#$ -cwd
#$ -j y
#$ -l h_rt=01:00:00
#$ -l h_vmem=4G,h_stack=100M

module purge
module load intel openmpi/intel cmaq

setenv M3DATA /home/$USER/scratch/cmaq
./run.bcon

# Empty line. Keep it

Running CCTM in parallel

In order to run CCTM in parallel, one needs to make modifications to the CCTM run script. First, make a copy of the script into your current working directory:

 $ cp $M3HOME/scripts/cctm/run.cctm ./

Then modify the script in a few places to allow it to run in parallel. Let's consider an example for a 4-process MPI job. Below are the lines that you would need to find in run.cctm and change their values appropriately:

  set PROC     = mpi
  setenv NPCOL_NPROW "2 2"; set NPROCS   = 4
# /usr/bin/time  $BLD/$EXEC
  mpirun $BLD/$EXEC

Please note the commented line above. Also note that mpirun does not require any options or parameters, because it will get the number of processes to run with from the scheduler. Below is an example of a submission script. The h_vmem is per process, so it can be reduced compared to the serial run.

#$ -S /bin/csh
#$ -cwd
#$ -j y
#$ -l h_rt=01:00:00
#$ -l h_vmem=2G,h_stack=100M
#$ -pe ompi* 4

module purge
module load intel openmpi/intel cmaq

setenv M3DATA /home/$USER/scratch/cmaq
./run.cctm

# Empty line. Keep it

Running MCIP on the sample data

If you have made a copy of the data directory as described above then you already have sample input data for MCIP. Here is a submissions script to run it:

#$ -S /bin/csh
#$ -cwd
#$ -j y
#$ -l h_rt=01:00:00
#$ -l h_vmem=2G

module purge
module load intel openmpi/intel cmaq

setenv M3DATA /home/$USER/scratch/cmaq/mcip
$M3HOME/scripts/mcip/run.mcip

# Empty line. Keep it

If you want to process any other data then you will need to modifying the run script for MCIP, and thus you will need to copy it into your working directory, make modifications and then call it from your submissions script. Here is how:

 $ cp $M3HOME/scripts/mcip/run.mcip ./

Edit run.mcip to your liking and then create a submission script like so:

#$ -S /bin/csh
#$ -cwd
#$ -j y
#$ -l h_rt=01:00:00
#$ -l h_vmem=2G

module purge
module load intel openmpi/intel cmaq

setenv M3DATA /home/$USER/scratch/cmaq/mcip
./run.mcip

# Empty line. Keep it