ANSYS

Legacy documentation This page describes a service provided by a retired ACENET system. Most ACENET services are currently provided by national systems, for which please visit https://docs.computecanada.ca.

Description

ANSYS in a collection of engineering simulation software.

Modulefile

ansys

License Agreement

ACENET has acquired licenses for ANSYS Academic Research CFD and ANSYS Academic Research HPC. In terms of modules, ACENET has Fluent and CFX (+ workbench). There are 5 academic licenses at Placentia, providing access to 4 processes each. In addition, 40 HPC licenses are available. So, if you submit a 10-process job, it will utilize 1 academic license and 6 HPC licenses.

Before using any of the ANSYS products for the first time at ACENET you must contact support via email (and also send a copy of your request to admin@ace-net.ca) with your username, the name of your project leader, and a statement that you agree to the terms and conditions set out in our license. When your statement has been filed your ACENET account will be granted the correct permissions for you to use Fluent.

You can tell this permission has been granted when "fluent" appears in the output of the "groups" command:

$ groups
jsmith fluent

The group is called "fluent" for historical reasons. You will be granted access to all ANSYS products we have available.

Running ANSYS products in parallel

There are several important consideration when running ANSYS products in parallel:

1. The preferred MPI implementation for ANSYS products is "Platform MPI" (distributed with the software) as it is supported across all ANSYS products and provides transparent access to a variety of interconnects. Some times it is default, other times it has to be requested explicitly.
2. The "Platform MPI" does not integrate seamlessly with Grid Engine and thus additional configuration is required.
   • You must request the 'mpich' parallel environment
   • The remote client shell must NOT be specified as 'ssh'
3. There is no need to specify the type of interconnect: the fastest will be selected automatically.
4. All necessary shell environment settings and configurations are provided via modulefiles.

Please see examples below how to run CFX and FLUENT in parallel.

CFX

Here is an example how to run CFX in parallel:

#$ -cwd
#$ -l h_rt=1:0:0
#$ -l h_vmem=2G
#$ -pe mpich 8

module load ansys

PAR_HOSTS=$(cat $PE_HOSTFILE | awk 'BEGIN {H=""; S=""}{for (i=1; i<=$2; i++) {H=$1 S H; S=","} }END {print H}')

cfx5solve -parallel -partition $NSLOTS -par-dist $PAR_HOSTS -double -def Benchmark.def -start-method "Platform MPI Distributed Parallel"

The above example uses the definition file called "Benchmark.def" as input for the simulation. This obviously needs to be adjusted to match the actual filename.

The simulation will then produce a result file with a ".res" file extension, which can later be opened in CFD-post (File / Load results).

FLUENT

Submission scripts

Here is an example how to run FLUENT as a serial job:

#$ -cwd
#$ -l h_rt=1:0:0
#$ -l h_vmem=2G

module load ansys
fluent 3d -g -i example.jou

Interactive work

Job setup and post-processing should be carried out using qrsh to get an interactive session on one of the compute nodes. Some examples:

$ module load ansys/15.0
$ qrsh -cwd -l h_rt=1:0:0,test=true fluent 3ddp
$ qrsh -cwd -l h_rt=1:0:0,test=true fluent 2ddp -post

When it comes time to iterate your model to a solution you should ensure your case and mesh files are saved, and then use a journal file to submit a non-interactive (or "batch") job to Grid Engine. You will have to create the journal file, either manually or using Fluent interactively. The journal file will look something like this:

file/read-case-data test001.cas
solve/iterate 10000
file/write-case-data
y
exit

Parallel Fluent and license tokens

You can get high-performance computing in Fluent by running in parallel. The licensing scheme requires two different types of license tokens for parallel jobs. These license tokens are represented in Grid Engine by requestable resources fluentall and fluent-par. If you include these resource requests in your submission script then Grid Engine can ensure that there are an appropriate number of license tokens available before starting your job. Unfortunately, the combined quirks of the two-token scheme from ANSYS and Grid Engine's per-slot request syntax makes it a little bit complicated to make the correct request.

Fluent requires one fluentall token per job, which permits up to four parallel processes. Parallel processes in excess of four require one fluent-par token each.

Here is an example script for a job using 16 processors:

#$ -cwd
#$ -l h_rt=1:0:0
#$ -l h_vmem=2G
#$ -pe mpich 16
#$ -l fluentall=0.0625
#$ -l fluent-par=0.75

module load ansys/15.0
fluent 3d -g -i example.jou

There are four important changes from the serial example given above:

1. You must ensure that your model and journal file are set up for the same number of processors.
2. You must request the mpich parallel environment.
3. -l fluentall=0.0625: Grid Engine multiplies all resource requests by the number of parallel slots, so in order to get the one fluentall token required, you must request 1/N tokens, where N is the number of parallel slots requested. In this example, N=16 and 1/16 = 0.0625, so you request fluentall=0.0625.
4. -l fluent-par=0.75: Fluent also requires one fluent-par token for each parallel slot over and above four slots. So for an N-slot parallel job, you need N-4 fluent-par tokens. Because resource requests are multiplied by the slot count N, the value of fluent-par should be (N-4)/N, which in this example is (16-4)/16 = 0.75. If you use 4 or fewer parallel slots, do not request any fluent-par tokens.