User Installed Software - StarCCM

StarCCM

Prerequisites

For versions older than 8.02.008-R8, these instructions does not work.

Procuring the software

Installing

Follow the installation instructions that comes with StartCCM itself.

Choose either a directory under /pfs/nobackup$HOME or $HOME/Public as installation directory, otherwise the batch nodes will not be able to access the software.

All versions

The submitfile should always contain the following:

hostlist --expand --append=':' --append-slurm-tasks=$SLURM_TASKS_PER_NODE $SLURM_JOB_NODELIST  > machinefile

Version 8 and 9

The submitfile should contain the following:

module add GCC/5.4.0-2.26 OpenMPI/1.6.5
export OMPI_MCA_btl_tcp_if_exclude=lo,ib0

Use these arguments to starccm+

-machinefile machinefile -mpi openmpi -np $SLURM_NTASKS -cpubind off

Submitfile example (Abisko)

This is an example of the important parts of a working submitfile.

#!/bin/bash
#SBATCH -A <your-project-id>
#SBATCH -n 48
#SBATCH --ntasks-per-node=12
#SBATCH -t 2:00:00

hostlist --expand --append=':' --append-slurm-tasks=$SLURM_TASKS_PER_NODE $SLURM_JOB_NODELIST  > machinefile

module add GCC/5.4.0-2.26 OpenMPI/1.6.5
export OMPI_MCA_btl_tcp_if_exclude=lo,ib0

STARDIR=<path-to-your-starccm-installation>

$STARDIR/star/bin/starccm+ -machinefile machinefile -mpi openmpi -np $SLURM_NTASKS -cpubind off -nbuserdir <path-to-your-starccm-user-dir> <any-other-starccm-parameters>

If whole nodes are allocated one can use other cpubind arguments, see your starccm documentation for options.

Choose --ntasks-per-node values as multiples of 6 on Abisko.

Version 11

Version 11 (at least 11.06.011-R8) is built with a too old GCC compiler to be able to use on of the site installed OpenMPI versions, therefore one must use the Intel MPI packaged with Star-CCM+ itself. There is also a problem with the default fabric choosen by that MPI version and the TCP fabric must be explicitly selected.

The important flags are therefore:

-machinefile machinefile -mpi intel -mpiflags -slurm -np $SLURM_NTASKS -fabric TCP

Submit file example

#!/bin/bash
#SBATCH -A <your-project-id>
#SBATCH -J star-ccm-v11
#SBATCH -n 56
#SBATCH -t 1:00:00

hostlist --expand --append=':' --append-slurm-tasks=$SLURM_TASKS_PER_NODE $SLURM_JOB_NODELIST  > machinefile

STARDIR=<path-to-your-starccm-installation>

$STARDIR/star/bin/starccm+ -machinefile machinefile -mpi intel -mpiflags -slurm -np $SLURM_NTASKS -fabric TCP -nbuserdir <path-to-your-starccm-user-dir> <any-other-starccm-parameters>

 

Updated: 2017-09-21, 11:05