The following instructions aims at setting up a baseline single-user environment for building MOOSE based applications in a job scheduling capable environment.

Pre-Reqs

What ever compiler you choose to use on your cluster (GCC/Clang, MPICH/OpenMPI), the minimum requirement, is that it must be C++11 compatible. If you are unsure, please consult with your system admins for your cluster on which compiler to use (and how to use it).

  • CMake. A modern version of CMake (>2.8) is required to build some of the meta packages we need to include in PETSc.

  • Python 2.7.x Development libraries. This normally is an easy addition to the system through the use of your package manager. Perhaps older distributions (such as RHEL 6) will need to manually install Python 2.7 and its accompanying development package.

PREFIX Setup

export CLUSTER_TEMP=`mktemp -d /tmp/cluster_temp.XXXXXX`
export PACKAGES_DIR=$HOME/moose-compilers

Note: The PACKAGES_DIR must reside in a location where all the compute nodes can access.

Note: The terminal you used to run that command, should be the terminal you use from now on while following the instructions below to completion.

Download Software

Follow one of the download steps below

  • Dwnload using Curl:
cd $CLUSTER_TEMP

curl -L -O http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.6.4.tar.gz
  • Download using Wget:
cd $CLUSTER_TEMP

wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.6.4.tar.gz

Extract Software

Extract PETSc:

cd $CLUSTER_TEMP
tar -xf petsc-3.6.4.tar.gz

Create MOOSE Profile

Make note of your $PACKAGES_DIR:

echo $PACKAGES_DIR

Edit the following file:

$HOME/.moose-profile

Add the following content to that file:

export PACKAGES_DIR=INSERT-PACKAGES-DIR-HERE!!!!

export CC=mpicc
export CXX=mpicxx
export F90=mpif90
export F77=mpif77
export FC=mpif90

export ARCH=gcc
export PETSC_DIR=$PACKAGES_DIR/petsc/petsc-3.6.4/$ARCH-opt

Note: You must insert your PACKAGES_DIR on the first line. If you are using something other than gcc you can change $ARCH. This is purely a cosmetic change so you can choose what you want (ie "intel"). To make the profile available in your terminal session throughout the rest of the instructions, run the following:

source ~/.moose-profile

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ------ VERY IMPORTANT ------ ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
The above will need to be executed for every new terminal session for which you perform work with MOOSE. If you want this to be automatic, add the above to your ~/.bash_profile (or ~/.bashrc or, which ever profile you use on your system)
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ------ VERY IMPORTANT ------ ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

Lets verify that the moose environment is available:

me@some_machine#>  echo $PETSC_DIR

That command should return what ever you decided your $PACKAGES_DIR to be, plus /petsc/petsc-3.6.4/gcc-opt

Build and Install PETSc

cd $CLUSTER_TEMP/petsc-3.6.4

./configure \
--prefix=$PETSC_DIR \
--download-hypre=1 \
--with-ssl=0 \
--with-debugging=no \
--with-pic=1 \
--with-shared-libraries=1 \
--with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--download-fblaslapack=1 \
--download-metis=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-scalapack=1 \
--download-mumps=1 \
CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 \
CFLAGS='-fPIC -fopenmp' \
CXXFLAGS='-fPIC -fopenmp' \
FFLAGS='-fPIC -fopenmp' \
FCFLAGS='-fPIC -fopenmp' \
F90FLAGS='-fPIC -fopenmp' \
F77FLAGS='-fPIC -fopenmp' \
PETSC_DIR=`pwd`

During the configure/build process, you will be prompted to enter the correct make / make install commands. Because this can be different from system to system, I leave that task to the reader.

Clean Up

Simple... clean all the temporary stuff:

rm -rf $CLUSTER_TEMP

You are done! You should now be able to continue to Step 2 on the Getting Started pages.