Follow these instructions to build a basic framework stack on Linux.

0. Pre-Reqs

  • CMake. A modern version of CMake is required to build PETSc. To verify you currently, or correctly have installed CMake, perform the following:
which cmake
cmake --version

If a path returns, you have CMake. CMake version 3.6 or greater is recommended. These instructions have not been tested with a version earlier than 3.6.

  • Python 2.7.x Development libraries. This normally is an easy addition to the system through the use of your package manager. Perhaps older distributions (such as RHEL 6) will need to manually install Python 2.7 and its accompanying development package.

1. Environment Setup

Next we need to have a complete and sane environment. Copy the following to the end of your ~/.bash_profile:

# Compiler Variables
export CC=mpicc
export CXX=mpicxx
export F77=mpif77
export FC=mpif90

# Library location
export PACKAGES_DIR=/opt/moose

# Helper variables
export GCC_HOME=$PACKAGES_DIR/gcc_6.2.0
export MPI_HOME=$PACKAGES_DIR/mpich/mpich_3.2
export PETSC_DIR=$PACKAGES_DIR/petsc/mpich_petsc-3.7.6/gcc-opt

# PATH
export PATH=$GCC_HOME/bin:$MPI_HOME/bin:$PATH

# LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$GCC_HOME/lib:$GCC_HOME/lib64:$GCC_HOME/lib/gcc/x86_64-pc-linux-gnu/6.2.0:$GCC_HOME/libexec/gcc/x86_64-pc-linux-gnu/6.2.0:$LD_LIBRARY_PATH

# JOB Count (the number of cores available on this machine)
export MOOSE_JOBS=4

Now close all of your open terminals and re-open them. This will allow the above information to be present in your environment.

Next make sure you have an area setup to download packages and a place to build packages that need to be built. We recommend using a ~/Downloads folder for your downloads and ~/projects/src, for your build directory. Create the build directory now:

mkdir -p ~/projects/src

Basic Compiler

In order to build a modern GNU compiler (C++11 compatible), we will need a basic one. On the Linux platform, this is normally already available to you. You can verify this by performing two which commands and verifying if anything returns:

which gcc
which g++

If nothing returns from either of those commands, you will need to first install a basic compiler (bootstrapping) before we can begin building modern compiler. Unfortunately, that is beyond the scope of this document, as every flavor of Linux has a different means of going about this process. Search the internets for 'how to install GCC on [insert linux distribution name]'.

2. GCC 6.2.0

If your current GCC compiler is >=4.8.4, you may skip step 2.

Download the following tarball: gcc-6.2.0.tar.gz. Note, this is a modified version of GCC. Basically, its GCC with the prereqs included in the tarball. If that is a concern to you (that this tarball is not a true GNU GCC tarball), then please proceed to the gnu.org web site and obtain a true copy of the gcc 6.2 tarball and perform the prereq steps as listed in their HOWTO. Again, the only difference with our tarball copy, is that we have performed the 'prereqs' steps, and then re-packaged the tarball. This is the same tarball that is used when building GCC in our redistributable package.

Build GCC

Extract the tarball, configure and build GCC

cd ~/projects/src
tar -xf ~/Downloads/gcc-6.2.0.tar.gz -C .
mkdir ~/projects/src/gcc-build
cd ~/projects/src/gcc-build

unset CC
unset CXX
unset FC
unset F90
unset F77

../gcc-6.2.0/configure --prefix=$GCC_HOME --disable-multilib --enable-languages=c,c++,fortran --enable-lto
make -j $MOOSE_JOBS
sudo -E make install

Verify you have correctly built/installed GCC by performing the following command:

which g++

If the above command does not return /opt/moose/gcc_6.2.0/bin/g++ then something went wrong and you should not proceed until this is fixed. Otherwise it will be a wast of time...

3. MPICH 3.2

We recommend compiling and using MPICH for your MPI needs.
Download mpich-3.2. Once downloaded, perform the following:

cd ~/projects/src
tar -xf ~/Downloads/mpich-3.2.tar.gz -C .
mkdir ~/projects/src/mpich-3.2/gcc-opt
cd ~/projects/src/mpich-3.2/gcc-opt

../configure --prefix=$MPI_HOME --enable-shared --enable-sharedlibs=clang --enable-fast=03 \
--enable-debuginfo --enable-totalview --enable-two-level-namespace CC=gcc CXX=g++ \
FC=gfortran F77=gfortran F90='' CFLAGS='' CXXFLAGS='' FFLAGS='' FCFLAGS='' F90FLAGS='' F77FLAGS=''

make -j $MOOSE_JOBS
sudo -E make install

Verify you have correctly built/installed MPICH by performing the following command:

which mpicc

If the above command does not return /opt/moose/mpich/mpich_3.2/bin/mpicc then something went wrong and you should not proceed until this is fixed. Otherwise it will be a wast of time...


4. PETSc

PETSc is our preferred linear and nonlinear solver package for MOOSE. If you are on a Linux system avoid getting a package distribution of PETSc because it may not come configured the way we want. Specifically it will most likely NOT be compiled with HYPRE support and will likely be tied to a different MPI installation.

Download petsc-3.7.6.tar.gz. Once downloaded, perform the following:

tar -xf ~/Downloads/petsc-3.7.6.tar.gz -C ~/projects/src
cd ~/projects/src/petsc-3.7.6
./configure \
--prefix=$PETSC_DIR \
--download-hypre=1 \
--with-ssl=0 \
--with-debugging=no \
--with-pic=1 \
--with-shared-libraries=1 \
--with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--download-fblaslapack=1 \
--download-metis=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-mumps=1 \
--download-scalapack=1 \
CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 \
CFLAGS='-fPIC -fopenmp' \
CXXFLAGS='-fPIC -fopenmp' \
FFLAGS='-fPIC -fopenmp' \
FCFLAGS='-fPIC -fopenmp' \
F90FLAGS='-fPIC -fopenmp' \
F77FLAGS='-fPIC -fopenmp' \
PETSC_DIR=`pwd`
  • You may be asked for your password during configure. This is because the above configure command is downloading and installing additional components needed (blas, lapack, etc).
  • Once configure has completed, follow the on-screen 'make' instructions, which will be specific to your machine.
  • Once make has completed, follow the on-screen 'make install' instructions, which will be specific to your machine. Be ready needing to use 'sudo' for the final step.

5. Finalizing

Close any running terminals (really just the one you were performing all of the above instructions from) and re-open them. This allow everything we just did, do be available for use.


That's it! We recommend that you browse your $PACKAGE_DIR directory and make sure that you see each of the library directories in there (GCC (if it was necessary to build), mpich, and PETSc). If not then you've probably done something wrong. Take a look back at the directions above and see what steps you may have missed. Otherwise you are set to continue on with Step #2 in the Getting Started Instructions