Skip to content

Lab1 Construct simple cluster

Keep in mind

Always redirect the output of your commands to a file. It helps to store informations in files. You may need them later to get information or debug.

# Redirect stdout and stderr to files
command > stdout.log 2> stderr.log

If you want to see the information in your terminal, you can use tee to print the output to both stdout and files.

# Redirect stdout and stderr to files and print them to terminal
command 2>&1 | tee stdout.log stderr.log

0. Create a virtual machine

Procedures of installation are skipped.

If you can't bear low display resolution in terminals, see Configuring Network and SSH to use SSH to connect to the virtual machine.

1. Build and install OpenMPI from source

Follow the official tutorial OpenMPI: FAQ:Building Open MPI to install OpenMPI.

tar -xvf openmpi-4.1.1.tar.gz
cd openmpi-4.1.1
./configure --prefix=/usr/local
make all install

You may need sudo privilege to execute make all install because it needs to change /usr/local/. After installation, the binary files are in /usr/local/bin and you don't need to change PATH.

Remember directory paths

You'll need to configure paths to OpenMPI when compiling and running MPI programs. In my case, the paths are:

  • binary: /usr/local/bin/
  • header files: /usr/local/include
  • hostfile: /usr/local/etc/openmpi-default-hostfile

After I've done the lab, I understood that I should have used an independent path to save the OpenMPI files and other components, so that they can be found more easily. For example, I can use /usr/local/openmpi-4.1.1 to save the files.

Test your OpenMPI installation

You can test your OpenMPI installation by executing mpirun:

mpirun --version

If you see the version information, then you've installed OpenMPI successfully.

If you want to test the compilation and execution of MPI programs, you can try to compile and run the hello.c. Follow the instructions on Using MPI with C

2. Build and install BLAS and CBLAS

Prepare tools:

sudo apt install build-essential gfortran cmake

Get the source code and extract them:

wget ""
wget ""
wget ""
tar xvf hpl-2.3.tar.gz -C hpl
tar xvf blas-3.11.0.tgz -C blas
tar xvf cblas.tgz -C cblas

Install BLAS

cd blas

Then you'll Find the line BLASLIB = ../../blas$(PLAT).a and change it to BLASLIB = /usr/local/lib/libblas.a. Then save and exit. Then execute make, get the binary file cblas_LINUX.a in the folder. Copy it to /usr/local/lib:

cp blas_LINUX.a /usr/local/lib/libblas.a

Run the test program for BLAS and passed:


Install CBLAS

cd ../cblas

Read the instructions in README to install CBLAS.

cp Makefile.LINUX

Find the line BLASLIB = ../../blas$(PLAT).a and change it to BLASLIB = /usr/local/lib/libblas.a. Then save and exit. Then execute make, get the binary file cblas_LINUX.a in the folder. Copy it to /usr/local/lib:

cp cblas_LINUX.a /usr/local/lib/libcblas.a
Error when compiling CBLAS

If you encountered the same error when executing make in CBLAS folder and stopped at testing process, receiving the error message like:

Error: rank mismatch in argument 'strue1' at (1) (rank-1 and scalar)


The reason is that there are some old-fashioned code deprecated by recent version of gcc-10. You can try to modify the Makefile in test folder, adding -std=legacy or -fallow-argument-mismatch to the flags.

Here are more information:

After edit, run make again and you'll see the test passed.


3. Build HPL

There are some difference between the experiment manual and INSTALL file, you should follow the manual.

cp setup/Make.Linux_PII_CBLAS Make.test

Edit some lines in Make.test:

# diff from Make.test and Make.Linux_PII_CBLAS
< ARCH         = test
> ARCH         = Linux_PII_CBLAS
< MPdir        =
< MPinc        = -I/usr/local/include
< MPlib        = /usr/local/lib/
> MPdir        = /usr/local/mpi
> MPinc        = -I$(MPdir)/include
> MPlib        = $(MPdir)/lib/libmpich.a
< LAdir        =
< LAinc        = #/home/bowling/CBLAS/include
< LAlib        = /usr/local/lib/libcblas.a /usr/local/lib/libblas.a /usr/lib/gcc/x86_64-linux-gnu/10/libgfortran.a /usr/lib/gcc/x86_64-linux-gnu/10/libquadmath.a /usr/lib/x86_64-linux-gnu/libm.a
> LAdir        = $(HOME)/netlib/ARCHIVES/Linux_PII
> LAinc        =
> LAlib        = $(LAdir)/libcblas.a $(LAdir)/libatlas.a
< CC           = /usr/local/bin/mpicc
> CC           = /usr/bin/gcc
< LINKER       = $(CC)
> LINKER       = /usr/bin/g77
Error in linking

You may encounter the error message like this:


Search the web and you'll find the libs containing missing symbols. They are:

  • /usr/lib/gcc/x86_64-linux-gun/10/libgfortran.a
  • /usr/lib/gcc/x86_64-linux-gun/10/libquadmath.a
  • /usr/lib/gcc/x86_64-linux-gun/10/libm.a

Searching for the missing library was hard, because it happens everywhere. Solutions to others may not take effect on you.

I'm still wondering why these libraries were not linked automatically, they are in the default PATH.

After make, you'll get the binary file xhpl in the folder bin/test.

4. Configuring network and SSH

To construct a cluster, we need to use NAT network to connect them together. The host cannot connect to virtual machine inside NAT network directly, but we can use port forwarding:

Port Forwarding

Now we can ssh into the master machine:

ssh -p 3022 bowling@localhost

5. Experiment

Error: cannot open file HPL.dat

You may encounter error like this:


Refer to this issue:

Final result:


Bonus: Docker

Using docker, we can skip the process of building everything. We only need to configure SSH connection and write HPL.dat, then enjoy. DOCKERS ARE AWESOME.

Find docker

I use the docker from ashael/hpl - Docker Image | Docker Hub, which contains configured hpl-2.2.

Here I've opened 4 containers from the same image:

docker image

Setting up connection

Dockers are connected within a bridge by default.

docker network

Attach to each of them and get their ip address:

docker ip

Select the host of as master, as slaves.

In master:

cat ~/.ssh/

In slaves:

mkdir /var/run/sshd
vim /etc/ssh/sshd_config

find these lines:

#ListenAddress ::
#AuthorizedKeyFile %h/.ssh/authorized_keys

uncomment them and save.

Then add master's public key to authorized_keys and start sshd.

docker ssh


Execute experiment

docker result

I found the computing speed in dockers is much faster then clusters. DOCKERS ARE EXTREMELY AWESOME.


Later in the course, instructors mentioned we can try to modify HPL.dat to get better performance and configure NFS to share files between nodes. I also tried to do these things.

Tuning HPL.dat

Refer to HPL Tuning.

Configuring NFS File System

NFS uses RPC protocol to share files between server and client.

Setup NFS Server and Client

Please refer to this passage: How to Set Up an NFS Mount on Debian 11 | DigitalOcean.

More details about NFS can be seen in:

Have a test

First install nfs-common and create mount point on clients:

sudo apt install nfs-common
sudo mkdir /hpc

mount error

If you encounter the following error:

mount: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

You need to install nfs-common on clients.

Add the following line to clients:

/etc/fstab /hpc nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

Then run the program and get the result: