View Source

{info}
This page contains some core skills a user must develop after gaining access to our HPC system.
{info}

h5. To connect to an HPC System

To access an HPC system via SSH-2 use:
PuTTY, OpenSSH, or SecureCRT; Example:
{code:bgColor=#c0c0c0}
ssh thekla@euclid.cyi.ac.cy
{code}

or

{code:bgColor=#c0c0c0}
ssh thekla@login.cytera.cyi.ac.cy
{code}

\\
To transfer files to and from the gird using SFTP use:
WinSCP, OpenSSH, or SecureFX; Example:
{code:bgColor=#c0c0c0}
scp /etc/resolv.conf euclid.cyi.ac.cy:tmp
{code}
\\

h5. Architecture
64bit code and 32bit code are both supported; what is optimal varies on a case by case basis. M2070 GPUs may be available. CPU architecture, CPU speed, memory, and networking may also be important factors. Use the list of queues (_qstat -q_) while making decisions about where to run jobs, especially if you want to make use of GPU resourses.

h5. $HOME & $WORK
When a job script is submitted PBS creates a shell to run the job on the assigned nodes. User $HOME directories are located on a network fiel system and are accessible from all of the nodes. Queued job should run in a directory on the assigned node which is part of the parallel file system, identified by the environment variable $WORK.

h5. To find cluster status and job summary
{code:bgColor=#c0c0c0}
qstat -q # Queue summary
pbsnodes # Detailed information per node
qstat # job list; can be very long
{code}

h5. Modules commands examples
{code:bgColor=#c0c0c0}
module avail # lists available modules
module list # lists currently loaded modules
module help # help on a specific module
module whatis # displays the changes that loading a specific module makes to the environment without actually loading it.
module load foo # loads a specific module
module unload # unloads a specific module
module clear # unloads all modules
{code}

h5. To start an interactive job
*On Euclid*
{code:bgColor=#c0c0c0}
qsub -I -q gpuq -l nodes=1:ppn=8,walltime=00:15:00
{code}

*On Cy-Tera*
{code:bgColor=#c0c0c0}
qsub -I -q cpui -l nodes=1:ppn=12,walltime=00:15:00
{code}

-I starts an interactive job
-q specifies a queue (cpuq or gpuq for Euclid and cpui or gpui for Cy-Tera)
-l number of nodes and cores per node needed by a job

This will open a shell on the first node assigned. This is useful for jobs that have graphical interfaces or setting up job scripts.

h5. Example MPI code: hello.c
{code}
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank;
int length;
char name[80];

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(name,&length);
printf(
"Hello MPI World! Proc %d out of %d on %s\n", rank, size, name);
MPI_Finalize();
return 0;
}
{code}

h5. To compile a parallel job
{code:bgColor=#c0c0c0}
module load openmpi
mpicc -o hello hello.c
{code}

h5. To create a job script
A job script is a list of instructions that tell PBS how to run your job. You can modify a job script in a text editor such as Vim, nano, joe, pico, emacs etc. You should do such work on the head node.

*On Euclid*
{code}
#!/bin/bash -l
# Fix: job name; merge stdout & stderr; environment; ask cores
#PBS -N mpi-hello_world
#PBS -j oe
#PBS -q batch
#PBS -l nodes=2:ppn=8

cd $PBS_O_WORKDIR

module load openmpi # without this, mpiexec may not be in path
mpiexec ./hello # execution depends on Software Environment

env|grep ^BC_ # print Baseline Configuration related variables
env|grep ^HOME
env|grep ^WORK
sleep 15 # Wait a bit otherwise the job runs too fast
{code}

*On Cy-Tera*
{code}
#!/bin/bash -l
# Fix: job name; merge stdout & stderr; environment; ask cores
#PBS -N mpi-hello_world
#PBS -j oe
#PBS -q cpuq
#PBS -l nodes=2:ppn=12,walltime=00:15:00

cd $PBS_O_WORKDIR

module load openmpi # without this, mpiexec may not be in path
mpiexec ./hello # execution depends on Software Environment

env|grep ^BC_ # print Baseline Configuration related variables
env|grep ^HOME
env|grep ^WORK
sleep 15 # Wait a bit otherwise the job runs too fast
{code}

h5. To submit a job
*On Euclid*
{code:bgColor=#c0c0c0}
qsub <name of script> # To submit a job in PBS
{code}

*On Cy-Tera*
{code:bgColor=#c0c0c0}
msub <name of script> # To submit a job in PBS
{code}

h5. Makefile Example
{code}
CC=g++
CFLAGS=-c -Wall
LDFLAGS=
SOURCES=main.cpp hello.cpp factorial.cpp
OBJECTS=$(SOURCES:.cpp=.o)
EXECUTABLE=hello

all: $(SOURCES) $(EXECUTABLE)

$(EXECUTABLE): $(OBJECTS)
$(CC) $(LDFLAGS) $(OBJECTS) -O $@

.cpp.o:
$(CC) $(FLAGS) $< -O $@
{code}

{info}
The examples mentioned hereby can be found in euclid:
/opt/examples/coreskills
{info}