CASIC HPCC User’s Guide
Version 2.2 – May 2nd, 2016
First steps with LSF
This section describes submitting simple jobs to LSF from the command line on hpc-login. LSF is the “workload scheduler” for this cluster. It receives job submissions from users, and decides where and when those jobs should run, given a) available resources and b) the jobs from other users.
We begin by assuming that you’ve already logged into this node:
This command tells you which version of LSF has been installed, and the name of the cluster, among other things
This command tells you which hosts are part of the cluster, and some details about them including their model name, amount of memory and swap, etc
This command shows you the hosts in the cluster in a different way. Here the hosts are shown with their current status, the number of jobs they’re running, and so forth
This command shows you the queues defined in LSF. Queues are a holding place for jobs. Most importantly, the queue to which you submit your job defines how important that job is, in comparison to other jobs.
An example set of queues have been defined for the cluster. A similar queue structure is used at many research institutions. This default queue structure prevents monopolization of the cluster by any one user for a long period of time. We would expect that Auburn’s experience with LSF will lead to changes to this queue structure in fairly short order.
The defined queues are:
debug: A one hour queue for testing, accessible by all users. This is the default queue: if you do not specify a queue to LSF on job submission, your job will be submitted to this queue.
short: Accessible by all users, for jobs running less than 2 days, with a maximum of 128 cores.
medium: Accessible by all users, for jobs running less than 5 days, with a maximum of 128 cores.
long: Accessible by all users, for jobs of any length, with a maximum of 208 cores.
bigmem: Resticted-access queue, for big memory/fast disk jobs. A single user can occupy up to 16 cores.
For full details on a particular queue, issue the command:
> bqueues -l
A default fairshare has also been defined for the cluster. What this means is that, over many days, every user will tend to get the same share of CPU time on the cluster. It does NOT mean that all users will get an equal allocation of the cluster at a specific moment in time.
We can now submit our first job to the cluster:
> bsub sleep 300
bsub is used to submit jobs to the cluster. You will get a jobid that is the reference for your job. From earlier in the discussion, this job was submitted to the default ‘short’ queue.
A bsub consists of two parts. The first part is the parameters to bsub (of which there are none here), and the command you’d like to run on the cluster. Everything after the final parameter to bsub is considered to constitute this command. Here, the command that LSF will run on the cluster is “sleep 300”.
bjobs shows you jobs that are ‘active’ in the cluster. That is, they are pending, running, exited or ended. Pending jobs are waiting in a queue, but have not been dispatched by LSF to run on the cluster yet. Running jobs are running on the cluster. Exited jobs ended with a non-zero exit code. Ended jobs ended with a zero exit code
> bjobs -l
Shows you a long listing of details for a specific job
> bsub -q long sleep 300
Submits the job to the long queue, where there are no runtime restrictions
> bsub -q long -n 1000 sleep 300
Submits a job to the long queue, and asks for 1000 slots. In LSF, slots are essentially the same as CPU cores; we’re telling LSF our job will consume 1000 cores. This job will be rejected due to the 500 slot limit on this queue
Finally, we can submit an RAxML job to the cluster. We assume that the files mentioned in these job submissions have been uploaded to your home directory, and that you have cd’ed into your home directory to begin:
> bsub -n 16 -R "span[hosts=1]" /tools/genome/raxmlHPC-PTHREADS-SSE3 -T 32 -s dataset_assembly.phy -n dataset_assembly.out -f a -p 12345 -x 12345 -N 100 -m PROTGAMMALGF -q partition_data.txt
There are many parts to this job submission:
bsub indicates that this is a job submission to LSF
- -n 16 indicates that this job will use 16 CPU cores on the cluster
- -R “span[hosts=1]”. This tells LSF that all 16 cores should come from a single host in the cluster.
- Everything afterwards is the RAxML command itself
- Remember that /tools/genome is universally accessible on the cluster. No matter where LSF decides to run your job, it will be able to access this path via GPFS.
- The files passed with -s and -q are assumed to be in your current working directory. Again, /home is universally accessible on the cluster
> bjobs -l
Depending on how things go, your job should ultimately run to completion. The output of the job will be placed in your home directory.
Finally, the most difficult job submission, a 500-way parallel job using MPI:
> export PATH=$PATH:/tools/openmpi-1.6.5/bin
> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/tools/openmpi-1.6.5/lib
> bsub -n 500 -R "span[ptile=16]" mpirun /tools/genome/raxmlHPC-MPI-SSE3 -s short-dataset_assembly.phy -n short-dataset_assembly.out -f a -p 12345 -x 12345 -N 100 -m PROTGAMMALGF
This job uses OpenMPI to run across multiple hosts:
- We’ve asked for 500 CPU cores. No single host can satisfy this.
- The -R “span[ptile=16]” string tells LSF to select 16 cores from every host on which the jobs runs. This prevents LSF from running 2 slots on one host, 8 slots on another, 13 on another, etc
- We prefix the RAxML command with mpirun; this is the standard way to start MPI jobs