Job Scheduling with LSF

Triton currently uses the LSF resource manager to schedule all compute resources. LSF (load sharing facility) supports over 1500 users and over 200,000 simultaneous job submissions. Jobs are submitted to queues, the software categories we define in the scheduler to organize work more efficiently. You can monitor your job status, queue position, and progress using LSF Commands.

Tip

Reserve an appropriate amount of resources through LSF for your jobs.

If you do not know the resources your jobs need, use the debug queue to benchmark your jobs. More on Pegasus Queues and LSF Job Scripts

Warning

Jobs with insufficient resource allocations interfere with cluster performance and the IDSC account responsible for those jobs may be suspended (Policies).

Tip

Stage data for running jobs exclusively in the /scratch file system, which is optimized for fast data access.

Any files used as input for your jobs must first be transferred to /scratch. See Pegasus Resource Allocations for more information. The /nethome file system is optimized for mass data storage and is therefore slower-access.

Warning

Using /nethome while running jobs degrades the performance of the entire system and the IDSC account responsible may be suspended*** (Policies).

Tip

Do not background processes with the & operator in LSF.

These spawned processes cannot be killed with bkill after the parent is gone.

Warning

Using the & operator while running jobs degrades the performance of the entire system and the IDSC account responsible may be suspended (Policies).

LSF Batch Jobs

Batch jobs are self-contained programs that require no intervention to run. Batch jobs are defined by resource requirements such as how many cores, how much memory, and how much time they need to complete. These requirements can be submitted via command line flags or a script file. Detailed information about LSF commands and example script files can be found later in this guide.

Create a job scriptfile

Include a job name -J, the information LSF needs to allocate resources to your job, and names for your output and error files.
```
scriptfile
#BSUB -J test
#BSUB -q normal
#BSUB -P myproject
#BSUB -o %J.out
...
```

Submit your job to the appropriate project and queue with bsub < scriptfile

Upon submission, a jobID and the queue name are returned.

[username@mgt3.summit hpc]$ pwd
/scratch/projects/hpc/
[username@mgt3.summit hpc]$ bsub < scriptfile
Job <391> is submitted to queue <normal>.

Monitor your jobs with bjobs

Flags can be used to specify a single job or another user’s jobs.

[username@mgt3.summit hpc]$ bjobs
JOBID  USER   STAT  QUEUE    FROM_HOST  EXEC_HOST   JOB_NAME  SUBMIT_TIME
391   usernam   RUN   normal  mgt3       16*t030     testjob   Apr  9 10:22

Examine job output files

Once your job has completed, view output information.

[username@mgt3.summit hpc]$ cat 391.out
Sender: LSF System <lsfadmin@t030>
Subject: Job 391: <test> in cluster <t1> Done
Job <test> was submitted from host <mgt3> by user <username> in cluster <t1> at Wed Apr  9 10:22:26 2025.
Job was executed on host(s) <16*t030>, in queue <normal>, as user <username> in cluster <t1>.
...