Ctrl+K

Job scheduler on BioHPC

Job scheduler on BioHPC#

A SLURM cluster cbsueccosl01 is maintained by Lars on behalf of Econ, on several nodes. Some are dedicated to the SLURM scheduler, others “borrowed”; the latter might not always be available.

Note

Since September 2026, there is also a bioslurm cluster which is available to subscribers, see the bioslurm page. Wherever these pages mention cbsueccosl01, you can replace bioslurm with some limitations.

As of September 24, 2026 at 06:01 PM, there are 660 “slots” (cpus) available for compute jobs (out of a maximum possible 692) - see Table.

Who can use#

Everybody in the ECCO group can submit jobs.

Current load#

The most current status (as per the date and time noted) is:

As of 2026-09-24 14:00:

CLUSTER: cbsueccosl01
PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
slow*          up   infinite      1    mix cbsuecco01
slow*          up   infinite      2   idle cbsuecco[07-08]
fast           up   infinite      2  alloc cbsuecco[09-10]
fast           up   infinite      2   idle cbsuecco[13-14]
lgmem          up   infinite      1   idle cbsuecco02
interactive    up   infinite      2    mix cbsueccosl[03-04]
scavenge       up   infinite      2  alloc cbsuecco[11-12]

For more details, see the SLURM Queue page. For explanation of the “Partition”, see Queues section below.

Manually query the latest availability#

To see availability at any point in time, type

sinfo --cluster cbsueccosl01

in a terminal window on the head node,[1] to obtain a result such as

$ sinfo --cluster cbsueccosl01
CLUSTER: cbsueccosl01
PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
slow*          up   infinite      3    mix cbsuecco[01,07-08]
fast           up   infinite      3    mix cbsuecco[09-10,14]
fast           up   infinite      1  alloc cbsuecco13
lgmem          up   infinite      1    mix cbsuecco02
interactive    up   infinite      2   idle cbsueccosl[03-04]
scavenge       up   infinite      2   idle cbsuecco[11-12]

which shows that currently, the 2 nodes in the interactive partition (queue) and the 2 nodes in the scavenge partition are idle (no jobs running), seven have some jobs running on them, but can still accept smaller jobs (mix means there are free CPUs), and one is completely used (alloc).

Note

sinfo and squeue only list the partitions you are allowed to submit to. Contributed nodes also belong to an owner partition reserved for the group that paid for them, which you will not see unless you are a member of that group, or unless you add the -a option (sinfo -a --cluster cbsueccosl01). See the scavenge partition below.

Queues#

The List of nodes shows various partitions. These are the job queues on SLURM.

  • slow is the default

  • all jobs submitted using srun (rather than sbatch) will be treated as interactive and sent to the interactive partition which has a limit of one CPU per job.

  • lgmem partition requires at least 256GB of RAM to be requested, and will then route to the node with the largest memory. Note that this is a very slow (old) node, so don’t do this if you don’t need it.

  • scavenge lets any ECCO user run on nodes that were paid for by one research group, for as long as that group is not using them. Jobs in this partition can be interrupted at any moment, so they need to be restartable: see the scavenge partition below.

  • There are no time limits on any partitions, default RAM per job is 4 GB.

  • In order to submit to a specific partition,

    • use the -p option with sbatch, e.g. sbatch -p lgmem run.sh.

    • specify the partition in the SBATCH file with #SBATCH --partition lgmem.

    • If you don’t specify a partition, it will be sent to the default (slow).

The scavenge partition#

Some nodes were funded by research groups, who are the “owners”. Their free compute cycles are contributed to the overall cluster, but owners retain the right to pull their availability at any point. Such nodes, marked scavenge in the list below, belong to two partitions (queues) simultaneously:

  • an owner partition, named after the owner’s netid , which only members of the owning group may submit to.

  • scavenge, open to all ECCO users, in the same priority tier as slow, fast and lgmem.

Note

Owner nodes sit in a higher priority tier than every other partition.

What happens when the owner submits a job#

Jobs in the scavenge queue can be “preemted”. Preemption only happens when it is needed. If there are free nodes in the owner’s queue, the owner’s job is allocated to that. Only jobs on the node the owner actually needs are preempted. Because owner jobs always take a whole node, every scavenge job on that node is killed.

What happens to pre-empted jobs?#

A preempted job

  1. has CANCELLED ... DUE TO PREEMPTION written to its output, and is sent SIGCONT followed by SIGTERM;

  2. has 30 seconds to save what it can, and is then killed with SIGKILL;

  3. is put back in the queue if you submitted it with --requeue, keeping its job ID and its partition list. It is held for two minutes (squeue gives reason BeginTime), then starts again from the top of the script, possibly on a different node. Without --requeue it is simply cancelled; --no-requeue makes that explicit.

Note

This cluster sets JobRequeue=0, so a preempted job is cancelled unless you submitted it with --requeue.

scontrol show partition scavenge reports GraceTime=120, but that setting only takes effect under PreemptMode=CANCEL. This partition uses PreemptMode=REQUEUE, where the interval between SIGTERM and SIGKILL is the cluster-wide KillWait, currently 30 seconds.

For a concrete job template that is safe to requeue under scavenge, see the scavenge-safe example in the SBATCH examples.

Choosing between partitions#

You can name several partitions, and SLURM will start your job in whichever one can run it first:

#SBATCH --partition=fast,scavenge

Listing fast first means SLURM uses fast whenever it can start the job there just as soon, and falls back to scavenge when that would start sooner. A job that does start in scavenge stays preemptable for as long as it runs there. Members of the owning group use the same mechanism the other way round, listing their own partition first (owner should be replaced by the actual partition name of the owning group).

#SBATCH --partition=owner,fast

List of nodes#

The following table shows the allocated nodes. Nodes marked flex may not be available, because an owner has reserved them. Nodes marked slurm are always available. Nodes in the scavenge partition are contributed nodes that anybody may use at low priority, and where jobs may be interrupted by the owner: see the scavenge partition.

Note

HT means “hyper-threading”, and effectively multiplies the number of cores by 2, but may not always lead to performance improvement. MATLAB ignores hyper-threading, and will only use the physical number of cores listed in the cores column. The various queues can be requested, but most jobs should use the default queue.

Note

🤖 For AI coding of your SLURM submission, you can point the agent at labordynamicsinstitute/ecco-notes for machine-readable version of this table.

Nodename allocation partition cores RAM local storage in TB model cores per CPU CPUs HT cpu benchmark (single thread) vintage
Loading... (need help?)

Size of the cluster#

Total cores possible across all SLURM nodes: 692
Total RAM possible across all SLURM nodes: 3840 GB

Total cores currently available across all SLURM nodes: 660
Total RAM currently available across all SLURM nodes: 3584 GB
Contents