Mapping Flywheel Job Priority to Slurm Scheduling

It can be convenient to impose a priority for scheduling a Slurm job. These directions demonstrate how to map the Flywheel Job Priorities (e.g. low, medium, high, critical) to scheduling a job on a Slurm Scheduler.

Configuration

Slurm Configuration

Depending on your Slurm Configuration, you may have basic or multifactor priority scheduling. The basic priority scheduling is a First-In-First-Out scheduler where both priorities and "nice" factors can be set explicitly. The multifactor priority scheduling dynamically sets the job priority on factors such as Job Size and Job Age. The user will only be able to set the "nice" factor for these jobs directly.

Your systems administrator will configure Slurm to have either of the above priority scheduling regimes. They will inform you of the best way to set the priority of the jobs at submission time.

Updating Cast Settings

An optional priority setting is illustrated in the ./examples/settings/priority_cast.yml file.

Settings for basic Slurm Scheduling

The following gives an example of increasing priority settings.

  map_job_priority: true
  fw_priority_map:
    low: 1
    medium: 2
    high: 3
    critical: 99

Adding the following directive to the custom script in settings/cast.yml gives the ability to submit jobs with the given priority above (see the example).

{% if job.priority %}#SBATCH --priority={{job.priority}}{% endif %}

Some Slurm configuration settings do not allow job priority to be set at submission time. Adjustments to Slurm job priority can be made on a Slurm login node in the following manner:

scontrol update job=<job-id> Priority=<any-integer>

This can depend on privileges for the user changing the permissions.

Consult with your systems administrator on how the Slurm Scheduler is configured and the appropriate values to set the priority mappings to.

Settings for multifactor Slurm Scheduling

With multifactor Slurm Scheduling the user will have control over the "nice" factor of their job submission. The Flywheel-to-Slurm priority mapping becomes a descending mapping of nice values:

  map_job_priority: true
  fw_priority_map:
    low: 100
    medium: 75
    high: 25
    critical: 1  # Must be non-zero

The SBATCH directive then becomes:

{% if job.priority %}#SBATCH --nice={{job.priority}}{% endif %}

Again, consult with your systems administrator on the best way to set the nice values.