Grid Batch Jobs

A batch job is a scheduled program that runs on the Grid, in the background, without further user interaction. A batch job is submitted to the Grid using a SLURM submission script that contains directives to pass along to the Grid nodes. The directives instruct the Grid nodes how to distribute resources for the program, e.g., memory, the number of CPUs or GPUs, and information about where to put program output.

The first step is to login to the SCRC environment.

Example Python Script – hello-world.py

Using a text editor such as vim, nano, or emacs, let’s create a python script ‘hello-world.py’.

import os
print("Hello World from the Stern Grid")

The SLURM Submission Script

#!/bin/bash                                                                                                            
#                                                                                                                      
# [hello-world.sbatch]                                                                                                         
#                                                                                                                      
# This SLURM submission script runs the python script 
# hello-world.py that prints the program's output 
# as well as error messages, if any.
#
# --------------------------------------------------------
# Submit this sbatch script to SLURM by typing at the linux 
# command line
#     sbatch hello-world.sbatch
#
# Use environment modules to specify software and version.  
# Before running this script, at the command line type
#     module avail
# to see what software and program languages are available
#
# Purge currently loaded modules then load the module you want.
# Change to your python program's directory
#

     module purge
     module load python/3.9.7
     cd myprogs

#
#
# slurm job scheduler directives begin with "#SBATCH"
#
#SBATCH --job-name=hello-world-job          # Job name
#SBATCH --time=00-00:10                     # Wall-clock time limit
#SBATCH --mem=4G                            # Request 4G RAM
#SBATCH --mail-type=END,FAIL                # email when job ENDs or FAILs
#SBATCH --mail-user=you@stern.nyu.edu       # email TO
#SBATCH --output=hello-world-job%j.out      # write output; "%j" = job ID
#
#
# **** Note: It is always advantages to specify conservative values
#       for run time and memory. Jobs requesting fewer resources are more
#       likely to be scheduled before jobs requesting more resources.
#
# This line tells slurm what program/script to run
#

python hello-world.py

Submit Job to the Grid

Use the sbatch command to run hello-world.py on the Grid. The sbatch command queues the job as it awaits resources to become available on the Grid.

$ sbatch hello-python.sbatch

Check Status of Job

Use the squeue command to see the status of your jobs.

$ squeue -u $USER