General Queue Job to SLURM Cluster script

Posted on 2011-06-16 13:29:54
by Geert Vandeweyer

Loading Content

General Queue Job to SLURM Cluster script

Posted on 2011-06-16 13:29:54
by Geert Vandeweyer

This script allows you to queue jobs on a SLURM cluster without writing custom batch-scripts.  The script takes the provided command and functions as a generic batch-script.  It then launches the command using sbatch. 

This means that instead of using srun, which is interactive, you can now use QueueJob to run a single command in a non-interactive way. 

Code: 

Create this script as 'QueueJob' and place it in a shared $PATH location on the cluster. 

#!/bin/bash

## QueueJob v1.0 
## Script to submit jobs to slurm-batch without 
##   the need to write your own scripts. 

## Two Step process: 
##   1. Take command and arguments, submit to this script preceeded with sbatch
##   2. use this script as the actual batch script running provided command

## Place this script in $PATH ON ALL NODES 



##################################################
## PRINT SHORT HELP IF NO ARGUMENTS ARE PRESENT ##
##################################################
if [[ $# == 0 ]] ; then
	echo "" 
	echo "  Provide a program and parameters to run on the cluster."
	echo "    eg :	QueueJob -o output.file.txt -- uptime"
	echo "       => this will get the uptime of a single (random) node and write it to output.file.txt"
	echo "  Try 'QueueJob -h' for the all options"
	echo ""
	echo ""
	exit;
fi		

if [[ "$1" == '-h' ]] ; then
	echo ""
	echo " ##############################################"
	echo " ## HOW-TO : Queue Jobs on the SLURM Cluster ##"
	echo " ##############################################"
	echo ""
	echo "	NAME: "
	echo "		QueueJob - Send Programs to the cluster queue in a non-interactive way"
	echo ""
	echo "	Synopsis:"
	echo "		QueueJob [slurm options] Executable [additional options]"
	echo ""
	echo "	Main available Slurm options:"
	echo "		-o file   : redirect output to the specified file. Default is slurm.<int>.out "
	echo "		-D path   : set working directory before execution. Default: current path."
	echo "		-c int    : cpu's per task: If the queued program uses multiple cpus,"
	echo "			    for optimal Node allocation."
	echo "		-n int	  : Number of tasks to launch. This option will run 'n' times the same program !"
	echo "		-J string : Job Name, will be used in job management (squeue etc)" 
	echo "		--mem-per-cpu=<MB> : Minimum Memory required for the job."
	echo "		--tmp=<MB>	   : Minimum amount of temporary disk space"
	echo ""
	echo "		See 'man sbatch' for a full overview of options"
	echo ""
	echo "          Seperate SLURM options from main command by double hyphen '--'"
	echo ""
	echo " Example:"
	echo "		QueueJob -o output.file.txt -- df -h ";
	echo "			=> Get information on the free disk space of the node this job is sent to,";
	echo "			   write output to output.file.txt in the current directory.";
	echo ""
	echo ""
	exit;
fi
########################################
## FIRST CASE : RESUBMIT USING SBATCH ##
########################################
pathtoscript=`which QueueJob`
if [ "$0" == "$pathtoscript" ] ; then
	args=("$@")
	command=""
	slurmopts=""
	index=0
	slurmfound=0
	## slurm options
	for (( f=0; f<$#; f++ )); do
		if [ ${args[$f]} == '--' ] ; then
			slurmfound=1;
			index=$f+1;
			break;
		fi;
		## escape single quotes
		case ${args[$f]} in 
			*\\\'* )
				## already escaped. skip
				;;
			*\'* )
				args[$f]=${args[$f]//\'/\\\'}
				;;
		esac
		## surround arguments with quotes if containing spaces
		case ${args[$f]} in 
     			*\ * )
				slurmopts="$slurmopts \"${args[$f]}\""
   			        ;;
       			*)
				slurmopts="$slurmopts ${args[$f]}"
           			;;
		esac

	done;
	if [ $slurmfound == 0 ] ; then
		slurmopts=""
	else 
		slurmopts=${slurmopts:1}	
	fi
	## program to run and parameters	
	for (( f=$index; f<$#; f++ )) do
		## escape single quotes
		case ${args[$f]} in 
			*\\\'* )
				## already escaped. skip
				;;
			*\'* )
				args[$f]=${args[$f]//\'/\\\'}
				;;
		esac
		## surround arguments with quotes if containing spaces
		case ${args[$f]} in 
     			*\ * )
				command="$command \"${args[$f]}\""
   			        ;;
       			*)
				command="$command ${args[$f]}"
           			;;
		esac
	done;
	command=${command:1}					# remove leading space from joining
	fullcommand="sbatch $slurmopts QueueJob $command" 	# add preceeding (slurm) options
	echo "Submitting Job to Cluster Queue."
	echo "  => $command"
	dump=`$fullcommand`

#############################################
## SECOND CASE : ACTUAL RUNNING OF PROGRAM ##
############################################# 
else
	args=("$@")
	command=""
	for (( f=0; f<$#; f++ )); do
		case ${args[$f]} in 
			*\\\'* )
				## already escaped. skip
				;;
			*\'* )
				args[$f]=${args[$f]//\'/\\\'}
				;;
		esac
		case ${args[$f]} in 
     			*\ * )
				command="$command \"${args[$f]}\""
   			        ;;
       			*)
				command="$command ${args[$f]}"
           			;;
		esac
	done;
	command=${command:1}		# remove leading space	
	echo "Running The Job."
	echo "   => $command"
	$command
fi 

Usage:

The following example will get the uptime from a single random node on the cluster, which meets the requirements of having 3Gb RAM, and will output the uptime to the file '~/slurm_out/example.out'

geert@ubuntu: QueueJob -o ~/slurm_out/example.out --mem-per-cpu=3000 -- uptime

I haven't done a lot of testing using this script, but it should work. Shortly after writing it I switched to Torque/PBS. 

BASH, Cluster, HPC, SLURM