DRMAA using Perl and Torque/PBS

Posted on 2011-11-18 13:47:30
by Geert Vandeweyer

Loading Content

DRMAA using Perl and Torque/PBS

Posted on 2011-11-18 13:47:30
by Geert Vandeweyer

I recently got the perl wrapper for DRMAA from Tim Harsch working with a torque/pbs cluster. Most of the functionality had worked for a longer time, but I couldn't get the DRMAA_NATIVE_SPECIFICATION working. I now have learned that the DRMAA implementation in the free torque/pbs release is too old to support this. So to get it working fully, you need to first install torque, then the pbs-drmaa modules from sourceforge. 

1. Installing torque/pbs

Find the latest release on the Cluster Resources website. I used v2.5.5, since that was listed as stable and recommended. 

geert@pbsmaster: wget 'http://www.clusterresources.com/downloads/torque/torque-2.5.5.tar.gz'
geert@pbsmaster: tar xzvf torque-2.5.5.tar.gz
geert@pbsmaster: cd torque-*
# following installs into /usr/local
geert@pbsmaster: ./configure && make && sudo make install 
# I had to update /etc/ld.so.conf
geert@pbsmaster: sudo echo '/usr/local/lib' >> /etc/ld.so.conf
geert@pbsmaster: sudo ldconfig
# add the startupscripts (ubuntu style)
geert@pbsmaster: sudo cp contrib/init.d/debian.pbs_mom /etc/init.d/pbs_mom
geert@pbsmaster: sudo update-rc.d pbs_mom defaults
geert@pbsmaster: sudo cp contrib/init.d/debian.pbs_server /etc/init.d/pbs_server
geert@pbsmaster: sudo update-rc.d pbs_server defaults

Continue by configuring the server as per documentation

2. Installing pbs-drmaa

Find the latest release on sourceforge. I tested with version 1.0.10. 

NOTE: I installed pbs-drmaa over torque server that had native drmaa support, and had no noticable complications. 

geert@pbsmaster: wget http://downloads.sourceforge.net/project/pbspro-drmaa/pbs-drmaa/1.0/pbs-drmaa-1.0.10.tar.gz
geert@pbsmaster: tar xzvf pbs-drmaa-1.0.10.tar.gz
geert@pbsmaster: cd pbs-drmaa-*
# following install into /usr/local, change if torque is locate elsewhere ( with ./configure --prefix=PREFIX)
geert@pbsmaster: ./configure && make
geert@pbsmaster: sudo make install 

3. Install Perl DRMAA interface

Download the Schedule::DRMAAc interface from Tim Harsch sourceforge page. Do not install form CPAN as that version only works on a sun grid engine. 

geert@pbsmaster: wget http://search.cpan.org/CPAN/authors/id/T/TH/THARSCH/Schedule-DRMAAc-0.81.tar.gz
geert@pbsmaster: tar xzvf Schedule-*
geert@pbsmaster: cd Schedule-*
## install swig from the distro repositories (ubuntu style)
geert@pbsmaster: sudo apt-get install swig
## prepare links & environment variable for scripts to work
geert@pbsmaster: ln -s /usr/local/include/drmaa.h drmaa.h
geert@pbsmaster: export SGE_ROOT='/usr/local'
## create the interface files
geert@pbsmaster: swig -perl -Wall -exportall Schedule_DRMAAc.i
## make and install
geert@pbsmaster: perl Makefile.PL
geert@pbsmaster: make && sudo make install

4. Test

If your pbs_server and pbs_mom is up and running and working (test with some qsub scripts first !), then you can use the perl interface with something like the following code: 

#!/usr/bin/perl

use Schedule::DRMAAc qw/ :all /;

# disable output buffer
$|++;

# initialise the drmaa interface
( $error, $diagnosis ) = drmaa_init( undef );
  die drmaa_strerror( $error ) . "\n" . $diagnosis if $error;

## define job
( $error, $jt, $diagnosis ) = drmaa_allocate_job_template();
  die drmaa_strerror( $error ) . "\n" . $diagnosis if $error;

## set job name
( $error, $diagnosis ) = drmaa_set_attribute( $jt, $DRMAA_JOB_NAME, "$username.plink.createtfam" );
  die drmaa_strerror( $error ) . "\n" . $diagnosis if $error;

## set output path
( $error, $diagnosis) = drmaa_set_attribute($jt,$DRMAA_OUTPUT_PATH , "/tmp/stdout.txt");
( $error, $diagnosis) = drmaa_set_attribute($jt,$DRMAA_ERROR_PATH , "/tmp/stderr.txt");

## set some native specifications to request resources in this case
# request 1 node with two processes and 3.4 Gb of memory. 
( $error, $diagnosis) = drmaa_set_attribute($jt, $DRMAA_NATIVE_SPECIFICATION, "-l nodes=1:ppn=2,mem=3400mb");
   die drmaa_strerror( $error ) . "\n" . $diagnosis if $error;

## set command
# command first waits 25 seconds (time to check qstat etc), then prints out a hello world
( $error, $diagnosis ) = drmaa_set_attribute( $jt, $DRMAA_REMOTE_COMMAND, "sleep 25; echo 'hello world'");

## submit the job
( $error, $jobid, $diagnosis ) = drmaa_run_job( $jt );
  die drmaa_strerror( $error ) . "\n" . $diagnosis if $error;

## wait to finish (now is time to check qstat for requested resources etc.
my @job_constant = ( $jobid );
( $error, $diagnosis ) = drmaa_synchronize( \@job_constant , $DRMAA_TIMEOUT_WAIT_FOREVER, 0 ); 
  die drmaa_strerror( $error ) . "\n" . $diagnosis if $error;

# after job is finished, shut down drmaa and exit
drmaa_exit();
exit;

Cluster, CPAN, drmaa, HPC, Installation, Perl, torque/pbs