Running Cromwell on AWS/Batch

Posted on 2018-11-21 11:12:25
by Geert Vandeweyer

Loading Content

Running Cromwell on AWS/Batch

Posted on 2018-11-21 11:12:25
by Geert Vandeweyer

Cromwell is one of the widely used workflow managers used in bioinformatics to run NGS analysis pipelines. Based on the WDL workflow language, it supports runnings tasks as regular scripts or as containers, on local infrastucture, HPC systems and various cloud systems.

As the official flavour of cromwell has retired the support for AWS, I contributed to an "AWS Friendly" flavour that runs the jobs on AWS/Batch. Some of the features added to this version are :

• Support for call caching using private ECR containers
• Support for FSx/EFS filesystems
• Retry handling for spot-kills
• Setting of ulimits in batch jobs
• Optimized launch templates to fix the "fetch_and_run is a directory" erros

As the docs get buried by the official, but outdated and nonfunctional, pages, here are some pointers:

• Documentation on AWS features : here
• Documentation on how to deploy cromwell on AWS : here

If you have questions, reach out or join the cromwell channel on slack !

AWS, Batch, bioinformatics, cloudformation, cromwell, cromwell, docker, GATK, Installation, NGS, terraform