Compress all fastq/vcf files in a storage tree with a single command

Posted on 2013-03-06 07:22:59
by Geert Vandeweyer

Loading Content

Compress all fastq/vcf files in a storage tree with a single command

Posted on 2013-03-06 07:22:59
by Geert Vandeweyer

this was my command of the day: 

find -type f | grep -P '(fastq|fastqsanger|vcf)$' | sed -e "s,^.,\'&," -e "s,\$,\'," | xargs pigz -9 -p 8

It saved me over a TB of disk space after running it on a ftp server used for long term storing of next-generation sequencing data, where users didn't have to compress before upload. 

This is the flow: 

1. find all files in the current dir + subdirs
2. Grep for files ending in fastq, fastqsanger or vcf using perl regexp
3. add quotes to the filenames using sed, to prevent issues with spaces, brackets etc
4. pass all found and escaped files to pigz using xargs, the parallel implementation of gzip, using 8 cores at best compression level

BASH, bioinformatics, compress, cotd