Compression

Description

File compression tools.

Modulefile

compression

Documentation

bzip2

Gzip

GNU Tar

XZ Utils

Zip

UnZip

zlib

PBZIP2

pigz

PXZ

Usage

$ module load compression

Parallel compression

If you are going to run these commands on the head node, then please respect our policies.

If you are going to be submitting jobs with these commands, then please use the 'openmp' parallel environment to request the resources, and the $NSLOTS environment variable to pass a number of threads.

PBZIP2

PBZIP2 has an option to set a number of threads, but defaults to using all system cores otherwise. Please note the lack of a space between the '-p' and the number of threads on the command line.

#$ -cwd
#$ -l h_rt=1:0:0
#$ -pe openmp 4

module load compression
pbzip2 -p$NSLOTS file

pigz

Pigz has an option to set a number of threads, but defaults to using all system cores otherwise.

#$ -cwd
#$ -l h_rt=1:0:0
#$ -pe openmp 4

module load compression
pigz -p $NSLOTS file

Pxz

Pxz has an option to set a number of threads, but defaults to using all system cores, contrary to the help page, otherwise. Moreover, the environment variable OMP_NUM_THREADS interferes with the command line option and always overrides it. The former is set to 1 by default on ACENET to avoid unintended OpenMP threading, it should be unset.

#$ -cwd
#$ -l h_rt=1:0:0
#$ -pe openmp 4

module load compression
unset OMP_NUM_THREADS
pxz -T $NSLOTS file