GridFTP

Currently non functional

In its current state at ACENET GridFTP no longer functions properly between all ACENET sites. We're working on a replacement that will be released in the near future.

Description

GridFTP is a file transfer solution provided by The Globus Alliance for high speed transfers. Using this protocol to transfer files between our clusters can speed up your transfers 5 to 14 fold, with transfer speeds from 50Mb/s to 90Mb/s, depending on the cluster and time of day.

Note, this can only be used between ACENET clusters, and cannot be used to transfer between your own computer and a cluster.

Location

/usr/local/globus

Version

4.2.1

How to set up

Please add the following lines into your dot profiles:

• for .bashrc

export GLOBUS_LOCATION=/usr/local/globus/
source $GLOBUS_LOCATION/etc/globus-user-env.sh

• for .cshrc

setenv GLOBUS_LOCATION /usr/local/globus/
source $GLOBUS_LOCATION/etc/globus-user-env.csh

How to use

gcp method - Use this

We have created a handy-dandy script to wrap around the globus-url-copy program that is used by GridFTP to transfer files between clusters. It's interface is similar to scp and will behave like it for the files. However, our options are more limited. Generally, a transfer can look like one of the three following examples:

gcp [options] /path/to/file/or/folder server.ace-net.ca:where/you/want/it
gcp [options] server.ace-net.ca:/path/to/file/or/folder where/you/want/it
gcp [options] server1.ace-net.ca:/path/to/file/or/folder server2.ace-net.ca:where/you/want/it

The first being a transfer from a local folder to a remote server, the second is transferring a file from a remote server to a local directory, and the third is copying a file between two remote servers.

Options can be:

-r

Recursively copy a directory

-h

Print the online help for gcp

-vb

Verbose, show the transfer speed and echo where the file is coming from and going to.

-V

Print the version of gcp

If there are any bugs or problems in this script, please contact support with a description of the problem and a copy of your terminal output.

globus-url-copy - Advanced Users

If you have no need to customize options please do not use this method, as it is difficult and cumbersome. Please use the gcp method above as it has taken out all of the guess work and testing you need to do to get this to work.

With that said, the globus-url-copy will allow you to customize your transfers to a much deeper level than our provided gcp script. The Globus Alliance has a very detailed guide (careful though, this guide is outdated) on the full use of their software, but most of it is irrelevant for use on ACENET's machines.

Generally, transfers between our machines will look like:

globus-url-copy [options] sshftp://server1.ace-net.ca/full/path/to/file(s) sshftp://server2.ace-net.ca/full/path/to/destination/folder

Where options can be one of:

• -vb
• -r
• -p #
• -tcp-bs #
• -dbg
• -list
• -rst

And the full path to the file starts with '/' so if you wanted to copy your '.bashrc' to another cluster you would have:

globus-url-copy sshftp://server1.ace-net.ca/home/username/.bashrc sshftp://server2.ace-net.ca/home/username/

The options are:

-vb

Verbose, shows the overall transfer speed in Mb/s

-r

Recursively copy a folder, needed to copy entire folders.

-p #

Parallelism, the number of simultaneous transfer threads that are initiated

-tcp-bs #

TCP Blocksize, changes the size of the TCP packets can (if the number is correct) speed up transfers as there (can be) less overhead.

-dbg

Debug, shows all of the FTP commands that are going on in the background, as well as the tags of the TCP blocks during the transfer.

-list

will list the contents of a folder or check if a file is on the server.

-rst

Restart, will attempt to restart broken FTP transfers

Known Issues

• placentia.ace-net.ca will not handle GridFTP (gcp) connections from other sites due to the configuration of the university firewall. If you plan to do file transfers with GridFTP, they should originate on althead. Files can be both sent and received, but only if the session is started on althead.