uberftp (1) - Linux Manuals

uberftp: GridFTP-enabled client

Command to display uberftp manual in Linux: $ man 1 uberftp

NAME

uberftp - GridFTP-enabled client

SYNOPSIS

uberftp [options] [host options] [host]

uberftp [options] [host options] host ``cmd''

uberftp [options] srcurl dsturl

uberftp [options] -f urlfile

uberftp [options] -cmd url

DESCRIPTION

uberftp is a GridFTP-enabled client that supports both interactive use and FTP commands on the uberftp command line to transfer files between two computers. It is intended for use with computers that have a GridFTP server installed. Uberftp supports GSI authentication, parallel data channels and striping. For more information about GridFTP, see the GridFTP URL in the "SEE ALSO" section below.

Only the first usage shown above will create an interactive session. If host is specified, uberftp immediately attempts to establish a connection to the GridFTP server on host, enters its command interpreter and awaits commands from the user. If host is not specified, uberftp immediately drops into the command interpreter without connecting to any GridFTP server.

The second usage option denotes a non interactive session in which ``cmd'' is a series of one or more commands as described in the COMMANDS section below to run once the control connection is established. These commands are run exactly as if they had been entered from the interactive prompt. This list must be enclosed in quotes. Multiple commands are semicolon or comma delimited. uberftp will execute these commands and then exit. Uberftp will exit upon the first error encountered.

The third and forth usage statements use the URL style format for specifying the source and destination for the files to transfer. The third usage statement places these URLs on the command line. The forth usage allows the user to specify multiple URL pairs in a separate file for Uberftp to transfer one at a time. The supported URL syntaxes are gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file and file://path.

The fifth usage statement allows for commands that take pathnames to accept URLs instead. The allowable commands are listed in the -cmds section below.

HOST OPTIONS

-P port

Connect to port instead of the default. The default for GSI authentication is 2811. The default for password authentication is 21.

-u user

Connect as user. This is useful for both password and GSI authentication mechanisms.

-p pass

Use pass when authenticating. If pass equals X, UberFTP will prompt for the password with character echoing turned off.

OPTIONS

-active: Use ACTIVE mode for data transfers.
-ascii: Use ASCII mode for data transfers.
-binary: Use BINARY mode for data transfers.
-blksize n: Set the internal buffer size to n.
-cksum [on|off]: Enable/Disable CRC checks after file transfers.
-cos name: Set the storage class of service to name. Used with HPSS installations. Use the class of service name default to allow the remote server to decide which class of service to use.
-d: Enable debugging. Same as '-debug 3'. Deprecated.
-debug n: Set the debug level to n.
-family name: Set the storage family to name. Use the family name default to allow the remote server to decide which family to use.
-glob [on|off]: Enable/Disable filename expansion.
-hash: Enable printing of hash marks during transfers.
-keepalive n: Send control channel keepalive messages every n seconds during data transfers.
-mode [E|S]: Switch the transfer mode to extended block (E) or streams mode (S).
-parallel n: Use n parallel data channels during extended block transfers.
-passive: Use PASSIVE mode for data transfers.
-pbsz n: Set the data protection buffer size to n n bytes.
-prot [C|S|E|P]: Set the data protection lelvel to clear (C), safe (S), confidential (E) or private (P).
-retry n: Retry commands that fail with transient errors n times.
-resume path: Retry the recursive transfer starting at path.
-tcpbuf n: Set the TCP read/write buffers to n bytes.
-wait: This will cause the client to wait for remote files to stage before attempting to transfer them.
-v: Print UberFTP version information and exit.
-version: Print UberFTP version information and exit.
-versions: Print version information about all used globus modules and exit.

Supported -cmds

-cat url: Print to stdout the contents of the remote file.
-chgrp [-r] group url: Set the group ownership of the remote object(s).
-chmod [-r] perms url: Set the permissions of the remote object(s).
-dir [-r] url: List the contents of the remote object.
-link url path: Create a hardlink named <path> to the remote object.
-ls [-r] url: List the contents of the remote object.
-mkdir url: Create the remote directory.
-rename url path: Rename the remote object to the given path.
-rm [-r] url: Remove the remote object(s).
-rmdir url: Remove the remote directory.
-size url: Return the size of the remote object.
-stage [-r] seconds url: Attempt to stage the remote object(s) over the time period given in seconds.
-symlink url path: Create a symlink named <path> to the remote object.

DEFAULT TRANSFER MODE

By default, without any special environment variables, command line options or commands, uberftp will transfer files in PASSIVE STREAMS mode. PASSIVE means that the client will initiate the data connection which is useful for users behind firewalls. STREAMS mode implies that GRIDFTP features including striping and parallel data connections are not used. In order to take advantage of these features with GridFTP capable servers, you must either change the mode directly using -m command line switch or the mode interactive command, or you can change the mode indirectly by specifying more than one parallel data connection using the -c command line switch or by using the parallel interactive command.

GETTING YOUR GSI PROXY

By default, uberftp requires a GSI certificate. If you do not already have a certificate, see the following web page to learn how to get one:

http://www.ncsa.uiuc.edu/UserInfo/Grid/Security/GetUserCert.html

Once you have a certificate, use the grid-proxy-init command to get a valid proxy.

COMMANDS

! [command]

Run the command using a shell on the local machine. If no command is given, invoke an interactive shell.

? [command]

If command is given, print a (hopefully) helpful blurb about it. Otherwise, list all commands.

active

Change to ACTIVE mode which causes the server to initiate the data connection. The default is PASSIVE mode unless the variable UBERFTP_ACTIVE_MODE is set in the environment. If you are behind a firewall you must use PASSIVE mode.

ascii

Change the data transfer type to ASCII which causes the server to do some simple transformations to the file being transferred. This is mostly useful for changing EOL (end of line) in text files when moving between platforms. This option is almost never necessary today. The default is BINARY mode also known as IMAGE mode.

binary

Change the data transfer type to BINARY (aka IMAGE) which causes the server to not perform transformations to the file being transferred. This is the default and is faster than an ASCII transfer.

blksize size

Change the size of the memory buffer used to read and write data to disks to size bytes. The default block size is 1024*1024 (1048576) bytes but it can be changed at compile time. The block size can be increased to improve file transfer performance. This is not related to the extended block mode block size used to determine the ratio of data to header for data transferred on the data channel.

bugs

Prints information regarding bug reporting and feature requests.

bye

Close all control and data connections and exit.

cat file1 [file2 ... filen]

Print the contents of the remote file(s) to stdout.

cdup

Change the remote working directory up one level.

cd [dir]

Change the remote working directory to dir. If dir is not given, the client will make every attempt to change to the user's home directory.

chgrp [-r] group object [object2 ... objectn]

Change group ownership on the remote object(s).
-r Recursively chgrp everything in the given directory.

chmod [-r] perms object [object2 ... objectn]

Change permissions on the remote object(s).
-r Recursively chmod everything in the given directory.

close

Close the control connection to the remote host.

cksum [on|off]

Enable file cksum comparison after each file transfer. This only works with NCSA's mass storage system.
on Enable checksum comparison
off Disable checksum comparison

cos name

Sets the HPSS class of service to name on the FTP service if the service supports it. If name is omitted, the current class of service is printed. Use the class of service name default to allow the remote server to decide which class of service to use.

dcau [N|A|S subject]

Change the data channel authentication settings. If the service does not support DCAU, these settings are ignored.
N Disabled dcau.
A Expect the remote identity to be mine. (Default)
S subject Expect the remote identity to be subject.

debug [0-3]

Turn debug statements on/off. If no value is given, this command will toggle between debug(2) and non debug(1) mode. Otherwise the debug level is set to the given level.
0 Only errors are printed
1 Default. Errors and some helpful messages are printed
2 Print useful control channel information
3 Print all information

family name

Sets the tape family to name on the FTP service if the service supports it. If name is omitted, the current family is printed. Use the family name default to allow the remote server to decide which family to use.

glob [on|off]

Enable or disable filename globbing. If no option is given, this command will toggle the current setting.
on Enable filename globbing
off Disable filename globbing

dir [-r] [target]

List the contents of the remote target directory. If target is not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.

get [-r] source [destination]

Retrieve file(s) from the remote service. If source implies multiple transfers, either through regular expressions or by using the recursive feature, then destination must be a directory. If destination is not specified, source is used.
-r Recursively transfer the given directory.

hash

Print hash marks during data transfers. This does not work during third party transfers.

help [command]

If command is given, print a helpful blurb about command. Otherwise, list all commands.

keepalive [seconds]

Attempts to keep the control channel from being blocked by firewalls during long data channel operations. UberFTP sends a NOOP command to the service at intervals equal to the specified number of seconds. Setting it to zero will disable keepalive. If seconds are not given, the current timeout is displayed. This feature is disabled by default.
seconds number of seconds between NOOPs. Disabled if zero.

lcat file1 [file2 ... filen]

Print the contents of the local file(s) to stdout.

lcd [dir]

Change the local working directory to dir. If dir is not given, the client will make every attempt to change to the user's home directory.

lcdup

Change the local working directory up one level.

lchgrp [-r] group object [object2 ... objectn]

Change group ownership on the local object(s).
-r Recursively chgrp everything in the given directory.

lchmod [-r] perms object [object2 ... objectn]

Change permissions on the local object(s).
-r Recursively chmod everything in the given directory.

lclose

Close the control connection to the local host.

ldir [-r] [target]

List the contents of the local target directory. If target is not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.

link [oldfile] [newfile]

Create a hardlink to oldfile named newfile on the remote service.

llink [oldfile] [newfile]

Create a hardlink to oldfile named newfile on the local service.

lls [-r] [target]

llscos

List the available class of services on the local server (HPSS only).

llsfam

List the available tape families on the local server (HPSS only).

lmkdir dir1 [dir2 ... dirn]

Create the local directory(ies).

lopen [-P port] [-u user] [-p pass | X] host

Opens a control channel to host and that host becomes the 'local' machine. After using lopen, all local (l*) commands perform their respective operations on host rather than the local machine. This is how third party transfers are accomplished. GSI authentication is used unless the -p option is used.
-P port Connect to port (Default 2811 for GSI, 21 for password).
-u user Connect as alternate user.
-p pass | X

    Use password pass when authenticating with host.

    If pass equals X, read the password from STDIN with

    character echoing turned off.
host Connect to host.

lpwd

Prints the current local working directory.

lrename src dst

Rename the local object src to dst.

lrm [-r] object1 [object1...objectn]

Removes the local file system object(s).
-r Recursively remove the given directory.

lrmdir dir1 [dir2...dirn]

Removes the given directories from the local service.

lquote cmd

Pass cmd to the local FTP service. This allows the user to use server-specific commands that are not available through the uberftp interface.

ls [-r] [target]

List the contents of the remote target directory. If [target] is not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.

lscos

List the available class of services on the remote server (HPSS only).

lsfam

List the available tape families on the remote server (HPSS only).

lsize file1 [file2...filen]

Prints the size of the given object(s).

lstage [-r] seconds object1 [object2...objectn]

Attempt to stage all matching files within the given number of seconds on the local service.
seconds number of seconds to attempt staging
-r Recursively stage all files in the given subdirectory.

lsymlink [oldfile] [newfile]

Create a symlink to oldfile named newfile on the local service.

mput [-r] object1 [object2...objectn]

Retrieve file(s) from the remote service. This is similiar to making multiple calls to get without specifying a destination.
-r Recursively transfer the given directory.

mkdir dir

Create the remote directory.

mode [E|S]

Toggle the data transfer mode between Streams mode and Extended Block mode. The default is Streams mode. If no option is given, it will display the current mode.
E Extended block mode
S Streams mode

mput [-r] object1 [object2...objectn]

Store file(s) to the remote service. This is similiar to making multiple calls to put without specifying a destination.
-r Recursively transfer the given directory.

open [-P port] [-u user] [-p pass | X] host

Opens a control channel to host and that host becomes the 'remote' machine. GSI authentication is used unless the -p option is used.
-P port Connect to port (Default 2811 for GSI, 21 for password).
-u user Connect as user.
-p pass | X

    Use password pass when authenticating with host.

    If pass equals X, read the password from STDIN with

    character echoing turned off.
host Connect to host.

order [type]

Changes the order of lists returned from ls and lls to the given scheme. If type is not given, the current order is displayed.
type Ordering scheme to use. Value options are:

     none  Do not order listings

     name  Order listings by name

     size  Order listings by size

     type  Order listings by type

parallel [number]

Set the number of parallel data connections to number. This is only useful for extended block mode transfers. The default number of data connections is one. If no number is given, the current setting for the number of parallel connects is printed.

passive

Change to PASSIVE mode which causes the client to initiate the data connection. This is the default mode unless the variable UBERFTP_ACTIVE_MODE is set in the environment. If you are behind a firewall you must use PASSIVE mode.

pbsz [size]

Change the length of the protection buffer. The protection buffer is used to encrypt data on the data channel. The length of the protection buffer represents the largest encoded message that is allowed on the data channel. By default, the protection buffer is grown to match the internal buffer used. For efficient transfers, pbsz should be sufficiently larger than blksize so that the wrapped buffer fits within the protection buffer. Otherwise, the blksize buffer is broken into multiple pieces so that each write is less than pbsz when wrapped. If pbsz is not given, the current size is displayed.
size length of protection buffer. 0 will set it to its default.

pget offset size srcfile [destfile]

Retrieve only the specified portion of the file(s). If srcfile is a regular expression and expands to multiple files, and destination is given, destination must refer to a directory.
offset Offset within the file
size Amount of data to retrieve
srcfile Name of remote file
destfile Name of local file. srcfile is used if destfile
is not specified

pput offset size srcfile [destfile]

Store only the specified portion of the file(s). If srcfile is a regular expression and expands to multiple files, and destination is given, destination must refer to a directory.
offset Offset within the file
size Amount of data to retrieve
srcfile Name of local file
destfile Name of remote file. srcfile is used if destfile

is not specified

prot [C|S|E|P]

This command configures the level of security on the data channel after data channel authentication has completed. Clear means that the data will not be protected. Safe means that the data will be integrity protected meaning that altered data will be detected. Confidential means that the data will be unreadable to third parties. Private mode means the data will be confidential and safe.
C Set protection level to clear.
S Set protection level to safe.
E Set protection level to confidential.
P Set protection level to private.

put [-r] source [destination]

Store file(s) to the remote service. If source implies multiple transfers, either through regular expressions or by using the recursive feature, then destination must be a directory. If destination is not specified, source is used.
-r Recursively transfer the given directory.

pwd

Prints the current working directory.

quit

Close all control and data connections and exit.

quote cmd

Pass cmd to the remote FTP service. This allows the user to use server-specific commands that are not available through the uberftp interface.

rename src dst

Rename the remote object src to dst.

retry [cnt]

Configures retry on failed commands that have transient errors. cnt represents the number of times a failed command is retried. A value of zero effectively disables retry. Zero is the default. If no value is given the current setting is displayed.
cnt Number of times a failed command is retried.

resume [-d] path

Sets a restart point for recursive transfers. If a long recursive transfer fails, you can set resume to the path that failed and UberFTP will skip all file and directory creations up to the given path.
path Path to resume transfer at. If path is not given, print the current

resume target.
-d Remove the current resume path.

rm [-r] object1 [object1...objectn]

Removes the remote file system object(s).
-r Recursively remove the given directory.

rmdir dir1 [dir2...dirn]

Removes the given directories from the remote service.

runique

Toggles the client to store files using unique names during put operations.

size file1 [file2...filen]

Prints the size of the given object(s).

stage [-r] seconds object1 [object2...objectn]

Attempt to stage all matching files within the given number of seconds on the remote service.
seconds number of seconds to attempt staging
-r Recursively stage all files in the given subdirectory.

sunique

Toggles the client to store files using unique names during get operations.

symlink [oldfile] [newfile]

Create a symlink to oldfile named newfile on the remote service.

tcpbuf [size]

Set the data channel TCP buffer size to size bytes. If size is not given, the current TCP buffer size will be printed.

versions

Prints the versions of all Globus modules being used.

wait

Toggles whether the client should wait for files to stage before attempting to retrieve them.

IMPROVING FILE TRANSFER PERFORMANCE

Use the active command to enable active mode FTP when using NCSA's UniTree mass storage system if possible since it will give much better file transfer performance. When tranferring files over long distances, use a large value (for example, 16777216) for tcpbuf. When there is high network traffic, you may be able to improve performance using the parallel command to increase the number of parallel data connections to 2-4.

THIRD-PARTY TRANSFERS

In order to perform a third-party transfer, you must log into two FTP servers. Typically, you connect to a single FTP service to "get" files to the local machine and "put" files to the remote service. For third-party transfers, you must connect to a second service thereby replacing the former local machine. In UberFTP terminology, it is referred to as "opening a new local service" since, from the perspective of the user, the new local service will appear as though the user initiated the FTP session from that machine.

All remote service commands have "l*" counterparts that allow you to specify that the command is to be performed on the local service, whether that service is the local machine or a new local service. So to open a new local service, use the "l*" version of the open command:

UberFTP> lopen mss.ncsa.teragrid.org
UberFTP> lclose
Once you have connected to both services, files can be transferred as before with the change that files you "get" go to the new local service and files you "put" are sent from the new local service.

CONTROLLING EPHEMERAL PORT SELECTION

By default, local port selection is managed by the operating system. However, you may wish to specify which ports UberFTP should use for incoming and out going connections. This is useful when dealing with firewalls.

Setting UBERFTP_TCP_PORT_RANGE in your environment will cause all inbound connections to use the specified port range. Likewise, setting UBERFTP_TCP_SOURCE_RANGE in your environment will cause all outbound connections to use the specified port range.

The environment variables GLOBUS_TCP_PORT_RANGE and GLOBUS_TCP_SOURCE_RANGE will also control the ephemeral port selection. These variables behave exactly as their UBERFTP counterparts and are available for backwards compatibility with older versions. The UBERFTP variables take precedence over the GLOBUS variables.

The values of the variables specify a port range, a minimum port number and a maximum port number, separated by either a comma or a space. For example, to set the inbound port range, you would set:

UBERFTP_TCP_PORT_RANGE=40000,50000

Using the space delimiter, this format is also acceptable:

UBERFTP_TCP_PORT_RANGE="40000 50000"

See your shell documentation for the proper syntax for settings variables within your environment.

Setting the ephemeral port range to an unusable range will cause UberFTP connections to fail. For instance, setting a port range from 10 to 100 with a non root process will fail on most operating systems.

EXIT VALUES

UberFTP will exit with a value of 0 if no errors occurred during the session, otherwise it will exit with a value of 1. In non interactive, commandline mode, it will exit after the first error occurs.

EXAMPLES

Set the environment variable to set active mode FTP (improves file transfer performance to the mass storage system). Connect to NCSA's UniTree mass storage system interactively from NCSA's TeraGrid cluster:

  setenv UBERFTP_ACTIVE_MODE on
  % uberftp mss.ncsa.teragrid.org
  ...
  220 UNIX Archive FTP server ready.
  230 User consult logged in.
  UberFTP>

Use the command-line interface to copy a file from NCSA's TeraGrid cluster to the UniTree mass storage system. (There is no need to set tcpbuf since it is over a LAN but active mode is turned on to improve file transfer performance to the mass storage system.):

uberftp mss.ncsa.teragrid.org \
"active; cd work; get file.tar"

Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster. (Note that tcpbuf is set to 16777216 since there is a long network latency between NCSA and SDSC):

uberftp tg-gridftp.sdsc.teragrid.org \
"tcpbuf 16777216; cd scr; put file.tar"