xferstats (8) - Linux Manuals

xferstats: compiles information about file transfers from logfiles

NAME

xferstats - compiles information about file transfers from logfiles

SYNOPSIS

xferstats [- | + ] [-c <config file>] [-T <number>] [[-+]C] [[-+]H] [[-+]n] [[-+]s] [[-+]r] [[-+]a] [[-+]g] [[-+]i] [[-+]u] [[-+]h] [[-+]m] [[-+]w] [[-+]M] [[-+]f [number]] [[-+]d [number]] [[-+]t [number]] [[-+]O [number]] [[-+]o [number]] [+L <number>] [+A] [-D <number>] [+D <number>] [+P <prefix>] [+B <timestamp>] [+E <timestamp>] [-v] [--version] [--help] [--] [filename(s)]

DESCRIPTION

xferstats is a utility that parses wu-ftpd, ncftpd, and (special) apache xferlogs (although it's not difficult to add support for other log types) and gives various information and totals about the data transferred.

OPTIONS

+
The single plus tells xferstats to get the xferlog from stdin. For example:
cat xferlog | xferstats +
-
The single hyphen tells xferstats to get the xferlog from a file. While this is the default behaviour, the configuration file may change the default, and this option allows you to override it. For example:
xferstats - /var/log/xferlog
-c configuration file
Specify an alternate path and filename for the configuration file. The packaged default is /etc/xferstats.cfg but your administrator may have changed this. A new default can be set by changing the appropriate variable in xferstats.h and recompiling.
-T log type
Specify the type of log file to be parsed. Currently, only three types are defined, wu-ftpd for wu-ftpd-style logs, ncftpd for ncftpd-style logs, and apache for special Apache logs. Note: you can also use "wu-ftp", "wuftp", "wuftpd", or "ncftp". In order for xferstats to parse Apache logfiles, you must use a special LogFile line: LogFormat "%{%a %b %d %X %Y}t %T %h %b %f %u". xferstats may or may not support regular Apache logfiles in the future. If this is something you want, please let me know.
-v, --version
Display what version of xferstats you're running.
--help
Display a quick summary of the command line options.
--
Use a -- (double hyphen) to indicate that you are finished giving parameters and will follow with a filename. This is particularly useful when you want to specify a filename that begins with a + or -

TOGGLE OPTIONS

The options in this section can be either forced on with +[option] or forced off with -[option]. Options in this section, like all other options, will always override any settings in the xferstats configuration file. Any arguments to toggle options should only be used when enabling an option. Supplying an argument when disabling an option will generate an error.

C
When enabled, xferstats will try to weed out corrupt log lines by strictly checking to make sure that the data is valid. If you --enable-debug_flow at compile time, xferstats will notify you when corrupt lines are detected.
H
When enabled, xferstats will generate HTML tables and graphs instead of text tables. In order to actually generate tables instead of broken graphic icons, you must put the graph JPGs in the directory pointed to by the GRAPH_PATH configuration command. For more information, see the CONFIGURATION FILE section.
n
When enabled, xferstats will refrain from adding <HTML>, <BODY>, and <TITLE> tags, so that the output can be inserted into pages containing custom headers.
s
When enabled, xferstats will output HTML as a single page (filename "xferstats.html") instead of a page for each section. This option also disables <HTML> and <BODY> tags, although it does produce a <TITLE> tag unless the NO_HTML_HEADERS option is also specified.
r
When enabled, xferstats will generate statistics about real user connections (but not necessarily only real user connections)
a
When enabled, xferstats will generate statistics about anonymous connections (but not necessarily only anonymous connections)
g
When enabled, xferstats will generate statistics about guest connections (but not necessarily only guest connections)
i
When enabled, xferstats will generate statistics about inbound (the product of a PUT) files (but not necessarily only inbound files)
u
When enabled, xferstats will generate statistics about outbound (the product of a GET) files (but not necessarily only outbound files)
h
When enabled, xferstats will generate a report listing traffic by hour
m
When enabled, xferstats will generate a report listing traffic by month (January, February, etc.)
S
When enabled, xferstats will generate a report listing traffic by file size
w
When enabled, xferstats will generate a report listing traffic by day of the week (Monday, Tuesday, etc.)
M
When enabled, xferstats will generate a report listing traffic by day of the month (1st, 2nd, 3rd, etc.)
f [number]
When enabled, xferstats will generate a report listing traffic by file. You can supply an optional argument that will determine how many file records xferstats displays (0 (zero) disables the limit)
d [number]
When enabled, xferstats will generate a report listing traffic by directory. You can supply an optional argument that will determine how many directory records xferstats displays (0 (zero) disables the limit). See the -D and +D options, as well as the DIR_DEPTH and DIR_FILTER configuration commands for more information.
t [number]
When enabled, xferstats will generate a report listing traffic by top-level domain. You can supply an optional argument that will determine how many TLD records xferstats displays (0 (zero) disables the limit)
O [number]
When enabled, xferstats will generate a report listing traffic by domain. You can supply an optional argument that will determine how many domain records xferstats displays (0 (zero) disables the limit)
o [number]
When enabled, xferstats will generate a report listing traffic by host. You can supply an optional argument that will determine how many host records xferstats displays (0 (zero) disables the limit)

OTHER OPTIONS

+A
Include all users, generate all reports
+L <number>
Limit the daily report to the last <number> days
-D <number>
Depth of path detail for directories (default 3)
+D <directory>
Generate reports on files located only in <directory>. For example:
+D /pub will report only on paths under /pub. Specifying multiple +D options will instruct xferstats to include data from all specified directories.
+P <pathprefix>
Strip the <prefix> from any pathnames that begin with it. This is the same as specifying a STRIP_PREFIX configuration file command. Keep in mind that the DIR_FILTER (or +D) is checked before the STRIP_PREFIX (or +P) when using these.
+B <timestamp>
Providing a +B argument instructs xferstats to only generate data on transfers occuring after that time. The time stamp format is "Jan DD HH:MM:ss YYYY". One may combine +B and +E to generate a report about a specific window in time.
+E <timestamp>
Providing a +E argument instructs xferstats to only generate data on transfers occuring before that time. The time stamp format is "Jan DD HH:MM:ss YYYY". One may combine +B and +E to generate a report about a specific window in time.

The default compile-time setting for the xferlog location is /var/log/xferlog but your administrator may have changed this. If xferstats reports an error, try specifying the filename as the last command line parameter.

CONFIGURATION FILE

The default configuration file path is /etc/xferstats.cfg although your administrator may have changed this.

Any text after a '#' is ignored, as long as the '#' begins a line or is prepended by whitespace.

Commands (LOGFILE, ANON_TRAFFIC, etc) are not case sensitive, although some arguments may be.

LOGFILE <filename>
sets the default log file to process. It can be overridden by adding "<filename>" at the end of the command line options. You may specify as many LOGFILE lines as you like; they will be processed in the order specified (although the order processed should not change the output). Example:
LOGFILE /var/log/xferlog LOGFILE /var/log/xferlog.1 LOGFILE /var/log/xferlog.2
STRICT_CHECK
takes no arguments. When enabled, xferstats will try to weed out corrupt log lines by strictly checking to make sure that the data is valid. If you --enable-debug_flow at compile time, xferstats will notify you when corrupt lines are detected. It can be enabled with "+C" or disabled with "-C" on the command line.
ANON_TRAFFIC
takes no arguments. When enabled, xferstats will generate statistics about anonymous connections (but not necessarily only anonymous connections). It can be enabled with "+a" or disabled with "-a" on the command line.
GUEST_TRAFFIC
takes no arguments. When enabled, xferstats will generate statistics about guest connections (but not necessarily only guest connections). This option has no effect when parsing non-wuftpd logfiles. It can be enabled with "+g" or disabled with "-g" on the command line.
REAL_TRAFFIC
takes no arguments. When enabled, xferstats will generate statistics about real user connections (but not necessarily only real user connections). It can be enabled with "+r" or disabled with "-r" on the command line.
INBOUND
takes no arguments. When enabled, xferstats will generate statistics about inbound (the product of a PUT) files (but not necessarily only inbound files). It can be enabled with "+i" or disabled with "-i" on the command line.
OUTBOUND
takes no arguments. When enabled, xferstats will generate statistics about outbound (the product of a GET) files (but not necessarily only outbound files). It can be enabled with "+u" or disabled with "-u" on the command line.
HOURLY_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic per hour. It can be enabled with "+h" or disabled with "-h" on the command line.
DOW_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic by day of the week (Monday, Tuesday, etc). It can be enabled with "+w" or disabled with "-w" on the command line.
DOM_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic by day of the month (1st, 2nd, etc). It can be enabled with "+M" or disabled with "-M" on the command line.
TLD_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic by top-level domain (com, edu, net, us, uk, etc). It can be enabled with "+t <number>" or disabled with "-t" on the command line.
DOMAIN_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic by domain (foo.com, blah.edu, etc). It can be enabled with "+O <number>" or disabled with "-O" on the command line.
HOST_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic by host. It can be enabled with "+o" or disabled with "-o" on the command line.
DIR_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic by directory. The directory depth that xferstats will keep track of can be controlled by the DIR_DEPTH configuration command or with the "-l <depth>" command line option. It can be enabled with "+d" or disabled with "-d" on the command line.
FILE_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic by file. It can be enabled with "+f" or disabled with "-f" on the command line.
MONTHLY_REPORT
takes no arguments. When enabled, xferstats will generate a report listing traffic per month (January, February, etc). It can be enabled with "+m" or disabled with "-m" on the command line.
HTML_OUTPUT
takes no arguments. When enabled, xferstats will output all reports into individual HTML files. Future versions will contain a configuration option to set the directory in which to place the HTML files. For now they are placed in the present working# directory. It can be enabled with "+H" or disabled with "-H" on the command line.
USE_STDIN
takes no arguments. When enabled, xferstats will, by default, read the logfile from the standard input instead of from a file on disk. It can be enabled with "+" or disabled with "-" on the command line.
LOG_TYPE
takes one argument, the type of log that will be processed by default. The argument can be either a number or a case insensitive string and are as follows:
wu-ftpd log: "1" or "wu-ftp" or "wu-ftpd" or "wuftp" or "wuftpd"
ncftpd log : "2" or "ncftp" or "ncftpd"
Apache log : "3" or "apache"
Therefore:
LOG_TYPE 1
LOG_TYPE wu-ftp
LOG_TYPE wu-ftpd
LOG_TYPE wuftp
LOG_TYPE wuftpd
are all equivalent.
Future versions will contain support for additional logfile types. It can be set on the command line with "-T <number>"
DIR_DEPTH
takes one argument, the number of subdirectories deep that xferstats should go when generating statistics by directory. This configuration command has no effect when DIR_REPORT has not been enabled. DIR_DEPTH has no upper bound. A DIR_DEPTH of '0' (zero) tells xferstats to process all subdirectories. The default value is 3. It can be set on the command line with "-D <number>". Example:
DIR_DEPTH 3
DIR_FILTER
takes one argument, the directory that xferstats should report on. xferstats will discard any data that does not have a path that begins with the value set by DIR_FILTER. This value is case-sensitive, as it is part of a path name. Specifying multiple statements will instruct xferstats to include data from all specified directories. It can be set on the command line with "+D <path>". Example:
DIR_FILTER /pub/foobar
NUMBER_FILE_STATS
takes one argument, the number of files to report on in the file statistics section. It must be a number greater than or equal to zero, with zero denoting no limit. It can be set on the command line with "+f <number>". The default is 50.
NUMBER_FILE_STATS 50
NUMBER_DIR_STATS
takes one argument, the number of directories to report on in the directory statistics section. It must be a number greater than or equal to zero, with zero denoting no limit. It can be set on the command line with "+d <number>". The default is 50.
NUMBER_DAILY_STATS
takes one argument, the number of days to report on in the daily statistics section. It must be a number greater than or equal to zero, with zero denoting no limit. It can be set on the command line with "+L <number>". The default is 50.
Note: the daily statistics section will always be displayed.
NUMBER_TLD_STATS
takes one argument, the number of top-level domains to report on in the TLD statistics section. It must be a number greater than or equal to zero, with zero denoting no limit. It can be set on the command line with "+t <number>". The default is 50.
NUMBER_DOMAIN_STATS
takes one argument, the number of top-level domains to report on in the domain statistics section. It must be a number greater than or equal to zero, with zero denoting no limit. It can be set on the command line with "+O <number>". The default is 50.
NUMBER_HOST_STATS
takes one argument, the number of hosts to report on in the host statistics section. It must be a number greater than or equal to zero, with zero denoting no limit. It can be set on the command line with "+o <number>". The default is 50.
MAX_REPORT_SIZE
takes one argument, the maximum size of any one output table. This is particularly useful to keep xferstats from generating gigantic tables that make graphical browsers choke. It will split tables to be no larger than# this value. It must be greater than or equal to ten, or zero, zero denoting no limit. It can be set on the command line with "+R <number>". The default is 30.
GRAPH_PATH
takes one argument, the location of the JPGs used to draw graphs in the HTML output. It can be an absolute path (ie, http://foo.bar.com/graphs/) or a relative path (ie, ../graphs/). You must include a trailing '/' or it will not work. This option cannot be set on the command line. The default is nothing (which will make the browser look for the graphs in the directory with the HTML). Example:
GRAPH_PATH ../graphs/
NO_HTML_HEADERS
takes no arguments. When enabled, xferstats will refrain from adding <HTML>, <BODY>, and <TITLE> tags, so that the output can be inserted into pages containing custom headers. It can be enabled with "+n" or disabled with "-n" on the command line.
REFRESH
takes one argument, the number of seconds after being displayed that the browser should refresh it. Unspecified or zero indicates no refresh.
SINGLE_PAGE
takes no arguments. When enabled, xferstats will output HTML as a single page (entitled "xferstats.html") instead of a page for each section. This option also disables <HTML> and <BODY> tags, although it does produce a <TITLE> tag unless the NO_HTML_HEADERS option is also specified. It can be enabled with "+s" or disabled with "-s" on the command line.
STRIP_PREFIX
takes one argument. When specified, xferstats will strip the given string from the beginning of any paths. Keep in mind when using this option with DIR_FILTER that the DIR_FILTER is applied -before- stripping, so that you must take this into account when specifying the DIR_FILTER string. This option cannot be set on the command line. Example:
STRIP_PREFIX /home/ftp

The next 11 entries are all basically the same--they define how the reports are to be sorted.

0 -- Sort by the number of bytes downloaded
1 -- Sort by the number of files downloaded
2 -- Sort by the "name". This varies by report; for the file, directory, TLD, etc reports, it sorts them alphabetically; for the file size report it lists the file sizes in alphabetical order; for the daily report, it lists the days in order. You get the idea. 3 -- Sort by average throughput, in K/s
FILE_SORT_PREF, DIR_SORT_PREF, DOMAIN_SORT_PREF, TLD_SORT_PREF, HOST_SORT_PREF, DOM_SORT_PREF, DOW_SORT_PREF, HOURLY_SORT_PREF, DAILY_SORT_PREF, SIZE_SORT_PREF, MONTHLY_SORT_PREF
CHUNK_INPUT
takes one argument. When specified, xferstats will parse the logfiles in chunks of the given size (in number of lines), process them, and continue. This will keep it from trying to parse the entire log at once, using up a lot of memory for large logs. Specifying too small a value will cause xferstats to waste a lot of time switching tasks, while too large a value may slow it down as your system swaps. Toy with this value a bit and see what works best for you. If left unspecified, xferstats will continue to parse the logs in one pass.

FILES

xferlog (generated by wu-ftpd or ncftpd)

/etc/xferstats.cfg contains all of the default settings and many settings which cannot be set from the command line.

If you installed the RPM version of xferstats, you'll find the JPGs for the graphs in /usr/share/xferstats/graphs.

BUGS

The JPG graph pieces included are not ideal; sometimes artifacts show up. I will eventually get around to making new GIFs or something.

Please send bug reports to phil [at] off.net If the xferlog that you're using is smaller than 1 megabyte, please send a gzipped copy, along with the exact parameters you're using.

ACKNOWLEDGEMENTS

I greatly appreciate the suggestions and assistance of Nat Friedman <ndf [at] mit.edu>.

Thanks to Mike Shaver <shaver [at] mozilla.org> and Jamie Zawinski <jwz [at] netscape.com> for coughing up gigantic mozilla.org logs to aid my testing.

Thanks to Phil Copeland <bryce [at] zen.ics.uwe.ac.uk> for getting on my back constantly to add new stuff, fix nagging bugs, and speed it up. Oh, and for using it, at least someone does :).

Extreme gratitude to Mark Hagler <hagler [at] hagler.org>, someone whom I've left off of this list for too long. Mark was kind enough to Amazon me a copy of O'Reillys _Pthreads Programming_ which was the catalyst for the threaded version and really helped get me interested in the project again. While the threaded version doesn't exist anymore, the book is still terribly useful for other projects.

LOCATION

xferstats is available from the following locations. The old addresses are valid, but are deprecated! Please use the new addresses whenever possible.
http://xferstats.off.net/
ftp://xferstats.off.net/pub/xferstats/

AUTHOR

xferstats is Copyright (c) 1997-2000 Phil Schwan <phil [at] off.net> and is released under the terms and provisions of the GNU Public License.

SEE ALSO

ftpd(8), ncftpd(8), xferlog(5)

Pages related to xferstats