pmlogextract (1) - Linux Manuals
pmlogextract: reduce, extract, concatenate
NAME
pmlogextract - reduce, extract, concatenate and merge Performance Co-Pilot archivesSYNOPSIS
pmlogextract [-dfwz] [-c configfile] [-S starttime] [-s samples] [-T endtime] [-v volsamples] [-Z timezone] input [...] outputDESCRIPTION
pmlogextract reads one or more Performance Co-Pilot (PCP) archive logs identified by input and creates a temporally merged and/or reduced PCP archive log in output. input is a comma-separated list of names, each of which may be the base name of an archive or the name of a directory containing one or more archives. The nature of merging is controlled by the number of input archive logs, while the nature of data reduction is controlled by the command line arguments. The input(s) must be sets of PCP archive logs created by pmlogger(1) with performance data collected from the same host, but usually over different time periods and possibly (although not usually) with different performance metrics being logged.If only one input is specified, then the default behavior simply copies the input set of PCP archive logs, into the output PCP archive log. When two or more sets of PCP archive logs are specified as input, the sets of logs are merged (or concatenated) and written to output.
In the output archive log a <mark> record will be inserted at a time just past the end of each of the input archive logs to indicate a possible temporal discontinuity between the end of one input archive log and the start of the next input archive log. See the MARK RECORDS section below for more information. There is no <mark> record after the end of the last (in temporal order) of the input archive logs.
COMMAND LINE OPTIONS
The command line options for pmlogextract are as follows:
- -c configfile
- Extract only the metrics specified in configfile from the input PCP archive log(s). The configfile syntax accepted by pmlogextract is explained in more detail in the Configuration File Syntax section.
- -d
- Desperate mode. Normally if a fatal error occurs, all trace of the partially written PCP archive output is removed. With the -d option, the output archive log is not removed.
- -f
- For most common uses, all of the input archive logs will have been collected in the same timezone. But if this is not the case, then pmlogextract must choose one of the timezones from the input archive logs to be used as the timezone for the output archive log. The default is to use the timezone from the last input archive log. The -f option forces the timezone from the first input archive log to be used.
- -S starttime
- Define the start of a time window to restrict the samples retrieved or specify a ``natural'' alignment of the output sample times; refer to PCPIntro(1). See also the -w option.
- -s samples
- The argument samples defines the number of samples to be written to output. If samples is 0 or -s is not specified, pmlogextract will sample until the end of the PCP archive log, or the end of the time window as specified by -T, whichever comes first. The -s option will override the -T option if it occurs sooner.
- -T endtime
- Define the termination of a time window to restrict the samples retrieved or specify a ``natural'' alignment of the output sample times; refer to PCPIntro(1). See also the -w option.
- -v volsamples
-
The
output
archive log is potentially a multi-volume data set, and the
-v
option causes
pmlogextract
to start a new volume after
volsamples
log records have been written to the archive log.
-
Independent of any -v option, each volume of an archive is limited to no more than 2^31 bytes, so pmlogextract will automatically create a new volume for the archive before this limit is reached.
-
- -w
- Where -S and -T specify a time window within the same day, the -w flag will cause the data within the time window to be extracted, for every day in the archive log. For example, the options -w -S @11:00 -T @15:00 specify that pmlogextract should include archive log records only for the periods from 11am to 3pm on each day. When -w is used, the output archive log will contain <mark> records to indicate the temporal discontinuity between the end of one time window and the start of the next.
- -Z timezone
- Use timezone when displaying the date and time. Timezone is in the format of the environment variable TZ as described in environ(7).
- -z
- Use the local timezone of the host from the input archive logs. The default is to initially use the timezone of the local host.
CONFIGURATION FILE SYNTAX
The configfile contains metrics of interest - only those metrics (or instances) mentioned explicitly or implicitly in the configuration file will be included in the output archive. Each specifications must begin on a new line, and may span multiple lines in the configuration file. Instances may also be specified, but they are optional. The format for each specification is
where
metric
may be a leaf or a non-leaf name in the Performance Metrics
Name Space (PMNS, see
pmns(5)).
If a metric refers to a non-leaf node in the PMNS,
pmlogextract
will recursively descend the PMNS and include all metrics
corresponding to descendent leaf nodes.
Instances are
optional, and may be specified as a list of one or more
space (or comma) separated names, numbers or strings (enclosed
in single or double quotes).
Elements in the list that are numbers are assumed to be internal
instance identifiers - see
pmGetInDom(3)
for more information.
If no instances are given, then
all
instances of the associated metric(s) will be extracted.
Any additional white space is ignored and
comments may be added with a `#' prefix.
A
<mark>
record contains a timestamp and no performance data and
is used to indicate that there is a time period
in the PCP archive log where we do not know the values of
any
performance metrics, because there was no
pmlogger(1)
collecting performance data during this period. Since these periods are
often associated with the restart of a service or
pmcd(1)
or a system, there may be considerable doubt as to the continuity of
performance data across this time period.
The rationale behind
<mark>
records may be demonstrated with an example.
Consider one input archive log that starts at 00:10 and ends at 09:15 on the
same day, and another input archive log that starts at 09:20 on the
same day and ends at 00:10 the following morning. This would be a very
common case for archives managed and rotated by
pmlogger_check(1)
and
pmlogger_daily(1).
The output archive log created by
pmlogextract
would contain:
The time period where the performance data is missing starts just after
09:15 and ends just before 09:20.
When the output archive log is processed with any of the PCP reporting
tools, the
<mark>
record is used to indicate a period of missing
data. For example using the output archive above, imagine one was reporting the average
I/O rate at 30 minute intervals
aligned on the hour and half-hour.
The I/O count metric is a counter, so the
average I/O rate requires two valid values from
consecutive sample times.
There would be
values for all the intervals ending at 09:00,
then no values at 09:30 because of the
<mark>
record, then no values at 10:00 because the ``prior'' value at 09:30 is not
available, then the rate would be reported again at 10:30 and continue
every 30 minutes until the last reported value at 01:00.
The presence of
<mark>
records in a PCP archive log can be established
using
pmdumplog(1)
where a timestamp and the annotation
<mark>
is used to indicate a
<mark>
record.
If any of these checks fail,
pmlogextract
reports the details and aborts without creating the
output
archive.
To address these semantic issues, use
pmlogrewrite(1)
to translate the
input
archives into equivalent archives with consistent metdadata before using
pmlogextract.
Should one of the input archive logs be corrupted (this can happen
if the
pmlogger
instance writing the log suddenly dies), then
pmlogextract
will detect and report the position of the corruption in the file,
and any subsequent information from that archive log will not be processed.
If any error is detected,
pmlogextract
will exit with a non-zero status.
CONFIGURATION FILE EXAMPLE
This is an example of a valid
configfile:
MARK RECORDS
When more than one input archive log contributes performance data to the
output archive log, then
<mark>
records are inserted to indicate a possible
discontinuity in the performance data.
00:10.000
...
09:15.000
09:15.001
09:20.000
...
01:10.000
METADATA CHECKS
When more than one
input
archive set is specified,
pmlogextract
performs a number of checks to ensure the metadata is consistent for
metrics appearing in more than one of the
input
archive sets.
These checks include:
FILES
For each of the
input
and
output
archive logs, several physical files are used.
PCP ENVIRONMENT
Environment variables with the prefix
PCP_
are used to parameterize the file and directory names
used by PCP.
On each installation, the file
/etc/pcp.conf
contains the local values for these variables.
The
$PCP_CONF
variable may be used to specify an alternative
configuration file,
as described in
pcp.conf(5).
DIAGNOSTICS
All error conditions detected by
pmlogextract
are reported on
stderr
with textual (if sometimes terse) explanation.
CAVEATS
The preamble metrics (pmcd.pmlogger.archive, pmcd.pmlogger.host,
and pmcd.pmlogger.port), which are automatically recorded by
pmlogger
at the start of the archive, may not be present in the archive output by
pmlogextract.
These metrics are only relevant while the archive is being created,
and have no significance once recording has finished.
SEE ALSO
PCPIntro(1),
pmdumplog(1),
pmlc(1),
pmlogger(1),
pmlogreduce(1),
pmlogrewrite(1),
pcp.conf(5)
and
pcp.env(5).