PCPIntro (1) - Linux Manuals
PCPIntro: introduction to the Performance Co-Pilot (PCP)
NAME
PCPIntro - introduction to the Performance Co-Pilot (PCP)INTRODUCTION
The Performance Co-Pilot (PCP) is a toolkit designed for monitoring and managing system-level performance. These services are distributed and scalable to accommodate the most complex system configurations and performance problems.PCP supports many different platforms, including (but not limited to) Linux, MacOSX, Solaris and Windows. From a high-level PCP can be considered to contain two classes of software utility:
- PCP Collectors
- These are the parts of PCP that collect and extract performance data from various sources, e.g. the operating system kernel.
- PCP Monitors
- These are the parts of PCP that display data collected from hosts (or archives) that have the PCP Collector installed. Many monitor tools are available as part of the core PCP release, while other (typically graphical) monitoring tools are available separately in the PCP GUI package.
This manual entry describes the high-level features and options common to most PCP utilities available on all platforms.
OVERVIEW
The PCP architecture is distributed in the sense that any PCP tool may be executing remotely. On the host (or hosts) being monitored, each domain of performance metrics, whether the kernel, a service layer, a database management system, a web server, an application, etc. requires a Performance Metrics Domain Agent (PMDA) which is responsible for collecting performance measurements from that domain. All PMDAs are controlled by the Performance Metrics Collector Daemon (pmcd(1)) on the same host.Client applications (the monitoring tools) connect to pmcd(1), which acts as a router for requests, by forwarding requests to the appropriate PMDA and returning the responses to the clients. Clients may also access performance data from sets of PCP archives (created using pmlogger(1)) for retrospective analysis.
Security philosophy
PCP redistributes a wealth of performance information within a host and across its networks. The following security philosophy underlies the setting of several defaults that control how much information is sent and received.By default, the information exposed by PMCD about a host is approximately of the same level of confidentiality as available to a completely unprivileged user on that host. So, performance data that is available to be read completely freely on a machine may be made available by PMCD to the network.
However, the host running PMCD and its network is not assumed to run only friendly applications. Therefore, write type operations, including from the local host, are not permitted by default.
These defaults may be overridden (expanded or reduced) in several ways, including by specifying network ACLs in pmcd.conf, activating non-default PMDAs, or by using PMCD connections that pass user credentials. For example, some PMDAs automatically provide greater information for particular credentialed users or groups.
Applications
The following performance monitoring applications are primarily console based, typically run directly from the command line, and are just a small subset of the tools available as part of the base PCP package.Each tool or command is documented completely in its own reference page.
- pmstat
- Outputs an ASCII high-level summary of system performance.
- pmie
- An inference engine that can evaluate predicate-action rules to perform alarms and automate system management tasks.
- pminfo
- Interrogate specific performance metrics and the metadata that describes them.
- pmlogger
- Generates PCP archives of performance metrics suitable for replay by most PCP tools.
- pmrep
- Highly customizable performance metrics reporter with support for various different output modes.
- pmval
- Simple periodic reporting for some or all instances of a performance metric, with optional VCR time control.
If the PCP GUI package is installed then the following additional tools are available.
- pmchart
- Displays trends over time of arbitrarily selected performance metrics from one or more hosts.
- pmtime
- Time control utility for coordinating the time between multiple tools (including pmchart and pmval).
- pmdumptext
- Produce ASCII reports for arbitrary combinations of performance metrics.
COMMON COMMAND LINE ARGUMENTS
There is a set of common command line arguments that are used consistently by most PCP tools.- -a archive
-
Performance metric information is retrospectively retrieved
from the set of Performance Co-Pilot (PCP) archives identified by
archive,
previously generated by
pmlogger(1).See
pcp-archive(5)
for format documentation.
The
-a
and
-h
options are mutually exclusive.
-
archive is a comma-separated list of names, each of which may be the name of a directory containing one or more archives, the base name common to all of the physical files created by an instance of pmlogger(1), or any one of the physical files, e.g. /path/to/myarchives (directory) or myarchive (base name) or myarchive.meta (the metadata file) or myarchive.index (the temporal index) or myarchive.0 (the first data volume of archive) or myarchive.0.bz2 or myarchive.0.bz (the first data volume compressed with bzip2(1)) or myarchive.0.gz or myarchive.0.Z or myarchive.0.z (the first data volume compressed with gzip(1)), myarchive.1 or myarchive.3.bz2 or myarchive.42.gz etc.
-
- -h hostname
- Unless directed to another host by the -h option, or to a set of archives by the -a option, the source of performance metrics will be the Performance Metrics Collector Daemon (PMCD) on the local host. Refer to the PMCD HOST SPECIFICATION section later for further details on the many options available when forming the hostname specification, as well as a detailed description of the default local host connection. The -a and -h options are mutually exclusive.
- -s samples
- The argument samples defines the number of samples to be retrieved and reported. If samples is 0 or -s is not specified, the application will sample and report continuously (in real time mode) or until the end of the set of PCP archives (in archive mode).
- -z
- Change the reporting timezone to the local timezone at the host that is the source of the performance metrics, as identified via either the -h or -a options.
- -Z timezone
- By default, applications report the time of day according to the local timezone on the system where the application is executed. The -Z option changes the timezone to timezone in the format of the environment variable TZ as described in environ(7).
INTERVAL SPECIFICATION AND ALIGNMENT
Most PCP tools operate with periodic sampling or reporting, and the -t and -A options may be used to control the duration of the sample interval and the alignment of the sample times.- -t interval
-
-
Set the update or reporting interval.
The interval argument is specified as a sequence of one or more elements of the form
number[units]
where number is an integer or floating point constant (parsed using strtod(3)) and the optional units is one of: seconds, second, secs, sec, s, minutes, minute, mins, min, m, hours, hour, h, days, day and d. If the unit is empty, second is assumed.In addition, the upper case (or mixed case) version of any of the above is also acceptable.
Spaces anywhere in the interval are ignored, so 4 days 6 hours 30 minutes, 4day6hour30min, 4d6h30m and 4d6.5h are all equivalent.
Multiple specifications are additive, e.g. ``1hour 15mins 30secs'' is interpreted as 3600+900+30 seconds.
-
Set the update or reporting interval.
- -A align
-
-
By default samples are not necessarily aligned on
any natural unit of time. The
-A
option may be used to force the initial sample to be aligned on the
boundary of a natural time unit.
For example
-A 1sec,
-A 30min
and
-A 1hour
specify alignment on whole seconds, half and whole hours respectively.
The align argument follows the syntax for an interval argument described above for the -t option.
Note that alignment occurs by advancing the time as required, and that -A acts as a modifier to advance both the start of the time window (see the next section) and the origin time (if the -O option is specified).
-
By default samples are not necessarily aligned on
any natural unit of time. The
-A
option may be used to force the initial sample to be aligned on the
boundary of a natural time unit.
For example
-A 1sec,
-A 30min
and
-A 1hour
specify alignment on whole seconds, half and whole hours respectively.
TIME WINDOW SPECIFICATION
Many PCP tools are designed to operate in some time window of interest, e.g. to define a termination time for real-time monitoring or to define a start and end time within a set of PCP archive logs.In the absence of the -O and -A options to specify an initial sample time origin and time alignment (see above), the PCP application will retrieve the first sample at the start of the time window.
The following options may be used to specify a time window of interest.
- -S starttime
-
-
By default the time window commences immediately in real-time mode,
or coincides with time at the start of the set of PCP archive logs
in archive mode.
The
-S
option may be used to specify a later time
for the start of the time window.
The starttime parameter may be given in one of three forms (interval is the same as for the -t option as described above, datetime is described below):
- interval
- To specify an offset from the current time (in real-time mode) or the beginning of a set of PCP archives (in archive mode) simply specify the interval of time as the argument. For example -S 30min will set the start of the time window to be exactly 30 minutes from now in real-time mode, or exactly 30 minutes from the start of a set of PCP archives.
- -interval
- To specify an offset from the end of a set of PCP archive logs, prefix the interval argument with a minus sign. In this case, the start of the time window precedes the time at the end of the set of archives by the given interval. For example -S -1hour will set the start of the time window to be exactly one hour before the time of the last sample in a set of PCP archive logs.
- @datetime
- To specify the calendar date and time (local time in the reporting timezone) for the start of the time window, use the datetime syntax preceded by an at sign. Refer to the datetime description below for detailed information.
-
By default the time window commences immediately in real-time mode,
or coincides with time at the start of the set of PCP archive logs
in archive mode.
The
-S
option may be used to specify a later time
for the start of the time window.
- -T endtime
-
-
By default the end of the time window is unbounded
(in real-time mode) or aligned with the time at the end of a set of PCP archive
logs (in archive mode).
The
-T
option may be used to specify an earlier time for
the end of the time window.
The endtime parameter may be given in one of three forms (interval is the same as for the -t option as described above, datetime is described below):
- interval
- To specify an offset from the start of the time window simply use the interval of time as the argument. For example -T 2h30m will set the end of the time window to be 2 hours and 30 minutes after the start of the time window.
- -interval
- To specify an offset back from the time at the end of a set of PCP archive logs, prefix the interval argument with a minus sign. For example -T -90m will set the end of the time window to be 90 minutes before the time of the last sample in a set of PCP archive logs.
- @datetime
- To specify the calendar date and time (local time in the reporting timezone) for the end of the time window, use the datetime syntax preceded by an at sign. Refer to the datetime description below for detailed information.
-
By default the end of the time window is unbounded
(in real-time mode) or aligned with the time at the end of a set of PCP archive
logs (in archive mode).
The
-T
option may be used to specify an earlier time for
the end of the time window.
- -O origin
-
-
By default samples are fetched from the start of the
time window (see description of
-S
option) to the end of the time window (see description of
-T
option).
The
-O
option allows the specification of an origin within the time window
to be used as the initial sample time. This
is useful for interactive use of a PCP tool with the
pmtime(1)
VCR replay facility.
The origin argument accepted by -O conforms to the same syntax and semantics as the starttime argument for the -T option.
For example -O -0 specifies that the initial position should be at the end of the time window; this is most useful when wishing to replay ``backwards'' within the time window.
-
By default samples are fetched from the start of the
time window (see description of
-S
option) to the end of the time window (see description of
-T
option).
The
-O
option allows the specification of an origin within the time window
to be used as the initial sample time. This
is useful for interactive use of a PCP tool with the
pmtime(1)
VCR replay facility.
The datetime argument for the
-O,
-S
and
-T
options consists of:
For any missing low order fields, the default value of
0 is assumed for hours, minutes and seconds, 1 for day of the month and Jan for months.
Hence, the following are equivalent:
-S '@ Mar 1996'
and
-S '@ Mar 1 00:00:00 1996'.
If any high order fields are missing, they are filled in by
starting with the
year, month and day from the current time (real-time mode) or
the time at the beginning of the set of PCP archive logs (archive mode)
and advancing the
time until it matches the fields that are specified.
So, for example if the time window starts by default at
``Mon Mar 4 13:07:47 1996'',
then
-S @13:10
corresponds to 13:10:00 on Mon Mar 4, 1996,
while
-S @10:00
corresponds to 10:00:00 on Tue Mar 5, 1996 (note this is the
following day).
For greater precision than afforded by
datetime(3),
the seconds component may be a floating point number.
If a timezone is not included in a
datetime
then there ares several interpretations available depending
on the other command line options used.
The default is to use the local timezone on the system where
the PCP tool is being run.
A
-Z
option specifies an explicit timezone, else a
-z
option changes the timezone to the local timezone at the host
that is the source of the performance metrics.
For all users and most applications, direct use of the PMIDs would be inappropriate
(e.g. this would limit the range of accessible metrics, make the code
hard to maintain, force the user interface to be particularly baroque,
etc.).
Hence a Performance Metrics Name Space (PMNS)
is used to provide external names and
a hierarchic classification for performance metrics.
A PMNS is
represented as a tree, with each node having a label, a pointer to
either a PMID (for leaf nodes) or a set of descendent
nodes in the PMNS (for non-leaf nodes).
A node label must begin with
an alphabetic character, followed by zero or more characters drawn
from the alphabetics, the digits and character `_' (underscore).
For alphabetic characters in a node label, upper and
lower case are distinguished.
By convention, the name of a performance metric is constructed by
concatenation of the node labels on a path through the PMNS from the
root node to a leaf node, with a ``.'' as a separator.
The root node in
the PMNS is unlabeled, so all names begin with the label associated
with one of the descendent nodes below the root node of the PMNS, e.g.
kernel.percpu.syscall.
Typically (although this is not a requirement)
there would be at most one name for each PMID in a PMNS.
For example
kernel.all.cpu.idle
and
disk.dev.read
are the unique names for two distinct performance
metrics, each with a unique PMID.
Groups of related PMIDs may be named
by naming a non-leaf node in the PMNS tree, e.g.
disk.
The default local PMNS used by
pmcd
is located at
$PCP_VAR_DIR/pmns/root
however the environment
variable
PMNS_DEFAULT
may be set to the full pathname of a different PMNS which will
then be used as the default local PMNS.
Most applications do not use the local PMNS directly,
but rather import parts of the PMNS as required from the
same place that performance metrics are fetched, i.e. from
pmcd(1)
for live monitoring or from a set of PCP archives for retrospective
monitoring.
To explore the PMNS
use
pminfo(1),
or if the PCP GUI package is installed the New Chart and Metric Search
windows within
pmchart(1).
If the source of performance metrics is real-time from
pmcd(1)
then the accepted
syntax is
If the source of performance metrics is a set of PCP archive logs then the
accepted syntax
is
The
host:,
archive/
and
[instance1,instance2,...]
components are all optional.
The
,
delimiter in the list of instance names
may be replaced by white space.
Special characters in
instance
names may be escaped by surrounding the name in double quotes or preceding
the character with a backslash.
White space is ignored everywhere except within a quoted
instance
name.
An empty
instance
is silently ignored, and in particular
``[]'' is the same as no
instance,
while ``[one,,,two]'' is parsed as specifying just
the two instances ``one'' and ``two''.
As a special case, if the
host
is the single character ``@'' then this refers to a
PM_CONTEXT_LOCAL
source, see
pmNewContext(3).
A secure
pmcd
connection requires use of certificate-based authentication.
The security features offered by
pmcd
and
pmproxy
are implemented using the Network Security Services (NSS) APIs and
utilities.
The NSS
certutil
tool can be used to create certificates suitable for establishing
trust between PCP monitor and collector hosts.
A complete description is beyond the scope of this document, refer
to the
PCP ENVIRONMENT,
FILES
and
SEE ALSO
sections for detailed information.
This includes links to tutorials on the steps involved in setting up the
available security features.
Names of remote hosts running the
pmcd(1)
daemon can of course also be provided to request a remote host be used.
The most basic form of
pmcd
host specification is a simple host name, possibly including the
domain name if necessary.
However, this can be extended in a number of ways to further refine
attributes of the connection made to
pmcd.
The
pmcd
port number and also optional
pmproxy(1)
hostname and its port number, can be given as part of the host
specification, since PCP version 3.0.
These supersede (and override) the old-style PMCD_PORT, PMPROXY_HOST
and PMPROXY_PORT environment variables.
The following are valid hostname specifications that specify connections to
pmcd
on host
nas1.servers.com
with/without a list of ports, with/without a
pmproxy(1)
connection through a firewall, and with IPv6 and IPv4 addresses as shown.
In addition, ``connection attributes'' can also be specified.
These include username, password (can be given interactively
and may depend on the authentication mechanism employed),
whether to target a specific running container, whether to use
secure (encrypted) or native (naked) protocol, and so on.
The previous examples all default to native protocol, and use
no authentication.
This can be altered, as in the following examples.
The choice of authentication method, and other resulting parameters like
username, optionally password, etc, depends on the SASL2 configuration
used by each (remote)
pmcd.
Tutorials are available specifying various aspects of configuring the
authentication module(s) used, these fine details are outside the scope
of this document.
In all situations, host names can be used interchangeably with IPv4 or IPv6
addressing (directly), as shown above. In the case of an IPv6 address, the
full address must be enclosed by square brackets and the scope (interface)
must also be specified.
The final
local:
example above is now the default for most tools.
This connection is an automatically authenticated local host connection
on all platforms that support Unix domain sockets. No password is required
and authentication is automatic. This is also the most efficient (lowest
overhead) communication channel available.
The difference between
unix:
and
local:
is that the former is a strict Unix domain socket specification (connection
fails if it cannot connect that way),
whereas the latter has a more forgiving fallback to using
localhost
(i.e. a regular Inet socket connection is used when Unix domain socket
connections are unavailable).
Note that most uses of these environment variables are optimized to
check the environment only the first time the variable might be used.
As the environment usually is not checked again, the only safe
strategy is to ensure all PCP-related environment variables are
set before the first call into any of the PCP libraries.
Each file in the resulting list is assumed to
contain
definitions of derived metrics as per the syntax described in
pmLoadDerivedConfig(3),
and these are loaded in order.
Derived metrics may be used to extend the available metrics with
new (derived) metrics using simple arithmetic expressions.
If
PCP_DERIVED_CONFIG
is set, the derived metric definitions are processed automatically
as each new source of performance metrics is established (i.e. each
time a
pmNewContext(3)
is called) or when requests are made against the PMNS.
Any component in the
$PCP_DERIVED_CONFIG
list or the expanded list of files that is not a file, or is not a directory
or is not accessible (due to permissions or a bad symbolic link) will
be silently ignored.
If
PCP_STDERR
is set to the literal value
DISPLAY
then all messages will be displayed in a dialog.
This is used for any tools launched from the a Desktop environment.
If
PCP_STDERR
is set to any other value, the value is assumed to
be a filename, and all messages will be written there.
The check on
pmcd's
readiness will wait up to
PMCD_WAIT_TIMEOUT
seconds.
If
pmcd
has a long startup time (such as on a very large
system), then
PMCD_WAIT_TIMEOUT
can be set to provide a maximum wait longer than the default 60 seconds.
The environment variable
PCP_COUNTER_WRAP
may be set to indicate that all such cases of a decreasing ``counter''
should be treated
as a counter overflow, and hence the values are assumed to have
wrapped once in the interval between consecutive samples.
This ``wrapping'' behavior was the default in earlier PCP versions, but
by default has been disabled in PCP release from version 1.3 on.
The following environment variables are relevant to installations
in which
pmlogger(1),
the PCP archive logger, is used.
If you have the PCP product installed, then the following
environment variables are relevant to the Performance Metrics
Domain Agents (PMDAs).
The previous behaviour was that
if this variable was set, then a context established with the
type
of
PM_CONTEXT_LOCAL
will have access to the ``proc'' PMDA to retrieve performance metrics
about individual processes.
The previous behaviour was that
if this variable was set, then a context established with the
type
of
PM_CONTEXT_LOCAL
will have access to the ``sample'' PMDA if this optional PMDA has
been installed locally.
In addition, if the PCP product is installed the following
files and directories are relevant.
If the PCP GUI package is installed, then the
following entries are also relevant:
If the secure sockets extensions have been enabled, then the
following references are also relevant:
Also refer to the books
Performance Co-Pilot User's and Administrator's Guide
and
Performance Co-Pilot Programmer's Guide
which can be found at http://www.pcp.io/
A date can be one of:
YY-MM-DD, MM/DD/YY, DD Month YYYY, or Month DD YYYY.
A time can be one of: HH:MM:SS, HH:MM. HH:MM can use either the
12 hour (via an am or pm suffix) or 24 hour convention.
A day of the
week can be a spelled out day of the week, optionally preceded by an
ordinal number such as second tuesday. A zone is a time zone value as
specified by the
tzselect(1)
command. A relative time can be a time
unit that is: preceded by a cardinal number such as 1 year or 2 months,
preceded by one of the time words this or last, or succeeded by the time word ago.
A relative time can also be one of the time words: yesterday, today, tomorrow, now.
Examples of datetime strings are:
1996-03-04 13:07:47 EST Mon,
1996-03-05 14:07:47 EST -1hour,
Mon Mar 4 13:07:47 1996,
Mar 4 1996,
Mar 4,
Mar,
13:07:50
or
13:08.
PERFORMANCE METRICS - NAMES AND IDENTIFIERS
The number of performance metric names supported by PCP on most
platforms ranges from many hundreds to several thousand.
The PCP libraries and applications use an internal
identification scheme that unambiguously associates a single
integer with each known performance metric.
This integer is known as the Performance Metric Identifier, or PMID.
Although not a requirement,
PMIDs tend to have global consistency across
all systems, so a particular performance metric usually has the same
PMID.
PERFORMANCE METRIC SPECIFICATIONS
In configuration files and (to a lesser extent) command line options,
metric specifications adhere to the following syntax rules.
SECURE PMCD CONNECTIONS
Since PCP version 3.6.11, a monitor can explicitly request
a secure connection to a collector host running
pmcd(1)
or
pmproxy(1)
using the PM_CTXFLAG_SECURE context flag.
If the PCP Collector host supports this feature - refer to the
pmcd.feature.secure metric for confirmation of this - a TLS/SSL
(Transport Layer Security or Secure Sockets Layer) connection
can be established which uses public key cryptography and related
techniques.
These features aim to prevent eavesdropping and data tampering
from a malicious third party, as well as providing server-side
authentication (confident identification of a server by a client)
which can be used to guard against man-in-the-middle attacks.
PMCD HOST SPECIFICATION
In the absence of an explicit host name specification, most tools
will default to the local host in live update mode.
In PCP releases since 3.8.4 onward, this results in an efficient
local protocol being selected - typically a Unix domain socket.
If this option is used (which can also be explicitly requested
via the
unix:
host specification described below), it is important to note that all
connections will be automatically authenticated. In other words, the
credentials of the user invoking a client tool will automatically be
made available to
pmcd(1)
and all of its PMDAs, on the users behalf, such that results can be
customized to the privilege levels of individual users.
$ pcp -h nas1.servers.com:44321,4321 [at] firewall.servers.com:44322
$ pcp -h nas1.servers.com:44321 [at] firewall.servers.com:44322
$ pcp -h nas1.servers.com:44321 [at] firewall.servers.com
$ pcp -h nas1.servers.com [at] firewall.servers.com
$ pcp -h nas1.servers.com:44321
$ pcp -h [fe80::2ad2:44ff:fe88:e4f1%p2p1]
$ pcp -h 192.168.0.103
$ pcp -h pcps://app2.servers.com?container=cae8e6edc0d5
$ pcp -h pcps://nas1.servers.com:44321?username=tanya&method=gssapi
$ pcp -h pcps://nas2.servers.com@firewalls.r.us?method=plain
$ pcp -h pcp://nas3.servers.com
$ pcp -h 192.168.0.103?container=cae8e6edc0d5,method=digest-md5
$ pcp -h unix:
$ pcp -h local:
ENVIRONMENT
In addition to the PCP run-time environment and configuration variables
described in the
PCP ENVIRONMENT
section below,
the following environment variables apply to all installations.
When
pmcd(1)
is started from
$PCP_RC_DIR/pcp
then the primary instance of
pmlogger(1)
will be started if the configuration flag
pmlogger
is
chkconfig(8)
enabled and
pmcd
is running and accepting connections.
FILES
PCP ENVIRONMENT
Environment variables with the prefix
PCP_
are used to parameterize the file and directory names
used by PCP.
On each installation, the file
/etc/pcp.conf
contains the local values for these variables.
The
$PCP_CONF
variable may be used to specify an alternative
configuration file,
as described in
pcp.conf(5).
SEE ALSO
pmcd(1),
pmie(1),
pmie_daily(1),
pminfo(1),
pmlc(1),
pmlogger(1),
pmlogger_daily(1),
pmrep(1),
pmstat(1),
pmval(1),
pcp(1),
pcp.conf(5),
pcp.env(5),
pmns(5)
and
chkconfig(8).
pmchart(1),
pmtime(1),
and
pmdumptext(1).
http://www.pcp.io/documentation.html
http://www.mozilla.org/projects/security/pki/nss/#documentation
http://www.mozilla.org/projects/security/pki/nss/tools/certutil.html