8. Customizing and Extending PCP Services

Performance Co-Pilot (PCP) has been developed to be fully extensible. The following sections summarize the various facilities provided to allow you to extend and customize PCP for your site:

Section 8.1, “PMDA Customization”, describes the procedure for customizing the summary PMDA to export derived metrics formed by aggregation of base PCP metrics from one or more collector hosts.

Section 8.2, “PCP Tool Customization”, describes the various options available for customizing and extending the basic PCP tools.

Section 8.3, “PMNS Management”, covers the concepts and tools provided for updating the PMNS (Performance Metrics Name Space).

Section 8.4, “PMDA Development”, details where to find further information to assist in the development of new PMDAs to extend the range of performance metrics available through the PCP infrastructure.

Section 8.5, “PCP Tool Development”, outlines how new tools may be developed to process performance data from the PCP infrastructure.

8.1. PMDA Customization

The generic procedures for installing and activating the optional PMDAs have been described in Section 2.3, “Managing Optional PMDAs”. In some cases, these procedures prompt the user for information based upon the local system or network configuration, application deployment, or processing profile to customize the PMDA and hence the performance metrics it exports.

The summary PMDA is a special case that warrants further discussion.

8.1.1. Customizing the Summary PMDA

The summary PMDA exports performance metrics derived from performance metrics made available by other PMDAs. It is described completely in the pmdasummary(1) man page.

The summary PMDA consists of two processes:

  1. pmie process

    Periodically samples the base metrics and compute values for the derived metrics. This dedicated instance of the PCP pmie inference engine is launched with special command line arguments by the main process. See Section 5.1, “Introduction to pmie”, for a complete discussion of the pmie feature set.

  2. main process

    Reads and buffers the values computed by the pmie process and makes them available to the Performance Metrics Collection Daemon (PMCD).

All of the metrics exported by the summary PMDA have a singular instance and the values are instantaneous; the exported value is the correct value as of the last time the corresponding expression was evaluated by the pmie process.

The summary PMDA resides in the ${PCP_PMDAS_DIR}/summary directory and may be installed with a default configuration by following the steps described in Section 2.3.1, “PMDA Installation on a PCP Collector Host”.

Alternatively, you may customize the summary PMDA to export your own derived performance metrics by following the steps in Procedure 8.1. Customizing the Summary PMDA:

Procedure 8.1. Customizing the Summary PMDA

  1. Check that the symbolic constant SYSSUMMARY is defined in the ${PCP_VAR_DIR}/pmns/stdpmid file. If it is not, perform the postinstall update of this file, as superuser:

    cd ${PCP_VAR_DIR}/pmns ./Make.stdpmid

  2. Choose Performance Metric Name Space (PMNS) names for the new metrics. These must begin with summary and follow the rules described in the pmns(5) man page. For example, you might use summary.fs.cache_write and summary.fs.cache_hit.

  3. Edit the pmns file in the ${PCP_PMDAS_DIR}/summary directory to add the new metric names in the format described in the pmns(5) man page. You must choose a unique performance metric identifier (PMID) for each metric. In the pmns file, these appear as SYSSUMMARY:0:x. The value of x is arbitrary in the range 0 to 1023 and unique in this file. Refer to Section 8.3, “PMNS Management”, for a further explanation of the rules governing PMNS updates.

    For example:

    summary {
          cpu
          disk
          netif
          fs              /*new*/
}
summary.fs {
          cache_write     SYSSUMMARY:0:10
          cache_hit       SYSSUMMARY:0:11
}

  4. Use the local test PMNS root and validate that the PMNS changes are correct.

    For example, enter this command:

    pminfo -n root -m summary.fs

    You see output similar to the following:

    summary.fs.cache_write PMID: 27.0.10
summary.fs.cache_hit PMID: 27.0.11

  5. Edit the ${PCP_PMDAS_DIR}/summary/expr.pmie file to add new pmie expressions. If the name to the left of the assignment operator (=) is one of the PMNS names, then the pmie expression to the right will be evaluated and returned by the summary PMDA. The expression must return a numeric value. Additional description of the pmie expression syntax may be found in Section 5.3, “Specification Language for pmie”.

    For example, consider this expression:

    // filesystem buffer cache hit percentages
prefix = "kernel.all.io";          // macro, not exported
summary.fs.cache_write =
                100 - 100 * $prefix.bwrite / $prefix.lwrite;
summary.fs.cache_hit =
                100 - 100 * $prefix.bread / $prefix.lread;

  6. Run pmie in debug mode to verify that the expressions are being evaluated correctly, and the values make sense.

    For example, enter this command:

    pmie -t 2sec -v expr.pmie

    You see output similar to the following:

    summary.fs.cache_write:      ?
summary.fs.cache_hit:      ?
summary.fs.cache_write:  45.83
summary.fs.cache_hit:   83.2
summary.fs.cache_write:  39.22
summary.fs.cache_hit:  84.51

  7. Install the new PMDA.

    From the ${PCP_PMDAS_DIR}/summary directory, use this command:

    ./Install

    You see the following output:

    Interval between summary expression evaluation (seconds)? [10] 10
Updating the Performance Metrics Name Space...
Installing pmchart view(s) ...
Terminate PMDA if already installed ...
Installing files ..
Updating the PMCD control file, and notifying PMCD ...
Wait 15 seconds for the agent to initialize ...
Check summary metrics have appeared ... 8 metrics and 8 values

  8. Check the metrics.

    For example, enter this command:

    pmval -t 5sec -s 8 summary.fs.cache_write

    You see a response similar to the following:

    metric:    summary.fs.cache_write
host:      localhost
semantics: instantaneous value
units:     none
samples:   8
interval:  5.00 sec
63.60132158590308
62.71878646441073
62.71878646441073
58.73968492123031
58.73968492123031
65.33822758259046
65.33822758259046
72.6099706744868

    Note that the values are being sampled here by pmval every 5 seconds, but pmie is passing only new values to the summary PMDA every 10 seconds. Both rates could be changed to suit the dynamics of your new metrics.

  9. You may now create pmchart views, pmie rules, and pmlogger configurations to monitor and archive your new performance metrics.

8.2. PCP Tool Customization

Performance Co-Pilot (PCP) has been designed and implemented with a philosophy that embraces the notion of toolkits and encourages extensibility.

In most cases, the PCP tools provide orthogonal services, based on external configuration files. It is the creation of new and modified configuration files that enables PCP users to customize tools quickly and meet the needs of the local environment, in many cases allowing personal preferences to be established for individual users on the same PCP monitor system.

The material in this section is intended to act as a checklist of pointers to detailed documentation found elsewhere in this guide, in the man pages, and in the files that are made available as part of the PCP installation.

8.2.1. Archive Logging Customization

The PCP archive logger is presented in Chapter 6, Archive Logging, and documented in the pmlogger(1) man page.

The following global files and directories influence the behavior of pmlogger:

${PCP_SYSCONF_DIR}/pmlogger

Enable/disable state for the primary logger facility using this command:

chkconfig pmlogger on

${PCP_SYSCONF_DIR}/pmlogger/config.default

The default pmlogger configuration file that is used for the primary logger when this facility is enabled.

${PCP_VAR_DIR}/config/pmlogconf/tools

Every PCP tool with a fixed group of performance metrics contributes a pmlogconf configuration file that includes each of the performance metrics used in the tool, for example, ${PCP_VAR_DIR}/config/pmlogconf/pmstat for pmstat.

${PCP_PMLOGGERCONTROL_PATH} or ${PCP_PMLOGGERCONTROL_PATH}.d files

Defines which PCP collector hosts require pmlogger to be launched on the local host, where the configuration file comes from, where the archive log files should be created, and pmlogger startup options.

These control files support the starting and stopping of multiple pmlogger instances that monitor local or remote hosts.

/etc/cron.d/pcp-pmlogger or ${PCP_VAR_DIR}/config/pmlogger/crontab

Default crontab entries that may be merged with the crontab entries for the pcp user to schedule the periodic execution of the archive log management scripts, for example, pmlogger_daily.

${PCP_LOG_DIR}/pmlogger/somehost

The default behavior of the archive log management scripts create archive log files for the host somehost in this directory.

${PCP_LOG_DIR}/pmlogger/somehost/Latest

A PCP archive folio for the most recent archive for the host somehost. This folio is created and maintained by the cron-driven periodic archive log management scripts, for example, pmlogger_check. Archive folios may be processed with the pmafm tool.

8.2.2. ⁠Inference Engine Customization

The PCP inference engine is presented in Chapter 5, Performance Metrics Inference Engine, and documented in the pmie(1) man page.

The following global files and directories influence the behavior of pmie:

${PCP_SYSCONF_DIR}/pmie

Controls the pmie daemon facility. Enable using this command:

chkconfig pmie on

${PCP_SYSCONF_DIR}/pmie/config.default

The pmie configuration file that is used for monitoring the local host when the pmie daemon facility is enabled in the default configuration. This file is created using pmieconf the first time the daemon facility is activated.

${PCP_PMIECONTROL_PATH} and ${PCP_PMIECONTROL_PATH}.d files

Defines which PCP collector hosts require a daemon pmie to be monitoring from the local host, where the configuration files comes from, where the pmie log file should be created, and pmie startup options.

These control files support the starting and stopping of multiple pmie instances that are each monitoring one or more hosts.

${PCP_VAR_DIR}/config/pmieconf/*/*

Each pmieconf rule definition can be found below one of these subdirectories.

/etc/cron.d/pcp-pmie or ${PCP_VAR_DIR}/config/pmie/crontab

Default crontab entries that may be merged with the crontab entries for the pcp user to schedule the periodic execution of the pmie_check and pmie_daily scripts, for verifying that pmie instances are running and logs rotated.

${PCP_LOG_DIR}/pmie/somehost

The default behavior of the ${PCP_RC_DIR}/pmie startup scripts create pmie log files for the host somehost in this directory.

pmie_check and pmie_daily

These commands are similar to the pmlogger support scripts, pmlogger_check and pmlogger_daily.

${PCP_TMP_DIR}/pmie

The statistics that pmie gathers are maintained in binary data structure files. These files can be found in the ${PCP_TMP_DIR}/pmie directory.

pmcd.pmie metrics

The PMCD PMDA exports information about executing pmie processes and their progress in terms of rule evaluations and action execution rates.

If pmie is running on a system with a PCP collector deployment, the pmcd PMDA exports these metrics via the pmcd.pmie group of metrics.

8.3. PMNS Management

This section describes the syntax, semantics, and processing framework for the external specification of a Performance Metrics Name Space (PMNS) as it might be loaded by the PMAPI routine pmLoadNameSpace; see the pmLoadNameSpace(3) man page. This is usually done only by pmcd, except in rare circumstances such as Section 8.1.1, “Customizing the Summary PMDA”.

The PMNS specification is a simple text source file that can be edited easily. For reasons of efficiency, a binary format is also supported; the utility pmnscomp translates the ASCII source format into binary format; see the pmnscomp(1) man page.

8.3.1. PMNS Processing Framework

The PMNS specification is initially passed through pmcpp(1). This means the following facilities may be used in the specification:

  • C-style comments

  • #include directives

  • #define directives and macro substitution

  • Conditional processing with #ifdef, #ifndef, #endif, and #undef

When pmcpp(1) is executed, the standard include directories are the current directory and ${PCP_VAR_DIR}/pmns, where some standard macros and default specifications may be found.

8.3.2. PMNS Syntax

Every PMNS is tree structured. The paths to the leaf nodes are the performance metric names. The general syntax for a non-leaf node in PMNS is as follows:

pathname {
        name    [pmid]
        ...
}

Here pathname is the full pathname from the root of the PMNS to this non-leaf node, with each component in the path separated by a period. The root node for the PMNS has the special name root, but the prefix string root. must be omitted from all other pathnames.

For example, refer to the PMNS shown in Figure 8.1. Small Performance Metrics Name Space (PMNS). The correct pathname for the rightmost non-leaf node is cpu.utilization, not root.cpu.utilization.

../_images/pmns-small2.svg

Figure 8.1. Small Performance Metrics Name Space (PMNS)

Each component in the pathname must begin with an alphabetic character and be followed by zero or more alphanumeric characters or the underscore (_) character. For alphabetic characters in a component, uppercase and lowercase are significant.

Non-leaf nodes in the PMNS may be defined in any order desired. The descendent nodes are defined by the set of names, relative to the pathname of their parent non-leaf node. For descendent nodes, leaf nodes have a pmid specification, but non-leaf nodes do not.

The syntax for the pmid specification was chosen to help manage the allocation of Performance Metric IDs (PMIDs) across disjoint and autonomous domains of administration and implementation. Each pmid consists of three integers separated by colons, for example, 14:27:11. This is intended to mirror the implementation hierarchy of performance metrics. The first integer identifies the domain in which the performance metric lies. Within a domain, related metrics are often grouped into clusters. The second integer identifies the cluster, and the third integer, the metric within the cluster.

The PMNS specification for Figure 8.1. Small Performance Metrics Name Space (PMNS) is shown in Example 8.1. PMNS Specification:

Example 8.1. PMNS Specification

/*
* PMNS Specification
*/
#define KERNEL 1
root {
       network
       cpu
}
#define NETWORK 26
network {
       interrupts    KERNEL:NETWORK:1
       packets
}
network.packets {
       in     KERNEL:NETWORK:35
       out    KERNEL:NETWORK:36
}
#define CPU 10
cpu {
       syscalls    KERNEL:CPU:10
       utilization
}
#define USER 20
#define SYSTEM 21
#define IDLE 22
cpu.utilization {
       user    KERNEL:CPU:USER
       sys     KERNEL:CPU:SYSTEM
       idle    KERNEL:CPU:IDLE
}

For complete documentation of the PMNS and associated utilities, see the pmns(5), pmnsadd(1), pmnsdel(1) and pmnsmerge(1) man pages.

8.4. PMDA Development

Performance Co-Pilot (PCP) is designed to be extensible at the collector site.

Application developers are encouraged to create new PMDAs to export performance metrics from the applications and service layers that are particularly relevant to a specific site, application suite, or processing environment.

These PMDAs use the routines of the libpcp_pmda library, which is discussed in detail in the Performance Co-Pilot Programmer’s Guide.

8.5. PCP Tool Development

Performance Co-Pilot (PCP) is designed to be extensible at the monitor site.

Application developers are encouraged to create new PCP client applications to monitor or display performance metrics in a manner that is particularly relevant to a specific site, application suite, or processing environment.

Client applications use the routines of the PMAPI (performance metrics application programming interface) described in the Performance Co-Pilot Programmer’s Guide. At the time of writing, native PMAPI interfaces are available for the C, C++ and Python languages.