2. Writing A PMDA

This chapter constitutes a programmer’s guide to writing a Performance Metrics Domain Agent (PMDA) for Performance Co-Pilot (PCP).

The presentation assumes the developer is using the standard PCP libpcp_pmda library, as documented in the PMDA(3) and associated man pages.

2.1. Implementing a PMDA

The job of a PMDA is to gather performance data and report them to the Performance Metrics Collection Daemon (PMCD) in response to requests from PCP monitoring tools routed to the PMDA via PMCD.

An important requirement for any PMDA is that it have low latency response to requests from PMCD. Either the PMDA must use a quick access method and a single thread of control, or it must have asynchronous refresh and two threads of control: one for communicating with PMCD, the other for updating the performance data.

The PMDA is typically acting as a gateway between the target domain (that is, the performance instrumentation in an application program or service) and the PCP framework. The PMDA may extract the information using one of a number of possible export options that include a shared memory segment or mmap file; a sequential log file (where the PMDA parses the tail of the log file to extract the information); a snapshot file (the PMDA rereads the file as required); or application-specific communication services (IPC).


The choice of export methodology is typically determined by the source of the instrumentation (the target domain) rather than by the PMDA.

Procedure 2.1. Creating a PMDA describes the suggested steps for designing and implementing a PMDA:

Procedure 2.1. Creating a PMDA

  1. Determine how to extract the metrics from the target domain.

  2. Select an appropriate architecture for the PMDA (daemon or DSO, IPC, pthreads or single threaded).

  3. Define the metrics and instances that the PMDA will support.

  4. Implement the functionality to extract the metric values.

  5. Assign Performance Metric Identifiers (PMIDs) for the metrics, along with names for the metrics in the Performance Metrics Name Space (PMNS). These concepts will be further expanded in Section 2.3, “Domains, Metrics, Instances and Labels

  6. Specify the help file and control data structures for metrics and instances that are required by the standard PMDA implementation library functions.

  7. Write code to supply the metrics and associated information to PMCD.

  8. Implement any PMDA-specific callbacks, and PMDA initialization functions.

  9. Exercise and test the PMDA with the purpose-built PMDA debugger; see the dbpmda(1) man page.

  10. Install and connect the PMDA to a running PMCD process; see the pmcd(1) man page.

  11. Where appropriate, define pmie rule templates suitable for alerting or notification systems. For more information, see the pmie(1) and pmieconf(1) man pages.

  12. Where appropriate, define pmlogger configuration templates suitable for creating PCP archives containing the new metrics. For more information, see the pmlogconf(1) and pmlogger(1) man pages.

2.2. ⁠PMDA Architecture

This section discusses the two methods of connecting a PMDA to a PMCD process:

  1. As a separate process using some interprocess communication (IPC) protocol.

  2. As a dynamically attached library (that is, a dynamic shared object or DSO).

2.2.1. Overview

All PMDAs are launched and controlled by the PMCD process on the local host. PMCD receives requests from the monitoring tools and forwards them to the PMDAs. Responses, when required, are returned through PMCD to the clients. The requests fall into a small number of categories, and the PMDA must handle each request type. For a DSO PMDA, each request type corresponds to a method in the agent. For a daemon PMDA, each request translates to a message or protocol data unit (PDU) that may be sent to a PMDA from PMCD.

For a daemon PMDA, the following request PDUs must be supported:


Request for metric values (see the pmFetch(3) man page.)


A list of instances required for the corresponding metrics in subsequent fetches (see the pmAddProfile(3) man page).


Request for a particular instance domain for instance descriptions (see the pmGetInDom(3) man page).


Request for metadata describing metrics (see the pmLookupDesc(3) man page).


Request for metric help text (see the pmLookupText(3) man page).


Values to store into metrics (see the pmStore(3) man page).

The following request PDUs may optionally be supported:


Request for metric names, given one or more identifiers (see the pmLookupName(3) man page.)


A list of immediate descendent nodes of a given namespace node (see the pmGetChildren(3) man page).


Request for a particular sub-tree of a given namespace node (see the pmTraversePMNS(3) man page).


Perform a reverse name lookup, mapping a metric identifier to a name (see the pmNameID(3) man page).


Handle connection attributes (key/value pairs), such as client credentials and other authentication information (see the __pmParseHostAttrsSpec(3) man page).


Request for metric labels (see the pmLookupLabels(3) man page).

Each PMDA is associated with a unique domain number that is encoded in the domain field of metric and instance identifiers, and PMCD uses the domain number to determine which PMDA can handle the components of any given client request.

2.2.2. DSO PMDA

Each PMDA is required to implement a function that handles each of the request types. By implementing these functions as library functions, a PMDA can be implemented as a dynamically shared object (DSO) and attached by PMCD at run time with a platform-specific call, such as dlopen; see the dlopen(3) man page. This eliminates the need for an IPC layer (typically a pipe) between each PMDA and PMCD, because each request becomes a function call rather than a message exchange. The required library functions are detailed in Section 2.5, “PMDA Interface”.

A PMDA that interacts with PMCD in this fashion must abide by a formal initialization protocol so that PMCD can discover the location of the library functions that are subsequently called with function pointers. When a DSO PMDA is installed, the PMCD configuration file, ${PCP_PMCDCONF_PATH}, is updated to reflect the domain and name of the PMDA, the location of the shared object, and the name of the initialization function. The initialization sequence is discussed in Section 2.6, “Initializing a PMDA”.

As superuser, install the simple PMDA as a DSO, as shown in Example 2.1. Simple PMDA as a DSO, and observe the changes in the PMCD configuration file. The output may differ slightly depending on the operating system you are using, any other PMDAs you have installed or any PMCD access controls you have in place.

Example 2.1. Simple PMDA as a DSO

# Performance Metrics Domain Specifications
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
root    1       pipe    binary          /var/lib/pcp/pmdas/root/pmdaroot
pmcd    2       dso     pmcd_init       ${PCP_PMDAS_DIR}/pmcd/pmda_pmcd.so
proc    3       pipe    binary          ${PCP_PMDAS_DIR}/linux/pmda_proc.so -d 3
linux   60      dso     linux_init      ${PCP_PMDAS_DIR}/linux/pmda_linux.so
mmv     70      dso     mmv_init        /var/lib/pcp/pmdas/mmv/pmda_mmv.so
simple  254     dso     simple_init     ${PCP_PMDAS_DIR}/simple/pmda_simple.so

As can be seen from the contents of ${PCP_PMCDCONF_PATH}, the DSO version of the simple PMDA is in a library named pmda_simple.so and has an initialization function called simple_init. The domain of the simple PMDA is 254, as shown in the column headed Id.


For some platforms the DSO file name will not be pmda_simple.so. On Mac OS X it is pmda_simple.dylib and on Windows it is pmda_simple.dll.

2.2.3. Daemon PMDA

A DSO PMDA provides the most efficient communication between the PMDA and PMCD. This approach has some disadvantages resulting from the DSO PMDA being the same process as PMCD:

  • An error or bug that causes a DSO PMDA to exit also causes PMCD to exit, which affects all connected client tools.

  • There is only one thread of control in PMCD; as a result, a computationally expensive PMDA, or worse, a PMDA that blocks for I/O, adversely affects the performance of PMCD.

  • PMCD runs as the “pcp” user; so all DSO PMDAs must also run as this user.

  • A memory leak in a DSO PMDA also causes a memory leak for PMCD.

Consequently, many PMDAs are implemented as a daemon process.

The libpcp_pmda library is designed to allow simple implementation of a PMDA that runs as a separate process. The library functions provide a message passing layer acting as a generic wrapper that accepts PDUs, makes library calls using the standard DSO PMDA interface, and sends PDUs. Therefore, you can implement a PMDA as a DSO and then install it as either a daemon or a DSO, depending on the presence or absence of the generic wrapper.

The PMCD process launches a daemon PMDA with fork and execv (or CreateProcess on Windows). You can easily connect a pipe to the PMDA using standard input and output. The PMCD process may also connect to a daemon PMDA using IPv4 or IPv6 TCP/IP, or UNIX domain sockets if the platform supports that; see the tcp(7), ip(7), ipv6(7) or unix(7) man pages.

As superuser, install the simple PMDA as a daemon process as shown in Example 2.2. Simple PMDA as a Daemon. Again, the output may differ due to operating system differences, other PMDAs already installed, or access control sections in the PMCD configuration file.

Example 2.2. Simple PMDA as a Daemon

The specification for the simple PMDA now states the connection type of pipe to PMCD and the executable image for the PMDA is ${PCP_PMDAS_DIR}/simple/pmdasimple, using domain number 253.

# cd ${PCP_PMDAS_DIR}/simple
# ./Install
Install simple as a daemon or dso agent? [daemon] daemon
PMCD should communicate with the daemon via pipe or socket? [pipe] pipe
# Performance Metrics Domain Specifications
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
root    1       pipe    binary          /var/lib/pcp/pmdas/root/pmdaroot
pmcd    2       dso     pmcd_init       ${PCP_PMDAS_DIR}/pmcd/pmda_pmcd.so
proc    3       pipe    binary          ${PCP_PMDAS_DIR}/linux/pmda_proc.so -d 3
linux   60      dso     linux_init      ${PCP_PMDAS_DIR}/linux/pmda_linux.so
mmv     70      dso     mmv_init        /var/lib/pcp/pmdas/mmv/pmda_mmv.so
simple  253     pipe    binary          ${PCP_PMDAS_DIR}/simple/pmdasimple -d 253

2.2.4. Caching PMDA

When either the cost or latency associated with collecting performance metrics is high, the PMDA implementer may choose to trade off the currency of the performance data to reduce the PMDA resource demands or the fetch latency time.

One scheme for doing this is called a caching PMDA, which periodically instantiates values for the performance metrics and responds to each request from PMCD with the most recently instantiated (or cached) values, as opposed to instantiating current values on demand when the PMCD asks for them.

The Cisco PMDA is an example of a caching PMDA. For additional information, see the contents of the ${PCP_PMDAS_DIR}/cisco directory and the pmdacisco(1) man page.

2.3. Domains, Metrics, Instances and Labels

This section defines metrics and instances, discusses how they should be designed for a particular target domain, and shows how to implement support for them.

The examples in this section are drawn from the trivial and simple PMDAs. Refer to the ${PCP_PMDAS_DIR}/trivial and ${PCP_PMDAS_DIR}/simple directories, respectively, where both binaries and source code are available.

2.3.1. Overview

Domains are autonomous performance areas, such as the operating system or a layered service or a particular application. Metrics are raw performance data for a domain, and typically quantify activity levels, resource utilization or quality of service. Instances are sets of related metrics, as for multiple processors, or multiple service classes, or multiple transaction types.

PCP employs the following simple and uniform data model to accommodate the demands of performance metrics drawn from multiple domains:

  • Each metric has an identifier that is unique across all metrics for all PMDAs on a particular host.

  • Externally, metrics are assigned names for user convenience–typically there is a 1:1 relationship between a metric name and a metric identifier.

  • The PMDA implementation determines if a particular metric has a singular value or a set of (zero or more) values. For instance, the metric hinv.ndisk counts the number of disks and has only one value on a host, whereas the metric disk.dev.total counts disk I/O operations and has one value for each disk on the host.

  • If a metric has a set of values, then members of the set are differentiated by instances. The set of instances associated with a metric is an instance domain. For example, the set of metrics disk.dev.total is defined over an instance domain that has one member per disk spindle.

The selection of metrics and instances is an important design decision for a PMDA implementer. The metrics and instances for a target domain should have the following qualities:

  • Obvious to a user

  • Consistent across the domain

  • Accurately representative of the operational and functional aspects of the domain

For each metric, you should also consider these questions:

  • How useful is this value?

  • What units give a good sense of scale?

  • What name gives a good description of the metric’s meaning?

  • Can this metric be combined with another to convey the same useful information?

As with all programming tasks, expect to refine the choice of metrics and instances several times during the development of the PMDA.

2.3.2. Domains

Each PMDA must be uniquely identified by PMCD so that requests from clients can be efficiently routed to the appropriate PMDA. The unique identifier, the PMDA’s domain, is encoded within the metrics and instance domain identifiers so that they are associated with the correct PMDA, and so that they are unique, regardless of the number of PMDAs that are connected to the PMCD process.

The default domain number for each PMDA is defined in ${PCP_VAR_DIR}/pmns/stdpmid. This file is a simple table of PMDA names and their corresponding domain number. However, a PMDA does not have to use this domain number–the file is only a guide to help avoid domain number clashes when PMDAs are installed and activated.

The domain number a PMDA uses is passed to the PMDA by PMCD when the PMDA is launched. Therefore, any data structures that require the PMDA’s domain number must be set up when the PMDA is initialized, rather than declared statically. The protocol for PMDA initialization provides a standard way for a PMDA to implement this run-time initialization.


Although uniqueness of the domain number in the ${PCP_PMCDCONF_PATH} control file used by PMCD is all that is required for successful starting of PMCD and the associated PMDAs, the developer of a new PMDA is encouraged to add the default domain number for each new PMDA to the ${PCP_VAR_DIR}/pmns/stdpmid.local file and then to run the Make.stdpmid script in ${PCP_VAR_DIR}/pmns to recreate ${PCP_VAR_DIR}/pmns/stdpmid; this file acts as a repository for documenting the known default domain numbers.

2.3.3. Metrics

A PMDA provides support for a collection of metrics. In addition to the obvious performance metrics, and the measures of time, activity and resource utilization, the metrics should also describe how the target domain has been configured, as this can greatly affect the correct interpretation of the observed performance. For example, metrics that describe network transfer rates should also describe the number and type of network interfaces connected to the host (hinv.ninterface, network.interface.speed, network.interface.duplex, and so on)

In addition, the metrics should describe how the PMDA has been configured. For example, if the PMDA was periodically probing a system to measure quality of service, there should be metrics for the delay between probes, the number of probes attempted, plus probe success and failure counters. It may also be appropriate to allow values to be stored (see the pmstore(1) man page) into the delay metric, so that the delay used by the PMDA can be altered dynamically. Data Structures

Each metric must be described in a pmDesc structure; see the pmLookupDesc(3) man page:

typedef struct {
    pmID        pmid;           /* unique identifier */
    int         type;           /* base data type */
    pmInDom     indom;          /* instance domain */
    int         sem;            /* semantics of value */
    pmUnits     units;          /* dimension and units */
} pmDesc;

This structure contains the following fields:


A unique identifier, Performance Metric Identifier (PMID), that differentiates this metric from other metrics across the union of all PMDAs


A data type indicator showing whether the format is an integer (32 or 64 bit, signed or unsigned); float; double; string; or arbitrary aggregate of binary data


An instance domain identifier that links this metric to an instance domain


An encoding of the value’s semantics (counter, instantaneous, or discrete)


A description of the value’s units based on dimension and scale in the three orthogonal dimensions of space, time, and count (or events)


This information can be observed for metrics from any active PMDA using pminfo command line options, for example:

$ pminfo -d -m network.interface.out.drops

network.interface.out.drops PMID: 60.3.11
    Data Type: 64-bit unsigned int  InDom: 60.3 0xf000003
    Semantics: counter  Units: count

Symbolic constants of the form PM_TYPE*, PM_SEM_*, PM_SPACE_*, PM_TIME_*, and PM_COUNT_* are defined in the <pcp/pmapi.h> header file. You may use them to initialize the elements of a pmDesc structure. The pmID type is an unsigned integer that can be safely cast to a __pmID_int structure, which contains fields defining the metric’s (PMDA’s) domain, cluster, and item number as shown in Example 2.3. __pmID_int Structure:

Example 2.3. __pmID_int Structure

typedef struct {
        int             flag:1;
        unsigned int    domain:9;
        unsigned int    cluster:12;
        unsigned int    item:10;
} __pmID_int;

For additional information, see the <pcp/libpcp.h> file.

The flag field should be ignored. The domain number should be set at run time when the PMDA is initialized. The PMDA_PMID macro defined in <pcp/pmapi.h> can be used to set the cluster and item fields at compile time, as these should always be known and fixed for a particular metric.


The three components of the PMID should correspond exactly to the three-part definition of the PMID for the corresponding metric in the PMNS described in Section 2.4.3, “Name Space”.

A table of pmdaMetric structures should be defined within the PMDA, with one structure per metric as shown in Example 2.4. pmdaMetric Structure.

Example 2.4. pmdaMetric Structure

typedef struct {
    void        *m_user;        /* for users external use */
    pmDesc      m_desc;         /* metric description */
} pmdaMetric;

This structure contains a pmDesc structure and a handle that allows PMDA-specific structures to be associated with each metric. For example, m_user could be a pointer to a global variable containing the metric value, or a pointer to a function that may be called to instantiate the metric’s value.

The trivial PMDA, shown in Example 2.5. Trivial PMDA, has only a singular metric (that is, no instance domain):

Example 2.5. Trivial PMDA

static pmdaMetric metrictab[] = {
/* time */
 { NULL,
     PMDA_PMUNITS(0, 1, 0, 0, PM_TIME_SEC, 0) }, },

This single metric (trivial.time) has the following:

  • A PMID with a cluster of 0 and an item of 1. Note that this is not yet a complete PMID, the domain number which identifies the PMDA will be combined with it at runtime.

  • An unsigned 32-bit integer (PM_TYPE_U32)

  • A singular value and hence no instance domain (PM_INDOM_NULL)

  • An instantaneous semantic value (PM_SEM_INSTANT)

  • Dimension “time” and the units “seconds” Semantics

The metric’s semantics describe how PCP tools should interpret the metric’s value. The following are the possible semantic types:

  • Counter (PM_SEM_COUNTER)

  • Instantaneous value (PM_SEM_INSTANT)

  • Discrete value (PM_SEM_DISCRETE)

A counter should be a value that monotonically increases (or monotonically decreases, which is less likely) with respect to time, so that the rate of change should be used in preference to the actual value. Rate conversion is not appropriate for metrics with instantaneous values, as the value is a snapshot and there is no basis for assuming any values that might have been observed between snapshots. Discrete is similar to instantaneous; however, once observed it is presumed the value will persist for an extended period (for example, system configuration, static tuning parameters and most metrics with non-numeric values).

For a given time interval covering six consecutive timestamps, each spanning two units of time, the metric values in Example 2.6. Effect of Semantics on a Metric are exported from a PMDA (“N/A” implies no value is available):

Example 2.6. Effect of Semantics on a Metric

Timestamps:         1   3   5   7   9  11
Value:             10  30  60  80  90 N/A

The default display of the values would be as follows:

Timestamps:         1   3   5   7   9  11
Counter           N/A  10  15  10   5 N/A
Instantaneous      10  30  60  80  90 N/A
Discrete           10  30  60  80  90  90

Note that these interpretations of metric semantics are performed by the monitor tool, automatically, before displaying a value and they are not transformations that the PMDA performs.

2.3.4. Instances

Singular metrics have only one value and no associated instance domain. Some metrics contain a set of values that share a common set of semantics for a specific instance, such as one value per processor, or one value per disk spindle, and so on.


The PMDA implementation is solely responsible for choosing the instance identifiers that differentiate instances within the instance domain. The PMDA is also responsible for ensuring the uniqueness of instance identifiers in any instance domain, as described in Section, “Instance Identification”. Instance Identification

Consistent interpretation of instances and instance domains require a few simple rules to be followed by PMDA authors. The PMDA library provides a series of pmdaCache routines to assist.

  • Each internal instance identifier (numeric) must be a unique 31-bit number.

  • The external instance name (string) must be unique.

  • When the instance name contains a space, the name to the left of the first space (the short name) must also be unique.

  • Where an external instance name corresponds to some object or entity, there is an expectation that the association between the name and the object is fixed.

  • It is preferable, although not mandatory, for the association between and external instance name (string) and internal instance identifier (numeric) to be persistent. N Dimensional Data

Where the performance data can be represented as scalar values (singular metrics) or one-dimensional arrays or lists (metrics with an instance domain), the PCP framework is more than adequate. In the case of metrics with an instance domain, each array or list element is associated with an instance from the instance domain.

To represent two or more dimensional arrays, the coordinates must be one of the following:

  • Mapped onto one dimensional coordinates.

  • Enumerated into the Performance Metrics Name Space (PMNS).

For example, this 2 x 3 array of values called M can be represented as instances 1,…, 6 for a metric M:

M[1]   M[2]   M[3]
M[4]   M[5]   M[6]

Or they can be represented as instances 1, 2, 3 for metric M1 and instances 1, 2, 3 for metric M2:

M1[1]  M1[2]  M1[3]
M2[1]  M2[2]  M2[3]

The PMDA implementer must decide and consistently export this encoding from the N-dimensional instrumentation to the 1-dimensional data model of the PCP. The use of metric label metadata - arbitrary key/value pairs - allows the implementer to capture the higher dimensions of the performance data.

In certain special cases (for example, such as for a histogram), it may be appropriate to export an array of values as raw binary data (the type encoding in the descriptor is PM_TYPE_AGGREGATE). However, this requires the development of special PMAPI client tools, because the standard PCP tools have no knowledge of the structure and interpretation of the binary data. The usual issues of platform-dependence must also be kept in mind for this case - endianness, word-size, alignment and so on - the (possibly remote) special PMAPI client tools may need this information in order to decode the data successfully. Data Structures

If the PMDA is required to support instance domains, then for each instance domain the unique internal instance identifier and external instance identifier should be defined using a pmdaInstid structure as shown in Example 2.7. pmdaInstid Structure:

Example 2.7. pmdaInstid Structure

typedef struct {
    int         i_inst;         /* internal instance identifier */
    char        *i_name;        /* external instance identifier */
} pmdaInstid;

The i_inst instance identifier must be a unique integer within a particular instance domain.

The complete instance domain description is specified in a pmdaIndom structure as shown in Example 2.8. pmdaIndom Structure:

Example 2.8. pmdaIndom Structure

typedef struct {
    pmInDom     it_indom;       /* indom, filled in */
    int         it_numinst;     /* number of instances */
    pmdaInstid  *it_set;        /* instance identifiers */
} pmdaIndom;

The it_indom element contains a pmInDom that must be unique across every PMDA. The other fields of the pmdaIndom structure are the number of instances in the instance domain and a pointer to an array of instance descriptions.

Example 2.9. __pmInDom_int Structure shows that the pmInDom can be safely cast to __pmInDom_int, which specifies the PMDA’s domain and the instance number within the PMDA:

Example 2.9. __pmInDom_int Structure

typedef struct {
        int             flag:1;
        unsigned int    domain:9;   /* the administrative PMD */
        unsigned int    serial:22;  /* unique within PMD */
} __pmInDom_int;

As with metrics, the PMDA domain number is not necessarily known until run time; so the domain field must be set up when the PMDA is initialized.

For information about how an instance domain may also be associated with more than one metric, see the pmdaInit(3) man page.

The simple PMDA, shown in Example 2.10. Simple PMDA, has five metrics and two instance domains of three instances.

Example 2.10. Simple PMDA

 * list of instances
static pmdaInstid color[] = {
    { 0, “red” }, { 1, “green” }, { 2, “blue” }
static pmdaInstid       *timenow = NULL;
static unsigned int     timesize = 0;
 * list of instance domains
static pmdaIndom indomtab[] = {
#define COLOR_INDOM     0
    { COLOR_INDOM, 3, color },
#define NOW_INDOM       1
   { NOW_INDOM, 0, NULL },
 * all metrics supported in this PMDA - one table entry for each
static pmdaMetric metrictab[] = {
/* numfetch */
    { NULL,
       PMDA_PMUNITS(0, 0, 0, 0, 0, 0) }, },
/* color */
    { NULL,
        PMDA_PMUNITS(0, 0, 0, 0, 0, 0) }, },
/* time.user */
    { NULL,
        PMDA_PMUNITS(0, 1, 0, 0, PM_TIME_SEC, 0) }, },
/* time.sys */
    { NULL,
        PMDA_PMUNITS(0, 1, 0, 0, PM_TIME_SEC, 0) }, },
/* now */
    { NULL,
        PMDA_PMUNITS(0, 0, 0, 0, 0, 0) }, },

The metric simple.color is associated, via COLOR_INDOM, with the first instance domain listed in indomtab. PMDA initialization assigns the correct domain portion of the instance domain identifier in indomtab[0].it_indom and metrictab[1].m_desc.indom. This instance domain has three instances: red, green, and blue.

The metric simple.now is associated, via NOW_INDOM, with the second instance domain listed in indomtab. PMDA initialization assigns the correct domain portion of the instance domain identifier in indomtab[1].it_indom and metrictab[4].m_desc.indom. This instance domain is dynamic and initially has no instances.

All other metrics are singular, as specified by PM_INDOM_NULL.

In some cases an instance domain may vary dynamically after PMDA initialization (for example, simple.now), and this requires some refinement of the default functions and data structures of the libpcp_pmda library. Briefly, this involves providing new functions that act as wrappers for pmdaInstance and pmdaFetch while understanding the dynamics of the instance domain, and then overriding the instance and fetch methods in the pmdaInterface structure during PMDA initialization.

For the simple PMDA, the wrapper functions are simple_fetch and simple_instance, and defaults are over-ridden by the following assignments in the simple_init function:

dp->version.any.fetch = simple_fetch;
dp->version.any.instance = simple_instance;

2.3.5. Labels

Metrics and instances can be further described through the use of metadata labels, which are arbitrary name:value pairs associated with individual metrics and instances. There are several applications of this concept, but one of the most important is the ability to differentiate the components of a multi-dimensional instance name, such as the case of the mem.zoneinfo.numa_hit metric which has one value per memory zone, per NUMA node.

Consider Example 2.11. Multi-dimensional Instance Domain Labels:

Example 2.11. Multi-dimensional Instance Domain Labels

$ pminfo -l mem.zoneinfo.numa_hit

   inst [0 or "DMA::node0"] labels {"device_type":["numa_node","memory"],"indom_name":"per zone per numa_node","numa_node":0,"zone":"DMA"}
   inst [1 or "Normal::node0"] labels {"device_type":["numa_node","memory"],"indom_name":"per zone per numa_node","numa_node":0,"zone":"Normal"}
   inst [2 or "DMA::node1"] labels {"device_type":["numa_node","memory"],"indom_name":"per zone per numa_node","numa_node":1,"zone":"DMA"}
   inst [3 or "Normal::node1"] labels {"device_type":["numa_node","memory"],"indom_name":"per zone per numa_node","numa_node":1,"zone":"Normal"}


The metric labels used here individually describe the memory zone and NUMA node associated with each instance.

The PMDA implementation is only partially responsible for choosing the label identifiers that differentiate components of metrics and instances within an instance domain. Label sets for a singleton metric or individual instance of a set-valued metric are formed from a label hierarchy, which includes global labels applied to all metrics and instances from one PMAPI context.

Labels are stored and communicated within PCP using JSONB format. This format is a restricted form of JSON suitable for indexing and other operations. In JSONB form, insignificant whitespace is discarded, and the order of label names is not preserved. Within the PMCS a lexicographically sorted key space is always maintained, however. Duplicate label names are not permitted. The label with highest precedence is the only one presented. If duplicate names are presented at the same hierarchy level, only one will be preserved (exactly which one wins is arbitrary, so do not rely on this). Label Hierarchy

The set of labels associated with any singleton metric or instance is formed by merging the sets of labels at each level of a hierarchy. The lower levels of the hierarchy have highest precedence when merging overlapping (duplicate) label names:

  • Global context labels (as reported by the pmcd.labels metric) are the lowest precedence. The PMDA implementor has no influence over labels at this level of the hierarchy, and these labels are typically supplied by pmcd from /etc/pcp/labels files.

  • Domain labels, for all metrics and instances of a PMDA, are the next highest precedence.

  • Instance Domain labels, associated with an InDom, are the next highest precedence.

  • Metric cluster labels, associated with a PMID cluster, are the next highest precedence.

  • Metric item labels, associated with an individual PMID, are the next highest precedence.

  • Instance labels, associated with a metric instance identifier, have the highest precedence. Data Structures

In any PMDA that supports labels at any level of the hierarchy, each individual label (one name:value pair) requires a pmLabel structure as shown in Example 2.12. pmLabel Structure:

Example 2.12. pmLabel Structure

typedef struct {
    uint     name : 16;      /* label name offset in JSONB string */
    uint     namelen : 8;    /* length of name excluding the null */
    uint     flags : 8;      /* information about this label */
    uint     value : 16;     /* offset of the label value */
    uint     valuelen : 16;  /* length of value in bytes */
} pmLabel;

The flags field is a bitfield identifying the hierarchy level and whether this name:value pair is intrinsic (optional) or extrinsic (part of the mandatory, identifying metadata for the metric or instance). All other fields are offsets and lengths in the JSONB string from an associated pmLabelSet structure.

Zero or more labels are specified via a label set, in a pmLabelSet structure as shown in Example 2.13. pmLabelSet Structure:

Example 2.13. pmLabelSet Structure

typedef struct {
    uint     inst;          /* PM_IN_NULL or the instance ID */
    int      nlabels;       /* count of labels or error code */
    char     *json;         /* JSONB formatted labels string */
    uint     jsonlen : 16;  /* JSON string length byte count */
    uint     padding : 16;  /* zero, reserved for future use */
    pmLabel  *labels;       /* indexing into the JSON string */
} pmLabelSet;

This provides information about the set of labels associated with an entity (context, domain, indom, metric cluster, item or instance). The entity will be from any one level of the label hierarchy. If at the lowest hierarchy level (which happens to be highest precedence - instances) then the inst field will contain an actual instance identifier instead of PM_IN_NULL.

For information about how a label can be associated with each level of the hierarchy, see the pmdaLabel(3) man page.

The simple PMDA, shown in Example 2.14. Simple PMDA, associates labels at the domain, indom and instance levels of the hierarchy.

Example 2.14. Simple PMDA

static int
simple_label(int ident, int type, pmLabelSet **lpp, pmdaExt *pmda)
    int         serial;

    switch (type) {
        pmdaAddLabels(lpp, "{"role":"testing"}");
    case PM_LABEL_INDOM:
        serial = pmInDom_serial((pmInDom)ident);
        if (serial == COLOR_INDOM) {
            pmdaAddLabels(lpp, "{"indom_name":"color"}");
            pmdaAddLabels(lpp, "{"model":"RGB"}");
        if (serial == NOW_INDOM) {
            pmdaAddLabels(lpp, "{"indom_name":"time"}");
            pmdaAddLabels(lpp, "{"unitsystem":"SI"}");
    case PM_LABEL_ITEM:
        /* no labels to add for these types, fall through */
    return pmdaLabel(ident, type, lpp, pmda);

static int
simple_labelCallBack(pmInDom indom, unsigned int inst, pmLabelSet **lp)
    struct timeslice *tsp;

    if (pmInDom_serial(indom) != NOW_INDOM)
        return 0;
    if (pmdaCacheLookup(indom, inst, NULL, (void *)&tsp) != PMDA_CACHE_ACTIVE)
        return 0;
    /* SI units label, value: sec (seconds), min (minutes), hour (hours) */
    return pmdaAddLabels(lp, "{"units":"%s"}", tsp-<tm_name);

The simple_labelCallBack function is called indirectly via pmdaLabel for each instance of the NOW_INDOM. PMDA initialization ensures these functions are registered with the global PMDA interface structure for use when handling label requests, by the following assignments in the simple_init function:

dp->version.seven.label = simple_label;
pmdaSetLabelCallBack(dp, simple_labelCallBack);

2.4. Other Issues

Other issues include extracting the information, latency and threads of control, Name Space, PMDA help text, and management of evolution within a PMDA.

2.4.1. Extracting the Information

A suggested approach to writing a PMDA is to write a standalone program to extract the values from the target domain and then incorporate this program into the PMDA framework. This approach avoids concurrent debugging of two distinct problems:

  1. Extraction of the data

  2. Communication with PMCD

These are some possible ways of exporting the data from the target domain:

  • Accumulate the performance data in a public shared memory segment.

  • Write the performance data to the end of a log file.

  • Periodically rewrite a file with the most recent values for the performance data.

  • Implement a protocol that allows a third party to connect to the target application, send a request, and receive new performance data.

  • If the data is in the operating system kernel, provide a kernel interface (preferred) to export the performance data.

Most of these approaches require some further data processing by the PMDA.

2.4.2. Latency and Threads of Control

The PCP protocols expect PMDAs to return the current values for performance metrics when requested, and with short delay (low latency). For some target domains, access to the underlying instrumentation may be costly or involve unpredictable delays (for example, if the real performance data is stored on some remote host or network device). In these cases, it may be necessary to separate probing for new performance data from servicing PMCD requests.

An architecture that has been used successfully for several PMDAs is to create one or more child processes to obtain information while the main process communicates with PMCD.

At the simplest deployment of this arrangement, the two processes may execute without synchronization. Threads have also been used as a more portable multithreading mechanism; see the pthreads(7) man page.

By contrast, a complex deployment would be one in which the refreshing of the metric values must be atomic, and this may require double buffering of the data structures. It also requires coordination between parent and child processes.


Since certain data structures used by the PMDA library are not thread-aware, only one PMDA thread of control should call PMDA library functions - this would typically be the thread servicing requests from PMCD.

One caveat about this style of caching PMDA–in this (special) case it is better if the PMDA converts counts to rates based upon consecutive periodic sampling from the underlying instrumentation. By exporting precomputed rate metrics with instantaneous semantics, the PMDA prevents the PCP monitor tools from computing their own rates upon consecutive PMCD fetches (which are likely to return identical values from a caching PMDA). The finer points of metric semantics are discussed in Section, “Semantics

2.4.3. Name Space

The PMNS file defines the name space of the PMDA. It is a simple text file that is used during installation to expand the Name Space of the PMCD process. The format of this file is described by the pmns(5) man page and its hierarchical nature, syntax, and helper tools are further described in the Performance Co-Pilot User’s and Administrator’s Guide.

Client processes will not be able to access the PMDA metrics if the PMNS file is not installed as part of the PMDA installation procedure on the collector host. The installed list of metric names and their corresponding PMIDs can be found in ${PCP_VAR_DIR}/pmns/root.

Example 2.15. pmns File for the Simple PMDA shows the simple PMDA, which has five metrics:

Three metrics immediately under the simple node

Two metrics under another non-terminal node called simple.time

Example 2.15. pmns File for the Simple PMDA

simple {
    numfetch    SIMPLE:0:0
    color       SIMPLE:0:1
    now         SIMPLE:2:4
simple.time {
    user        SIMPLE:1:2
    sys         SIMPLE:1:3

Metrics that have different clusters do not have to be specified in different subtrees of the PMNS. Example 2.16. Alternate pmns File for the Simple PMDA shows an alternative PMNS for the simple PMDA:

Example 2.16. Alternate pmns File for the Simple PMDA

simple {
    numfetch    SIMPLE:0:0
    color       SIMPLE:0:1
    usertime    SIMPLE:1:2
    systime     SIMPLE:1:3

In this example, the SIMPLE macro is replaced by the domain number listed in ${PCP_VAR_DIR}/pmns/stdpmid for the corresponding PMDA during installation (for the simple PMDA, this would normally be the value 253).

If the PMDA implementer so chooses, all or a subset of the metric names and identifiers can be specified programatically. In this situation, a special asterisk syntax is used to denote those subtrees which are to be handles this way. Example 2.17. Dynamic metrics pmns File for the Simple PMDA shows this dynamic namespace syntax, for all metrics in the simple PMDA:

Example 2.17. Dynamic metrics pmns File for the Simple PMDA

simple         SIMPLE:*:*

In this example, like the one before, the SIMPLE macro is replaced by the domain number, and all (simple.*) metric namespace operations must be handled by the PMDA. This is in contrast to the static metric name model earlier, where the host-wide PMNS file is updated and used by PMCD, acting on behalf of the agent.

2.4.4. PMDA Help Text

For each metric defined within a PMDA, the PMDA developer is strongly encouraged to provide both terse and extended help text to describe the metric, and perhaps provide hints about the expected value ranges.

The help text is used to describe each metric in the visualization tools and pminfo with the -T option. The help text, such as the help text for the simple PMDA in Example 2.18. Help Text for the Simple PMDA, is specified in a specially formatted file, normally called help. This file is converted to the expected run-time format using the newhelp command; see the newhelp(1) man page. Converted help text files are usually placed in the PMDA’s directory below ${PCP_PMDAS_DIR} as part of the PMDA installation procedure.

Example 2.18. Help Text for the Simple PMDA

The two instance domains and five metrics have a short and a verbose description. Each entry begins with a line that starts with the character “@” and is followed by either the metric name (simple.numfetch) or a symbolic reference to the instance domain number (SIMPLE.1), followed by the short description. The verbose description is on the following lines, terminated by the next line starting with “@” or end of file:

@ SIMPLE.0 Instance domain “colour” for simple PMDA
Universally 3 instances, “red” (0), “green” (1) and “blue” (3).

@ SIMPLE.1 Dynamic instance domain “time” for simple PMDA
An instance domain is computed on-the-fly for exporting current time
information. Refer to the help text for simple.now for more details.

@ simple.numfetch Number of pmFetch operations.
The cumulative number of pmFetch operations directed to “simple” PMDA.

This counter may be modified with pmstore(1).

@ simple.color Metrics which increment with each fetch
This metric has 3 instances, designated “red”, “green” and “blue”.

The value of the metric is monotonic increasing in the range 0 to
255, then back to 0.  The different instances have different starting
values, namely 0 (red), 100 (green) and 200 (blue).

The metric values my be altered using pmstore(1).

@ simple.time.user Time agent has spent executing user code
The time in seconds that the CPU has spent executing agent user code.

@ simple.time.sys Time agent has spent executing system code
The time in seconds that the CPU has spent executing agent system code.

@ simple.now Time of day with a configurable instance domain
The value reflects the current time of day through a dynamically
reconfigurable instance domain.  On each metric value fetch request,
the agent checks to see whether the configuration file in
${PCP_PMDAS_DIR}/simple/simple.conf has been modified - if it has then
the file is re-parsed and the instance domain for this metric is again
constructed according to its contents.

This configuration file contains a single line of comma-separated time
tokens from this set:
  “sec”  (seconds after the minute),
  “min”  (minutes after the hour),
  “hour” (hour since midnight).

An example configuration file could be:  sec,min,hour
and in this case the simple.now metric would export values for the
three instances “sec”, “min” and “hour” corresponding respectively to
the components seconds, minutes and hours of the current time of day.

The instance domain reflects each token present in the file, and the
values reflect the time at which the PMDA processes the fetch.

2.4.5. Management of Evolution within a PMDA

Evolution of a PMDA, or more particularly the underlying instrumentation to which it provides access, over time naturally results in the appearance of new metrics and the disappearance of old metrics. This creates potential problems for PMAPI clients and PCP tools that may be required to interact with both new and former versions of the PMDA.

The following guidelines are intended to help reduce the complexity of implementing a PMDA in the face of evolutionary change, while maintaining predictability and semantic coherence for tools using the PMAPI, and for end users of those tools.

  • Try to support as full a range of metrics as possible in every version of the PMDA. In this context, support means responding sensibly to requests, even if the underlying instrumentation is not available.

  • If a metric is not supported in a given version of the underlying instrumentation, the PMDA should respond to pmLookupDesc requests with a pmDesc structure whose type field has the special value PM_TYPE_NOSUPPORT. Values of fields other than pmid and type are immaterial, but Example 2.19. Setting Values is typically benign:

Example 2.19. Setting Values

pmDesc dummy = {
     .pmid  = PMDA_PMID(3,0),           /* pmid, fill this in */
     .type  = PM_TYPE_NOSUPPORT,        /* this is the important part */
     .indom = PM_INDOM_NULL,            /* singular,causes no problems */
     .sem   = 0,                        /* no semantics */
     .units = PMDA_PMUNITS(0,0,0,0,0,0) /* no units */
  • If a metric lacks support in a particular version of the underlying instrumentation, the PMDA should respond to pmFetch requests with a pmResult in which no values are returned for the unsupported metric. This is marginally friendlier than the other semantically acceptable option of returning an illegal PMID error or PM_ERR_PMID.

  • Help text should be updated with annotations to describe different versions of the underlying product, or product configuration options, for which a specific metric is available. This is so pmLookupText can always respond correctly.

  • The pmStore operation should fail with return status of PM_ERR_PERMISSION if a user or application tries to amend the value of an unsupported metric.

  • The value extraction, conversion, and printing functions (pmExtractValue, pmConvScale, pmAtomStr, pmTypeStr, and pmPrintValue) return the PM_ERR_CONV error or an appropriate diagnostic string, if an attempt is made to operate on a value for which type is PM_TYPE_NOSUPPORT.

  • If performance tools take note of the type field in the pmDesc structure, they should not manipulate values for unsupported metrics. Even if tools ignore type in the metric’s description, following these development guidelines ensures that no misleading value is ever returned; so there is no reason to call the extraction, conversion, and printing functions.

2.5. PMDA Interface

This section describes an interface for the request handling callbacks in a PMDA. This interface is used by PMCD for communicating with DSO PMDAs and is also used by daemon PMDAs with pmdaMain.

2.5.1. Overview

Both daemon and DSO PMDAs must handle multiple request types from PMCD. A daemon PMDA communicates with PMCD using the PDU protocol, while a DSO PMDA defines callbacks for each request type. To avoid duplicating this PDU processing (in the case of a PMDA that can be installed either as a daemon or as a DSO), and to allow a consistent framework, pmdaMain can be used by a daemon PMDA as a wrapper to handle the communication protocol using the same callbacks as a DSO PMDA. This allows a PMDA to be built as both a daemon and a DSO, and then to be installed as either.

To further simplify matters, default callbacks are declared in <pcp/pmda.h>:












Each callback takes a pmdaExt structure as its last argument. This structure contains all the information that is required by the default callbacks in most cases. The one exception is pmdaFetch, which needs an additional callback to instantiate the current value for each supported combination of a performance metric and an instance.

Therefore, for most PMDAs all the communication with PMCD is automatically handled by functions in libpcp.so and libpcp_pmda.so. Trivial PMDA

The trivial PMDA uses all of the default callbacks as shown in Example 2.20. Request Handling Callbacks in the Trivial PMDA. The additional callback for pmdaFetch is defined as trivial_fetchCallBack:

Example 2.20. Request Handling Callbacks in the Trivial PMDA

static int
trivial_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
   __pmID_int      *idp = (__pmID_int *)&(mdesc->m_desc.pmid);

   if (idp->cluster != 0 || idp->item != 0)
       return PM_ERR_PMID;
   if (inst != PM_IN_NULL)
       return PM_ERR_INST;
   atom->l = time(NULL);
   return 0;

This function checks that the PMID and instance are valid, and then places the metric value for the current time into the pmAtomValue structure.

The callback is set up by a call to pmdaSetFetchCallBack in trivial_init. As a rule of thumb, the API routines with named ending with CallBack are helpers for the higher PDU handling routines like pmdaFetch. The latter are set directly using the PMDA Interface Structures, as described in Section 2.5.2, “PMDA Structures”. Simple PMDA

The simple PMDA callback for pmdaFetch is more complicated because it supports more metrics, some metrics are instantiated with each fetch, and one instance domain is dynamic. The default pmdaFetch callback, shown in Example 2.21. Request Handling Callbacks in the Simple PMDA, is replaced by simple_fetch in simple_init, which increments the number of fetches and updates the instance domain for INDOM_NOW before calling pmdaFetch:

Example 2.21. Request Handling Callbacks in the Simple PMDA

static int
simple_fetch(int numpmid, pmID pmidlist[], pmResult **resp, pmdaExt *pmda)
    return pmdaFetch(numpmid, pmidlist, resp, pmda);

The callback for pmdaFetch is defined as simple_fetchCallBack. The PMID is extracted from the pmdaMetric structure, and if valid, the appropriate field in the pmAtomValue structure is set. The available types and associated fields are described further in Section 3.4, “Performance Metric Descriptions” and Example 3.18. pmAtomValue Structure.


Note that PMID validity checking need only check the cluster and item numbers, the domain number is guaranteed to be valid and the PMDA should make no assumptions about the actual domain number being used at this point.

The simple.numfetch metric has no instance domain and is easily handled first as shown in Example 2.22. simple.numfetch Metric:

Example 2.22. simple.numfetch Metric

static int
simple_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
   int             i;
   static int      oldfetch;
   static double   usr, sys;
   __pmID_int      *idp = (__pmID_int *)&(mdesc->m_desc.pmid);

   if (inst != PM_IN_NULL &&
       !(idp->cluster == 0 && idp->item == 1) &&
       !(idp->cluster == 2 && idp->item == 4))
       return PM_ERR_INST;
   if (idp->cluster == 0) {
       if (idp->item == 0) {                   /* simple.numfetch */
           atom->l = numfetch;

In Example 2.23. simple.color Metric, the inst parameter is used to specify which instance is required for the simple.color metric:

Example 2.23. simple.color Metric

else if (idp->item == 1) {              /* simple.color */
      switch (inst) {
      case 0:                             /* red */
          red = (red + 1) % 256;
          atom->l = red;
      case 1:                             /* green */
          green = (green + 1) % 256;
          atom->l = green;
      case 2:                             /* blue */
          blue = (blue + 1) % 256;
          atom->l = blue;
          return PM_ERR_INST;
     return PM_ERR_PMID;

In Example 2.24. simple.time Metric, the simple.time metric is in a second cluster and has a simple optimization to reduce the overhead of calling times twice on the same fetch and return consistent values from a single call to times when both metrics simple.time.user and simple.time.sys are requested in a single pmFetch. The previous fetch count is used to determine if the usr and sys values should be updated:

Example 2.24. simple.time Metric

else if (idp->cluster == 1) {               /* simple.time */
    if (oldfetch < numfetch) {
        __pmProcessRunTimes(&usr, &sys);
        oldfetch = numfetch;
    if (idp->item == 2)                     /* simple.time.user */
        atom->d = usr;
    else if (idp->item == 3)                /* simple.time.sys */
        atom->d = sys;
        return PM_ERR_PMID;

In Example 2.25. simple.now Metric, the simple.now metric is in a third cluster and uses inst again to select a specific instance from the INDOM_NOW instance domain. The values associated with instances in this instance domain are managed using the pmdaCache(3) helper routines, which provide efficient interfaces for managing more complex instance domains:

Example 2.25. simple.now Metric

else if (idp->cluster == 2) {
    if (idp->item == 4) {                 /* simple.now */
        struct timeslice *tsp;
        sts = pmdaCacheLookup(*now_indom, inst, NULL, (void *)&tsp);
        if (sts != PMDA_CACHE_ACTIVE) {
            if (sts < 0)
                pmNotifyErr(LOG_ERR, "pmdaCacheLookup failed: inst=%d: %s",
                              inst, pmErrStr(sts));
            return PM_ERR_INST;
        atom->l = tsp->tm_field;
        return PM_ERR_PMID;
} simple_store in the Simple PMDA

The simple PMDA permits some of the metrics it supports to be modified by pmStore as shown in Example 2.26. simple_store in the Simple PMDA. For additional information, see the pmstore(1) and pmStore(3) man pages.

Example 2.26. simple_store in the Simple PMDA

The pmdaStore callback (which returns PM_ERR_PERMISSION to indicate no metrics can be altered) is replaced by simple_store in simple_init. This replacement function must take the same arguments so that it can be assigned to the function pointer in the pmdaInterface structure.

The function traverses the pmResult and checks the cluster and unit of each PMID to ensure that it corresponds to a metric that can be changed. Checks are made on the values to ensure they are within range before being assigned to variables in the PMDA that hold the current values for exported metrics:

static int
simple_store(pmResult *result, pmdaExt *pmda)
    int         i, j, val, sts = 0;
    pmAtomValue av;
    pmValueSet  *vsp = NULL;
    __pmID_int  *pmidp = NULL;

    /* a store request may affect multiple metrics at once */
    for (i = 0; i < result->numpmid; i++) {
        vsp = result->vset[i];
        pmidp = (__pmID_int *)&vsp->pmid;
        if (pmidp->cluster == 0) {  /* storable metrics are cluster 0 */
            switch (pmidp->item) {
            case 0:                           /* simple.numfetch */
                val = vsp->vlist[0].value.lval;
                if (val < 0) {
                    sts = PM_ERR_SIGN;
                    val = 0;
                numfetch = val;
            case 1:                             /* simple.color */
                /* a store request may affect multiple instances at once */
                for (j = 0; j < vsp->numval && sts == 0; j++) {
                    val = vsp->vlist[j].value.lval;
                    if (val < 0) {
                        sts = PM_ERR_SIGN;
                        val = 0;
                    } if (val > 255) {
                        sts = PM_ERR_CONV;
                        val = 255;

The simple.color metric has an instance domain that must be searched because any or all instances may be specified. Any instances that are not supported in this instance domain should cause an error value of PM_ERR_INST to be returned as shown in Example 2.27. simple.color and PM_ERR_INST Errors:

Example 2.27. simple.color and PM_ERR_INST Errors

switch (vsp->vlist[j].inst) {
case 0:                         /* red */
    red = val;
case 1:                         /* green */
    green = val;
case 2:                         /* blue */
    blue = val;
    sts = PM_ERR_INST;

Any other PMIDs in cluster 0 that are not supported by the simple PMDA should result in an error value of PM_ERR_PMID as shown in Example 2.28. PM_ERR_PMID Errors:

Example 2.28. PM_ERR_PMID Errors

            sts = PM_ERR_PMID;

Any metrics that cannot be altered should generate an error value of PM_ERR_PERMISSION, and metrics not supported by the PMDA should result in an error value of PM_ERR_PMID as shown in Example 2.29. PM_ERR_PERMISSION and PM_ERR_PMID Errors:

Example 2.29. PM_ERR_PERMISSION and PM_ERR_PMID Errors

        else if ((pmidp->cluster == 1 &&
                 (pmidp->item == 2 || pmidp->item == 3)) ||
                 (pmidp->cluster == 2 && pmidp->item == 4)) {
            sts = PM_ERR_PERMISSION;
        else {
            sts = PM_ERR_PMID;
    return sts;

The structure pmdaExt pmda argument is not used by the simple_store function above.


When using storable metrics, it is important to consider the implications. It is possible pmlogger is actively sampling the metric being modified, for example, which may cause unexpected results to be persisted in an archive. Consider also the use of client credentials, available via the attribute callback of the pmdaInterface structure, to appropriately limit access to any modifications that might be made via your storable metrics. Return Codes for pmdaFetch Callbacks

In PMDA_INTERFACE_1 and PMDA_INTERFACE_2, the return codes for the pmdaFetch callback function are defined:



< 0

Error code (for example, PM_ERR_PMID, PM_ERR_INST or PM_ERR_AGAIN)



In PMDA_INTERFACE_3 and all later versions, the return codes for the pmdaFetch callback function are defined:



< 0

Error code (for example, PM_ERR_PMID, PM_ERR_INST)


Metric value not currently available

> 0


2.5.2. PMDA Structures

PMDA structures used with the pcp_pmda library are defined in <pcp/pmda.h>. Example 2.30. pmdaInterface Structure Header and Example 2.32. pmdaExt Structure describe the pmdaInterface and pmdaExt structures.

Example 2.30. pmdaInterface Structure Header

The callbacks must be specified in a pmdaInterface structure:

typedef struct {
    int domain;     /* set/return performance metrics domain id here */
    struct {
        unsigned int pmda_interface : 8;  /* PMDA DSO version */
        unsigned int pmapi_version : 8;   /* PMAPI version */
        unsigned int flags : 16;          /* optional feature flags */
    } comm;             /* set/return communication and version info */
    int status;         /* return initialization status here */
    union {

This structure is passed by PMCD to a DSO PMDA as an argument to the initialization function. This structure supports multiple (binary-compatible) versions–the second and subsequent versions have support for the pmdaExt structure. Protocol version one is for backwards compatibility only, and should not be used in any new PMDA.

To date there have been six revisions of the interface structure:

  1. Version two added the pmdaExt structure, as mentioned above.

  2. Version three changed the fetch callback return code semantics, as mentioned in Section, “Return Codes for pmdaFetch Callbacks”.

  3. Version four added support for dynamic metric names, where the PMDA is able to create and remove metric names on-the-fly in response to changes in the performance domain (pmdaPMID, pmdaName, pmdaChildren interfaces)

  4. Version five added support for per-client contexts, where the PMDA is able to track arrival and disconnection of PMAPI client tools via PMCD (pmdaGetContext helper routine). At the same time, support for PM_TYPE_EVENT metrics was implemented, which relies on the per-client context concepts (pmdaEvent* helper routines).

  5. Version six added support for authenticated client contexts, where the PMDA is informed of user credentials and other PMCD attributes of the connection between individual PMAPI clients and PMCD (pmdaAttribute interface)

  6. Version seven added support for metadata labels, where the PMDA is able to associate name:value pairs in a hierarchy such that additional metadata, above and beyond the metric descriptors, is associated with metrics and instances (pmdaLabel interface)

Example 2.31. pmdaInterface Structure, Latest Version

    union {
         * PMDA_INTERFACE7
        struct {
            pmdaExt *ext;
            int     (*profile)(pmdaInProfile *, pmdaExt *);
            int     (*fetch)(int, pmID *, pmResult **, pmdaExt *);
            int     (*desc)(pmID, pmDesc *, pmdaExt *);
            int     (*instance)(pmInDom, int, char *, pmdaInResult **, pmdaExt *);
            int     (*text)(int, int, char **, pmdaExt *);
            int     (*store)(pmResult *, pmdaExt *);
            int     (*pmid)(const char *, pmID *, pmdaExt *);
            int     (*name)(pmID, char ***, pmdaExt *);
            int     (*children)(const char *, int, char ***, int **, pmdaExt *);
            int     (*attribute)(int, int, const char *, int, pmdaExt *);
            int     (*label)(int, int, pmLabelSet **, pmdaExt *);
        } seven;
    } version;
} pmdaInterface;


Each new interface version is always defined as a superset of those that preceded it, only adds fields at the end of the new structure in the union, and is always binary backwards-compatible. And thus it shall remain. For brevity, we have shown only the latest interface version (seven) above, but all prior versions still exist, build, and function. In other words, PMDAs built against earlier versions of this header structure (and PMDA library) function correctly with the latest version of the PMDA library.

Example 2.32. pmdaExt Structure

Additional PMDA information must be specified in a pmdaExt structure:

typedef struct {
    unsigned int e_flags;       /* PMDA_EXT_FLAG_* bit field */
    void        *e_ext;         /* used internally within libpcp_pmda */
    char        *e_sockname;    /* socket name to pmcd */
    char        *e_name;        /* name of this pmda */
    char        *e_logfile;     /* path to log file */
    char        *e_helptext;    /* path to help text */
    int         e_status;       /* =0 is OK */
    int         e_infd;         /* input file descriptor from pmcd */
    int         e_outfd;        /* output file descriptor to pmcd */
    int         e_port;         /* port to pmcd */
    int         e_singular;     /* =0 for singular values */
    int         e_ordinal;      /* >=0 for non-singular values */
    int         e_direct;       /* =1 if pmid map to meta table */
    int         e_domain;       /* metrics domain */
    int         e_nmetrics;     /* number of metrics */
    int         e_nindoms;      /* number of instance domains */
    int         e_help;         /* help text comes via this handle */
    pmProfile   *e_prof;        /* last received profile */
    pmdaIoType  e_io;           /* connection type to pmcd */
    pmdaIndom   *e_indoms;      /* instance domain table */
    pmdaIndom   *e_idp;         /* instance domain expansion */
    pmdaMetric  *e_metrics;     /* metric description table */
    pmdaResultCallBack e_resultCallBack; /* to clean up pmResult after fetch */
    pmdaFetchCallBack  e_fetchCallBack;  /* to assign metric values in fetch */
    pmdaCheckCallBack  e_checkCallBack;  /* callback on receipt of a PDU */
    pmdaDoneCallBack   e_doneCallBack;   /* callback after PDU is processed */
    /* added for PMDA_INTERFACE_5 */
    int         e_context;      /* client context id from pmcd */
    pmdaEndContextCallBack e_endCallBack;  /* callback after client context closed */
    /* added for PMDA_INTERFACE_7 */
    pmdaLabelCallBack  e_labelCallBack;  /* callback to lookup metric instance labels */
} pmdaExt;

The pmdaExt structure contains filenames, pointers to tables, and some variables shared by several functions in the pcp_pmda library. All fields of the pmdaInterface and pmdaExt structures can be correctly set by PMDA initialization functions; see the pmdaDaemon(3), pmdaDSO(3), pmdaGetOptions(3), pmdaInit(3), and pmdaConnect(3) man pages for a full description of how various fields in these structures may be set or used by pcp_pmda library functions.

2.6. Initializing a PMDA

Several functions are provided to simplify the initialization of a PMDA. These functions, if used, must be called in a strict order so that the PMDA can operate correctly.

2.6.1. Overview

The initialization process for a PMDA involves opening help text files, assigning callback function pointers, adjusting the metric and instance identifiers to the correct domains, and much more. The initialization of a daemon PMDA also differs significantly from a DSO PMDA, since the pmdaInterface structure is initialized by main or the PMCD process, respectively.

2.6.2. Common Initialization

As described in Section 2.2.2, “DSO PMDA”, an initialization function is provided by a DSO PMDA and called by PMCD. Using the standard PMDA wrappers, the same function can also be used as part of the daemon PMDA initialization. This PMDA initialization function performs the following tasks:

  • Assigning callback functions to the function pointer interface of pmdaInterface

  • Assigning pointers to the metric and instance tables from pmdaExt

  • Opening the help text files

  • Assigning the domain number to the instance domains

  • Correlating metrics with their instance domains

If the PMDA uses the common data structures defined for the pcp_pmda library, most of these requirements can be handled by the default pmdaInit function; see the pmdaInit(3) man page.

Because the initialization function is the only initialization opportunity for a DSO PMDA, the common initialization function should also perform any DSO-specific functions that are required. A default implementation of this functionality is provided by the pmdaDSO function; see the pmdaDSO(3) man page. Trivial PMDA

Example 2.33. Initialization in the Trivial PMDA shows the trivial PMDA, which has no instances (that is, all metrics have singular values) and a single callback. This callback is for the pmdaFetch function called trivial_fetchCallBack; see the pmdaFetch(3) man page:

Example 2.33. Initialization in the Trivial PMDA

static char     *username;
static int      isDSO = 1;              /* ==0 if I am a daemon */

void trivial_init(pmdaInterface *dp)
    if (isDSO)
        pmdaDSO(dp, PMDA_INTERFACE_2, “trivial DSO”,

    if (dp->status != 0)

    pmdaSetFetchCallBack(dp, trivial_fetchCallBack);
    pmdaInit(dp, NULL, 0,
             metrictab, sizeof(metrictab)/sizeof(metrictab[0]));

The trivial PMDA can execute as either a DSO or daemon PMDA. A default installation installs it as a daemon, however, and the main routine clears isDSO and sets username accordingly.

The trivial_init routine provides the opportunity to do any extra DSO or daemon setup before calling the library pmdaInit. In the example, the help text is setup for DSO mode and the daemon is switched to run as an unprivileged user (default is root, but it is generally good form for PMDAs to run with the least privileges possible). If dp->status is non-zero after the pmdaDSO call, the PMDA will be removed by PMCD and cannot safely continue to use the pmdaInterface structure. Simple PMDA

In Example 2.34. Initialization in the Simple PMDA, the simple PMDA uses its own callbacks to handle PDU_FETCH and PDU_RESULT request PDUs (for pmFetch and pmStore operations respectively), as well as providing pmdaFetch with the callback simple_fetchCallBack.

Example 2.34. Initialization in the Simple PMDA

static int      isDSO = 1;              /* =0 I am a daemon */
static char     *username;

void simple_init(pmdaInterface *dp)
    if (isDSO)
        pmdaDSO(dp, PMDA_INTERFACE_7, “simple DSO”,

    if (dp->status != 0)

    dp->version.any.fetch = simple_fetch;
    dp->version.any.store = simple_store;
    dp->version.any.instance = simple_instance;
    dp->version.seven.label = simple_label;
    pmdaSetFetchCallBack(dp, simple_fetchCallBack);
    pmdaSetLabelCallBack(dp, simple_labelCallBack);
    pmdaInit(dp, indomtab, sizeof(indomtab)/sizeof(indomtab[0]),
             metrictab, sizeof(metrictab)/sizeof(metrictab[0]));

Once again, the simple PMDA may be installed either as a daemon PMDA or a DSO PMDA. The static variable isDSO indicates whether the PMDA is running as a DSO or as a daemon. A daemon PMDA always changes the value of this variable to 0 in main, for PMDAs that can operate in both modes.

Remember also, as described earlier, simple_fetch is dealing with a single request for (possibly many) values for metrics from the PMDA, and simple_fetchCallBack is its little helper, dealing with just one metric and one instance (optionally, if the metric happens to have an instance domain) within that larger request.

2.6.3. Daemon Initialization

In addition to the initialization function that can be shared by a DSO and a daemon PMDA, a daemon PMDA must also meet the following requirements:

  • Create the pmdaInterface structure that is passed to the initialization function

  • Parse any command-line arguments

  • Open a log file (a DSO PMDA uses PMCD’s log file)

  • Set up the IPC connection between the PMDA and the PMCD process

  • Handle incoming PDUs

All these requirements can be handled by default initialization functions in the pcp_pmda library; see the pmdaDaemon(3), pmdaGetOptions(3), pmdaOpenLog(3), pmdaConnect(3), and pmdaMain(3) man pages.


Optionally, a daemon PMDA may wish to reduce or change its privilege level, as seen in Example 2.33. Initialization in the Trivial PMDA and Example 2.34. Initialization in the Simple PMDA. Some performance domains require the extraction process to run as a specific user in order to access the instrumentation. Many domains require the default root level of access for a daemon PMDA.

The simple PMDA specifies the command-line arguments it accepts using pmdaGetOptions, as shown in Example 2.35. main in the Simple PMDA. For additional information, see the pmdaGetOptions(3) man page.

Example 2.35. main in the Simple PMDA

static pmLongOptions longopts[] = {
    PMDA_OPTIONS_TEXT(“\nExactly one of the following options may appear:”),
static pmdaOptions opts = {
    .short_options = “D:d:i:l:pu:U:6:?”,
    .long_options = longopts,

main(int argc, char **argv)
    pmdaInterface       dispatch;

    isDSO = 0;
    pmdaDaemon(&dispatch, PMDA_INTERFACE_7, pmGetProgname(), SIMPLE,
               “simple.log”, “${PCP_PMDAS_DIR}/simple/help”);

    pmdaGetOptions(argc, argv, &opts, &dispatch);
    if (opts.errors) {
    if (opts.username)
        username = opts.username;



The conditions under which pmdaMain will return are either unexpected error conditions (often from failed initialisation, which would already have been logged), or when PMCD closes the connection to the PMDA. In all cases the correct action to take is simply to exit cleanly, possibly after any final cleanup the PMDA may need to perform.

2.7. Testing and Debugging a PMDA

Ensuring the correct operation of a PMDA can be difficult, because the responsibility of providing metrics to the requesting PMCD process and simultaneously retrieving values from the target domain requires nearly real-time communication with two modules beyond the PMDA’s control. Some tools are available to assist in this important task.

2.7.1. Overview

Thoroughly testing a PMDA with PMCD is difficult, although testing a daemon PMDA is marginally simpler than testing a DSO PMDA. If a DSO PMDA exits, PMCD also exits because they share a single address space and control thread.

The difficulty in using PMCD to test a daemon PMDA results from PMCD requiring timely replies from the PMDA in response to request PDUs. Although a timeout period can be set in ${PCP_PMCDOPTIONS_PATH}, attaching a debugger (such as gdb) to the PMDA process might cause an already running PMCD to close its connection with the PMDA. If timeouts are disabled, PMCD could wait forever to connect with the PMDA.

If you suspect a PMDA has been terminated due to a timeout failure, check the PMCD log file, usually ${PCP_LOG_DIR}/pmcd/pmcd.log.

A more robust way of testing a PMDA is to use the dbpmda tool, which is similar to PMCD except that dbpmda provides complete control over the PDUs that are sent to the PMDA, and there are no time limits–it is essentially an interactive debugger for exercising a PMDA. See the dbpmda(3) man page for details.

In addition, careful use of PCP debugging flags can produce useful information concerning a PMDA’s behavior; see the PMAPI(3) and pmdbg(1) man pages for a discussion of the PCP debugging and tracing framework.

2.7.2. Debugging Information

You can activate debugging options in PMCD and most other PCP tools with the -D command-line option. Supported options can be listed with the pmdbg command; see the pmdbg(1) man page. Setting the debug options for PMCD in ${PCP_PMCDOPTIONS_PATH} might generate too much information to be useful, especially if there are other clients and PMDAs connected to the PMCD process.

The PMCD debugging options can also be changed dynamically by storing a new value into the metric pmcd.control.debug:

# pmstore pmcd.control.debug 5

Most of the pcp_pmda library functions log additional information if the libpmda option is set within the PMDA; see the PMDA(3) man page. The command-line argument -D is trapped by pmdaGetOptions to set the global debugging control options. Adding tests within the PMDA for the appl0, appl1 and appl2 trace flags permits different levels of information to be logged to the PMDA’s log file.

All diagnostic, debugging, and tracing output from a PMDA should be written to the standard error stream.

Adding this segment of code to the simple_store metric causes a timestamped log message to be sent to the current log file whenever pmstore attempts to change simple.numfetch and the PCP debugging options have the appl0 option set as shown in Example 2.36. simple.numfetch in the Simple PMDA:

Example 2.36. simple.numfetch in the Simple PMDA

case 0: /* simple.numfetch */
     val = vsp->vlist[0].value.lval;
     if (val < 0) {
         sts = PM_ERR_SIGN;
         val = 0;
     if (pmDebugOptions.appl0__) {
               "simple: %d stored into numfetch", val);
     numfetch = val;

For a description of pmstore, see the pmstore(1) man page.

2.7.3. dbpmda Debug Utility

The dbpmda utility provides a simple interface to the PDU communication protocol. It allows daemon and DSO PMDAs to be tested with most request types, while the PMDA process may be monitored with a debugger, tracing utilities, and other diagnostic tools. The dbpmda(1) man page contains a sample session with the simple PMDA.

2.8. Integration of a PMDA

Several steps are required to install (or remove) a PMDA from a production PMCD environment without affecting the operation of other PMDAs or related visualization and logging tools.

The PMDA typically would have its own directory below ${PCP_PMDAS_DIR} into which several files would be installed. In the description in Section 2.8.1, “Installing a PMDA”, the PMDA of interest is assumed to be known by the name newbie, hence the PMDA directory would be ${PCP_PMDAS_DIR}/newbie.


Any installation or removal of a PMDA involves updating files and directories that are typically well protected. Hence the procedures described in this section must be executed as the superuser.

2.8.1. Installing a PMDA

A PMDA is fully installed when these tasks are completed:

  • Help text has been installed in a place where the PMDA can find it, usually in the PMDA directory ${PCP_PMDAS_DIR}/newbie.

  • The name space has been updated in the ${PCP_VAR_DIR}/pmns directory.

  • The PMDA binary has been installed, usually in the directory ${PCP_PMDAS_DIR}/newbie.

  • The ${PCP_PMCDCONF_PATH} file has been updated.

  • The PMCD process has been restarted or notified (with a SIGHUP signal) that the new PMDA exists.

The Makefile should include an install target to compile and link the PMDA (as a DSO, or a daemon or both) in the PMDA directory. The clobber target should remove any files created as a by-product of the install target.

You may wish to use ${PCP_PMDAS_DIR}/simple/Makefile as a template for constructing a new PMDA Makefile; changing the assignment of IAM from simple to newbie would account for most of the required changes.

The Install script should make use of the generic procedures defined in the script ${PCP_SHARE_DIR}/lib/pmdaproc.sh, and may be as straightforward as the one used for the trivial PMDA, shown in Example 2.37. Install Script for the Trivial PMDA:

Example 2.37. Install Script for the Trivial PMDA

. ${PCP_DIR}/etc/pcp.env
. ${PCP_SHARE_DIR}/lib/pmdaproc.sh


The variables, shown in Table 2.1. Variables to Control Behavior of Generic pmdaproc.sh Procedures, may be assigned values to modify the behavior of the pmdaSetup and pmdainstall procedures from ${PCP_SHARE_DIR}/lib/pmdaproc.sh.

Table 2.1. Variables to Control Behavior of Generic pmdaproc.sh Procedures

Shell Variable




Name of the PMDA; assignment to this variable is mandatory. Example: iam=newbie


Can this PMDA be installed as a DSO?



Can this PMDA be installed as a daemon?



Is this PMDA a perl script?



Is this PMDA a python script?



If installed as a daemon PMDA, is the default IPC via pipes?



If installed as a daemon PMDA, is the default IPC via an Internet socket?



If installed as a daemon PMDA, and the IPC method uses an Internet socket, the default port number.


IPC style for PDU exchanges involving a daemon PMDA; binary or text.



Delay in seconds between installing PMDA and checking if metrics are available.



Additional command-line arguments passed to a daemon PMDA.


The name of the PMNS file (by default relative to the PMDA directory).



First-level name for this PMDA’s metrics in the PMNS.



The name of the help file (by default relative to the PMDA directory).



The name of the executable for a daemon PMDA.



The name of the shared library for a DSO PMDA.



The name of the initialization function for a DSO PMDA.



The numerical PMDA domain number (from domain.h).


The symbolic name of the PMDA domain number (from domain.h).


Exit status for the shell script


In addition, the variable do_check will be set to reflect the intention to check the availability of the metrics once the PMDA is installed. By default this variable is true however the command-line option -Q to Install may be used to set the variable to false.

Obviously, for anything but the most trivial PMDA, after calling the pmdaSetup procedure, the Install script should also prompt for any PMDA-specific parameters, which are typically accumulated in the args variable and used by the pmdainstall procedure.

The detailed operation of the pmdainstall procedure involves the following tasks:

  • Using default assignments, and interaction where ambiguity exists, determine the PMDA type (DSO or daemon) and the IPC parameters, if any.

  • Copy the $pmns_source file, replacing symbolic references to SYMDOM by the desired numeric domain number from domain.

  • Merge the PMDA’s name space into the PCP name space at the non-leaf node identified by $pmns_name.

  • If any pmchart views can be found (files with names ending in “.pmchart”), copy these to the standard directory (${PCP_VAR_DIR}/config/pmchart) with the “.pmchart” suffix removed.

  • Create new help files from $help_source after replacing symbolic references to SYMDOM by the desired numeric domain number from domain.

  • Terminate the old daemon PMDA, if any.

  • Use the Makefile to build the appropriate executables.

  • Add the PMDA specification to PMCD’s configuration file (${PCP_PMCDCONF_PATH}).

  • Notify PMCD. To minimize the impact on the services PMCD provides, sending a SIGHUP to PMCD forces it to reread the configuration file and start, restart, or remove any PMDAs that have changed since the file was last read. However, if the newly installed PMDA must run using a different privilege level to PMCD then PMCD must be restarted. This is because PMCD runs unprivileged after initially starting the PMDAs.

  • Check that the metrics from the new PMDA are available.

There are some PMDA changes that may trick PMCD into thinking nothing has changed, and not restarting the PMDA. Most notable are changes to the PMDA executable. In these cases, you may need to explicitly remove the PMDA as described in Section 2.8.2, “Removing a PMDA”, or more drastically, restart PMCD as follows:

# ${PCP_RC_DIR}/pcp start

The files ${PCP_PMDAS_DIR}/*/Install provide a wealth of examples that may be used to construct a new PMDA Install script.

2.8.2. Removing a PMDA

The simplest way to stop a PMDA from running, apart from killing the process, is to remove the entry from ${PCP_PMCDCONF_PATH} and signal PMCD (with SIGHUP) to reread its configuration file. To completely remove a PMDA requires the reverse process of the installation, including an update of the Performance Metrics Name Space (PMNS).

This typically involves a Remove script in the PMDA directory that uses the same common procedures as the Install script described Section 2.8.1, “Installing a PMDA”.

The ${PCP_PMDAS_DIR}/*/Remove files provide a wealth of examples that may be used to construct a new PMDA Remove script.

2.8.3. Configuring PCP Tools

Most PCP tools have their own configuration file format for specifying which metrics to view or to log. By using canned configuration files that monitor key metrics of the new PMDA, users can quickly see the performance of the target system, as characterized by key metrics in the new PMDA.

Any configuration files that are created should be kept with the PMDA and installed into the appropriate directories when the PMDA is installed.

As with all PCP customization, some of the most valuable tools can be created by defining views, scenes, and control-panel layouts that combine related performance metrics from multiple PMDAs or multiple hosts.

Metrics suitable for on-going logging can be specified in templated pmlogger configuration files for pmlogconf to automatically add to the pmlogger_daily recorded set; see the pmlogger(1), pmlogconf(1) and pmlogger_daily(1) man pages.

Parameterized alarm configurations can be created using the pmieconf facilities; see the pmieconf(1) and pmie(1) man pages.