pmdaopenmetrics(1) — Linux manual page

NAME | SYNOPSIS | DESCRIPTION | CONFIGURATION SOURCES | URL SOURCES | METRIC FILTERING | LABEL FILTERING | SCRIPTED SOURCES | SELinux CONSIDERATIONS | METRIC NAMING | DYNAMIC METRIC NAMES | METADATA | CONTROL METRICS | LIMITATIONS | INSTALLATION | FILES | PCP ENVIRONMENT | SEE ALSO | COLOPHON

PMDAOPENMETRICS(1)       General Commands Manual      PMDAOPENMETRICS(1)

NAME         top

       pmdaopenmetrics - OpenMetrics PMDA

SYNOPSIS         top

       $PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics [-D] [-n] [-c config]
       [-d domain] [-l logfile] [-r root] [-t timeout] [-u user]

DESCRIPTION         top

       pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA)
       which dynamically creates PCP metrics from configured OpenMetrics
       endpoints, which provide HTTP based access to application
       metrics.  The PMDA essentially implements a bridge between
       Prometheus and PCP, allowing PCP to easily ingest performance
       data from more than 650 registered end-points and many other
       application specific end-points.

       The default config directory is
       $PCP_PMDAS_DIR/openmetrics/config.d/, see ``CONFIGURATION
       SOURCES'' below.  The default URL fetch timeout is 2 seconds.
       The default user, if not specified with the -u option, is the
       current user.  If the -n option is given, the list of
       configuration files will not be sorted prior to processing.  This
       list is sorted by default but that can be expensive if there are
       a large number of configuration files (URLs and/or scripts).

       If the -D option is given, additional diagnostic messages will be
       written to the PMDA log file, which is
       $PCP_LOG_DIR/pmcd/openmetrics.log by default (see also -lbelow).
       In addition, the metric openmetrics.control.debug controls the
       same debug flag and can be set with the following command:
            pmstore openmetrics.control.debug value
       where value is either 1 (to enable verbose log messages) or 0 (to
       disable verbose log messages).  This is particularly useful for
       examining the http headers passed to each fetch request, filter
       settings and other processing details that are logged when the
       debugging flag is enabled.

       The -d option may be used to override the default performance
       metrics domain number, which defaults to 144.  It is strongly
       recommended not to change this.  The domain number should be
       different for every PMDA on the one host, and the same domain
       number should be used for pmdaopenmetrics PMDA on all hosts.  See
       also the -r option, which allows the root of the dynamic
       namespace to be changed from the default openmetrics.

       The -l option may be used to specify logfile as the destination
       for PMDA messages instead of the default,
       $PCP_LOG_DIR/pmcd/openmetrics.log.  As a special case, logfile
       may be "-" to send messages to the stderr stream instead, e.g.
       -l-.  This would normally be the stderr stream for the parent
       process, pmcd(1), which may itself have redirected stderr.  This
       redirection is normally most useful in a containerized
       environment, or when using dbpmda(1).

       The -r option allows the root of the dynamic namespace to be
       changed to root from the default, openmetrics.  In conjunction
       with other command line options, this allows pmdaopenmetrics to
       be deployed as a different PMDA with distinct metrics namespace
       and metrics domain on the same host system.  Note that all PMDAs
       require a unique domain number so the -d option must also be
       specified.  Use of the -r option may also change the defaults for
       some other command line options, e.g. the default log file name
       and the default configuration directory.

CONFIGURATION SOURCES         top

       As it runs, pmdaopenmetrics periodically recursively scans the
       $PCP_PMDAS_DIR/openmetrics/config.d directory (or the directory
       specified with the -c option), looking for source URL files
       (*.url) and executable scripts or binaries.  Any files that do
       not have the .url suffix or are not executable, are ignored -
       this allows documentation files such as "README" and non-
       executable "common" script function definitions to be present
       without being considered as config files.

       A remote server does not have to be up or stay running - the PMDA
       tolerates remote URLs that may come and go over time.  The PMDA
       will relay data and metadata when/if they are available, and will
       return errors when/if they are down.  PCP metric IDs, internal
       and external instance domain identifiers are persisted and will
       be restored when individual metric sources become available
       and/or when the PMDA is restarted.  In addition, the PMDA checks
       directory modification times and will rescan for new or changed
       configuration files dynamically.  It is not necessary to restart
       the PMDA when adding, removing or changing configuration files.

URL SOURCES         top

       Each file with the .url suffix found in the config directory or
       sub-directory contains one complete HTTP or HTTPS URL at which
       pmdaopenmetrics can reach a OpenMetrics endpoint.  Local file
       access is also supported with a conventional
       file:///somepath/somefile URL, in which case somepath/somefile
       should contain openmetrics formatted metric data.

       The first line of a .url config file should be the URL, as
       described above.  Subsequent lines, if any, are prefixed with a
       keyword that can be used to alter the http GET request.  A
       keyword must end with ':' (colon) and the text extends to the end
       of the line.  Comment lines that start with # and blank lines are
       ignored.  The only currently supported keywords are HEADER: and
       FILTER:.

       HEADER: headername: value ... to end of line
       Adds headername and its value to the headers passed in the http
       GET request for the configured URL.  An example configuration
       file that provides 3 commonly used headers and an authentication
       token might be :

          http://somehost/path/endpoint.html
          # this is a comment
          HEADER: Accept: text/html
          HEADER: Keep-Alive: 300
          HEADER: Connection: keep-alive
          HEADER: Authorization: token ABCDEF1234567890

       As mentioned above, header values extend to the end of the line.
       They may contain any valid characters, including colons.
       Multiple spaces will be collapsed to a single space, and leading
       and trailing spaces are trimmed.  A common use for headers is to
       configure a proxy agent and the assorted parameters it may
       require.

METRIC FILTERING         top

       Metric filtering is a configuration file feature that allows
       ingested metrics to be included or excluded, i.e. filtered.  This
       is useful because most end-points return multiple metrics, and
       usually only some are interesting for monitoring purposes.  The
       syntax is:
       FILTER: INCLUDE METRIC regex
       or
       FILTER: EXCLUDE METRIC regex
       Dynamically created metric names that match regex will be either
       included or excluded in the name space, as specified.  Note that
       only the PMNS leaf component of the metric name (as ingested from
       the URL source) is compared with the regex pattern.  The simple
       rule is that the first matching filter regex for a particular
       metric leaf name is the rule that prevails.  If no filter regex
       matches (or there are no filters), then the metric is included by
       default, i.e. the default filter if none are specified is FILTER:
       INCLUDE METRIC .*  This is backward compatible with older
       versions of the configuration file that did not support filters.
       Multiple FILTER: lines would normally be used, e.g. to include
       some metrics but exclude all others, use FILTER: EXCLUDE METRIC
       .*  as the last of several filters that include the desired
       metrics.  Conversely, to exclude some metrics but include all
       others, use FILTER: EXCLUDE METRIC regex.  In this case it's not
       necessary (though doesn't hurt) to specify the final FILTER:
       INCLUDE METRIC .*  because, as stated above, any metric that does
       not match any filter regex will be included by default.

LABEL FILTERING         top

       Label filtering uses similar FILTER: syntax and semantics as
       metric filtering.  FILTER: EXCLUDE LABEL regex will delete all
       labels with label name matching regex from all metrics defined by
       the configuration file.  The same rules as for metric filters
       apply for label filters too - an implicit rule: FILTER: INCLUDE
       LABEL .*  applies to all labels that do not match any earlier
       label filter rule.  FILTER: OPTIONAL LABEL regex specifies that
       matching label names are to be included in the returned metric
       labelsets (i.e. included), but are not to be used as part of the
       the external instance names.  All included labels that are not
       optional (i.e. the intrinsic labels) will be concatenated
       together and used for external instance naming.  In addition,
       non-intrinsic labels (i.e. labels tagged as OPTIONAL) will have
       the PM_LABEL_OPTIONAL flag set in the labelsets returned by notes
       callbacks.  This flag affects how the labels are used in certain
       clients.  For further details, see pmLookupLabels(3) and related
       man pages for further details.  Note that external instance names
       begin with the unique numeric internal instance identifier
       followed by a space, so external instance names are always
       unique.

       Caution is needed with label filtering because by default, all
       labels are used to construct the PCP instance name.  By excluding
       some labels (or changing them to optional), the instance names
       will change.  In addition, excluding all labels for a particular
       metric changes that metric to be singular, i.e. have no instance
       domain.  By excluding some labels, different instances returned
       by the URL or scripted configuration entry for the same metric
       may become duplicates.  When such duplicates occur, the last
       duplicate instance returned by the end-point URL or script
       prevails over any earlier instances.  For these reasons, it is
       recommended that label filtering rules be configured when the
       configuration file is first defined, and not changed thereafter.
       If a label filtering change is required, the configuration file
       should be renamed, which effectively defines a new metric (or set
       of peer metrics as returned by the URL or script), with the new
       (or changed) instance naming.

       Unrecognized keywords in configuration files are reported in the
       PMDA log file but otherwise ignored.

SCRIPTED SOURCES         top

       Executable scripts present in the
       $PCP_PMDAS_DIR/openmetrics/config.d directory or sub-directories
       will be executed and the stdout stream containing openmetrics
       formatted metric data will be parsed as though it had come from a
       URL or file.  The stderr stream from a script will be sent to the
       PMDA log file, which by default can be found in
       $(PCP_LOG_DIR)/pmcd/openmetrics.log.

       Note that scripted sources do not support label or metric
       filtering (as described above for URL sources) - they can simply
       do their own filtering in the script itself with sed(1), awk(1),
       or whatever tool is desired.

       A simple example of a scripted config entry follows:

          #! /bin/sh
          awk '{
              print("# HELP loadavg local load average")
              print("# TYPE loadavg gauge")
              printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
              printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
              printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
          }' /proc/loadavg

       This script produces the following OpenMetrics-formatted metric
       data when run:

          # HELP loadavg local load average
          # TYPE loadavg gauge
          loadavg {interval="1-minute"} 0.12
          loadavg {interval="5-minute"} 0.27
          loadavg {interval="15-minute"} 0.54

       If the above script was saved and made executable in a file named
       $PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh then this
       would result in a new PCP metric named
       openmetrics.local.system.loadavg which would have three instances
       for the current load average values: 1-minute, 5-minute and
       15-minute.

       Scripted config entries may produce more than one PCP leaf metric
       name.  For example, the above "system.sh" script could also
       export other metrics such as CPU statistics, by reading
       /proc/stat on the local system.  Such additional metrics would
       appear as peer metrics in the same PCP metric subtree.  In the
       case of CPU counters, the metric type definition should be
       counter, not gauge.  For full details of the openmetrics
       exposition formats, see
       https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md .

SELinux CONSIDERATIONS         top

       Scripted config files are executed by the pmdaopenmetrics PMDA
       with the same SELinux context and policy as the local pmcd(1).
       For simple scripts, such as the load average example described
       above, this is normally fine.  However AVC errors may result for
       scripts that make library or system calls that are restricted by
       the prevailing SELinux context and policies.  In these cases it
       is not feasible to unilaterally grant pmcd or it's PMDAs an
       unconfined execution policy.  In these site specific cases it
       will be necessary to create a local SELinux policy module.  This
       can be done by capturing the AVC record(s) from the local audit
       log, generate a local policy module using audit2allow, and then
       load the new module using semodule, e.g. as follows :

              $ sudo grep '^type=AVC.*pcp' /var/log/audit/audit.log \
              | audit2allow -M mypolicy
              $ sudo semodule -i mypolicy.pp

       If these local policies need to be persistent across reboots,
       then a scriptlet similar to the above example may be added to the
       local pmcd RC file (typically /etc/pcp/pmcd/rc.local).  For
       further details, see audit2allow(1) and semodule(1).

METRIC NAMING         top

       All metrics from a file named JOB.*  will be exported as PCP
       metrics with the openmetrics.JOB metric name prefix.  Therefore,
       the JOB name must be a valid non-leaf name for PCP PMNS metric
       names.  If the JOB name has multiple dot-separated components,
       the resulting PMNS names will include those components and care
       is needed to ensure there are no overlapping definitions, e.g.
       metrics returned by JOB.response may overlap or conflict with
       metrics returned by JOB.response.time.

       Config file entries (URLs or scripts) found in subdirectories of
       the config directory will also result in hierarchical metric
       names.  For example, a config file named
       $PCP_PMDAS_DIR/openmetrics/config.d/mysource/latency/get.url will
       result in metrics being created (by fetching that source URL)
       below openmetrics.mysource.latency.get in the PCP namespace.
       Scripts found in subdirectories of the config directory similarly
       result in hierarchical PCP metric names.

DYNAMIC METRIC NAMES         top

       As described above, changes and new additions can be made to
       files in the configuration directory without having to restart
       the PMDA.  These changes are detected automatically and the PCP
       metric names below openmetrics in the PMNS will be updated
       accordingly, i.e. new metrics will be dynamically added and/or
       existing metrics removed.  In addition, pmdaopenmetrics honors
       the PMCD_NAMES_CHANGE pmFetch(3) protocol that was introduced in
       PCP version 4.0.  In particular, if openmetrics metrics are being
       logged by a PCP version 4.0 or later pmlogger(1), new metrics
       that appear as a result of changes in the PMDA configuration
       directory will automatically start to be logged, provided the
       root of the openmetrics PMDA namespace is configured for logging
       in the pmlogger configuration file.  See pmlogger(1) for details.
       An example of such a pmlogger configuration file is :

          log mandatory on 2 second {
               # log all metrics below the root of the openmetrics namespace
               openmetrics
          }

METADATA         top

       Metric data returned by URL or scripted configuration files may
       contain metadata that can be used by the openmetrics PMDA to
       specify the semantics, data type, scaling and units of
       dynamically created metrics.  This metadata is prefixed with #
       PCP5 or # PCP in the ingested metric data.  For additional
       information about PCP metadata, see pmLookupDesc(3) and
       pmParseUnitsStr(3) and examples in shipped configuration files.

       In-line "PCP5" metadata must be supplied by the metrics source
       end-pont (URL or script).  An alternative is to specify this in
       the URL configuration file directly, which has the advantage of
       not having to modify the source/end-point if the metadata is
       incorrect or missing.  Metadata specified in the URL
       configuration file over-rides any in-line metadata.

       The configuration file syntax for specifying metadata is:
       METADATA: regex type indom semantics units ... to EOL
       Where:
       METADATA: is literal
       regex is an extended regular expression to match one or more
       metric names returned by the URL,
       type is one of the PCP numeric types (32, u32, 64, u64, float or
       double),
       indom is an arbitrary instance domain name, or PM_INDOM_NULL,
       semantics is either counter, instant or discrete and
       units is either none or a string extending to end of line that is
       parseable by pmParseUnitsStr(3), i.e. the units, dimensions and
       scaling to be used for matching metrics.

       An example configuration file that ingests metrics from the
       Grafana /metrics end-point on localhost, filters out all metrics
       returned by that URL except for grafana_api_response_status_total
       and then specifies the metric type is an unsigned 32 bit integer
       with a non-singular instance domain named response and counter
       semantics with units of count.

       http://localhost:3000/metrics 
       FILTER: INCLUDE METRIC grafana_api_response_status_total
       FILTER: EXCLUDE METRIC .*
       METADATA: grafana_api_response_status_total u32 response counter
       count

       Note that the name in the indom field is presently ignored unless
       it is PM_INDOM_NULL, in which case the metric has no instance
       domain (i.e. singular), even if it has labels which would
       otherwise be used for instance naming.

CONTROL METRICS         top

       The PMDA maintains special control metrics, as described below.
       Apart from openmetrics.control.debug, each of these metrics has
       one instance for each configured metric source.  All of these
       metrics have integer values with counter semantics, except
       openmetrics.control.status, which has a string value.  It is
       important to note that fetching any of the openmetrics.control
       metrics will only update the counters and status values if the
       corresponding URL is actually fetched.  If the source URL is not
       fetched, the control metric values do not trigger a refresh and
       the control values reported represent the most recent fetch of
       each corresponding source.

       The instance domain for the openmetrics.control metrics is
       adjusted dynamically as new sources are discovered.  If there are
       no sources configured, the metric names are still defined but the
       instance domain will be empty and a fetch will return no values.

       openmetrics.control.status
              A string representing the status of the last fetch of the
              corresponding source.  This will generally be success for
              an http response code of 200.  This metric can be used for
              service availability monitoring - provided, as stated
              above, the corresponding source URL is fetched too.

       openmetrics.control.status_code
              This metric is similar to openmetrics.control.status
              except that it is the integer response code of the last
              fetch.  A value of 200 usually signifies success and any
              other value failure.  This metric can also be used for
              service availability monitoring, with the same caveats as
              openmetrics.control.status.

       openmetrics.control.calls
              total number of times each configured metric source has
              been fetched (if it's a URL) or executed (if it's a
              script), since the PMDA started.  This metric has counter
              semantics and would normally be converted to a rate/second
              by client tools.

       openmetrics.control.fetch_time
              Total time in milliseconds that each configured metric
              source has taken to return a document, excluding the time
              to parse the document.  This metric has counter semantics
              and would normally be rate converted by client tools but
              is also useful in raw form as the accumulated parse time
              since the PMDA was started.

       openmetrics.control.parse_time
              Total time in milliseconds that each configured metric
              source has taken to parse each document, excluding the
              time to fetch the document.  This metric has counter
              semantics and would normally be rate converted by client
              tools but is also useful in raw form as the accumulated
              parse time since the PMDA was started.

       When converted to a rate, the calls metric represents the average
       fetch rate of each source over the sampling interval (time delta
       between samples).  The fetch_time and parse_time counters, when
       converted to a rate, represent the average fetch and parsing
       latency (respectfully), during the sampling interval.

       The openmetrics.control.debug metric has a singular value,
       defaulting to 0.  If a non-zero value is stored into this metric
       using pmstore(1), additional debug messages will be written to
       the PMDA log file.

LIMITATIONS         top

       pmdaopenmetrics and libpcp internals impose some numerical
       constraints about the number of sources (4095), metrics (1024)
       within each source, and instances for each metric (4194304).

INSTALLATION         top

       Install the OpenMetrics PMDA by using the Install script as root:

           # cd $PCP_PMDAS_DIR/openmetrics
           # ./Install

       To uninstall, the following must be done as root:

           # cd $PCP_PMDAS_DIR/openmetrics
           # ./Remove

       pmdaopenmetrics is launched by pmcd(1) and should never be
       executed directly.  The Install and Remove scripts notify pmcd
       when the agent is installed or removed.

       When scripts and .url files are added, removed or changed in the
       configuration directory, it is usually not necessary to restart
       the PMDA - the changes will be detected and managed on subsequent
       requests to the PMDA.

FILES         top

       $PCP_PMDAS_DIR/openmetrics/Install
           installation script for the pmdaopenmetrics agent

       $PCP_PMDAS_DIR/openmetrics/Remove
           undo installation script for the pmdaopenmetrics agent

       $PCP_PMDAS_DIR/openmetrics/config.d/
           contains URLs and scripts used by the pmdaopenmetrics agent
           as sources of openmetrics metric data.

       $PCP_LOG_DIR/pmcd/openmetrics.log
           default log file for error messages from pmdaopenmetrics

       $PCP_VAR_DIR/config/144.*
           files containing internal tables for metric and instance ID
           number persistence (domain 144).

PCP ENVIRONMENT         top

       Environment variables with the prefix PCP_ are used to
       parameterize the file and directory names used by PCP.  On each
       installation, the file /etc/pcp.conf contains the local values
       for these variables.  The $PCP_CONF variable may be used to
       specify an alternative configuration file, as described in
       pcp.conf(5).

SEE ALSO         top

       PCPIntro(1), audit2allow(1), pmcd(1), pminfo(1), pmlogger(1),
       pmstore(1), PMWEBAPI(3), pmFetch(3), pmLookupLabels(3),
       semodule(1), and
       https://prometheus.io/docs/instrumenting/exposition_formats .

COLOPHON         top

       This page is part of the PCP (Performance Co-Pilot) project.
       Information about the project can be found at 
       ⟨http://www.pcp.io/⟩.  If you have a bug report for this manual
       page, send it to pcp@groups.io.  This page was obtained from the
       project's upstream Git repository
       ⟨https://github.com/performancecopilot/pcp.git⟩ on 2021-08-27.
       (At that time, the date of the most recent commit that was found
       in the repository was 2021-08-27.)  If you discover any rendering
       problems in this HTML version of the page, or you believe there
       is a better or more up-to-date source for the page, or you have
       corrections or improvements to the information in this COLOPHON
       (which is not part of the original manual page), send a mail to
       man-pages@man7.org

Performance Co-Pilot               PCP                PMDAOPENMETRICS(1)

Pages that refer to this page: htop(1)pmlogger(1)pmdasenderror(3)pmwebapi(3)