perf-probe(1) — Linux manual page


PERF-PROBE(1)                  perf Manual                 PERF-PROBE(1)

NAME         top

       perf-probe - Define new dynamic tracepoints

SYNOPSIS         top

       perf probe [options] --add=PROBE [...]
       perf probe [options] PROBE
       perf probe [options] --del=[GROUP:]EVENT [...]
       perf probe --list[=[GROUP:]EVENT]
       perf probe [options] --line=LINE
       perf probe [options] --vars=PROBEPOINT
       perf probe [options] --funcs
       perf probe [options] --definition=PROBE [...]

DESCRIPTION         top

       This command defines dynamic tracepoint events, by symbol and
       registers without debuginfo, or by C expressions (C line numbers,
       C function names, and C local variables) with debuginfo.

OPTIONS         top

       -k, --vmlinux=PATH
           Specify vmlinux path which has debuginfo (Dwarf binary). Only
           when using this with --definition, you can give an offline
           vmlinux file.

       -m, --module=MODNAME|PATH
           Specify module name in which perf-probe searches probe points
           or lines. If a path of module file is passed, perf-probe
           treat it as an offline module (this means you can add a probe
           on a module which has not been loaded yet).

       -s, --source=PATH
           Specify path to kernel source.

       -v, --verbose
           Be more verbose (show parsed arguments, etc). Can not use
           with -q.

       -q, --quiet
           Do not show any warnings or messages. Can not use with -v.

       -a, --add=
           Define a probe event (see PROBE SYNTAX for detail).

       -d, --del=
           Delete probe events. This accepts glob wildcards(*, ?) and
           character classes(e.g. [a-z], [!A-Z]).

       -l, --list[=[GROUP:]EVENT]
           List up current probe events. This can also accept filtering
           patterns of event names. When this is used with --cache, perf
           shows all cached probes instead of the live probes.

       -L, --line=
           Show source code lines which can be probed. This needs an
           argument which specifies a range of the source code. (see
           LINE SYNTAX for detail)

       -V, --vars=
           Show available local variables at given probe point. The
           argument syntax is same as PROBE SYNTAX, but NO ARGs.

           (Only for --vars) Show external defined variables in addition
           to local variables.

           (Only for --add) Search only for non-inlined functions. The
           functions which do not have instances are ignored.

       -F, --funcs[=FILTER]
           Show available functions in given module or kernel. With
           -x/--exec, can also list functions in a user space executable
           / shared library. This also can accept a FILTER rule

       -D, --definition=
           Show trace-event definition converted from given probe-event
           instead of write it into tracing/[k,u]probe_events.

           (Only for --vars and --funcs) Set filter. FILTER is a
           combination of glob pattern, see FILTER PATTERN for detail.
           Default FILTER is "!k???tab_* & !crc_*" for --vars, and "!_*"
           for --funcs. If several filters are specified, only the last
           filter is used.

       -f, --force
           Forcibly add events with existing name.

       -n, --dry-run
           Dry run. With this option, --add and --del doesn’t execute
           actual adding and removal operations.

           (With --add) Cache the probes. Any events which successfully
           added are also stored in the cache file. (With --list) Show
           cached probes. (With --del) Remove cached probes.

           Set the maximum number of probe points for an event. Default
           is 128.

       --target-ns=PID: Obtain mount namespace information from the
       target pid. This is used when creating a uprobe for a process
       that resides in a different mount namespace from the perf(1)

       -x, --exec=PATH
           Specify path to the executable or shared library file for
           user space tracing. Can also be used with --funcs option.

           Demangle application symbols. --no-demangle is also available
           for disabling demangling.

           Demangle kernel symbols. --no-demangle-kernel is also
           available for disabling kernel demangling.

       In absence of -m/-x options, perf probe checks if the first
       argument after the options is an absolute path name. If its an
       absolute path, perf probe uses it as a target module/target user
       space binary to probe.

PROBE SYNTAX         top

       Probe points are defined by following syntax.

           1) Define event based on function name
            [[GROUP:]EVENT=]FUNC[@SRC][:RLN|+OFFS|%return|;PTN] [ARG ...]

           2) Define event based on source file with line number
            [[GROUP:]EVENT=]SRC:ALN [ARG ...]

           3) Define event based on source file with lazy pattern
            [[GROUP:]EVENT=]SRC;PTN [ARG ...]

           4) Pre-defined SDT events or cached event with name

       EVENT specifies the name of new event, if omitted, it will be set
       the name of the probed function, and for return probes, a
       "__return" suffix is automatically added to the function name.
       You can also specify a group name by GROUP, if omitted, set probe
       is used for kprobe and probe_<bin> is used for uprobe. Note that
       using existing group name can conflict with other events.
       Especially, using the group name reserved for kernel modules can
       hide embedded events in the modules. FUNC specifies a probed
       function name, and it may have one of the following options;
       +OFFS is the offset from function entry address in bytes, :RLN is
       the relative-line number from function entry line, and %return
       means that it probes function return. And ;PTN means lazy
       matching pattern (see LAZY MATCHING). Note that ;PTN must be the
       end of the probe point definition. In addition, @SRC specifies a
       source file which has that function. It is also possible to
       specify a probe point by the source line number or lazy matching
       by using SRC:ALN or SRC;PTN syntax, where SRC is the source file
       path, :ALN is the line number and ;PTN is the lazy matching
       pattern. ARG specifies the arguments of this probe point, (see
       PROBE ARGUMENT). SDTEVENT and PROVIDER is the pre-defined event
       name which is defined by user SDT (Statically Defined Tracing) or
       the pre-cached probes with event name. Note that before using the
       SDT event, the target binary (on which SDT events are defined)
       must be scanned by perf-buildid-cache(1) to make SDT events as
       cached events.

       For details of the SDT, see below. 


       In the probe syntax, =, @, +, : and ; are treated as a special
       character. You can use a backslash (\) to escape the special
       characters. This is useful if you need to probe on a specific
       versioned symbols, like @GLIBC_... suffixes, or also you need to
       specify a source file which includes the special characters. Note
       that usually single backslash is consumed by shell, so you might
       need to pass double backslash (\\) or wrapping with single quotes
       ('AAA\@BBB'). See EXAMPLES how it is used.

PROBE ARGUMENT         top

       Each probe argument follows below syntax.


       NAME specifies the name of this argument (optional). You can use
       the name of local variable, local data structure member (e.g.
       var→field, var.field2), local array with fixed index (e.g.
       array[1], var→array[0], var→pointer[2]), or kprobe-tracer
       argument format (e.g. $retval, %ax, etc). Note that the name of
       this argument will be set as the last member name if you specify
       a local data structure member (e.g. field2 for
       var→field1.field2.) $vars and $params special arguments are also
       available for NAME, $vars is expanded to the local variables
       (including function parameters) which can access at given probe
       point. $params is expanded to only the function parameters. TYPE
       casts the type of this argument (optional). If omitted, perf
       probe automatically set the type based on debuginfo (*).
       Currently, basic types (u8/u16/u32/u64/s8/s16/s32/s64),
       hexadecimal integers (x/x8/x16/x32/x64), signedness casting
       (u/s), "string" and bitfield are supported. (see TYPES for
       detail) On x86 systems %REG is always the short form of the
       register: for example %AX. %RAX or %EAX is not valid. "@user" is
       a special attribute which means the LOCALVAR will be treated as a
       user-space memory. This is only valid for kprobe event.

TYPES         top

       Basic types (u8/u16/u32/u64/s8/s16/s32/s64) and hexadecimal
       integers (x8/x16/x32/x64) are integer types. Prefix s and u means
       those types are signed and unsigned respectively, and x means
       that is shown in hexadecimal format. Traced arguments are shown
       in decimal (sNN/uNN) or hex (xNN). You can also use s or u to
       specify only signedness and leave its size auto-detected by perf
       probe. Moreover, you can use x to explicitly specify to be shown
       in hexadecimal (the size is also auto-detected). String type is a
       special type, which fetches a "null-terminated" string from
       kernel space. This means it will fail and store NULL if the
       string container has been paged out. You can specify string type
       only for the local variable or structure member which is an array
       of or a pointer to char or unsigned char type. Bitfield is
       another special type, which takes 3 parameters, bit-width,
       bit-offset, and container-size (usually 32). The syntax is;


LINE SYNTAX         top

       Line range is described by following syntax.


       FUNC specifies the function name of showing lines. RLN is the
       start line number from function entry line, and RLN2 is the end
       line number. As same as probe syntax, SRC means the source file
       path, ALN is start line number, and ALN2 is end line number in
       the file. It is also possible to specify how many lines to show
       by using NUM. Moreover, FUNC@SRC combination is good for
       searching a specific function when several functions share same
       name. So, "source.c:100-120" shows lines between 100th to 120th
       in source.c file. And "func:10+20" shows 20 lines from 10th line
       of func function.

LAZY MATCHING         top

       The lazy line matching is similar to glob matching but ignoring
       spaces in both of pattern and target. So this accepts
       wildcards(*, ?) and character classes(e.g. [a-z], [!A-Z]).

       e.g. a=* can matches a=b, a = b, a == b and so on.

       This provides some sort of flexibility and robustness to probe
       point definitions against minor code changes. For example, actual
       10th line of schedule() can be moved easily by modifying
       schedule(), but the same line matching rq=cpu_rq* may still exist
       in the function.)

FILTER PATTERN         top

       The filter pattern is a glob matching pattern(s) to filter
       variables. In addition, you can use "!" for specifying filter-out
       rule. You also can give several rules combined with "&" or "|",
       and fold those rules as one rule by using "(" ")".

       e.g. With --filter "foo* | bar*", perf probe -V shows variables
       which start with "foo" or "bar". With --filter "!foo* & *bar",
       perf probe -V shows variables which don’t start with "foo" and
       end with "bar", like "fizzbar". But "foobar" is filtered out.

EXAMPLES         top

       Display which lines in schedule() can be probed:

           ./perf probe --line schedule

       Add a probe on schedule() function 12th line with recording cpu
       local variable:

           ./perf probe schedule:12 cpu
           ./perf probe --add='schedule:12 cpu'

       Add one or more probes which has the name start with "schedule".

           ./perf probe schedule*
           ./perf probe --add='schedule*'

       Add probes on lines in schedule() function which calls

           ./perf probe 'schedule;update_rq_clock*'
           ./perf probe --add='schedule;update_rq_clock*'

       Delete all probes on schedule().

           ./perf probe --del='schedule*'

       Add probes at zfree() function on /bin/zsh

           ./perf probe -x /bin/zsh zfree or ./perf probe /bin/zsh zfree

       Add probes at malloc() function on libc

           ./perf probe -x /lib/ malloc or ./perf probe /lib/ malloc

       Add a uprobe to a target process running in a different mount

           ./perf probe --target-ns <target pid> -x /lib64/ malloc

       Add a USDT probe to a target process running in a different mount

           ./perf probe --target-ns <target pid> -x /usr/lib/jvm/java-1.8.0-openjdk- %sdt_hotspot:thread__sleep__end

       Add a probe on specific versioned symbol by backslash escape

           ./perf probe -x /lib64/ 'malloc_get_state\@GLIBC_2.2.5'

       Add a probe in a source file using special characters by
       backslash escape

           ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'


       Since perf probe depends on ftrace (tracefs) and kallsyms
       (/proc/kallsyms), you have to care about the permission and some
       sysctl knobs.

       •   Since tracefs and kallsyms requires root or privileged user
           to access it, the following perf probe commands also require
           it; --add, --del, --list (except for --cache option)

       •   The system admin can remount the tracefs with 755 (sudo mount
           -o remount,mode=755 /sys/kernel/tracing/) to allow
           unprivileged user to run the perf probe --list command.

       •   /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also
           prevents perf probe to retrieve the important information
           from kallsyms. You also need to set to 1 (restrict non
           CAP_SYSLOG users) for the above commands. Since the
           user-space probe doesn’t need to access kallsyms, this is
           only for probing the kernel function (kprobes).

       •   Since the perf probe commands read the vmlinux (for kernel)
           and/or the debuginfo file (including user-space application),
           you need to ensure that you can read those files.

SEE ALSO         top

       perf-trace(1), perf-record(1), perf-buildid-cache(1)

COLOPHON         top

       This page is part of the perf (Performance analysis tools for
       Linux (in Linux source tree)) project.  Information about the
       project can be found at 
       ⟨⟩.  If you have a
       bug report for this manual page, send it to  This page was obtained from the
       project's upstream Git repository
       on 2023-12-22.  (At that time, the date of the most recent commit
       that was found in the repository was 2023-12-21.)  If you
       discover any rendering problems in this HTML version of the page,
       or you believe there is a better or more up-to-date source for
       the page, or you have corrections or improvements to the
       information in this COLOPHON (which is not part of the original
       manual page), send a mail to

perf                           2023-01-19                  PERF-PROBE(1)

Pages that refer to this page: perf(1)perf-buildid-cache(1)