2
0
mirror of https://github.com/edk2-porting/linux-next.git synced 2024-12-29 23:53:55 +08:00
linux-next/tools/perf/util/probe-finder.h

122 lines
3.3 KiB
C
Raw Normal View History

perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
#ifndef _PROBE_FINDER_H
#define _PROBE_FINDER_H
#include <stdbool.h>
#include "intlist.h"
#include "probe-event.h"
#include "sane_ctype.h"
#define MAX_PROBE_BUFFER 1024
#define MAX_PROBES 128
#define MAX_PROBE_ARGS 128
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
perf probe: Support $params special probe argument $params is similar to $vars but matches only function parameters not local variables. Thus, this is useful for tracing function parameter changing or tracing function call with parameters. Testing it: # perf probe tcp_sendmsg '$params' Added new event: probe:tcp_sendmsg (on tcp_sendmsg with $params) You can now use it in all perf tools, such as: perf record -e probe:tcp_sendmsg -aR sleep 1 # perf probe -l probe:tcp_sendmsg (on tcp_sendmsg@acme/git/linux/net/ipv4/tcp.c with iocb sk msg size) # perf record -a -e probe:* press some random letters to generate TCP (sshd) traffic... ^C[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.223 MB perf.data (6 samples) ] # perf script sshd 6385 [2] 3.907529: probe:tcp_sendmsg: iocb=0xffff8800ac4cfe70 sk=0xffff88042196c140 msg=0xffff8800ac4cfda8 size=0x24 sshd 6385 [2] 4.138973: probe:tcp_sendmsg: iocb=0xffff8800ac4cfe70 sk=0xffff88042196c140 msg=0xffff8800ac4cfda8 size=0x24 sshd 6385 [2] 4.378966: probe:tcp_sendmsg: iocb=0xffff8800ac4cfe70 sk=0xffff88042196c140 msg=0xffff8800ac4cfda8 size=0x24 sshd 6385 [2] 4.603681: probe:tcp_sendmsg: iocb=0xffff8800ac4cfe70 sk=0xffff88042196c140 msg=0xffff8800ac4cfda8 size=0x24 sshd 6385 [2] 4.818455: probe:tcp_sendmsg: iocb=0xffff8800ac4cfe70 sk=0xffff88042196c140 msg=0xffff8800ac4cfda8 size=0x24 sshd 6385 [2] 5.043603: probe:tcp_sendmsg: iocb=0xffff8800ac4cfe70 sk=0xffff88042196c140 msg=0xffff8800ac4cfda8 size=0x24 # cat /sys/kernel/debug/tracing/events/probe/tcp_sendmsg/format name: tcp_sendmsg ID: 1927 format: field:unsigned short common_type; offset:0; size:2; signed:0; field:unsigned char common_flags; offset:2; size:1; signed:0; field:unsigned char common_preempt_count; offset:3; size:1; signed:0; field:int common_pid; offset:4; size:4; signed:1; field:unsigned long __probe_ip; offset:8; size:8; signed:0; field:u64 iocb; offset:16; size:8; signed:0; field:u64 sk; offset:24; size:8; signed:0; field:u64 msg; offset:32; size:8; signed:0; field:u64 size; offset:40; size:8; signed:0; print fmt: "(%lx) iocb=0x%Lx sk=0x%Lx msg=0x%Lx size=0x%Lx", REC->__probe_ip, REC->iocb, REC->sk, REC->msg, REC->size # Do some system wide tracing of this probe + write syscalls: # perf trace -e write --ev probe:* --filter-pids 6385 462.612 (0.010 ms): bash/19153 write(fd: 1</dev/pts/1>, buf: 0x7f7556c78000, count: 29 ) = 29 462.701 (0.027 ms): sshd/19152 write(fd: 3<socket:[63117]>, buf: 0x7f78dd12e160, count: 68 ) ... 462.701 ( ): probe:tcp_sendmsg:(ffffffff8163db30) iocb=0xffff8803ebec7e70 sk=0xffff88042196ab80 msg=0xffff8803ebec7da8 size=0x44) 462.710 (0.035 ms): sshd/19152 ... [continued]: write()) = 68 462.787 (0.009 ms): bash/19153 write(fd: 2</dev/pts/1>, buf: 0x7f7556c77000, count: 22 ) = 22 462.865 (0.002 ms): sshd/19152 write(fd: 3<socket:[63117]>, buf: 0x7f78dd12e160, count: 68 ) ... 462.865 ( ): probe:tcp_sendmsg:(ffffffff8163db30) iocb=0xffff8803ebec7e70 sk=0xffff88042196ab80 msg=0xffff8803ebec7da8 size=0x44) 462.873 (0.010 ms): sshd/19152 ... [continued]: write()) = 68 Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: David Ahern <dsahern@gmail.com> Cc: Hemant Kumar <hemant@linux.vnet.ibm.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/20150506124653.4961.59806.stgit@localhost.localdomain [ Add some examples to the changelog message showing how to use it ] Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2015-05-06 20:46:53 +08:00
#define PROBE_ARG_VARS "$vars"
#define PROBE_ARG_PARAMS "$params"
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
static inline int is_c_varname(const char *name)
{
/* TODO */
return isalpha(name[0]) || name[0] == '_';
}
#ifdef HAVE_DWARF_SUPPORT
#include "dwarf-aux.h"
/* TODO: export debuginfo data structure even if no dwarf support */
/* debug information structure */
struct debuginfo {
Dwarf *dbg;
perf probe: Fix probing symbols with optimization suffix Fix perf probe to probe on some symbols which have some optimzation suffixes, e.g. ".part", ".isra", and ".constprop". To fix this issue, instead of using the DIE name, perf probe uses the symbol name found by dwfl_module_addrsym(). This also involves a perf probe --vars operation update which now shows the symbol name instead of the DIE name. Without this patch, putting a probe on an inlined function which was compiled with a suffixed symbol will fail like this: $ perf probe -v getname_flags probe-definition(0): getname_flags symbol:getname_flags file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (6 entries long) Using /lib/modules/3.11.0+/build/vmlinux for symbols found inline addr: 0xffffffff8119bb70 Probe point found: getname_flags+0 found inline addr: 0xffffffff8119bcb6 Probe point found: getname+6 found inline addr: 0xffffffff811a06a6 Probe point found: user_path_at_empty+6 find 3 probe_trace_events. Opening /sys/kernel/debug//tracing/kprobe_events write=1 Added new events: Writing event: p:probe/getname_flags getname_flags+0 Failed to write event: No such file or directory Error: Failed to add events. (-1) Because the debuginfo knows only the original (non suffix) symbol name, it uses the original symbol for probe address but the kernel (kallsyms) knows only suffixed symbol. Then, the kernel rejects that original symbol. This patch uses dwfl_module_addrsym() to get the correct (suffixed) symbol from symtab when a probe point is found. Reported-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/20130925131616.31632.46658.stgit@udc4-manage.rcp.hitachi.co.jp Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-09-25 21:16:16 +08:00
Dwfl_Module *mod;
Dwfl *dwfl;
Dwarf_Addr bias;
};
/* This also tries to open distro debuginfo */
struct debuginfo *debuginfo__new(const char *path);
void debuginfo__delete(struct debuginfo *dbg);
/* Find probe_trace_events specified by perf_probe_event from debuginfo */
int debuginfo__find_trace_events(struct debuginfo *dbg,
struct perf_probe_event *pev,
struct probe_trace_event **tevs);
/* Find a perf_probe_point from debuginfo */
int debuginfo__find_probe_point(struct debuginfo *dbg, unsigned long addr,
struct perf_probe_point *ppt);
int debuginfo__get_text_offset(struct debuginfo *dbg, Dwarf_Addr *offs,
bool adjust_offset);
/* Find a line range */
int debuginfo__find_line_range(struct debuginfo *dbg, struct line_range *lr);
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
/* Find available variables */
int debuginfo__find_available_vars_at(struct debuginfo *dbg,
struct perf_probe_event *pev,
struct variable_list **vls);
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
/* Find a src file from a DWARF tag path */
int get_real_path(const char *raw_path, const char *comp_dir,
char **new_path);
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
struct probe_finder {
struct perf_probe_event *pev; /* Target probe event */
/* Callback when a probe point is found */
int (*callback)(Dwarf_Die *sc_die, struct probe_finder *pf);
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
/* For function searching */
int lno; /* Line number */
Dwarf_Addr addr; /* Address */
const char *fname; /* Real file name */
Dwarf_Die cu_die; /* Current CU */
perf probe: Add fastpath to do lookup by function name v3 -> v2: - Make pubname_search_cb more generic - Add fastpath to find_probes also v2 -> v1: - Don't compare file names with cu_find_realpath(...), instead, compare them with the name returned by dwarf_decl_file(sp_die) The vmlinux file may have thousands of CUs. We can lookup function name from .debug_pubnames section to avoid the slow loop on CUs. 1. Improvement data for find_line_range ./perf stat -e cycles -r 10 -- ./perf probe -k /home/mlin/vmlinux \ -s /home/mlin/linux-2.6 \ --line csum_partial_copy_to_user > tmp.log before patch applied ===================== 847,988,276 cycles 0.355075856 seconds time elapsed after patch applied ===================== 206,102,622 cycles 0.086883555 seconds time elapsed 2. Improvement data for find_probes ./perf stat -e cycles -r 10 -- ./perf probe -k /home/mlin/vmlinux \ -s /home/mlin/linux-2.6 \ --vars csum_partial_copy_to_user > tmp.log before patch applied ===================== 848,490,844 cycles 0.355307901 seconds time elapsed after patch applied ===================== 205,684,469 cycles 0.086694010 seconds time elapsed Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: linux-kernel <linux-kernel@vger.kernel.org> LKML-Reference: <1301041668.14111.52.camel@minggr.sh.intel.com> Signed-off-by: Lin Ming <ming.m.lin@intel.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2011-03-25 16:27:48 +08:00
Dwarf_Die sp_die;
struct intlist *lcache; /* Line cache for lazy match */
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
/* For variable searching */
#if _ELFUTILS_PREREQ(0, 142)
perf probe: Search both .eh_frame and .debug_frame sections for probe location 'perf probe' through debuginfo__find_probes() in util/probe-finder.c checks for the functions' frame descriptions in either .eh_frame section of an ELF or the .debug_frame. The check is based on whether either one of these sections is present. Depending on distro, toolchain defaults, architetcutre, build flags, etc., CFI might be found in either .eh_frame and/or .debug_frame. Sometimes, it may happen that, .eh_frame, even if present, may not be complete and may miss some descriptions. Therefore, to be sure, to find the CFI covering an address we will always have to investigate both if available. For e.g., in powerpc, this may happen: $ gcc -g bin.c -o bin $ objdump --dwarf ./bin <1><145>: Abbrev Number: 7 (DW_TAG_subprogram) <146> DW_AT_external : 1 <146> DW_AT_name : (indirect string, offset: 0x9e): main <14a> DW_AT_decl_file : 1 <14b> DW_AT_decl_line : 39 <14c> DW_AT_prototyped : 1 <14c> DW_AT_type : <0x57> <150> DW_AT_low_pc : 0x100007b8 If the .eh_frame and .debug_frame are checked for the same binary, we will find that, .eh_frame (although present) doesn't contain a description for "main" function. But, .debug_frame has a description: 000000d8 00000024 00000000 FDE cie=00000000 pc=100007b8..10000838 DW_CFA_advance_loc: 16 to 100007c8 DW_CFA_def_cfa_offset: 144 DW_CFA_offset_extended_sf: r65 at cfa+16 ... Due to this (since, perf checks whether .eh_frame is present and goes on searching for that address inside that frame), perf is unable to process the probes: # perf probe -x ./bin main Failed to get call frame on 0x100007b8 Error: Failed to add events. To avoid this issue, we need to check both the sections (.eh_frame and .debug_frame), which is done in this patch. Note that, we can always force everything into both .eh_frame and .debug_frame by: $ gcc bin.c -fasynchronous-unwind-tables -fno-dwarf2-cfi-asm -g -o bin Signed-off-by: Hemant Kumar <hemant@linux.vnet.ibm.com> Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: linuxppc-dev@lists.ozlabs.org Cc: Mark Wielaard <mjw@redhat.com> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com> Link: http://lkml.kernel.org/r/1454426806-13974-1-git-send-email-hemant@linux.vnet.ibm.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2016-02-02 23:26:46 +08:00
/* Call Frame Information from .eh_frame */
Dwarf_CFI *cfi_eh;
/* Call Frame Information from .debug_frame */
Dwarf_CFI *cfi_dbg;
#endif
Dwarf_Op *fb_ops; /* Frame base attribute */
unsigned int machine; /* Target machine arch */
struct perf_probe_arg *pvar; /* Current target variable */
struct probe_trace_arg *tvar; /* Current result variable */
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
};
perf probe: Support --line option to show probable source-code lines Add --line option to support showing probable source-code lines. perf probe --line SRC:LN[-LN|+NUM] or perf probe --line FUNC[:LN[-LN|+NUM]] This option shows source-code with line number if the line can be probed. Lines without line number (and blue color) means that the line can not be probed, because debuginfo doesn't have the information of those lines. The argument specifies the range of lines, "source.c:100-120" shows lines between 100th to l20th in source.c file. And "func:10+20" shows 20 lines from 10th line of func function. e.g. # ./perf probe --line kernel/sched.c:1080 <kernel/sched.c:1080> * * called with rq->lock held and irqs disabled */ static void hrtick_start(struct rq *rq, u64 delay) { struct hrtimer *timer = &rq->hrtick_timer; 1086 ktime_t time = ktime_add_ns(timer->base->get_time(), delay); hrtimer_set_expires(timer, time); 1090 if (rq == this_rq()) { 1091 hrtimer_restart(timer); 1092 } else if (!rq->hrtick_csd_pending) { 1093 __smp_call_function_single(cpu_of(rq), &rq->hrtick_csd, 1094 rq->hrtick_csd_pending = 1; If you specifying function name, this shows function-relative line number. # ./perf probe --line schedule <schedule:0> asmlinkage void __sched schedule(void) 1 { struct task_struct *prev, *next; unsigned long *switch_count; struct rq *rq; int cpu; need_resched: preempt_disable(); 9 cpu = smp_processor_id(); 10 rq = cpu_rq(cpu); 11 rcu_sched_qs(cpu); 12 prev = rq->curr; 13 switch_count = &prev->nivcsw; Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> LKML-Reference: <20100106144534.27218.77939.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>
2010-01-06 22:45:34 +08:00
struct trace_event_finder {
struct probe_finder pf;
perf probe: Fix probing symbols with optimization suffix Fix perf probe to probe on some symbols which have some optimzation suffixes, e.g. ".part", ".isra", and ".constprop". To fix this issue, instead of using the DIE name, perf probe uses the symbol name found by dwfl_module_addrsym(). This also involves a perf probe --vars operation update which now shows the symbol name instead of the DIE name. Without this patch, putting a probe on an inlined function which was compiled with a suffixed symbol will fail like this: $ perf probe -v getname_flags probe-definition(0): getname_flags symbol:getname_flags file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (6 entries long) Using /lib/modules/3.11.0+/build/vmlinux for symbols found inline addr: 0xffffffff8119bb70 Probe point found: getname_flags+0 found inline addr: 0xffffffff8119bcb6 Probe point found: getname+6 found inline addr: 0xffffffff811a06a6 Probe point found: user_path_at_empty+6 find 3 probe_trace_events. Opening /sys/kernel/debug//tracing/kprobe_events write=1 Added new events: Writing event: p:probe/getname_flags getname_flags+0 Failed to write event: No such file or directory Error: Failed to add events. (-1) Because the debuginfo knows only the original (non suffix) symbol name, it uses the original symbol for probe address but the kernel (kallsyms) knows only suffixed symbol. Then, the kernel rejects that original symbol. This patch uses dwfl_module_addrsym() to get the correct (suffixed) symbol from symtab when a probe point is found. Reported-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/20130925131616.31632.46658.stgit@udc4-manage.rcp.hitachi.co.jp Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-09-25 21:16:16 +08:00
Dwfl_Module *mod; /* For solving symbols */
struct probe_trace_event *tevs; /* Found trace events */
int ntevs; /* Number of trace events */
int max_tevs; /* Max number of trace events */
};
struct available_var_finder {
struct probe_finder pf;
perf probe: Fix probing symbols with optimization suffix Fix perf probe to probe on some symbols which have some optimzation suffixes, e.g. ".part", ".isra", and ".constprop". To fix this issue, instead of using the DIE name, perf probe uses the symbol name found by dwfl_module_addrsym(). This also involves a perf probe --vars operation update which now shows the symbol name instead of the DIE name. Without this patch, putting a probe on an inlined function which was compiled with a suffixed symbol will fail like this: $ perf probe -v getname_flags probe-definition(0): getname_flags symbol:getname_flags file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (6 entries long) Using /lib/modules/3.11.0+/build/vmlinux for symbols found inline addr: 0xffffffff8119bb70 Probe point found: getname_flags+0 found inline addr: 0xffffffff8119bcb6 Probe point found: getname+6 found inline addr: 0xffffffff811a06a6 Probe point found: user_path_at_empty+6 find 3 probe_trace_events. Opening /sys/kernel/debug//tracing/kprobe_events write=1 Added new events: Writing event: p:probe/getname_flags getname_flags+0 Failed to write event: No such file or directory Error: Failed to add events. (-1) Because the debuginfo knows only the original (non suffix) symbol name, it uses the original symbol for probe address but the kernel (kallsyms) knows only suffixed symbol. Then, the kernel rejects that original symbol. This patch uses dwfl_module_addrsym() to get the correct (suffixed) symbol from symtab when a probe point is found. Reported-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/20130925131616.31632.46658.stgit@udc4-manage.rcp.hitachi.co.jp Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-09-25 21:16:16 +08:00
Dwfl_Module *mod; /* For solving symbols */
struct variable_list *vls; /* Found variable lists */
int nvls; /* Number of variable lists */
int max_vls; /* Max no. of variable lists */
bool child; /* Search child scopes */
};
perf probe: Support --line option to show probable source-code lines Add --line option to support showing probable source-code lines. perf probe --line SRC:LN[-LN|+NUM] or perf probe --line FUNC[:LN[-LN|+NUM]] This option shows source-code with line number if the line can be probed. Lines without line number (and blue color) means that the line can not be probed, because debuginfo doesn't have the information of those lines. The argument specifies the range of lines, "source.c:100-120" shows lines between 100th to l20th in source.c file. And "func:10+20" shows 20 lines from 10th line of func function. e.g. # ./perf probe --line kernel/sched.c:1080 <kernel/sched.c:1080> * * called with rq->lock held and irqs disabled */ static void hrtick_start(struct rq *rq, u64 delay) { struct hrtimer *timer = &rq->hrtick_timer; 1086 ktime_t time = ktime_add_ns(timer->base->get_time(), delay); hrtimer_set_expires(timer, time); 1090 if (rq == this_rq()) { 1091 hrtimer_restart(timer); 1092 } else if (!rq->hrtick_csd_pending) { 1093 __smp_call_function_single(cpu_of(rq), &rq->hrtick_csd, 1094 rq->hrtick_csd_pending = 1; If you specifying function name, this shows function-relative line number. # ./perf probe --line schedule <schedule:0> asmlinkage void __sched schedule(void) 1 { struct task_struct *prev, *next; unsigned long *switch_count; struct rq *rq; int cpu; need_resched: preempt_disable(); 9 cpu = smp_processor_id(); 10 rq = cpu_rq(cpu); 11 rcu_sched_qs(cpu); 12 prev = rq->curr; 13 switch_count = &prev->nivcsw; Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> LKML-Reference: <20100106144534.27218.77939.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>
2010-01-06 22:45:34 +08:00
struct line_finder {
struct line_range *lr; /* Target line range */
const char *fname; /* File name */
int lno_s; /* Start line number */
int lno_e; /* End line number */
Dwarf_Die cu_die; /* Current CU */
perf probe: Add fastpath to do lookup by function name v3 -> v2: - Make pubname_search_cb more generic - Add fastpath to find_probes also v2 -> v1: - Don't compare file names with cu_find_realpath(...), instead, compare them with the name returned by dwarf_decl_file(sp_die) The vmlinux file may have thousands of CUs. We can lookup function name from .debug_pubnames section to avoid the slow loop on CUs. 1. Improvement data for find_line_range ./perf stat -e cycles -r 10 -- ./perf probe -k /home/mlin/vmlinux \ -s /home/mlin/linux-2.6 \ --line csum_partial_copy_to_user > tmp.log before patch applied ===================== 847,988,276 cycles 0.355075856 seconds time elapsed after patch applied ===================== 206,102,622 cycles 0.086883555 seconds time elapsed 2. Improvement data for find_probes ./perf stat -e cycles -r 10 -- ./perf probe -k /home/mlin/vmlinux \ -s /home/mlin/linux-2.6 \ --vars csum_partial_copy_to_user > tmp.log before patch applied ===================== 848,490,844 cycles 0.355307901 seconds time elapsed after patch applied ===================== 205,684,469 cycles 0.086694010 seconds time elapsed Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: linux-kernel <linux-kernel@vger.kernel.org> LKML-Reference: <1301041668.14111.52.camel@minggr.sh.intel.com> Signed-off-by: Lin Ming <ming.m.lin@intel.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2011-03-25 16:27:48 +08:00
Dwarf_Die sp_die;
perf probe: Support --line option to show probable source-code lines Add --line option to support showing probable source-code lines. perf probe --line SRC:LN[-LN|+NUM] or perf probe --line FUNC[:LN[-LN|+NUM]] This option shows source-code with line number if the line can be probed. Lines without line number (and blue color) means that the line can not be probed, because debuginfo doesn't have the information of those lines. The argument specifies the range of lines, "source.c:100-120" shows lines between 100th to l20th in source.c file. And "func:10+20" shows 20 lines from 10th line of func function. e.g. # ./perf probe --line kernel/sched.c:1080 <kernel/sched.c:1080> * * called with rq->lock held and irqs disabled */ static void hrtick_start(struct rq *rq, u64 delay) { struct hrtimer *timer = &rq->hrtick_timer; 1086 ktime_t time = ktime_add_ns(timer->base->get_time(), delay); hrtimer_set_expires(timer, time); 1090 if (rq == this_rq()) { 1091 hrtimer_restart(timer); 1092 } else if (!rq->hrtick_csd_pending) { 1093 __smp_call_function_single(cpu_of(rq), &rq->hrtick_csd, 1094 rq->hrtick_csd_pending = 1; If you specifying function name, this shows function-relative line number. # ./perf probe --line schedule <schedule:0> asmlinkage void __sched schedule(void) 1 { struct task_struct *prev, *next; unsigned long *switch_count; struct rq *rq; int cpu; need_resched: preempt_disable(); 9 cpu = smp_processor_id(); 10 rq = cpu_rq(cpu); 11 rcu_sched_qs(cpu); 12 prev = rq->curr; 13 switch_count = &prev->nivcsw; Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> LKML-Reference: <20100106144534.27218.77939.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>
2010-01-06 22:45:34 +08:00
int found;
};
#endif /* HAVE_DWARF_SUPPORT */
perf: Add perf probe subcommand, a kprobe-event setup helper Add perf probe subcommand that implements a kprobe-event setup helper to the perf command. This allows user to define kprobe events using C expressions (C line numbers, C function names, and C local variables). Usage ----- perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...] -k, --vmlinux <file> vmlinux/module pathname -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]> probe point definition, where p: kprobe probe r: kretprobe probe GRP: Group name (optional) NAME: Event name FUNC: Function name OFFS: Offset from function entry (in byte) SRC: Source code path LINE: Line number ARG: Probe argument (local variable name or kprobe-tracer argument format is supported.) Changes in v4: - Add _GNU_SOURCE macro for strndup(). Changes in v3: - Remove -r option because perf always be used for online kernel. - Check malloc/calloc results. Changes in v2: - Check synthesized string length. - Rename perf kprobe to perf probe. - Use spaces for separator and update usage comment. - Check error paths in parse_probepoint(). - Check optimized-out variables. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> Cc: Jim Keniston <jkenisto@us.ibm.com> Cc: Frank Ch. Eigler <fche@redhat.com> LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
2009-10-09 05:17:38 +08:00
#endif /*_PROBE_FINDER_H */