linux/tools/perf
Arnaldo Carvalho de Melo 0a6545bda2 perf trace: Show only failing syscalls
For instance:

  # perf probe "vfs_getname=getname_flags:72 pathname=result->name:string"
  Added new event:
    probe:vfs_getname    (on getname_flags:72 with pathname=result->name:string)

  You can now use it in all perf tools, such as:

	  perf record -e probe:vfs_getname -aR sleep 1

  # perf trace --failure sleep 1
     0.043 ( 0.010 ms): sleep/10978 access(filename: /etc/ld.so.preload, mode: R) = -1 ENOENT No such file or directory

For reference, here are all the syscalls in this case:

  # perf trace sleep 1
         ? (         ): sleep/10976  ... [continued]: execve()) = 0
       0.027 ( 0.001 ms): sleep/10976 brk() = 0x55bdc2d04000
       0.044 ( 0.010 ms): sleep/10976 access(filename: /etc/ld.so.preload, mode: R) = -1 ENOENT No such file or directory
       0.057 ( 0.006 ms): sleep/10976 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC) = 3
       0.064 ( 0.002 ms): sleep/10976 fstat(fd: 3, statbuf: 0x7fffac22b370) = 0
       0.067 ( 0.003 ms): sleep/10976 mmap(len: 111457, prot: READ, flags: PRIVATE, fd: 3) = 0x7feec8615000
       0.071 ( 0.001 ms): sleep/10976 close(fd: 3) = 0
       0.080 ( 0.007 ms): sleep/10976 openat(dfd: CWD, filename: /lib64/libc.so.6, flags: CLOEXEC) = 3
       0.088 ( 0.002 ms): sleep/10976 read(fd: 3, buf: 0x7fffac22b538, count: 832) = 832
       0.092 ( 0.001 ms): sleep/10976 fstat(fd: 3, statbuf: 0x7fffac22b3d0) = 0
       0.094 ( 0.002 ms): sleep/10976 mmap(len: 8192, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS) = 0x7feec8613000
       0.099 ( 0.004 ms): sleep/10976 mmap(len: 3889792, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3) = 0x7feec8057000
       0.104 ( 0.007 ms): sleep/10976 mprotect(start: 0x7feec8203000, len: 2097152) = 0
       0.112 ( 0.005 ms): sleep/10976 mmap(addr: 0x7feec8403000, len: 24576, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3, off: 1753088) = 0x7feec8403000
       0.120 ( 0.003 ms): sleep/10976 mmap(addr: 0x7feec8409000, len: 14976, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS|FIXED) = 0x7feec8409000
       0.128 ( 0.001 ms): sleep/10976 close(fd: 3) = 0
       0.139 ( 0.001 ms): sleep/10976 arch_prctl(option: 4098, arg2: 140663540761856) = 0
       0.186 ( 0.004 ms): sleep/10976 mprotect(start: 0x7feec8403000, len: 16384, prot: READ) = 0
       0.204 ( 0.003 ms): sleep/10976 mprotect(start: 0x55bdc0ec3000, len: 4096, prot: READ) = 0
       0.209 ( 0.004 ms): sleep/10976 mprotect(start: 0x7feec8631000, len: 4096, prot: READ) = 0
       0.214 ( 0.010 ms): sleep/10976 munmap(addr: 0x7feec8615000, len: 111457) = 0
       0.269 ( 0.001 ms): sleep/10976 brk() = 0x55bdc2d04000
       0.271 ( 0.002 ms): sleep/10976 brk(brk: 0x55bdc2d25000) = 0x55bdc2d25000
       0.274 ( 0.001 ms): sleep/10976 brk() = 0x55bdc2d25000
       0.278 ( 0.007 ms): sleep/10976 open(filename: /usr/lib/locale/locale-archive, flags: CLOEXEC) = 3
       0.288 ( 0.001 ms): sleep/10976 fstat(fd: 3</usr/lib/locale/locale-archive>, statbuf: 0x7feec8408aa0) = 0
       0.290 ( 0.003 ms): sleep/10976 mmap(len: 113045344, prot: READ, flags: PRIVATE, fd: 3) = 0x7feec1488000
       0.297 ( 0.001 ms): sleep/10976 close(fd: 3</usr/lib/locale/locale-archive>) = 0
       0.325 (1000.193 ms): sleep/10976 nanosleep(rqtp: 0x7fffac22c0b0) = 0
    1000.560 ( 0.006 ms): sleep/10976 close(fd: 1) = 0
    1000.573 ( 0.005 ms): sleep/10976 close(fd: 2) = 0
    1000.596 (         ): sleep/10976 exit_group()
  #

And can be done systemwide, etc, with backtraces:

  # perf trace --max-stack=16 --failure sleep 1
     0.048 ( 0.015 ms): sleep/11092 access(filename: /etc/ld.so.preload, mode: R) = -1 ENOENT No such file or directory
                                       __access (inlined)
                                       dl_main (/usr/lib64/ld-2.26.so)
  #

Or for some specific syscalls:

  # perf trace --max-stack=16 -e openat --failure cat /tmp/rien
  cat: /tmp/rien: No such file or directory
       0.251 ( 0.012 ms): cat/11106 openat(dfd: CWD, filename: /tmp/rien) = -1 ENOENT No such file or directory
                                         __libc_open64 (inlined)
                                         main (/usr/bin/cat)
                                         __libc_start_main (/usr/lib64/libc-2.26.so)
                                         _start (/usr/bin/cat)
  #

Look for inotify* syscalls that fail, system wide, for 2 seconds, with backtraces:

  # perf trace -a --max-stack=16 --failure -e inotify* sleep 2
   819.165 ( 0.058 ms): gmain/1724 inotify_add_watch(fd: 8<anon_inode:inotify>, pathname: /home/acme/~, mask: 16789454) = -1 ENOENT No such file or directory
                                       __GI_inotify_add_watch (inlined)
                                       _ik_watch (/usr/lib64/libgio-2.0.so.0.5400.3)
                                       _ip_start_watching (/usr/lib64/libgio-2.0.so.0.5400.3)
                                       im_scan_missing (/usr/lib64/libgio-2.0.so.0.5400.3)
                                       g_timeout_dispatch (/usr/lib64/libglib-2.0.so.0.5400.3)
                                       g_main_context_dispatch (/usr/lib64/libglib-2.0.so.0.5400.3)
                                       g_main_context_iterate.isra.23 (/usr/lib64/libglib-2.0.so.0.5400.3)
                                       g_main_context_iteration (/usr/lib64/libglib-2.0.so.0.5400.3)
                                       glib_worker_main (/usr/lib64/libglib-2.0.so.0.5400.3)
                                       g_thread_proxy (/usr/lib64/libglib-2.0.so.0.5400.3)
                                       start_thread (/usr/lib64/libpthread-2.26.so)
                                       __GI___clone (inlined)
  #

Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: https://lkml.kernel.org/n/tip-8f7d3mngaxvi7tlzloz3n7cs@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2018-04-02 07:57:37 -03:00
..
arch perf annotate: Pass function descriptor to its instruction parsing routines 2018-03-21 16:19:41 -03:00
bench perf perf: Remove duplicate includes 2017-12-27 12:15:49 -03:00
Documentation perf trace: Show only failing syscalls 2018-04-02 07:57:37 -03:00
jvmti perf jvmti: Generate correct debug information for inlined code 2017-12-18 11:54:08 -03:00
pmu-events perf vendor events s390: Add JSON files for IBM z14 2018-03-27 13:13:39 -03:00
python perf python: Make twatch.py work with both python2 and python3 2018-02-19 12:28:08 -03:00
scripts perf tools: Add Python 3 support 2018-02-19 12:28:23 -03:00
tests perf tests bp_account: Fix build with clang-6 2018-03-19 13:51:54 -03:00
trace perf trace beauty flock: Move to separate object file 2018-01-25 06:37:31 -03:00
ui perf annotate: Support jumping from one function to another 2018-03-23 16:46:18 -03:00
util perf tools: Add a "dso_size" sort order 2018-04-02 07:57:37 -03:00
.gitignore perf tools: Add trace/beauty/generated/ into .gitignore 2018-02-05 13:58:02 -03:00
Build perf trace: Remove audit-libs dependency if syscall tables are present 2018-01-23 09:51:38 -03:00
builtin-annotate.c perf annotate: Introduce --ignore-vmlinux command line option 2018-03-21 12:53:42 -03:00
builtin-bench.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
builtin-buildid-cache.c perf buildid-cache: Update help text for purge command 2017-11-16 14:49:54 -03:00
builtin-buildid-list.c Merge branch 'linus' into perf/core, to fix conflicts 2017-11-07 10:30:18 +01:00
builtin-c2c.c perf c2c report: Add cacheline address count column 2018-03-16 13:53:38 -03:00
builtin-config.c Merge branch 'linus' into perf/core, to fix conflicts 2017-11-07 10:30:18 +01:00
builtin-data.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
builtin-diff.c Merge branch 'linus' into perf/core, to fix conflicts 2017-11-07 10:30:18 +01:00
builtin-evlist.c Merge branch 'linus' into perf/core, to fix conflicts 2017-11-07 10:30:18 +01:00
builtin-ftrace.c perf ftrace: Append an EOL when write tracing files 2018-02-19 09:49:12 -03:00
builtin-help.c perf trace: Remove audit-libs dependency if syscall tables are present 2018-01-23 09:51:38 -03:00
builtin-inject.c perf tools: Get rid of unused 'swapped' parameter from perf_event__synthesize_sample() 2018-01-18 09:01:23 -03:00
builtin-kallsyms.c perf tools: Including missing inttypes.h header 2017-04-19 13:01:46 -03:00
builtin-kmem.c mm: remove __GFP_COLD 2017-11-15 18:21:06 -08:00
builtin-kvm.c perf mmap: Simplify perf_mmap__read_init() 2018-03-08 10:05:53 -03:00
builtin-list.c Merge branch 'linus' into perf/core, to fix conflicts 2017-11-07 10:30:18 +01:00
builtin-lock.c Merge branch 'linus' into perf/core, to fix conflicts 2017-11-07 10:30:18 +01:00
builtin-mem.c Merge branch 'linus' into perf/core, to fix conflicts 2017-11-07 10:30:18 +01:00
builtin-probe.c perf buildid-cache: Support binary objects from other namespaces 2017-07-18 23:14:11 -03:00
builtin-record.c perf record: Synthesize features before events in pipe mode 2018-03-16 13:56:50 -03:00
builtin-report.c perf report: Introduce --ignore-vmlinux command line option 2018-03-21 12:53:42 -03:00
builtin-sched.c perf sched map: Re-annotate shortname if thread comm changed 2018-03-07 10:22:26 -03:00
builtin-script.c perf tools: Fix snprint warnings for gcc 8 2018-03-19 10:00:43 -03:00
builtin-stat.c perf stat: Fix core dump when flag T is used 2018-03-16 13:55:29 -03:00
builtin-timechart.c perf tools: Add struct perf_data_file 2017-10-30 13:37:37 -03:00
builtin-top.c perf annotate: Move the default annotate options to the library 2018-03-21 12:53:40 -03:00
builtin-trace.c perf trace: Show only failing syscalls 2018-04-02 07:57:37 -03:00
builtin-version.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
builtin.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
check-headers.sh perf build: Fix check-headers.sh opts assignment 2018-03-27 13:13:38 -03:00
command-list.txt perf tools: Missing c2c command in command-list 2017-03-13 10:59:31 -03:00
CREDITS
design.txt
Makefile License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
Makefile.config perf tools arm64: Add libdw DWARF post unwind support for ARM64 2018-03-16 13:53:46 -03:00
Makefile.perf perf tools: Update tags with .cpp files 2018-03-08 11:30:47 -03:00
MANIFEST perf tools: Get all of tools/{arch,include}/ in the MANIFEST 2017-09-25 10:39:43 -03:00
perf-archive.sh License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
perf-completion.sh perf tools: Auto-complete for events with ':' 2017-12-27 12:16:00 -03:00
perf-read-vdso.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
perf-sys.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
perf-with-kcore.sh perf tools: Fix perf-with-kcore handling of arguments containing spaces 2015-08-06 16:48:27 -03:00
perf.c perf trace: Remove audit-libs dependency if syscall tables are present 2018-01-23 09:51:38 -03:00
perf.h perf record: Fix crash in pipe mode 2018-03-05 11:52:41 -03:00