linux/kernel/trace
Petr Mladek d303de1fcf tracing: Initialize iter->seq after zeroing in tracing_read_pipe()
A customer reported the following softlockup:

[899688.160002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [test.sh:16464]
[899688.160002] CPU: 0 PID: 16464 Comm: test.sh Not tainted 4.12.14-6.23-azure #1 SLE12-SP4
[899688.160002] RIP: 0010:up_write+0x1a/0x30
[899688.160002] Kernel panic - not syncing: softlockup: hung tasks
[899688.160002] RIP: 0010:up_write+0x1a/0x30
[899688.160002] RSP: 0018:ffffa86784d4fde8 EFLAGS: 00000257 ORIG_RAX: ffffffffffffff12
[899688.160002] RAX: ffffffff970fea00 RBX: 0000000000000001 RCX: 0000000000000000
[899688.160002] RDX: ffffffff00000001 RSI: 0000000000000080 RDI: ffffffff970fea00
[899688.160002] RBP: ffffffffffffffff R08: ffffffffffffffff R09: 0000000000000000
[899688.160002] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8b59014720d8
[899688.160002] R13: ffff8b59014720c0 R14: ffff8b5901471090 R15: ffff8b5901470000
[899688.160002]  tracing_read_pipe+0x336/0x3c0
[899688.160002]  __vfs_read+0x26/0x140
[899688.160002]  vfs_read+0x87/0x130
[899688.160002]  SyS_read+0x42/0x90
[899688.160002]  do_syscall_64+0x74/0x160

It caught the process in the middle of trace_access_unlock(). There is
no loop. So, it must be looping in the caller tracing_read_pipe()
via the "waitagain" label.

Crashdump analyze uncovered that iter->seq was completely zeroed
at this point, including iter->seq.seq.size. It means that
print_trace_line() was never able to print anything and
there was no forward progress.

The culprit seems to be in the code:

	/* reset all but tr, trace, and overruns */
	memset(&iter->seq, 0,
	       sizeof(struct trace_iterator) -
	       offsetof(struct trace_iterator, seq));

It was added by the commit 53d0aa7730 ("ftrace:
add logic to record overruns"). It was v2.6.27-rc1.
It was the time when iter->seq looked like:

     struct trace_seq {
	unsigned char		buffer[PAGE_SIZE];
	unsigned int		len;
     };

There was no "size" variable and zeroing was perfectly fine.

The solution is to reinitialize the structure after or without
zeroing.

Link: http://lkml.kernel.org/r/20191011142134.11997-1-pmladek@suse.com

Signed-off-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
2019-10-12 20:49:34 -04:00
..
blktrace.c blktrace: no need to check return value of debugfs_create functions 2019-06-03 15:39:39 +02:00
bpf_trace.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2019-09-28 17:47:33 -07:00
fgraph.c ftrace: Look up the address of return_to_handler() using helpers 2019-09-18 12:24:47 +10:00
ftrace_internal.h treewide: Rename rcu_dereference_raw_notrace() to _check() 2019-08-01 14:16:21 -07:00
ftrace.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
Kconfig Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next 2019-09-18 12:34:53 -07:00
Makefile tracing: Add unified dynamic event framework 2018-12-08 20:54:09 -05:00
power-traces.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
preemptirq_delay_test.c tracing: Use trace_clock_local() for looping in preemptirq_delay_test.c 2018-10-17 15:35:33 -04:00
ring_buffer_benchmark.c tracing: Use CONFIG_PREEMPTION 2019-07-31 19:03:35 +02:00
ring_buffer.c ring-buffer: Remove HAVE_64BIT_ALIGNED_ACCESS 2019-05-28 09:36:19 -04:00
rpm-traces.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
trace_benchmark.c rcu: Rename cond_resched_rcu_qs() to cond_resched_tasks_rcu_qs() 2018-05-15 10:27:29 -07:00
trace_benchmark.h tracing: Fix SPDX format headers to use C++ style comments 2018-08-16 19:08:06 -04:00
trace_branch.c x86/uaccess, ftrace: Fix ftrace_likely_update() vs. SMAP 2019-04-03 11:02:24 +02:00
trace_clock.c tracing: Add SPDX License format tags to tracing files 2018-08-16 19:08:06 -04:00
trace_dynevent.c tracing: Add tracing_check_open_get_tr() 2019-10-12 20:44:07 -04:00
trace_dynevent.h tracing/dynevent: Pass extra arguments to match operation 2019-08-31 12:19:38 -04:00
trace_entries.h tracing: Change the function format to display function names by perf 2019-02-11 14:53:43 -05:00
trace_event_perf.c tracing: Pass type into tracing_generic_entry_update() 2019-07-16 15:14:48 -04:00
trace_events_filter_test.h tracing: Fix SPDX format headers to use C++ style comments 2018-08-16 19:08:06 -04:00
trace_events_filter.c tracing: Have error path in predicate_parse() free its allocated memory 2019-09-28 17:13:39 -04:00
trace_events_hist.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_events_trigger.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_events.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_export.c tracing: Add trigger file for trace_markers tracefs/ftrace/print 2018-05-29 08:28:53 -04:00
trace_functions_graph.c fgraph: Remove redundant ftrace_graph_notrace_addr() test 2019-07-30 21:50:03 -04:00
trace_functions.c tracing: Update stack trace skipping for ORC unwinder 2018-01-23 15:57:00 -05:00
trace_hwlat.c tracing/hwlat: Don't ignore outer-loop duration when calculating max_latency 2019-10-12 20:49:33 -04:00
trace_irqsoff.c The biggest change for this release is in the histogram code. 2019-03-11 17:01:32 -07:00
trace_kdb.c tracing: Silence GCC 9 array bounds warning 2019-05-25 23:04:30 -04:00
trace_kprobe_selftest.c selftest/ftrace: Move kprobe selftest function to separate compile unit 2018-07-30 18:41:04 -04:00
trace_kprobe_selftest.h tracing: Fix SPDX format headers to use C++ style comments 2018-08-16 19:08:06 -04:00
trace_kprobe.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_mmiotrace.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
trace_nop.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
trace_output.c tracing: Be more clever when dumping hex in __print_hex() 2019-09-17 11:21:28 -04:00
trace_output.h tracing: Fix SPDX format headers to use C++ style comments 2018-08-16 19:08:06 -04:00
trace_preemptirq.c kprobes: Prohibit probing on hardirq tracers 2019-02-13 08:16:40 +01:00
trace_printk.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_probe_tmpl.h tracing/probe: Support user-space dereference 2019-05-25 23:04:42 -04:00
trace_probe.c tracing/probe: Fix to check the difference of nr_args before adding probe 2019-09-28 17:07:53 -04:00
trace_probe.h tracing/probe: Reject exactly same probe event 2019-09-19 11:09:16 -04:00
trace_sched_switch.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
trace_sched_wakeup.c sched/core: Convert get_task_struct() to return the task 2019-07-25 15:51:54 +02:00
trace_selftest_dynamic.c ftrace: Mark function tracer test functions noinline/noclone 2018-01-23 15:57:29 -05:00
trace_selftest.c function_graph: Have selftest also emulate tr->reset() as it did with tr->init() 2019-04-21 19:46:56 -04:00
trace_seq.c tracing: Add SPDX License format tags to tracing files 2018-08-16 19:08:06 -04:00
trace_stack.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_stat.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_stat.h tracing: Fix SPDX format headers to use C++ style comments 2018-08-16 19:08:06 -04:00
trace_syscalls.c syscalls: Remove start and number from syscall_get_arguments() args 2019-04-05 09:26:43 -04:00
trace_uprobe.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace.c tracing: Initialize iter->seq after zeroing in tracing_read_pipe() 2019-10-12 20:49:34 -04:00
trace.h tracing: Add tracing_check_open_get_tr() 2019-10-12 20:44:07 -04:00
tracing_map.c tracing: Add SPDX License format tags to tracing files 2018-08-16 19:08:06 -04:00
tracing_map.h tracing: Fix SPDX format headers to use C++ style comments 2018-08-16 19:08:06 -04:00