linux/kernel
Nicholas Piggin 9d08fce64d workqueue: Improve scalability of workqueue watchdog touch
[ Upstream commit 98f887f820 ]

On a ~2000 CPU powerpc system, hard lockups have been observed in the
workqueue code when stop_machine runs (in this case due to CPU hotplug).
This is due to lots of CPUs spinning in multi_cpu_stop, calling
touch_nmi_watchdog() which ends up calling wq_watchdog_touch().
wq_watchdog_touch() writes to the global variable wq_watchdog_touched,
and that can find itself in the same cacheline as other important
workqueue data, which slows down operations to the point of lockups.

In the case of the following abridged trace, worker_pool_idr was in
the hot line, causing the lockups to always appear at idr_find.

  watchdog: CPU 1125 self-detected hard LOCKUP @ idr_find
  Call Trace:
  get_work_pool
  __queue_work
  call_timer_fn
  run_timer_softirq
  __do_softirq
  do_softirq_own_stack
  irq_exit
  timer_interrupt
  decrementer_common_virt
  * interrupt: 900 (timer) at multi_cpu_stop
  multi_cpu_stop
  cpu_stopper_thread
  smpboot_thread_fn
  kthread

Fix this by having wq_watchdog_touch() only write to the line if the
last time a touch was recorded exceeds 1/4 of the watchdog threshold.

Reported-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-09-12 11:07:52 +02:00
..
bpf bpf: Eliminate remaining "make W=1" warnings in kernel/bpf/btf.o 2024-08-19 05:44:59 +02:00
cgroup cgroup: Protect css->cgroup write under css_set_lock 2024-09-12 11:07:48 +02:00
configs
debug kdb: Use the passed prompt in kdb_position_cursor() 2024-08-19 05:45:22 +02:00
dma dma-mapping: benchmark: Don't starve others when doing the test 2024-09-12 11:07:48 +02:00
entry entry: Respect changes to system call number by trace_sys_enter() 2024-04-10 16:18:46 +02:00
events perf/aux: Fix AUX buffer serialization 2024-09-12 11:07:51 +02:00
futex futex: Don't include process MM in futex key on no-MMU 2023-11-20 11:08:13 +01:00
gcov gcov: add support for GCC 14 2024-07-05 09:14:34 +02:00
irq genirq/irqdesc: Honor caller provided affinity in alloc_desc() 2024-08-19 05:45:46 +02:00
kcsan kcsan: Don't expect 64 bits atomic builtins from 32 bits architectures 2023-07-23 13:47:12 +02:00
livepatch livepatch: Fix missing newline character in klp_resolve_symbols() 2023-11-20 11:08:25 +01:00
locking rtmutex: Drop rt_mutex::wait_lock before scheduling 2024-09-12 11:07:43 +02:00
power PM: suspend: Set mem_sleep_current during kernel command line setup 2024-04-10 16:18:36 +02:00
printk printk: Update @console_may_schedule in console_trylock_spinning() 2024-04-10 16:18:47 +02:00
rcu rcu-tasks: Fix show_rcu_tasks_trace_gp_kthread buffer overflow 2024-09-12 11:07:42 +02:00
sched sched/smt: Fix unbalance sched_smt_present dec/inc 2024-08-19 05:45:47 +02:00
time hrtimer: Prevent queuing of hrtimer without a function callback 2024-09-04 13:23:28 +02:00
trace tracing: Avoid possible softlockup in tracing_iter_reset() 2024-09-12 11:07:44 +02:00
.gitignore
acct.c acct: fix potential integer overflow in encode_comp_t() 2022-12-31 13:14:40 +01:00
async.c async: Introduce async_schedule_dev_nocall() 2024-02-23 08:54:25 +01:00
audit_fsnotify.c fsnotify: make allow_dups a property of the group 2024-04-10 16:19:02 +02:00
audit_tree.c fsnotify: pass flags argument to fsnotify_alloc_group() 2024-04-10 16:19:02 +02:00
audit_watch.c fsnotify: pass flags argument to fsnotify_alloc_group() 2024-04-10 16:19:02 +02:00
audit.c audit: Send netlink ACK before setting connection in auditd_set 2024-02-23 08:54:37 +01:00
audit.h audit: log AUDIT_TIME_* records only from rules 2022-04-08 14:23:06 +02:00
auditfilter.c ima: Avoid blocking in RCU read-side critical section 2024-07-18 13:07:34 +02:00
auditsc.c audit: fix possible soft lockup in __audit_inode_child() 2023-09-19 12:22:39 +02:00
backtracetest.c
bounds.c bounds: Use the right number of bits for power-of-two CONFIG_NR_CPUS 2024-05-02 16:24:50 +02:00
capability.c
cfi.c cfi: Fix __cfi_slowpath_diag RCU usage with cpuidle 2022-06-22 14:22:04 +02:00
compat.c sched_getaffinity: don't assume 'cpumask_size()' is fully initialized 2023-04-05 11:24:53 +02:00
configs.c
context_tracking.c
cpu_pm.c PM: cpu: Make notifier chain use a raw_spinlock_t 2021-08-16 18:55:32 +02:00
cpu.c cpu/hotplug: Fix dynstate assignment in __cpuhp_setup_state_cpuslocked() 2024-07-05 09:14:48 +02:00
crash_core.c kernel/crash_core: suppress unknown crashkernel parameter warning 2021-12-29 12:28:49 +01:00
crash_dump.c
cred.c cred: switch to using atomic_long_t 2023-12-20 15:17:37 +01:00
delayacct.c
dma.c
exec_domain.c
exit.c mm: optimize the redundant loop of mm_update_owner_next() 2024-07-18 13:07:32 +02:00
extable.c
fail_function.c kernel/fail_function: fix memory leak with using debugfs_lookup() 2023-03-11 13:57:38 +01:00
fork.c kernel/fork: beware of __put_task_struct() calling context 2023-09-23 11:09:55 +02:00
freezer.c
gen_kheaders.sh kheaders: explicitly define file modes for archived headers 2024-07-05 09:14:37 +02:00
groups.c
hung_task.c Merge branch 'akpm' (patches from Andrew) 2021-07-02 12:08:10 -07:00
iomem.c
irq_work.c
jump_label.c jump_label: Fix jump_label_text_reserved() vs __init 2021-07-05 10:46:20 +02:00
kallsyms.c kallsyms: Make kallsyms_on_each_symbol generally available 2023-12-13 18:36:45 +01:00
kcmp.c
Kconfig.freezer
Kconfig.hz
Kconfig.locks locking/rwlock: Provide RT variant 2021-08-17 17:50:51 +02:00
Kconfig.preempt sched/core: Disable CONFIG_SCHED_CORE by default 2021-06-28 22:43:05 +02:00
kcov.c kcov: properly check for softirq context 2024-08-19 05:45:46 +02:00
kexec_core.c kexec: fix a memory leak in crash_shrink_memory() 2023-07-23 13:46:52 +02:00
kexec_elf.c
kexec_file.c kexec: support purgatories with .text.hot sections 2023-06-21 15:59:14 +02:00
kexec_internal.h panic, kexec: make __crash_kexec() NMI safe 2023-04-20 12:13:57 +02:00
kexec.c kernel: kexec: copy user-array safely 2023-11-28 16:56:16 +00:00
kheaders.c kheaders: Use array declaration instead of char 2023-05-11 23:00:17 +09:00
kmod.c
kprobes.c kprobes: Fix to check symbol prefixes correctly 2024-08-19 05:45:42 +02:00
ksysfs.c kexec: turn all kexec_mutex acquisitions into trylocks 2023-04-20 12:13:57 +02:00
kthread.c exit: Implement kthread_exit 2024-04-10 16:18:55 +02:00
latencytop.c
Makefile futex: Move to kernel/futex/ 2022-12-31 13:14:04 +01:00
module_signature.c
module_signing.c
module-internal.h
module.c NFSD: Remove svc_serv_ops::svo_module 2024-04-10 16:19:01 +02:00
notifier.c notifier: Remove atomic_notifier_call_chain_robust() 2021-08-16 18:55:32 +02:00
nsproxy.c memcg: enable accounting for new namesapces and struct nsproxy 2021-09-03 09:58:12 -07:00
padata.c padata: Fix possible divide-by-0 panic in padata_mt_helper() 2024-08-19 05:45:47 +02:00
panic.c panic: Flush kernel log buffer at the end 2024-04-13 13:01:43 +02:00
params.c params: lift param_set_uint_minmax to common code 2021-08-16 14:42:22 +02:00
pid_namespace.c zap_pid_ns_processes: clear TIF_NOTIFY_SIGNAL along with TIF_SIGPENDING 2024-07-05 09:14:24 +02:00
pid.c kernel/pid.c: implement additional checks upon pidfd_create() parameters 2021-08-10 12:53:07 +02:00
profile.c profiling: remove profile=sleep support 2024-08-19 05:45:39 +02:00
ptrace.c ptrace: Reimplement PTRACE_KILL by always sending SIGKILL 2022-06-09 10:22:29 +02:00
range.c
reboot.c kernel/reboot: emergency_restart: Set correct system_state 2023-11-28 16:56:31 +00:00
regset.c
relay.c relayfs: fix out-of-bounds access in relay_file_read 2023-05-11 23:00:18 +09:00
resource_kunit.c
resource.c dax/kmem: Fix leak of memory-hotplug resources 2023-03-10 09:40:08 +01:00
rseq.c rseq: Remove broken uapi field layout on 32-bit little endian 2022-04-08 14:23:10 +02:00
scftorture.c scftorture: Forgive memory-allocation failure if KASAN 2023-09-23 11:09:55 +02:00
scs.c scs: Release kasan vmalloc poison in scs_free process 2021-11-18 19:16:29 +01:00
seccomp.c seccomp: Invalidate seccomp mode to catch death failures 2022-02-16 12:56:38 +01:00
signal.c kernel: rerun task_work while freezing in get_signal() 2024-08-19 05:45:22 +02:00
smp.c smp: Add missing destroy_work_on_stack() call in smp_call_on_cpu() 2024-09-12 11:07:49 +02:00
smpboot.c smpboot: Replace deprecated CPU-hotplug functions. 2021-08-10 14:57:42 +02:00
smpboot.h
softirq.c softirq: Fix suspicious RCU usage in __do_softirq() 2024-06-16 13:39:15 +02:00
stackleak.c gcc-plugins/stackleak: Use noinstr in favor of notrace 2022-02-23 12:03:07 +01:00
stacktrace.c stacktrace: move filter_irq_stacks() to kernel/stacktrace.c 2022-04-13 20:59:28 +02:00
static_call_inline.c static_call: Don't make __static_call_return0 static 2022-04-13 20:59:28 +02:00
static_call.c static_call: Don't make __static_call_return0 static 2022-04-13 20:59:28 +02:00
stop_machine.c
sys_ni.c syscalls: fix compat_sys_io_pgetevents_time64 usage 2024-07-05 09:14:50 +02:00
sys.c getrusage: use sig->stats_lock rather than lock_task_sighand() 2024-03-15 10:48:22 -04:00
sysctl-test.c kernel/sysctl-test: Remove some casts which are no-longer required 2021-06-23 16:41:24 -06:00
sysctl.c sched/rt: Disallow writing invalid values to sched_rt_period_us 2024-03-01 13:21:43 +01:00
task_work.c task_work: Introduce task_work_cancel() again 2024-08-19 05:45:13 +02:00
taskstats.c
test_kprobes.c
torture.c torture: Fix hang during kthread shutdown phase 2023-08-30 16:18:19 +02:00
tracepoint.c tracepoint: Fix kerneldoc comments 2021-08-16 11:39:51 -04:00
tsacct.c taskstats: Cleanup the use of task->exit_code 2022-01-27 11:05:35 +01:00
ucount.c ucounts: Handle wrapping in is_ucounts_overlimit 2022-02-23 12:03:20 +01:00
uid16.c
uid16.h
umh.c
up.c
user_namespace.c ucounts: Fix systemd LimitNPROC with private users regression 2022-03-08 19:12:42 +01:00
user-return-notifier.c
user.c fs/epoll: use a per-cpu counter for user's watches count 2021-09-08 11:50:27 -07:00
usermode_driver.c Merge branch 'work.namei' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2021-07-03 11:41:14 -07:00
utsname_sysctl.c
utsname.c
watch_queue.c kernel: watch_queue: copy user-array safely 2023-11-28 16:56:16 +00:00
watchdog_hld.c watchdog/perf: properly initialize the turbo mode timestamp and rearm counter 2024-08-19 05:45:20 +02:00
watchdog.c watchdog: move softlockup_panic back to early_param 2023-11-28 16:56:28 +00:00
workqueue_internal.h workqueue: Assign a color to barrier work items 2021-08-17 07:49:10 -10:00
workqueue.c workqueue: Improve scalability of workqueue watchdog touch 2024-09-12 11:07:52 +02:00