linux/arch/powerpc/lib
Nicholas Piggin aa65ff6b18 powerpc/64s: Implement queued spinlocks and rwlocks
These have shown significantly improved performance and fairness when
spinlock contention is moderate to high on very large systems.

With this series including subsequent patches, on a 16 socket 1536
thread POWER9, a stress test such as same-file open/close from all
CPUs gets big speedups, 11620op/s aggregate with simple spinlocks vs
384158op/s (33x faster), where the difference in throughput between
the fastest and slowest thread goes from 7x to 1.4x.

Thanks to the fast path being identical in terms of atomics and
barriers (after a subsequent optimisation patch), single threaded
performance is not changed (no measurable difference).

On smaller systems, performance and fairness seems to be generally
improved. Using dbench on tmpfs as a test (that starts to run into
kernel spinlock contention), a 2-socket OpenPOWER POWER9 system was
tested with bare metal and KVM guest configurations. Results can be
found here:

https://github.com/linuxppc/issues/issues/305#issuecomment-663487453

Observations are:

- Queued spinlocks are equal when contention is insignificant, as
  expected and as measured with microbenchmarks.

- When there is contention, on bare metal queued spinlocks have better
  throughput and max latency at all points.

- When virtualised, queued spinlocks are slightly worse approaching
  peak throughput, but significantly better throughput and max latency
  at all points beyond peak, until queued spinlock maximum latency
  rises when clients are 2x vCPUs.

The regressions haven't been analysed very well yet, there are a lot
of things that can be tuned, particularly the paravirtualised locking,
but the numbers already look like a good net win even on relatively
small systems.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Waiman Long <longman@redhat.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/20200724131423.1362108-4-npiggin@gmail.com
2020-07-27 00:01:23 +10:00
..
alloc.c treewide: add checks for the return value of memblock_alloc*() 2019-03-12 10:04:02 -07:00
checksum_32.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
checksum_64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
checksum_wrappers.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156 2019-05-30 11:26:35 -07:00
code-patching.c mm: don't include asm/pgtable.h if linux/mm.h is already included 2020-06-09 09:39:13 -07:00
copy_32.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
copypage_64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
copypage_power7.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156 2019-05-30 11:26:35 -07:00
copyuser_64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
copyuser_power7.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156 2019-05-30 11:26:35 -07:00
crtsavres.S powerpc/64: Do not create new section for save/restore functions 2017-05-30 14:59:51 +10:00
div64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
error-inject.c powerpc: Add support for function error injection 2018-10-20 13:26:43 +11:00
feature-fixups-test.S powerpc: Test prefixed instructions in feature fixups 2020-05-19 00:11:02 +10:00
feature-fixups.c powerpc: Add ppc_inst_next() 2020-05-26 23:36:51 +10:00
hweight_64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156 2019-05-30 11:26:35 -07:00
inst.c maccess: rename probe_user_{read,write} to copy_{from,to}_user_nofault 2020-06-17 10:57:41 -07:00
ldstfp.S powerpc updates for 5.3 2019-07-13 16:08:36 -07:00
locks.c powerpc/pseries: Move some PAPR paravirt functions to their own file 2020-07-26 23:34:26 +10:00
Makefile powerpc/64s: Implement queued spinlocks and rwlocks 2020-07-27 00:01:23 +10:00
mem_64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
memcmp_32.S powerpc/lib: optimise PPC32 memcmp 2018-06-04 00:39:21 +10:00
memcmp_64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
memcpy_64.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
memcpy_mcsafe_64.S powerpc/memcpy: Fix stack corruption for smaller sizes 2019-09-12 09:27:00 +10:00
memcpy_power7.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156 2019-05-30 11:26:35 -07:00
pmem.c powerpc/pmem: Avoid the barrier in flush routines 2020-07-16 13:00:23 +10:00
quad.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
rheap.c treewide: kmalloc() -> kmalloc_array() 2018-06-12 16:19:22 -07:00
sstep.c powerpc/sstep: Fix incorrect CONFIG symbol in scv handling 2020-07-24 23:16:49 +10:00
string_32.S powerpc: Fix __clear_user() with KUAP enabled 2019-12-16 23:19:44 +11:00
string_64.S powerpc: Fix __clear_user() with KUAP enabled 2019-12-16 23:19:44 +11:00
string.S treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
strlen_32.S powerpc/lib: Implement strlen() in assembly for PPC32 2018-08-07 21:49:30 +10:00
test_code-patching.S powerpc: Test prefixed code patching 2020-05-19 00:11:02 +10:00
test_emulate_step_exec_instr.S powerpc/sstep: Add tests for Prefixed Add Immediate 2020-07-23 17:25:21 +10:00
test_emulate_step.c powerpc/test_emulate_sstep: Fix build error 2020-07-24 10:41:19 +10:00
vmx-helper.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156 2019-05-30 11:26:35 -07:00
xor_vmx_glue.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
xor_vmx.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156 2019-05-30 11:26:35 -07:00
xor_vmx.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00