linux/arch/x86
Sean Christopherson ed69a6cb70 KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast()
Acquire the per-VM slots_lock when zapping all shadow pages as part of
toggling nx_huge_pages.  The fast zap algorithm relies on exclusivity
(via slots_lock) to identify obsolete vs. valid shadow pages, because it
uses a single bit for its generation number. Holding slots_lock also
obviates the need to acquire a read lock on the VM's srcu.

Failing to take slots_lock when toggling nx_huge_pages allows multiple
instances of kvm_mmu_zap_all_fast() to run concurrently, as the other
user, KVM_SET_USER_MEMORY_REGION, does not take the global kvm_lock.
(kvm_mmu_zap_all_fast() does take kvm->mmu_lock, but it can be
temporarily dropped by kvm_zap_obsolete_pages(), so it is not enough
to enforce exclusivity).

Concurrent fast zap instances causes obsolete shadow pages to be
incorrectly identified as valid due to the single bit generation number
wrapping, which results in stale shadow pages being left in KVM's MMU
and leads to all sorts of undesirable behavior.
The bug is easily confirmed by running with CONFIG_PROVE_LOCKING and
toggling nx_huge_pages via its module param.

Note, until commit 4ae5acbc4936 ("KVM: x86/mmu: Take slots_lock when
using kvm_mmu_zap_all_fast()", 2019-11-13) the fast zap algorithm used
an ulong-sized generation instead of relying on exclusivity for
correctness, but all callers except the recently added set_nx_huge_pages()
needed to hold slots_lock anyways.  Therefore, this patch does not have
to be backported to stable kernels.

Given that toggling nx_huge_pages is by no means a fast path, force it
to conform to the current approach instead of reintroducing the previous
generation count.

Fixes: b8e8c8303f ("kvm: mmu: ITLB_MULTIHIT mitigation", but NOT FOR STABLE)
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-11-14 16:21:22 +01:00
..
boot x86, efi: Never relocate kernel below lowest acceptable address 2019-10-31 09:40:19 +01:00
configs x86/defconfigs: Remove useless UEVENT_HELPER_PATH 2019-06-21 19:22:08 +02:00
crypto crypto: x86/aes-ni - use AES library instead of single-use AES cipher 2019-09-09 23:48:41 +10:00
entry Kbuild updates for v5.4 2019-09-20 08:36:47 -07:00
events perf/x86/uncore: Fix event group support 2019-10-28 11:02:01 +01:00
hyperv x86/hyperv: Make vapic support x2apic mode 2019-10-15 10:57:09 +02:00
ia32 clone: fix CLONE_PIDFD support 2019-07-14 20:36:12 +02:00
include kvm: x86: mmu: Recovery of shattered NX large pages 2019-11-04 20:26:00 +01:00
kernel Merge branch 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2019-11-12 10:53:24 -08:00
kvm KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast() 2019-11-14 16:21:22 +01:00
lib x86/asm: Fix MWAITX C-state hint value 2019-10-08 13:25:24 +02:00
math-emu x86/fpu/math-emu: Address fallthrough warnings 2019-08-12 20:35:05 +02:00
mm Merge branch 'next-lockdown' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security 2019-09-28 08:14:15 -07:00
net bpf: fix x64 JIT code generation for jmp to 1st insn 2019-08-01 13:12:09 -07:00
oprofile
pci dma-mapping updates for 5.4: 2019-09-19 13:27:23 -07:00
platform efi/x86: Do not clean dummy variable in kexec path 2019-10-07 15:24:36 +02:00
power Merge branch 'x86-apic-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2019-09-17 12:04:39 -07:00
purgatory Merge branch 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2019-09-28 13:37:41 -07:00
ras RAS/CEC: Add CONFIG_RAS_CEC_DEBUG and move CEC debug features there 2019-06-08 17:39:24 +02:00
realmode x86/realmode: Remove trampoline_status 2019-07-22 11:30:18 +02:00
tools Merge branch 'x86-paravirt-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2019-07-08 17:34:44 -07:00
um um: Use real DMA barriers 2019-09-15 21:37:14 +02:00
video treewide: Add SPDX license identifier - Makefile/Kconfig 2019-05-21 10:50:46 +02:00
xen xen: patch for 5.4-rc5 2019-10-26 06:32:12 -04:00
.gitignore
Kbuild treewide: Add SPDX license identifier - Kbuild 2019-05-30 11:32:33 -07:00
Kconfig x86/tsx: Add config options to set tsx=on|off|auto 2019-10-28 09:12:18 +01:00
Kconfig.cpu x86/cpu: Create Zhaoxin processors architecture support file 2019-06-22 11:45:57 +02:00
Kconfig.debug x86, perf: Fix the dependency of the x86 insn decoder selftest 2019-09-02 20:05:58 +02:00
Makefile x86/build: Add -Wnoaddress-of-packed-member to REALMODE_CFLAGS, to silence GCC9 build warning 2019-08-28 17:31:31 +02:00
Makefile_32.cpu
Makefile.um