2019-06-04 16:11:32 +08:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-only
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
/*
|
|
|
|
* Kernel-based Virtual Machine driver for Linux
|
|
|
|
*
|
|
|
|
* This module enables machines with Intel VT-x extensions to run virtual
|
|
|
|
* machines without emulation or binary translation.
|
|
|
|
*
|
|
|
|
* MMU support
|
|
|
|
*
|
|
|
|
* Copyright (C) 2006 Qumranet, Inc.
|
2010-10-06 20:23:22 +08:00
|
|
|
* Copyright 2010 Red Hat, Inc. and/or its affiliates.
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
*
|
|
|
|
* Authors:
|
|
|
|
* Yaniv Kamay <yaniv@qumranet.com>
|
|
|
|
* Avi Kivity <avi@qumranet.com>
|
|
|
|
*/
|
2007-06-29 02:15:57 +08:00
|
|
|
|
2010-10-14 17:22:46 +08:00
|
|
|
#include "irq.h"
|
2020-05-21 13:57:49 +08:00
|
|
|
#include "ioapic.h"
|
2007-12-14 09:35:10 +08:00
|
|
|
#include "mmu.h"
|
2020-06-23 04:20:31 +08:00
|
|
|
#include "mmu_internal.h"
|
2010-01-21 21:31:49 +08:00
|
|
|
#include "x86.h"
|
2009-06-01 03:58:47 +08:00
|
|
|
#include "kvm_cache_regs.h"
|
2020-02-19 07:29:49 +08:00
|
|
|
#include "kvm_emulate.h"
|
2014-05-07 20:32:50 +08:00
|
|
|
#include "cpuid.h"
|
2007-06-29 02:15:57 +08:00
|
|
|
|
2007-12-16 17:02:48 +08:00
|
|
|
#include <linux/kvm_host.h>
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/string.h>
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/highmem.h>
|
2016-07-14 08:19:00 +08:00
|
|
|
#include <linux/moduleparam.h>
|
|
|
|
#include <linux/export.h>
|
2007-11-26 20:08:14 +08:00
|
|
|
#include <linux/swap.h>
|
2008-02-23 22:44:30 +08:00
|
|
|
#include <linux/hugetlb.h>
|
2008-02-23 01:21:37 +08:00
|
|
|
#include <linux/compiler.h>
|
2009-12-24 00:35:21 +08:00
|
|
|
#include <linux/srcu.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/slab.h>
|
2017-02-09 01:51:30 +08:00
|
|
|
#include <linux/sched/signal.h>
|
2010-05-31 14:28:19 +08:00
|
|
|
#include <linux/uaccess.h>
|
kvm: x86: reduce collisions in mmu_page_hash
When using two-dimensional paging, the mmu_page_hash (which provides
lookups for existing kvm_mmu_page structs), becomes imbalanced; with
too many collisions in buckets 0 and 512. This has been seen to cause
mmu_lock to be held for multiple milliseconds in kvm_mmu_get_page on
VMs with a large amount of RAM mapped with 4K pages.
The current hash function uses the lower 10 bits of gfn to index into
mmu_page_hash. When doing shadow paging, gfn is the address of the
guest page table being shadow. These tables are 4K-aligned, which
makes the low bits of gfn a good hash. However, with two-dimensional
paging, no guest page tables are being shadowed, so gfn is the base
address that is mapped by the table. Thus page tables (level=1) have
a 2MB aligned gfn, page directories (level=2) have a 1GB aligned gfn,
etc. This means hashes will only differ in their 10th bit.
hash_64() provides a better hash. For example, on a VM with ~200G
(99458 direct=1 kvm_mmu_page structs):
hash max_mmu_page_hash_collisions
--------------------------------------------
low 10 bits 49847
hash_64 105
perfect 97
While we're changing the hash, increase the table size by 4x to better
support large VMs (further reduces number of collisions in 200G VM to
29).
Note that hash_64() does not provide a good distribution prior to commit
ef703f49a6c5 ("Eliminate bad hash multipliers from hash_32() and
hash_64()").
Signed-off-by: David Matlack <dmatlack@google.com>
Change-Id: I5aa6b13c834722813c6cca46b8b1ed6f53368ade
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-12-20 05:58:25 +08:00
|
|
|
#include <linux/hash.h>
|
2016-12-07 08:46:16 +08:00
|
|
|
#include <linux/kern_levels.h>
|
2019-11-05 03:26:00 +08:00
|
|
|
#include <linux/kthread.h>
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2007-06-29 02:15:57 +08:00
|
|
|
#include <asm/page.h>
|
2019-11-20 22:33:57 +08:00
|
|
|
#include <asm/memtype.h>
|
2007-06-29 02:15:57 +08:00
|
|
|
#include <asm/cmpxchg.h>
|
2019-02-01 04:24:44 +08:00
|
|
|
#include <asm/e820/api.h>
|
2007-11-21 20:08:40 +08:00
|
|
|
#include <asm/io.h>
|
2008-11-18 05:03:13 +08:00
|
|
|
#include <asm/vmx.h>
|
2016-02-24 17:51:11 +08:00
|
|
|
#include <asm/kvm_page_track.h>
|
2017-07-14 09:30:40 +08:00
|
|
|
#include "trace.h"
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
extern bool itlb_multihit_kvm_mitigation;
|
|
|
|
|
|
|
|
static int __read_mostly nx_huge_pages = -1;
|
2019-11-13 22:47:06 +08:00
|
|
|
#ifdef CONFIG_PREEMPT_RT
|
|
|
|
/* Recovery can cause latency spikes, disable it for PREEMPT_RT. */
|
|
|
|
static uint __read_mostly nx_huge_pages_recovery_ratio = 0;
|
|
|
|
#else
|
2019-11-05 03:26:00 +08:00
|
|
|
static uint __read_mostly nx_huge_pages_recovery_ratio = 60;
|
2019-11-13 22:47:06 +08:00
|
|
|
#endif
|
2019-11-04 19:22:02 +08:00
|
|
|
|
|
|
|
static int set_nx_huge_pages(const char *val, const struct kernel_param *kp);
|
2019-11-05 03:26:00 +08:00
|
|
|
static int set_nx_huge_pages_recovery_ratio(const char *val, const struct kernel_param *kp);
|
2019-11-04 19:22:02 +08:00
|
|
|
|
|
|
|
static struct kernel_param_ops nx_huge_pages_ops = {
|
|
|
|
.set = set_nx_huge_pages,
|
|
|
|
.get = param_get_bool,
|
|
|
|
};
|
|
|
|
|
2019-11-05 03:26:00 +08:00
|
|
|
static struct kernel_param_ops nx_huge_pages_recovery_ratio_ops = {
|
|
|
|
.set = set_nx_huge_pages_recovery_ratio,
|
|
|
|
.get = param_get_uint,
|
|
|
|
};
|
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
module_param_cb(nx_huge_pages, &nx_huge_pages_ops, &nx_huge_pages, 0644);
|
|
|
|
__MODULE_PARM_TYPE(nx_huge_pages, "bool");
|
2019-11-05 03:26:00 +08:00
|
|
|
module_param_cb(nx_huge_pages_recovery_ratio, &nx_huge_pages_recovery_ratio_ops,
|
|
|
|
&nx_huge_pages_recovery_ratio, 0644);
|
|
|
|
__MODULE_PARM_TYPE(nx_huge_pages_recovery_ratio, "uint");
|
2019-11-04 19:22:02 +08:00
|
|
|
|
2020-03-21 05:28:28 +08:00
|
|
|
static bool __read_mostly force_flush_and_sync_on_reuse;
|
|
|
|
module_param_named(flush_on_reuse, force_flush_and_sync_on_reuse, bool, 0644);
|
|
|
|
|
2008-02-07 20:47:41 +08:00
|
|
|
/*
|
|
|
|
* When setting this variable to true it enables Two-Dimensional-Paging
|
|
|
|
* where the hardware walks 2 page tables:
|
|
|
|
* 1. the guest-virtual to guest-physical
|
|
|
|
* 2. while doing 1. it walks guest-physical to host-physical
|
|
|
|
* If the hardware supports that we don't need to do shadow paging.
|
|
|
|
*/
|
2008-02-23 01:21:37 +08:00
|
|
|
bool tdp_enabled = false;
|
2008-02-07 20:47:41 +08:00
|
|
|
|
2020-03-03 07:57:03 +08:00
|
|
|
static int max_page_level __read_mostly;
|
|
|
|
|
2010-08-30 18:22:53 +08:00
|
|
|
enum {
|
|
|
|
AUDIT_PRE_PAGE_FAULT,
|
|
|
|
AUDIT_POST_PAGE_FAULT,
|
|
|
|
AUDIT_PRE_PTE_WRITE,
|
2010-09-27 18:09:29 +08:00
|
|
|
AUDIT_POST_PTE_WRITE,
|
|
|
|
AUDIT_PRE_SYNC,
|
|
|
|
AUDIT_POST_SYNC
|
2010-08-30 18:22:53 +08:00
|
|
|
};
|
2007-01-06 08:36:56 +08:00
|
|
|
|
2010-08-30 18:22:53 +08:00
|
|
|
#undef MMU_DEBUG
|
2007-01-06 08:36:56 +08:00
|
|
|
|
|
|
|
#ifdef MMU_DEBUG
|
2013-10-02 22:56:16 +08:00
|
|
|
static bool dbg = 0;
|
|
|
|
module_param(dbg, bool, 0644);
|
2007-01-06 08:36:56 +08:00
|
|
|
|
|
|
|
#define pgprintk(x...) do { if (dbg) printk(x); } while (0)
|
|
|
|
#define rmap_printk(x...) do { if (dbg) printk(x); } while (0)
|
2013-10-02 22:56:16 +08:00
|
|
|
#define MMU_WARN_ON(x) WARN_ON(x)
|
2007-01-06 08:36:56 +08:00
|
|
|
#else
|
|
|
|
#define pgprintk(x...) do { } while (0)
|
|
|
|
#define rmap_printk(x...) do { } while (0)
|
2013-10-02 22:56:16 +08:00
|
|
|
#define MMU_WARN_ON(x) do { } while (0)
|
2007-04-25 14:17:25 +08:00
|
|
|
#endif
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2010-08-22 19:12:48 +08:00
|
|
|
#define PTE_PREFETCH_NUM 8
|
|
|
|
|
2012-06-07 18:26:07 +08:00
|
|
|
#define PT_FIRST_AVAIL_BITS_SHIFT 10
|
2019-09-24 18:43:08 +08:00
|
|
|
#define PT64_SECOND_AVAIL_BITS_SHIFT 54
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The mask used to denote special SPTEs, which can be either MMIO SPTEs or
|
|
|
|
* Access Tracking SPTEs.
|
|
|
|
*/
|
|
|
|
#define SPTE_SPECIAL_MASK (3ULL << 52)
|
|
|
|
#define SPTE_AD_ENABLED_MASK (0ULL << 52)
|
|
|
|
#define SPTE_AD_DISABLED_MASK (1ULL << 52)
|
2019-09-27 00:47:59 +08:00
|
|
|
#define SPTE_AD_WRPROT_ONLY_MASK (2ULL << 52)
|
2019-09-24 18:43:08 +08:00
|
|
|
#define SPTE_MMIO_MASK (3ULL << 52)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
|
|
|
#define PT64_LEVEL_BITS 9
|
|
|
|
|
|
|
|
#define PT64_LEVEL_SHIFT(level) \
|
2007-10-08 21:02:08 +08:00
|
|
|
(PAGE_SHIFT + (level - 1) * PT64_LEVEL_BITS)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
|
|
|
#define PT64_INDEX(address, level)\
|
|
|
|
(((address) >> PT64_LEVEL_SHIFT(level)) & ((1 << PT64_LEVEL_BITS) - 1))
|
|
|
|
|
|
|
|
|
|
|
|
#define PT32_LEVEL_BITS 10
|
|
|
|
|
|
|
|
#define PT32_LEVEL_SHIFT(level) \
|
2007-10-08 21:02:08 +08:00
|
|
|
(PAGE_SHIFT + (level - 1) * PT32_LEVEL_BITS)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2009-07-27 22:30:45 +08:00
|
|
|
#define PT32_LVL_OFFSET_MASK(level) \
|
|
|
|
(PT32_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \
|
|
|
|
* PT32_LEVEL_BITS))) - 1))
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
|
|
|
#define PT32_INDEX(address, level)\
|
|
|
|
(((address) >> PT32_LEVEL_SHIFT(level)) & ((1 << PT32_LEVEL_BITS) - 1))
|
|
|
|
|
|
|
|
|
2019-01-15 12:28:40 +08:00
|
|
|
#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK
|
|
|
|
#define PT64_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1))
|
|
|
|
#else
|
|
|
|
#define PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1))
|
|
|
|
#endif
|
2009-07-27 22:30:45 +08:00
|
|
|
#define PT64_LVL_ADDR_MASK(level) \
|
|
|
|
(PT64_BASE_ADDR_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \
|
|
|
|
* PT64_LEVEL_BITS))) - 1))
|
|
|
|
#define PT64_LVL_OFFSET_MASK(level) \
|
|
|
|
(PT64_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \
|
|
|
|
* PT64_LEVEL_BITS))) - 1))
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
|
|
|
#define PT32_BASE_ADDR_MASK PAGE_MASK
|
|
|
|
#define PT32_DIR_BASE_ADDR_MASK \
|
|
|
|
(PAGE_MASK & ~((1ULL << (PAGE_SHIFT + PT32_LEVEL_BITS)) - 1))
|
2009-07-27 22:30:45 +08:00
|
|
|
#define PT32_LVL_ADDR_MASK(level) \
|
|
|
|
(PAGE_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \
|
|
|
|
* PT32_LEVEL_BITS))) - 1))
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2013-08-05 16:07:14 +08:00
|
|
|
#define PT64_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \
|
2017-07-18 05:10:27 +08:00
|
|
|
| shadow_x_mask | shadow_nx_mask | shadow_me_mask)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2007-12-09 22:15:46 +08:00
|
|
|
#define ACC_EXEC_MASK 1
|
|
|
|
#define ACC_WRITE_MASK PT_WRITABLE_MASK
|
|
|
|
#define ACC_USER_MASK PT_USER_MASK
|
|
|
|
#define ACC_ALL (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK)
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
/* The mask for the R/X bits in EPT PTEs */
|
|
|
|
#define PT64_EPT_READABLE_MASK 0x1ull
|
|
|
|
#define PT64_EPT_EXECUTABLE_MASK 0x4ull
|
|
|
|
|
2009-12-31 18:10:16 +08:00
|
|
|
#include <trace/events/kvm.h>
|
|
|
|
|
2012-06-20 15:58:58 +08:00
|
|
|
#define SPTE_HOST_WRITEABLE (1ULL << PT_FIRST_AVAIL_BITS_SHIFT)
|
|
|
|
#define SPTE_MMU_WRITEABLE (1ULL << (PT_FIRST_AVAIL_BITS_SHIFT + 1))
|
2009-09-24 02:47:17 +08:00
|
|
|
|
2008-08-21 22:49:56 +08:00
|
|
|
#define SHADOW_PT_INDEX(addr, level) PT64_INDEX(addr, level)
|
|
|
|
|
2012-03-21 22:49:39 +08:00
|
|
|
/* make pte_list_desc fit well in cache line */
|
|
|
|
#define PTE_LIST_EXT 3
|
|
|
|
|
2017-08-17 21:03:32 +08:00
|
|
|
/*
|
|
|
|
* Return values of handle_mmio_page_fault and mmu.page_fault:
|
|
|
|
* RET_PF_RETRY: let CPU fault again on the address.
|
|
|
|
* RET_PF_EMULATE: mmio page fault, emulate the instruction directly.
|
|
|
|
*
|
|
|
|
* For handle_mmio_page_fault only:
|
|
|
|
* RET_PF_INVALID: the spte is invalid, let the real page fault path update it.
|
|
|
|
*/
|
|
|
|
enum {
|
|
|
|
RET_PF_RETRY = 0,
|
|
|
|
RET_PF_EMULATE = 1,
|
|
|
|
RET_PF_INVALID = 2,
|
|
|
|
};
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
struct pte_list_desc {
|
|
|
|
u64 *sptes[PTE_LIST_EXT];
|
|
|
|
struct pte_list_desc *more;
|
2007-01-06 08:36:38 +08:00
|
|
|
};
|
|
|
|
|
2008-12-25 20:39:47 +08:00
|
|
|
struct kvm_shadow_walk_iterator {
|
|
|
|
u64 addr;
|
|
|
|
hpa_t shadow_addr;
|
|
|
|
u64 *sptep;
|
2011-07-12 03:32:54 +08:00
|
|
|
int level;
|
2008-12-25 20:39:47 +08:00
|
|
|
unsigned index;
|
|
|
|
};
|
|
|
|
|
2018-06-28 05:59:16 +08:00
|
|
|
#define for_each_shadow_entry_using_root(_vcpu, _root, _addr, _walker) \
|
|
|
|
for (shadow_walk_init_using_root(&(_walker), (_vcpu), \
|
|
|
|
(_root), (_addr)); \
|
|
|
|
shadow_walk_okay(&(_walker)); \
|
|
|
|
shadow_walk_next(&(_walker)))
|
|
|
|
|
|
|
|
#define for_each_shadow_entry(_vcpu, _addr, _walker) \
|
2008-12-25 20:39:47 +08:00
|
|
|
for (shadow_walk_init(&(_walker), _vcpu, _addr); \
|
|
|
|
shadow_walk_okay(&(_walker)); \
|
|
|
|
shadow_walk_next(&(_walker)))
|
|
|
|
|
2011-07-12 03:32:13 +08:00
|
|
|
#define for_each_shadow_entry_lockless(_vcpu, _addr, _walker, spte) \
|
|
|
|
for (shadow_walk_init(&(_walker), _vcpu, _addr); \
|
|
|
|
shadow_walk_okay(&(_walker)) && \
|
|
|
|
({ spte = mmu_spte_get_lockless(_walker.sptep); 1; }); \
|
|
|
|
__shadow_walk_next(&(_walker), spte))
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
static struct kmem_cache *pte_list_desc_cache;
|
2007-05-30 17:34:53 +08:00
|
|
|
static struct kmem_cache *mmu_page_header_cache;
|
KVM: create aggregate kvm_total_used_mmu_pages value
Of slab shrinkers, the VM code says:
* Note that 'shrink' will be passed nr_to_scan == 0 when the VM is
* querying the cache size, so a fastpath for that case is appropriate.
and it *means* it. Look at how it calls the shrinkers:
nr_before = (*shrinker->shrink)(0, gfp_mask);
shrink_ret = (*shrinker->shrink)(this_scan, gfp_mask);
So, if you do anything stupid in your shrinker, the VM will doubly
punish you.
The mmu_shrink() function takes the global kvm_lock, then acquires
every VM's kvm->mmu_lock in sequence. If we have 100 VMs, then
we're going to take 101 locks. We do it twice, so each call takes
202 locks. If we're under memory pressure, we can have each cpu
trying to do this. It can get really hairy, and we've seen lock
spinning in mmu_shrink() be the dominant entry in profiles.
This is guaranteed to optimize at least half of those lock
aquisitions away. It removes the need to take any of the locks
when simply trying to count objects.
A 'percpu_counter' can be a large object, but we only have one
of these for the entire system. There are not any better
alternatives at the moment, especially ones that handle CPU
hotplug.
Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
Signed-off-by: Tim Pepper <lnxninja@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2010-08-20 09:11:37 +08:00
|
|
|
static struct percpu_counter kvm_total_used_mmu_pages;
|
2007-04-15 21:31:09 +08:00
|
|
|
|
2008-04-25 21:13:50 +08:00
|
|
|
static u64 __read_mostly shadow_nx_mask;
|
|
|
|
static u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */
|
|
|
|
static u64 __read_mostly shadow_user_mask;
|
|
|
|
static u64 __read_mostly shadow_accessed_mask;
|
|
|
|
static u64 __read_mostly shadow_dirty_mask;
|
2017-07-01 08:26:30 +08:00
|
|
|
static u64 __read_mostly shadow_mmio_value;
|
2019-08-02 04:35:22 +08:00
|
|
|
static u64 __read_mostly shadow_mmio_access_mask;
|
2016-07-13 06:18:49 +08:00
|
|
|
static u64 __read_mostly shadow_present_mask;
|
2017-07-18 05:10:27 +08:00
|
|
|
static u64 __read_mostly shadow_me_mask;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
/*
|
2019-09-24 18:43:08 +08:00
|
|
|
* SPTEs used by MMUs without A/D bits are marked with SPTE_AD_DISABLED_MASK;
|
|
|
|
* shadow_acc_track_mask is the set of bits to be cleared in non-accessed
|
|
|
|
* pages.
|
2016-12-07 08:46:16 +08:00
|
|
|
*/
|
|
|
|
static u64 __read_mostly shadow_acc_track_mask;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The mask/shift to use for saving the original R/X bits when marking the PTE
|
|
|
|
* as not-present for access tracking purposes. We do not save the W bit as the
|
|
|
|
* PTEs being access tracked also need to be dirty tracked, so the W bit will be
|
|
|
|
* restored only when a write is attempted to the page.
|
|
|
|
*/
|
|
|
|
static const u64 shadow_acc_track_saved_bits_mask = PT64_EPT_READABLE_MASK |
|
|
|
|
PT64_EPT_EXECUTABLE_MASK;
|
|
|
|
static const u64 shadow_acc_track_saved_bits_shift = PT64_SECOND_AVAIL_BITS_SHIFT;
|
|
|
|
|
2018-08-15 01:15:34 +08:00
|
|
|
/*
|
|
|
|
* This mask must be set on all non-zero Non-Present or Reserved SPTEs in order
|
|
|
|
* to guard against L1TF attacks.
|
|
|
|
*/
|
|
|
|
static u64 __read_mostly shadow_nonpresent_or_rsvd_mask;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The number of high-order 1 bits to use in the mask above.
|
|
|
|
*/
|
|
|
|
static const u64 shadow_nonpresent_or_rsvd_mask_len = 5;
|
|
|
|
|
KVM: x86: fix L1TF's MMIO GFN calculation
One defense against L1TF in KVM is to always set the upper five bits
of the *legal* physical address in the SPTEs for non-present and
reserved SPTEs, e.g. MMIO SPTEs. In the MMIO case, the GFN of the
MMIO SPTE may overlap with the upper five bits that are being usurped
to defend against L1TF. To preserve the GFN, the bits of the GFN that
overlap with the repurposed bits are shifted left into the reserved
bits, i.e. the GFN in the SPTE will be split into high and low parts.
When retrieving the GFN from the MMIO SPTE, e.g. to check for an MMIO
access, get_mmio_spte_gfn() unshifts the affected bits and restores
the original GFN for comparison. Unfortunately, get_mmio_spte_gfn()
neglects to mask off the reserved bits in the SPTE that were used to
store the upper chunk of the GFN. As a result, KVM fails to detect
MMIO accesses whose GPA overlaps the repurprosed bits, which in turn
causes guest panics and hangs.
Fix the bug by generating a mask that covers the lower chunk of the
GFN, i.e. the bits that aren't shifted by the L1TF mitigation. The
alternative approach would be to explicitly zero the five reserved
bits that are used to store the upper chunk of the GFN, but that
requires additional run-time computation and makes an already-ugly
bit of code even more inscrutable.
I considered adding a WARN_ON_ONCE(low_phys_bits-1 <= PAGE_SHIFT) to
warn if GENMASK_ULL() generated a nonsensical value, but that seemed
silly since that would mean a system that supports VMX has less than
18 bits of physical address space...
Reported-by: Sakari Ailus <sakari.ailus@iki.fi>
Fixes: d9b47449c1a1 ("kvm: x86: Set highest physical address bits in non-present/reserved SPTEs")
Cc: Junaid Shahid <junaids@google.com>
Cc: Jim Mattson <jmattson@google.com>
Cc: stable@vger.kernel.org
Reviewed-by: Junaid Shahid <junaids@google.com>
Tested-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-09-26 04:20:00 +08:00
|
|
|
/*
|
|
|
|
* In some cases, we need to preserve the GFN of a non-present or reserved
|
|
|
|
* SPTE when we usurp the upper five bits of the physical address space to
|
|
|
|
* defend against L1TF, e.g. for MMIO SPTEs. To preserve the GFN, we'll
|
|
|
|
* shift bits of the GFN that overlap with shadow_nonpresent_or_rsvd_mask
|
|
|
|
* left into the reserved bits, i.e. the GFN in the SPTE will be split into
|
|
|
|
* high and low parts. This mask covers the lower bits of the GFN.
|
|
|
|
*/
|
|
|
|
static u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask;
|
|
|
|
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
/*
|
|
|
|
* The number of non-reserved physical address bits irrespective of features
|
|
|
|
* that repurpose legal bits, e.g. MKTME.
|
|
|
|
*/
|
|
|
|
static u8 __read_mostly shadow_phys_bits;
|
KVM: x86: fix L1TF's MMIO GFN calculation
One defense against L1TF in KVM is to always set the upper five bits
of the *legal* physical address in the SPTEs for non-present and
reserved SPTEs, e.g. MMIO SPTEs. In the MMIO case, the GFN of the
MMIO SPTE may overlap with the upper five bits that are being usurped
to defend against L1TF. To preserve the GFN, the bits of the GFN that
overlap with the repurposed bits are shifted left into the reserved
bits, i.e. the GFN in the SPTE will be split into high and low parts.
When retrieving the GFN from the MMIO SPTE, e.g. to check for an MMIO
access, get_mmio_spte_gfn() unshifts the affected bits and restores
the original GFN for comparison. Unfortunately, get_mmio_spte_gfn()
neglects to mask off the reserved bits in the SPTE that were used to
store the upper chunk of the GFN. As a result, KVM fails to detect
MMIO accesses whose GPA overlaps the repurprosed bits, which in turn
causes guest panics and hangs.
Fix the bug by generating a mask that covers the lower chunk of the
GFN, i.e. the bits that aren't shifted by the L1TF mitigation. The
alternative approach would be to explicitly zero the five reserved
bits that are used to store the upper chunk of the GFN, but that
requires additional run-time computation and makes an already-ugly
bit of code even more inscrutable.
I considered adding a WARN_ON_ONCE(low_phys_bits-1 <= PAGE_SHIFT) to
warn if GENMASK_ULL() generated a nonsensical value, but that seemed
silly since that would mean a system that supports VMX has less than
18 bits of physical address space...
Reported-by: Sakari Ailus <sakari.ailus@iki.fi>
Fixes: d9b47449c1a1 ("kvm: x86: Set highest physical address bits in non-present/reserved SPTEs")
Cc: Junaid Shahid <junaids@google.com>
Cc: Jim Mattson <jmattson@google.com>
Cc: stable@vger.kernel.org
Reviewed-by: Junaid Shahid <junaids@google.com>
Tested-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-09-26 04:20:00 +08:00
|
|
|
|
2011-07-12 03:33:44 +08:00
|
|
|
static void mmu_spte_set(u64 *sptep, u64 spte);
|
2019-07-01 18:22:57 +08:00
|
|
|
static bool is_executable_pte(u64 spte);
|
2018-06-28 05:59:07 +08:00
|
|
|
static union kvm_mmu_page_role
|
|
|
|
kvm_mmu_calc_root_page_role(struct kvm_vcpu *vcpu);
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2019-07-01 18:22:57 +08:00
|
|
|
#define CREATE_TRACE_POINTS
|
|
|
|
#include "mmutrace.h"
|
|
|
|
|
2018-12-06 21:21:08 +08:00
|
|
|
|
|
|
|
static inline bool kvm_available_flush_tlb_with_range(void)
|
|
|
|
{
|
2020-03-22 04:26:00 +08:00
|
|
|
return kvm_x86_ops.tlb_remote_flush_with_range;
|
2018-12-06 21:21:08 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void kvm_flush_remote_tlbs_with_range(struct kvm *kvm,
|
|
|
|
struct kvm_tlb_range *range)
|
|
|
|
{
|
|
|
|
int ret = -ENOTSUPP;
|
|
|
|
|
2020-03-22 04:26:00 +08:00
|
|
|
if (range && kvm_x86_ops.tlb_remote_flush_with_range)
|
|
|
|
ret = kvm_x86_ops.tlb_remote_flush_with_range(kvm, range);
|
2018-12-06 21:21:08 +08:00
|
|
|
|
|
|
|
if (ret)
|
|
|
|
kvm_flush_remote_tlbs(kvm);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
|
|
|
|
u64 start_gfn, u64 pages)
|
|
|
|
{
|
|
|
|
struct kvm_tlb_range range;
|
|
|
|
|
|
|
|
range.start_gfn = start_gfn;
|
|
|
|
range.pages = pages;
|
|
|
|
|
|
|
|
kvm_flush_remote_tlbs_with_range(kvm, &range);
|
|
|
|
}
|
|
|
|
|
2020-05-19 17:04:49 +08:00
|
|
|
void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 access_mask)
|
2011-07-12 03:33:44 +08:00
|
|
|
{
|
2019-08-02 04:35:22 +08:00
|
|
|
BUG_ON((u64)(unsigned)access_mask != access_mask);
|
2020-05-19 17:34:41 +08:00
|
|
|
WARN_ON(mmio_value & (shadow_nonpresent_or_rsvd_mask << shadow_nonpresent_or_rsvd_mask_len));
|
|
|
|
WARN_ON(mmio_value & shadow_nonpresent_or_rsvd_lower_gfn_mask);
|
2019-09-24 18:43:08 +08:00
|
|
|
shadow_mmio_value = mmio_value | SPTE_MMIO_MASK;
|
2019-08-02 04:35:22 +08:00
|
|
|
shadow_mmio_access_mask = access_mask;
|
2011-07-12 03:33:44 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_mask);
|
|
|
|
|
2019-08-02 04:35:23 +08:00
|
|
|
static bool is_mmio_spte(u64 spte)
|
|
|
|
{
|
2020-05-19 17:04:49 +08:00
|
|
|
return (spte & SPTE_SPECIAL_MASK) == SPTE_MMIO_MASK;
|
2019-08-02 04:35:23 +08:00
|
|
|
}
|
|
|
|
|
2017-07-01 08:26:31 +08:00
|
|
|
static inline bool sp_ad_disabled(struct kvm_mmu_page *sp)
|
|
|
|
{
|
|
|
|
return sp->role.ad_disabled;
|
|
|
|
}
|
|
|
|
|
2019-09-27 00:47:59 +08:00
|
|
|
static inline bool kvm_vcpu_ad_need_write_protect(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* When using the EPT page-modification log, the GPAs in the log
|
|
|
|
* would come from L2 rather than L1. Therefore, we need to rely
|
|
|
|
* on write protection to record dirty pages. This also bypasses
|
|
|
|
* PML, since writes now result in a vmexit.
|
|
|
|
*/
|
|
|
|
return vcpu->arch.mmu == &vcpu->arch.guest_mmu;
|
|
|
|
}
|
|
|
|
|
2017-07-01 08:26:31 +08:00
|
|
|
static inline bool spte_ad_enabled(u64 spte)
|
|
|
|
{
|
2019-08-02 04:35:23 +08:00
|
|
|
MMU_WARN_ON(is_mmio_spte(spte));
|
2019-09-27 00:47:59 +08:00
|
|
|
return (spte & SPTE_SPECIAL_MASK) != SPTE_AD_DISABLED_MASK;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool spte_ad_need_write_protect(u64 spte)
|
|
|
|
{
|
|
|
|
MMU_WARN_ON(is_mmio_spte(spte));
|
|
|
|
return (spte & SPTE_SPECIAL_MASK) != SPTE_AD_ENABLED_MASK;
|
2017-07-01 08:26:31 +08:00
|
|
|
}
|
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
static bool is_nx_huge_page_enabled(void)
|
|
|
|
{
|
|
|
|
return READ_ONCE(nx_huge_pages);
|
|
|
|
}
|
|
|
|
|
2017-07-01 08:26:31 +08:00
|
|
|
static inline u64 spte_shadow_accessed_mask(u64 spte)
|
|
|
|
{
|
2019-08-02 04:35:23 +08:00
|
|
|
MMU_WARN_ON(is_mmio_spte(spte));
|
2017-07-01 08:26:31 +08:00
|
|
|
return spte_ad_enabled(spte) ? shadow_accessed_mask : 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline u64 spte_shadow_dirty_mask(u64 spte)
|
|
|
|
{
|
2019-08-02 04:35:23 +08:00
|
|
|
MMU_WARN_ON(is_mmio_spte(spte));
|
2017-07-01 08:26:31 +08:00
|
|
|
return spte_ad_enabled(spte) ? shadow_dirty_mask : 0;
|
|
|
|
}
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
static inline bool is_access_track_spte(u64 spte)
|
|
|
|
{
|
2017-07-01 08:26:31 +08:00
|
|
|
return !spte_ad_enabled(spte) && (spte & shadow_acc_track_mask) == 0;
|
2016-12-07 08:46:16 +08:00
|
|
|
}
|
|
|
|
|
2013-06-07 16:51:24 +08:00
|
|
|
/*
|
2019-02-06 05:01:16 +08:00
|
|
|
* Due to limited space in PTEs, the MMIO generation is a 19 bit subset of
|
|
|
|
* the memslots generation and is derived as follows:
|
2014-08-19 06:46:06 +08:00
|
|
|
*
|
2019-02-06 05:01:18 +08:00
|
|
|
* Bits 0-8 of the MMIO generation are propagated to spte bits 3-11
|
|
|
|
* Bits 9-18 of the MMIO generation are propagated to spte bits 52-61
|
2019-02-06 05:01:16 +08:00
|
|
|
*
|
2019-02-06 05:01:18 +08:00
|
|
|
* The KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS flag is intentionally not included in
|
|
|
|
* the MMIO generation number, as doing so would require stealing a bit from
|
|
|
|
* the "real" generation number and thus effectively halve the maximum number
|
|
|
|
* of MMIO generations that can be handled before encountering a wrap (which
|
|
|
|
* requires a full MMU zap). The flag is instead explicitly queried when
|
|
|
|
* checking for MMIO spte cache hits.
|
2013-06-07 16:51:24 +08:00
|
|
|
*/
|
2020-01-19 03:09:03 +08:00
|
|
|
#define MMIO_SPTE_GEN_MASK GENMASK_ULL(17, 0)
|
2013-06-07 16:51:24 +08:00
|
|
|
|
2019-02-06 05:01:16 +08:00
|
|
|
#define MMIO_SPTE_GEN_LOW_START 3
|
|
|
|
#define MMIO_SPTE_GEN_LOW_END 11
|
|
|
|
#define MMIO_SPTE_GEN_LOW_MASK GENMASK_ULL(MMIO_SPTE_GEN_LOW_END, \
|
|
|
|
MMIO_SPTE_GEN_LOW_START)
|
2013-06-07 16:51:24 +08:00
|
|
|
|
2020-01-19 03:09:03 +08:00
|
|
|
#define MMIO_SPTE_GEN_HIGH_START PT64_SECOND_AVAIL_BITS_SHIFT
|
|
|
|
#define MMIO_SPTE_GEN_HIGH_END 62
|
2019-02-06 05:01:16 +08:00
|
|
|
#define MMIO_SPTE_GEN_HIGH_MASK GENMASK_ULL(MMIO_SPTE_GEN_HIGH_END, \
|
|
|
|
MMIO_SPTE_GEN_HIGH_START)
|
2020-01-19 03:09:03 +08:00
|
|
|
|
2019-02-06 05:01:15 +08:00
|
|
|
static u64 generation_mmio_spte_mask(u64 gen)
|
2013-06-07 16:51:24 +08:00
|
|
|
{
|
|
|
|
u64 mask;
|
|
|
|
|
2019-02-06 05:01:16 +08:00
|
|
|
WARN_ON(gen & ~MMIO_SPTE_GEN_MASK);
|
2020-01-19 03:09:03 +08:00
|
|
|
BUILD_BUG_ON((MMIO_SPTE_GEN_HIGH_MASK | MMIO_SPTE_GEN_LOW_MASK) & SPTE_SPECIAL_MASK);
|
2013-06-07 16:51:24 +08:00
|
|
|
|
2019-02-06 05:01:16 +08:00
|
|
|
mask = (gen << MMIO_SPTE_GEN_LOW_START) & MMIO_SPTE_GEN_LOW_MASK;
|
|
|
|
mask |= (gen << MMIO_SPTE_GEN_HIGH_START) & MMIO_SPTE_GEN_HIGH_MASK;
|
2013-06-07 16:51:24 +08:00
|
|
|
return mask;
|
|
|
|
}
|
|
|
|
|
2019-02-06 05:01:15 +08:00
|
|
|
static u64 get_mmio_spte_generation(u64 spte)
|
2013-06-07 16:51:24 +08:00
|
|
|
{
|
2019-02-06 05:01:15 +08:00
|
|
|
u64 gen;
|
2013-06-07 16:51:24 +08:00
|
|
|
|
2019-02-06 05:01:16 +08:00
|
|
|
gen = (spte & MMIO_SPTE_GEN_LOW_MASK) >> MMIO_SPTE_GEN_LOW_START;
|
|
|
|
gen |= (spte & MMIO_SPTE_GEN_HIGH_MASK) >> MMIO_SPTE_GEN_HIGH_START;
|
2013-06-07 16:51:24 +08:00
|
|
|
return gen;
|
|
|
|
}
|
|
|
|
|
2020-02-04 07:09:10 +08:00
|
|
|
static u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)
|
2011-07-12 03:33:44 +08:00
|
|
|
{
|
2020-02-04 07:09:10 +08:00
|
|
|
|
2019-02-06 05:01:16 +08:00
|
|
|
u64 gen = kvm_vcpu_memslots(vcpu)->generation & MMIO_SPTE_GEN_MASK;
|
2013-06-07 16:51:26 +08:00
|
|
|
u64 mask = generation_mmio_spte_mask(gen);
|
2018-08-15 01:15:34 +08:00
|
|
|
u64 gpa = gfn << PAGE_SHIFT;
|
2013-03-12 16:44:40 +08:00
|
|
|
|
2019-08-02 04:35:22 +08:00
|
|
|
access &= shadow_mmio_access_mask;
|
2018-08-15 01:15:34 +08:00
|
|
|
mask |= shadow_mmio_value | access;
|
|
|
|
mask |= gpa | shadow_nonpresent_or_rsvd_mask;
|
|
|
|
mask |= (gpa & shadow_nonpresent_or_rsvd_mask)
|
|
|
|
<< shadow_nonpresent_or_rsvd_mask_len;
|
2013-06-07 16:51:24 +08:00
|
|
|
|
2020-02-04 07:09:10 +08:00
|
|
|
return mask;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
|
|
|
|
unsigned int access)
|
|
|
|
{
|
|
|
|
u64 mask = make_mmio_spte(vcpu, gfn, access);
|
|
|
|
unsigned int gen = get_mmio_spte_generation(mask);
|
|
|
|
|
|
|
|
access = mask & ACC_ALL;
|
|
|
|
|
2013-06-07 16:51:26 +08:00
|
|
|
trace_mark_mmio_spte(sptep, gfn, access, gen);
|
2013-06-07 16:51:24 +08:00
|
|
|
mmu_spte_set(sptep, mask);
|
2011-07-12 03:33:44 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static gfn_t get_mmio_spte_gfn(u64 spte)
|
|
|
|
{
|
KVM: x86: fix L1TF's MMIO GFN calculation
One defense against L1TF in KVM is to always set the upper five bits
of the *legal* physical address in the SPTEs for non-present and
reserved SPTEs, e.g. MMIO SPTEs. In the MMIO case, the GFN of the
MMIO SPTE may overlap with the upper five bits that are being usurped
to defend against L1TF. To preserve the GFN, the bits of the GFN that
overlap with the repurposed bits are shifted left into the reserved
bits, i.e. the GFN in the SPTE will be split into high and low parts.
When retrieving the GFN from the MMIO SPTE, e.g. to check for an MMIO
access, get_mmio_spte_gfn() unshifts the affected bits and restores
the original GFN for comparison. Unfortunately, get_mmio_spte_gfn()
neglects to mask off the reserved bits in the SPTE that were used to
store the upper chunk of the GFN. As a result, KVM fails to detect
MMIO accesses whose GPA overlaps the repurprosed bits, which in turn
causes guest panics and hangs.
Fix the bug by generating a mask that covers the lower chunk of the
GFN, i.e. the bits that aren't shifted by the L1TF mitigation. The
alternative approach would be to explicitly zero the five reserved
bits that are used to store the upper chunk of the GFN, but that
requires additional run-time computation and makes an already-ugly
bit of code even more inscrutable.
I considered adding a WARN_ON_ONCE(low_phys_bits-1 <= PAGE_SHIFT) to
warn if GENMASK_ULL() generated a nonsensical value, but that seemed
silly since that would mean a system that supports VMX has less than
18 bits of physical address space...
Reported-by: Sakari Ailus <sakari.ailus@iki.fi>
Fixes: d9b47449c1a1 ("kvm: x86: Set highest physical address bits in non-present/reserved SPTEs")
Cc: Junaid Shahid <junaids@google.com>
Cc: Jim Mattson <jmattson@google.com>
Cc: stable@vger.kernel.org
Reviewed-by: Junaid Shahid <junaids@google.com>
Tested-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-09-26 04:20:00 +08:00
|
|
|
u64 gpa = spte & shadow_nonpresent_or_rsvd_lower_gfn_mask;
|
2018-08-15 01:15:34 +08:00
|
|
|
|
|
|
|
gpa |= (spte >> shadow_nonpresent_or_rsvd_mask_len)
|
|
|
|
& shadow_nonpresent_or_rsvd_mask;
|
|
|
|
|
|
|
|
return gpa >> PAGE_SHIFT;
|
2011-07-12 03:33:44 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned get_mmio_spte_access(u64 spte)
|
|
|
|
{
|
2019-08-02 04:35:22 +08:00
|
|
|
return spte & shadow_mmio_access_mask;
|
2011-07-12 03:33:44 +08:00
|
|
|
}
|
|
|
|
|
2015-04-08 21:39:23 +08:00
|
|
|
static bool set_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,
|
2020-02-04 07:09:09 +08:00
|
|
|
kvm_pfn_t pfn, unsigned int access)
|
2011-07-12 03:33:44 +08:00
|
|
|
{
|
|
|
|
if (unlikely(is_noslot_pfn(pfn))) {
|
2015-04-08 21:39:23 +08:00
|
|
|
mark_mmio_spte(vcpu, sptep, gfn, access);
|
2011-07-12 03:33:44 +08:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
KVM: Allow not-present guest page faults to bypass kvm
There are two classes of page faults trapped by kvm:
- host page faults, where the fault is needed to allow kvm to install
the shadow pte or update the guest accessed and dirty bits
- guest page faults, where the guest has faulted and kvm simply injects
the fault back into the guest to handle
The second class, guest page faults, is pure overhead. We can eliminate
some of it on vmx using the following evil trick:
- when we set up a shadow page table entry, if the corresponding guest pte
is not present, set up the shadow pte as not present
- if the guest pte _is_ present, mark the shadow pte as present but also
set one of the reserved bits in the shadow pte
- tell the vmx hardware not to trap faults which have the present bit clear
With this, normal page-not-present faults go directly to the guest,
bypassing kvm entirely.
Unfortunately, this trick only works on Intel hardware, as AMD lacks a
way to discriminate among page faults based on error code. It is also
a little risky since it uses reserved bits which might become unreserved
in the future, so a module parameter is provided to disable it.
Signed-off-by: Avi Kivity <avi@qumranet.com>
2007-09-17 00:58:32 +08:00
|
|
|
|
2015-04-08 21:39:23 +08:00
|
|
|
static bool check_mmio_spte(struct kvm_vcpu *vcpu, u64 spte)
|
2013-06-07 16:51:26 +08:00
|
|
|
{
|
2019-02-06 05:01:16 +08:00
|
|
|
u64 kvm_gen, spte_gen, gen;
|
2013-06-07 16:51:27 +08:00
|
|
|
|
2019-02-06 05:01:16 +08:00
|
|
|
gen = kvm_vcpu_memslots(vcpu)->generation;
|
|
|
|
if (unlikely(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS))
|
|
|
|
return false;
|
2013-06-07 16:51:27 +08:00
|
|
|
|
2019-02-06 05:01:16 +08:00
|
|
|
kvm_gen = gen & MMIO_SPTE_GEN_MASK;
|
2013-06-07 16:51:27 +08:00
|
|
|
spte_gen = get_mmio_spte_generation(spte);
|
|
|
|
|
|
|
|
trace_check_mmio_spte(spte, kvm_gen, spte_gen);
|
|
|
|
return likely(kvm_gen == spte_gen);
|
2013-06-07 16:51:26 +08:00
|
|
|
}
|
|
|
|
|
2017-07-01 08:26:29 +08:00
|
|
|
/*
|
|
|
|
* Sets the shadow PTE masks used by the MMU.
|
|
|
|
*
|
|
|
|
* Assumptions:
|
|
|
|
* - Setting either @accessed_mask or @dirty_mask requires setting both
|
|
|
|
* - At least one of @accessed_mask or @acc_track_mask must be set
|
|
|
|
*/
|
2008-04-25 21:13:50 +08:00
|
|
|
void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask,
|
2016-12-07 08:46:16 +08:00
|
|
|
u64 dirty_mask, u64 nx_mask, u64 x_mask, u64 p_mask,
|
2017-07-18 05:10:27 +08:00
|
|
|
u64 acc_track_mask, u64 me_mask)
|
2008-04-25 21:13:50 +08:00
|
|
|
{
|
2017-07-01 08:26:29 +08:00
|
|
|
BUG_ON(!dirty_mask != !accessed_mask);
|
|
|
|
BUG_ON(!accessed_mask && !acc_track_mask);
|
2019-09-24 18:43:08 +08:00
|
|
|
BUG_ON(acc_track_mask & SPTE_SPECIAL_MASK);
|
2016-12-22 12:29:29 +08:00
|
|
|
|
2008-04-25 21:13:50 +08:00
|
|
|
shadow_user_mask = user_mask;
|
|
|
|
shadow_accessed_mask = accessed_mask;
|
|
|
|
shadow_dirty_mask = dirty_mask;
|
|
|
|
shadow_nx_mask = nx_mask;
|
|
|
|
shadow_x_mask = x_mask;
|
2016-07-13 06:18:49 +08:00
|
|
|
shadow_present_mask = p_mask;
|
2016-12-07 08:46:16 +08:00
|
|
|
shadow_acc_track_mask = acc_track_mask;
|
2017-07-18 05:10:27 +08:00
|
|
|
shadow_me_mask = me_mask;
|
2008-04-25 21:13:50 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_set_mask_ptes);
|
|
|
|
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
static u8 kvm_get_shadow_phys_bits(void)
|
|
|
|
{
|
|
|
|
/*
|
2019-12-04 22:50:27 +08:00
|
|
|
* boot_cpu_data.x86_phys_bits is reduced when MKTME or SME are detected
|
|
|
|
* in CPU detection code, but the processor treats those reduced bits as
|
|
|
|
* 'keyID' thus they are not reserved bits. Therefore KVM needs to look at
|
|
|
|
* the physical address bits reported by CPUID.
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
*/
|
2019-12-04 22:50:27 +08:00
|
|
|
if (likely(boot_cpu_data.extended_cpuid_level >= 0x80000008))
|
|
|
|
return cpuid_eax(0x80000008) & 0xff;
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
|
2019-12-04 22:50:27 +08:00
|
|
|
/*
|
|
|
|
* Quite weird to have VMX or SVM but not MAXPHYADDR; probably a VM with
|
|
|
|
* custom CPUID. Proceed with whatever the kernel found since these features
|
|
|
|
* aren't virtualizable (SME/SEV also require CPUIDs higher than 0x80000008).
|
|
|
|
*/
|
|
|
|
return boot_cpu_data.x86_phys_bits;
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
}
|
|
|
|
|
2018-08-15 01:15:34 +08:00
|
|
|
static void kvm_mmu_reset_all_pte_masks(void)
|
2016-12-07 08:46:16 +08:00
|
|
|
{
|
KVM: x86: fix L1TF's MMIO GFN calculation
One defense against L1TF in KVM is to always set the upper five bits
of the *legal* physical address in the SPTEs for non-present and
reserved SPTEs, e.g. MMIO SPTEs. In the MMIO case, the GFN of the
MMIO SPTE may overlap with the upper five bits that are being usurped
to defend against L1TF. To preserve the GFN, the bits of the GFN that
overlap with the repurposed bits are shifted left into the reserved
bits, i.e. the GFN in the SPTE will be split into high and low parts.
When retrieving the GFN from the MMIO SPTE, e.g. to check for an MMIO
access, get_mmio_spte_gfn() unshifts the affected bits and restores
the original GFN for comparison. Unfortunately, get_mmio_spte_gfn()
neglects to mask off the reserved bits in the SPTE that were used to
store the upper chunk of the GFN. As a result, KVM fails to detect
MMIO accesses whose GPA overlaps the repurprosed bits, which in turn
causes guest panics and hangs.
Fix the bug by generating a mask that covers the lower chunk of the
GFN, i.e. the bits that aren't shifted by the L1TF mitigation. The
alternative approach would be to explicitly zero the five reserved
bits that are used to store the upper chunk of the GFN, but that
requires additional run-time computation and makes an already-ugly
bit of code even more inscrutable.
I considered adding a WARN_ON_ONCE(low_phys_bits-1 <= PAGE_SHIFT) to
warn if GENMASK_ULL() generated a nonsensical value, but that seemed
silly since that would mean a system that supports VMX has less than
18 bits of physical address space...
Reported-by: Sakari Ailus <sakari.ailus@iki.fi>
Fixes: d9b47449c1a1 ("kvm: x86: Set highest physical address bits in non-present/reserved SPTEs")
Cc: Junaid Shahid <junaids@google.com>
Cc: Jim Mattson <jmattson@google.com>
Cc: stable@vger.kernel.org
Reviewed-by: Junaid Shahid <junaids@google.com>
Tested-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-09-26 04:20:00 +08:00
|
|
|
u8 low_phys_bits;
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
shadow_user_mask = 0;
|
|
|
|
shadow_accessed_mask = 0;
|
|
|
|
shadow_dirty_mask = 0;
|
|
|
|
shadow_nx_mask = 0;
|
|
|
|
shadow_x_mask = 0;
|
|
|
|
shadow_present_mask = 0;
|
|
|
|
shadow_acc_track_mask = 0;
|
2018-08-15 01:15:34 +08:00
|
|
|
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
shadow_phys_bits = kvm_get_shadow_phys_bits();
|
|
|
|
|
2018-08-15 01:15:34 +08:00
|
|
|
/*
|
|
|
|
* If the CPU has 46 or less physical address bits, then set an
|
|
|
|
* appropriate mask to guard against L1TF attacks. Otherwise, it is
|
|
|
|
* assumed that the CPU is not vulnerable to L1TF.
|
2019-05-03 16:40:25 +08:00
|
|
|
*
|
|
|
|
* Some Intel CPUs address the L1 cache using more PA bits than are
|
|
|
|
* reported by CPUID. Use the PA width of the L1 cache when possible
|
|
|
|
* to achieve more effective mitigation, e.g. if system RAM overlaps
|
|
|
|
* the most significant bits of legal physical address space.
|
2018-08-15 01:15:34 +08:00
|
|
|
*/
|
2019-05-03 16:40:25 +08:00
|
|
|
shadow_nonpresent_or_rsvd_mask = 0;
|
2020-05-19 17:34:41 +08:00
|
|
|
low_phys_bits = boot_cpu_data.x86_phys_bits;
|
|
|
|
if (boot_cpu_has_bug(X86_BUG_L1TF) &&
|
|
|
|
!WARN_ON_ONCE(boot_cpu_data.x86_cache_bits >=
|
|
|
|
52 - shadow_nonpresent_or_rsvd_mask_len)) {
|
|
|
|
low_phys_bits = boot_cpu_data.x86_cache_bits
|
|
|
|
- shadow_nonpresent_or_rsvd_mask_len;
|
2018-08-15 01:15:34 +08:00
|
|
|
shadow_nonpresent_or_rsvd_mask =
|
2020-05-19 17:34:41 +08:00
|
|
|
rsvd_bits(low_phys_bits, boot_cpu_data.x86_cache_bits - 1);
|
|
|
|
}
|
2019-05-03 16:40:25 +08:00
|
|
|
|
KVM: x86: fix L1TF's MMIO GFN calculation
One defense against L1TF in KVM is to always set the upper five bits
of the *legal* physical address in the SPTEs for non-present and
reserved SPTEs, e.g. MMIO SPTEs. In the MMIO case, the GFN of the
MMIO SPTE may overlap with the upper five bits that are being usurped
to defend against L1TF. To preserve the GFN, the bits of the GFN that
overlap with the repurposed bits are shifted left into the reserved
bits, i.e. the GFN in the SPTE will be split into high and low parts.
When retrieving the GFN from the MMIO SPTE, e.g. to check for an MMIO
access, get_mmio_spte_gfn() unshifts the affected bits and restores
the original GFN for comparison. Unfortunately, get_mmio_spte_gfn()
neglects to mask off the reserved bits in the SPTE that were used to
store the upper chunk of the GFN. As a result, KVM fails to detect
MMIO accesses whose GPA overlaps the repurprosed bits, which in turn
causes guest panics and hangs.
Fix the bug by generating a mask that covers the lower chunk of the
GFN, i.e. the bits that aren't shifted by the L1TF mitigation. The
alternative approach would be to explicitly zero the five reserved
bits that are used to store the upper chunk of the GFN, but that
requires additional run-time computation and makes an already-ugly
bit of code even more inscrutable.
I considered adding a WARN_ON_ONCE(low_phys_bits-1 <= PAGE_SHIFT) to
warn if GENMASK_ULL() generated a nonsensical value, but that seemed
silly since that would mean a system that supports VMX has less than
18 bits of physical address space...
Reported-by: Sakari Ailus <sakari.ailus@iki.fi>
Fixes: d9b47449c1a1 ("kvm: x86: Set highest physical address bits in non-present/reserved SPTEs")
Cc: Junaid Shahid <junaids@google.com>
Cc: Jim Mattson <jmattson@google.com>
Cc: stable@vger.kernel.org
Reviewed-by: Junaid Shahid <junaids@google.com>
Tested-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-09-26 04:20:00 +08:00
|
|
|
shadow_nonpresent_or_rsvd_lower_gfn_mask =
|
|
|
|
GENMASK_ULL(low_phys_bits - 1, PAGE_SHIFT);
|
2016-12-07 08:46:16 +08:00
|
|
|
}
|
|
|
|
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
static int is_cpuid_PSE36(void)
|
|
|
|
{
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2007-01-26 16:56:41 +08:00
|
|
|
static int is_nx(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2010-01-21 21:31:50 +08:00
|
|
|
return vcpu->arch.efer & EFER_NX;
|
2007-01-26 16:56:41 +08:00
|
|
|
}
|
|
|
|
|
KVM: Allow not-present guest page faults to bypass kvm
There are two classes of page faults trapped by kvm:
- host page faults, where the fault is needed to allow kvm to install
the shadow pte or update the guest accessed and dirty bits
- guest page faults, where the guest has faulted and kvm simply injects
the fault back into the guest to handle
The second class, guest page faults, is pure overhead. We can eliminate
some of it on vmx using the following evil trick:
- when we set up a shadow page table entry, if the corresponding guest pte
is not present, set up the shadow pte as not present
- if the guest pte _is_ present, mark the shadow pte as present but also
set one of the reserved bits in the shadow pte
- tell the vmx hardware not to trap faults which have the present bit clear
With this, normal page-not-present faults go directly to the guest,
bypassing kvm entirely.
Unfortunately, this trick only works on Intel hardware, as AMD lacks a
way to discriminate among page faults based on error code. It is also
a little risky since it uses reserved bits which might become unreserved
in the future, so a module parameter is provided to disable it.
Signed-off-by: Avi Kivity <avi@qumranet.com>
2007-09-17 00:58:32 +08:00
|
|
|
static int is_shadow_present_pte(u64 pte)
|
|
|
|
{
|
2016-12-07 08:46:16 +08:00
|
|
|
return (pte != 0) && !is_mmio_spte(pte);
|
KVM: Allow not-present guest page faults to bypass kvm
There are two classes of page faults trapped by kvm:
- host page faults, where the fault is needed to allow kvm to install
the shadow pte or update the guest accessed and dirty bits
- guest page faults, where the guest has faulted and kvm simply injects
the fault back into the guest to handle
The second class, guest page faults, is pure overhead. We can eliminate
some of it on vmx using the following evil trick:
- when we set up a shadow page table entry, if the corresponding guest pte
is not present, set up the shadow pte as not present
- if the guest pte _is_ present, mark the shadow pte as present but also
set one of the reserved bits in the shadow pte
- tell the vmx hardware not to trap faults which have the present bit clear
With this, normal page-not-present faults go directly to the guest,
bypassing kvm entirely.
Unfortunately, this trick only works on Intel hardware, as AMD lacks a
way to discriminate among page faults based on error code. It is also
a little risky since it uses reserved bits which might become unreserved
in the future, so a module parameter is provided to disable it.
Signed-off-by: Avi Kivity <avi@qumranet.com>
2007-09-17 00:58:32 +08:00
|
|
|
}
|
|
|
|
|
2008-02-23 22:44:30 +08:00
|
|
|
static int is_large_pte(u64 pte)
|
|
|
|
{
|
|
|
|
return pte & PT_PAGE_SIZE_MASK;
|
|
|
|
}
|
|
|
|
|
2009-06-10 23:27:03 +08:00
|
|
|
static int is_last_spte(u64 pte, int level)
|
|
|
|
{
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level == PG_LEVEL_4K)
|
2009-06-10 23:27:03 +08:00
|
|
|
return 1;
|
2009-07-27 22:30:44 +08:00
|
|
|
if (is_large_pte(pte))
|
2009-06-10 23:27:03 +08:00
|
|
|
return 1;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-12-22 12:29:32 +08:00
|
|
|
static bool is_executable_pte(u64 spte)
|
|
|
|
{
|
|
|
|
return (spte & (shadow_x_mask | shadow_nx_mask)) == shadow_x_mask;
|
|
|
|
}
|
|
|
|
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
static kvm_pfn_t spte_to_pfn(u64 pte)
|
2008-03-23 21:06:23 +08:00
|
|
|
{
|
2008-04-03 03:46:56 +08:00
|
|
|
return (pte & PT64_BASE_ADDR_MASK) >> PAGE_SHIFT;
|
2008-03-23 21:06:23 +08:00
|
|
|
}
|
|
|
|
|
2007-11-21 19:54:47 +08:00
|
|
|
static gfn_t pse36_gfn_delta(u32 gpte)
|
|
|
|
{
|
|
|
|
int shift = 32 - PT32_DIR_PSE36_SHIFT - PAGE_SHIFT;
|
|
|
|
|
|
|
|
return (gpte & PT32_DIR_PSE36_MASK) << shift;
|
|
|
|
}
|
|
|
|
|
2011-07-12 03:31:28 +08:00
|
|
|
#ifdef CONFIG_X86_64
|
2009-06-10 19:24:23 +08:00
|
|
|
static void __set_spte(u64 *sptep, u64 spte)
|
2007-05-31 20:46:04 +08:00
|
|
|
{
|
2016-05-11 23:04:29 +08:00
|
|
|
WRITE_ONCE(*sptep, spte);
|
2007-05-31 20:46:04 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:31:28 +08:00
|
|
|
static void __update_clear_spte_fast(u64 *sptep, u64 spte)
|
2010-06-06 19:48:06 +08:00
|
|
|
{
|
2016-05-11 23:04:29 +08:00
|
|
|
WRITE_ONCE(*sptep, spte);
|
2011-07-12 03:31:28 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static u64 __update_clear_spte_slow(u64 *sptep, u64 spte)
|
|
|
|
{
|
|
|
|
return xchg(sptep, spte);
|
|
|
|
}
|
2011-07-12 03:32:13 +08:00
|
|
|
|
|
|
|
static u64 __get_spte_lockless(u64 *sptep)
|
|
|
|
{
|
locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE()
Please do not apply this to mainline directly, instead please re-run the
coccinelle script shown below and apply its output.
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't harmful, and changing them results in
churn.
However, for some features, the read/write distinction is critical to
correct operation. To distinguish these cases, separate read/write
accessors must be used. This patch migrates (most) remaining
ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following
coccinelle script:
----
// Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and
// WRITE_ONCE()
// $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davem@davemloft.net
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-24 05:07:29 +08:00
|
|
|
return READ_ONCE(*sptep);
|
2011-07-12 03:32:13 +08:00
|
|
|
}
|
2010-06-06 19:48:06 +08:00
|
|
|
#else
|
2011-07-12 03:31:28 +08:00
|
|
|
union split_spte {
|
|
|
|
struct {
|
|
|
|
u32 spte_low;
|
|
|
|
u32 spte_high;
|
|
|
|
};
|
|
|
|
u64 spte;
|
|
|
|
};
|
2010-06-06 19:48:06 +08:00
|
|
|
|
2011-07-12 03:32:13 +08:00
|
|
|
static void count_spte_clear(u64 *sptep, u64 spte)
|
|
|
|
{
|
2020-06-23 04:20:33 +08:00
|
|
|
struct kvm_mmu_page *sp = sptep_to_sp(sptep);
|
2011-07-12 03:32:13 +08:00
|
|
|
|
|
|
|
if (is_shadow_present_pte(spte))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Ensure the spte is completely set before we increase the count */
|
|
|
|
smp_wmb();
|
|
|
|
sp->clear_spte_count++;
|
|
|
|
}
|
|
|
|
|
2011-07-12 03:31:28 +08:00
|
|
|
static void __set_spte(u64 *sptep, u64 spte)
|
|
|
|
{
|
|
|
|
union split_spte *ssptep, sspte;
|
2010-06-06 19:48:06 +08:00
|
|
|
|
2011-07-12 03:31:28 +08:00
|
|
|
ssptep = (union split_spte *)sptep;
|
|
|
|
sspte = (union split_spte)spte;
|
|
|
|
|
|
|
|
ssptep->spte_high = sspte.spte_high;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we map the spte from nonpresent to present, We should store
|
|
|
|
* the high bits firstly, then set present bit, so cpu can not
|
|
|
|
* fetch this spte while we are setting the spte.
|
|
|
|
*/
|
|
|
|
smp_wmb();
|
|
|
|
|
2016-05-11 23:04:29 +08:00
|
|
|
WRITE_ONCE(ssptep->spte_low, sspte.spte_low);
|
2010-06-06 19:48:06 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:31:28 +08:00
|
|
|
static void __update_clear_spte_fast(u64 *sptep, u64 spte)
|
|
|
|
{
|
|
|
|
union split_spte *ssptep, sspte;
|
|
|
|
|
|
|
|
ssptep = (union split_spte *)sptep;
|
|
|
|
sspte = (union split_spte)spte;
|
|
|
|
|
2016-05-11 23:04:29 +08:00
|
|
|
WRITE_ONCE(ssptep->spte_low, sspte.spte_low);
|
2011-07-12 03:31:28 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If we map the spte from present to nonpresent, we should clear
|
|
|
|
* present bit firstly to avoid vcpu fetch the old high bits.
|
|
|
|
*/
|
|
|
|
smp_wmb();
|
|
|
|
|
|
|
|
ssptep->spte_high = sspte.spte_high;
|
2011-07-12 03:32:13 +08:00
|
|
|
count_spte_clear(sptep, spte);
|
2011-07-12 03:31:28 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static u64 __update_clear_spte_slow(u64 *sptep, u64 spte)
|
|
|
|
{
|
|
|
|
union split_spte *ssptep, sspte, orig;
|
|
|
|
|
|
|
|
ssptep = (union split_spte *)sptep;
|
|
|
|
sspte = (union split_spte)spte;
|
|
|
|
|
|
|
|
/* xchg acts as a barrier before the setting of the high bits */
|
|
|
|
orig.spte_low = xchg(&ssptep->spte_low, sspte.spte_low);
|
2011-09-19 12:19:51 +08:00
|
|
|
orig.spte_high = ssptep->spte_high;
|
|
|
|
ssptep->spte_high = sspte.spte_high;
|
2011-07-12 03:32:13 +08:00
|
|
|
count_spte_clear(sptep, spte);
|
2011-07-12 03:31:28 +08:00
|
|
|
|
|
|
|
return orig.spte;
|
|
|
|
}
|
2011-07-12 03:32:13 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The idea using the light way get the spte on x86_32 guest is from
|
2019-07-12 11:56:49 +08:00
|
|
|
* gup_get_pte (mm/gup.c).
|
2013-06-19 17:09:20 +08:00
|
|
|
*
|
|
|
|
* An spte tlb flush may be pending, because kvm_set_pte_rmapp
|
|
|
|
* coalesces them and we are running out of the MMU lock. Therefore
|
|
|
|
* we need to protect against in-progress updates of the spte.
|
|
|
|
*
|
|
|
|
* Reading the spte while an update is in progress may get the old value
|
|
|
|
* for the high part of the spte. The race is fine for a present->non-present
|
|
|
|
* change (because the high part of the spte is ignored for non-present spte),
|
|
|
|
* but for a present->present change we must reread the spte.
|
|
|
|
*
|
|
|
|
* All such changes are done in two steps (present->non-present and
|
|
|
|
* non-present->present), hence it is enough to count the number of
|
|
|
|
* present->non-present updates: if it changed while reading the spte,
|
|
|
|
* we might have hit the race. This is done using clear_spte_count.
|
2011-07-12 03:32:13 +08:00
|
|
|
*/
|
|
|
|
static u64 __get_spte_lockless(u64 *sptep)
|
|
|
|
{
|
2020-06-23 04:20:33 +08:00
|
|
|
struct kvm_mmu_page *sp = sptep_to_sp(sptep);
|
2011-07-12 03:32:13 +08:00
|
|
|
union split_spte spte, *orig = (union split_spte *)sptep;
|
|
|
|
int count;
|
|
|
|
|
|
|
|
retry:
|
|
|
|
count = sp->clear_spte_count;
|
|
|
|
smp_rmb();
|
|
|
|
|
|
|
|
spte.spte_low = orig->spte_low;
|
|
|
|
smp_rmb();
|
|
|
|
|
|
|
|
spte.spte_high = orig->spte_high;
|
|
|
|
smp_rmb();
|
|
|
|
|
|
|
|
if (unlikely(spte.spte_low != orig->spte_low ||
|
|
|
|
count != sp->clear_spte_count))
|
|
|
|
goto retry;
|
|
|
|
|
|
|
|
return spte.spte;
|
|
|
|
}
|
2011-07-12 03:31:28 +08:00
|
|
|
#endif
|
|
|
|
|
2016-12-07 08:46:11 +08:00
|
|
|
static bool spte_can_locklessly_be_made_writable(u64 spte)
|
2012-06-20 15:59:18 +08:00
|
|
|
{
|
2013-01-30 22:45:00 +08:00
|
|
|
return (spte & (SPTE_HOST_WRITEABLE | SPTE_MMU_WRITEABLE)) ==
|
|
|
|
(SPTE_HOST_WRITEABLE | SPTE_MMU_WRITEABLE);
|
2012-06-20 15:59:18 +08:00
|
|
|
}
|
|
|
|
|
2010-08-02 16:14:04 +08:00
|
|
|
static bool spte_has_volatile_bits(u64 spte)
|
|
|
|
{
|
2016-12-07 08:46:16 +08:00
|
|
|
if (!is_shadow_present_pte(spte))
|
|
|
|
return false;
|
|
|
|
|
2012-06-20 15:59:18 +08:00
|
|
|
/*
|
2016-02-24 07:34:30 +08:00
|
|
|
* Always atomically update spte if it can be updated
|
2012-06-20 15:59:18 +08:00
|
|
|
* out of mmu-lock, it can ensure dirty bit is not lost,
|
|
|
|
* also, it can help us to get a stable is_writable_pte()
|
|
|
|
* to ensure tlb flush is not missed.
|
|
|
|
*/
|
2016-12-07 08:46:16 +08:00
|
|
|
if (spte_can_locklessly_be_made_writable(spte) ||
|
|
|
|
is_access_track_spte(spte))
|
2012-06-20 15:59:18 +08:00
|
|
|
return true;
|
|
|
|
|
2017-07-01 08:26:31 +08:00
|
|
|
if (spte_ad_enabled(spte)) {
|
2016-12-07 08:46:16 +08:00
|
|
|
if ((spte & shadow_accessed_mask) == 0 ||
|
|
|
|
(is_writable_pte(spte) && (spte & shadow_dirty_mask) == 0))
|
|
|
|
return true;
|
|
|
|
}
|
2010-08-02 16:14:04 +08:00
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
return false;
|
2010-08-02 16:14:04 +08:00
|
|
|
}
|
|
|
|
|
2016-12-07 08:46:13 +08:00
|
|
|
static bool is_accessed_spte(u64 spte)
|
2010-08-02 16:15:08 +08:00
|
|
|
{
|
2017-07-01 08:26:31 +08:00
|
|
|
u64 accessed_mask = spte_shadow_accessed_mask(spte);
|
|
|
|
|
|
|
|
return accessed_mask ? spte & accessed_mask
|
|
|
|
: !is_access_track_spte(spte);
|
2010-08-02 16:15:08 +08:00
|
|
|
}
|
|
|
|
|
2016-12-07 08:46:13 +08:00
|
|
|
static bool is_dirty_spte(u64 spte)
|
2015-01-09 16:44:30 +08:00
|
|
|
{
|
2017-07-01 08:26:31 +08:00
|
|
|
u64 dirty_mask = spte_shadow_dirty_mask(spte);
|
|
|
|
|
|
|
|
return dirty_mask ? spte & dirty_mask : spte & PT_WRITABLE_MASK;
|
2015-01-09 16:44:30 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:30:35 +08:00
|
|
|
/* Rules for using mmu_spte_set:
|
|
|
|
* Set the sptep from nonpresent to present.
|
|
|
|
* Note: the sptep being assigned *must* be either not present
|
|
|
|
* or in a state where the hardware will not attempt to update
|
|
|
|
* the spte.
|
|
|
|
*/
|
|
|
|
static void mmu_spte_set(u64 *sptep, u64 new_spte)
|
|
|
|
{
|
|
|
|
WARN_ON(is_shadow_present_pte(*sptep));
|
|
|
|
__set_spte(sptep, new_spte);
|
|
|
|
}
|
|
|
|
|
2016-12-07 08:46:14 +08:00
|
|
|
/*
|
|
|
|
* Update the SPTE (excluding the PFN), but do not track changes in its
|
|
|
|
* accessed/dirty status.
|
2011-07-12 03:30:35 +08:00
|
|
|
*/
|
2016-12-07 08:46:14 +08:00
|
|
|
static u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte)
|
2010-06-06 20:46:44 +08:00
|
|
|
{
|
2012-06-20 15:59:18 +08:00
|
|
|
u64 old_spte = *sptep;
|
2010-08-02 16:15:08 +08:00
|
|
|
|
2015-11-20 16:44:55 +08:00
|
|
|
WARN_ON(!is_shadow_present_pte(new_spte));
|
2010-06-06 20:46:44 +08:00
|
|
|
|
2012-06-20 15:58:33 +08:00
|
|
|
if (!is_shadow_present_pte(old_spte)) {
|
|
|
|
mmu_spte_set(sptep, new_spte);
|
2016-12-07 08:46:14 +08:00
|
|
|
return old_spte;
|
2012-06-20 15:58:33 +08:00
|
|
|
}
|
2010-08-02 16:15:08 +08:00
|
|
|
|
2012-06-20 15:59:18 +08:00
|
|
|
if (!spte_has_volatile_bits(old_spte))
|
2011-07-12 03:31:28 +08:00
|
|
|
__update_clear_spte_fast(sptep, new_spte);
|
2010-08-02 16:15:08 +08:00
|
|
|
else
|
2011-07-12 03:31:28 +08:00
|
|
|
old_spte = __update_clear_spte_slow(sptep, new_spte);
|
2010-08-02 16:15:08 +08:00
|
|
|
|
2016-12-07 08:46:13 +08:00
|
|
|
WARN_ON(spte_to_pfn(old_spte) != spte_to_pfn(new_spte));
|
|
|
|
|
2016-12-07 08:46:14 +08:00
|
|
|
return old_spte;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Rules for using mmu_spte_update:
|
|
|
|
* Update the state bits, it means the mapped pfn is not changed.
|
|
|
|
*
|
|
|
|
* Whenever we overwrite a writable spte with a read-only one we
|
|
|
|
* should flush remote TLBs. Otherwise rmap_write_protect
|
|
|
|
* will find a read-only spte, even though the writable spte
|
|
|
|
* might be cached on a CPU's TLB, the return value indicates this
|
|
|
|
* case.
|
|
|
|
*
|
|
|
|
* Returns true if the TLB needs to be flushed
|
|
|
|
*/
|
|
|
|
static bool mmu_spte_update(u64 *sptep, u64 new_spte)
|
|
|
|
{
|
|
|
|
bool flush = false;
|
|
|
|
u64 old_spte = mmu_spte_update_no_track(sptep, new_spte);
|
|
|
|
|
|
|
|
if (!is_shadow_present_pte(old_spte))
|
|
|
|
return false;
|
|
|
|
|
2012-06-20 15:59:18 +08:00
|
|
|
/*
|
|
|
|
* For the spte updated out of mmu-lock is safe, since
|
2016-02-24 07:34:30 +08:00
|
|
|
* we always atomically update it, see the comments in
|
2012-06-20 15:59:18 +08:00
|
|
|
* spte_has_volatile_bits().
|
|
|
|
*/
|
2016-12-07 08:46:11 +08:00
|
|
|
if (spte_can_locklessly_be_made_writable(old_spte) &&
|
2014-04-17 17:06:15 +08:00
|
|
|
!is_writable_pte(new_spte))
|
2016-12-07 08:46:13 +08:00
|
|
|
flush = true;
|
2010-08-02 16:15:08 +08:00
|
|
|
|
2015-01-09 16:44:30 +08:00
|
|
|
/*
|
2016-12-07 08:46:13 +08:00
|
|
|
* Flush TLB when accessed/dirty states are changed in the page tables,
|
2015-01-09 16:44:30 +08:00
|
|
|
* to guarantee consistency between TLB and page tables.
|
|
|
|
*/
|
|
|
|
|
2016-12-07 08:46:13 +08:00
|
|
|
if (is_accessed_spte(old_spte) && !is_accessed_spte(new_spte)) {
|
|
|
|
flush = true;
|
2010-08-02 16:15:08 +08:00
|
|
|
kvm_set_pfn_accessed(spte_to_pfn(old_spte));
|
2016-12-07 08:46:13 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
if (is_dirty_spte(old_spte) && !is_dirty_spte(new_spte)) {
|
|
|
|
flush = true;
|
2010-08-02 16:15:08 +08:00
|
|
|
kvm_set_pfn_dirty(spte_to_pfn(old_spte));
|
2016-12-07 08:46:13 +08:00
|
|
|
}
|
2012-06-20 15:58:33 +08:00
|
|
|
|
2016-12-07 08:46:13 +08:00
|
|
|
return flush;
|
2010-06-06 20:46:44 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:30:35 +08:00
|
|
|
/*
|
|
|
|
* Rules for using mmu_spte_clear_track_bits:
|
|
|
|
* It sets the sptep from present to nonpresent, and track the
|
|
|
|
* state bits, it is used to clear the last level sptep.
|
2016-12-07 08:46:13 +08:00
|
|
|
* Returns non-zero if the PTE was previously valid.
|
2011-07-12 03:30:35 +08:00
|
|
|
*/
|
|
|
|
static int mmu_spte_clear_track_bits(u64 *sptep)
|
|
|
|
{
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
kvm_pfn_t pfn;
|
2011-07-12 03:30:35 +08:00
|
|
|
u64 old_spte = *sptep;
|
|
|
|
|
|
|
|
if (!spte_has_volatile_bits(old_spte))
|
2011-07-12 03:31:28 +08:00
|
|
|
__update_clear_spte_fast(sptep, 0ull);
|
2011-07-12 03:30:35 +08:00
|
|
|
else
|
2011-07-12 03:31:28 +08:00
|
|
|
old_spte = __update_clear_spte_slow(sptep, 0ull);
|
2011-07-12 03:30:35 +08:00
|
|
|
|
2015-11-20 16:44:55 +08:00
|
|
|
if (!is_shadow_present_pte(old_spte))
|
2011-07-12 03:30:35 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
pfn = spte_to_pfn(old_spte);
|
2012-07-17 21:52:52 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* KVM does not hold the refcount of the page used by
|
|
|
|
* kvm mmu, before reclaiming the page, we should
|
|
|
|
* unmap it from mmu first.
|
|
|
|
*/
|
2014-11-10 16:33:56 +08:00
|
|
|
WARN_ON(!kvm_is_reserved_pfn(pfn) && !page_count(pfn_to_page(pfn)));
|
2012-07-17 21:52:52 +08:00
|
|
|
|
2016-12-07 08:46:13 +08:00
|
|
|
if (is_accessed_spte(old_spte))
|
2011-07-12 03:30:35 +08:00
|
|
|
kvm_set_pfn_accessed(pfn);
|
2016-12-07 08:46:13 +08:00
|
|
|
|
|
|
|
if (is_dirty_spte(old_spte))
|
2011-07-12 03:30:35 +08:00
|
|
|
kvm_set_pfn_dirty(pfn);
|
2016-12-07 08:46:13 +08:00
|
|
|
|
2011-07-12 03:30:35 +08:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Rules for using mmu_spte_clear_no_track:
|
|
|
|
* Directly clear spte without caring the state bits of sptep,
|
|
|
|
* it is used to set the upper level spte.
|
|
|
|
*/
|
|
|
|
static void mmu_spte_clear_no_track(u64 *sptep)
|
|
|
|
{
|
2011-07-12 03:31:28 +08:00
|
|
|
__update_clear_spte_fast(sptep, 0ull);
|
2011-07-12 03:30:35 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:32:13 +08:00
|
|
|
static u64 mmu_spte_get_lockless(u64 *sptep)
|
|
|
|
{
|
|
|
|
return __get_spte_lockless(sptep);
|
|
|
|
}
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
static u64 mark_spte_for_access_track(u64 spte)
|
|
|
|
{
|
2017-07-01 08:26:31 +08:00
|
|
|
if (spte_ad_enabled(spte))
|
2016-12-07 08:46:16 +08:00
|
|
|
return spte & ~shadow_accessed_mask;
|
|
|
|
|
2017-07-01 08:26:31 +08:00
|
|
|
if (is_access_track_spte(spte))
|
2016-12-07 08:46:16 +08:00
|
|
|
return spte;
|
|
|
|
|
|
|
|
/*
|
2016-12-22 12:29:31 +08:00
|
|
|
* Making an Access Tracking PTE will result in removal of write access
|
|
|
|
* from the PTE. So, verify that we will be able to restore the write
|
|
|
|
* access in the fast page fault path later on.
|
2016-12-07 08:46:16 +08:00
|
|
|
*/
|
|
|
|
WARN_ONCE((spte & PT_WRITABLE_MASK) &&
|
|
|
|
!spte_can_locklessly_be_made_writable(spte),
|
|
|
|
"kvm: Writable SPTE is not locklessly dirty-trackable\n");
|
|
|
|
|
|
|
|
WARN_ONCE(spte & (shadow_acc_track_saved_bits_mask <<
|
|
|
|
shadow_acc_track_saved_bits_shift),
|
|
|
|
"kvm: Access Tracking saved bit locations are not zero\n");
|
|
|
|
|
|
|
|
spte |= (spte & shadow_acc_track_saved_bits_mask) <<
|
|
|
|
shadow_acc_track_saved_bits_shift;
|
|
|
|
spte &= ~shadow_acc_track_mask;
|
|
|
|
|
|
|
|
return spte;
|
|
|
|
}
|
|
|
|
|
2016-12-22 12:29:32 +08:00
|
|
|
/* Restore an acc-track PTE back to a regular PTE */
|
|
|
|
static u64 restore_acc_track_spte(u64 spte)
|
|
|
|
{
|
|
|
|
u64 new_spte = spte;
|
|
|
|
u64 saved_bits = (spte >> shadow_acc_track_saved_bits_shift)
|
|
|
|
& shadow_acc_track_saved_bits_mask;
|
|
|
|
|
2017-07-01 08:26:31 +08:00
|
|
|
WARN_ON_ONCE(spte_ad_enabled(spte));
|
2016-12-22 12:29:32 +08:00
|
|
|
WARN_ON_ONCE(!is_access_track_spte(spte));
|
|
|
|
|
|
|
|
new_spte &= ~shadow_acc_track_mask;
|
|
|
|
new_spte &= ~(shadow_acc_track_saved_bits_mask <<
|
|
|
|
shadow_acc_track_saved_bits_shift);
|
|
|
|
new_spte |= saved_bits;
|
|
|
|
|
|
|
|
return new_spte;
|
|
|
|
}
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
/* Returns the Accessed status of the PTE and resets it at the same time. */
|
|
|
|
static bool mmu_spte_age(u64 *sptep)
|
|
|
|
{
|
|
|
|
u64 spte = mmu_spte_get_lockless(sptep);
|
|
|
|
|
|
|
|
if (!is_accessed_spte(spte))
|
|
|
|
return false;
|
|
|
|
|
2017-07-01 08:26:31 +08:00
|
|
|
if (spte_ad_enabled(spte)) {
|
2016-12-07 08:46:16 +08:00
|
|
|
clear_bit((ffs(shadow_accessed_mask) - 1),
|
|
|
|
(unsigned long *)sptep);
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Capture the dirty status of the page, so that it doesn't get
|
|
|
|
* lost when the SPTE is marked for access tracking.
|
|
|
|
*/
|
|
|
|
if (is_writable_pte(spte))
|
|
|
|
kvm_set_pfn_dirty(spte_to_pfn(spte));
|
|
|
|
|
|
|
|
spte = mark_spte_for_access_track(spte);
|
|
|
|
mmu_spte_update_no_track(sptep, spte);
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2011-07-12 03:32:13 +08:00
|
|
|
static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2012-05-14 20:44:06 +08:00
|
|
|
/*
|
|
|
|
* Prevent page table teardown by making any free-er wait during
|
|
|
|
* kvm_flush_remote_tlbs() IPI to all active vcpus.
|
|
|
|
*/
|
|
|
|
local_irq_disable();
|
2016-03-13 11:10:25 +08:00
|
|
|
|
2012-05-14 20:44:06 +08:00
|
|
|
/*
|
|
|
|
* Make sure a following spte read is not reordered ahead of the write
|
|
|
|
* to vcpu->mode.
|
|
|
|
*/
|
2016-03-13 11:10:25 +08:00
|
|
|
smp_store_mb(vcpu->mode, READING_SHADOW_PAGE_TABLES);
|
2011-07-12 03:32:13 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2012-05-14 20:44:06 +08:00
|
|
|
/*
|
|
|
|
* Make sure the write to vcpu->mode is not reordered in front of
|
2018-09-07 13:45:02 +08:00
|
|
|
* reads to sptes. If it does, kvm_mmu_commit_zap_page() can see us
|
2012-05-14 20:44:06 +08:00
|
|
|
* OUTSIDE_GUEST_MODE and proceed to free the shadow page table.
|
|
|
|
*/
|
2016-03-13 11:10:25 +08:00
|
|
|
smp_store_release(&vcpu->mode, OUTSIDE_GUEST_MODE);
|
2012-05-14 20:44:06 +08:00
|
|
|
local_irq_enable();
|
2011-07-12 03:32:13 +08:00
|
|
|
}
|
|
|
|
|
2007-01-06 08:36:54 +08:00
|
|
|
static int mmu_topup_memory_cache(struct kvm_mmu_memory_cache *cache,
|
2007-09-10 16:28:17 +08:00
|
|
|
struct kmem_cache *base_cache, int min)
|
2007-01-06 08:36:53 +08:00
|
|
|
{
|
|
|
|
void *obj;
|
|
|
|
|
|
|
|
if (cache->nobjs >= min)
|
2007-01-06 08:36:54 +08:00
|
|
|
return 0;
|
2007-01-06 08:36:53 +08:00
|
|
|
while (cache->nobjs < ARRAY_SIZE(cache->objects)) {
|
2019-02-12 03:02:50 +08:00
|
|
|
obj = kmem_cache_zalloc(base_cache, GFP_KERNEL_ACCOUNT);
|
2007-01-06 08:36:53 +08:00
|
|
|
if (!obj)
|
2018-09-04 23:57:32 +08:00
|
|
|
return cache->nobjs >= min ? 0 : -ENOMEM;
|
2007-01-06 08:36:53 +08:00
|
|
|
cache->objects[cache->nobjs++] = obj;
|
|
|
|
}
|
2007-01-06 08:36:54 +08:00
|
|
|
return 0;
|
2007-01-06 08:36:53 +08:00
|
|
|
}
|
|
|
|
|
2011-09-22 16:53:17 +08:00
|
|
|
static int mmu_memory_cache_free_objects(struct kvm_mmu_memory_cache *cache)
|
|
|
|
{
|
|
|
|
return cache->nobjs;
|
|
|
|
}
|
|
|
|
|
2010-05-13 10:06:02 +08:00
|
|
|
static void mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc,
|
|
|
|
struct kmem_cache *cache)
|
2007-01-06 08:36:53 +08:00
|
|
|
{
|
|
|
|
while (mc->nobjs)
|
2010-05-13 10:06:02 +08:00
|
|
|
kmem_cache_free(cache, mc->objects[--mc->nobjs]);
|
2007-01-06 08:36:53 +08:00
|
|
|
}
|
|
|
|
|
2007-07-20 13:18:27 +08:00
|
|
|
static int mmu_topup_memory_cache_page(struct kvm_mmu_memory_cache *cache,
|
2007-09-10 16:28:17 +08:00
|
|
|
int min)
|
2007-07-20 13:18:27 +08:00
|
|
|
{
|
2011-03-04 19:01:10 +08:00
|
|
|
void *page;
|
2007-07-20 13:18:27 +08:00
|
|
|
|
|
|
|
if (cache->nobjs >= min)
|
|
|
|
return 0;
|
|
|
|
while (cache->nobjs < ARRAY_SIZE(cache->objects)) {
|
2018-07-27 07:37:45 +08:00
|
|
|
page = (void *)__get_free_page(GFP_KERNEL_ACCOUNT);
|
2007-07-20 13:18:27 +08:00
|
|
|
if (!page)
|
2018-09-04 23:57:32 +08:00
|
|
|
return cache->nobjs >= min ? 0 : -ENOMEM;
|
2011-03-04 19:01:10 +08:00
|
|
|
cache->objects[cache->nobjs++] = page;
|
2007-07-20 13:18:27 +08:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void mmu_free_memory_cache_page(struct kvm_mmu_memory_cache *mc)
|
|
|
|
{
|
|
|
|
while (mc->nobjs)
|
2007-07-21 14:06:46 +08:00
|
|
|
free_page((unsigned long)mc->objects[--mc->nobjs]);
|
2007-07-20 13:18:27 +08:00
|
|
|
}
|
|
|
|
|
2007-09-10 16:28:17 +08:00
|
|
|
static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu)
|
2007-01-06 08:36:53 +08:00
|
|
|
{
|
2007-01-06 08:36:54 +08:00
|
|
|
int r;
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
r = mmu_topup_memory_cache(&vcpu->arch.mmu_pte_list_desc_cache,
|
2011-05-15 23:27:08 +08:00
|
|
|
pte_list_desc_cache, 8 + PTE_PREFETCH_NUM);
|
2007-05-30 17:34:53 +08:00
|
|
|
if (r)
|
|
|
|
goto out;
|
2007-12-13 23:50:52 +08:00
|
|
|
r = mmu_topup_memory_cache_page(&vcpu->arch.mmu_page_cache, 8);
|
2007-05-30 17:34:53 +08:00
|
|
|
if (r)
|
|
|
|
goto out;
|
2007-12-13 23:50:52 +08:00
|
|
|
r = mmu_topup_memory_cache(&vcpu->arch.mmu_page_header_cache,
|
2007-09-10 16:28:17 +08:00
|
|
|
mmu_page_header_cache, 4);
|
2007-01-06 08:36:54 +08:00
|
|
|
out:
|
|
|
|
return r;
|
2007-01-06 08:36:53 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2011-05-15 23:26:20 +08:00
|
|
|
mmu_free_memory_cache(&vcpu->arch.mmu_pte_list_desc_cache,
|
|
|
|
pte_list_desc_cache);
|
2007-12-13 23:50:52 +08:00
|
|
|
mmu_free_memory_cache_page(&vcpu->arch.mmu_page_cache);
|
2010-05-13 10:06:02 +08:00
|
|
|
mmu_free_memory_cache(&vcpu->arch.mmu_page_header_cache,
|
|
|
|
mmu_page_header_cache);
|
2007-01-06 08:36:53 +08:00
|
|
|
}
|
|
|
|
|
2012-05-29 22:54:26 +08:00
|
|
|
static void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc)
|
2007-01-06 08:36:53 +08:00
|
|
|
{
|
|
|
|
void *p;
|
|
|
|
|
|
|
|
BUG_ON(!mc->nobjs);
|
|
|
|
p = mc->objects[--mc->nobjs];
|
|
|
|
return p;
|
|
|
|
}
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
static struct pte_list_desc *mmu_alloc_pte_list_desc(struct kvm_vcpu *vcpu)
|
2007-01-06 08:36:53 +08:00
|
|
|
{
|
2012-05-29 22:54:26 +08:00
|
|
|
return mmu_memory_cache_alloc(&vcpu->arch.mmu_pte_list_desc_cache);
|
2007-01-06 08:36:53 +08:00
|
|
|
}
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
static void mmu_free_pte_list_desc(struct pte_list_desc *pte_list_desc)
|
2007-01-06 08:36:53 +08:00
|
|
|
{
|
2011-05-15 23:26:20 +08:00
|
|
|
kmem_cache_free(pte_list_desc_cache, pte_list_desc);
|
2007-01-06 08:36:53 +08:00
|
|
|
}
|
|
|
|
|
2010-05-26 16:49:59 +08:00
|
|
|
static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index)
|
|
|
|
{
|
|
|
|
if (!sp->role.direct)
|
|
|
|
return sp->gfns[index];
|
|
|
|
|
|
|
|
return sp->gfn + (index << ((sp->role.level - 1) * PT64_LEVEL_BITS));
|
|
|
|
}
|
|
|
|
|
|
|
|
static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)
|
|
|
|
{
|
2019-06-30 20:36:21 +08:00
|
|
|
if (!sp->role.direct) {
|
2010-05-26 16:49:59 +08:00
|
|
|
sp->gfns[index] = gfn;
|
2019-06-30 20:36:21 +08:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (WARN_ON(gfn != kvm_mmu_page_get_gfn(sp, index)))
|
|
|
|
pr_err_ratelimited("gfn mismatch under direct page %llx "
|
|
|
|
"(expected %llx, got %llx)\n",
|
|
|
|
sp->gfn,
|
|
|
|
kvm_mmu_page_get_gfn(sp, index), gfn);
|
2010-05-26 16:49:59 +08:00
|
|
|
}
|
|
|
|
|
2008-02-23 22:44:30 +08:00
|
|
|
/*
|
2010-12-07 11:59:07 +08:00
|
|
|
* Return the pointer to the large page information for a given gfn,
|
|
|
|
* handling slots that are not large page aligned.
|
2008-02-23 22:44:30 +08:00
|
|
|
*/
|
2010-12-07 11:59:07 +08:00
|
|
|
static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn,
|
|
|
|
struct kvm_memory_slot *slot,
|
|
|
|
int level)
|
2008-02-23 22:44:30 +08:00
|
|
|
{
|
|
|
|
unsigned long idx;
|
|
|
|
|
2012-02-08 11:59:10 +08:00
|
|
|
idx = gfn_to_index(gfn, slot->base_gfn, level);
|
2012-02-08 12:02:18 +08:00
|
|
|
return &slot->arch.lpage_info[level - 2][idx];
|
2008-02-23 22:44:30 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:07 +08:00
|
|
|
static void update_gfn_disallow_lpage_count(struct kvm_memory_slot *slot,
|
|
|
|
gfn_t gfn, int count)
|
|
|
|
{
|
|
|
|
struct kvm_lpage_info *linfo;
|
|
|
|
int i;
|
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
for (i = PG_LEVEL_2M; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
|
2016-02-24 17:51:07 +08:00
|
|
|
linfo = lpage_info_slot(gfn, slot, i);
|
|
|
|
linfo->disallow_lpage += count;
|
|
|
|
WARN_ON(linfo->disallow_lpage < 0);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void kvm_mmu_gfn_disallow_lpage(struct kvm_memory_slot *slot, gfn_t gfn)
|
|
|
|
{
|
|
|
|
update_gfn_disallow_lpage_count(slot, gfn, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
void kvm_mmu_gfn_allow_lpage(struct kvm_memory_slot *slot, gfn_t gfn)
|
|
|
|
{
|
|
|
|
update_gfn_disallow_lpage_count(slot, gfn, -1);
|
|
|
|
}
|
|
|
|
|
2015-05-19 22:29:22 +08:00
|
|
|
static void account_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
|
2008-02-23 22:44:30 +08:00
|
|
|
{
|
2015-05-18 21:03:39 +08:00
|
|
|
struct kvm_memslots *slots;
|
2009-07-27 22:30:43 +08:00
|
|
|
struct kvm_memory_slot *slot;
|
2015-05-19 22:29:22 +08:00
|
|
|
gfn_t gfn;
|
2008-02-23 22:44:30 +08:00
|
|
|
|
2016-02-24 17:51:14 +08:00
|
|
|
kvm->arch.indirect_shadow_pages++;
|
2015-05-19 22:29:22 +08:00
|
|
|
gfn = sp->gfn;
|
2015-05-18 21:03:39 +08:00
|
|
|
slots = kvm_memslots_for_spte_role(kvm, sp->role);
|
|
|
|
slot = __gfn_to_memslot(slots, gfn);
|
2016-02-24 17:51:14 +08:00
|
|
|
|
|
|
|
/* the non-leaf shadow pages are keeping readonly. */
|
2020-04-28 08:54:22 +08:00
|
|
|
if (sp->role.level > PG_LEVEL_4K)
|
2016-02-24 17:51:14 +08:00
|
|
|
return kvm_slot_page_track_add_page(kvm, slot, gfn,
|
|
|
|
KVM_PAGE_TRACK_WRITE);
|
|
|
|
|
2016-02-24 17:51:07 +08:00
|
|
|
kvm_mmu_gfn_disallow_lpage(slot, gfn);
|
2008-02-23 22:44:30 +08:00
|
|
|
}
|
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
static void account_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp)
|
|
|
|
{
|
|
|
|
if (sp->lpage_disallowed)
|
|
|
|
return;
|
|
|
|
|
|
|
|
++kvm->stat.nx_lpage_splits;
|
2019-11-05 03:26:00 +08:00
|
|
|
list_add_tail(&sp->lpage_disallowed_link,
|
|
|
|
&kvm->arch.lpage_disallowed_mmu_pages);
|
2019-11-04 19:22:02 +08:00
|
|
|
sp->lpage_disallowed = true;
|
|
|
|
}
|
|
|
|
|
2015-05-19 22:29:22 +08:00
|
|
|
static void unaccount_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
|
2008-02-23 22:44:30 +08:00
|
|
|
{
|
2015-05-18 21:03:39 +08:00
|
|
|
struct kvm_memslots *slots;
|
2009-07-27 22:30:43 +08:00
|
|
|
struct kvm_memory_slot *slot;
|
2015-05-19 22:29:22 +08:00
|
|
|
gfn_t gfn;
|
2008-02-23 22:44:30 +08:00
|
|
|
|
2016-02-24 17:51:14 +08:00
|
|
|
kvm->arch.indirect_shadow_pages--;
|
2015-05-19 22:29:22 +08:00
|
|
|
gfn = sp->gfn;
|
2015-05-18 21:03:39 +08:00
|
|
|
slots = kvm_memslots_for_spte_role(kvm, sp->role);
|
|
|
|
slot = __gfn_to_memslot(slots, gfn);
|
2020-04-28 08:54:22 +08:00
|
|
|
if (sp->role.level > PG_LEVEL_4K)
|
2016-02-24 17:51:14 +08:00
|
|
|
return kvm_slot_page_track_remove_page(kvm, slot, gfn,
|
|
|
|
KVM_PAGE_TRACK_WRITE);
|
|
|
|
|
2016-02-24 17:51:07 +08:00
|
|
|
kvm_mmu_gfn_allow_lpage(slot, gfn);
|
2008-02-23 22:44:30 +08:00
|
|
|
}
|
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
static void unaccount_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp)
|
|
|
|
{
|
|
|
|
--kvm->stat.nx_lpage_splits;
|
|
|
|
sp->lpage_disallowed = false;
|
2019-11-05 03:26:00 +08:00
|
|
|
list_del(&sp->lpage_disallowed_link);
|
2019-11-04 19:22:02 +08:00
|
|
|
}
|
|
|
|
|
2011-03-09 15:43:00 +08:00
|
|
|
static struct kvm_memory_slot *
|
|
|
|
gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu, gfn_t gfn,
|
|
|
|
bool no_dirty_log)
|
2008-02-23 22:44:30 +08:00
|
|
|
{
|
|
|
|
struct kvm_memory_slot *slot;
|
2011-03-09 15:43:00 +08:00
|
|
|
|
2015-04-08 21:39:23 +08:00
|
|
|
slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
|
2020-01-21 23:16:32 +08:00
|
|
|
if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
|
|
|
|
return NULL;
|
|
|
|
if (no_dirty_log && slot->dirty_bitmap)
|
|
|
|
return NULL;
|
2011-03-09 15:43:00 +08:00
|
|
|
|
|
|
|
return slot;
|
|
|
|
}
|
|
|
|
|
2007-09-27 20:11:22 +08:00
|
|
|
/*
|
2015-11-20 16:41:28 +08:00
|
|
|
* About rmap_head encoding:
|
2007-01-06 08:36:38 +08:00
|
|
|
*
|
2015-11-20 16:41:28 +08:00
|
|
|
* If the bit zero of rmap_head->val is clear, then it points to the only spte
|
|
|
|
* in this rmap chain. Otherwise, (rmap_head->val & ~1) points to a struct
|
2011-05-15 23:26:20 +08:00
|
|
|
* pte_list_desc containing more mappings.
|
2015-11-20 16:41:28 +08:00
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Returns the number of pointers in the rmap chain, not counting the new one.
|
2007-01-06 08:36:38 +08:00
|
|
|
*/
|
2011-05-15 23:26:20 +08:00
|
|
|
static int pte_list_add(struct kvm_vcpu *vcpu, u64 *spte,
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head)
|
2007-01-06 08:36:38 +08:00
|
|
|
{
|
2011-05-15 23:26:20 +08:00
|
|
|
struct pte_list_desc *desc;
|
2009-08-06 02:43:58 +08:00
|
|
|
int i, count = 0;
|
2007-01-06 08:36:38 +08:00
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
if (!rmap_head->val) {
|
2011-05-15 23:26:20 +08:00
|
|
|
rmap_printk("pte_list_add: %p %llx 0->1\n", spte, *spte);
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head->val = (unsigned long)spte;
|
|
|
|
} else if (!(rmap_head->val & 1)) {
|
2011-05-15 23:26:20 +08:00
|
|
|
rmap_printk("pte_list_add: %p %llx 1->many\n", spte, *spte);
|
|
|
|
desc = mmu_alloc_pte_list_desc(vcpu);
|
2015-11-20 16:41:28 +08:00
|
|
|
desc->sptes[0] = (u64 *)rmap_head->val;
|
2009-06-10 19:24:23 +08:00
|
|
|
desc->sptes[1] = spte;
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head->val = (unsigned long)desc | 1;
|
2010-09-18 08:41:02 +08:00
|
|
|
++count;
|
2007-01-06 08:36:38 +08:00
|
|
|
} else {
|
2011-05-15 23:26:20 +08:00
|
|
|
rmap_printk("pte_list_add: %p %llx many->many\n", spte, *spte);
|
2015-11-20 16:41:28 +08:00
|
|
|
desc = (struct pte_list_desc *)(rmap_head->val & ~1ul);
|
2011-05-15 23:26:20 +08:00
|
|
|
while (desc->sptes[PTE_LIST_EXT-1] && desc->more) {
|
2007-01-06 08:36:38 +08:00
|
|
|
desc = desc->more;
|
2011-05-15 23:26:20 +08:00
|
|
|
count += PTE_LIST_EXT;
|
2009-08-06 02:43:58 +08:00
|
|
|
}
|
2011-05-15 23:26:20 +08:00
|
|
|
if (desc->sptes[PTE_LIST_EXT-1]) {
|
|
|
|
desc->more = mmu_alloc_pte_list_desc(vcpu);
|
2007-01-06 08:36:38 +08:00
|
|
|
desc = desc->more;
|
|
|
|
}
|
2009-06-10 19:24:23 +08:00
|
|
|
for (i = 0; desc->sptes[i]; ++i)
|
2010-09-18 08:41:02 +08:00
|
|
|
++count;
|
2009-06-10 19:24:23 +08:00
|
|
|
desc->sptes[i] = spte;
|
2007-01-06 08:36:38 +08:00
|
|
|
}
|
2009-08-06 02:43:58 +08:00
|
|
|
return count;
|
2007-01-06 08:36:38 +08:00
|
|
|
}
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
static void
|
2015-11-20 16:41:28 +08:00
|
|
|
pte_list_desc_remove_entry(struct kvm_rmap_head *rmap_head,
|
|
|
|
struct pte_list_desc *desc, int i,
|
|
|
|
struct pte_list_desc *prev_desc)
|
2007-01-06 08:36:38 +08:00
|
|
|
{
|
|
|
|
int j;
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
for (j = PTE_LIST_EXT - 1; !desc->sptes[j] && j > i; --j)
|
2007-01-06 08:36:38 +08:00
|
|
|
;
|
2009-06-10 19:24:23 +08:00
|
|
|
desc->sptes[i] = desc->sptes[j];
|
|
|
|
desc->sptes[j] = NULL;
|
2007-01-06 08:36:38 +08:00
|
|
|
if (j != 0)
|
|
|
|
return;
|
|
|
|
if (!prev_desc && !desc->more)
|
2019-12-05 11:40:16 +08:00
|
|
|
rmap_head->val = 0;
|
2007-01-06 08:36:38 +08:00
|
|
|
else
|
|
|
|
if (prev_desc)
|
|
|
|
prev_desc->more = desc->more;
|
|
|
|
else
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head->val = (unsigned long)desc->more | 1;
|
2011-05-15 23:26:20 +08:00
|
|
|
mmu_free_pte_list_desc(desc);
|
2007-01-06 08:36:38 +08:00
|
|
|
}
|
|
|
|
|
2018-10-04 10:04:22 +08:00
|
|
|
static void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head)
|
2007-01-06 08:36:38 +08:00
|
|
|
{
|
2011-05-15 23:26:20 +08:00
|
|
|
struct pte_list_desc *desc;
|
|
|
|
struct pte_list_desc *prev_desc;
|
2007-01-06 08:36:38 +08:00
|
|
|
int i;
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
if (!rmap_head->val) {
|
2018-10-04 10:04:22 +08:00
|
|
|
pr_err("%s: %p 0->BUG\n", __func__, spte);
|
2007-01-06 08:36:38 +08:00
|
|
|
BUG();
|
2015-11-20 16:41:28 +08:00
|
|
|
} else if (!(rmap_head->val & 1)) {
|
2018-10-04 10:04:22 +08:00
|
|
|
rmap_printk("%s: %p 1->0\n", __func__, spte);
|
2015-11-20 16:41:28 +08:00
|
|
|
if ((u64 *)rmap_head->val != spte) {
|
2018-10-04 10:04:22 +08:00
|
|
|
pr_err("%s: %p 1->BUG\n", __func__, spte);
|
2007-01-06 08:36:38 +08:00
|
|
|
BUG();
|
|
|
|
}
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head->val = 0;
|
2007-01-06 08:36:38 +08:00
|
|
|
} else {
|
2018-10-04 10:04:22 +08:00
|
|
|
rmap_printk("%s: %p many->many\n", __func__, spte);
|
2015-11-20 16:41:28 +08:00
|
|
|
desc = (struct pte_list_desc *)(rmap_head->val & ~1ul);
|
2007-01-06 08:36:38 +08:00
|
|
|
prev_desc = NULL;
|
|
|
|
while (desc) {
|
2015-11-20 16:41:28 +08:00
|
|
|
for (i = 0; i < PTE_LIST_EXT && desc->sptes[i]; ++i) {
|
2009-06-10 19:24:23 +08:00
|
|
|
if (desc->sptes[i] == spte) {
|
2015-11-20 16:41:28 +08:00
|
|
|
pte_list_desc_remove_entry(rmap_head,
|
|
|
|
desc, i, prev_desc);
|
2007-01-06 08:36:38 +08:00
|
|
|
return;
|
|
|
|
}
|
2015-11-20 16:41:28 +08:00
|
|
|
}
|
2007-01-06 08:36:38 +08:00
|
|
|
prev_desc = desc;
|
|
|
|
desc = desc->more;
|
|
|
|
}
|
2018-10-04 10:04:22 +08:00
|
|
|
pr_err("%s: %p many->many\n", __func__, spte);
|
2007-01-06 08:36:38 +08:00
|
|
|
BUG();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-10-04 10:04:23 +08:00
|
|
|
static void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep)
|
|
|
|
{
|
|
|
|
mmu_spte_clear_track_bits(sptep);
|
|
|
|
__pte_list_remove(sptep, rmap_head);
|
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level,
|
|
|
|
struct kvm_memory_slot *slot)
|
2011-05-15 23:26:20 +08:00
|
|
|
{
|
2012-07-02 16:57:17 +08:00
|
|
|
unsigned long idx;
|
2011-05-15 23:26:20 +08:00
|
|
|
|
2012-07-02 16:57:17 +08:00
|
|
|
idx = gfn_to_index(gfn, slot->base_gfn, level);
|
2020-04-28 08:54:22 +08:00
|
|
|
return &slot->arch.rmap[level - PG_LEVEL_4K][idx];
|
2011-05-15 23:26:20 +08:00
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static struct kvm_rmap_head *gfn_to_rmap(struct kvm *kvm, gfn_t gfn,
|
|
|
|
struct kvm_mmu_page *sp)
|
2011-11-14 17:22:28 +08:00
|
|
|
{
|
2015-05-18 21:03:39 +08:00
|
|
|
struct kvm_memslots *slots;
|
2011-11-14 17:22:28 +08:00
|
|
|
struct kvm_memory_slot *slot;
|
|
|
|
|
2015-05-18 21:03:39 +08:00
|
|
|
slots = kvm_memslots_for_spte_role(kvm, sp->role);
|
|
|
|
slot = __gfn_to_memslot(slots, gfn);
|
2015-05-18 21:11:46 +08:00
|
|
|
return __gfn_to_rmap(gfn, sp->role.level, slot);
|
2011-11-14 17:22:28 +08:00
|
|
|
}
|
|
|
|
|
2011-09-22 16:53:17 +08:00
|
|
|
static bool rmap_can_add(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
|
|
|
struct kvm_mmu_memory_cache *cache;
|
|
|
|
|
|
|
|
cache = &vcpu->arch.mmu_pte_list_desc_cache;
|
|
|
|
return mmu_memory_cache_free_objects(cache);
|
|
|
|
}
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)
|
|
|
|
{
|
|
|
|
struct kvm_mmu_page *sp;
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head;
|
2011-05-15 23:26:20 +08:00
|
|
|
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(spte);
|
2011-05-15 23:26:20 +08:00
|
|
|
kvm_mmu_page_set_gfn(sp, spte - sp->spt, gfn);
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head = gfn_to_rmap(vcpu->kvm, gfn, sp);
|
|
|
|
return pte_list_add(vcpu, spte, rmap_head);
|
2011-05-15 23:26:20 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void rmap_remove(struct kvm *kvm, u64 *spte)
|
|
|
|
{
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
gfn_t gfn;
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head;
|
2011-05-15 23:26:20 +08:00
|
|
|
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(spte);
|
2011-05-15 23:26:20 +08:00
|
|
|
gfn = kvm_mmu_page_get_gfn(sp, spte - sp->spt);
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head = gfn_to_rmap(kvm, gfn, sp);
|
2018-10-04 10:04:22 +08:00
|
|
|
__pte_list_remove(spte, rmap_head);
|
2011-05-15 23:26:20 +08:00
|
|
|
}
|
|
|
|
|
2012-03-21 22:50:34 +08:00
|
|
|
/*
|
|
|
|
* Used by the following functions to iterate through the sptes linked by a
|
|
|
|
* rmap. All fields are private and not assumed to be used outside.
|
|
|
|
*/
|
|
|
|
struct rmap_iterator {
|
|
|
|
/* private fields */
|
|
|
|
struct pte_list_desc *desc; /* holds the sptep if not NULL */
|
|
|
|
int pos; /* index of the sptep */
|
|
|
|
};
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Iteration must be started by this function. This should also be used after
|
|
|
|
* removing/dropping sptes from the rmap link because in such cases the
|
2019-12-06 16:20:18 +08:00
|
|
|
* information in the iterator may not be valid.
|
2012-03-21 22:50:34 +08:00
|
|
|
*
|
|
|
|
* Returns sptep if found, NULL otherwise.
|
|
|
|
*/
|
2015-11-20 16:41:28 +08:00
|
|
|
static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head,
|
|
|
|
struct rmap_iterator *iter)
|
2012-03-21 22:50:34 +08:00
|
|
|
{
|
2015-11-20 16:45:44 +08:00
|
|
|
u64 *sptep;
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
if (!rmap_head->val)
|
2012-03-21 22:50:34 +08:00
|
|
|
return NULL;
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
if (!(rmap_head->val & 1)) {
|
2012-03-21 22:50:34 +08:00
|
|
|
iter->desc = NULL;
|
2015-11-20 16:45:44 +08:00
|
|
|
sptep = (u64 *)rmap_head->val;
|
|
|
|
goto out;
|
2012-03-21 22:50:34 +08:00
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
iter->desc = (struct pte_list_desc *)(rmap_head->val & ~1ul);
|
2012-03-21 22:50:34 +08:00
|
|
|
iter->pos = 0;
|
2015-11-20 16:45:44 +08:00
|
|
|
sptep = iter->desc->sptes[iter->pos];
|
|
|
|
out:
|
|
|
|
BUG_ON(!is_shadow_present_pte(*sptep));
|
|
|
|
return sptep;
|
2012-03-21 22:50:34 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Must be used with a valid iterator: e.g. after rmap_get_first().
|
|
|
|
*
|
|
|
|
* Returns sptep if found, NULL otherwise.
|
|
|
|
*/
|
|
|
|
static u64 *rmap_get_next(struct rmap_iterator *iter)
|
|
|
|
{
|
2015-11-20 16:45:44 +08:00
|
|
|
u64 *sptep;
|
|
|
|
|
2012-03-21 22:50:34 +08:00
|
|
|
if (iter->desc) {
|
|
|
|
if (iter->pos < PTE_LIST_EXT - 1) {
|
|
|
|
++iter->pos;
|
|
|
|
sptep = iter->desc->sptes[iter->pos];
|
|
|
|
if (sptep)
|
2015-11-20 16:45:44 +08:00
|
|
|
goto out;
|
2012-03-21 22:50:34 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
iter->desc = iter->desc->more;
|
|
|
|
|
|
|
|
if (iter->desc) {
|
|
|
|
iter->pos = 0;
|
|
|
|
/* desc->sptes[0] cannot be NULL */
|
2015-11-20 16:45:44 +08:00
|
|
|
sptep = iter->desc->sptes[iter->pos];
|
|
|
|
goto out;
|
2012-03-21 22:50:34 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return NULL;
|
2015-11-20 16:45:44 +08:00
|
|
|
out:
|
|
|
|
BUG_ON(!is_shadow_present_pte(*sptep));
|
|
|
|
return sptep;
|
2012-03-21 22:50:34 +08:00
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
#define for_each_rmap_spte(_rmap_head_, _iter_, _spte_) \
|
|
|
|
for (_spte_ = rmap_get_first(_rmap_head_, _iter_); \
|
2015-11-20 16:45:44 +08:00
|
|
|
_spte_; _spte_ = rmap_get_next(_iter_))
|
2015-05-13 14:42:20 +08:00
|
|
|
|
2011-07-12 03:28:04 +08:00
|
|
|
static void drop_spte(struct kvm *kvm, u64 *sptep)
|
2010-07-16 11:28:09 +08:00
|
|
|
{
|
2011-07-12 03:30:35 +08:00
|
|
|
if (mmu_spte_clear_track_bits(sptep))
|
2010-10-25 21:58:22 +08:00
|
|
|
rmap_remove(kvm, sptep);
|
2010-06-06 19:31:27 +08:00
|
|
|
}
|
|
|
|
|
2012-06-20 15:57:39 +08:00
|
|
|
|
|
|
|
static bool __drop_large_spte(struct kvm *kvm, u64 *sptep)
|
|
|
|
{
|
|
|
|
if (is_large_pte(*sptep)) {
|
2020-06-23 04:20:33 +08:00
|
|
|
WARN_ON(sptep_to_sp(sptep)->role.level == PG_LEVEL_4K);
|
2012-06-20 15:57:39 +08:00
|
|
|
drop_spte(kvm, sptep);
|
|
|
|
--kvm->stat.lpages;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void drop_large_spte(struct kvm_vcpu *vcpu, u64 *sptep)
|
|
|
|
{
|
2018-12-06 21:21:09 +08:00
|
|
|
if (__drop_large_spte(vcpu->kvm, sptep)) {
|
2020-06-23 04:20:33 +08:00
|
|
|
struct kvm_mmu_page *sp = sptep_to_sp(sptep);
|
2018-12-06 21:21:09 +08:00
|
|
|
|
|
|
|
kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
|
|
|
|
KVM_PAGES_PER_HPAGE(sp->role.level));
|
|
|
|
}
|
2012-06-20 15:57:39 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2012-06-20 15:58:58 +08:00
|
|
|
* Write-protect on the specified @sptep, @pt_protect indicates whether
|
2014-04-17 17:06:14 +08:00
|
|
|
* spte write-protection is caused by protecting shadow page table.
|
2012-06-20 15:58:58 +08:00
|
|
|
*
|
2014-09-22 10:31:38 +08:00
|
|
|
* Note: write protection is difference between dirty logging and spte
|
2012-06-20 15:58:58 +08:00
|
|
|
* protection:
|
|
|
|
* - for dirty logging, the spte can be set to writable at anytime if
|
|
|
|
* its dirty bitmap is properly set.
|
|
|
|
* - for spte protection, the spte can be writable only after unsync-ing
|
|
|
|
* shadow page.
|
2012-06-20 15:57:39 +08:00
|
|
|
*
|
2014-04-17 17:06:14 +08:00
|
|
|
* Return true if tlb need be flushed.
|
2012-06-20 15:57:39 +08:00
|
|
|
*/
|
2016-08-03 04:32:37 +08:00
|
|
|
static bool spte_write_protect(u64 *sptep, bool pt_protect)
|
2012-06-20 15:57:15 +08:00
|
|
|
{
|
|
|
|
u64 spte = *sptep;
|
|
|
|
|
2012-06-20 15:58:58 +08:00
|
|
|
if (!is_writable_pte(spte) &&
|
2016-12-07 08:46:11 +08:00
|
|
|
!(pt_protect && spte_can_locklessly_be_made_writable(spte)))
|
2012-06-20 15:57:15 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
rmap_printk("rmap_write_protect: spte %p %llx\n", sptep, *sptep);
|
|
|
|
|
2012-06-20 15:58:58 +08:00
|
|
|
if (pt_protect)
|
|
|
|
spte &= ~SPTE_MMU_WRITEABLE;
|
2012-06-20 15:57:15 +08:00
|
|
|
spte = spte & ~PT_WRITABLE_MASK;
|
2012-06-20 15:58:58 +08:00
|
|
|
|
2014-04-17 17:06:14 +08:00
|
|
|
return mmu_spte_update(sptep, spte);
|
2012-06-20 15:57:15 +08:00
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static bool __rmap_write_protect(struct kvm *kvm,
|
|
|
|
struct kvm_rmap_head *rmap_head,
|
2013-01-08 18:44:09 +08:00
|
|
|
bool pt_protect)
|
2007-10-16 20:42:30 +08:00
|
|
|
{
|
2012-03-21 22:50:34 +08:00
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
2012-06-20 15:57:15 +08:00
|
|
|
bool flush = false;
|
2007-01-06 08:36:43 +08:00
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
for_each_rmap_spte(rmap_head, &iter, sptep)
|
2016-08-03 04:32:37 +08:00
|
|
|
flush |= spte_write_protect(sptep, pt_protect);
|
2008-03-21 00:17:24 +08:00
|
|
|
|
2012-06-20 15:57:15 +08:00
|
|
|
return flush;
|
2012-03-01 18:31:22 +08:00
|
|
|
}
|
|
|
|
|
2016-08-03 04:32:37 +08:00
|
|
|
static bool spte_clear_dirty(u64 *sptep)
|
2015-01-28 10:54:24 +08:00
|
|
|
{
|
|
|
|
u64 spte = *sptep;
|
|
|
|
|
|
|
|
rmap_printk("rmap_clear_dirty: spte %p %llx\n", sptep, *sptep);
|
|
|
|
|
2019-09-27 00:47:59 +08:00
|
|
|
MMU_WARN_ON(!spte_ad_enabled(spte));
|
2015-01-28 10:54:24 +08:00
|
|
|
spte &= ~shadow_dirty_mask;
|
|
|
|
return mmu_spte_update(sptep, spte);
|
|
|
|
}
|
|
|
|
|
2019-09-27 00:47:59 +08:00
|
|
|
static bool spte_wrprot_for_clear_dirty(u64 *sptep)
|
2017-07-01 08:26:31 +08:00
|
|
|
{
|
|
|
|
bool was_writable = test_and_clear_bit(PT_WRITABLE_SHIFT,
|
|
|
|
(unsigned long *)sptep);
|
2019-09-27 00:47:59 +08:00
|
|
|
if (was_writable && !spte_ad_enabled(*sptep))
|
2017-07-01 08:26:31 +08:00
|
|
|
kvm_set_pfn_dirty(spte_to_pfn(*sptep));
|
|
|
|
|
|
|
|
return was_writable;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Gets the GFN ready for another round of dirty logging by clearing the
|
|
|
|
* - D bit on ad-enabled SPTEs, and
|
|
|
|
* - W bit on ad-disabled SPTEs.
|
|
|
|
* Returns true iff any D or W bits were cleared.
|
|
|
|
*/
|
2015-11-20 16:41:28 +08:00
|
|
|
static bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head)
|
2015-01-28 10:54:24 +08:00
|
|
|
{
|
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
|
|
|
bool flush = false;
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
for_each_rmap_spte(rmap_head, &iter, sptep)
|
2019-09-27 00:47:59 +08:00
|
|
|
if (spte_ad_need_write_protect(*sptep))
|
|
|
|
flush |= spte_wrprot_for_clear_dirty(sptep);
|
2017-07-01 08:26:31 +08:00
|
|
|
else
|
2019-09-27 00:47:59 +08:00
|
|
|
flush |= spte_clear_dirty(sptep);
|
2015-01-28 10:54:24 +08:00
|
|
|
|
|
|
|
return flush;
|
|
|
|
}
|
|
|
|
|
2016-08-03 04:32:37 +08:00
|
|
|
static bool spte_set_dirty(u64 *sptep)
|
2015-01-28 10:54:24 +08:00
|
|
|
{
|
|
|
|
u64 spte = *sptep;
|
|
|
|
|
|
|
|
rmap_printk("rmap_set_dirty: spte %p %llx\n", sptep, *sptep);
|
|
|
|
|
2019-09-27 00:47:59 +08:00
|
|
|
/*
|
2020-03-22 04:26:00 +08:00
|
|
|
* Similar to the !kvm_x86_ops.slot_disable_log_dirty case,
|
2019-09-27 00:47:59 +08:00
|
|
|
* do not bother adding back write access to pages marked
|
|
|
|
* SPTE_AD_WRPROT_ONLY_MASK.
|
|
|
|
*/
|
2015-01-28 10:54:24 +08:00
|
|
|
spte |= shadow_dirty_mask;
|
|
|
|
|
|
|
|
return mmu_spte_update(sptep, spte);
|
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static bool __rmap_set_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head)
|
2015-01-28 10:54:24 +08:00
|
|
|
{
|
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
|
|
|
bool flush = false;
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
for_each_rmap_spte(rmap_head, &iter, sptep)
|
2017-07-01 08:26:31 +08:00
|
|
|
if (spte_ad_enabled(*sptep))
|
|
|
|
flush |= spte_set_dirty(sptep);
|
2015-01-28 10:54:24 +08:00
|
|
|
|
|
|
|
return flush;
|
|
|
|
}
|
|
|
|
|
2012-03-01 18:32:16 +08:00
|
|
|
/**
|
2015-01-28 10:54:23 +08:00
|
|
|
* kvm_mmu_write_protect_pt_masked - write protect selected PT level pages
|
2012-03-01 18:32:16 +08:00
|
|
|
* @kvm: kvm instance
|
|
|
|
* @slot: slot to protect
|
|
|
|
* @gfn_offset: start of the BITS_PER_LONG pages we care about
|
|
|
|
* @mask: indicates which pages we should protect
|
|
|
|
*
|
|
|
|
* Used when we do not need to care about huge page mappings: e.g. during dirty
|
|
|
|
* logging we do not have any such mappings.
|
|
|
|
*/
|
2015-01-28 10:54:23 +08:00
|
|
|
static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
|
2012-03-01 18:32:16 +08:00
|
|
|
struct kvm_memory_slot *slot,
|
|
|
|
gfn_t gfn_offset, unsigned long mask)
|
2012-03-01 18:31:22 +08:00
|
|
|
{
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head;
|
2012-03-01 18:31:22 +08:00
|
|
|
|
2012-03-01 18:32:16 +08:00
|
|
|
while (mask) {
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head = __gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask),
|
2020-04-28 08:54:22 +08:00
|
|
|
PG_LEVEL_4K, slot);
|
2015-11-20 16:41:28 +08:00
|
|
|
__rmap_write_protect(kvm, rmap_head, false);
|
2008-02-23 22:44:30 +08:00
|
|
|
|
2012-03-01 18:32:16 +08:00
|
|
|
/* clear the first set bit */
|
|
|
|
mask &= mask - 1;
|
|
|
|
}
|
2007-01-06 08:36:43 +08:00
|
|
|
}
|
|
|
|
|
2015-01-28 10:54:24 +08:00
|
|
|
/**
|
2017-07-01 08:26:31 +08:00
|
|
|
* kvm_mmu_clear_dirty_pt_masked - clear MMU D-bit for PT level pages, or write
|
|
|
|
* protect the page if the D-bit isn't supported.
|
2015-01-28 10:54:24 +08:00
|
|
|
* @kvm: kvm instance
|
|
|
|
* @slot: slot to clear D-bit
|
|
|
|
* @gfn_offset: start of the BITS_PER_LONG pages we care about
|
|
|
|
* @mask: indicates which pages we should clear D-bit
|
|
|
|
*
|
|
|
|
* Used for PML to re-log the dirty GPAs after userspace querying dirty_bitmap.
|
|
|
|
*/
|
|
|
|
void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm,
|
|
|
|
struct kvm_memory_slot *slot,
|
|
|
|
gfn_t gfn_offset, unsigned long mask)
|
|
|
|
{
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head;
|
2015-01-28 10:54:24 +08:00
|
|
|
|
|
|
|
while (mask) {
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head = __gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask),
|
2020-04-28 08:54:22 +08:00
|
|
|
PG_LEVEL_4K, slot);
|
2015-11-20 16:41:28 +08:00
|
|
|
__rmap_clear_dirty(kvm, rmap_head);
|
2015-01-28 10:54:24 +08:00
|
|
|
|
|
|
|
/* clear the first set bit */
|
|
|
|
mask &= mask - 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_clear_dirty_pt_masked);
|
|
|
|
|
2015-01-28 10:54:23 +08:00
|
|
|
/**
|
|
|
|
* kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected
|
|
|
|
* PT level pages.
|
|
|
|
*
|
|
|
|
* It calls kvm_mmu_write_protect_pt_masked to write protect selected pages to
|
|
|
|
* enable dirty logging for them.
|
|
|
|
*
|
|
|
|
* Used when we do not need to care about huge page mappings: e.g. during dirty
|
|
|
|
* logging we do not have any such mappings.
|
|
|
|
*/
|
|
|
|
void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
|
|
|
|
struct kvm_memory_slot *slot,
|
|
|
|
gfn_t gfn_offset, unsigned long mask)
|
|
|
|
{
|
2020-03-22 04:26:00 +08:00
|
|
|
if (kvm_x86_ops.enable_log_dirty_pt_masked)
|
|
|
|
kvm_x86_ops.enable_log_dirty_pt_masked(kvm, slot, gfn_offset,
|
2015-01-28 10:54:27 +08:00
|
|
|
mask);
|
|
|
|
else
|
|
|
|
kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask);
|
2015-01-28 10:54:23 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:08 +08:00
|
|
|
bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
|
|
|
|
struct kvm_memory_slot *slot, u64 gfn)
|
2011-11-14 17:24:50 +08:00
|
|
|
{
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head;
|
2012-03-01 18:32:16 +08:00
|
|
|
int i;
|
2012-06-20 15:56:53 +08:00
|
|
|
bool write_protected = false;
|
2011-11-14 17:24:50 +08:00
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
for (i = PG_LEVEL_4K; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head = __gfn_to_rmap(gfn, i, slot);
|
2016-02-24 17:51:08 +08:00
|
|
|
write_protected |= __rmap_write_protect(kvm, rmap_head, true);
|
2012-03-01 18:32:16 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return write_protected;
|
2011-11-14 17:24:50 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:08 +08:00
|
|
|
static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn)
|
|
|
|
{
|
|
|
|
struct kvm_memory_slot *slot;
|
|
|
|
|
|
|
|
slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
|
|
|
|
return kvm_mmu_slot_gfn_write_protect(vcpu->kvm, slot, gfn);
|
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static bool kvm_zap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head)
|
2008-07-25 22:24:52 +08:00
|
|
|
{
|
2012-03-21 22:50:34 +08:00
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
2015-05-13 14:42:25 +08:00
|
|
|
bool flush = false;
|
2008-07-25 22:24:52 +08:00
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
while ((sptep = rmap_get_first(rmap_head, &iter))) {
|
2015-05-13 14:42:25 +08:00
|
|
|
rmap_printk("%s: spte %p %llx.\n", __func__, sptep, *sptep);
|
2012-03-21 22:50:34 +08:00
|
|
|
|
2018-10-04 10:04:23 +08:00
|
|
|
pte_list_remove(rmap_head, sptep);
|
2015-05-13 14:42:25 +08:00
|
|
|
flush = true;
|
2008-07-25 22:24:52 +08:00
|
|
|
}
|
2012-03-21 22:50:34 +08:00
|
|
|
|
2015-05-13 14:42:25 +08:00
|
|
|
return flush;
|
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static int kvm_unmap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
|
2015-05-13 14:42:25 +08:00
|
|
|
struct kvm_memory_slot *slot, gfn_t gfn, int level,
|
|
|
|
unsigned long data)
|
|
|
|
{
|
2015-11-20 16:41:28 +08:00
|
|
|
return kvm_zap_rmapp(kvm, rmap_head);
|
2008-07-25 22:24:52 +08:00
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static int kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
|
2014-09-24 03:34:54 +08:00
|
|
|
struct kvm_memory_slot *slot, gfn_t gfn, int level,
|
|
|
|
unsigned long data)
|
2009-09-24 02:47:18 +08:00
|
|
|
{
|
2012-03-21 22:50:34 +08:00
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
2009-09-24 02:47:18 +08:00
|
|
|
int need_flush = 0;
|
2012-03-21 22:50:34 +08:00
|
|
|
u64 new_spte;
|
2009-09-24 02:47:18 +08:00
|
|
|
pte_t *ptep = (pte_t *)data;
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
kvm_pfn_t new_pfn;
|
2009-09-24 02:47:18 +08:00
|
|
|
|
|
|
|
WARN_ON(pte_huge(*ptep));
|
|
|
|
new_pfn = pte_pfn(*ptep);
|
2012-03-21 22:50:34 +08:00
|
|
|
|
2015-05-13 14:42:20 +08:00
|
|
|
restart:
|
2015-11-20 16:41:28 +08:00
|
|
|
for_each_rmap_spte(rmap_head, &iter, sptep) {
|
2014-09-24 03:34:54 +08:00
|
|
|
rmap_printk("kvm_set_pte_rmapp: spte %p %llx gfn %llx (%d)\n",
|
2016-12-07 08:46:16 +08:00
|
|
|
sptep, *sptep, gfn, level);
|
2012-03-21 22:50:34 +08:00
|
|
|
|
2009-09-24 02:47:18 +08:00
|
|
|
need_flush = 1;
|
2012-03-21 22:50:34 +08:00
|
|
|
|
2009-09-24 02:47:18 +08:00
|
|
|
if (pte_write(*ptep)) {
|
2018-10-04 10:04:23 +08:00
|
|
|
pte_list_remove(rmap_head, sptep);
|
2015-05-13 14:42:20 +08:00
|
|
|
goto restart;
|
2009-09-24 02:47:18 +08:00
|
|
|
} else {
|
2012-03-21 22:50:34 +08:00
|
|
|
new_spte = *sptep & ~PT64_BASE_ADDR_MASK;
|
2009-09-24 02:47:18 +08:00
|
|
|
new_spte |= (u64)new_pfn << PAGE_SHIFT;
|
|
|
|
|
|
|
|
new_spte &= ~PT_WRITABLE_MASK;
|
|
|
|
new_spte &= ~SPTE_HOST_WRITEABLE;
|
2016-12-07 08:46:16 +08:00
|
|
|
|
|
|
|
new_spte = mark_spte_for_access_track(new_spte);
|
2012-03-21 22:50:34 +08:00
|
|
|
|
|
|
|
mmu_spte_clear_track_bits(sptep);
|
|
|
|
mmu_spte_set(sptep, new_spte);
|
2009-09-24 02:47:18 +08:00
|
|
|
}
|
|
|
|
}
|
2012-03-21 22:50:34 +08:00
|
|
|
|
2018-12-06 21:21:12 +08:00
|
|
|
if (need_flush && kvm_available_flush_tlb_with_range()) {
|
|
|
|
kvm_flush_remote_tlbs_with_address(kvm, gfn, 1);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-12-06 21:21:11 +08:00
|
|
|
return need_flush;
|
2009-09-24 02:47:18 +08:00
|
|
|
}
|
|
|
|
|
2015-05-13 14:42:22 +08:00
|
|
|
struct slot_rmap_walk_iterator {
|
|
|
|
/* input fields. */
|
|
|
|
struct kvm_memory_slot *slot;
|
|
|
|
gfn_t start_gfn;
|
|
|
|
gfn_t end_gfn;
|
|
|
|
int start_level;
|
|
|
|
int end_level;
|
|
|
|
|
|
|
|
/* output fields. */
|
|
|
|
gfn_t gfn;
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap;
|
2015-05-13 14:42:22 +08:00
|
|
|
int level;
|
|
|
|
|
|
|
|
/* private field. */
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *end_rmap;
|
2015-05-13 14:42:22 +08:00
|
|
|
};
|
|
|
|
|
|
|
|
static void
|
|
|
|
rmap_walk_init_level(struct slot_rmap_walk_iterator *iterator, int level)
|
|
|
|
{
|
|
|
|
iterator->level = level;
|
|
|
|
iterator->gfn = iterator->start_gfn;
|
|
|
|
iterator->rmap = __gfn_to_rmap(iterator->gfn, level, iterator->slot);
|
|
|
|
iterator->end_rmap = __gfn_to_rmap(iterator->end_gfn, level,
|
|
|
|
iterator->slot);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
slot_rmap_walk_init(struct slot_rmap_walk_iterator *iterator,
|
|
|
|
struct kvm_memory_slot *slot, int start_level,
|
|
|
|
int end_level, gfn_t start_gfn, gfn_t end_gfn)
|
|
|
|
{
|
|
|
|
iterator->slot = slot;
|
|
|
|
iterator->start_level = start_level;
|
|
|
|
iterator->end_level = end_level;
|
|
|
|
iterator->start_gfn = start_gfn;
|
|
|
|
iterator->end_gfn = end_gfn;
|
|
|
|
|
|
|
|
rmap_walk_init_level(iterator, iterator->start_level);
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool slot_rmap_walk_okay(struct slot_rmap_walk_iterator *iterator)
|
|
|
|
{
|
|
|
|
return !!iterator->rmap;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void slot_rmap_walk_next(struct slot_rmap_walk_iterator *iterator)
|
|
|
|
{
|
|
|
|
if (++iterator->rmap <= iterator->end_rmap) {
|
|
|
|
iterator->gfn += (1UL << KVM_HPAGE_GFN_SHIFT(iterator->level));
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (++iterator->level > iterator->end_level) {
|
|
|
|
iterator->rmap = NULL;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
rmap_walk_init_level(iterator, iterator->level);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define for_each_slot_rmap_range(_slot_, _start_level_, _end_level_, \
|
|
|
|
_start_gfn, _end_gfn, _iter_) \
|
|
|
|
for (slot_rmap_walk_init(_iter_, _slot_, _start_level_, \
|
|
|
|
_end_level_, _start_gfn, _end_gfn); \
|
|
|
|
slot_rmap_walk_okay(_iter_); \
|
|
|
|
slot_rmap_walk_next(_iter_))
|
|
|
|
|
KVM: MMU: Make kvm_handle_hva() handle range of addresses
When guest's memory is backed by THP pages, MMU notifier needs to call
kvm_unmap_hva(), which in turn leads to kvm_handle_hva(), in a loop to
invalidate a range of pages which constitute one huge page:
for each page
for each memslot
if page is in memslot
unmap using rmap
This means although every page in that range is expected to be found in
the same memslot, we are forced to check unrelated memslots many times.
If the guest has more memslots, the situation will become worse.
Furthermore, if the range does not include any pages in the guest's
memory, the loop over the pages will just consume extra time.
This patch, together with the following patches, solves this problem by
introducing kvm_handle_hva_range() which makes the loop look like this:
for each memslot
for each page in memslot
unmap using rmap
In this new processing, the actual work is converted to a loop over rmap
which is much more cache friendly than before.
Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
Cc: Alexander Graf <agraf@suse.de>
Cc: Paul Mackerras <paulus@samba.org>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
2012-07-02 16:55:48 +08:00
|
|
|
static int kvm_handle_hva_range(struct kvm *kvm,
|
|
|
|
unsigned long start,
|
|
|
|
unsigned long end,
|
|
|
|
unsigned long data,
|
|
|
|
int (*handler)(struct kvm *kvm,
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head,
|
2012-07-02 16:57:59 +08:00
|
|
|
struct kvm_memory_slot *slot,
|
2014-09-24 03:34:54 +08:00
|
|
|
gfn_t gfn,
|
|
|
|
int level,
|
KVM: MMU: Make kvm_handle_hva() handle range of addresses
When guest's memory is backed by THP pages, MMU notifier needs to call
kvm_unmap_hva(), which in turn leads to kvm_handle_hva(), in a loop to
invalidate a range of pages which constitute one huge page:
for each page
for each memslot
if page is in memslot
unmap using rmap
This means although every page in that range is expected to be found in
the same memslot, we are forced to check unrelated memslots many times.
If the guest has more memslots, the situation will become worse.
Furthermore, if the range does not include any pages in the guest's
memory, the loop over the pages will just consume extra time.
This patch, together with the following patches, solves this problem by
introducing kvm_handle_hva_range() which makes the loop look like this:
for each memslot
for each page in memslot
unmap using rmap
In this new processing, the actual work is converted to a loop over rmap
which is much more cache friendly than before.
Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
Cc: Alexander Graf <agraf@suse.de>
Cc: Paul Mackerras <paulus@samba.org>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
2012-07-02 16:55:48 +08:00
|
|
|
unsigned long data))
|
2008-07-25 22:24:52 +08:00
|
|
|
{
|
2009-12-24 00:35:21 +08:00
|
|
|
struct kvm_memslots *slots;
|
2011-11-24 17:39:18 +08:00
|
|
|
struct kvm_memory_slot *memslot;
|
2015-05-13 14:42:22 +08:00
|
|
|
struct slot_rmap_walk_iterator iterator;
|
|
|
|
int ret = 0;
|
2015-05-18 19:33:16 +08:00
|
|
|
int i;
|
2009-12-24 00:35:21 +08:00
|
|
|
|
2015-05-18 19:33:16 +08:00
|
|
|
for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
|
|
|
|
slots = __kvm_memslots(kvm, i);
|
|
|
|
kvm_for_each_memslot(memslot, slots) {
|
|
|
|
unsigned long hva_start, hva_end;
|
|
|
|
gfn_t gfn_start, gfn_end;
|
2008-07-25 22:24:52 +08:00
|
|
|
|
2015-05-18 19:33:16 +08:00
|
|
|
hva_start = max(start, memslot->userspace_addr);
|
|
|
|
hva_end = min(end, memslot->userspace_addr +
|
|
|
|
(memslot->npages << PAGE_SHIFT));
|
|
|
|
if (hva_start >= hva_end)
|
|
|
|
continue;
|
|
|
|
/*
|
|
|
|
* {gfn(page) | page intersects with [hva_start, hva_end)} =
|
|
|
|
* {gfn_start, gfn_start+1, ..., gfn_end-1}.
|
|
|
|
*/
|
|
|
|
gfn_start = hva_to_gfn_memslot(hva_start, memslot);
|
|
|
|
gfn_end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, memslot);
|
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
for_each_slot_rmap_range(memslot, PG_LEVEL_4K,
|
2020-04-28 08:54:21 +08:00
|
|
|
KVM_MAX_HUGEPAGE_LEVEL,
|
2015-05-18 19:33:16 +08:00
|
|
|
gfn_start, gfn_end - 1,
|
|
|
|
&iterator)
|
|
|
|
ret |= handler(kvm, iterator.rmap, memslot,
|
|
|
|
iterator.gfn, iterator.level, data);
|
|
|
|
}
|
2008-07-25 22:24:52 +08:00
|
|
|
}
|
|
|
|
|
2012-07-02 16:58:48 +08:00
|
|
|
return ret;
|
2008-07-25 22:24:52 +08:00
|
|
|
}
|
|
|
|
|
KVM: MMU: Make kvm_handle_hva() handle range of addresses
When guest's memory is backed by THP pages, MMU notifier needs to call
kvm_unmap_hva(), which in turn leads to kvm_handle_hva(), in a loop to
invalidate a range of pages which constitute one huge page:
for each page
for each memslot
if page is in memslot
unmap using rmap
This means although every page in that range is expected to be found in
the same memslot, we are forced to check unrelated memslots many times.
If the guest has more memslots, the situation will become worse.
Furthermore, if the range does not include any pages in the guest's
memory, the loop over the pages will just consume extra time.
This patch, together with the following patches, solves this problem by
introducing kvm_handle_hva_range() which makes the loop look like this:
for each memslot
for each page in memslot
unmap using rmap
In this new processing, the actual work is converted to a loop over rmap
which is much more cache friendly than before.
Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
Cc: Alexander Graf <agraf@suse.de>
Cc: Paul Mackerras <paulus@samba.org>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
2012-07-02 16:55:48 +08:00
|
|
|
static int kvm_handle_hva(struct kvm *kvm, unsigned long hva,
|
|
|
|
unsigned long data,
|
2015-11-20 16:41:28 +08:00
|
|
|
int (*handler)(struct kvm *kvm,
|
|
|
|
struct kvm_rmap_head *rmap_head,
|
2012-07-02 16:57:59 +08:00
|
|
|
struct kvm_memory_slot *slot,
|
2014-09-24 03:34:54 +08:00
|
|
|
gfn_t gfn, int level,
|
KVM: MMU: Make kvm_handle_hva() handle range of addresses
When guest's memory is backed by THP pages, MMU notifier needs to call
kvm_unmap_hva(), which in turn leads to kvm_handle_hva(), in a loop to
invalidate a range of pages which constitute one huge page:
for each page
for each memslot
if page is in memslot
unmap using rmap
This means although every page in that range is expected to be found in
the same memslot, we are forced to check unrelated memslots many times.
If the guest has more memslots, the situation will become worse.
Furthermore, if the range does not include any pages in the guest's
memory, the loop over the pages will just consume extra time.
This patch, together with the following patches, solves this problem by
introducing kvm_handle_hva_range() which makes the loop look like this:
for each memslot
for each page in memslot
unmap using rmap
In this new processing, the actual work is converted to a loop over rmap
which is much more cache friendly than before.
Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
Cc: Alexander Graf <agraf@suse.de>
Cc: Paul Mackerras <paulus@samba.org>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
2012-07-02 16:55:48 +08:00
|
|
|
unsigned long data))
|
|
|
|
{
|
|
|
|
return kvm_handle_hva_range(kvm, hva, hva + 1, data, handler);
|
2008-07-25 22:24:52 +08:00
|
|
|
}
|
|
|
|
|
2012-07-02 16:56:33 +08:00
|
|
|
int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end)
|
|
|
|
{
|
|
|
|
return kvm_handle_hva_range(kvm, start, end, 0, kvm_unmap_rmapp);
|
|
|
|
}
|
|
|
|
|
2018-12-06 21:21:10 +08:00
|
|
|
int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte)
|
2009-09-24 02:47:18 +08:00
|
|
|
{
|
2018-12-06 21:21:11 +08:00
|
|
|
return kvm_handle_hva(kvm, hva, (unsigned long)&pte, kvm_set_pte_rmapp);
|
2008-07-25 22:24:52 +08:00
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static int kvm_age_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
|
2014-09-24 03:34:54 +08:00
|
|
|
struct kvm_memory_slot *slot, gfn_t gfn, int level,
|
|
|
|
unsigned long data)
|
2008-07-25 22:24:52 +08:00
|
|
|
{
|
2012-03-21 22:50:34 +08:00
|
|
|
u64 *sptep;
|
2012-06-03 16:34:08 +08:00
|
|
|
struct rmap_iterator uninitialized_var(iter);
|
2008-07-25 22:24:52 +08:00
|
|
|
int young = 0;
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
for_each_rmap_spte(rmap_head, &iter, sptep)
|
|
|
|
young |= mmu_spte_age(sptep);
|
2015-05-13 14:42:20 +08:00
|
|
|
|
2014-09-24 03:34:54 +08:00
|
|
|
trace_kvm_age_page(gfn, level, slot, young);
|
2008-07-25 22:24:52 +08:00
|
|
|
return young;
|
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static int kvm_test_age_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
|
2014-09-24 03:34:54 +08:00
|
|
|
struct kvm_memory_slot *slot, gfn_t gfn,
|
|
|
|
int level, unsigned long data)
|
2011-01-14 07:47:10 +08:00
|
|
|
{
|
2012-03-21 22:50:34 +08:00
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
2011-01-14 07:47:10 +08:00
|
|
|
|
2016-12-07 08:46:13 +08:00
|
|
|
for_each_rmap_spte(rmap_head, &iter, sptep)
|
|
|
|
if (is_accessed_spte(*sptep))
|
|
|
|
return 1;
|
|
|
|
return 0;
|
2011-01-14 07:47:10 +08:00
|
|
|
}
|
|
|
|
|
2009-08-06 02:43:58 +08:00
|
|
|
#define RMAP_RECYCLE_THRESHOLD 1000
|
|
|
|
|
2009-07-27 22:30:44 +08:00
|
|
|
static void rmap_recycle(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)
|
2009-08-06 02:43:58 +08:00
|
|
|
{
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head;
|
2009-07-27 22:30:44 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(spte);
|
2009-08-06 02:43:58 +08:00
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
rmap_head = gfn_to_rmap(vcpu->kvm, gfn, sp);
|
2009-08-06 02:43:58 +08:00
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
kvm_unmap_rmapp(vcpu->kvm, rmap_head, NULL, gfn, sp->role.level, 0);
|
2018-12-06 21:21:09 +08:00
|
|
|
kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
|
|
|
|
KVM_PAGES_PER_HPAGE(sp->role.level));
|
2009-08-06 02:43:58 +08:00
|
|
|
}
|
|
|
|
|
2014-09-23 05:54:42 +08:00
|
|
|
int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end)
|
2008-07-25 22:24:52 +08:00
|
|
|
{
|
2014-09-23 05:54:42 +08:00
|
|
|
return kvm_handle_hva_range(kvm, start, end, 0, kvm_age_rmapp);
|
2008-07-25 22:24:52 +08:00
|
|
|
}
|
|
|
|
|
2011-01-14 07:47:10 +08:00
|
|
|
int kvm_test_age_hva(struct kvm *kvm, unsigned long hva)
|
|
|
|
{
|
|
|
|
return kvm_handle_hva(kvm, hva, 0, kvm_test_age_rmapp);
|
|
|
|
}
|
|
|
|
|
2007-04-25 14:17:25 +08:00
|
|
|
#ifdef MMU_DEBUG
|
2007-05-06 20:50:58 +08:00
|
|
|
static int is_empty_shadow_page(u64 *spt)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2007-01-06 08:36:50 +08:00
|
|
|
u64 *pos;
|
|
|
|
u64 *end;
|
|
|
|
|
2007-05-06 20:50:58 +08:00
|
|
|
for (pos = spt, end = pos + PAGE_SIZE / sizeof(u64); pos != end; pos++)
|
2008-05-20 21:21:13 +08:00
|
|
|
if (is_shadow_present_pte(*pos)) {
|
2008-03-04 04:59:56 +08:00
|
|
|
printk(KERN_ERR "%s: %p %llx\n", __func__,
|
2007-01-06 08:36:50 +08:00
|
|
|
pos, *pos);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
return 0;
|
2007-01-06 08:36:50 +08:00
|
|
|
}
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
return 1;
|
|
|
|
}
|
2007-04-25 14:17:25 +08:00
|
|
|
#endif
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
KVM: create aggregate kvm_total_used_mmu_pages value
Of slab shrinkers, the VM code says:
* Note that 'shrink' will be passed nr_to_scan == 0 when the VM is
* querying the cache size, so a fastpath for that case is appropriate.
and it *means* it. Look at how it calls the shrinkers:
nr_before = (*shrinker->shrink)(0, gfp_mask);
shrink_ret = (*shrinker->shrink)(this_scan, gfp_mask);
So, if you do anything stupid in your shrinker, the VM will doubly
punish you.
The mmu_shrink() function takes the global kvm_lock, then acquires
every VM's kvm->mmu_lock in sequence. If we have 100 VMs, then
we're going to take 101 locks. We do it twice, so each call takes
202 locks. If we're under memory pressure, we can have each cpu
trying to do this. It can get really hairy, and we've seen lock
spinning in mmu_shrink() be the dominant entry in profiles.
This is guaranteed to optimize at least half of those lock
aquisitions away. It removes the need to take any of the locks
when simply trying to count objects.
A 'percpu_counter' can be a large object, but we only have one
of these for the entire system. There are not any better
alternatives at the moment, especially ones that handle CPU
hotplug.
Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
Signed-off-by: Tim Pepper <lnxninja@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2010-08-20 09:11:37 +08:00
|
|
|
/*
|
|
|
|
* This value is the sum of all of the kvm instances's
|
|
|
|
* kvm->arch.n_used_mmu_pages values. We need a global,
|
|
|
|
* aggregate version in order to make the slab shrinker
|
|
|
|
* faster
|
|
|
|
*/
|
2019-04-09 02:07:30 +08:00
|
|
|
static inline void kvm_mod_used_mmu_pages(struct kvm *kvm, unsigned long nr)
|
KVM: create aggregate kvm_total_used_mmu_pages value
Of slab shrinkers, the VM code says:
* Note that 'shrink' will be passed nr_to_scan == 0 when the VM is
* querying the cache size, so a fastpath for that case is appropriate.
and it *means* it. Look at how it calls the shrinkers:
nr_before = (*shrinker->shrink)(0, gfp_mask);
shrink_ret = (*shrinker->shrink)(this_scan, gfp_mask);
So, if you do anything stupid in your shrinker, the VM will doubly
punish you.
The mmu_shrink() function takes the global kvm_lock, then acquires
every VM's kvm->mmu_lock in sequence. If we have 100 VMs, then
we're going to take 101 locks. We do it twice, so each call takes
202 locks. If we're under memory pressure, we can have each cpu
trying to do this. It can get really hairy, and we've seen lock
spinning in mmu_shrink() be the dominant entry in profiles.
This is guaranteed to optimize at least half of those lock
aquisitions away. It removes the need to take any of the locks
when simply trying to count objects.
A 'percpu_counter' can be a large object, but we only have one
of these for the entire system. There are not any better
alternatives at the moment, especially ones that handle CPU
hotplug.
Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
Signed-off-by: Tim Pepper <lnxninja@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2010-08-20 09:11:37 +08:00
|
|
|
{
|
|
|
|
kvm->arch.n_used_mmu_pages += nr;
|
|
|
|
percpu_counter_add(&kvm_total_used_mmu_pages, nr);
|
|
|
|
}
|
|
|
|
|
2013-01-30 22:45:05 +08:00
|
|
|
static void kvm_mmu_free_page(struct kvm_mmu_page *sp)
|
2007-01-06 08:36:49 +08:00
|
|
|
{
|
2013-10-02 22:56:16 +08:00
|
|
|
MMU_WARN_ON(!is_empty_shadow_page(sp->spt));
|
2010-06-04 21:53:54 +08:00
|
|
|
hlist_del(&sp->hash_link);
|
2011-07-12 03:27:14 +08:00
|
|
|
list_del(&sp->link);
|
|
|
|
free_page((unsigned long)sp->spt);
|
2013-01-30 22:45:05 +08:00
|
|
|
if (!sp->role.direct)
|
|
|
|
free_page((unsigned long)sp->gfns);
|
2010-05-13 10:06:02 +08:00
|
|
|
kmem_cache_free(mmu_page_header_cache, sp);
|
2007-01-06 08:36:49 +08:00
|
|
|
}
|
|
|
|
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
static unsigned kvm_page_table_hashfn(gfn_t gfn)
|
|
|
|
{
|
kvm: x86: reduce collisions in mmu_page_hash
When using two-dimensional paging, the mmu_page_hash (which provides
lookups for existing kvm_mmu_page structs), becomes imbalanced; with
too many collisions in buckets 0 and 512. This has been seen to cause
mmu_lock to be held for multiple milliseconds in kvm_mmu_get_page on
VMs with a large amount of RAM mapped with 4K pages.
The current hash function uses the lower 10 bits of gfn to index into
mmu_page_hash. When doing shadow paging, gfn is the address of the
guest page table being shadow. These tables are 4K-aligned, which
makes the low bits of gfn a good hash. However, with two-dimensional
paging, no guest page tables are being shadowed, so gfn is the base
address that is mapped by the table. Thus page tables (level=1) have
a 2MB aligned gfn, page directories (level=2) have a 1GB aligned gfn,
etc. This means hashes will only differ in their 10th bit.
hash_64() provides a better hash. For example, on a VM with ~200G
(99458 direct=1 kvm_mmu_page structs):
hash max_mmu_page_hash_collisions
--------------------------------------------
low 10 bits 49847
hash_64 105
perfect 97
While we're changing the hash, increase the table size by 4x to better
support large VMs (further reduces number of collisions in 200G VM to
29).
Note that hash_64() does not provide a good distribution prior to commit
ef703f49a6c5 ("Eliminate bad hash multipliers from hash_32() and
hash_64()").
Signed-off-by: David Matlack <dmatlack@google.com>
Change-Id: I5aa6b13c834722813c6cca46b8b1ed6f53368ade
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-12-20 05:58:25 +08:00
|
|
|
return hash_64(gfn, KVM_MMU_HASH_SHIFT);
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
}
|
|
|
|
|
2007-01-06 08:36:53 +08:00
|
|
|
static void mmu_page_add_parent_pte(struct kvm_vcpu *vcpu,
|
2007-11-21 21:28:32 +08:00
|
|
|
struct kvm_mmu_page *sp, u64 *parent_pte)
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
{
|
|
|
|
if (!parent_pte)
|
|
|
|
return;
|
|
|
|
|
2011-05-15 23:27:08 +08:00
|
|
|
pte_list_add(vcpu, parent_pte, &sp->parent_ptes);
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
}
|
|
|
|
|
2007-11-21 21:28:32 +08:00
|
|
|
static void mmu_page_remove_parent_pte(struct kvm_mmu_page *sp,
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
u64 *parent_pte)
|
|
|
|
{
|
2018-10-04 10:04:22 +08:00
|
|
|
__pte_list_remove(parent_pte, &sp->parent_ptes);
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
}
|
|
|
|
|
2011-05-15 23:28:29 +08:00
|
|
|
static void drop_parent_pte(struct kvm_mmu_page *sp,
|
|
|
|
u64 *parent_pte)
|
|
|
|
{
|
|
|
|
mmu_page_remove_parent_pte(sp, parent_pte);
|
2011-07-12 03:30:35 +08:00
|
|
|
mmu_spte_clear_no_track(parent_pte);
|
2011-05-15 23:28:29 +08:00
|
|
|
}
|
|
|
|
|
2015-11-20 16:46:29 +08:00
|
|
|
static struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, int direct)
|
2008-09-24 00:18:36 +08:00
|
|
|
{
|
2011-05-15 23:27:08 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2013-03-21 18:33:43 +08:00
|
|
|
|
2012-05-29 22:54:26 +08:00
|
|
|
sp = mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache);
|
|
|
|
sp->spt = mmu_memory_cache_alloc(&vcpu->arch.mmu_page_cache);
|
2011-05-15 23:27:08 +08:00
|
|
|
if (!direct)
|
2012-05-29 22:54:26 +08:00
|
|
|
sp->gfns = mmu_memory_cache_alloc(&vcpu->arch.mmu_page_cache);
|
2011-05-15 23:27:08 +08:00
|
|
|
set_page_private(virt_to_page(sp->spt), (unsigned long)sp);
|
2019-09-13 10:46:02 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* active_mmu_pages must be a FIFO list, as kvm_zap_obsolete_pages()
|
|
|
|
* depends on valid pages being added to the head of the list. See
|
|
|
|
* comments in kvm_zap_obsolete_pages().
|
|
|
|
*/
|
2019-09-13 10:46:11 +08:00
|
|
|
sp->mmu_valid_gen = vcpu->kvm->arch.mmu_valid_gen;
|
2011-05-15 23:27:08 +08:00
|
|
|
list_add(&sp->link, &vcpu->kvm->arch.active_mmu_pages);
|
|
|
|
kvm_mod_used_mmu_pages(vcpu->kvm, +1);
|
|
|
|
return sp;
|
2008-09-24 00:18:36 +08:00
|
|
|
}
|
|
|
|
|
2011-05-15 23:27:08 +08:00
|
|
|
static void mark_unsync(u64 *spte);
|
2010-06-11 21:35:15 +08:00
|
|
|
static void kvm_mmu_mark_parents_unsync(struct kvm_mmu_page *sp)
|
2008-09-24 00:18:40 +08:00
|
|
|
{
|
2015-11-26 20:15:38 +08:00
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
|
|
|
|
|
|
|
for_each_rmap_spte(&sp->parent_ptes, &iter, sptep) {
|
|
|
|
mark_unsync(sptep);
|
|
|
|
}
|
2008-09-24 00:18:40 +08:00
|
|
|
}
|
|
|
|
|
2011-05-15 23:27:08 +08:00
|
|
|
static void mark_unsync(u64 *spte)
|
2008-09-24 00:18:40 +08:00
|
|
|
{
|
2011-05-15 23:27:08 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2010-06-11 21:35:15 +08:00
|
|
|
unsigned int index;
|
2008-09-24 00:18:40 +08:00
|
|
|
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(spte);
|
2010-06-11 21:35:15 +08:00
|
|
|
index = spte - sp->spt;
|
|
|
|
if (__test_and_set_bit(index, sp->unsync_child_bitmap))
|
2008-09-24 00:18:40 +08:00
|
|
|
return;
|
2010-06-11 21:35:15 +08:00
|
|
|
if (sp->unsync_children++)
|
2008-09-24 00:18:40 +08:00
|
|
|
return;
|
2010-06-11 21:35:15 +08:00
|
|
|
kvm_mmu_mark_parents_unsync(sp);
|
2008-09-24 00:18:40 +08:00
|
|
|
}
|
|
|
|
|
2008-09-24 00:18:33 +08:00
|
|
|
static int nonpaging_sync_page(struct kvm_vcpu *vcpu,
|
2010-11-19 17:04:03 +08:00
|
|
|
struct kvm_mmu_page *sp)
|
2008-09-24 00:18:33 +08:00
|
|
|
{
|
2016-02-24 18:07:14 +08:00
|
|
|
return 0;
|
2008-09-24 00:18:33 +08:00
|
|
|
}
|
|
|
|
|
2011-03-09 15:43:51 +08:00
|
|
|
static void nonpaging_update_pte(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu_page *sp, u64 *spte,
|
2011-03-28 10:29:27 +08:00
|
|
|
const void *pte)
|
2011-03-09 15:43:51 +08:00
|
|
|
{
|
|
|
|
WARN_ON(1);
|
|
|
|
}
|
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
#define KVM_PAGE_ARRAY_NR 16
|
|
|
|
|
|
|
|
struct kvm_mmu_pages {
|
|
|
|
struct mmu_page_and_offset {
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
unsigned int idx;
|
|
|
|
} page[KVM_PAGE_ARRAY_NR];
|
|
|
|
unsigned int nr;
|
|
|
|
};
|
|
|
|
|
2009-02-21 09:19:13 +08:00
|
|
|
static int mmu_pages_add(struct kvm_mmu_pages *pvec, struct kvm_mmu_page *sp,
|
|
|
|
int idx)
|
2008-09-24 00:18:39 +08:00
|
|
|
{
|
2008-12-02 08:32:02 +08:00
|
|
|
int i;
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
if (sp->unsync)
|
|
|
|
for (i=0; i < pvec->nr; i++)
|
|
|
|
if (pvec->page[i].sp == sp)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
pvec->page[pvec->nr].sp = sp;
|
|
|
|
pvec->page[pvec->nr].idx = idx;
|
|
|
|
pvec->nr++;
|
|
|
|
return (pvec->nr == KVM_PAGE_ARRAY_NR);
|
|
|
|
}
|
|
|
|
|
2015-11-20 16:43:13 +08:00
|
|
|
static inline void clear_unsync_child_bit(struct kvm_mmu_page *sp, int idx)
|
|
|
|
{
|
|
|
|
--sp->unsync_children;
|
|
|
|
WARN_ON((int)sp->unsync_children < 0);
|
|
|
|
__clear_bit(idx, sp->unsync_child_bitmap);
|
|
|
|
}
|
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
static int __mmu_unsync_walk(struct kvm_mmu_page *sp,
|
|
|
|
struct kvm_mmu_pages *pvec)
|
|
|
|
{
|
|
|
|
int i, ret, nr_unsync_leaf = 0;
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2011-11-29 13:02:45 +08:00
|
|
|
for_each_set_bit(i, sp->unsync_child_bitmap, 512) {
|
2010-06-11 21:34:04 +08:00
|
|
|
struct kvm_mmu_page *child;
|
2008-09-24 00:18:39 +08:00
|
|
|
u64 ent = sp->spt[i];
|
|
|
|
|
2015-11-20 16:43:13 +08:00
|
|
|
if (!is_shadow_present_pte(ent) || is_large_pte(ent)) {
|
|
|
|
clear_unsync_child_bit(sp, i);
|
|
|
|
continue;
|
|
|
|
}
|
2010-06-11 21:34:04 +08:00
|
|
|
|
2020-06-23 04:20:34 +08:00
|
|
|
child = to_shadow_page(ent & PT64_BASE_ADDR_MASK);
|
2010-06-11 21:34:04 +08:00
|
|
|
|
|
|
|
if (child->unsync_children) {
|
|
|
|
if (mmu_pages_add(pvec, child, i))
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
ret = __mmu_unsync_walk(child, pvec);
|
2015-11-20 16:43:13 +08:00
|
|
|
if (!ret) {
|
|
|
|
clear_unsync_child_bit(sp, i);
|
|
|
|
continue;
|
|
|
|
} else if (ret > 0) {
|
2010-06-11 21:34:04 +08:00
|
|
|
nr_unsync_leaf += ret;
|
2015-11-20 16:43:13 +08:00
|
|
|
} else
|
2010-06-11 21:34:04 +08:00
|
|
|
return ret;
|
|
|
|
} else if (child->unsync) {
|
|
|
|
nr_unsync_leaf++;
|
|
|
|
if (mmu_pages_add(pvec, child, i))
|
|
|
|
return -ENOSPC;
|
|
|
|
} else
|
2015-11-20 16:43:13 +08:00
|
|
|
clear_unsync_child_bit(sp, i);
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
return nr_unsync_leaf;
|
|
|
|
}
|
|
|
|
|
2016-02-24 16:46:06 +08:00
|
|
|
#define INVALID_INDEX (-1)
|
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
static int mmu_unsync_walk(struct kvm_mmu_page *sp,
|
|
|
|
struct kvm_mmu_pages *pvec)
|
|
|
|
{
|
2016-02-23 20:54:25 +08:00
|
|
|
pvec->nr = 0;
|
2008-12-02 08:32:02 +08:00
|
|
|
if (!sp->unsync_children)
|
|
|
|
return 0;
|
|
|
|
|
2016-02-24 16:46:06 +08:00
|
|
|
mmu_pages_add(pvec, sp, INVALID_INDEX);
|
2008-12-02 08:32:02 +08:00
|
|
|
return __mmu_unsync_walk(sp, pvec);
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void kvm_unlink_unsync_page(struct kvm *kvm, struct kvm_mmu_page *sp)
|
|
|
|
{
|
|
|
|
WARN_ON(!sp->unsync);
|
2010-04-28 11:55:06 +08:00
|
|
|
trace_kvm_mmu_sync_page(sp);
|
2008-09-24 00:18:39 +08:00
|
|
|
sp->unsync = 0;
|
|
|
|
--kvm->stat.mmu_unsync;
|
|
|
|
}
|
|
|
|
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
static bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,
|
|
|
|
struct list_head *invalid_list);
|
2010-06-04 21:53:54 +08:00
|
|
|
static void kvm_mmu_commit_zap_page(struct kvm *kvm,
|
|
|
|
struct list_head *invalid_list);
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2020-06-24 03:40:26 +08:00
|
|
|
#define for_each_valid_sp(_kvm, _sp, _list) \
|
|
|
|
hlist_for_each_entry(_sp, _list, hash_link) \
|
2019-09-13 10:46:03 +08:00
|
|
|
if (is_obsolete_sp((_kvm), (_sp))) { \
|
2016-12-21 07:25:57 +08:00
|
|
|
} else
|
2013-03-06 15:05:07 +08:00
|
|
|
|
|
|
|
#define for_each_gfn_indirect_valid_sp(_kvm, _sp, _gfn) \
|
2020-06-24 03:40:26 +08:00
|
|
|
for_each_valid_sp(_kvm, _sp, \
|
|
|
|
&(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)]) \
|
2016-12-21 07:25:57 +08:00
|
|
|
if ((_sp)->gfn != (_gfn) || (_sp)->role.direct) {} else
|
2010-06-04 21:53:07 +08:00
|
|
|
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
static inline bool is_ept_sp(struct kvm_mmu_page *sp)
|
|
|
|
{
|
|
|
|
return sp->role.cr0_wp && sp->role.smap_andnot_wp;
|
|
|
|
}
|
|
|
|
|
2010-06-11 21:30:36 +08:00
|
|
|
/* @sp->gfn should be write-protected at the call site */
|
2016-02-24 18:07:14 +08:00
|
|
|
static bool __kvm_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
|
|
|
|
struct list_head *invalid_list)
|
2008-09-24 00:18:39 +08:00
|
|
|
{
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
if ((!is_ept_sp(sp) && sp->role.gpte_is_8_bytes != !!is_pae(vcpu)) ||
|
|
|
|
vcpu->arch.mmu->sync_page(vcpu, sp) == 0) {
|
2010-06-04 21:55:29 +08:00
|
|
|
kvm_mmu_prepare_zap_page(vcpu->kvm, sp, invalid_list);
|
2016-02-24 18:07:14 +08:00
|
|
|
return false;
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 18:07:14 +08:00
|
|
|
return true;
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
|
|
|
|
2019-02-06 05:01:20 +08:00
|
|
|
static bool kvm_mmu_remote_flush_or_zap(struct kvm *kvm,
|
|
|
|
struct list_head *invalid_list,
|
|
|
|
bool remote_flush)
|
|
|
|
{
|
2019-04-13 10:55:41 +08:00
|
|
|
if (!remote_flush && list_empty(invalid_list))
|
2019-02-06 05:01:20 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
if (!list_empty(invalid_list))
|
|
|
|
kvm_mmu_commit_zap_page(kvm, invalid_list);
|
|
|
|
else
|
|
|
|
kvm_flush_remote_tlbs(kvm);
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2016-02-24 17:03:27 +08:00
|
|
|
static void kvm_mmu_flush_or_zap(struct kvm_vcpu *vcpu,
|
|
|
|
struct list_head *invalid_list,
|
|
|
|
bool remote_flush, bool local_flush)
|
2010-05-15 18:51:24 +08:00
|
|
|
{
|
2019-02-06 05:01:20 +08:00
|
|
|
if (kvm_mmu_remote_flush_or_zap(vcpu->kvm, invalid_list, remote_flush))
|
2016-02-24 17:03:27 +08:00
|
|
|
return;
|
2010-06-04 21:55:29 +08:00
|
|
|
|
2019-02-06 05:01:20 +08:00
|
|
|
if (local_flush)
|
2020-03-21 05:28:21 +08:00
|
|
|
kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
|
2010-05-15 18:51:24 +08:00
|
|
|
}
|
|
|
|
|
2011-11-30 17:43:24 +08:00
|
|
|
#ifdef CONFIG_KVM_MMU_AUDIT
|
|
|
|
#include "mmu_audit.c"
|
|
|
|
#else
|
|
|
|
static void kvm_mmu_audit(struct kvm_vcpu *vcpu, int point) { }
|
|
|
|
static void mmu_audit_disable(void) { }
|
|
|
|
#endif
|
|
|
|
|
2019-09-13 10:46:02 +08:00
|
|
|
static bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
|
|
|
|
{
|
2019-09-13 10:46:03 +08:00
|
|
|
return sp->role.invalid ||
|
|
|
|
unlikely(sp->mmu_valid_gen != kvm->arch.mmu_valid_gen);
|
2019-09-13 10:46:02 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 18:07:14 +08:00
|
|
|
static bool kvm_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
|
2010-06-04 21:55:29 +08:00
|
|
|
struct list_head *invalid_list)
|
2010-05-15 18:51:24 +08:00
|
|
|
{
|
2016-02-24 17:28:01 +08:00
|
|
|
kvm_unlink_unsync_page(vcpu->kvm, sp);
|
|
|
|
return __kvm_sync_page(vcpu, sp, invalid_list);
|
2010-05-15 18:51:24 +08:00
|
|
|
}
|
|
|
|
|
2010-05-24 15:41:33 +08:00
|
|
|
/* @gfn should be write-protected at the call site */
|
2016-02-24 18:26:10 +08:00
|
|
|
static bool kvm_sync_pages(struct kvm_vcpu *vcpu, gfn_t gfn,
|
|
|
|
struct list_head *invalid_list)
|
2010-05-24 15:41:33 +08:00
|
|
|
{
|
|
|
|
struct kvm_mmu_page *s;
|
2016-02-24 18:26:10 +08:00
|
|
|
bool ret = false;
|
2010-05-24 15:41:33 +08:00
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-28 09:06:00 +08:00
|
|
|
for_each_gfn_indirect_valid_sp(vcpu->kvm, s, gfn) {
|
2010-06-04 21:53:07 +08:00
|
|
|
if (!s->unsync)
|
2010-05-24 15:41:33 +08:00
|
|
|
continue;
|
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
WARN_ON(s->role.level != PG_LEVEL_4K);
|
2016-02-24 18:26:10 +08:00
|
|
|
ret |= kvm_sync_page(vcpu, s, invalid_list);
|
2010-05-24 15:41:33 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 18:26:10 +08:00
|
|
|
return ret;
|
2010-05-24 15:41:33 +08:00
|
|
|
}
|
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
struct mmu_page_path {
|
2017-08-24 20:27:54 +08:00
|
|
|
struct kvm_mmu_page *parent[PT64_ROOT_MAX_LEVEL];
|
|
|
|
unsigned int idx[PT64_ROOT_MAX_LEVEL];
|
2008-09-24 00:18:39 +08:00
|
|
|
};
|
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
#define for_each_sp(pvec, sp, parents, i) \
|
2016-02-23 20:54:25 +08:00
|
|
|
for (i = mmu_pages_first(&pvec, &parents); \
|
2008-12-02 08:32:02 +08:00
|
|
|
i < pvec.nr && ({ sp = pvec.page[i].sp; 1;}); \
|
|
|
|
i = mmu_pages_next(&pvec, &parents, i))
|
|
|
|
|
2009-02-21 09:19:13 +08:00
|
|
|
static int mmu_pages_next(struct kvm_mmu_pages *pvec,
|
|
|
|
struct mmu_page_path *parents,
|
|
|
|
int i)
|
2008-12-02 08:32:02 +08:00
|
|
|
{
|
|
|
|
int n;
|
|
|
|
|
|
|
|
for (n = i+1; n < pvec->nr; n++) {
|
|
|
|
struct kvm_mmu_page *sp = pvec->page[n].sp;
|
2016-02-23 20:54:25 +08:00
|
|
|
unsigned idx = pvec->page[n].idx;
|
|
|
|
int level = sp->role.level;
|
2008-12-02 08:32:02 +08:00
|
|
|
|
2016-02-23 20:54:25 +08:00
|
|
|
parents->idx[level-1] = idx;
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level == PG_LEVEL_4K)
|
2016-02-23 20:54:25 +08:00
|
|
|
break;
|
2008-12-02 08:32:02 +08:00
|
|
|
|
2016-02-23 20:54:25 +08:00
|
|
|
parents->parent[level-2] = sp;
|
2008-12-02 08:32:02 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return n;
|
|
|
|
}
|
|
|
|
|
2016-02-23 20:54:25 +08:00
|
|
|
static int mmu_pages_first(struct kvm_mmu_pages *pvec,
|
|
|
|
struct mmu_page_path *parents)
|
|
|
|
{
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
int level;
|
|
|
|
|
|
|
|
if (pvec->nr == 0)
|
|
|
|
return 0;
|
|
|
|
|
2016-02-24 16:46:06 +08:00
|
|
|
WARN_ON(pvec->page[0].idx != INVALID_INDEX);
|
|
|
|
|
2016-02-23 20:54:25 +08:00
|
|
|
sp = pvec->page[0].sp;
|
|
|
|
level = sp->role.level;
|
2020-04-28 08:54:22 +08:00
|
|
|
WARN_ON(level == PG_LEVEL_4K);
|
2016-02-23 20:54:25 +08:00
|
|
|
|
|
|
|
parents->parent[level-2] = sp;
|
|
|
|
|
|
|
|
/* Also set up a sentinel. Further entries in pvec are all
|
|
|
|
* children of sp, so this element is never overwritten.
|
|
|
|
*/
|
|
|
|
parents->parent[level-1] = NULL;
|
|
|
|
return mmu_pages_next(pvec, parents, 0);
|
|
|
|
}
|
|
|
|
|
2009-02-21 09:19:13 +08:00
|
|
|
static void mmu_pages_clear_parents(struct mmu_page_path *parents)
|
2008-09-24 00:18:39 +08:00
|
|
|
{
|
2008-12-02 08:32:02 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
unsigned int level = 0;
|
|
|
|
|
|
|
|
do {
|
|
|
|
unsigned int idx = parents->idx[level];
|
|
|
|
sp = parents->parent[level];
|
|
|
|
if (!sp)
|
|
|
|
return;
|
|
|
|
|
2016-02-24 16:46:06 +08:00
|
|
|
WARN_ON(idx == INVALID_INDEX);
|
2015-11-20 16:43:13 +08:00
|
|
|
clear_unsync_child_bit(sp, idx);
|
2008-12-02 08:32:02 +08:00
|
|
|
level++;
|
2016-02-23 20:54:25 +08:00
|
|
|
} while (!sp->unsync_children);
|
2008-12-02 08:32:02 +08:00
|
|
|
}
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
static void mmu_sync_children(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu_page *parent)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
struct mmu_page_path parents;
|
|
|
|
struct kvm_mmu_pages pages;
|
2010-06-04 21:55:29 +08:00
|
|
|
LIST_HEAD(invalid_list);
|
2016-02-25 17:47:38 +08:00
|
|
|
bool flush = false;
|
2008-12-02 08:32:02 +08:00
|
|
|
|
|
|
|
while (mmu_unsync_walk(parent, &pages)) {
|
2012-06-20 15:56:53 +08:00
|
|
|
bool protected = false;
|
2008-12-02 08:32:03 +08:00
|
|
|
|
|
|
|
for_each_sp(pages, sp, parents, i)
|
2015-04-08 21:39:23 +08:00
|
|
|
protected |= rmap_write_protect(vcpu, sp->gfn);
|
2008-12-02 08:32:03 +08:00
|
|
|
|
2016-02-25 17:47:38 +08:00
|
|
|
if (protected) {
|
2008-12-02 08:32:03 +08:00
|
|
|
kvm_flush_remote_tlbs(vcpu->kvm);
|
2016-02-25 17:47:38 +08:00
|
|
|
flush = false;
|
|
|
|
}
|
2008-12-02 08:32:03 +08:00
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
for_each_sp(pages, sp, parents, i) {
|
2016-02-24 18:07:14 +08:00
|
|
|
flush |= kvm_sync_page(vcpu, sp, &invalid_list);
|
2008-12-02 08:32:02 +08:00
|
|
|
mmu_pages_clear_parents(&parents);
|
|
|
|
}
|
2016-02-25 17:47:38 +08:00
|
|
|
if (need_resched() || spin_needbreak(&vcpu->kvm->mmu_lock)) {
|
|
|
|
kvm_mmu_flush_or_zap(vcpu, &invalid_list, false, flush);
|
|
|
|
cond_resched_lock(&vcpu->kvm->mmu_lock);
|
|
|
|
flush = false;
|
|
|
|
}
|
2008-12-02 08:32:02 +08:00
|
|
|
}
|
2016-02-25 17:47:38 +08:00
|
|
|
|
|
|
|
kvm_mmu_flush_or_zap(vcpu, &invalid_list, false, flush);
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
|
|
|
|
KVM: MMU: improve write flooding detected
Detecting write-flooding does not work well, when we handle page written, if
the last speculative spte is not accessed, we treat the page is
write-flooding, however, we can speculative spte on many path, such as pte
prefetch, page synced, that means the last speculative spte may be not point
to the written page and the written page can be accessed via other sptes, so
depends on the Accessed bit of the last speculative spte is not enough
Instead of detected page accessed, we can detect whether the spte is accessed
after it is written, if the spte is not accessed but it is written frequently,
we treat is not a page table or it not used for a long time
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-09-22 16:58:36 +08:00
|
|
|
static void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp)
|
|
|
|
{
|
2016-02-24 17:51:12 +08:00
|
|
|
atomic_set(&sp->write_flooding_count, 0);
|
KVM: MMU: improve write flooding detected
Detecting write-flooding does not work well, when we handle page written, if
the last speculative spte is not accessed, we treat the page is
write-flooding, however, we can speculative spte on many path, such as pte
prefetch, page synced, that means the last speculative spte may be not point
to the written page and the written page can be accessed via other sptes, so
depends on the Accessed bit of the last speculative spte is not enough
Instead of detected page accessed, we can detect whether the spte is accessed
after it is written, if the spte is not accessed but it is written frequently,
we treat is not a page table or it not used for a long time
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-09-22 16:58:36 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void clear_sp_write_flooding_count(u64 *spte)
|
|
|
|
{
|
2020-06-23 04:20:33 +08:00
|
|
|
__clear_sp_write_flooding_count(sptep_to_sp(spte));
|
KVM: MMU: improve write flooding detected
Detecting write-flooding does not work well, when we handle page written, if
the last speculative spte is not accessed, we treat the page is
write-flooding, however, we can speculative spte on many path, such as pte
prefetch, page synced, that means the last speculative spte may be not point
to the written page and the written page can be accessed via other sptes, so
depends on the Accessed bit of the last speculative spte is not enough
Instead of detected page accessed, we can detect whether the spte is accessed
after it is written, if the spte is not accessed but it is written frequently,
we treat is not a page table or it not used for a long time
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-09-22 16:58:36 +08:00
|
|
|
}
|
|
|
|
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
|
|
|
|
gfn_t gfn,
|
|
|
|
gva_t gaddr,
|
|
|
|
unsigned level,
|
2009-01-11 19:02:10 +08:00
|
|
|
int direct,
|
2020-02-04 07:09:09 +08:00
|
|
|
unsigned int access)
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
{
|
2020-06-24 03:40:27 +08:00
|
|
|
bool direct_mmu = vcpu->arch.mmu->direct_map;
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
union kvm_mmu_page_role role;
|
2020-06-24 03:40:26 +08:00
|
|
|
struct hlist_head *sp_list;
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
unsigned quadrant;
|
2010-05-24 15:41:33 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
bool need_sync = false;
|
2016-02-24 18:26:10 +08:00
|
|
|
bool flush = false;
|
2016-12-21 07:25:57 +08:00
|
|
|
int collisions = 0;
|
2016-02-24 18:26:10 +08:00
|
|
|
LIST_HEAD(invalid_list);
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
|
2018-10-09 03:28:10 +08:00
|
|
|
role = vcpu->arch.mmu->mmu_role.base;
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
role.level = level;
|
2009-01-11 19:02:10 +08:00
|
|
|
role.direct = direct;
|
2010-03-14 16:16:40 +08:00
|
|
|
if (role.direct)
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
role.gpte_is_8_bytes = true;
|
2007-12-09 23:00:02 +08:00
|
|
|
role.access = access;
|
2020-06-24 03:40:27 +08:00
|
|
|
if (!direct_mmu && vcpu->arch.mmu->root_level <= PT32_ROOT_LEVEL) {
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
quadrant = gaddr >> (PAGE_SHIFT + (PT64_PT_BITS * level));
|
|
|
|
quadrant &= (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1;
|
|
|
|
role.quadrant = quadrant;
|
|
|
|
}
|
2020-06-24 03:40:26 +08:00
|
|
|
|
|
|
|
sp_list = &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)];
|
|
|
|
for_each_valid_sp(vcpu->kvm, sp, sp_list) {
|
2016-12-21 07:25:57 +08:00
|
|
|
if (sp->gfn != gfn) {
|
|
|
|
collisions++;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2010-06-04 21:53:07 +08:00
|
|
|
if (!need_sync && sp->unsync)
|
|
|
|
need_sync = true;
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2010-06-04 21:53:07 +08:00
|
|
|
if (sp->role.word != role.word)
|
|
|
|
continue;
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2020-06-24 03:40:27 +08:00
|
|
|
if (direct_mmu)
|
|
|
|
goto trace_get_page;
|
|
|
|
|
2016-02-24 18:26:10 +08:00
|
|
|
if (sp->unsync) {
|
|
|
|
/* The page is good, but __kvm_sync_page might still end
|
|
|
|
* up zapping it. If so, break in order to rebuild it.
|
|
|
|
*/
|
|
|
|
if (!__kvm_sync_page(vcpu, sp, &invalid_list))
|
|
|
|
break;
|
|
|
|
|
|
|
|
WARN_ON(!list_empty(&invalid_list));
|
2020-03-21 05:28:21 +08:00
|
|
|
kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
|
2016-02-24 18:26:10 +08:00
|
|
|
}
|
KVM: MMU: don't write-protect if have new mapping to unsync page
Two cases maybe happen in kvm_mmu_get_page() function:
- one case is, the goal sp is already in cache, if the sp is unsync,
we only need update it to assure this mapping is valid, but not
mark it sync and not write-protect sp->gfn since it not broke unsync
rule(one shadow page for a gfn)
- another case is, the goal sp not existed, we need create a new sp
for gfn, i.e, gfn (may)has another shadow page, to keep unsync rule,
we should sync(mark sync and write-protect) gfn's unsync shadow page.
After enabling multiple unsync shadows, we sync those shadow pages
only when the new sp not allow to become unsync(also for the unsyc
rule, the new rule is: allow all pte page become unsync)
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2010-05-15 18:52:34 +08:00
|
|
|
|
2015-11-26 20:14:34 +08:00
|
|
|
if (sp->unsync_children)
|
2020-03-21 05:28:21 +08:00
|
|
|
kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
|
KVM: MMU: don't write-protect if have new mapping to unsync page
Two cases maybe happen in kvm_mmu_get_page() function:
- one case is, the goal sp is already in cache, if the sp is unsync,
we only need update it to assure this mapping is valid, but not
mark it sync and not write-protect sp->gfn since it not broke unsync
rule(one shadow page for a gfn)
- another case is, the goal sp not existed, we need create a new sp
for gfn, i.e, gfn (may)has another shadow page, to keep unsync rule,
we should sync(mark sync and write-protect) gfn's unsync shadow page.
After enabling multiple unsync shadows, we sync those shadow pages
only when the new sp not allow to become unsync(also for the unsyc
rule, the new rule is: allow all pte page become unsync)
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2010-05-15 18:52:34 +08:00
|
|
|
|
KVM: MMU: improve write flooding detected
Detecting write-flooding does not work well, when we handle page written, if
the last speculative spte is not accessed, we treat the page is
write-flooding, however, we can speculative spte on many path, such as pte
prefetch, page synced, that means the last speculative spte may be not point
to the written page and the written page can be accessed via other sptes, so
depends on the Accessed bit of the last speculative spte is not enough
Instead of detected page accessed, we can detect whether the spte is accessed
after it is written, if the spte is not accessed but it is written frequently,
we treat is not a page table or it not used for a long time
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-09-22 16:58:36 +08:00
|
|
|
__clear_sp_write_flooding_count(sp);
|
2020-06-24 03:40:27 +08:00
|
|
|
|
|
|
|
trace_get_page:
|
2010-06-04 21:53:07 +08:00
|
|
|
trace_kvm_mmu_get_page(sp, false);
|
2016-12-21 07:25:57 +08:00
|
|
|
goto out;
|
2010-06-04 21:53:07 +08:00
|
|
|
}
|
2015-11-20 16:46:29 +08:00
|
|
|
|
2007-12-19 01:47:18 +08:00
|
|
|
++vcpu->kvm->stat.mmu_cache_miss;
|
2015-11-20 16:46:29 +08:00
|
|
|
|
|
|
|
sp = kvm_mmu_alloc_page(vcpu, direct);
|
|
|
|
|
2007-11-21 21:28:32 +08:00
|
|
|
sp->gfn = gfn;
|
|
|
|
sp->role = role;
|
2020-06-24 03:40:26 +08:00
|
|
|
hlist_add_head(&sp->hash_link, sp_list);
|
2009-01-11 19:02:10 +08:00
|
|
|
if (!direct) {
|
2016-02-24 17:51:14 +08:00
|
|
|
/*
|
|
|
|
* we should do write protection before syncing pages
|
|
|
|
* otherwise the content of the synced shadow page may
|
|
|
|
* be inconsistent with guest page table.
|
|
|
|
*/
|
|
|
|
account_shadowed(vcpu->kvm, sp);
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level == PG_LEVEL_4K && rmap_write_protect(vcpu, gfn))
|
2018-12-06 21:21:09 +08:00
|
|
|
kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn, 1);
|
2010-05-24 15:41:33 +08:00
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level > PG_LEVEL_4K && need_sync)
|
2016-02-24 18:26:10 +08:00
|
|
|
flush |= kvm_sync_pages(vcpu, gfn, &invalid_list);
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
2015-12-18 17:54:49 +08:00
|
|
|
clear_page(sp->spt);
|
2009-07-06 20:58:14 +08:00
|
|
|
trace_kvm_mmu_get_page(sp, true);
|
2016-02-24 18:26:10 +08:00
|
|
|
|
|
|
|
kvm_mmu_flush_or_zap(vcpu, &invalid_list, false, flush);
|
2016-12-21 07:25:57 +08:00
|
|
|
out:
|
|
|
|
if (collisions > vcpu->kvm->stat.max_mmu_page_hash_collisions)
|
|
|
|
vcpu->kvm->stat.max_mmu_page_hash_collisions = collisions;
|
2007-11-21 21:28:32 +08:00
|
|
|
return sp;
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:16 +08:00
|
|
|
static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator,
|
|
|
|
struct kvm_vcpu *vcpu, hpa_t root,
|
|
|
|
u64 addr)
|
2008-12-25 20:39:47 +08:00
|
|
|
{
|
|
|
|
iterator->addr = addr;
|
2018-06-28 05:59:16 +08:00
|
|
|
iterator->shadow_addr = root;
|
2018-10-09 03:28:05 +08:00
|
|
|
iterator->level = vcpu->arch.mmu->shadow_root_level;
|
2010-09-10 23:31:00 +08:00
|
|
|
|
2017-08-24 20:27:54 +08:00
|
|
|
if (iterator->level == PT64_ROOT_4LEVEL &&
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->root_level < PT64_ROOT_4LEVEL &&
|
|
|
|
!vcpu->arch.mmu->direct_map)
|
2010-09-10 23:31:00 +08:00
|
|
|
--iterator->level;
|
|
|
|
|
2008-12-25 20:39:47 +08:00
|
|
|
if (iterator->level == PT32E_ROOT_LEVEL) {
|
2018-06-28 05:59:16 +08:00
|
|
|
/*
|
|
|
|
* prev_root is currently only used for 64-bit hosts. So only
|
|
|
|
* the active root_hpa is valid here.
|
|
|
|
*/
|
2018-10-09 03:28:05 +08:00
|
|
|
BUG_ON(root != vcpu->arch.mmu->root_hpa);
|
2018-06-28 05:59:16 +08:00
|
|
|
|
2008-12-25 20:39:47 +08:00
|
|
|
iterator->shadow_addr
|
2018-10-09 03:28:05 +08:00
|
|
|
= vcpu->arch.mmu->pae_root[(addr >> 30) & 3];
|
2008-12-25 20:39:47 +08:00
|
|
|
iterator->shadow_addr &= PT64_BASE_ADDR_MASK;
|
|
|
|
--iterator->level;
|
|
|
|
if (!iterator->shadow_addr)
|
|
|
|
iterator->level = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:16 +08:00
|
|
|
static void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator,
|
|
|
|
struct kvm_vcpu *vcpu, u64 addr)
|
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
shadow_walk_init_using_root(iterator, vcpu, vcpu->arch.mmu->root_hpa,
|
2018-06-28 05:59:16 +08:00
|
|
|
addr);
|
|
|
|
}
|
|
|
|
|
2008-12-25 20:39:47 +08:00
|
|
|
static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)
|
|
|
|
{
|
2020-04-28 08:54:22 +08:00
|
|
|
if (iterator->level < PG_LEVEL_4K)
|
2008-12-25 20:39:47 +08:00
|
|
|
return false;
|
2009-06-11 23:07:41 +08:00
|
|
|
|
2008-12-25 20:39:47 +08:00
|
|
|
iterator->index = SHADOW_PT_INDEX(iterator->addr, iterator->level);
|
|
|
|
iterator->sptep = ((u64 *)__va(iterator->shadow_addr)) + iterator->index;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2011-07-12 03:32:13 +08:00
|
|
|
static void __shadow_walk_next(struct kvm_shadow_walk_iterator *iterator,
|
|
|
|
u64 spte)
|
2008-12-25 20:39:47 +08:00
|
|
|
{
|
2011-07-12 03:32:13 +08:00
|
|
|
if (is_last_spte(spte, iterator->level)) {
|
2011-07-12 03:21:17 +08:00
|
|
|
iterator->level = 0;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2011-07-12 03:32:13 +08:00
|
|
|
iterator->shadow_addr = spte & PT64_BASE_ADDR_MASK;
|
2008-12-25 20:39:47 +08:00
|
|
|
--iterator->level;
|
|
|
|
}
|
|
|
|
|
2011-07-12 03:32:13 +08:00
|
|
|
static void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator)
|
|
|
|
{
|
2017-08-25 02:51:23 +08:00
|
|
|
__shadow_walk_next(iterator, *iterator->sptep);
|
2011-07-12 03:32:13 +08:00
|
|
|
}
|
|
|
|
|
2015-11-26 20:14:34 +08:00
|
|
|
static void link_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep,
|
|
|
|
struct kvm_mmu_page *sp)
|
2010-07-13 19:27:04 +08:00
|
|
|
{
|
|
|
|
u64 spte;
|
|
|
|
|
2016-07-13 06:18:49 +08:00
|
|
|
BUILD_BUG_ON(VMX_EPT_WRITABLE_MASK != PT_WRITABLE_MASK);
|
2013-08-05 16:07:13 +08:00
|
|
|
|
2016-07-13 06:18:49 +08:00
|
|
|
spte = __pa(sp->spt) | shadow_present_mask | PT_WRITABLE_MASK |
|
2017-07-18 05:10:27 +08:00
|
|
|
shadow_user_mask | shadow_x_mask | shadow_me_mask;
|
2017-07-01 08:26:31 +08:00
|
|
|
|
|
|
|
if (sp_ad_disabled(sp))
|
2019-09-24 18:43:08 +08:00
|
|
|
spte |= SPTE_AD_DISABLED_MASK;
|
2017-07-01 08:26:31 +08:00
|
|
|
else
|
|
|
|
spte |= shadow_accessed_mask;
|
2013-02-05 15:28:02 +08:00
|
|
|
|
2011-07-12 03:30:35 +08:00
|
|
|
mmu_spte_set(sptep, spte);
|
2015-11-26 20:14:34 +08:00
|
|
|
|
|
|
|
mmu_page_add_parent_pte(vcpu, sp, sptep);
|
|
|
|
|
|
|
|
if (sp->unsync_children || sp->unsync)
|
|
|
|
mark_unsync(sptep);
|
2010-07-13 19:27:04 +08:00
|
|
|
}
|
|
|
|
|
2010-07-13 19:27:07 +08:00
|
|
|
static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep,
|
|
|
|
unsigned direct_access)
|
|
|
|
{
|
|
|
|
if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep)) {
|
|
|
|
struct kvm_mmu_page *child;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* For the direct sp, if the guest pte's dirty bit
|
|
|
|
* changed form clean to dirty, it will corrupt the
|
|
|
|
* sp's access: allow writable in the read-only sp,
|
|
|
|
* so we should update the spte at this point to get
|
|
|
|
* a new sp with the correct access.
|
|
|
|
*/
|
2020-06-23 04:20:34 +08:00
|
|
|
child = to_shadow_page(*sptep & PT64_BASE_ADDR_MASK);
|
2010-07-13 19:27:07 +08:00
|
|
|
if (child->role.access == direct_access)
|
|
|
|
return;
|
|
|
|
|
2011-05-15 23:28:29 +08:00
|
|
|
drop_parent_pte(child, sptep);
|
2018-12-06 21:21:09 +08:00
|
|
|
kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
|
2010-07-13 19:27:07 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-09-22 16:56:06 +08:00
|
|
|
static bool mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp,
|
2011-05-15 23:27:52 +08:00
|
|
|
u64 *spte)
|
|
|
|
{
|
|
|
|
u64 pte;
|
|
|
|
struct kvm_mmu_page *child;
|
|
|
|
|
|
|
|
pte = *spte;
|
|
|
|
if (is_shadow_present_pte(pte)) {
|
2011-09-22 16:56:06 +08:00
|
|
|
if (is_last_spte(pte, sp->role.level)) {
|
2011-07-12 03:28:04 +08:00
|
|
|
drop_spte(kvm, spte);
|
2011-09-22 16:56:06 +08:00
|
|
|
if (is_large_pte(pte))
|
|
|
|
--kvm->stat.lpages;
|
|
|
|
} else {
|
2020-06-23 04:20:34 +08:00
|
|
|
child = to_shadow_page(pte & PT64_BASE_ADDR_MASK);
|
2011-05-15 23:28:29 +08:00
|
|
|
drop_parent_pte(child, spte);
|
2011-05-15 23:27:52 +08:00
|
|
|
}
|
2011-09-22 16:56:06 +08:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (is_mmio_spte(pte))
|
2011-07-12 03:33:44 +08:00
|
|
|
mmu_spte_clear_no_track(spte);
|
2011-07-12 03:28:04 +08:00
|
|
|
|
2011-09-22 16:56:06 +08:00
|
|
|
return false;
|
2011-05-15 23:27:52 +08:00
|
|
|
}
|
|
|
|
|
2007-07-17 18:04:56 +08:00
|
|
|
static void kvm_mmu_page_unlink_children(struct kvm *kvm,
|
2007-11-21 21:28:32 +08:00
|
|
|
struct kvm_mmu_page *sp)
|
2007-01-06 08:36:45 +08:00
|
|
|
{
|
2007-01-06 08:36:46 +08:00
|
|
|
unsigned i;
|
|
|
|
|
2011-05-15 23:27:52 +08:00
|
|
|
for (i = 0; i < PT64_ENT_PER_PAGE; ++i)
|
|
|
|
mmu_page_zap_pte(kvm, sp, sp->spt + i);
|
2007-01-06 08:36:45 +08:00
|
|
|
}
|
|
|
|
|
2008-07-11 22:59:46 +08:00
|
|
|
static void kvm_mmu_unlink_parents(struct kvm *kvm, struct kvm_mmu_page *sp)
|
2007-01-06 08:36:45 +08:00
|
|
|
{
|
2012-03-21 22:50:34 +08:00
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
2007-01-06 08:36:45 +08:00
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
while ((sptep = rmap_get_first(&sp->parent_ptes, &iter)))
|
2012-03-21 22:50:34 +08:00
|
|
|
drop_parent_pte(sp, sptep);
|
2008-07-11 22:59:46 +08:00
|
|
|
}
|
|
|
|
|
2008-12-02 08:32:02 +08:00
|
|
|
static int mmu_zap_unsync_children(struct kvm *kvm,
|
2010-06-04 21:53:54 +08:00
|
|
|
struct kvm_mmu_page *parent,
|
|
|
|
struct list_head *invalid_list)
|
2008-09-24 00:18:39 +08:00
|
|
|
{
|
2008-12-02 08:32:02 +08:00
|
|
|
int i, zapped = 0;
|
|
|
|
struct mmu_page_path parents;
|
|
|
|
struct kvm_mmu_pages pages;
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (parent->role.level == PG_LEVEL_4K)
|
2008-09-24 00:18:39 +08:00
|
|
|
return 0;
|
2008-12-02 08:32:02 +08:00
|
|
|
|
|
|
|
while (mmu_unsync_walk(parent, &pages)) {
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
|
|
|
|
for_each_sp(pages, sp, parents, i) {
|
2010-06-04 21:53:54 +08:00
|
|
|
kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
|
2008-12-02 08:32:02 +08:00
|
|
|
mmu_pages_clear_parents(&parents);
|
2010-04-16 16:34:42 +08:00
|
|
|
zapped++;
|
2008-12-02 08:32:02 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return zapped;
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
|
|
|
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
static bool __kvm_mmu_prepare_zap_page(struct kvm *kvm,
|
|
|
|
struct kvm_mmu_page *sp,
|
|
|
|
struct list_head *invalid_list,
|
|
|
|
int *nr_zapped)
|
2008-07-11 22:59:46 +08:00
|
|
|
{
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
bool list_unstable;
|
2009-07-06 20:58:14 +08:00
|
|
|
|
2010-06-04 21:53:54 +08:00
|
|
|
trace_kvm_mmu_prepare_zap_page(sp);
|
2008-07-11 22:59:46 +08:00
|
|
|
++kvm->stat.mmu_shadow_zapped;
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
*nr_zapped = mmu_zap_unsync_children(kvm, sp, invalid_list);
|
2007-11-21 21:28:32 +08:00
|
|
|
kvm_mmu_page_unlink_children(kvm, sp);
|
2008-07-11 22:59:46 +08:00
|
|
|
kvm_mmu_unlink_parents(kvm, sp);
|
2013-05-31 08:36:22 +08:00
|
|
|
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
/* Zapping children means active_mmu_pages has become unstable. */
|
|
|
|
list_unstable = *nr_zapped;
|
|
|
|
|
2009-01-11 19:02:10 +08:00
|
|
|
if (!sp->role.invalid && !sp->role.direct)
|
2015-05-19 22:29:22 +08:00
|
|
|
unaccount_shadowed(kvm, sp);
|
2013-05-31 08:36:22 +08:00
|
|
|
|
2008-09-24 00:18:39 +08:00
|
|
|
if (sp->unsync)
|
|
|
|
kvm_unlink_unsync_page(kvm, sp);
|
2007-11-21 21:28:32 +08:00
|
|
|
if (!sp->root_count) {
|
2010-05-05 09:03:49 +08:00
|
|
|
/* Count self */
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
(*nr_zapped)++;
|
2020-06-24 03:35:39 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Already invalid pages (previously active roots) are not on
|
|
|
|
* the active page list. See list_del() in the "else" case of
|
|
|
|
* !sp->root_count.
|
|
|
|
*/
|
|
|
|
if (sp->role.invalid)
|
|
|
|
list_add(&sp->link, invalid_list);
|
|
|
|
else
|
|
|
|
list_move(&sp->link, invalid_list);
|
2011-07-12 03:26:40 +08:00
|
|
|
kvm_mod_used_mmu_pages(kvm, -1);
|
2008-02-21 03:47:24 +08:00
|
|
|
} else {
|
2020-06-24 03:35:39 +08:00
|
|
|
/*
|
|
|
|
* Remove the active root from the active page list, the root
|
|
|
|
* will be explicitly freed when the root_count hits zero.
|
|
|
|
*/
|
|
|
|
list_del(&sp->link);
|
2013-05-31 08:36:30 +08:00
|
|
|
|
2019-09-13 10:46:10 +08:00
|
|
|
/*
|
|
|
|
* Obsolete pages cannot be used on any vCPUs, see the comment
|
|
|
|
* in kvm_mmu_zap_all_fast(). Note, is_obsolete_sp() also
|
|
|
|
* treats invalid shadow pages as being obsolete.
|
|
|
|
*/
|
|
|
|
if (!is_obsolete_sp(kvm, sp))
|
2013-05-31 08:36:30 +08:00
|
|
|
kvm_reload_remote_mmus(kvm);
|
2008-02-21 03:47:24 +08:00
|
|
|
}
|
2010-06-04 21:53:54 +08:00
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
if (sp->lpage_disallowed)
|
|
|
|
unaccount_huge_nx_page(kvm, sp);
|
|
|
|
|
2010-06-04 21:53:54 +08:00
|
|
|
sp->role.invalid = 1;
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
return list_unstable;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,
|
|
|
|
struct list_head *invalid_list)
|
|
|
|
{
|
|
|
|
int nr_zapped;
|
|
|
|
|
|
|
|
__kvm_mmu_prepare_zap_page(kvm, sp, invalid_list, &nr_zapped);
|
|
|
|
return nr_zapped;
|
2007-01-06 08:36:45 +08:00
|
|
|
}
|
|
|
|
|
2010-06-04 21:53:54 +08:00
|
|
|
static void kvm_mmu_commit_zap_page(struct kvm *kvm,
|
|
|
|
struct list_head *invalid_list)
|
|
|
|
{
|
2013-03-06 15:05:52 +08:00
|
|
|
struct kvm_mmu_page *sp, *nsp;
|
2010-06-04 21:53:54 +08:00
|
|
|
|
|
|
|
if (list_empty(invalid_list))
|
|
|
|
return;
|
|
|
|
|
2012-05-14 20:44:06 +08:00
|
|
|
/*
|
2016-03-13 11:10:24 +08:00
|
|
|
* We need to make sure everyone sees our modifications to
|
|
|
|
* the page tables and see changes to vcpu->mode here. The barrier
|
|
|
|
* in the kvm_flush_remote_tlbs() achieves this. This pairs
|
|
|
|
* with vcpu_enter_guest and walk_shadow_page_lockless_begin/end.
|
|
|
|
*
|
|
|
|
* In addition, kvm_flush_remote_tlbs waits for all vcpus to exit
|
|
|
|
* guest mode and/or lockless shadow page table walks.
|
2012-05-14 20:44:06 +08:00
|
|
|
*/
|
|
|
|
kvm_flush_remote_tlbs(kvm);
|
2011-07-12 03:32:13 +08:00
|
|
|
|
2013-03-06 15:05:52 +08:00
|
|
|
list_for_each_entry_safe(sp, nsp, invalid_list, link) {
|
2010-06-04 21:53:54 +08:00
|
|
|
WARN_ON(!sp->role.invalid || sp->root_count);
|
2011-07-12 03:26:40 +08:00
|
|
|
kvm_mmu_free_page(sp);
|
2013-03-06 15:05:52 +08:00
|
|
|
}
|
2010-06-04 21:53:54 +08:00
|
|
|
}
|
|
|
|
|
2020-06-24 03:35:40 +08:00
|
|
|
static unsigned long kvm_mmu_zap_oldest_mmu_pages(struct kvm *kvm,
|
|
|
|
unsigned long nr_to_zap)
|
2019-12-07 07:57:15 +08:00
|
|
|
{
|
2020-06-24 03:35:40 +08:00
|
|
|
unsigned long total_zapped = 0;
|
|
|
|
struct kvm_mmu_page *sp, *tmp;
|
2019-12-07 07:57:15 +08:00
|
|
|
LIST_HEAD(invalid_list);
|
2020-06-24 03:35:40 +08:00
|
|
|
bool unstable;
|
|
|
|
int nr_zapped;
|
2019-12-07 07:57:15 +08:00
|
|
|
|
2020-06-24 03:35:40 +08:00
|
|
|
if (list_empty(&kvm->arch.active_mmu_pages))
|
2019-12-07 07:57:15 +08:00
|
|
|
return 0;
|
|
|
|
|
2020-06-24 03:35:40 +08:00
|
|
|
restart:
|
|
|
|
list_for_each_entry_safe(sp, tmp, &kvm->arch.active_mmu_pages, link) {
|
|
|
|
/*
|
|
|
|
* Don't zap active root pages, the page itself can't be freed
|
|
|
|
* and zapping it will just force vCPUs to realloc and reload.
|
|
|
|
*/
|
|
|
|
if (sp->root_count)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
unstable = __kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list,
|
|
|
|
&nr_zapped);
|
|
|
|
total_zapped += nr_zapped;
|
|
|
|
if (total_zapped >= nr_to_zap)
|
2019-12-07 07:57:15 +08:00
|
|
|
break;
|
|
|
|
|
2020-06-24 03:35:40 +08:00
|
|
|
if (unstable)
|
|
|
|
goto restart;
|
2019-12-07 07:57:15 +08:00
|
|
|
}
|
2020-06-24 03:35:40 +08:00
|
|
|
|
|
|
|
kvm_mmu_commit_zap_page(kvm, &invalid_list);
|
|
|
|
|
|
|
|
kvm->stat.mmu_recycled += total_zapped;
|
|
|
|
return total_zapped;
|
|
|
|
}
|
|
|
|
|
2020-06-23 04:20:30 +08:00
|
|
|
static inline unsigned long kvm_mmu_available_pages(struct kvm *kvm)
|
|
|
|
{
|
|
|
|
if (kvm->arch.n_max_mmu_pages > kvm->arch.n_used_mmu_pages)
|
|
|
|
return kvm->arch.n_max_mmu_pages -
|
|
|
|
kvm->arch.n_used_mmu_pages;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2020-06-24 03:35:40 +08:00
|
|
|
static int make_mmu_pages_available(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
|
|
|
unsigned long avail = kvm_mmu_available_pages(vcpu->kvm);
|
|
|
|
|
|
|
|
if (likely(avail >= KVM_MIN_FREE_MMU_PAGES))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
kvm_mmu_zap_oldest_mmu_pages(vcpu->kvm, KVM_REFILL_PAGES - avail);
|
2019-12-07 07:57:15 +08:00
|
|
|
|
|
|
|
if (!kvm_mmu_available_pages(vcpu->kvm))
|
|
|
|
return -ENOSPC;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2007-10-03 00:52:55 +08:00
|
|
|
/*
|
|
|
|
* Changing the number of mmu pages allocated to the vm
|
2010-08-20 09:11:28 +08:00
|
|
|
* Note: if goal_nr_mmu_pages is too small, you will get dead lock
|
2007-10-03 00:52:55 +08:00
|
|
|
*/
|
2019-04-09 02:07:30 +08:00
|
|
|
void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long goal_nr_mmu_pages)
|
2007-10-03 00:52:55 +08:00
|
|
|
{
|
2013-01-08 18:46:07 +08:00
|
|
|
spin_lock(&kvm->mmu_lock);
|
|
|
|
|
2010-08-20 09:11:28 +08:00
|
|
|
if (kvm->arch.n_used_mmu_pages > goal_nr_mmu_pages) {
|
2020-06-24 03:35:40 +08:00
|
|
|
kvm_mmu_zap_oldest_mmu_pages(kvm, kvm->arch.n_used_mmu_pages -
|
|
|
|
goal_nr_mmu_pages);
|
2007-10-03 00:52:55 +08:00
|
|
|
|
2010-08-20 09:11:28 +08:00
|
|
|
goal_nr_mmu_pages = kvm->arch.n_used_mmu_pages;
|
2007-10-03 00:52:55 +08:00
|
|
|
}
|
|
|
|
|
2010-08-20 09:11:28 +08:00
|
|
|
kvm->arch.n_max_mmu_pages = goal_nr_mmu_pages;
|
2013-01-08 18:46:07 +08:00
|
|
|
|
|
|
|
spin_unlock(&kvm->mmu_lock);
|
2007-10-03 00:52:55 +08:00
|
|
|
}
|
|
|
|
|
2011-09-22 17:02:48 +08:00
|
|
|
int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn)
|
2007-01-06 08:36:45 +08:00
|
|
|
{
|
2007-11-21 21:28:32 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2010-06-04 21:55:29 +08:00
|
|
|
LIST_HEAD(invalid_list);
|
2007-01-06 08:36:45 +08:00
|
|
|
int r;
|
|
|
|
|
2010-08-28 19:19:42 +08:00
|
|
|
pgprintk("%s: looking for gfn %llx\n", __func__, gfn);
|
2007-01-06 08:36:45 +08:00
|
|
|
r = 0;
|
2011-09-22 17:02:48 +08:00
|
|
|
spin_lock(&kvm->mmu_lock);
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-28 09:06:00 +08:00
|
|
|
for_each_gfn_indirect_valid_sp(kvm, sp, gfn) {
|
2010-08-28 19:19:42 +08:00
|
|
|
pgprintk("%s: gfn %llx role %x\n", __func__, gfn,
|
2010-06-04 21:53:07 +08:00
|
|
|
sp->role.word);
|
|
|
|
r = 1;
|
2010-06-04 21:56:11 +08:00
|
|
|
kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
|
2010-06-04 21:53:07 +08:00
|
|
|
}
|
2010-06-04 21:55:29 +08:00
|
|
|
kvm_mmu_commit_zap_page(kvm, &invalid_list);
|
2011-09-22 17:02:48 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
|
2007-01-06 08:36:45 +08:00
|
|
|
return r;
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
}
|
2011-09-22 17:02:48 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_unprotect_page);
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
|
2016-02-24 17:51:15 +08:00
|
|
|
static void kvm_unsync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
|
2010-05-24 15:40:07 +08:00
|
|
|
{
|
|
|
|
trace_kvm_mmu_unsync_page(sp);
|
|
|
|
++vcpu->kvm->stat.mmu_unsync;
|
|
|
|
sp->unsync = 1;
|
|
|
|
|
|
|
|
kvm_mmu_mark_parents_unsync(sp);
|
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:11 +08:00
|
|
|
static bool mmu_need_write_protect(struct kvm_vcpu *vcpu, gfn_t gfn,
|
|
|
|
bool can_unsync)
|
2008-09-24 00:18:39 +08:00
|
|
|
{
|
2016-02-24 17:51:15 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2008-09-24 00:18:39 +08:00
|
|
|
|
2016-02-24 17:51:11 +08:00
|
|
|
if (kvm_page_track_is_active(vcpu, gfn, KVM_PAGE_TRACK_WRITE))
|
|
|
|
return true;
|
2010-05-24 15:40:07 +08:00
|
|
|
|
2016-02-24 17:51:15 +08:00
|
|
|
for_each_gfn_indirect_valid_sp(vcpu->kvm, sp, gfn) {
|
KVM: MMU: fix writable sync sp mapping
While we sync many unsync sp at one time(in mmu_sync_children()),
we may mapping the spte writable, it's dangerous, if one unsync
sp's mapping gfn is another unsync page's gfn.
For example:
SP1.pte[0] = P
SP2.gfn's pfn = P
[SP1.pte[0] = SP2.gfn's pfn]
First, we write protected SP1 and SP2, but SP1 and SP2 are still the
unsync sp.
Then, sync SP1 first, it will detect SP1.pte[0].gfn only has one unsync-sp,
that is SP2, so it will mapping it writable, but we plan to sync SP2 soon,
at this point, the SP2->unsync is not reliable since later we sync SP2 but
SP2->gfn is already writable.
So the final result is: SP2 is the sync page but SP2.gfn is writable.
This bug will corrupt guest's page table, fixed by mark read-only mapping
if the mapped gfn has shadow pages.
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
2010-06-30 16:02:02 +08:00
|
|
|
if (!can_unsync)
|
2016-02-24 17:51:11 +08:00
|
|
|
return true;
|
KVM: MMU: fix writable sync sp mapping
While we sync many unsync sp at one time(in mmu_sync_children()),
we may mapping the spte writable, it's dangerous, if one unsync
sp's mapping gfn is another unsync page's gfn.
For example:
SP1.pte[0] = P
SP2.gfn's pfn = P
[SP1.pte[0] = SP2.gfn's pfn]
First, we write protected SP1 and SP2, but SP1 and SP2 are still the
unsync sp.
Then, sync SP1 first, it will detect SP1.pte[0].gfn only has one unsync-sp,
that is SP2, so it will mapping it writable, but we plan to sync SP2 soon,
at this point, the SP2->unsync is not reliable since later we sync SP2 but
SP2->gfn is already writable.
So the final result is: SP2 is the sync page but SP2.gfn is writable.
This bug will corrupt guest's page table, fixed by mark read-only mapping
if the mapped gfn has shadow pages.
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
2010-06-30 16:02:02 +08:00
|
|
|
|
2016-02-24 17:51:15 +08:00
|
|
|
if (sp->unsync)
|
|
|
|
continue;
|
2010-05-24 15:40:07 +08:00
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
WARN_ON(sp->role.level != PG_LEVEL_4K);
|
2016-02-24 17:51:15 +08:00
|
|
|
kvm_unsync_page(vcpu, sp);
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
2016-02-24 17:51:11 +08:00
|
|
|
|
2018-06-28 05:59:05 +08:00
|
|
|
/*
|
|
|
|
* We need to ensure that the marking of unsync pages is visible
|
|
|
|
* before the SPTE is updated to allow writes because
|
|
|
|
* kvm_mmu_sync_roots() checks the unsync flags without holding
|
|
|
|
* the MMU lock and so can race with this. If the SPTE was updated
|
|
|
|
* before the page had been marked as unsync-ed, something like the
|
|
|
|
* following could happen:
|
|
|
|
*
|
|
|
|
* CPU 1 CPU 2
|
|
|
|
* ---------------------------------------------------------------------
|
|
|
|
* 1.2 Host updates SPTE
|
|
|
|
* to be writable
|
|
|
|
* 2.1 Guest writes a GPTE for GVA X.
|
|
|
|
* (GPTE being in the guest page table shadowed
|
|
|
|
* by the SP from CPU 1.)
|
|
|
|
* This reads SPTE during the page table walk.
|
|
|
|
* Since SPTE.W is read as 1, there is no
|
|
|
|
* fault.
|
|
|
|
*
|
|
|
|
* 2.2 Guest issues TLB flush.
|
|
|
|
* That causes a VM Exit.
|
|
|
|
*
|
|
|
|
* 2.3 kvm_mmu_sync_pages() reads sp->unsync.
|
|
|
|
* Since it is false, so it just returns.
|
|
|
|
*
|
|
|
|
* 2.4 Guest accesses GVA X.
|
|
|
|
* Since the mapping in the SP was not updated,
|
|
|
|
* so the old mapping for GVA X incorrectly
|
|
|
|
* gets used.
|
|
|
|
* 1.1 Host marks SP
|
|
|
|
* as unsync
|
|
|
|
* (sp->unsync = true)
|
|
|
|
*
|
|
|
|
* The write barrier below ensures that 1.1 happens before 1.2 and thus
|
|
|
|
* the situation in 2.4 does not arise. The implicit barrier in 2.2
|
|
|
|
* pairs with this write barrier.
|
|
|
|
*/
|
|
|
|
smp_wmb();
|
|
|
|
|
2016-02-24 17:51:11 +08:00
|
|
|
return false;
|
2008-09-24 00:18:39 +08:00
|
|
|
}
|
|
|
|
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
static bool kvm_is_mmio_pfn(kvm_pfn_t pfn)
|
2015-07-07 21:03:18 +08:00
|
|
|
{
|
|
|
|
if (pfn_valid(pfn))
|
2017-12-20 15:29:29 +08:00
|
|
|
return !is_zero_pfn(pfn) && PageReserved(pfn_to_page(pfn)) &&
|
|
|
|
/*
|
|
|
|
* Some reserved pages, such as those from NVDIMM
|
|
|
|
* DAX devices, are not for MMIO, and can be mapped
|
|
|
|
* with cached memory type for better performance.
|
|
|
|
* However, the above check misconceives those pages
|
|
|
|
* as MMIO, and results in KVM mapping them with UC
|
|
|
|
* memory type, which would hurt the performance.
|
|
|
|
* Therefore, we check the host memory type in addition
|
|
|
|
* and only treat UC/UC-/WC pages as MMIO.
|
|
|
|
*/
|
|
|
|
(!pat_enabled() || pat_pfn_immune_to_uc_mtrr(pfn));
|
2015-07-07 21:03:18 +08:00
|
|
|
|
2019-02-01 04:24:44 +08:00
|
|
|
return !e820__mapped_raw_any(pfn_to_hpa(pfn),
|
|
|
|
pfn_to_hpa(pfn + 1) - 1,
|
|
|
|
E820_TYPE_RAM);
|
2015-07-07 21:03:18 +08:00
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:04 +08:00
|
|
|
/* Bits which may be returned by set_spte() */
|
|
|
|
#define SET_SPTE_WRITE_PROTECTED_PT BIT(0)
|
|
|
|
#define SET_SPTE_NEED_REMOTE_TLB_FLUSH BIT(1)
|
|
|
|
|
2009-06-10 19:24:23 +08:00
|
|
|
static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep,
|
2020-02-04 07:09:09 +08:00
|
|
|
unsigned int pte_access, int level,
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
gfn_t gfn, kvm_pfn_t pfn, bool speculative,
|
2010-11-19 17:03:22 +08:00
|
|
|
bool can_unsync, bool host_writable)
|
2007-12-09 23:40:31 +08:00
|
|
|
{
|
2016-07-13 06:18:49 +08:00
|
|
|
u64 spte = 0;
|
2008-09-24 00:18:30 +08:00
|
|
|
int ret = 0;
|
2017-07-01 08:26:31 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2008-10-09 16:01:57 +08:00
|
|
|
|
2015-04-08 21:39:23 +08:00
|
|
|
if (set_mmio_spte(vcpu, sptep, gfn, pfn, pte_access))
|
2011-07-12 03:33:44 +08:00
|
|
|
return 0;
|
|
|
|
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(sptep);
|
2017-07-01 08:26:31 +08:00
|
|
|
if (sp_ad_disabled(sp))
|
2019-09-24 18:43:08 +08:00
|
|
|
spte |= SPTE_AD_DISABLED_MASK;
|
2019-09-27 00:47:59 +08:00
|
|
|
else if (kvm_vcpu_ad_need_write_protect(vcpu))
|
|
|
|
spte |= SPTE_AD_WRPROT_ONLY_MASK;
|
2017-07-01 08:26:31 +08:00
|
|
|
|
2016-07-13 06:18:51 +08:00
|
|
|
/*
|
|
|
|
* For the EPT case, shadow_present_mask is 0 if hardware
|
|
|
|
* supports exec-only page table entries. In that case,
|
|
|
|
* ACC_USER_MASK and shadow_user_mask are used to represent
|
|
|
|
* read access. See FNAME(gpte_access) in paging_tmpl.h.
|
|
|
|
*/
|
2016-07-13 06:18:49 +08:00
|
|
|
spte |= shadow_present_mask;
|
2008-03-18 17:05:52 +08:00
|
|
|
if (!speculative)
|
2017-07-01 08:26:31 +08:00
|
|
|
spte |= spte_shadow_accessed_mask(spte);
|
2011-07-12 03:24:39 +08:00
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level > PG_LEVEL_4K && (pte_access & ACC_EXEC_MASK) &&
|
2019-11-04 19:22:02 +08:00
|
|
|
is_nx_huge_page_enabled()) {
|
|
|
|
pte_access &= ~ACC_EXEC_MASK;
|
|
|
|
}
|
|
|
|
|
2008-04-25 21:13:50 +08:00
|
|
|
if (pte_access & ACC_EXEC_MASK)
|
|
|
|
spte |= shadow_x_mask;
|
|
|
|
else
|
|
|
|
spte |= shadow_nx_mask;
|
2012-06-20 15:58:58 +08:00
|
|
|
|
2007-12-09 23:40:31 +08:00
|
|
|
if (pte_access & ACC_USER_MASK)
|
2008-04-25 21:13:50 +08:00
|
|
|
spte |= shadow_user_mask;
|
2012-06-20 15:58:58 +08:00
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level > PG_LEVEL_4K)
|
2008-02-23 22:44:30 +08:00
|
|
|
spte |= PT_PAGE_SIZE_MASK;
|
2010-09-13 22:45:28 +08:00
|
|
|
if (tdp_enabled)
|
2020-03-22 04:26:00 +08:00
|
|
|
spte |= kvm_x86_ops.get_mt_mask(vcpu, gfn,
|
2015-07-07 21:03:18 +08:00
|
|
|
kvm_is_mmio_pfn(pfn));
|
2007-12-09 23:40:31 +08:00
|
|
|
|
2010-11-19 17:03:22 +08:00
|
|
|
if (host_writable)
|
2009-09-24 02:47:17 +08:00
|
|
|
spte |= SPTE_HOST_WRITEABLE;
|
2010-12-23 16:09:29 +08:00
|
|
|
else
|
|
|
|
pte_access &= ~ACC_WRITE_MASK;
|
2009-09-24 02:47:17 +08:00
|
|
|
|
2018-03-09 07:17:31 +08:00
|
|
|
if (!kvm_is_mmio_pfn(pfn))
|
|
|
|
spte |= shadow_me_mask;
|
|
|
|
|
2008-04-03 03:46:56 +08:00
|
|
|
spte |= (u64)pfn << PAGE_SHIFT;
|
2007-12-09 23:40:31 +08:00
|
|
|
|
2013-01-08 14:36:04 +08:00
|
|
|
if (pte_access & ACC_WRITE_MASK) {
|
2012-06-20 15:58:58 +08:00
|
|
|
spte |= PT_WRITABLE_MASK | SPTE_MMU_WRITEABLE;
|
2007-12-09 23:40:31 +08:00
|
|
|
|
2008-11-25 22:58:07 +08:00
|
|
|
/*
|
|
|
|
* Optimization: for pte sync, if spte was writable the hash
|
|
|
|
* lookup is unnecessary (and expensive). Write protection
|
|
|
|
* is responsibility of mmu_get_page / kvm_sync_page.
|
|
|
|
* Same reasoning can be applied to dirty page accounting.
|
|
|
|
*/
|
2010-01-18 17:45:10 +08:00
|
|
|
if (!can_unsync && is_writable_pte(*sptep))
|
2008-11-25 22:58:07 +08:00
|
|
|
goto set_pte;
|
|
|
|
|
2008-09-24 00:18:39 +08:00
|
|
|
if (mmu_need_write_protect(vcpu, gfn, can_unsync)) {
|
2010-08-28 19:19:42 +08:00
|
|
|
pgprintk("%s: found shadow page for %llx, marking ro\n",
|
2008-03-04 04:59:56 +08:00
|
|
|
__func__, gfn);
|
2018-06-28 05:59:04 +08:00
|
|
|
ret |= SET_SPTE_WRITE_PROTECTED_PT;
|
2007-12-09 23:40:31 +08:00
|
|
|
pte_access &= ~ACC_WRITE_MASK;
|
2012-06-20 15:58:58 +08:00
|
|
|
spte &= ~(PT_WRITABLE_MASK | SPTE_MMU_WRITEABLE);
|
2007-12-09 23:40:31 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-01-28 10:54:25 +08:00
|
|
|
if (pte_access & ACC_WRITE_MASK) {
|
2015-04-08 21:39:23 +08:00
|
|
|
kvm_vcpu_mark_page_dirty(vcpu, gfn);
|
2017-07-01 08:26:31 +08:00
|
|
|
spte |= spte_shadow_dirty_mask(spte);
|
2015-01-28 10:54:25 +08:00
|
|
|
}
|
2007-12-09 23:40:31 +08:00
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
if (speculative)
|
|
|
|
spte = mark_spte_for_access_track(spte);
|
|
|
|
|
2008-09-24 00:18:32 +08:00
|
|
|
set_pte:
|
2012-06-20 15:58:33 +08:00
|
|
|
if (mmu_spte_update(sptep, spte))
|
2018-06-28 05:59:04 +08:00
|
|
|
ret |= SET_SPTE_NEED_REMOTE_TLB_FLUSH;
|
2008-09-24 00:18:30 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2020-02-04 07:09:09 +08:00
|
|
|
static int mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep,
|
|
|
|
unsigned int pte_access, int write_fault, int level,
|
|
|
|
gfn_t gfn, kvm_pfn_t pfn, bool speculative,
|
|
|
|
bool host_writable)
|
2008-09-24 00:18:30 +08:00
|
|
|
{
|
|
|
|
int was_rmapped = 0;
|
2009-08-06 02:43:58 +08:00
|
|
|
int rmap_count;
|
2018-06-28 05:59:04 +08:00
|
|
|
int set_spte_ret;
|
2017-08-17 21:03:32 +08:00
|
|
|
int ret = RET_PF_RETRY;
|
2018-07-24 16:17:07 +08:00
|
|
|
bool flush = false;
|
2008-09-24 00:18:30 +08:00
|
|
|
|
2013-02-05 15:27:27 +08:00
|
|
|
pgprintk("%s: spte %llx write_fault %d gfn %llx\n", __func__,
|
|
|
|
*sptep, write_fault, gfn);
|
2008-09-24 00:18:30 +08:00
|
|
|
|
2015-11-20 16:44:55 +08:00
|
|
|
if (is_shadow_present_pte(*sptep)) {
|
2008-09-24 00:18:30 +08:00
|
|
|
/*
|
|
|
|
* If we overwrite a PTE page pointer with a 2MB PMD, unlink
|
|
|
|
* the parent of the now unreachable PTE.
|
|
|
|
*/
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level > PG_LEVEL_4K && !is_large_pte(*sptep)) {
|
2008-09-24 00:18:30 +08:00
|
|
|
struct kvm_mmu_page *child;
|
2009-06-10 19:24:23 +08:00
|
|
|
u64 pte = *sptep;
|
2008-09-24 00:18:30 +08:00
|
|
|
|
2020-06-23 04:20:34 +08:00
|
|
|
child = to_shadow_page(pte & PT64_BASE_ADDR_MASK);
|
2011-05-15 23:28:29 +08:00
|
|
|
drop_parent_pte(child, sptep);
|
2018-07-24 16:17:07 +08:00
|
|
|
flush = true;
|
2009-06-10 19:24:23 +08:00
|
|
|
} else if (pfn != spte_to_pfn(*sptep)) {
|
2010-08-28 19:19:42 +08:00
|
|
|
pgprintk("hfn old %llx new %llx\n",
|
2009-06-10 19:24:23 +08:00
|
|
|
spte_to_pfn(*sptep), pfn);
|
2011-07-12 03:28:04 +08:00
|
|
|
drop_spte(vcpu->kvm, sptep);
|
2018-07-24 16:17:07 +08:00
|
|
|
flush = true;
|
2009-02-18 21:08:59 +08:00
|
|
|
} else
|
|
|
|
was_rmapped = 1;
|
2008-09-24 00:18:30 +08:00
|
|
|
}
|
2009-07-27 22:30:44 +08:00
|
|
|
|
2018-06-28 05:59:04 +08:00
|
|
|
set_spte_ret = set_spte(vcpu, sptep, pte_access, level, gfn, pfn,
|
|
|
|
speculative, true, host_writable);
|
|
|
|
if (set_spte_ret & SET_SPTE_WRITE_PROTECTED_PT) {
|
2008-09-24 00:18:30 +08:00
|
|
|
if (write_fault)
|
2017-08-17 21:03:32 +08:00
|
|
|
ret = RET_PF_EMULATE;
|
2020-03-21 05:28:21 +08:00
|
|
|
kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
|
2008-09-24 00:18:31 +08:00
|
|
|
}
|
2018-12-06 21:21:09 +08:00
|
|
|
|
2018-07-24 16:17:07 +08:00
|
|
|
if (set_spte_ret & SET_SPTE_NEED_REMOTE_TLB_FLUSH || flush)
|
2018-12-06 21:21:09 +08:00
|
|
|
kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn,
|
|
|
|
KVM_PAGES_PER_HPAGE(level));
|
2008-09-24 00:18:30 +08:00
|
|
|
|
2015-11-20 16:44:05 +08:00
|
|
|
if (unlikely(is_mmio_spte(*sptep)))
|
2017-08-17 21:03:32 +08:00
|
|
|
ret = RET_PF_EMULATE;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2009-06-10 19:24:23 +08:00
|
|
|
pgprintk("%s: setting spte %llx\n", __func__, *sptep);
|
2019-07-01 18:22:57 +08:00
|
|
|
trace_kvm_mmu_set_spte(level, gfn, sptep);
|
2009-06-10 19:24:23 +08:00
|
|
|
if (!was_rmapped && is_large_pte(*sptep))
|
2008-02-23 22:44:30 +08:00
|
|
|
++vcpu->kvm->stat.lpages;
|
|
|
|
|
2011-07-12 03:22:01 +08:00
|
|
|
if (is_shadow_present_pte(*sptep)) {
|
|
|
|
if (!was_rmapped) {
|
|
|
|
rmap_count = rmap_add(vcpu, sptep, gfn);
|
|
|
|
if (rmap_count > RMAP_RECYCLE_THRESHOLD)
|
|
|
|
rmap_recycle(vcpu, sptep, gfn);
|
|
|
|
}
|
2007-12-09 23:40:31 +08:00
|
|
|
}
|
2012-08-03 15:42:10 +08:00
|
|
|
|
2017-08-17 21:03:32 +08:00
|
|
|
return ret;
|
2007-12-09 23:40:31 +08:00
|
|
|
}
|
|
|
|
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
static kvm_pfn_t pte_prefetch_gfn_to_pfn(struct kvm_vcpu *vcpu, gfn_t gfn,
|
2010-08-22 19:12:48 +08:00
|
|
|
bool no_dirty_log)
|
|
|
|
{
|
|
|
|
struct kvm_memory_slot *slot;
|
|
|
|
|
2011-03-09 15:43:00 +08:00
|
|
|
slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, no_dirty_log);
|
2012-07-17 21:54:11 +08:00
|
|
|
if (!slot)
|
2012-08-03 15:37:54 +08:00
|
|
|
return KVM_PFN_ERR_FAULT;
|
2010-08-22 19:12:48 +08:00
|
|
|
|
2012-08-21 10:59:12 +08:00
|
|
|
return gfn_to_pfn_memslot_atomic(slot, gfn);
|
2010-08-22 19:12:48 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu_page *sp,
|
|
|
|
u64 *start, u64 *end)
|
|
|
|
{
|
|
|
|
struct page *pages[PTE_PREFETCH_NUM];
|
2015-05-19 22:01:50 +08:00
|
|
|
struct kvm_memory_slot *slot;
|
2020-02-04 07:09:09 +08:00
|
|
|
unsigned int access = sp->role.access;
|
2010-08-22 19:12:48 +08:00
|
|
|
int i, ret;
|
|
|
|
gfn_t gfn;
|
|
|
|
|
|
|
|
gfn = kvm_mmu_page_get_gfn(sp, start - sp->spt);
|
2015-05-19 22:01:50 +08:00
|
|
|
slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, access & ACC_WRITE_MASK);
|
|
|
|
if (!slot)
|
2010-08-22 19:12:48 +08:00
|
|
|
return -1;
|
|
|
|
|
2015-05-19 22:01:50 +08:00
|
|
|
ret = gfn_to_page_many_atomic(slot, gfn, pages, end - start);
|
2010-08-22 19:12:48 +08:00
|
|
|
if (ret <= 0)
|
|
|
|
return -1;
|
|
|
|
|
2019-01-04 08:22:21 +08:00
|
|
|
for (i = 0; i < ret; i++, gfn++, start++) {
|
2015-11-20 16:44:05 +08:00
|
|
|
mmu_set_spte(vcpu, start, access, 0, sp->role.level, gfn,
|
|
|
|
page_to_pfn(pages[i]), true, true);
|
2019-01-04 08:22:21 +08:00
|
|
|
put_page(pages[i]);
|
|
|
|
}
|
2010-08-22 19:12:48 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu_page *sp, u64 *sptep)
|
|
|
|
{
|
|
|
|
u64 *spte, *start = NULL;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
WARN_ON(!sp->role.direct);
|
|
|
|
|
|
|
|
i = (sptep - sp->spt) & ~(PTE_PREFETCH_NUM - 1);
|
|
|
|
spte = sp->spt + i;
|
|
|
|
|
|
|
|
for (i = 0; i < PTE_PREFETCH_NUM; i++, spte++) {
|
2011-07-12 03:28:04 +08:00
|
|
|
if (is_shadow_present_pte(*spte) || spte == sptep) {
|
2010-08-22 19:12:48 +08:00
|
|
|
if (!start)
|
|
|
|
continue;
|
|
|
|
if (direct_pte_prefetch_many(vcpu, sp, start, spte) < 0)
|
|
|
|
break;
|
|
|
|
start = NULL;
|
|
|
|
} else if (!start)
|
|
|
|
start = spte;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
|
|
|
|
{
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(sptep);
|
2017-07-01 08:26:31 +08:00
|
|
|
|
2010-08-22 19:12:48 +08:00
|
|
|
/*
|
2017-07-01 08:26:31 +08:00
|
|
|
* Without accessed bits, there's no way to distinguish between
|
|
|
|
* actually accessed translations and prefetched, so disable pte
|
|
|
|
* prefetch if accessed bits aren't available.
|
2010-08-22 19:12:48 +08:00
|
|
|
*/
|
2017-07-01 08:26:31 +08:00
|
|
|
if (sp_ad_disabled(sp))
|
2010-08-22 19:12:48 +08:00
|
|
|
return;
|
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (sp->role.level > PG_LEVEL_4K)
|
2010-08-22 19:12:48 +08:00
|
|
|
return;
|
|
|
|
|
|
|
|
__direct_pte_prefetch(vcpu, sp, sptep);
|
|
|
|
}
|
|
|
|
|
2020-01-09 04:24:41 +08:00
|
|
|
static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn,
|
2020-01-09 04:24:46 +08:00
|
|
|
kvm_pfn_t pfn, struct kvm_memory_slot *slot)
|
2020-01-09 04:24:41 +08:00
|
|
|
{
|
|
|
|
unsigned long hva;
|
|
|
|
pte_t *pte;
|
|
|
|
int level;
|
|
|
|
|
2020-01-09 04:24:48 +08:00
|
|
|
if (!PageCompound(pfn_to_page(pfn)) && !kvm_is_zone_device_pfn(pfn))
|
2020-04-28 08:54:22 +08:00
|
|
|
return PG_LEVEL_4K;
|
2020-01-09 04:24:41 +08:00
|
|
|
|
2020-01-09 04:24:46 +08:00
|
|
|
/*
|
|
|
|
* Note, using the already-retrieved memslot and __gfn_to_hva_memslot()
|
|
|
|
* is not solely for performance, it's also necessary to avoid the
|
|
|
|
* "writable" check in __gfn_to_hva_many(), which will always fail on
|
|
|
|
* read-only memslots due to gfn_to_hva() assuming writes. Earlier
|
|
|
|
* page fault steps have already verified the guest isn't writing a
|
|
|
|
* read-only memslot.
|
|
|
|
*/
|
2020-01-09 04:24:41 +08:00
|
|
|
hva = __gfn_to_hva_memslot(slot, gfn);
|
|
|
|
|
|
|
|
pte = lookup_address_in_mm(vcpu->kvm->mm, hva, &level);
|
|
|
|
if (unlikely(!pte))
|
2020-04-28 08:54:22 +08:00
|
|
|
return PG_LEVEL_4K;
|
2020-01-09 04:24:41 +08:00
|
|
|
|
|
|
|
return level;
|
|
|
|
}
|
|
|
|
|
2020-01-09 04:24:43 +08:00
|
|
|
static int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn,
|
|
|
|
int max_level, kvm_pfn_t *pfnp)
|
2019-12-07 07:57:25 +08:00
|
|
|
{
|
2020-01-09 04:24:46 +08:00
|
|
|
struct kvm_memory_slot *slot;
|
2020-01-09 04:24:47 +08:00
|
|
|
struct kvm_lpage_info *linfo;
|
2019-12-07 07:57:25 +08:00
|
|
|
kvm_pfn_t pfn = *pfnp;
|
2020-01-09 04:24:40 +08:00
|
|
|
kvm_pfn_t mask;
|
2020-01-09 04:24:43 +08:00
|
|
|
int level;
|
2020-01-09 04:24:40 +08:00
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (unlikely(max_level == PG_LEVEL_4K))
|
|
|
|
return PG_LEVEL_4K;
|
2020-01-09 04:24:40 +08:00
|
|
|
|
2020-01-09 04:24:48 +08:00
|
|
|
if (is_error_noslot_pfn(pfn) || kvm_is_reserved_pfn(pfn))
|
2020-04-28 08:54:22 +08:00
|
|
|
return PG_LEVEL_4K;
|
2020-01-09 04:24:40 +08:00
|
|
|
|
2020-01-09 04:24:46 +08:00
|
|
|
slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, true);
|
|
|
|
if (!slot)
|
2020-04-28 08:54:22 +08:00
|
|
|
return PG_LEVEL_4K;
|
2020-01-09 04:24:46 +08:00
|
|
|
|
2020-03-03 07:57:03 +08:00
|
|
|
max_level = min(max_level, max_page_level);
|
2020-04-28 08:54:22 +08:00
|
|
|
for ( ; max_level > PG_LEVEL_4K; max_level--) {
|
2020-01-09 04:24:47 +08:00
|
|
|
linfo = lpage_info_slot(gfn, slot, max_level);
|
|
|
|
if (!linfo->disallow_lpage)
|
2020-01-09 04:24:46 +08:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (max_level == PG_LEVEL_4K)
|
|
|
|
return PG_LEVEL_4K;
|
2020-01-09 04:24:46 +08:00
|
|
|
|
|
|
|
level = host_pfn_mapping_level(vcpu, gfn, pfn, slot);
|
2020-04-28 08:54:22 +08:00
|
|
|
if (level == PG_LEVEL_4K)
|
2020-01-09 04:24:43 +08:00
|
|
|
return level;
|
2020-01-09 04:24:40 +08:00
|
|
|
|
2020-01-09 04:24:41 +08:00
|
|
|
level = min(level, max_level);
|
2019-12-07 07:57:25 +08:00
|
|
|
|
|
|
|
/*
|
2020-01-09 04:24:40 +08:00
|
|
|
* mmu_notifier_retry() was successful and mmu_lock is held, so
|
|
|
|
* the pmd can't be split from under us.
|
2019-12-07 07:57:25 +08:00
|
|
|
*/
|
2020-01-09 04:24:40 +08:00
|
|
|
mask = KVM_PAGES_PER_HPAGE(level) - 1;
|
|
|
|
VM_BUG_ON((gfn & mask) != (pfn & mask));
|
|
|
|
*pfnp = pfn & ~mask;
|
2020-01-09 04:24:43 +08:00
|
|
|
|
|
|
|
return level;
|
2019-12-07 07:57:25 +08:00
|
|
|
}
|
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
static void disallowed_hugepage_adjust(struct kvm_shadow_walk_iterator it,
|
|
|
|
gfn_t gfn, kvm_pfn_t *pfnp, int *levelp)
|
|
|
|
{
|
|
|
|
int level = *levelp;
|
|
|
|
u64 spte = *it.sptep;
|
|
|
|
|
2020-04-28 08:54:22 +08:00
|
|
|
if (it.level == level && level > PG_LEVEL_4K &&
|
2019-11-04 19:22:02 +08:00
|
|
|
is_nx_huge_page_enabled() &&
|
|
|
|
is_shadow_present_pte(spte) &&
|
|
|
|
!is_large_pte(spte)) {
|
|
|
|
/*
|
|
|
|
* A small SPTE exists for this pfn, but FNAME(fetch)
|
|
|
|
* and __direct_map would like to create a large PTE
|
|
|
|
* instead: just force them to go down another level,
|
|
|
|
* patching back for them into pfn the next 9 bits of
|
|
|
|
* the address.
|
|
|
|
*/
|
|
|
|
u64 page_mask = KVM_PAGES_PER_HPAGE(level) - KVM_PAGES_PER_HPAGE(level - 1);
|
|
|
|
*pfnp |= gfn & page_mask;
|
|
|
|
(*levelp)--;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-06-24 19:06:21 +08:00
|
|
|
static int __direct_map(struct kvm_vcpu *vcpu, gpa_t gpa, int write,
|
2020-01-09 04:24:43 +08:00
|
|
|
int map_writable, int max_level, kvm_pfn_t pfn,
|
|
|
|
bool prefault, bool account_disallowed_nx_lpage)
|
2008-08-23 00:28:04 +08:00
|
|
|
{
|
2019-06-24 19:06:21 +08:00
|
|
|
struct kvm_shadow_walk_iterator it;
|
2008-08-23 00:28:04 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2020-01-09 04:24:43 +08:00
|
|
|
int level, ret;
|
2019-06-24 19:06:21 +08:00
|
|
|
gfn_t gfn = gpa >> PAGE_SHIFT;
|
|
|
|
gfn_t base_gfn = gfn;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2019-12-07 07:57:28 +08:00
|
|
|
if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root_hpa)))
|
2019-06-24 19:06:21 +08:00
|
|
|
return RET_PF_RETRY;
|
2013-12-20 01:28:51 +08:00
|
|
|
|
2020-01-09 04:24:43 +08:00
|
|
|
level = kvm_mmu_hugepage_adjust(vcpu, gfn, max_level, &pfn);
|
2019-12-07 07:57:26 +08:00
|
|
|
|
2019-07-01 18:22:57 +08:00
|
|
|
trace_kvm_mmu_spte_requested(gpa, level, pfn);
|
2019-06-24 19:06:21 +08:00
|
|
|
for_each_shadow_entry(vcpu, gpa, it) {
|
2019-11-04 19:22:02 +08:00
|
|
|
/*
|
|
|
|
* We cannot overwrite existing page tables with an NX
|
|
|
|
* large page, as the leaf could be executable.
|
|
|
|
*/
|
|
|
|
disallowed_hugepage_adjust(it, gfn, &pfn, &level);
|
|
|
|
|
2019-06-24 19:06:21 +08:00
|
|
|
base_gfn = gfn & ~(KVM_PAGES_PER_HPAGE(it.level) - 1);
|
|
|
|
if (it.level == level)
|
2008-12-25 20:54:25 +08:00
|
|
|
break;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2019-06-24 19:06:21 +08:00
|
|
|
drop_large_spte(vcpu, it.sptep);
|
|
|
|
if (!is_shadow_present_pte(*it.sptep)) {
|
|
|
|
sp = kvm_mmu_get_page(vcpu, base_gfn, it.addr,
|
|
|
|
it.level - 1, true, ACC_ALL);
|
2010-05-26 16:48:25 +08:00
|
|
|
|
2019-06-24 19:06:21 +08:00
|
|
|
link_shadow_page(vcpu, it.sptep, sp);
|
2019-12-07 07:57:23 +08:00
|
|
|
if (account_disallowed_nx_lpage)
|
2019-11-04 19:22:02 +08:00
|
|
|
account_huge_nx_page(vcpu->kvm, sp);
|
2008-12-25 20:54:25 +08:00
|
|
|
}
|
|
|
|
}
|
2019-06-24 19:06:21 +08:00
|
|
|
|
|
|
|
ret = mmu_set_spte(vcpu, it.sptep, ACC_ALL,
|
|
|
|
write, level, base_gfn, pfn, prefault,
|
|
|
|
map_writable);
|
|
|
|
direct_pte_prefetch(vcpu, it.sptep);
|
|
|
|
++vcpu->stat.pf_fixed;
|
|
|
|
return ret;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2010-10-08 16:24:15 +08:00
|
|
|
static void kvm_send_hwpoison_signal(unsigned long address, struct task_struct *tsk)
|
2010-05-31 14:28:19 +08:00
|
|
|
{
|
2018-04-17 03:23:27 +08:00
|
|
|
send_sig_mceerr(BUS_MCEERR_AR, (void __user *)address, PAGE_SHIFT, tsk);
|
2010-05-31 14:28:19 +08:00
|
|
|
}
|
|
|
|
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
|
2010-05-31 14:28:19 +08:00
|
|
|
{
|
2012-08-21 11:02:51 +08:00
|
|
|
/*
|
|
|
|
* Do not cache the mmio info caused by writing the readonly gfn
|
|
|
|
* into the spte otherwise read access on readonly gfn also can
|
|
|
|
* caused mmio page fault and treat it as mmio access.
|
|
|
|
*/
|
|
|
|
if (pfn == KVM_PFN_ERR_RO_FAULT)
|
2017-08-17 21:03:32 +08:00
|
|
|
return RET_PF_EMULATE;
|
2012-08-21 11:02:51 +08:00
|
|
|
|
2012-08-03 15:38:36 +08:00
|
|
|
if (pfn == KVM_PFN_ERR_HWPOISON) {
|
2015-04-08 21:39:23 +08:00
|
|
|
kvm_send_hwpoison_signal(kvm_vcpu_gfn_to_hva(vcpu, gfn), current);
|
2017-08-17 21:03:32 +08:00
|
|
|
return RET_PF_RETRY;
|
2011-07-12 03:29:38 +08:00
|
|
|
}
|
2010-07-08 01:16:45 +08:00
|
|
|
|
Revert "KVM: X86: Fix SMRAM accessing even if VM is shutdown"
The bug that led to commit 95e057e25892eaa48cad1e2d637b80d0f1a4fac5
was a benign warning (no adverse affects other than the warning
itself) that was detected by syzkaller. Further inspection shows
that the WARN_ON in question, in handle_ept_misconfig(), is
unnecessary and flawed (this was also briefly discussed in the
original patch: https://patchwork.kernel.org/patch/10204649).
* The WARN_ON is unnecessary as kvm_mmu_page_fault() will WARN
if reserved bits are set in the SPTEs, i.e. it covers the case
where an EPT misconfig occurred because of a KVM bug.
* The WARN_ON is flawed because it will fire on any system error
code that is hit while handling the fault, e.g. -ENOMEM can be
returned by mmu_topup_memory_caches() while handling a legitmate
MMIO EPT misconfig.
The original behavior of returning -EFAULT when userspace munmaps
an HVA without first removing the memslot is correct and desirable,
i.e. KVM is letting userspace know it has generated a bad address.
Returning RET_PF_EMULATE masks the WARN_ON in the EPT misconfig path,
but does not fix the underlying bug, i.e. the WARN_ON is bogus.
Furthermore, returning RET_PF_EMULATE has the unwanted side effect of
causing KVM to attempt to emulate an instruction on any page fault
with an invalid HVA translation, e.g. a not-present EPT violation
on a VM_PFNMAP VMA whose fault handler failed to insert a PFN.
* There is no guarantee that the fault is directly related to the
instruction, i.e. the fault could have been triggered by a side
effect memory access in the guest, e.g. while vectoring a #DB or
writing a tracing record. This could cause KVM to effectively
mask the fault if KVM doesn't model the behavior leading to the
fault, i.e. emulation could succeed and resume the guest.
* If emulation does fail, KVM will return EMULATION_FAILED instead
of -EFAULT, which is a red herring as the user will either debug
a bogus emulation attempt or scratch their head wondering why we
were attempting emulation in the first place.
TL;DR: revert to returning -EFAULT and remove the bogus WARN_ON in
handle_ept_misconfig in a future patch.
This reverts commit 95e057e25892eaa48cad1e2d637b80d0f1a4fac5.
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-03-30 05:48:30 +08:00
|
|
|
return -EFAULT;
|
2010-05-31 14:28:19 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:29:38 +08:00
|
|
|
static bool handle_abnormal_pfn(struct kvm_vcpu *vcpu, gva_t gva, gfn_t gfn,
|
2020-02-04 07:09:09 +08:00
|
|
|
kvm_pfn_t pfn, unsigned int access,
|
|
|
|
int *ret_val)
|
2011-07-12 03:29:38 +08:00
|
|
|
{
|
|
|
|
/* The pfn is invalid, report the error! */
|
2012-10-16 20:10:59 +08:00
|
|
|
if (unlikely(is_error_pfn(pfn))) {
|
2011-07-12 03:29:38 +08:00
|
|
|
*ret_val = kvm_handle_bad_page(vcpu, gfn, pfn);
|
2016-02-23 22:28:51 +08:00
|
|
|
return true;
|
2011-07-12 03:29:38 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:33:44 +08:00
|
|
|
if (unlikely(is_noslot_pfn(pfn)))
|
2019-08-02 04:35:22 +08:00
|
|
|
vcpu_cache_mmio_info(vcpu, gva, gfn,
|
|
|
|
access & shadow_mmio_access_mask);
|
2011-07-12 03:29:38 +08:00
|
|
|
|
2016-02-23 22:28:51 +08:00
|
|
|
return false;
|
2011-07-12 03:29:38 +08:00
|
|
|
}
|
|
|
|
|
2013-07-30 21:01:59 +08:00
|
|
|
static bool page_fault_can_be_fast(u32 error_code)
|
2012-06-20 15:59:18 +08:00
|
|
|
{
|
2013-07-18 12:52:37 +08:00
|
|
|
/*
|
|
|
|
* Do not fix the mmio spte with invalid generation number which
|
|
|
|
* need to be updated by slow page fault path.
|
|
|
|
*/
|
|
|
|
if (unlikely(error_code & PFERR_RSVD_MASK))
|
|
|
|
return false;
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
/* See if the page fault is due to an NX violation */
|
|
|
|
if (unlikely(((error_code & (PFERR_FETCH_MASK | PFERR_PRESENT_MASK))
|
|
|
|
== (PFERR_FETCH_MASK | PFERR_PRESENT_MASK))))
|
|
|
|
return false;
|
|
|
|
|
2012-06-20 15:59:18 +08:00
|
|
|
/*
|
2016-12-07 08:46:16 +08:00
|
|
|
* #PF can be fast if:
|
|
|
|
* 1. The shadow page table entry is not present, which could mean that
|
|
|
|
* the fault is potentially caused by access tracking (if enabled).
|
|
|
|
* 2. The shadow page table entry is present and the fault
|
|
|
|
* is caused by write-protect, that means we just need change the W
|
|
|
|
* bit of the spte which can be done out of mmu-lock.
|
|
|
|
*
|
|
|
|
* However, if access tracking is disabled we know that a non-present
|
|
|
|
* page must be a genuine page fault where we have to create a new SPTE.
|
|
|
|
* So, if access tracking is disabled, we return true only for write
|
|
|
|
* accesses to a present page.
|
2012-06-20 15:59:18 +08:00
|
|
|
*/
|
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
return shadow_acc_track_mask != 0 ||
|
|
|
|
((error_code & (PFERR_WRITE_MASK | PFERR_PRESENT_MASK))
|
|
|
|
== (PFERR_WRITE_MASK | PFERR_PRESENT_MASK));
|
2012-06-20 15:59:18 +08:00
|
|
|
}
|
|
|
|
|
2016-12-07 08:46:12 +08:00
|
|
|
/*
|
|
|
|
* Returns true if the SPTE was fixed successfully. Otherwise,
|
|
|
|
* someone else modified the SPTE from its original value.
|
|
|
|
*/
|
2012-06-20 15:59:18 +08:00
|
|
|
static bool
|
2014-04-17 17:06:13 +08:00
|
|
|
fast_pf_fix_direct_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
|
2016-12-22 12:29:32 +08:00
|
|
|
u64 *sptep, u64 old_spte, u64 new_spte)
|
2012-06-20 15:59:18 +08:00
|
|
|
{
|
|
|
|
gfn_t gfn;
|
|
|
|
|
|
|
|
WARN_ON(!sp->role.direct);
|
|
|
|
|
2015-01-28 10:54:25 +08:00
|
|
|
/*
|
|
|
|
* Theoretically we could also set dirty bit (and flush TLB) here in
|
|
|
|
* order to eliminate unnecessary PML logging. See comments in
|
|
|
|
* set_spte. But fast_page_fault is very unlikely to happen with PML
|
|
|
|
* enabled, so we do not do this. This might result in the same GPA
|
|
|
|
* to be logged in PML buffer again when the write really happens, and
|
|
|
|
* eventually to be called by mark_page_dirty twice. But it's also no
|
|
|
|
* harm. This also avoids the TLB flush needed after setting dirty bit
|
|
|
|
* so non-PML cases won't be impacted.
|
|
|
|
*
|
|
|
|
* Compare with set_spte where instead shadow_dirty_mask is set.
|
|
|
|
*/
|
2016-12-07 08:46:16 +08:00
|
|
|
if (cmpxchg64(sptep, old_spte, new_spte) != old_spte)
|
2016-12-07 08:46:12 +08:00
|
|
|
return false;
|
|
|
|
|
2016-12-22 12:29:32 +08:00
|
|
|
if (is_writable_pte(new_spte) && !is_writable_pte(old_spte)) {
|
2016-12-07 08:46:16 +08:00
|
|
|
/*
|
|
|
|
* The gfn of direct spte is stable since it is
|
|
|
|
* calculated by sp->gfn.
|
|
|
|
*/
|
|
|
|
gfn = kvm_mmu_page_get_gfn(sp, sptep - sp->spt);
|
|
|
|
kvm_vcpu_mark_page_dirty(vcpu, gfn);
|
|
|
|
}
|
2012-06-20 15:59:18 +08:00
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2016-12-22 12:29:32 +08:00
|
|
|
static bool is_access_allowed(u32 fault_err_code, u64 spte)
|
|
|
|
{
|
|
|
|
if (fault_err_code & PFERR_FETCH_MASK)
|
|
|
|
return is_executable_pte(spte);
|
|
|
|
|
|
|
|
if (fault_err_code & PFERR_WRITE_MASK)
|
|
|
|
return is_writable_pte(spte);
|
|
|
|
|
|
|
|
/* Fault was on Read access */
|
|
|
|
return spte & PT_PRESENT_MASK;
|
|
|
|
}
|
|
|
|
|
2012-06-20 15:59:18 +08:00
|
|
|
/*
|
|
|
|
* Return value:
|
|
|
|
* - true: let the vcpu to access on the same address again.
|
|
|
|
* - false: let the real page fault path to fix it.
|
|
|
|
*/
|
2020-01-09 04:24:42 +08:00
|
|
|
static bool fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
|
2012-06-20 15:59:18 +08:00
|
|
|
u32 error_code)
|
|
|
|
{
|
|
|
|
struct kvm_shadow_walk_iterator iterator;
|
2014-04-17 17:06:13 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2016-12-07 08:46:12 +08:00
|
|
|
bool fault_handled = false;
|
2012-06-20 15:59:18 +08:00
|
|
|
u64 spte = 0ull;
|
2016-12-07 08:46:12 +08:00
|
|
|
uint retry_count = 0;
|
2012-06-20 15:59:18 +08:00
|
|
|
|
2013-07-30 21:01:59 +08:00
|
|
|
if (!page_fault_can_be_fast(error_code))
|
2012-06-20 15:59:18 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
walk_shadow_page_lockless_begin(vcpu);
|
|
|
|
|
2016-12-07 08:46:12 +08:00
|
|
|
do {
|
2016-12-22 12:29:32 +08:00
|
|
|
u64 new_spte;
|
2012-06-20 15:59:18 +08:00
|
|
|
|
2019-12-07 07:57:14 +08:00
|
|
|
for_each_shadow_entry_lockless(vcpu, cr2_or_gpa, iterator, spte)
|
2020-01-09 04:24:42 +08:00
|
|
|
if (!is_shadow_present_pte(spte))
|
2016-12-22 12:29:30 +08:00
|
|
|
break;
|
|
|
|
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(iterator.sptep);
|
2016-12-07 08:46:12 +08:00
|
|
|
if (!is_last_spte(spte, sp->role.level))
|
|
|
|
break;
|
2012-06-20 15:59:18 +08:00
|
|
|
|
2016-12-07 08:46:12 +08:00
|
|
|
/*
|
2016-12-07 08:46:16 +08:00
|
|
|
* Check whether the memory access that caused the fault would
|
|
|
|
* still cause it if it were to be performed right now. If not,
|
|
|
|
* then this is a spurious fault caused by TLB lazily flushed,
|
|
|
|
* or some other CPU has already fixed the PTE after the
|
|
|
|
* current CPU took the fault.
|
2016-12-07 08:46:12 +08:00
|
|
|
*
|
|
|
|
* Need not check the access of upper level table entries since
|
|
|
|
* they are always ACC_ALL.
|
|
|
|
*/
|
2016-12-22 12:29:32 +08:00
|
|
|
if (is_access_allowed(error_code, spte)) {
|
|
|
|
fault_handled = true;
|
|
|
|
break;
|
|
|
|
}
|
2016-12-07 08:46:16 +08:00
|
|
|
|
2016-12-22 12:29:32 +08:00
|
|
|
new_spte = spte;
|
|
|
|
|
|
|
|
if (is_access_track_spte(spte))
|
|
|
|
new_spte = restore_acc_track_spte(new_spte);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Currently, to simplify the code, write-protection can
|
|
|
|
* be removed in the fast path only if the SPTE was
|
|
|
|
* write-protected for dirty-logging or access tracking.
|
|
|
|
*/
|
|
|
|
if ((error_code & PFERR_WRITE_MASK) &&
|
2020-02-15 10:44:22 +08:00
|
|
|
spte_can_locklessly_be_made_writable(spte)) {
|
2016-12-22 12:29:32 +08:00
|
|
|
new_spte |= PT_WRITABLE_MASK;
|
2016-12-07 08:46:16 +08:00
|
|
|
|
|
|
|
/*
|
2016-12-22 12:29:32 +08:00
|
|
|
* Do not fix write-permission on the large spte. Since
|
|
|
|
* we only dirty the first page into the dirty-bitmap in
|
|
|
|
* fast_pf_fix_direct_spte(), other pages are missed
|
|
|
|
* if its slot has dirty logging enabled.
|
|
|
|
*
|
|
|
|
* Instead, we let the slow page fault path create a
|
|
|
|
* normal spte to fix the access.
|
|
|
|
*
|
|
|
|
* See the comments in kvm_arch_commit_memory_region().
|
2016-12-07 08:46:16 +08:00
|
|
|
*/
|
2020-04-28 08:54:22 +08:00
|
|
|
if (sp->role.level > PG_LEVEL_4K)
|
2016-12-07 08:46:16 +08:00
|
|
|
break;
|
2016-12-07 08:46:12 +08:00
|
|
|
}
|
2012-06-20 15:59:18 +08:00
|
|
|
|
2016-12-07 08:46:16 +08:00
|
|
|
/* Verify that the fault can be handled in the fast path */
|
2016-12-22 12:29:32 +08:00
|
|
|
if (new_spte == spte ||
|
|
|
|
!is_access_allowed(error_code, new_spte))
|
2016-12-07 08:46:12 +08:00
|
|
|
break;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Currently, fast page fault only works for direct mapping
|
|
|
|
* since the gfn is not stable for indirect shadow page. See
|
2020-04-15 00:48:36 +08:00
|
|
|
* Documentation/virt/kvm/locking.rst to get more detail.
|
2016-12-07 08:46:12 +08:00
|
|
|
*/
|
|
|
|
fault_handled = fast_pf_fix_direct_spte(vcpu, sp,
|
2016-12-07 08:46:16 +08:00
|
|
|
iterator.sptep, spte,
|
2016-12-22 12:29:32 +08:00
|
|
|
new_spte);
|
2016-12-07 08:46:12 +08:00
|
|
|
if (fault_handled)
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (++retry_count > 4) {
|
|
|
|
printk_once(KERN_WARNING
|
|
|
|
"kvm: Fast #PF retrying more than 4 times.\n");
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
} while (true);
|
2014-04-17 17:06:14 +08:00
|
|
|
|
2019-12-07 07:57:14 +08:00
|
|
|
trace_fast_page_fault(vcpu, cr2_or_gpa, error_code, iterator.sptep,
|
2016-12-07 08:46:12 +08:00
|
|
|
spte, fault_handled);
|
2012-06-20 15:59:18 +08:00
|
|
|
walk_shadow_page_lockless_end(vcpu);
|
|
|
|
|
2016-12-07 08:46:12 +08:00
|
|
|
return fault_handled;
|
2012-06-20 15:59:18 +08:00
|
|
|
}
|
|
|
|
|
2018-05-05 02:37:11 +08:00
|
|
|
static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
|
|
|
|
struct list_head *invalid_list)
|
2007-01-06 08:36:40 +08:00
|
|
|
{
|
2007-11-21 21:28:32 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
2007-01-06 08:36:40 +08:00
|
|
|
|
2018-05-05 02:37:11 +08:00
|
|
|
if (!VALID_PAGE(*root_hpa))
|
2007-06-05 17:17:03 +08:00
|
|
|
return;
|
2013-05-16 16:55:51 +08:00
|
|
|
|
2020-06-23 04:20:34 +08:00
|
|
|
sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
|
2018-05-05 02:37:11 +08:00
|
|
|
--sp->root_count;
|
|
|
|
if (!sp->root_count && sp->role.invalid)
|
|
|
|
kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
|
2007-01-06 08:36:40 +08:00
|
|
|
|
2018-05-05 02:37:11 +08:00
|
|
|
*root_hpa = INVALID_PAGE;
|
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:17 +08:00
|
|
|
/* roots_to_free must be some combination of the KVM_MMU_ROOT_* flags */
|
2018-10-09 03:28:07 +08:00
|
|
|
void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
|
|
|
|
ulong roots_to_free)
|
2018-05-05 02:37:11 +08:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
LIST_HEAD(invalid_list);
|
2018-06-28 05:59:17 +08:00
|
|
|
bool free_active_root = roots_to_free & KVM_MMU_ROOT_CURRENT;
|
2018-05-05 02:37:11 +08:00
|
|
|
|
2018-06-28 05:59:20 +08:00
|
|
|
BUILD_BUG_ON(KVM_MMU_NUM_PREV_ROOTS >= BITS_PER_LONG);
|
2018-05-05 02:37:11 +08:00
|
|
|
|
2018-06-28 05:59:17 +08:00
|
|
|
/* Before acquiring the MMU lock, see if we need to do any real work. */
|
2018-06-28 05:59:20 +08:00
|
|
|
if (!(free_active_root && VALID_PAGE(mmu->root_hpa))) {
|
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
|
|
|
|
if ((roots_to_free & KVM_MMU_ROOT_PREVIOUS(i)) &&
|
|
|
|
VALID_PAGE(mmu->prev_roots[i].hpa))
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (i == KVM_MMU_NUM_PREV_ROOTS)
|
|
|
|
return;
|
|
|
|
}
|
2013-05-16 16:55:51 +08:00
|
|
|
|
|
|
|
spin_lock(&vcpu->kvm->mmu_lock);
|
2007-01-06 08:36:40 +08:00
|
|
|
|
2018-06-28 05:59:20 +08:00
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
|
|
|
|
if (roots_to_free & KVM_MMU_ROOT_PREVIOUS(i))
|
|
|
|
mmu_free_root_page(vcpu->kvm, &mmu->prev_roots[i].hpa,
|
|
|
|
&invalid_list);
|
2018-06-28 05:59:06 +08:00
|
|
|
|
2018-06-28 05:59:17 +08:00
|
|
|
if (free_active_root) {
|
|
|
|
if (mmu->shadow_root_level >= PT64_ROOT_4LEVEL &&
|
|
|
|
(mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) {
|
|
|
|
mmu_free_root_page(vcpu->kvm, &mmu->root_hpa,
|
|
|
|
&invalid_list);
|
|
|
|
} else {
|
|
|
|
for (i = 0; i < 4; ++i)
|
|
|
|
if (mmu->pae_root[i] != 0)
|
|
|
|
mmu_free_root_page(vcpu->kvm,
|
|
|
|
&mmu->pae_root[i],
|
|
|
|
&invalid_list);
|
|
|
|
mmu->root_hpa = INVALID_PAGE;
|
|
|
|
}
|
2020-03-21 05:28:32 +08:00
|
|
|
mmu->root_pgd = 0;
|
2007-01-06 08:36:40 +08:00
|
|
|
}
|
2018-05-05 02:37:11 +08:00
|
|
|
|
2010-06-04 21:55:29 +08:00
|
|
|
kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
|
2007-12-21 08:18:26 +08:00
|
|
|
spin_unlock(&vcpu->kvm->mmu_lock);
|
2007-01-06 08:36:40 +08:00
|
|
|
}
|
2018-05-05 02:37:11 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_free_roots);
|
2007-01-06 08:36:40 +08:00
|
|
|
|
2009-05-13 05:55:45 +08:00
|
|
|
static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
if (!kvm_is_visible_gfn(vcpu->kvm, root_gfn)) {
|
2010-05-10 17:34:53 +08:00
|
|
|
kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
|
2009-05-13 05:55:45 +08:00
|
|
|
ret = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2020-04-28 10:37:14 +08:00
|
|
|
static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, gva_t gva,
|
|
|
|
u8 level, bool direct)
|
2010-09-10 23:30:59 +08:00
|
|
|
{
|
|
|
|
struct kvm_mmu_page *sp;
|
2020-04-28 10:37:14 +08:00
|
|
|
|
|
|
|
spin_lock(&vcpu->kvm->mmu_lock);
|
|
|
|
|
|
|
|
if (make_mmu_pages_available(vcpu)) {
|
|
|
|
spin_unlock(&vcpu->kvm->mmu_lock);
|
|
|
|
return INVALID_PAGE;
|
|
|
|
}
|
|
|
|
sp = kvm_mmu_get_page(vcpu, gfn, gva, level, direct, ACC_ALL);
|
|
|
|
++sp->root_count;
|
|
|
|
|
|
|
|
spin_unlock(&vcpu->kvm->mmu_lock);
|
|
|
|
return __pa(sp->spt);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
|
|
|
u8 shadow_root_level = vcpu->arch.mmu->shadow_root_level;
|
|
|
|
hpa_t root;
|
2010-10-04 00:51:39 +08:00
|
|
|
unsigned i;
|
2010-09-10 23:30:59 +08:00
|
|
|
|
2020-04-28 10:37:14 +08:00
|
|
|
if (shadow_root_level >= PT64_ROOT_4LEVEL) {
|
|
|
|
root = mmu_alloc_root(vcpu, 0, 0, shadow_root_level, true);
|
|
|
|
if (!VALID_PAGE(root))
|
2017-12-05 14:21:30 +08:00
|
|
|
return -ENOSPC;
|
2020-04-28 10:37:14 +08:00
|
|
|
vcpu->arch.mmu->root_hpa = root;
|
|
|
|
} else if (shadow_root_level == PT32E_ROOT_LEVEL) {
|
2010-09-10 23:30:59 +08:00
|
|
|
for (i = 0; i < 4; ++i) {
|
2020-04-28 10:37:14 +08:00
|
|
|
MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->pae_root[i]));
|
2010-09-10 23:30:59 +08:00
|
|
|
|
2020-04-28 10:37:14 +08:00
|
|
|
root = mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT),
|
|
|
|
i << 30, PT32_ROOT_LEVEL, true);
|
|
|
|
if (!VALID_PAGE(root))
|
2017-12-05 14:21:30 +08:00
|
|
|
return -ENOSPC;
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->pae_root[i] = root | PT_PRESENT_MASK;
|
2010-09-10 23:30:59 +08:00
|
|
|
}
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root);
|
2010-09-10 23:30:59 +08:00
|
|
|
} else
|
|
|
|
BUG();
|
2020-02-29 06:52:39 +08:00
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
/* root_pgd is ignored for direct MMUs. */
|
|
|
|
vcpu->arch.mmu->root_pgd = 0;
|
2010-09-10 23:30:59 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)
|
2007-01-06 08:36:40 +08:00
|
|
|
{
|
2010-09-10 23:31:00 +08:00
|
|
|
u64 pdptr, pm_mask;
|
2020-03-21 05:28:32 +08:00
|
|
|
gfn_t root_gfn, root_pgd;
|
2020-04-28 10:37:14 +08:00
|
|
|
hpa_t root;
|
2010-09-10 23:31:00 +08:00
|
|
|
int i;
|
2007-01-06 08:36:51 +08:00
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
root_pgd = vcpu->arch.mmu->get_guest_pgd(vcpu);
|
|
|
|
root_gfn = root_pgd >> PAGE_SHIFT;
|
2007-01-06 08:36:40 +08:00
|
|
|
|
2010-09-10 23:30:59 +08:00
|
|
|
if (mmu_check_root(vcpu, root_gfn))
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Do we shadow a long mode page table? If so we need to
|
|
|
|
* write-protect the guests page table root.
|
|
|
|
*/
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->root_level >= PT64_ROOT_4LEVEL) {
|
2020-04-28 10:37:14 +08:00
|
|
|
MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->root_hpa));
|
2010-09-10 23:30:59 +08:00
|
|
|
|
2020-04-28 10:37:14 +08:00
|
|
|
root = mmu_alloc_root(vcpu, root_gfn, 0,
|
|
|
|
vcpu->arch.mmu->shadow_root_level, false);
|
|
|
|
if (!VALID_PAGE(root))
|
2017-12-05 14:21:30 +08:00
|
|
|
return -ENOSPC;
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->root_hpa = root;
|
2020-03-21 05:28:32 +08:00
|
|
|
goto set_root_pgd;
|
2007-01-06 08:36:40 +08:00
|
|
|
}
|
2010-09-02 23:29:45 +08:00
|
|
|
|
2010-09-10 23:30:59 +08:00
|
|
|
/*
|
|
|
|
* We shadow a 32 bit page table. This may be a legacy 2-level
|
2010-09-10 23:31:00 +08:00
|
|
|
* or a PAE 3-level page table. In either case we need to be aware that
|
|
|
|
* the shadow page table may be a PAE or a long mode page table.
|
2010-09-10 23:30:59 +08:00
|
|
|
*/
|
2010-09-10 23:31:00 +08:00
|
|
|
pm_mask = PT_PRESENT_MASK;
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL)
|
2010-09-10 23:31:00 +08:00
|
|
|
pm_mask |= PT_ACCESSED_MASK | PT_WRITABLE_MASK | PT_USER_MASK;
|
|
|
|
|
2007-01-06 08:36:40 +08:00
|
|
|
for (i = 0; i < 4; ++i) {
|
2020-04-28 10:37:14 +08:00
|
|
|
MMU_WARN_ON(VALID_PAGE(vcpu->arch.mmu->pae_root[i]));
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->root_level == PT32E_ROOT_LEVEL) {
|
|
|
|
pdptr = vcpu->arch.mmu->get_pdptr(vcpu, i);
|
2016-07-13 06:18:50 +08:00
|
|
|
if (!(pdptr & PT_PRESENT_MASK)) {
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->pae_root[i] = 0;
|
2007-04-12 22:35:58 +08:00
|
|
|
continue;
|
|
|
|
}
|
2009-06-01 03:58:47 +08:00
|
|
|
root_gfn = pdptr >> PAGE_SHIFT;
|
2010-09-02 23:29:45 +08:00
|
|
|
if (mmu_check_root(vcpu, root_gfn))
|
|
|
|
return 1;
|
2010-04-27 08:00:05 +08:00
|
|
|
}
|
2010-05-04 17:58:32 +08:00
|
|
|
|
2020-04-28 10:37:14 +08:00
|
|
|
root = mmu_alloc_root(vcpu, root_gfn, i << 30,
|
|
|
|
PT32_ROOT_LEVEL, false);
|
|
|
|
if (!VALID_PAGE(root))
|
|
|
|
return -ENOSPC;
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->pae_root[i] = root | pm_mask;
|
2007-01-06 08:36:40 +08:00
|
|
|
}
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->pae_root);
|
2010-09-10 23:31:00 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If we shadow a 32 bit page table with a long mode page
|
|
|
|
* table we enter this path.
|
|
|
|
*/
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->shadow_root_level == PT64_ROOT_4LEVEL) {
|
|
|
|
if (vcpu->arch.mmu->lm_root == NULL) {
|
2010-09-10 23:31:00 +08:00
|
|
|
/*
|
|
|
|
* The additional page necessary for this is only
|
|
|
|
* allocated on demand.
|
|
|
|
*/
|
|
|
|
|
|
|
|
u64 *lm_root;
|
|
|
|
|
2019-02-12 03:02:50 +08:00
|
|
|
lm_root = (void*)get_zeroed_page(GFP_KERNEL_ACCOUNT);
|
2010-09-10 23:31:00 +08:00
|
|
|
if (lm_root == NULL)
|
|
|
|
return 1;
|
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
lm_root[0] = __pa(vcpu->arch.mmu->pae_root) | pm_mask;
|
2010-09-10 23:31:00 +08:00
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->lm_root = lm_root;
|
2010-09-10 23:31:00 +08:00
|
|
|
}
|
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->root_hpa = __pa(vcpu->arch.mmu->lm_root);
|
2010-09-10 23:31:00 +08:00
|
|
|
}
|
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
set_root_pgd:
|
|
|
|
vcpu->arch.mmu->root_pgd = root_pgd;
|
2019-02-23 00:45:01 +08:00
|
|
|
|
2009-05-13 05:55:45 +08:00
|
|
|
return 0;
|
2007-01-06 08:36:40 +08:00
|
|
|
}
|
|
|
|
|
2010-09-10 23:30:59 +08:00
|
|
|
static int mmu_alloc_roots(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->direct_map)
|
2010-09-10 23:30:59 +08:00
|
|
|
return mmu_alloc_direct_roots(vcpu);
|
|
|
|
else
|
|
|
|
return mmu_alloc_shadow_roots(vcpu);
|
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:05 +08:00
|
|
|
void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu)
|
2008-09-24 00:18:34 +08:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->direct_map)
|
2010-09-10 23:31:00 +08:00
|
|
|
return;
|
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
if (!VALID_PAGE(vcpu->arch.mmu->root_hpa))
|
2008-09-24 00:18:34 +08:00
|
|
|
return;
|
2010-09-27 18:09:29 +08:00
|
|
|
|
2014-08-19 06:46:07 +08:00
|
|
|
vcpu_clear_mmio_info(vcpu, MMIO_GVA_ANY);
|
2018-06-28 05:59:05 +08:00
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->root_level >= PT64_ROOT_4LEVEL) {
|
|
|
|
hpa_t root = vcpu->arch.mmu->root_hpa;
|
2020-06-23 04:20:34 +08:00
|
|
|
sp = to_shadow_page(root);
|
2018-06-28 05:59:05 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Even if another CPU was marking the SP as unsync-ed
|
|
|
|
* simultaneously, any guest page table changes are not
|
|
|
|
* guaranteed to be visible anyway until this VCPU issues a TLB
|
|
|
|
* flush strictly after those changes are made. We only need to
|
|
|
|
* ensure that the other CPU sets these flags before any actual
|
|
|
|
* changes to the page tables are made. The comments in
|
|
|
|
* mmu_need_write_protect() describe what could go wrong if this
|
|
|
|
* requirement isn't satisfied.
|
|
|
|
*/
|
|
|
|
if (!smp_load_acquire(&sp->unsync) &&
|
|
|
|
!smp_load_acquire(&sp->unsync_children))
|
|
|
|
return;
|
|
|
|
|
|
|
|
spin_lock(&vcpu->kvm->mmu_lock);
|
|
|
|
kvm_mmu_audit(vcpu, AUDIT_PRE_SYNC);
|
|
|
|
|
2008-09-24 00:18:34 +08:00
|
|
|
mmu_sync_children(vcpu, sp);
|
2018-06-28 05:59:05 +08:00
|
|
|
|
2011-11-28 20:41:00 +08:00
|
|
|
kvm_mmu_audit(vcpu, AUDIT_POST_SYNC);
|
2018-06-28 05:59:05 +08:00
|
|
|
spin_unlock(&vcpu->kvm->mmu_lock);
|
2008-09-24 00:18:34 +08:00
|
|
|
return;
|
|
|
|
}
|
2018-06-28 05:59:05 +08:00
|
|
|
|
|
|
|
spin_lock(&vcpu->kvm->mmu_lock);
|
|
|
|
kvm_mmu_audit(vcpu, AUDIT_PRE_SYNC);
|
|
|
|
|
2008-09-24 00:18:34 +08:00
|
|
|
for (i = 0; i < 4; ++i) {
|
2018-10-09 03:28:05 +08:00
|
|
|
hpa_t root = vcpu->arch.mmu->pae_root[i];
|
2008-09-24 00:18:34 +08:00
|
|
|
|
2009-05-13 05:55:45 +08:00
|
|
|
if (root && VALID_PAGE(root)) {
|
2008-09-24 00:18:34 +08:00
|
|
|
root &= PT64_BASE_ADDR_MASK;
|
2020-06-23 04:20:34 +08:00
|
|
|
sp = to_shadow_page(root);
|
2008-09-24 00:18:34 +08:00
|
|
|
mmu_sync_children(vcpu, sp);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:05 +08:00
|
|
|
kvm_mmu_audit(vcpu, AUDIT_POST_SYNC);
|
2008-12-02 08:32:04 +08:00
|
|
|
spin_unlock(&vcpu->kvm->mmu_lock);
|
2008-09-24 00:18:34 +08:00
|
|
|
}
|
2013-08-05 16:07:17 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_sync_roots);
|
2008-09-24 00:18:34 +08:00
|
|
|
|
2019-12-07 07:57:14 +08:00
|
|
|
static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, gpa_t vaddr,
|
2010-11-22 23:53:26 +08:00
|
|
|
u32 access, struct x86_exception *exception)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2010-11-22 23:53:26 +08:00
|
|
|
if (exception)
|
|
|
|
exception->error_code = 0;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
return vaddr;
|
|
|
|
}
|
|
|
|
|
2019-12-07 07:57:14 +08:00
|
|
|
static gpa_t nonpaging_gva_to_gpa_nested(struct kvm_vcpu *vcpu, gpa_t vaddr,
|
2010-11-22 23:53:26 +08:00
|
|
|
u32 access,
|
|
|
|
struct x86_exception *exception)
|
2010-09-10 23:30:50 +08:00
|
|
|
{
|
2010-11-22 23:53:26 +08:00
|
|
|
if (exception)
|
|
|
|
exception->error_code = 0;
|
2014-09-02 19:23:06 +08:00
|
|
|
return vcpu->arch.nested_mmu.translate_gpa(vcpu, vaddr, access, exception);
|
2010-09-10 23:30:50 +08:00
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:25 +08:00
|
|
|
static bool
|
|
|
|
__is_rsvd_bits_set(struct rsvd_bits_validate *rsvd_check, u64 pte, int level)
|
|
|
|
{
|
KVM: x86/mmu: Micro-optimize nEPT's bad memptype/XWR checks
Rework the handling of nEPT's bad memtype/XWR checks to micro-optimize
the checks as much as possible. Move the check to a separate helper,
__is_bad_mt_xwr(), which allows the guest_rsvd_check usage in
paging_tmpl.h to omit the check entirely for paging32/64 (bad_mt_xwr is
always zero for non-nEPT) while retaining the bitwise-OR of the current
code for the shadow_zero_check in walk_shadow_page_get_mmio_spte().
Add a comment for the bitwise-OR usage in the mmio spte walk to avoid
future attempts to "fix" the code, which is what prompted this
optimization in the first place[*].
Opportunistically remove the superfluous '!= 0' and parantheses, and
use BIT_ULL() instead of open coding its equivalent.
The net effect is that code generation is largely unchanged for
walk_shadow_page_get_mmio_spte(), marginally better for
ept_prefetch_invalid_gpte(), and significantly improved for
paging32/64_prefetch_invalid_gpte().
Note, walk_shadow_page_get_mmio_spte() can't use a templated version of
the memtype/XRW as it works on the host's shadow PTEs, e.g. checks that
KVM hasn't borked its EPT tables. Even if it could be templated, the
benefits of having a single implementation far outweight the few uops
that would be saved for NPT or non-TDP paging, e.g. most compilers
inline it all the way to up kvm_mmu_page_fault().
[*] https://lkml.kernel.org/r/20200108001859.25254-1-sean.j.christopherson@intel.com
Cc: Jim Mattson <jmattson@google.com>
Cc: David Laight <David.Laight@ACULAB.COM>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-01-10 07:06:40 +08:00
|
|
|
int bit7 = (pte >> 7) & 1;
|
2015-08-05 12:04:25 +08:00
|
|
|
|
KVM: x86/mmu: Micro-optimize nEPT's bad memptype/XWR checks
Rework the handling of nEPT's bad memtype/XWR checks to micro-optimize
the checks as much as possible. Move the check to a separate helper,
__is_bad_mt_xwr(), which allows the guest_rsvd_check usage in
paging_tmpl.h to omit the check entirely for paging32/64 (bad_mt_xwr is
always zero for non-nEPT) while retaining the bitwise-OR of the current
code for the shadow_zero_check in walk_shadow_page_get_mmio_spte().
Add a comment for the bitwise-OR usage in the mmio spte walk to avoid
future attempts to "fix" the code, which is what prompted this
optimization in the first place[*].
Opportunistically remove the superfluous '!= 0' and parantheses, and
use BIT_ULL() instead of open coding its equivalent.
The net effect is that code generation is largely unchanged for
walk_shadow_page_get_mmio_spte(), marginally better for
ept_prefetch_invalid_gpte(), and significantly improved for
paging32/64_prefetch_invalid_gpte().
Note, walk_shadow_page_get_mmio_spte() can't use a templated version of
the memtype/XRW as it works on the host's shadow PTEs, e.g. checks that
KVM hasn't borked its EPT tables. Even if it could be templated, the
benefits of having a single implementation far outweight the few uops
that would be saved for NPT or non-TDP paging, e.g. most compilers
inline it all the way to up kvm_mmu_page_fault().
[*] https://lkml.kernel.org/r/20200108001859.25254-1-sean.j.christopherson@intel.com
Cc: Jim Mattson <jmattson@google.com>
Cc: David Laight <David.Laight@ACULAB.COM>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-01-10 07:06:40 +08:00
|
|
|
return pte & rsvd_check->rsvd_bits_mask[bit7][level-1];
|
2015-08-05 12:04:25 +08:00
|
|
|
}
|
|
|
|
|
KVM: x86/mmu: Micro-optimize nEPT's bad memptype/XWR checks
Rework the handling of nEPT's bad memtype/XWR checks to micro-optimize
the checks as much as possible. Move the check to a separate helper,
__is_bad_mt_xwr(), which allows the guest_rsvd_check usage in
paging_tmpl.h to omit the check entirely for paging32/64 (bad_mt_xwr is
always zero for non-nEPT) while retaining the bitwise-OR of the current
code for the shadow_zero_check in walk_shadow_page_get_mmio_spte().
Add a comment for the bitwise-OR usage in the mmio spte walk to avoid
future attempts to "fix" the code, which is what prompted this
optimization in the first place[*].
Opportunistically remove the superfluous '!= 0' and parantheses, and
use BIT_ULL() instead of open coding its equivalent.
The net effect is that code generation is largely unchanged for
walk_shadow_page_get_mmio_spte(), marginally better for
ept_prefetch_invalid_gpte(), and significantly improved for
paging32/64_prefetch_invalid_gpte().
Note, walk_shadow_page_get_mmio_spte() can't use a templated version of
the memtype/XRW as it works on the host's shadow PTEs, e.g. checks that
KVM hasn't borked its EPT tables. Even if it could be templated, the
benefits of having a single implementation far outweight the few uops
that would be saved for NPT or non-TDP paging, e.g. most compilers
inline it all the way to up kvm_mmu_page_fault().
[*] https://lkml.kernel.org/r/20200108001859.25254-1-sean.j.christopherson@intel.com
Cc: Jim Mattson <jmattson@google.com>
Cc: David Laight <David.Laight@ACULAB.COM>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-01-10 07:06:40 +08:00
|
|
|
static bool __is_bad_mt_xwr(struct rsvd_bits_validate *rsvd_check, u64 pte)
|
2015-08-05 12:04:25 +08:00
|
|
|
{
|
KVM: x86/mmu: Micro-optimize nEPT's bad memptype/XWR checks
Rework the handling of nEPT's bad memtype/XWR checks to micro-optimize
the checks as much as possible. Move the check to a separate helper,
__is_bad_mt_xwr(), which allows the guest_rsvd_check usage in
paging_tmpl.h to omit the check entirely for paging32/64 (bad_mt_xwr is
always zero for non-nEPT) while retaining the bitwise-OR of the current
code for the shadow_zero_check in walk_shadow_page_get_mmio_spte().
Add a comment for the bitwise-OR usage in the mmio spte walk to avoid
future attempts to "fix" the code, which is what prompted this
optimization in the first place[*].
Opportunistically remove the superfluous '!= 0' and parantheses, and
use BIT_ULL() instead of open coding its equivalent.
The net effect is that code generation is largely unchanged for
walk_shadow_page_get_mmio_spte(), marginally better for
ept_prefetch_invalid_gpte(), and significantly improved for
paging32/64_prefetch_invalid_gpte().
Note, walk_shadow_page_get_mmio_spte() can't use a templated version of
the memtype/XRW as it works on the host's shadow PTEs, e.g. checks that
KVM hasn't borked its EPT tables. Even if it could be templated, the
benefits of having a single implementation far outweight the few uops
that would be saved for NPT or non-TDP paging, e.g. most compilers
inline it all the way to up kvm_mmu_page_fault().
[*] https://lkml.kernel.org/r/20200108001859.25254-1-sean.j.christopherson@intel.com
Cc: Jim Mattson <jmattson@google.com>
Cc: David Laight <David.Laight@ACULAB.COM>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-01-10 07:06:40 +08:00
|
|
|
return rsvd_check->bad_mt_xwr & BIT_ULL(pte & 0x3f);
|
2015-08-05 12:04:25 +08:00
|
|
|
}
|
|
|
|
|
2016-02-22 16:23:40 +08:00
|
|
|
static bool mmio_info_in_cache(struct kvm_vcpu *vcpu, u64 addr, bool direct)
|
2011-07-12 03:33:44 +08:00
|
|
|
{
|
2017-08-18 00:36:58 +08:00
|
|
|
/*
|
|
|
|
* A nested guest cannot use the MMIO cache if it is using nested
|
|
|
|
* page tables, because cr2 is a nGPA while the cache stores GPAs.
|
|
|
|
*/
|
|
|
|
if (mmu_is_nested(vcpu))
|
|
|
|
return false;
|
|
|
|
|
2011-07-12 03:33:44 +08:00
|
|
|
if (direct)
|
|
|
|
return vcpu_match_mmio_gpa(vcpu, addr);
|
|
|
|
|
|
|
|
return vcpu_match_mmio_gva(vcpu, addr);
|
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:26 +08:00
|
|
|
/* return true if reserved bit is detected on spte. */
|
|
|
|
static bool
|
|
|
|
walk_shadow_page_get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep)
|
2011-07-12 03:33:44 +08:00
|
|
|
{
|
|
|
|
struct kvm_shadow_walk_iterator iterator;
|
2017-08-24 20:27:54 +08:00
|
|
|
u64 sptes[PT64_ROOT_MAX_LEVEL], spte = 0ull;
|
KVM: x86/mmu: Micro-optimize nEPT's bad memptype/XWR checks
Rework the handling of nEPT's bad memtype/XWR checks to micro-optimize
the checks as much as possible. Move the check to a separate helper,
__is_bad_mt_xwr(), which allows the guest_rsvd_check usage in
paging_tmpl.h to omit the check entirely for paging32/64 (bad_mt_xwr is
always zero for non-nEPT) while retaining the bitwise-OR of the current
code for the shadow_zero_check in walk_shadow_page_get_mmio_spte().
Add a comment for the bitwise-OR usage in the mmio spte walk to avoid
future attempts to "fix" the code, which is what prompted this
optimization in the first place[*].
Opportunistically remove the superfluous '!= 0' and parantheses, and
use BIT_ULL() instead of open coding its equivalent.
The net effect is that code generation is largely unchanged for
walk_shadow_page_get_mmio_spte(), marginally better for
ept_prefetch_invalid_gpte(), and significantly improved for
paging32/64_prefetch_invalid_gpte().
Note, walk_shadow_page_get_mmio_spte() can't use a templated version of
the memtype/XRW as it works on the host's shadow PTEs, e.g. checks that
KVM hasn't borked its EPT tables. Even if it could be templated, the
benefits of having a single implementation far outweight the few uops
that would be saved for NPT or non-TDP paging, e.g. most compilers
inline it all the way to up kvm_mmu_page_fault().
[*] https://lkml.kernel.org/r/20200108001859.25254-1-sean.j.christopherson@intel.com
Cc: Jim Mattson <jmattson@google.com>
Cc: David Laight <David.Laight@ACULAB.COM>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-01-10 07:06:40 +08:00
|
|
|
struct rsvd_bits_validate *rsvd_check;
|
2015-08-05 12:04:26 +08:00
|
|
|
int root, leaf;
|
|
|
|
bool reserved = false;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
KVM: x86/mmu: Micro-optimize nEPT's bad memptype/XWR checks
Rework the handling of nEPT's bad memtype/XWR checks to micro-optimize
the checks as much as possible. Move the check to a separate helper,
__is_bad_mt_xwr(), which allows the guest_rsvd_check usage in
paging_tmpl.h to omit the check entirely for paging32/64 (bad_mt_xwr is
always zero for non-nEPT) while retaining the bitwise-OR of the current
code for the shadow_zero_check in walk_shadow_page_get_mmio_spte().
Add a comment for the bitwise-OR usage in the mmio spte walk to avoid
future attempts to "fix" the code, which is what prompted this
optimization in the first place[*].
Opportunistically remove the superfluous '!= 0' and parantheses, and
use BIT_ULL() instead of open coding its equivalent.
The net effect is that code generation is largely unchanged for
walk_shadow_page_get_mmio_spte(), marginally better for
ept_prefetch_invalid_gpte(), and significantly improved for
paging32/64_prefetch_invalid_gpte().
Note, walk_shadow_page_get_mmio_spte() can't use a templated version of
the memtype/XRW as it works on the host's shadow PTEs, e.g. checks that
KVM hasn't borked its EPT tables. Even if it could be templated, the
benefits of having a single implementation far outweight the few uops
that would be saved for NPT or non-TDP paging, e.g. most compilers
inline it all the way to up kvm_mmu_page_fault().
[*] https://lkml.kernel.org/r/20200108001859.25254-1-sean.j.christopherson@intel.com
Cc: Jim Mattson <jmattson@google.com>
Cc: David Laight <David.Laight@ACULAB.COM>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-01-10 07:06:40 +08:00
|
|
|
rsvd_check = &vcpu->arch.mmu->shadow_zero_check;
|
2014-01-04 03:09:32 +08:00
|
|
|
|
2011-07-12 03:33:44 +08:00
|
|
|
walk_shadow_page_lockless_begin(vcpu);
|
2015-08-05 12:04:26 +08:00
|
|
|
|
2015-09-06 22:24:50 +08:00
|
|
|
for (shadow_walk_init(&iterator, vcpu, addr),
|
|
|
|
leaf = root = iterator.level;
|
2015-08-05 12:04:26 +08:00
|
|
|
shadow_walk_okay(&iterator);
|
|
|
|
__shadow_walk_next(&iterator, spte)) {
|
|
|
|
spte = mmu_spte_get_lockless(iterator.sptep);
|
|
|
|
|
|
|
|
sptes[leaf - 1] = spte;
|
2015-09-06 22:24:50 +08:00
|
|
|
leaf--;
|
2015-08-05 12:04:26 +08:00
|
|
|
|
2011-07-12 03:33:44 +08:00
|
|
|
if (!is_shadow_present_pte(spte))
|
|
|
|
break;
|
2015-08-05 12:04:26 +08:00
|
|
|
|
KVM: x86/mmu: Micro-optimize nEPT's bad memptype/XWR checks
Rework the handling of nEPT's bad memtype/XWR checks to micro-optimize
the checks as much as possible. Move the check to a separate helper,
__is_bad_mt_xwr(), which allows the guest_rsvd_check usage in
paging_tmpl.h to omit the check entirely for paging32/64 (bad_mt_xwr is
always zero for non-nEPT) while retaining the bitwise-OR of the current
code for the shadow_zero_check in walk_shadow_page_get_mmio_spte().
Add a comment for the bitwise-OR usage in the mmio spte walk to avoid
future attempts to "fix" the code, which is what prompted this
optimization in the first place[*].
Opportunistically remove the superfluous '!= 0' and parantheses, and
use BIT_ULL() instead of open coding its equivalent.
The net effect is that code generation is largely unchanged for
walk_shadow_page_get_mmio_spte(), marginally better for
ept_prefetch_invalid_gpte(), and significantly improved for
paging32/64_prefetch_invalid_gpte().
Note, walk_shadow_page_get_mmio_spte() can't use a templated version of
the memtype/XRW as it works on the host's shadow PTEs, e.g. checks that
KVM hasn't borked its EPT tables. Even if it could be templated, the
benefits of having a single implementation far outweight the few uops
that would be saved for NPT or non-TDP paging, e.g. most compilers
inline it all the way to up kvm_mmu_page_fault().
[*] https://lkml.kernel.org/r/20200108001859.25254-1-sean.j.christopherson@intel.com
Cc: Jim Mattson <jmattson@google.com>
Cc: David Laight <David.Laight@ACULAB.COM>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-01-10 07:06:40 +08:00
|
|
|
/*
|
|
|
|
* Use a bitwise-OR instead of a logical-OR to aggregate the
|
|
|
|
* reserved bit and EPT's invalid memtype/XWR checks to avoid
|
|
|
|
* adding a Jcc in the loop.
|
|
|
|
*/
|
|
|
|
reserved |= __is_bad_mt_xwr(rsvd_check, spte) |
|
|
|
|
__is_rsvd_bits_set(rsvd_check, spte, iterator.level);
|
2015-08-05 12:04:26 +08:00
|
|
|
}
|
|
|
|
|
2011-07-12 03:33:44 +08:00
|
|
|
walk_shadow_page_lockless_end(vcpu);
|
|
|
|
|
2015-08-05 12:04:26 +08:00
|
|
|
if (reserved) {
|
|
|
|
pr_err("%s: detect reserved bits on spte, addr 0x%llx, dump hierarchy:\n",
|
|
|
|
__func__, addr);
|
2015-09-06 22:24:50 +08:00
|
|
|
while (root > leaf) {
|
2015-08-05 12:04:26 +08:00
|
|
|
pr_err("------ spte 0x%llx level %d.\n",
|
|
|
|
sptes[root - 1], root);
|
|
|
|
root--;
|
|
|
|
}
|
|
|
|
}
|
KVM: x86/mmu: Move root_hpa validity checks to top of page fault handler
Add a check on root_hpa at the beginning of the page fault handler to
consolidate several checks on root_hpa that are scattered throughout the
page fault code. This is a preparatory step towards eventually removing
such checks altogether, or at the very least WARNing if an invalid root
is encountered. Remove only the checks that can be easily audited to
confirm that root_hpa cannot be invalidated between their current
location and the new check in kvm_mmu_page_fault(), and aren't currently
protected by mmu_lock, i.e. keep the checks in __direct_map() and
FNAME(fetch) for the time being.
The root_hpa checks that are consolidate were all added by commit
37f6a4e237303 ("KVM: x86: handle invalid root_hpa everywhere")
which was a follow up to a bug fix for __direct_map(), commit
989c6b34f6a94 ("KVM: MMU: handle invalid root_hpa at __direct_map")
At the time, nested VMX had, in hindsight, crazy handling of nested
interrupts and would trigger a nested VM-Exit in ->interrupt_allowed(),
and thus unexpectedly reset the MMU in flows such as can_do_async_pf().
Now that the wonky nested VM-Exit behavior is gone, the root_hpa checks
are bogus and confusing, e.g. it's not at all obvious what they actually
protect against, and at first glance they appear to be broken since many
of them run without holding mmu_lock.
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-12-07 07:57:27 +08:00
|
|
|
|
2015-08-05 12:04:26 +08:00
|
|
|
*sptep = spte;
|
|
|
|
return reserved;
|
2011-07-12 03:33:44 +08:00
|
|
|
}
|
|
|
|
|
2017-08-18 00:36:56 +08:00
|
|
|
static int handle_mmio_page_fault(struct kvm_vcpu *vcpu, u64 addr, bool direct)
|
2011-07-12 03:33:44 +08:00
|
|
|
{
|
|
|
|
u64 spte;
|
2015-08-05 12:04:26 +08:00
|
|
|
bool reserved;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2016-02-22 16:23:40 +08:00
|
|
|
if (mmio_info_in_cache(vcpu, addr, direct))
|
2017-08-17 21:03:32 +08:00
|
|
|
return RET_PF_EMULATE;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2015-08-05 12:04:26 +08:00
|
|
|
reserved = walk_shadow_page_get_mmio_spte(vcpu, addr, &spte);
|
2015-11-04 20:41:21 +08:00
|
|
|
if (WARN_ON(reserved))
|
2017-08-17 21:03:32 +08:00
|
|
|
return -EINVAL;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
|
|
|
if (is_mmio_spte(spte)) {
|
|
|
|
gfn_t gfn = get_mmio_spte_gfn(spte);
|
2020-02-04 07:09:09 +08:00
|
|
|
unsigned int access = get_mmio_spte_access(spte);
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2015-04-08 21:39:23 +08:00
|
|
|
if (!check_mmio_spte(vcpu, spte))
|
2017-08-17 21:03:32 +08:00
|
|
|
return RET_PF_INVALID;
|
2013-06-07 16:51:26 +08:00
|
|
|
|
2011-07-12 03:33:44 +08:00
|
|
|
if (direct)
|
|
|
|
addr = 0;
|
2011-07-12 03:34:24 +08:00
|
|
|
|
|
|
|
trace_handle_mmio_page_fault(addr, gfn, access);
|
2011-07-12 03:33:44 +08:00
|
|
|
vcpu_cache_mmio_info(vcpu, addr, gfn, access);
|
2017-08-17 21:03:32 +08:00
|
|
|
return RET_PF_EMULATE;
|
2011-07-12 03:33:44 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the page table is zapped by other cpus, let CPU fault again on
|
|
|
|
* the address.
|
|
|
|
*/
|
2017-08-17 21:03:32 +08:00
|
|
|
return RET_PF_RETRY;
|
2011-07-12 03:33:44 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:11 +08:00
|
|
|
static bool page_fault_handle_page_track(struct kvm_vcpu *vcpu,
|
|
|
|
u32 error_code, gfn_t gfn)
|
|
|
|
{
|
|
|
|
if (unlikely(error_code & PFERR_RSVD_MASK))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
if (!(error_code & PFERR_PRESENT_MASK) ||
|
|
|
|
!(error_code & PFERR_WRITE_MASK))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* guest is writing the page which is write tracked which can
|
|
|
|
* not be fixed by page fault handler.
|
|
|
|
*/
|
|
|
|
if (kvm_page_track_is_active(vcpu, gfn, KVM_PAGE_TRACK_WRITE))
|
|
|
|
return true;
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:12 +08:00
|
|
|
static void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr)
|
|
|
|
{
|
|
|
|
struct kvm_shadow_walk_iterator iterator;
|
|
|
|
u64 spte;
|
|
|
|
|
|
|
|
walk_shadow_page_lockless_begin(vcpu);
|
|
|
|
for_each_shadow_entry_lockless(vcpu, addr, iterator, spte) {
|
|
|
|
clear_sp_write_flooding_count(iterator.sptep);
|
|
|
|
if (!is_shadow_present_pte(spte))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
walk_shadow_page_lockless_end(vcpu);
|
|
|
|
}
|
|
|
|
|
2020-06-15 20:13:34 +08:00
|
|
|
static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
|
|
|
|
gfn_t gfn)
|
2010-10-14 17:22:46 +08:00
|
|
|
{
|
|
|
|
struct kvm_arch_async_pf arch;
|
2010-12-07 10:35:25 +08:00
|
|
|
|
2010-10-14 17:22:53 +08:00
|
|
|
arch.token = (vcpu->arch.apf.id++ << 12) | vcpu->vcpu_id;
|
2010-10-14 17:22:46 +08:00
|
|
|
arch.gfn = gfn;
|
2018-10-09 03:28:05 +08:00
|
|
|
arch.direct_map = vcpu->arch.mmu->direct_map;
|
2020-03-03 10:02:39 +08:00
|
|
|
arch.cr3 = vcpu->arch.mmu->get_guest_pgd(vcpu);
|
2010-10-14 17:22:46 +08:00
|
|
|
|
2019-12-07 07:57:17 +08:00
|
|
|
return kvm_setup_async_pf(vcpu, cr2_or_gpa,
|
|
|
|
kvm_vcpu_gfn_to_hva(vcpu, gfn), &arch);
|
2010-10-14 17:22:46 +08:00
|
|
|
}
|
|
|
|
|
2010-12-07 10:48:06 +08:00
|
|
|
static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,
|
2019-12-07 07:57:17 +08:00
|
|
|
gpa_t cr2_or_gpa, kvm_pfn_t *pfn, bool write,
|
|
|
|
bool *writable)
|
2010-10-14 17:22:46 +08:00
|
|
|
{
|
2020-04-16 21:48:07 +08:00
|
|
|
struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
|
2010-10-14 17:22:46 +08:00
|
|
|
bool async;
|
|
|
|
|
2020-04-16 21:48:07 +08:00
|
|
|
/* Don't expose private memslots to L2. */
|
|
|
|
if (is_guest_mode(vcpu) && !kvm_is_visible_memslot(slot)) {
|
2018-05-10 05:02:05 +08:00
|
|
|
*pfn = KVM_PFN_NOSLOT;
|
2020-04-16 05:44:13 +08:00
|
|
|
*writable = false;
|
2018-05-10 05:02:05 +08:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2015-04-02 17:20:48 +08:00
|
|
|
async = false;
|
|
|
|
*pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write, writable);
|
2010-10-14 17:22:46 +08:00
|
|
|
if (!async)
|
|
|
|
return false; /* *pfn has correct page already */
|
|
|
|
|
2017-06-09 11:13:40 +08:00
|
|
|
if (!prefault && kvm_can_do_async_pf(vcpu)) {
|
2019-12-07 07:57:17 +08:00
|
|
|
trace_kvm_try_async_get_page(cr2_or_gpa, gfn);
|
2010-10-14 17:22:46 +08:00
|
|
|
if (kvm_find_async_pf_gfn(vcpu, gfn)) {
|
2019-12-07 07:57:17 +08:00
|
|
|
trace_kvm_async_pf_doublefault(cr2_or_gpa, gfn);
|
2010-10-14 17:22:46 +08:00
|
|
|
kvm_make_request(KVM_REQ_APF_HALT, vcpu);
|
|
|
|
return true;
|
2019-12-07 07:57:17 +08:00
|
|
|
} else if (kvm_arch_setup_async_pf(vcpu, cr2_or_gpa, gfn))
|
2010-10-14 17:22:46 +08:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2015-04-02 17:20:48 +08:00
|
|
|
*pfn = __gfn_to_pfn_memslot(slot, gfn, false, NULL, write, writable);
|
2010-10-14 17:22:46 +08:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2019-12-07 07:57:24 +08:00
|
|
|
static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
|
|
|
|
bool prefault, int max_level, bool is_tdp)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2019-12-07 07:57:16 +08:00
|
|
|
bool write = error_code & PFERR_WRITE_MASK;
|
|
|
|
bool exec = error_code & PFERR_FETCH_MASK;
|
|
|
|
bool lpage_disallowed = exec && is_nx_huge_page_enabled();
|
2019-12-07 07:57:24 +08:00
|
|
|
bool map_writable;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2019-12-07 07:57:24 +08:00
|
|
|
gfn_t gfn = gpa >> PAGE_SHIFT;
|
|
|
|
unsigned long mmu_seq;
|
|
|
|
kvm_pfn_t pfn;
|
2020-01-09 04:24:43 +08:00
|
|
|
int r;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2016-02-24 17:51:11 +08:00
|
|
|
if (page_fault_handle_page_track(vcpu, error_code, gfn))
|
2017-08-17 21:03:32 +08:00
|
|
|
return RET_PF_EMULATE;
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2007-01-06 08:36:54 +08:00
|
|
|
r = mmu_topup_memory_caches(vcpu);
|
|
|
|
if (r)
|
|
|
|
return r;
|
2007-01-06 08:36:53 +08:00
|
|
|
|
2019-12-07 07:57:24 +08:00
|
|
|
if (lpage_disallowed)
|
2020-04-28 08:54:22 +08:00
|
|
|
max_level = PG_LEVEL_4K;
|
2019-12-07 07:57:16 +08:00
|
|
|
|
2020-01-09 04:24:42 +08:00
|
|
|
if (fast_page_fault(vcpu, gpa, error_code))
|
2019-12-07 07:57:16 +08:00
|
|
|
return RET_PF_RETRY;
|
|
|
|
|
|
|
|
mmu_seq = vcpu->kvm->mmu_notifier_seq;
|
|
|
|
smp_rmb();
|
|
|
|
|
|
|
|
if (try_async_pf(vcpu, prefault, gfn, gpa, &pfn, write, &map_writable))
|
|
|
|
return RET_PF_RETRY;
|
|
|
|
|
2019-12-07 07:57:24 +08:00
|
|
|
if (handle_abnormal_pfn(vcpu, is_tdp ? 0 : gpa, gfn, pfn, ACC_ALL, &r))
|
2019-12-07 07:57:16 +08:00
|
|
|
return r;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2019-12-07 07:57:16 +08:00
|
|
|
r = RET_PF_RETRY;
|
|
|
|
spin_lock(&vcpu->kvm->mmu_lock);
|
|
|
|
if (mmu_notifier_retry(vcpu->kvm, mmu_seq))
|
|
|
|
goto out_unlock;
|
2020-06-24 03:35:42 +08:00
|
|
|
r = make_mmu_pages_available(vcpu);
|
|
|
|
if (r)
|
2019-12-07 07:57:16 +08:00
|
|
|
goto out_unlock;
|
2020-01-09 04:24:43 +08:00
|
|
|
r = __direct_map(vcpu, gpa, write, map_writable, max_level, pfn,
|
2019-12-07 07:57:26 +08:00
|
|
|
prefault, is_tdp && lpage_disallowed);
|
2019-12-07 07:57:24 +08:00
|
|
|
|
2019-12-07 07:57:16 +08:00
|
|
|
out_unlock:
|
|
|
|
spin_unlock(&vcpu->kvm->mmu_lock);
|
|
|
|
kvm_release_pfn_clean(pfn);
|
|
|
|
return r;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2019-12-07 07:57:24 +08:00
|
|
|
static int nonpaging_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa,
|
|
|
|
u32 error_code, bool prefault)
|
|
|
|
{
|
|
|
|
pgprintk("%s: gva %lx error %x\n", __func__, gpa, error_code);
|
|
|
|
|
|
|
|
/* This path builds a PAE pagetable, we can map 2mb pages at maximum. */
|
|
|
|
return direct_page_fault(vcpu, gpa & PAGE_MASK, error_code, prefault,
|
2020-04-28 08:54:22 +08:00
|
|
|
PG_LEVEL_2M, false);
|
2019-12-07 07:57:24 +08:00
|
|
|
}
|
|
|
|
|
2017-07-14 09:30:40 +08:00
|
|
|
int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,
|
2017-08-12 00:36:43 +08:00
|
|
|
u64 fault_address, char *insn, int insn_len)
|
2017-07-14 09:30:40 +08:00
|
|
|
{
|
|
|
|
int r = 1;
|
2020-05-07 22:36:02 +08:00
|
|
|
u32 flags = vcpu->arch.apf.host_apf_flags;
|
2017-07-14 09:30:40 +08:00
|
|
|
|
2019-12-07 07:57:14 +08:00
|
|
|
#ifndef CONFIG_X86_64
|
|
|
|
/* A 64-bit CR2 should be impossible on 32-bit KVM. */
|
|
|
|
if (WARN_ON_ONCE(fault_address >> 32))
|
|
|
|
return -EFAULT;
|
|
|
|
#endif
|
|
|
|
|
x86/KVM/VMX: Add L1D flush logic
Add the logic for flushing L1D on VMENTER. The flush depends on the static
key being enabled and the new l1tf_flush_l1d flag being set.
The flags is set:
- Always, if the flush module parameter is 'always'
- Conditionally at:
- Entry to vcpu_run(), i.e. after executing user space
- From the sched_in notifier, i.e. when switching to a vCPU thread.
- From vmexit handlers which are considered unsafe, i.e. where
sensitive data can be brought into L1D:
- The emulator, which could be a good target for other speculative
execution-based threats,
- The MMU, which can bring host page tables in the L1 cache.
- External interrupts
- Nested operations that require the MMU (see above). That is
vmptrld, vmptrst, vmclear,vmwrite,vmread.
- When handling invept,invvpid
[ tglx: Split out from combo patch and reduced to a single flag ]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
2018-07-02 19:07:14 +08:00
|
|
|
vcpu->arch.l1tf_flush_l1d = true;
|
2020-05-07 22:36:02 +08:00
|
|
|
if (!flags) {
|
2017-07-14 09:30:40 +08:00
|
|
|
trace_kvm_page_fault(fault_address, error_code);
|
|
|
|
|
2017-08-12 00:36:43 +08:00
|
|
|
if (kvm_event_needs_reinjection(vcpu))
|
2017-07-14 09:30:40 +08:00
|
|
|
kvm_mmu_unprotect_page_virt(vcpu, fault_address);
|
|
|
|
r = kvm_mmu_page_fault(vcpu, fault_address, error_code, insn,
|
|
|
|
insn_len);
|
2020-05-07 22:36:02 +08:00
|
|
|
} else if (flags & KVM_PV_REASON_PAGE_NOT_PRESENT) {
|
2020-05-25 22:41:17 +08:00
|
|
|
vcpu->arch.apf.host_apf_flags = 0;
|
2017-07-14 09:30:40 +08:00
|
|
|
local_irq_disable();
|
2020-03-07 07:42:06 +08:00
|
|
|
kvm_async_pf_task_wait_schedule(fault_address);
|
2017-07-14 09:30:40 +08:00
|
|
|
local_irq_enable();
|
2020-05-07 22:36:02 +08:00
|
|
|
} else {
|
|
|
|
WARN_ONCE(1, "Unexpected host async PF flags: %x\n", flags);
|
2017-07-14 09:30:40 +08:00
|
|
|
}
|
2020-05-07 22:36:02 +08:00
|
|
|
|
2017-07-14 09:30:40 +08:00
|
|
|
return r;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_handle_page_fault);
|
|
|
|
|
2020-02-07 06:14:34 +08:00
|
|
|
int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
|
|
|
|
bool prefault)
|
2008-02-07 20:47:44 +08:00
|
|
|
{
|
2019-12-07 07:57:18 +08:00
|
|
|
int max_level;
|
2008-02-07 20:47:44 +08:00
|
|
|
|
2020-04-28 08:54:21 +08:00
|
|
|
for (max_level = KVM_MAX_HUGEPAGE_LEVEL;
|
2020-04-28 08:54:22 +08:00
|
|
|
max_level > PG_LEVEL_4K;
|
2019-12-07 07:57:18 +08:00
|
|
|
max_level--) {
|
|
|
|
int page_num = KVM_PAGES_PER_HPAGE(max_level);
|
2019-12-07 07:57:24 +08:00
|
|
|
gfn_t base = (gpa >> PAGE_SHIFT) & ~(page_num - 1);
|
2011-07-12 03:33:44 +08:00
|
|
|
|
2019-12-07 07:57:18 +08:00
|
|
|
if (kvm_mtrr_check_gfn_range_consistency(vcpu, base, page_num))
|
|
|
|
break;
|
2015-10-16 16:06:02 +08:00
|
|
|
}
|
2009-07-27 22:30:44 +08:00
|
|
|
|
2019-12-07 07:57:24 +08:00
|
|
|
return direct_page_fault(vcpu, gpa, error_code, prefault,
|
|
|
|
max_level, true);
|
2008-02-07 20:47:44 +08:00
|
|
|
}
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void nonpaging_init_context(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
|
|
|
context->page_fault = nonpaging_page_fault;
|
|
|
|
context->gva_to_gpa = nonpaging_gva_to_gpa;
|
2008-09-24 00:18:33 +08:00
|
|
|
context->sync_page = nonpaging_sync_page;
|
2020-03-24 08:42:57 +08:00
|
|
|
context->invlpg = NULL;
|
2011-03-09 15:43:51 +08:00
|
|
|
context->update_pte = nonpaging_update_pte;
|
[PATCH] KVM: MMU: Shadow page table caching
Define a hashtable for caching shadow page tables. Look up the cache on
context switch (cr3 change) or during page faults.
The key to the cache is a combination of
- the guest page table frame number
- the number of paging levels in the guest
* we can cache real mode, 32-bit mode, pae, and long mode page
tables simultaneously. this is useful for smp bootup.
- the guest page table table
* some kernels use a page as both a page table and a page directory. this
allows multiple shadow pages to exist for that page, one per level
- the "quadrant"
* 32-bit mode page tables span 4MB, whereas a shadow page table spans
2MB. similarly, a 32-bit page directory spans 4GB, while a shadow
page directory spans 1GB. the quadrant allows caching up to 4 shadow page
tables for one guest page in one level.
- a "metaphysical" bit
* for real mode, and for pse pages, there is no guest page table, so set
the bit to avoid write protecting the page.
Signed-off-by: Avi Kivity <avi@qumranet.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2007-01-06 08:36:43 +08:00
|
|
|
context->root_level = 0;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
context->shadow_root_level = PT32E_ROOT_LEVEL;
|
2010-09-10 23:30:39 +08:00
|
|
|
context->direct_map = true;
|
2010-09-10 23:31:01 +08:00
|
|
|
context->nx = false;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
static inline bool is_root_usable(struct kvm_mmu_root_info *root, gpa_t pgd,
|
2020-02-29 06:52:40 +08:00
|
|
|
union kvm_mmu_page_role role)
|
|
|
|
{
|
2020-03-21 05:28:32 +08:00
|
|
|
return (role.direct || pgd == root->pgd) &&
|
2020-06-23 04:20:34 +08:00
|
|
|
VALID_PAGE(root->hpa) && to_shadow_page(root->hpa) &&
|
|
|
|
role.word == to_shadow_page(root->hpa)->role.word;
|
2020-02-29 06:52:40 +08:00
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:20 +08:00
|
|
|
/*
|
2020-03-21 05:28:32 +08:00
|
|
|
* Find out if a previously cached root matching the new pgd/role is available.
|
2018-06-28 05:59:20 +08:00
|
|
|
* The current root is also inserted into the cache.
|
|
|
|
* If a matching root was found, it is assigned to kvm_mmu->root_hpa and true is
|
|
|
|
* returned.
|
|
|
|
* Otherwise, the LRU root from the cache is assigned to kvm_mmu->root_hpa and
|
|
|
|
* false is returned. This root should now be freed by the caller.
|
|
|
|
*/
|
2020-03-21 05:28:32 +08:00
|
|
|
static bool cached_root_available(struct kvm_vcpu *vcpu, gpa_t new_pgd,
|
2018-06-28 05:59:20 +08:00
|
|
|
union kvm_mmu_page_role new_role)
|
|
|
|
{
|
|
|
|
uint i;
|
|
|
|
struct kvm_mmu_root_info root;
|
2018-10-09 03:28:05 +08:00
|
|
|
struct kvm_mmu *mmu = vcpu->arch.mmu;
|
2018-06-28 05:59:20 +08:00
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
root.pgd = mmu->root_pgd;
|
2018-06-28 05:59:20 +08:00
|
|
|
root.hpa = mmu->root_hpa;
|
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
if (is_root_usable(&root, new_pgd, new_role))
|
2020-02-29 06:52:40 +08:00
|
|
|
return true;
|
|
|
|
|
2018-06-28 05:59:20 +08:00
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) {
|
|
|
|
swap(root, mmu->prev_roots[i]);
|
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
if (is_root_usable(&root, new_pgd, new_role))
|
2018-06-28 05:59:20 +08:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
mmu->root_hpa = root.hpa;
|
2020-03-21 05:28:32 +08:00
|
|
|
mmu->root_pgd = root.pgd;
|
2018-06-28 05:59:20 +08:00
|
|
|
|
|
|
|
return i < KVM_MMU_NUM_PREV_ROOTS;
|
|
|
|
}
|
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
static bool fast_pgd_switch(struct kvm_vcpu *vcpu, gpa_t new_pgd,
|
2020-03-21 05:28:26 +08:00
|
|
|
union kvm_mmu_page_role new_role)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
struct kvm_mmu *mmu = vcpu->arch.mmu;
|
2018-06-28 05:59:06 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* For now, limit the fast switch to 64-bit hosts+VMs in order to avoid
|
|
|
|
* having to deal with PDPTEs. We may add support for 32-bit hosts/VMs
|
|
|
|
* later if necessary.
|
|
|
|
*/
|
|
|
|
if (mmu->shadow_root_level >= PT64_ROOT_4LEVEL &&
|
2020-03-21 05:28:26 +08:00
|
|
|
mmu->root_level >= PT64_ROOT_4LEVEL)
|
2020-03-21 05:28:32 +08:00
|
|
|
return !mmu_check_root(vcpu, new_pgd >> PAGE_SHIFT) &&
|
|
|
|
cached_root_available(vcpu, new_pgd, new_role);
|
2018-06-28 05:59:06 +08:00
|
|
|
|
|
|
|
return false;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
static void __kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd,
|
2018-06-28 05:59:15 +08:00
|
|
|
union kvm_mmu_page_role new_role,
|
2020-03-21 05:28:27 +08:00
|
|
|
bool skip_tlb_flush, bool skip_mmu_sync)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2020-03-21 05:28:32 +08:00
|
|
|
if (!fast_pgd_switch(vcpu, new_pgd, new_role)) {
|
2020-03-21 05:28:26 +08:00
|
|
|
kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* It's possible that the cached previous root page is obsolete because
|
|
|
|
* of a change in the MMU generation number. However, changing the
|
|
|
|
* generation number is accompanied by KVM_REQ_MMU_RELOAD, which will
|
|
|
|
* free the root set here and allocate a new one.
|
|
|
|
*/
|
|
|
|
kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
|
|
|
|
|
2020-03-21 05:28:28 +08:00
|
|
|
if (!skip_mmu_sync || force_flush_and_sync_on_reuse)
|
2020-03-21 05:28:26 +08:00
|
|
|
kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
|
2020-03-21 05:28:28 +08:00
|
|
|
if (!skip_tlb_flush || force_flush_and_sync_on_reuse)
|
2020-03-21 05:28:26 +08:00
|
|
|
kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The last MMIO access's GVA and GPA are cached in the VCPU. When
|
|
|
|
* switching to a new CR3, that GVA->GPA mapping may no longer be
|
|
|
|
* valid. So clear any cached MMIO info even when we don't need to sync
|
|
|
|
* the shadow page tables.
|
|
|
|
*/
|
|
|
|
vcpu_clear_mmio_info(vcpu, MMIO_GVA_ANY);
|
|
|
|
|
2020-06-23 04:20:34 +08:00
|
|
|
__clear_sp_write_flooding_count(to_shadow_page(vcpu->arch.mmu->root_hpa));
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd, bool skip_tlb_flush,
|
2020-03-21 05:28:27 +08:00
|
|
|
bool skip_mmu_sync)
|
2018-06-28 05:59:09 +08:00
|
|
|
{
|
2020-03-21 05:28:32 +08:00
|
|
|
__kvm_mmu_new_pgd(vcpu, new_pgd, kvm_mmu_calc_root_page_role(vcpu),
|
2020-03-21 05:28:27 +08:00
|
|
|
skip_tlb_flush, skip_mmu_sync);
|
2018-06-28 05:59:09 +08:00
|
|
|
}
|
2020-03-21 05:28:32 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd);
|
2018-06-28 05:59:09 +08:00
|
|
|
|
2010-09-10 23:30:42 +08:00
|
|
|
static unsigned long get_cr3(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2010-12-05 23:30:00 +08:00
|
|
|
return kvm_read_cr3(vcpu);
|
2010-09-10 23:30:42 +08:00
|
|
|
}
|
|
|
|
|
2015-04-08 21:39:23 +08:00
|
|
|
static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,
|
2020-02-04 07:09:09 +08:00
|
|
|
unsigned int access, int *nr_present)
|
2011-07-12 03:33:44 +08:00
|
|
|
{
|
|
|
|
if (unlikely(is_mmio_spte(*sptep))) {
|
|
|
|
if (gfn != get_mmio_spte_gfn(*sptep)) {
|
|
|
|
mmu_spte_clear_no_track(sptep);
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
(*nr_present)++;
|
2015-04-08 21:39:23 +08:00
|
|
|
mark_mmio_spte(vcpu, sptep, gfn, access);
|
2011-07-12 03:33:44 +08:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2016-02-23 19:51:19 +08:00
|
|
|
static inline bool is_last_gpte(struct kvm_mmu *mmu,
|
|
|
|
unsigned level, unsigned gpte)
|
2012-09-13 01:46:56 +08:00
|
|
|
{
|
2016-02-23 19:51:19 +08:00
|
|
|
/*
|
|
|
|
* The RHS has bit 7 set iff level < mmu->last_nonleaf_level.
|
|
|
|
* If it is clear, there are no large pages at this level, so clear
|
|
|
|
* PT_PAGE_SIZE_MASK in gpte if that is the case.
|
|
|
|
*/
|
|
|
|
gpte &= level - mmu->last_nonleaf_level;
|
|
|
|
|
2017-10-05 17:10:23 +08:00
|
|
|
/*
|
2020-04-28 08:54:22 +08:00
|
|
|
* PG_LEVEL_4K always terminates. The RHS has bit 7 set
|
|
|
|
* iff level <= PG_LEVEL_4K, which for our purpose means
|
|
|
|
* level == PG_LEVEL_4K; set PT_PAGE_SIZE_MASK in gpte then.
|
2017-10-05 17:10:23 +08:00
|
|
|
*/
|
2020-04-28 08:54:22 +08:00
|
|
|
gpte |= level - PG_LEVEL_4K - 1;
|
2017-10-05 17:10:23 +08:00
|
|
|
|
2016-02-23 19:51:19 +08:00
|
|
|
return gpte & PT_PAGE_SIZE_MASK;
|
2012-09-13 01:46:56 +08:00
|
|
|
}
|
|
|
|
|
2013-08-05 16:07:12 +08:00
|
|
|
#define PTTYPE_EPT 18 /* arbitrary */
|
|
|
|
#define PTTYPE PTTYPE_EPT
|
|
|
|
#include "paging_tmpl.h"
|
|
|
|
#undef PTTYPE
|
|
|
|
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
#define PTTYPE 64
|
|
|
|
#include "paging_tmpl.h"
|
|
|
|
#undef PTTYPE
|
|
|
|
|
|
|
|
#define PTTYPE 32
|
|
|
|
#include "paging_tmpl.h"
|
|
|
|
#undef PTTYPE
|
|
|
|
|
2015-08-05 12:04:22 +08:00
|
|
|
static void
|
|
|
|
__reset_rsvds_bits_mask(struct kvm_vcpu *vcpu,
|
|
|
|
struct rsvd_bits_validate *rsvd_check,
|
|
|
|
int maxphyaddr, int level, bool nx, bool gbpages,
|
2015-09-23 05:02:14 +08:00
|
|
|
bool pse, bool amd)
|
2009-03-30 16:21:08 +08:00
|
|
|
{
|
|
|
|
u64 exb_bit_rsvd = 0;
|
2014-05-07 20:32:50 +08:00
|
|
|
u64 gbpages_bit_rsvd = 0;
|
2014-09-02 19:24:12 +08:00
|
|
|
u64 nonleaf_bit8_rsvd = 0;
|
2009-03-30 16:21:08 +08:00
|
|
|
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->bad_mt_xwr = 0;
|
2013-08-06 17:00:32 +08:00
|
|
|
|
2015-08-05 12:04:22 +08:00
|
|
|
if (!nx)
|
2009-03-30 16:21:08 +08:00
|
|
|
exb_bit_rsvd = rsvd_bits(63, 63);
|
2015-08-05 12:04:22 +08:00
|
|
|
if (!gbpages)
|
2014-05-07 20:32:50 +08:00
|
|
|
gbpages_bit_rsvd = rsvd_bits(7, 7);
|
2014-09-02 19:24:12 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Non-leaf PML4Es and PDPEs reserve bit 8 (which would be the G bit for
|
|
|
|
* leaf entries) on AMD CPUs only.
|
|
|
|
*/
|
2015-09-23 05:02:14 +08:00
|
|
|
if (amd)
|
2014-09-02 19:24:12 +08:00
|
|
|
nonleaf_bit8_rsvd = rsvd_bits(8, 8);
|
|
|
|
|
2015-08-05 12:04:22 +08:00
|
|
|
switch (level) {
|
2009-03-30 16:21:08 +08:00
|
|
|
case PT32_ROOT_LEVEL:
|
|
|
|
/* no rsvd bits for 2 level 4K page table entries */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][1] = 0;
|
|
|
|
rsvd_check->rsvd_bits_mask[0][0] = 0;
|
|
|
|
rsvd_check->rsvd_bits_mask[1][0] =
|
|
|
|
rsvd_check->rsvd_bits_mask[0][0];
|
2010-03-19 17:58:53 +08:00
|
|
|
|
2015-08-05 12:04:22 +08:00
|
|
|
if (!pse) {
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][1] = 0;
|
2010-03-19 17:58:53 +08:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2009-03-30 16:21:08 +08:00
|
|
|
if (is_cpuid_PSE36())
|
|
|
|
/* 36bits PSE 4MB page */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][1] = rsvd_bits(17, 21);
|
2009-03-30 16:21:08 +08:00
|
|
|
else
|
|
|
|
/* 32 bits PSE 4MB page */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][1] = rsvd_bits(13, 21);
|
2009-03-30 16:21:08 +08:00
|
|
|
break;
|
|
|
|
case PT32E_ROOT_LEVEL:
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][2] =
|
2009-03-31 23:03:45 +08:00
|
|
|
rsvd_bits(maxphyaddr, 63) |
|
2014-04-04 11:31:04 +08:00
|
|
|
rsvd_bits(5, 8) | rsvd_bits(1, 2); /* PDPTE */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][1] = exb_bit_rsvd |
|
2009-04-02 10:28:37 +08:00
|
|
|
rsvd_bits(maxphyaddr, 62); /* PDE */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][0] = exb_bit_rsvd |
|
2009-03-30 16:21:08 +08:00
|
|
|
rsvd_bits(maxphyaddr, 62); /* PTE */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][1] = exb_bit_rsvd |
|
2009-03-30 16:21:08 +08:00
|
|
|
rsvd_bits(maxphyaddr, 62) |
|
|
|
|
rsvd_bits(13, 20); /* large page */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][0] =
|
|
|
|
rsvd_check->rsvd_bits_mask[0][0];
|
2009-03-30 16:21:08 +08:00
|
|
|
break;
|
2017-08-24 20:27:55 +08:00
|
|
|
case PT64_ROOT_5LEVEL:
|
|
|
|
rsvd_check->rsvd_bits_mask[0][4] = exb_bit_rsvd |
|
|
|
|
nonleaf_bit8_rsvd | rsvd_bits(7, 7) |
|
|
|
|
rsvd_bits(maxphyaddr, 51);
|
|
|
|
rsvd_check->rsvd_bits_mask[1][4] =
|
|
|
|
rsvd_check->rsvd_bits_mask[0][4];
|
2019-01-26 02:23:17 +08:00
|
|
|
/* fall through */
|
2017-08-24 20:27:54 +08:00
|
|
|
case PT64_ROOT_4LEVEL:
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][3] = exb_bit_rsvd |
|
|
|
|
nonleaf_bit8_rsvd | rsvd_bits(7, 7) |
|
2009-04-02 10:28:37 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][2] = exb_bit_rsvd |
|
2020-06-30 19:07:20 +08:00
|
|
|
gbpages_bit_rsvd |
|
2009-03-30 16:21:08 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][1] = exb_bit_rsvd |
|
|
|
|
rsvd_bits(maxphyaddr, 51);
|
|
|
|
rsvd_check->rsvd_bits_mask[0][0] = exb_bit_rsvd |
|
|
|
|
rsvd_bits(maxphyaddr, 51);
|
|
|
|
rsvd_check->rsvd_bits_mask[1][3] =
|
|
|
|
rsvd_check->rsvd_bits_mask[0][3];
|
|
|
|
rsvd_check->rsvd_bits_mask[1][2] = exb_bit_rsvd |
|
2014-05-07 20:32:50 +08:00
|
|
|
gbpages_bit_rsvd | rsvd_bits(maxphyaddr, 51) |
|
2009-07-27 22:30:45 +08:00
|
|
|
rsvd_bits(13, 29);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][1] = exb_bit_rsvd |
|
2009-04-02 10:28:37 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51) |
|
|
|
|
rsvd_bits(13, 20); /* large page */
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][0] =
|
|
|
|
rsvd_check->rsvd_bits_mask[0][0];
|
2009-03-30 16:21:08 +08:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:22 +08:00
|
|
|
static void reset_rsvds_bits_mask(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context)
|
|
|
|
{
|
|
|
|
__reset_rsvds_bits_mask(vcpu, &context->guest_rsvd_check,
|
|
|
|
cpuid_maxphyaddr(vcpu), context->root_level,
|
2017-08-05 06:12:49 +08:00
|
|
|
context->nx,
|
|
|
|
guest_cpuid_has(vcpu, X86_FEATURE_GBPAGES),
|
2020-03-05 09:34:33 +08:00
|
|
|
is_pse(vcpu),
|
|
|
|
guest_cpuid_is_amd_or_hygon(vcpu));
|
2015-08-05 12:04:22 +08:00
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:23 +08:00
|
|
|
static void
|
|
|
|
__reset_rsvds_bits_mask_ept(struct rsvd_bits_validate *rsvd_check,
|
|
|
|
int maxphyaddr, bool execonly)
|
2013-08-06 17:00:32 +08:00
|
|
|
{
|
2015-09-23 16:34:26 +08:00
|
|
|
u64 bad_mt_xwr;
|
2013-08-06 17:00:32 +08:00
|
|
|
|
2017-08-24 20:27:55 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][4] =
|
|
|
|
rsvd_bits(maxphyaddr, 51) | rsvd_bits(3, 7);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][3] =
|
2013-08-06 17:00:32 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51) | rsvd_bits(3, 7);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][2] =
|
2013-08-06 17:00:32 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51) | rsvd_bits(3, 6);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][1] =
|
2013-08-06 17:00:32 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51) | rsvd_bits(3, 6);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[0][0] = rsvd_bits(maxphyaddr, 51);
|
2013-08-06 17:00:32 +08:00
|
|
|
|
|
|
|
/* large page */
|
2017-08-24 20:27:55 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][4] = rsvd_check->rsvd_bits_mask[0][4];
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][3] = rsvd_check->rsvd_bits_mask[0][3];
|
|
|
|
rsvd_check->rsvd_bits_mask[1][2] =
|
2013-08-06 17:00:32 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51) | rsvd_bits(12, 29);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][1] =
|
2013-08-06 17:00:32 +08:00
|
|
|
rsvd_bits(maxphyaddr, 51) | rsvd_bits(12, 20);
|
2015-08-05 12:04:21 +08:00
|
|
|
rsvd_check->rsvd_bits_mask[1][0] = rsvd_check->rsvd_bits_mask[0][0];
|
2013-08-06 17:00:32 +08:00
|
|
|
|
2015-09-23 16:34:26 +08:00
|
|
|
bad_mt_xwr = 0xFFull << (2 * 8); /* bits 3..5 must not be 2 */
|
|
|
|
bad_mt_xwr |= 0xFFull << (3 * 8); /* bits 3..5 must not be 3 */
|
|
|
|
bad_mt_xwr |= 0xFFull << (7 * 8); /* bits 3..5 must not be 7 */
|
|
|
|
bad_mt_xwr |= REPEAT_BYTE(1ull << 2); /* bits 0..2 must not be 010 */
|
|
|
|
bad_mt_xwr |= REPEAT_BYTE(1ull << 6); /* bits 0..2 must not be 110 */
|
|
|
|
if (!execonly) {
|
|
|
|
/* bits 0..2 must not be 100 unless VMX capabilities allow it */
|
|
|
|
bad_mt_xwr |= REPEAT_BYTE(1ull << 4);
|
2013-08-06 17:00:32 +08:00
|
|
|
}
|
2015-09-23 16:34:26 +08:00
|
|
|
rsvd_check->bad_mt_xwr = bad_mt_xwr;
|
2013-08-06 17:00:32 +08:00
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:23 +08:00
|
|
|
static void reset_rsvds_bits_mask_ept(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context, bool execonly)
|
|
|
|
{
|
|
|
|
__reset_rsvds_bits_mask_ept(&context->guest_rsvd_check,
|
|
|
|
cpuid_maxphyaddr(vcpu), execonly);
|
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:24 +08:00
|
|
|
/*
|
|
|
|
* the page table on host is the shadow page table for the page
|
|
|
|
* table in guest or amd nested guest, its mmu features completely
|
|
|
|
* follow the features in guest.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context)
|
|
|
|
{
|
2018-10-09 03:28:10 +08:00
|
|
|
bool uses_nx = context->nx ||
|
|
|
|
context->mmu_role.base.smep_andnot_wp;
|
2017-08-26 04:55:40 +08:00
|
|
|
struct rsvd_bits_validate *shadow_zero_check;
|
|
|
|
int i;
|
2016-03-09 21:28:02 +08:00
|
|
|
|
2015-09-23 05:02:14 +08:00
|
|
|
/*
|
|
|
|
* Passing "true" to the last argument is okay; it adds a check
|
|
|
|
* on bit 8 of the SPTEs which KVM doesn't use anyway.
|
|
|
|
*/
|
2017-08-26 04:55:40 +08:00
|
|
|
shadow_zero_check = &context->shadow_zero_check;
|
|
|
|
__reset_rsvds_bits_mask(vcpu, shadow_zero_check,
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
shadow_phys_bits,
|
2016-03-09 21:28:02 +08:00
|
|
|
context->shadow_root_level, uses_nx,
|
2017-08-05 06:12:49 +08:00
|
|
|
guest_cpuid_has(vcpu, X86_FEATURE_GBPAGES),
|
|
|
|
is_pse(vcpu), true);
|
2017-08-26 04:55:40 +08:00
|
|
|
|
|
|
|
if (!shadow_me_mask)
|
|
|
|
return;
|
|
|
|
|
|
|
|
for (i = context->shadow_root_level; --i >= 0;) {
|
|
|
|
shadow_zero_check->rsvd_bits_mask[0][i] &= ~shadow_me_mask;
|
|
|
|
shadow_zero_check->rsvd_bits_mask[1][i] &= ~shadow_me_mask;
|
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:24 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(reset_shadow_zero_bits_mask);
|
|
|
|
|
2015-09-23 05:02:14 +08:00
|
|
|
static inline bool boot_cpu_is_amd(void)
|
|
|
|
{
|
|
|
|
WARN_ON_ONCE(!tdp_enabled);
|
|
|
|
return shadow_x_mask == 0;
|
|
|
|
}
|
|
|
|
|
2015-08-05 12:04:24 +08:00
|
|
|
/*
|
|
|
|
* the direct page table on host, use as much mmu features as
|
|
|
|
* possible, however, kvm currently does not do execution-protection.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
reset_tdp_shadow_zero_bits_mask(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context)
|
|
|
|
{
|
2017-08-26 04:55:40 +08:00
|
|
|
struct rsvd_bits_validate *shadow_zero_check;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
shadow_zero_check = &context->shadow_zero_check;
|
|
|
|
|
2015-09-23 05:02:14 +08:00
|
|
|
if (boot_cpu_is_amd())
|
2017-08-26 04:55:40 +08:00
|
|
|
__reset_rsvds_bits_mask(vcpu, shadow_zero_check,
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
shadow_phys_bits,
|
2015-08-05 12:04:24 +08:00
|
|
|
context->shadow_root_level, false,
|
2016-03-29 23:41:58 +08:00
|
|
|
boot_cpu_has(X86_FEATURE_GBPAGES),
|
|
|
|
true, true);
|
2015-08-05 12:04:24 +08:00
|
|
|
else
|
2017-08-26 04:55:40 +08:00
|
|
|
__reset_rsvds_bits_mask_ept(shadow_zero_check,
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
shadow_phys_bits,
|
2015-08-05 12:04:24 +08:00
|
|
|
false);
|
|
|
|
|
2017-08-26 04:55:40 +08:00
|
|
|
if (!shadow_me_mask)
|
|
|
|
return;
|
|
|
|
|
|
|
|
for (i = context->shadow_root_level; --i >= 0;) {
|
|
|
|
shadow_zero_check->rsvd_bits_mask[0][i] &= ~shadow_me_mask;
|
|
|
|
shadow_zero_check->rsvd_bits_mask[1][i] &= ~shadow_me_mask;
|
|
|
|
}
|
2015-08-05 12:04:24 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* as the comments in reset_shadow_zero_bits_mask() except it
|
|
|
|
* is the shadow page table for intel nested guest.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
reset_ept_shadow_zero_bits_mask(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context, bool execonly)
|
|
|
|
{
|
|
|
|
__reset_rsvds_bits_mask_ept(&context->shadow_zero_check,
|
kvm: x86: Fix reserved bits related calculation errors caused by MKTME
Intel MKTME repurposes several high bits of physical address as 'keyID'
for memory encryption thus effectively reduces platform's maximum
physical address bits. Exactly how many bits are reduced is configured
by BIOS. To honor such HW behavior, the repurposed bits are reduced from
cpuinfo_x86->x86_phys_bits when MKTME is detected in CPU detection.
Similarly, AMD SME/SEV also reduces physical address bits for memory
encryption, and cpuinfo->x86_phys_bits is reduced too when SME/SEV is
detected, so for both MKTME and SME/SEV, boot_cpu_data.x86_phys_bits
doesn't hold physical address bits reported by CPUID anymore.
Currently KVM treats bits from boot_cpu_data.x86_phys_bits to 51 as
reserved bits, but it's not true anymore for MKTME, since MKTME treats
those reduced bits as 'keyID', but not reserved bits. Therefore
boot_cpu_data.x86_phys_bits cannot be used to calculate reserved bits
anymore, although we can still use it for AMD SME/SEV since SME/SEV
treats the reduced bits differently -- they are treated as reserved
bits, the same as other reserved bits in page table entity [1].
Fix by introducing a new 'shadow_phys_bits' variable in KVM x86 MMU code
to store the effective physical bits w/o reserved bits -- for MKTME,
it equals to physical address reported by CPUID, and for SME/SEV, it is
boot_cpu_data.x86_phys_bits.
Note that for the physical address bits reported to guest should remain
unchanged -- KVM should report physical address reported by CPUID to
guest, but not boot_cpu_data.x86_phys_bits. Because for Intel MKTME,
there's no harm if guest sets up 'keyID' bits in guest page table (since
MKTME only works at physical address level), and KVM doesn't even expose
MKTME to guest. Arguably, for AMD SME/SEV, guest is aware of SEV thus it
should adjust boot_cpu_data.x86_phys_bits when it detects SEV, therefore
KVM should still reports physcial address reported by CPUID to guest.
Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-05-03 18:08:53 +08:00
|
|
|
shadow_phys_bits, execonly);
|
2015-08-05 12:04:24 +08:00
|
|
|
}
|
|
|
|
|
2017-08-24 23:37:25 +08:00
|
|
|
#define BYTE_MASK(access) \
|
|
|
|
((1 & (access) ? 2 : 0) | \
|
|
|
|
(2 & (access) ? 4 : 0) | \
|
|
|
|
(3 & (access) ? 8 : 0) | \
|
|
|
|
(4 & (access) ? 16 : 0) | \
|
|
|
|
(5 & (access) ? 32 : 0) | \
|
|
|
|
(6 & (access) ? 64 : 0) | \
|
|
|
|
(7 & (access) ? 128 : 0))
|
|
|
|
|
|
|
|
|
2015-05-11 22:55:21 +08:00
|
|
|
static void update_permission_bitmask(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *mmu, bool ept)
|
KVM: MMU: Optimize pte permission checks
walk_addr_generic() permission checks are a maze of branchy code, which is
performed four times per lookup. It depends on the type of access, efer.nxe,
cr0.wp, cr4.smep, and in the near future, cr4.smap.
Optimize this away by precalculating all variants and storing them in a
bitmap. The bitmap is recalculated when rarely-changing variables change
(cr0, cr4) and is indexed by the often-changing variables (page fault error
code, pte access permissions).
The permission check is moved to the end of the loop, otherwise an SMEP
fault could be reported as a false positive, when PDE.U=1 but PTE.U=0.
Noted by Xiao Guangrong.
The result is short, branch-free code.
Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2012-09-12 19:52:00 +08:00
|
|
|
{
|
2017-08-24 23:37:25 +08:00
|
|
|
unsigned byte;
|
|
|
|
|
|
|
|
const u8 x = BYTE_MASK(ACC_EXEC_MASK);
|
|
|
|
const u8 w = BYTE_MASK(ACC_WRITE_MASK);
|
|
|
|
const u8 u = BYTE_MASK(ACC_USER_MASK);
|
|
|
|
|
|
|
|
bool cr4_smep = kvm_read_cr4_bits(vcpu, X86_CR4_SMEP) != 0;
|
|
|
|
bool cr4_smap = kvm_read_cr4_bits(vcpu, X86_CR4_SMAP) != 0;
|
|
|
|
bool cr0_wp = is_write_protection(vcpu);
|
KVM: MMU: Optimize pte permission checks
walk_addr_generic() permission checks are a maze of branchy code, which is
performed four times per lookup. It depends on the type of access, efer.nxe,
cr0.wp, cr4.smep, and in the near future, cr4.smap.
Optimize this away by precalculating all variants and storing them in a
bitmap. The bitmap is recalculated when rarely-changing variables change
(cr0, cr4) and is indexed by the often-changing variables (page fault error
code, pte access permissions).
The permission check is moved to the end of the loop, otherwise an SMEP
fault could be reported as a false positive, when PDE.U=1 but PTE.U=0.
Noted by Xiao Guangrong.
The result is short, branch-free code.
Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2012-09-12 19:52:00 +08:00
|
|
|
|
|
|
|
for (byte = 0; byte < ARRAY_SIZE(mmu->permissions); ++byte) {
|
2017-08-24 23:37:25 +08:00
|
|
|
unsigned pfec = byte << 1;
|
|
|
|
|
2014-04-01 17:46:34 +08:00
|
|
|
/*
|
2017-08-24 23:37:25 +08:00
|
|
|
* Each "*f" variable has a 1 bit for each UWX value
|
|
|
|
* that causes a fault with the given PFEC.
|
2014-04-01 17:46:34 +08:00
|
|
|
*/
|
KVM: MMU: Optimize pte permission checks
walk_addr_generic() permission checks are a maze of branchy code, which is
performed four times per lookup. It depends on the type of access, efer.nxe,
cr0.wp, cr4.smep, and in the near future, cr4.smap.
Optimize this away by precalculating all variants and storing them in a
bitmap. The bitmap is recalculated when rarely-changing variables change
(cr0, cr4) and is indexed by the often-changing variables (page fault error
code, pte access permissions).
The permission check is moved to the end of the loop, otherwise an SMEP
fault could be reported as a false positive, when PDE.U=1 but PTE.U=0.
Noted by Xiao Guangrong.
The result is short, branch-free code.
Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2012-09-12 19:52:00 +08:00
|
|
|
|
2017-08-24 23:37:25 +08:00
|
|
|
/* Faults from writes to non-writable pages */
|
2019-07-12 17:12:30 +08:00
|
|
|
u8 wf = (pfec & PFERR_WRITE_MASK) ? (u8)~w : 0;
|
2017-08-24 23:37:25 +08:00
|
|
|
/* Faults from user mode accesses to supervisor pages */
|
2019-07-12 17:12:30 +08:00
|
|
|
u8 uf = (pfec & PFERR_USER_MASK) ? (u8)~u : 0;
|
2017-08-24 23:37:25 +08:00
|
|
|
/* Faults from fetches of non-executable pages*/
|
2019-07-12 17:12:30 +08:00
|
|
|
u8 ff = (pfec & PFERR_FETCH_MASK) ? (u8)~x : 0;
|
2017-08-24 23:37:25 +08:00
|
|
|
/* Faults from kernel mode fetches of user pages */
|
|
|
|
u8 smepf = 0;
|
|
|
|
/* Faults from kernel mode accesses of user pages */
|
|
|
|
u8 smapf = 0;
|
|
|
|
|
|
|
|
if (!ept) {
|
|
|
|
/* Faults from kernel mode accesses to user pages */
|
|
|
|
u8 kf = (pfec & PFERR_USER_MASK) ? 0 : u;
|
|
|
|
|
|
|
|
/* Not really needed: !nx will cause pte.nx to fault */
|
|
|
|
if (!mmu->nx)
|
|
|
|
ff = 0;
|
|
|
|
|
|
|
|
/* Allow supervisor writes if !cr0.wp */
|
|
|
|
if (!cr0_wp)
|
|
|
|
wf = (pfec & PFERR_USER_MASK) ? wf : 0;
|
|
|
|
|
|
|
|
/* Disallow supervisor fetches of user code if cr4.smep */
|
|
|
|
if (cr4_smep)
|
|
|
|
smepf = (pfec & PFERR_FETCH_MASK) ? kf : 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* SMAP:kernel-mode data accesses from user-mode
|
|
|
|
* mappings should fault. A fault is considered
|
|
|
|
* as a SMAP violation if all of the following
|
2018-10-04 23:45:00 +08:00
|
|
|
* conditions are true:
|
2017-08-24 23:37:25 +08:00
|
|
|
* - X86_CR4_SMAP is set in CR4
|
|
|
|
* - A user page is accessed
|
|
|
|
* - The access is not a fetch
|
|
|
|
* - Page fault in kernel mode
|
|
|
|
* - if CPL = 3 or X86_EFLAGS_AC is clear
|
|
|
|
*
|
|
|
|
* Here, we cover the first three conditions.
|
|
|
|
* The fourth is computed dynamically in permission_fault();
|
|
|
|
* PFERR_RSVD_MASK bit will be set in PFEC if the access is
|
|
|
|
* *not* subject to SMAP restrictions.
|
|
|
|
*/
|
|
|
|
if (cr4_smap)
|
|
|
|
smapf = (pfec & (PFERR_RSVD_MASK|PFERR_FETCH_MASK)) ? 0 : kf;
|
KVM: MMU: Optimize pte permission checks
walk_addr_generic() permission checks are a maze of branchy code, which is
performed four times per lookup. It depends on the type of access, efer.nxe,
cr0.wp, cr4.smep, and in the near future, cr4.smap.
Optimize this away by precalculating all variants and storing them in a
bitmap. The bitmap is recalculated when rarely-changing variables change
(cr0, cr4) and is indexed by the often-changing variables (page fault error
code, pte access permissions).
The permission check is moved to the end of the loop, otherwise an SMEP
fault could be reported as a false positive, when PDE.U=1 but PTE.U=0.
Noted by Xiao Guangrong.
The result is short, branch-free code.
Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2012-09-12 19:52:00 +08:00
|
|
|
}
|
2017-08-24 23:37:25 +08:00
|
|
|
|
|
|
|
mmu->permissions[byte] = ff | uf | wf | smepf | smapf;
|
KVM: MMU: Optimize pte permission checks
walk_addr_generic() permission checks are a maze of branchy code, which is
performed four times per lookup. It depends on the type of access, efer.nxe,
cr0.wp, cr4.smep, and in the near future, cr4.smap.
Optimize this away by precalculating all variants and storing them in a
bitmap. The bitmap is recalculated when rarely-changing variables change
(cr0, cr4) and is indexed by the often-changing variables (page fault error
code, pte access permissions).
The permission check is moved to the end of the loop, otherwise an SMEP
fault could be reported as a false positive, when PDE.U=1 but PTE.U=0.
Noted by Xiao Guangrong.
The result is short, branch-free code.
Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2012-09-12 19:52:00 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-03-22 16:51:19 +08:00
|
|
|
/*
|
|
|
|
* PKU is an additional mechanism by which the paging controls access to
|
|
|
|
* user-mode addresses based on the value in the PKRU register. Protection
|
|
|
|
* key violations are reported through a bit in the page fault error code.
|
|
|
|
* Unlike other bits of the error code, the PK bit is not known at the
|
|
|
|
* call site of e.g. gva_to_gpa; it must be computed directly in
|
|
|
|
* permission_fault based on two bits of PKRU, on some machine state (CR4,
|
|
|
|
* CR0, EFER, CPL), and on other bits of the error code and the page tables.
|
|
|
|
*
|
|
|
|
* In particular the following conditions come from the error code, the
|
|
|
|
* page tables and the machine state:
|
|
|
|
* - PK is always zero unless CR4.PKE=1 and EFER.LMA=1
|
|
|
|
* - PK is always zero if RSVD=1 (reserved bit set) or F=1 (instruction fetch)
|
|
|
|
* - PK is always zero if U=0 in the page tables
|
|
|
|
* - PKRU.WD is ignored if CR0.WP=0 and the access is a supervisor access.
|
|
|
|
*
|
|
|
|
* The PKRU bitmask caches the result of these four conditions. The error
|
|
|
|
* code (minus the P bit) and the page table's U bit form an index into the
|
|
|
|
* PKRU bitmask. Two bits of the PKRU bitmask are then extracted and ANDed
|
|
|
|
* with the two bits of the PKRU register corresponding to the protection key.
|
|
|
|
* For the first three conditions above the bits will be 00, thus masking
|
|
|
|
* away both AD and WD. For all reads or if the last condition holds, WD
|
|
|
|
* only will be masked away.
|
|
|
|
*/
|
|
|
|
static void update_pkru_bitmask(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
|
|
|
|
bool ept)
|
|
|
|
{
|
|
|
|
unsigned bit;
|
|
|
|
bool wp;
|
|
|
|
|
|
|
|
if (ept) {
|
|
|
|
mmu->pkru_mask = 0;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* PKEY is enabled only if CR4.PKE and EFER.LMA are both set. */
|
|
|
|
if (!kvm_read_cr4_bits(vcpu, X86_CR4_PKE) || !is_long_mode(vcpu)) {
|
|
|
|
mmu->pkru_mask = 0;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
wp = is_write_protection(vcpu);
|
|
|
|
|
|
|
|
for (bit = 0; bit < ARRAY_SIZE(mmu->permissions); ++bit) {
|
|
|
|
unsigned pfec, pkey_bits;
|
|
|
|
bool check_pkey, check_write, ff, uf, wf, pte_user;
|
|
|
|
|
|
|
|
pfec = bit << 1;
|
|
|
|
ff = pfec & PFERR_FETCH_MASK;
|
|
|
|
uf = pfec & PFERR_USER_MASK;
|
|
|
|
wf = pfec & PFERR_WRITE_MASK;
|
|
|
|
|
|
|
|
/* PFEC.RSVD is replaced by ACC_USER_MASK. */
|
|
|
|
pte_user = pfec & PFERR_RSVD_MASK;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Only need to check the access which is not an
|
|
|
|
* instruction fetch and is to a user page.
|
|
|
|
*/
|
|
|
|
check_pkey = (!ff && pte_user);
|
|
|
|
/*
|
|
|
|
* write access is controlled by PKRU if it is a
|
|
|
|
* user access or CR0.WP = 1.
|
|
|
|
*/
|
|
|
|
check_write = check_pkey && wf && (uf || wp);
|
|
|
|
|
|
|
|
/* PKRU.AD stops both read and write access. */
|
|
|
|
pkey_bits = !!check_pkey;
|
|
|
|
/* PKRU.WD stops write access. */
|
|
|
|
pkey_bits |= (!!check_write) << 1;
|
|
|
|
|
|
|
|
mmu->pkru_mask |= (pkey_bits & 3) << pfec;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-02-23 19:51:19 +08:00
|
|
|
static void update_last_nonleaf_level(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)
|
2012-09-13 01:46:56 +08:00
|
|
|
{
|
2016-02-23 19:51:19 +08:00
|
|
|
unsigned root_level = mmu->root_level;
|
|
|
|
|
|
|
|
mmu->last_nonleaf_level = root_level;
|
|
|
|
if (root_level == PT32_ROOT_LEVEL && is_pse(vcpu))
|
|
|
|
mmu->last_nonleaf_level++;
|
2012-09-13 01:46:56 +08:00
|
|
|
}
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void paging64_init_context_common(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context,
|
|
|
|
int level)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2010-09-10 23:31:01 +08:00
|
|
|
context->nx = is_nx(vcpu);
|
2012-03-05 23:53:06 +08:00
|
|
|
context->root_level = level;
|
2010-09-10 23:31:01 +08:00
|
|
|
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, context);
|
2013-08-06 17:00:32 +08:00
|
|
|
update_permission_bitmask(vcpu, context, false);
|
2016-03-22 16:51:19 +08:00
|
|
|
update_pkru_bitmask(vcpu, context, false);
|
2016-02-23 19:51:19 +08:00
|
|
|
update_last_nonleaf_level(vcpu, context);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2013-10-02 22:56:16 +08:00
|
|
|
MMU_WARN_ON(!is_pae(vcpu));
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
context->page_fault = paging64_page_fault;
|
|
|
|
context->gva_to_gpa = paging64_gva_to_gpa;
|
2008-09-24 00:18:33 +08:00
|
|
|
context->sync_page = paging64_sync_page;
|
2008-09-24 00:18:35 +08:00
|
|
|
context->invlpg = paging64_invlpg;
|
2011-03-09 15:43:51 +08:00
|
|
|
context->update_pte = paging64_update_pte;
|
2007-01-06 08:36:40 +08:00
|
|
|
context->shadow_root_level = level;
|
2010-09-10 23:30:39 +08:00
|
|
|
context->direct_map = false;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void paging64_init_context(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context)
|
2007-01-06 08:36:40 +08:00
|
|
|
{
|
2017-08-24 20:27:55 +08:00
|
|
|
int root_level = is_la57_mode(vcpu) ?
|
|
|
|
PT64_ROOT_5LEVEL : PT64_ROOT_4LEVEL;
|
|
|
|
|
|
|
|
paging64_init_context_common(vcpu, context, root_level);
|
2007-01-06 08:36:40 +08:00
|
|
|
}
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void paging32_init_context(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2010-09-10 23:31:01 +08:00
|
|
|
context->nx = false;
|
2012-03-05 23:53:06 +08:00
|
|
|
context->root_level = PT32_ROOT_LEVEL;
|
2010-09-10 23:31:01 +08:00
|
|
|
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, context);
|
2013-08-06 17:00:32 +08:00
|
|
|
update_permission_bitmask(vcpu, context, false);
|
2016-03-22 16:51:19 +08:00
|
|
|
update_pkru_bitmask(vcpu, context, false);
|
2016-02-23 19:51:19 +08:00
|
|
|
update_last_nonleaf_level(vcpu, context);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
|
|
|
context->page_fault = paging32_page_fault;
|
|
|
|
context->gva_to_gpa = paging32_gva_to_gpa;
|
2008-09-24 00:18:33 +08:00
|
|
|
context->sync_page = paging32_sync_page;
|
2008-09-24 00:18:35 +08:00
|
|
|
context->invlpg = paging32_invlpg;
|
2011-03-09 15:43:51 +08:00
|
|
|
context->update_pte = paging32_update_pte;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
context->shadow_root_level = PT32E_ROOT_LEVEL;
|
2010-09-10 23:30:39 +08:00
|
|
|
context->direct_map = false;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void paging32E_init_context(struct kvm_vcpu *vcpu,
|
|
|
|
struct kvm_mmu *context)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2013-10-02 22:56:13 +08:00
|
|
|
paging64_init_context_common(vcpu, context, PT32E_ROOT_LEVEL);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2018-10-09 03:28:11 +08:00
|
|
|
static union kvm_mmu_extended_role kvm_calc_mmu_role_ext(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
|
|
|
union kvm_mmu_extended_role ext = {0};
|
|
|
|
|
2018-10-09 03:28:12 +08:00
|
|
|
ext.cr0_pg = !!is_paging(vcpu);
|
2019-05-01 01:33:26 +08:00
|
|
|
ext.cr4_pae = !!is_pae(vcpu);
|
2018-10-09 03:28:11 +08:00
|
|
|
ext.cr4_smep = !!kvm_read_cr4_bits(vcpu, X86_CR4_SMEP);
|
|
|
|
ext.cr4_smap = !!kvm_read_cr4_bits(vcpu, X86_CR4_SMAP);
|
|
|
|
ext.cr4_pse = !!is_pse(vcpu);
|
|
|
|
ext.cr4_pke = !!kvm_read_cr4_bits(vcpu, X86_CR4_PKE);
|
2019-02-01 00:09:23 +08:00
|
|
|
ext.maxphyaddr = cpuid_maxphyaddr(vcpu);
|
2018-10-09 03:28:11 +08:00
|
|
|
|
|
|
|
ext.valid = 1;
|
|
|
|
|
|
|
|
return ext;
|
|
|
|
}
|
|
|
|
|
2018-10-09 03:28:12 +08:00
|
|
|
static union kvm_mmu_role kvm_calc_mmu_role_common(struct kvm_vcpu *vcpu,
|
|
|
|
bool base_only)
|
|
|
|
{
|
|
|
|
union kvm_mmu_role role = {0};
|
|
|
|
|
|
|
|
role.base.access = ACC_ALL;
|
|
|
|
role.base.nxe = !!is_nx(vcpu);
|
|
|
|
role.base.cr0_wp = is_write_protection(vcpu);
|
|
|
|
role.base.smm = is_smm(vcpu);
|
|
|
|
role.base.guest_mode = is_guest_mode(vcpu);
|
|
|
|
|
|
|
|
if (base_only)
|
|
|
|
return role;
|
|
|
|
|
|
|
|
role.ext = kvm_calc_mmu_role_ext(vcpu);
|
|
|
|
|
|
|
|
return role;
|
|
|
|
}
|
|
|
|
|
|
|
|
static union kvm_mmu_role
|
|
|
|
kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, bool base_only)
|
2018-06-28 05:59:07 +08:00
|
|
|
{
|
2018-10-09 03:28:12 +08:00
|
|
|
union kvm_mmu_role role = kvm_calc_mmu_role_common(vcpu, base_only);
|
2018-06-28 05:59:07 +08:00
|
|
|
|
2018-10-09 03:28:12 +08:00
|
|
|
role.base.ad_disabled = (shadow_accessed_mask == 0);
|
2020-05-02 12:32:34 +08:00
|
|
|
role.base.level = vcpu->arch.tdp_level;
|
2018-10-09 03:28:12 +08:00
|
|
|
role.base.direct = true;
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
role.base.gpte_is_8_bytes = true;
|
2018-06-28 05:59:07 +08:00
|
|
|
|
|
|
|
return role;
|
|
|
|
}
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)
|
2008-02-07 20:47:44 +08:00
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
struct kvm_mmu *context = vcpu->arch.mmu;
|
2018-10-09 03:28:12 +08:00
|
|
|
union kvm_mmu_role new_role =
|
|
|
|
kvm_calc_tdp_mmu_root_page_role(vcpu, false);
|
2008-02-07 20:47:44 +08:00
|
|
|
|
2018-10-09 03:28:12 +08:00
|
|
|
if (new_role.as_u64 == context->mmu_role.as_u64)
|
|
|
|
return;
|
|
|
|
|
|
|
|
context->mmu_role.as_u64 = new_role.as_u64;
|
2020-02-07 06:14:34 +08:00
|
|
|
context->page_fault = kvm_tdp_page_fault;
|
2008-09-24 00:18:33 +08:00
|
|
|
context->sync_page = nonpaging_sync_page;
|
2020-03-24 08:42:57 +08:00
|
|
|
context->invlpg = NULL;
|
2011-03-09 15:43:51 +08:00
|
|
|
context->update_pte = nonpaging_update_pte;
|
2020-05-02 12:32:34 +08:00
|
|
|
context->shadow_root_level = vcpu->arch.tdp_level;
|
2010-09-10 23:30:39 +08:00
|
|
|
context->direct_map = true;
|
2020-03-03 10:02:39 +08:00
|
|
|
context->get_guest_pgd = get_cr3;
|
2011-07-28 16:36:17 +08:00
|
|
|
context->get_pdptr = kvm_pdptr_read;
|
2010-09-10 23:30:43 +08:00
|
|
|
context->inject_page_fault = kvm_inject_page_fault;
|
2008-02-07 20:47:44 +08:00
|
|
|
|
|
|
|
if (!is_paging(vcpu)) {
|
2010-09-10 23:31:01 +08:00
|
|
|
context->nx = false;
|
2008-02-07 20:47:44 +08:00
|
|
|
context->gva_to_gpa = nonpaging_gva_to_gpa;
|
|
|
|
context->root_level = 0;
|
|
|
|
} else if (is_long_mode(vcpu)) {
|
2010-09-10 23:31:01 +08:00
|
|
|
context->nx = is_nx(vcpu);
|
2017-08-24 20:27:55 +08:00
|
|
|
context->root_level = is_la57_mode(vcpu) ?
|
|
|
|
PT64_ROOT_5LEVEL : PT64_ROOT_4LEVEL;
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, context);
|
|
|
|
context->gva_to_gpa = paging64_gva_to_gpa;
|
2008-02-07 20:47:44 +08:00
|
|
|
} else if (is_pae(vcpu)) {
|
2010-09-10 23:31:01 +08:00
|
|
|
context->nx = is_nx(vcpu);
|
2008-02-07 20:47:44 +08:00
|
|
|
context->root_level = PT32E_ROOT_LEVEL;
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, context);
|
|
|
|
context->gva_to_gpa = paging64_gva_to_gpa;
|
2008-02-07 20:47:44 +08:00
|
|
|
} else {
|
2010-09-10 23:31:01 +08:00
|
|
|
context->nx = false;
|
2008-02-07 20:47:44 +08:00
|
|
|
context->root_level = PT32_ROOT_LEVEL;
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, context);
|
|
|
|
context->gva_to_gpa = paging32_gva_to_gpa;
|
2008-02-07 20:47:44 +08:00
|
|
|
}
|
|
|
|
|
2013-08-06 17:00:32 +08:00
|
|
|
update_permission_bitmask(vcpu, context, false);
|
2016-03-22 16:51:19 +08:00
|
|
|
update_pkru_bitmask(vcpu, context, false);
|
2016-02-23 19:51:19 +08:00
|
|
|
update_last_nonleaf_level(vcpu, context);
|
2015-08-05 12:04:24 +08:00
|
|
|
reset_tdp_shadow_zero_bits_mask(vcpu, context);
|
2008-02-07 20:47:44 +08:00
|
|
|
}
|
|
|
|
|
2018-10-09 03:28:12 +08:00
|
|
|
static union kvm_mmu_role
|
|
|
|
kvm_calc_shadow_mmu_root_page_role(struct kvm_vcpu *vcpu, bool base_only)
|
|
|
|
{
|
|
|
|
union kvm_mmu_role role = kvm_calc_mmu_role_common(vcpu, base_only);
|
|
|
|
|
|
|
|
role.base.smep_andnot_wp = role.ext.cr4_smep &&
|
|
|
|
!is_write_protection(vcpu);
|
|
|
|
role.base.smap_andnot_wp = role.ext.cr4_smap &&
|
|
|
|
!is_write_protection(vcpu);
|
|
|
|
role.base.direct = !is_paging(vcpu);
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
role.base.gpte_is_8_bytes = !!is_pae(vcpu);
|
2018-06-28 05:59:07 +08:00
|
|
|
|
|
|
|
if (!is_long_mode(vcpu))
|
2018-10-09 03:28:12 +08:00
|
|
|
role.base.level = PT32E_ROOT_LEVEL;
|
2018-06-28 05:59:07 +08:00
|
|
|
else if (is_la57_mode(vcpu))
|
2018-10-09 03:28:12 +08:00
|
|
|
role.base.level = PT64_ROOT_5LEVEL;
|
2018-06-28 05:59:07 +08:00
|
|
|
else
|
2018-10-09 03:28:12 +08:00
|
|
|
role.base.level = PT64_ROOT_4LEVEL;
|
2018-06-28 05:59:07 +08:00
|
|
|
|
|
|
|
return role;
|
|
|
|
}
|
|
|
|
|
2020-05-19 18:18:31 +08:00
|
|
|
void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu, u32 cr0, u32 cr4, u32 efer)
|
2018-06-28 05:59:07 +08:00
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
struct kvm_mmu *context = vcpu->arch.mmu;
|
2018-10-09 03:28:12 +08:00
|
|
|
union kvm_mmu_role new_role =
|
|
|
|
kvm_calc_shadow_mmu_root_page_role(vcpu, false);
|
|
|
|
|
|
|
|
if (new_role.as_u64 == context->mmu_role.as_u64)
|
|
|
|
return;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2020-05-19 18:18:31 +08:00
|
|
|
if (!(cr0 & X86_CR0_PG))
|
2013-10-02 22:56:13 +08:00
|
|
|
nonpaging_init_context(vcpu, context);
|
2020-05-19 18:18:31 +08:00
|
|
|
else if (efer & EFER_LMA)
|
2013-10-02 22:56:13 +08:00
|
|
|
paging64_init_context(vcpu, context);
|
2020-05-19 18:18:31 +08:00
|
|
|
else if (cr4 & X86_CR4_PAE)
|
2013-10-02 22:56:13 +08:00
|
|
|
paging32E_init_context(vcpu, context);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
else
|
2013-10-02 22:56:13 +08:00
|
|
|
paging32_init_context(vcpu, context);
|
2008-12-22 01:20:09 +08:00
|
|
|
|
2018-10-09 03:28:12 +08:00
|
|
|
context->mmu_role.as_u64 = new_role.as_u64;
|
2015-08-05 12:04:24 +08:00
|
|
|
reset_shadow_zero_bits_mask(vcpu, context);
|
2010-09-10 23:30:44 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_init_shadow_mmu);
|
|
|
|
|
2018-10-09 03:28:11 +08:00
|
|
|
static union kvm_mmu_role
|
|
|
|
kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty,
|
2020-03-03 10:02:36 +08:00
|
|
|
bool execonly, u8 level)
|
2018-06-28 05:59:07 +08:00
|
|
|
{
|
2019-03-08 07:27:43 +08:00
|
|
|
union kvm_mmu_role role = {0};
|
2018-10-09 03:28:08 +08:00
|
|
|
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
/* SMM flag is inherited from root_mmu */
|
|
|
|
role.base.smm = vcpu->arch.root_mmu.mmu_role.base.smm;
|
2018-06-28 05:59:07 +08:00
|
|
|
|
2020-03-03 10:02:36 +08:00
|
|
|
role.base.level = level;
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
role.base.gpte_is_8_bytes = true;
|
2018-10-09 03:28:11 +08:00
|
|
|
role.base.direct = false;
|
|
|
|
role.base.ad_disabled = !accessed_dirty;
|
|
|
|
role.base.guest_mode = true;
|
|
|
|
role.base.access = ACC_ALL;
|
2018-06-28 05:59:07 +08:00
|
|
|
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
/*
|
|
|
|
* WP=1 and NOT_WP=1 is an impossible combination, use WP and the
|
|
|
|
* SMAP variation to denote shadow EPT entries.
|
|
|
|
*/
|
|
|
|
role.base.cr0_wp = true;
|
|
|
|
role.base.smap_andnot_wp = true;
|
|
|
|
|
2019-03-08 07:27:43 +08:00
|
|
|
role.ext = kvm_calc_mmu_role_ext(vcpu);
|
2018-10-09 03:28:11 +08:00
|
|
|
role.ext.execonly = execonly;
|
2018-06-28 05:59:07 +08:00
|
|
|
|
|
|
|
return role;
|
|
|
|
}
|
|
|
|
|
2017-03-30 17:55:30 +08:00
|
|
|
void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly,
|
2018-06-28 05:59:11 +08:00
|
|
|
bool accessed_dirty, gpa_t new_eptp)
|
2013-08-05 16:07:16 +08:00
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
struct kvm_mmu *context = vcpu->arch.mmu;
|
2020-03-03 10:02:36 +08:00
|
|
|
u8 level = vmx_eptp_page_walk_level(new_eptp);
|
2018-10-09 03:28:11 +08:00
|
|
|
union kvm_mmu_role new_role =
|
|
|
|
kvm_calc_shadow_ept_root_page_role(vcpu, accessed_dirty,
|
2020-03-03 10:02:36 +08:00
|
|
|
execonly, level);
|
2018-10-09 03:28:11 +08:00
|
|
|
|
2020-03-21 05:28:32 +08:00
|
|
|
__kvm_mmu_new_pgd(vcpu, new_eptp, new_role.base, true, true);
|
2018-10-09 03:28:11 +08:00
|
|
|
|
|
|
|
if (new_role.as_u64 == context->mmu_role.as_u64)
|
|
|
|
return;
|
2013-10-02 22:56:14 +08:00
|
|
|
|
2020-03-03 10:02:36 +08:00
|
|
|
context->shadow_root_level = level;
|
2013-08-05 16:07:16 +08:00
|
|
|
|
|
|
|
context->nx = true;
|
2017-03-30 17:55:30 +08:00
|
|
|
context->ept_ad = accessed_dirty;
|
2013-08-05 16:07:16 +08:00
|
|
|
context->page_fault = ept_page_fault;
|
|
|
|
context->gva_to_gpa = ept_gva_to_gpa;
|
|
|
|
context->sync_page = ept_sync_page;
|
|
|
|
context->invlpg = ept_invlpg;
|
|
|
|
context->update_pte = ept_update_pte;
|
2020-03-03 10:02:36 +08:00
|
|
|
context->root_level = level;
|
2013-08-05 16:07:16 +08:00
|
|
|
context->direct_map = false;
|
2018-10-09 03:28:11 +08:00
|
|
|
context->mmu_role.as_u64 = new_role.as_u64;
|
2018-10-09 03:28:06 +08:00
|
|
|
|
2013-08-05 16:07:16 +08:00
|
|
|
update_permission_bitmask(vcpu, context, true);
|
2016-03-22 16:51:19 +08:00
|
|
|
update_pkru_bitmask(vcpu, context, true);
|
2017-10-05 17:10:22 +08:00
|
|
|
update_last_nonleaf_level(vcpu, context);
|
2013-08-05 16:07:16 +08:00
|
|
|
reset_rsvds_bits_mask_ept(vcpu, context, execonly);
|
2015-08-05 12:04:24 +08:00
|
|
|
reset_ept_shadow_zero_bits_mask(vcpu, context, execonly);
|
2013-08-05 16:07:16 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_init_shadow_ept_mmu);
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void init_kvm_softmmu(struct kvm_vcpu *vcpu)
|
2010-09-10 23:30:44 +08:00
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
struct kvm_mmu *context = vcpu->arch.mmu;
|
2013-10-02 22:56:14 +08:00
|
|
|
|
2020-05-19 18:18:31 +08:00
|
|
|
kvm_init_shadow_mmu(vcpu,
|
|
|
|
kvm_read_cr0_bits(vcpu, X86_CR0_PG),
|
|
|
|
kvm_read_cr4_bits(vcpu, X86_CR4_PAE),
|
|
|
|
vcpu->arch.efer);
|
|
|
|
|
2020-03-03 10:02:39 +08:00
|
|
|
context->get_guest_pgd = get_cr3;
|
2013-10-02 22:56:14 +08:00
|
|
|
context->get_pdptr = kvm_pdptr_read;
|
|
|
|
context->inject_page_fault = kvm_inject_page_fault;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu)
|
2010-09-10 23:30:54 +08:00
|
|
|
{
|
2018-10-09 03:28:13 +08:00
|
|
|
union kvm_mmu_role new_role = kvm_calc_mmu_role_common(vcpu, false);
|
2010-09-10 23:30:54 +08:00
|
|
|
struct kvm_mmu *g_context = &vcpu->arch.nested_mmu;
|
|
|
|
|
2018-10-09 03:28:13 +08:00
|
|
|
if (new_role.as_u64 == g_context->mmu_role.as_u64)
|
|
|
|
return;
|
|
|
|
|
|
|
|
g_context->mmu_role.as_u64 = new_role.as_u64;
|
2020-03-03 10:02:39 +08:00
|
|
|
g_context->get_guest_pgd = get_cr3;
|
2011-07-28 16:36:17 +08:00
|
|
|
g_context->get_pdptr = kvm_pdptr_read;
|
2010-09-10 23:30:54 +08:00
|
|
|
g_context->inject_page_fault = kvm_inject_page_fault;
|
|
|
|
|
2020-03-24 08:42:57 +08:00
|
|
|
/*
|
|
|
|
* L2 page tables are never shadowed, so there is no need to sync
|
|
|
|
* SPTEs.
|
|
|
|
*/
|
|
|
|
g_context->invlpg = NULL;
|
|
|
|
|
2010-09-10 23:30:54 +08:00
|
|
|
/*
|
2018-10-09 03:28:05 +08:00
|
|
|
* Note that arch.mmu->gva_to_gpa translates l2_gpa to l1_gpa using
|
2015-12-31 00:26:17 +08:00
|
|
|
* L1's nested page tables (e.g. EPT12). The nested translation
|
|
|
|
* of l2_gva to l1_gpa is done by arch.nested_mmu.gva_to_gpa using
|
|
|
|
* L2's page tables as the first level of translation and L1's
|
|
|
|
* nested page tables as the second level of translation. Basically
|
|
|
|
* the gva_to_gpa functions between mmu and nested_mmu are swapped.
|
2010-09-10 23:30:54 +08:00
|
|
|
*/
|
|
|
|
if (!is_paging(vcpu)) {
|
2010-09-10 23:31:01 +08:00
|
|
|
g_context->nx = false;
|
2010-09-10 23:30:54 +08:00
|
|
|
g_context->root_level = 0;
|
|
|
|
g_context->gva_to_gpa = nonpaging_gva_to_gpa_nested;
|
|
|
|
} else if (is_long_mode(vcpu)) {
|
2010-09-10 23:31:01 +08:00
|
|
|
g_context->nx = is_nx(vcpu);
|
2017-08-24 20:27:55 +08:00
|
|
|
g_context->root_level = is_la57_mode(vcpu) ?
|
|
|
|
PT64_ROOT_5LEVEL : PT64_ROOT_4LEVEL;
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, g_context);
|
2010-09-10 23:30:54 +08:00
|
|
|
g_context->gva_to_gpa = paging64_gva_to_gpa_nested;
|
|
|
|
} else if (is_pae(vcpu)) {
|
2010-09-10 23:31:01 +08:00
|
|
|
g_context->nx = is_nx(vcpu);
|
2010-09-10 23:30:54 +08:00
|
|
|
g_context->root_level = PT32E_ROOT_LEVEL;
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, g_context);
|
2010-09-10 23:30:54 +08:00
|
|
|
g_context->gva_to_gpa = paging64_gva_to_gpa_nested;
|
|
|
|
} else {
|
2010-09-10 23:31:01 +08:00
|
|
|
g_context->nx = false;
|
2010-09-10 23:30:54 +08:00
|
|
|
g_context->root_level = PT32_ROOT_LEVEL;
|
2012-03-05 23:53:06 +08:00
|
|
|
reset_rsvds_bits_mask(vcpu, g_context);
|
2010-09-10 23:30:54 +08:00
|
|
|
g_context->gva_to_gpa = paging32_gva_to_gpa_nested;
|
|
|
|
}
|
|
|
|
|
2013-08-06 17:00:32 +08:00
|
|
|
update_permission_bitmask(vcpu, g_context, false);
|
2016-03-22 16:51:19 +08:00
|
|
|
update_pkru_bitmask(vcpu, g_context, false);
|
2016-02-23 19:51:19 +08:00
|
|
|
update_last_nonleaf_level(vcpu, g_context);
|
2010-09-10 23:30:54 +08:00
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:10 +08:00
|
|
|
void kvm_init_mmu(struct kvm_vcpu *vcpu, bool reset_roots)
|
2008-02-07 20:47:44 +08:00
|
|
|
{
|
2018-06-28 05:59:10 +08:00
|
|
|
if (reset_roots) {
|
2018-06-28 05:59:20 +08:00
|
|
|
uint i;
|
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->root_hpa = INVALID_PAGE;
|
2018-06-28 05:59:20 +08:00
|
|
|
|
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->prev_roots[i] = KVM_MMU_ROOT_INFO_INVALID;
|
2018-06-28 05:59:10 +08:00
|
|
|
}
|
|
|
|
|
2010-09-10 23:30:54 +08:00
|
|
|
if (mmu_is_nested(vcpu))
|
2014-12-23 20:39:46 +08:00
|
|
|
init_kvm_nested_mmu(vcpu);
|
2010-09-10 23:30:54 +08:00
|
|
|
else if (tdp_enabled)
|
2014-12-23 20:39:46 +08:00
|
|
|
init_kvm_tdp_mmu(vcpu);
|
2008-02-07 20:47:44 +08:00
|
|
|
else
|
2014-12-23 20:39:46 +08:00
|
|
|
init_kvm_softmmu(vcpu);
|
2008-02-07 20:47:44 +08:00
|
|
|
}
|
2018-06-28 05:59:10 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_init_mmu);
|
2008-02-07 20:47:44 +08:00
|
|
|
|
2018-06-28 05:59:07 +08:00
|
|
|
static union kvm_mmu_page_role
|
|
|
|
kvm_mmu_calc_root_page_role(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2018-10-09 03:28:12 +08:00
|
|
|
union kvm_mmu_role role;
|
|
|
|
|
2018-06-28 05:59:07 +08:00
|
|
|
if (tdp_enabled)
|
2018-10-09 03:28:12 +08:00
|
|
|
role = kvm_calc_tdp_mmu_root_page_role(vcpu, true);
|
2018-06-28 05:59:07 +08:00
|
|
|
else
|
2018-10-09 03:28:12 +08:00
|
|
|
role = kvm_calc_shadow_mmu_root_page_role(vcpu, true);
|
|
|
|
|
|
|
|
return role.base;
|
2018-06-28 05:59:07 +08:00
|
|
|
}
|
2008-02-07 20:47:44 +08:00
|
|
|
|
2013-10-02 22:56:13 +08:00
|
|
|
void kvm_mmu_reset_context(struct kvm_vcpu *vcpu)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2013-10-02 22:56:12 +08:00
|
|
|
kvm_mmu_unload(vcpu);
|
2018-06-28 05:59:10 +08:00
|
|
|
kvm_init_mmu(vcpu, true);
|
2007-06-04 20:58:30 +08:00
|
|
|
}
|
2007-10-10 14:26:45 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_reset_context);
|
2007-06-04 20:58:30 +08:00
|
|
|
|
|
|
|
int kvm_mmu_load(struct kvm_vcpu *vcpu)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2007-01-06 08:36:53 +08:00
|
|
|
int r;
|
|
|
|
|
2007-01-06 08:36:54 +08:00
|
|
|
r = mmu_topup_memory_caches(vcpu);
|
2007-06-04 20:58:30 +08:00
|
|
|
if (r)
|
|
|
|
goto out;
|
2009-05-13 05:55:45 +08:00
|
|
|
r = mmu_alloc_roots(vcpu);
|
2013-05-09 14:45:12 +08:00
|
|
|
kvm_mmu_sync_roots(vcpu);
|
2009-05-13 05:55:45 +08:00
|
|
|
if (r)
|
|
|
|
goto out;
|
2020-03-05 16:52:50 +08:00
|
|
|
kvm_mmu_load_pgd(vcpu);
|
2020-03-21 05:28:21 +08:00
|
|
|
kvm_x86_ops.tlb_flush_current(vcpu);
|
2007-01-06 08:36:53 +08:00
|
|
|
out:
|
|
|
|
return r;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
2007-06-04 20:58:30 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_load);
|
|
|
|
|
|
|
|
void kvm_mmu_unload(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2018-10-09 03:28:08 +08:00
|
|
|
kvm_mmu_free_roots(vcpu, &vcpu->arch.root_mmu, KVM_MMU_ROOTS_ALL);
|
|
|
|
WARN_ON(VALID_PAGE(vcpu->arch.root_mmu.root_hpa));
|
|
|
|
kvm_mmu_free_roots(vcpu, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL);
|
|
|
|
WARN_ON(VALID_PAGE(vcpu->arch.guest_mmu.root_hpa));
|
2007-06-04 20:58:30 +08:00
|
|
|
}
|
2010-09-10 23:31:03 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_unload);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2007-05-01 21:53:31 +08:00
|
|
|
static void mmu_pte_write_new_pte(struct kvm_vcpu *vcpu,
|
2011-03-28 10:29:27 +08:00
|
|
|
struct kvm_mmu_page *sp, u64 *spte,
|
|
|
|
const void *new)
|
2007-05-01 21:53:31 +08:00
|
|
|
{
|
2020-04-28 08:54:22 +08:00
|
|
|
if (sp->role.level != PG_LEVEL_4K) {
|
2009-07-27 22:30:46 +08:00
|
|
|
++vcpu->kvm->stat.mmu_pde_zapped;
|
|
|
|
return;
|
2008-06-12 07:32:40 +08:00
|
|
|
}
|
2007-05-01 21:53:31 +08:00
|
|
|
|
2007-11-18 22:37:07 +08:00
|
|
|
++vcpu->kvm->stat.mmu_pte_updated;
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu->update_pte(vcpu, sp, spte, new);
|
2007-05-01 21:53:31 +08:00
|
|
|
}
|
|
|
|
|
2007-11-21 08:06:21 +08:00
|
|
|
static bool need_remote_flush(u64 old, u64 new)
|
|
|
|
{
|
|
|
|
if (!is_shadow_present_pte(old))
|
|
|
|
return false;
|
|
|
|
if (!is_shadow_present_pte(new))
|
|
|
|
return true;
|
|
|
|
if ((old ^ new) & PT64_BASE_ADDR_MASK)
|
|
|
|
return true;
|
2013-08-05 16:07:14 +08:00
|
|
|
old ^= shadow_nx_mask;
|
|
|
|
new ^= shadow_nx_mask;
|
2007-11-21 08:06:21 +08:00
|
|
|
return (old & ~new & PT64_PERM_MASK) != 0;
|
|
|
|
}
|
|
|
|
|
2011-09-22 16:57:23 +08:00
|
|
|
static u64 mmu_pte_write_fetch_gpte(struct kvm_vcpu *vcpu, gpa_t *gpa,
|
2018-11-01 05:53:57 +08:00
|
|
|
int *bytes)
|
2007-01-06 08:36:44 +08:00
|
|
|
{
|
2018-11-01 05:53:57 +08:00
|
|
|
u64 gentry = 0;
|
2011-09-22 16:57:23 +08:00
|
|
|
int r;
|
2010-03-15 19:59:53 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Assume that the pte write on a page table of the same type
|
2011-03-04 19:00:00 +08:00
|
|
|
* as the current vcpu paging mode since we update the sptes only
|
|
|
|
* when they have the same mode.
|
2010-03-15 19:59:53 +08:00
|
|
|
*/
|
2011-09-22 16:57:23 +08:00
|
|
|
if (is_pae(vcpu) && *bytes == 4) {
|
2010-03-15 19:59:53 +08:00
|
|
|
/* Handle a 32-bit guest writing two halves of a 64-bit gpte */
|
2011-09-22 16:57:23 +08:00
|
|
|
*gpa &= ~(gpa_t)7;
|
|
|
|
*bytes = 8;
|
2010-03-15 19:59:57 +08:00
|
|
|
}
|
|
|
|
|
2018-11-01 05:53:57 +08:00
|
|
|
if (*bytes == 4 || *bytes == 8) {
|
|
|
|
r = kvm_vcpu_read_guest_atomic(vcpu, *gpa, &gentry, *bytes);
|
|
|
|
if (r)
|
|
|
|
gentry = 0;
|
2010-03-15 19:59:53 +08:00
|
|
|
}
|
|
|
|
|
2011-09-22 16:57:23 +08:00
|
|
|
return gentry;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we're seeing too many writes to a page, it may no longer be a page table,
|
|
|
|
* or we may be forking, in which case it is better to unmap the page.
|
|
|
|
*/
|
2011-12-16 18:18:10 +08:00
|
|
|
static bool detect_write_flooding(struct kvm_mmu_page *sp)
|
2011-09-22 16:57:23 +08:00
|
|
|
{
|
KVM: MMU: improve write flooding detected
Detecting write-flooding does not work well, when we handle page written, if
the last speculative spte is not accessed, we treat the page is
write-flooding, however, we can speculative spte on many path, such as pte
prefetch, page synced, that means the last speculative spte may be not point
to the written page and the written page can be accessed via other sptes, so
depends on the Accessed bit of the last speculative spte is not enough
Instead of detected page accessed, we can detect whether the spte is accessed
after it is written, if the spte is not accessed but it is written frequently,
we treat is not a page table or it not used for a long time
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-09-22 16:58:36 +08:00
|
|
|
/*
|
|
|
|
* Skip write-flooding detected for the sp whose level is 1, because
|
|
|
|
* it can become unsync, then the guest page is not write-protected.
|
|
|
|
*/
|
2020-04-28 08:54:22 +08:00
|
|
|
if (sp->role.level == PG_LEVEL_4K)
|
KVM: MMU: improve write flooding detected
Detecting write-flooding does not work well, when we handle page written, if
the last speculative spte is not accessed, we treat the page is
write-flooding, however, we can speculative spte on many path, such as pte
prefetch, page synced, that means the last speculative spte may be not point
to the written page and the written page can be accessed via other sptes, so
depends on the Accessed bit of the last speculative spte is not enough
Instead of detected page accessed, we can detect whether the spte is accessed
after it is written, if the spte is not accessed but it is written frequently,
we treat is not a page table or it not used for a long time
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-09-22 16:58:36 +08:00
|
|
|
return false;
|
2010-04-16 16:35:54 +08:00
|
|
|
|
2016-02-24 17:51:12 +08:00
|
|
|
atomic_inc(&sp->write_flooding_count);
|
|
|
|
return atomic_read(&sp->write_flooding_count) >= 3;
|
2011-09-22 16:57:23 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Misaligned accesses are too much trouble to fix up; also, they usually
|
|
|
|
* indicate a page is not used as a page table.
|
|
|
|
*/
|
|
|
|
static bool detect_write_misaligned(struct kvm_mmu_page *sp, gpa_t gpa,
|
|
|
|
int bytes)
|
|
|
|
{
|
|
|
|
unsigned offset, pte_size, misaligned;
|
|
|
|
|
|
|
|
pgprintk("misaligned: gpa %llx bytes %d role %x\n",
|
|
|
|
gpa, bytes, sp->role.word);
|
|
|
|
|
|
|
|
offset = offset_in_page(gpa);
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
pte_size = sp->role.gpte_is_8_bytes ? 8 : 4;
|
2011-09-22 16:57:55 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Sometimes, the OS only writes the last one bytes to update status
|
|
|
|
* bits, for example, in linux, andb instruction is used in clear_bit().
|
|
|
|
*/
|
|
|
|
if (!(offset & (pte_size - 1)) && bytes == 1)
|
|
|
|
return false;
|
|
|
|
|
2011-09-22 16:57:23 +08:00
|
|
|
misaligned = (offset ^ (offset + bytes - 1)) & ~(pte_size - 1);
|
|
|
|
misaligned |= bytes < 4;
|
|
|
|
|
|
|
|
return misaligned;
|
|
|
|
}
|
|
|
|
|
|
|
|
static u64 *get_written_sptes(struct kvm_mmu_page *sp, gpa_t gpa, int *nspte)
|
|
|
|
{
|
|
|
|
unsigned page_offset, quadrant;
|
|
|
|
u64 *spte;
|
|
|
|
int level;
|
|
|
|
|
|
|
|
page_offset = offset_in_page(gpa);
|
|
|
|
level = sp->role.level;
|
|
|
|
*nspte = 1;
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
if (!sp->role.gpte_is_8_bytes) {
|
2011-09-22 16:57:23 +08:00
|
|
|
page_offset <<= 1; /* 32->64 */
|
|
|
|
/*
|
|
|
|
* A 32-bit pde maps 4MB while the shadow pdes map
|
|
|
|
* only 2MB. So we need to double the offset again
|
|
|
|
* and zap two pdes instead of one.
|
|
|
|
*/
|
|
|
|
if (level == PT32_ROOT_LEVEL) {
|
|
|
|
page_offset &= ~7; /* kill rounding error */
|
|
|
|
page_offset <<= 1;
|
|
|
|
*nspte = 2;
|
|
|
|
}
|
|
|
|
quadrant = page_offset >> PAGE_SHIFT;
|
|
|
|
page_offset &= ~PAGE_MASK;
|
|
|
|
if (quadrant != sp->role.quadrant)
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
spte = &sp->spt[page_offset / sizeof(*spte)];
|
|
|
|
return spte;
|
|
|
|
}
|
|
|
|
|
2020-03-03 10:02:34 +08:00
|
|
|
/*
|
|
|
|
* Ignore various flags when determining if a SPTE can be immediately
|
|
|
|
* overwritten for the current MMU.
|
|
|
|
* - level: explicitly checked in mmu_pte_write_new_pte(), and will never
|
|
|
|
* match the current MMU role, as MMU's level tracks the root level.
|
|
|
|
* - access: updated based on the new guest PTE
|
|
|
|
* - quadrant: handled by get_written_sptes()
|
|
|
|
* - invalid: always false (loop only walks valid shadow pages)
|
|
|
|
*/
|
|
|
|
static const union kvm_mmu_page_role role_ign = {
|
|
|
|
.level = 0xf,
|
|
|
|
.access = 0x7,
|
|
|
|
.quadrant = 0x3,
|
|
|
|
.invalid = 0x1,
|
|
|
|
};
|
|
|
|
|
2016-02-24 17:51:16 +08:00
|
|
|
static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
|
2016-10-25 15:50:42 +08:00
|
|
|
const u8 *new, int bytes,
|
|
|
|
struct kvm_page_track_notifier_node *node)
|
2011-09-22 16:57:23 +08:00
|
|
|
{
|
|
|
|
gfn_t gfn = gpa >> PAGE_SHIFT;
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
LIST_HEAD(invalid_list);
|
|
|
|
u64 entry, gentry, *spte;
|
|
|
|
int npte;
|
2016-02-24 18:21:55 +08:00
|
|
|
bool remote_flush, local_flush;
|
2011-09-22 16:57:23 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If we don't have indirect shadow pages, it means no page is
|
|
|
|
* write-protected, so we can exit simply.
|
|
|
|
*/
|
locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE()
Please do not apply this to mainline directly, instead please re-run the
coccinelle script shown below and apply its output.
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't harmful, and changing them results in
churn.
However, for some features, the read/write distinction is critical to
correct operation. To distinguish these cases, separate read/write
accessors must be used. This patch migrates (most) remaining
ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following
coccinelle script:
----
// Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and
// WRITE_ONCE()
// $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davem@davemloft.net
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-24 05:07:29 +08:00
|
|
|
if (!READ_ONCE(vcpu->kvm->arch.indirect_shadow_pages))
|
2011-09-22 16:57:23 +08:00
|
|
|
return;
|
|
|
|
|
2016-02-24 18:21:55 +08:00
|
|
|
remote_flush = local_flush = false;
|
2011-09-22 16:57:23 +08:00
|
|
|
|
|
|
|
pgprintk("%s: gpa %llx bytes %d\n", __func__, gpa, bytes);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* No need to care whether allocation memory is successful
|
|
|
|
* or not since pte prefetch is skiped if it does not have
|
|
|
|
* enough objects in the cache.
|
|
|
|
*/
|
|
|
|
mmu_topup_memory_caches(vcpu);
|
|
|
|
|
|
|
|
spin_lock(&vcpu->kvm->mmu_lock);
|
2018-11-01 05:53:57 +08:00
|
|
|
|
|
|
|
gentry = mmu_pte_write_fetch_gpte(vcpu, &gpa, &bytes);
|
|
|
|
|
2011-09-22 16:57:23 +08:00
|
|
|
++vcpu->kvm->stat.mmu_pte_write;
|
2011-11-28 20:41:00 +08:00
|
|
|
kvm_mmu_audit(vcpu, AUDIT_PRE_PTE_WRITE);
|
2011-09-22 16:57:23 +08:00
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-28 09:06:00 +08:00
|
|
|
for_each_gfn_indirect_valid_sp(vcpu->kvm, sp, gfn) {
|
KVM: MMU: improve write flooding detected
Detecting write-flooding does not work well, when we handle page written, if
the last speculative spte is not accessed, we treat the page is
write-flooding, however, we can speculative spte on many path, such as pte
prefetch, page synced, that means the last speculative spte may be not point
to the written page and the written page can be accessed via other sptes, so
depends on the Accessed bit of the last speculative spte is not enough
Instead of detected page accessed, we can detect whether the spte is accessed
after it is written, if the spte is not accessed but it is written frequently,
we treat is not a page table or it not used for a long time
Signed-off-by: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-09-22 16:58:36 +08:00
|
|
|
if (detect_write_misaligned(sp, gpa, bytes) ||
|
2011-12-16 18:18:10 +08:00
|
|
|
detect_write_flooding(sp)) {
|
2016-02-24 18:21:55 +08:00
|
|
|
kvm_mmu_prepare_zap_page(vcpu->kvm, sp, &invalid_list);
|
2007-11-18 22:37:07 +08:00
|
|
|
++vcpu->kvm->stat.mmu_flooded;
|
2007-01-06 08:36:48 +08:00
|
|
|
continue;
|
|
|
|
}
|
2011-09-22 16:57:23 +08:00
|
|
|
|
|
|
|
spte = get_written_sptes(sp, gpa, &npte);
|
|
|
|
if (!spte)
|
|
|
|
continue;
|
|
|
|
|
2010-06-04 21:56:59 +08:00
|
|
|
local_flush = true;
|
2007-03-08 23:13:32 +08:00
|
|
|
while (npte--) {
|
2018-10-09 03:28:10 +08:00
|
|
|
u32 base_role = vcpu->arch.mmu->mmu_role.base.word;
|
|
|
|
|
2007-11-21 08:06:21 +08:00
|
|
|
entry = *spte;
|
2011-05-15 23:27:52 +08:00
|
|
|
mmu_page_zap_pte(vcpu->kvm, sp, spte);
|
2010-07-16 11:19:51 +08:00
|
|
|
if (gentry &&
|
2020-03-03 10:02:34 +08:00
|
|
|
!((sp->role.word ^ base_role) & ~role_ign.word) &&
|
|
|
|
rmap_can_add(vcpu))
|
2011-03-28 10:29:27 +08:00
|
|
|
mmu_pte_write_new_pte(vcpu, sp, spte, &gentry);
|
2013-01-30 22:45:01 +08:00
|
|
|
if (need_remote_flush(entry, *spte))
|
2010-06-04 21:56:59 +08:00
|
|
|
remote_flush = true;
|
2007-03-08 23:13:32 +08:00
|
|
|
++spte;
|
2007-01-06 08:36:45 +08:00
|
|
|
}
|
|
|
|
}
|
2016-02-24 18:21:55 +08:00
|
|
|
kvm_mmu_flush_or_zap(vcpu, &invalid_list, remote_flush, local_flush);
|
2011-11-28 20:41:00 +08:00
|
|
|
kvm_mmu_audit(vcpu, AUDIT_POST_PTE_WRITE);
|
2007-12-21 08:18:26 +08:00
|
|
|
spin_unlock(&vcpu->kvm->mmu_lock);
|
2007-01-06 08:36:44 +08:00
|
|
|
}
|
|
|
|
|
2007-01-06 08:36:45 +08:00
|
|
|
int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva)
|
|
|
|
{
|
2007-12-21 08:18:22 +08:00
|
|
|
gpa_t gpa;
|
|
|
|
int r;
|
2007-01-06 08:36:45 +08:00
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->direct_map)
|
2009-08-27 18:37:06 +08:00
|
|
|
return 0;
|
|
|
|
|
2010-02-10 20:21:32 +08:00
|
|
|
gpa = kvm_mmu_gva_to_gpa_read(vcpu, gva, NULL);
|
2007-12-21 08:18:22 +08:00
|
|
|
|
|
|
|
r = kvm_mmu_unprotect_page(vcpu->kvm, gpa >> PAGE_SHIFT);
|
2011-09-22 17:02:48 +08:00
|
|
|
|
2007-12-21 08:18:22 +08:00
|
|
|
return r;
|
2007-01-06 08:36:45 +08:00
|
|
|
}
|
2008-07-19 13:57:05 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_unprotect_page_virt);
|
2007-01-06 08:36:45 +08:00
|
|
|
|
2019-12-07 07:57:14 +08:00
|
|
|
int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
|
2010-12-21 18:12:07 +08:00
|
|
|
void *insn, int insn_len)
|
2007-10-29 00:48:59 +08:00
|
|
|
{
|
2020-02-19 07:03:08 +08:00
|
|
|
int r, emulation_type = EMULTYPE_PF;
|
2018-10-09 03:28:05 +08:00
|
|
|
bool direct = vcpu->arch.mmu->direct_map;
|
2007-10-29 00:48:59 +08:00
|
|
|
|
2019-12-07 07:57:29 +08:00
|
|
|
if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root_hpa)))
|
KVM: x86/mmu: Move root_hpa validity checks to top of page fault handler
Add a check on root_hpa at the beginning of the page fault handler to
consolidate several checks on root_hpa that are scattered throughout the
page fault code. This is a preparatory step towards eventually removing
such checks altogether, or at the very least WARNing if an invalid root
is encountered. Remove only the checks that can be easily audited to
confirm that root_hpa cannot be invalidated between their current
location and the new check in kvm_mmu_page_fault(), and aren't currently
protected by mmu_lock, i.e. keep the checks in __direct_map() and
FNAME(fetch) for the time being.
The root_hpa checks that are consolidate were all added by commit
37f6a4e237303 ("KVM: x86: handle invalid root_hpa everywhere")
which was a follow up to a bug fix for __direct_map(), commit
989c6b34f6a94 ("KVM: MMU: handle invalid root_hpa at __direct_map")
At the time, nested VMX had, in hindsight, crazy handling of nested
interrupts and would trigger a nested VM-Exit in ->interrupt_allowed(),
and thus unexpectedly reset the MMU in flows such as can_do_async_pf().
Now that the wonky nested VM-Exit behavior is gone, the root_hpa checks
are bogus and confusing, e.g. it's not at all obvious what they actually
protect against, and at first glance they appear to be broken since many
of them run without holding mmu_lock.
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-12-07 07:57:27 +08:00
|
|
|
return RET_PF_RETRY;
|
|
|
|
|
2017-08-17 21:03:32 +08:00
|
|
|
r = RET_PF_INVALID;
|
2016-02-22 16:23:41 +08:00
|
|
|
if (unlikely(error_code & PFERR_RSVD_MASK)) {
|
2019-12-07 07:57:14 +08:00
|
|
|
r = handle_mmio_page_fault(vcpu, cr2_or_gpa, direct);
|
2018-08-24 04:56:50 +08:00
|
|
|
if (r == RET_PF_EMULATE)
|
2016-02-22 16:23:41 +08:00
|
|
|
goto emulate;
|
|
|
|
}
|
2007-10-29 00:48:59 +08:00
|
|
|
|
2017-08-17 21:03:32 +08:00
|
|
|
if (r == RET_PF_INVALID) {
|
2020-02-07 06:14:34 +08:00
|
|
|
r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa,
|
|
|
|
lower_32_bits(error_code), false);
|
2017-08-17 21:03:32 +08:00
|
|
|
WARN_ON(r == RET_PF_INVALID);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (r == RET_PF_RETRY)
|
|
|
|
return 1;
|
2007-10-29 00:48:59 +08:00
|
|
|
if (r < 0)
|
2016-02-22 16:23:41 +08:00
|
|
|
return r;
|
2007-10-29 00:48:59 +08:00
|
|
|
|
2016-11-24 01:01:38 +08:00
|
|
|
/*
|
|
|
|
* Before emulating the instruction, check if the error code
|
|
|
|
* was due to a RO violation while translating the guest page.
|
|
|
|
* This can occur when using nested virtualization with nested
|
|
|
|
* paging in both guests. If true, we simply unprotect the page
|
|
|
|
* and resume the guest.
|
|
|
|
*/
|
2018-10-09 03:28:05 +08:00
|
|
|
if (vcpu->arch.mmu->direct_map &&
|
2016-11-28 21:39:58 +08:00
|
|
|
(error_code & PFERR_NESTED_GUEST_PAGE) == PFERR_NESTED_GUEST_PAGE) {
|
2019-12-07 07:57:14 +08:00
|
|
|
kvm_mmu_unprotect_page(vcpu->kvm, gpa_to_gfn(cr2_or_gpa));
|
2016-11-24 01:01:38 +08:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2018-08-24 04:56:50 +08:00
|
|
|
/*
|
|
|
|
* vcpu->arch.mmu.page_fault returned RET_PF_EMULATE, but we can still
|
|
|
|
* optimistically try to just unprotect the page and let the processor
|
|
|
|
* re-execute the instruction that caused the page fault. Do not allow
|
|
|
|
* retrying MMIO emulation, as it's not only pointless but could also
|
|
|
|
* cause us to enter an infinite loop because the processor will keep
|
KVM: x86: Do not re-{try,execute} after failed emulation in L2
Commit a6f177efaa58 ("KVM: Reenter guest after emulation failure if
due to access to non-mmio address") added reexecute_instruction() to
handle the scenario where two (or more) vCPUS race to write a shadowed
page, i.e. reexecute_instruction() is intended to return true if and
only if the instruction being emulated was accessing a shadowed page.
As L0 is only explicitly shadowing L1 tables, an emulation failure of
a nested VM instruction cannot be due to a race to write a shadowed
page and so should never be re-executed.
This fixes an issue where an "MMIO" emulation failure[1] in L2 is all
but guaranteed to result in an infinite loop when TDP is enabled.
Because "cr2" is actually an L2 GPA when TDP is enabled, calling
kvm_mmu_gva_to_gpa_write() to translate cr2 in the non-direct mapped
case (L2 is never direct mapped) will almost always yield UNMAPPED_GVA
and cause reexecute_instruction() to immediately return true. The
!mmio_info_in_cache() check in kvm_mmu_page_fault() doesn't catch this
case because mmio_info_in_cache() returns false for a nested MMU (the
MMIO caching currently handles L1 only, e.g. to cache nested guests'
GPAs we'd have to manually flush the cache when switching between
VMs and when L1 updated its page tables controlling the nested guest).
Way back when, commit 68be0803456b ("KVM: x86: never re-execute
instruction with enabled tdp") changed reexecute_instruction() to
always return false when using TDP under the assumption that KVM would
only get into the emulator for MMIO. Commit 95b3cf69bdf8 ("KVM: x86:
let reexecute_instruction work for tdp") effectively reverted that
behavior in order to handle the scenario where emulation failed due to
an access from L1 to the shadow page tables for L2, but it didn't
account for the case where emulation failed in L2 with TDP enabled.
All of the above logic also applies to retry_instruction(), added by
commit 1cb3f3ae5a38 ("KVM: x86: retry non-page-table writing
instructions"). An indefinite loop in retry_instruction() should be
impossible as it protects against retrying the same instruction over
and over, but it's still correct to not retry an L2 instruction in
the first place.
Fix the immediate issue by adding a check for a nested guest when
determining whether or not to allow retry in kvm_mmu_page_fault().
In addition to fixing the immediate bug, add WARN_ON_ONCE in the
retry functions since they are not designed to handle nested cases,
i.e. they need to be modified even if there is some scenario in the
future where we want to allow retrying a nested guest.
[1] This issue was encountered after commit 3a2936dedd20 ("kvm: mmu:
Don't expose private memslots to L2") changed the page fault path
to return KVM_PFN_NOSLOT when translating an L2 access to a
prive memslot. Returning KVM_PFN_NOSLOT is semantically correct
when we want to hide a memslot from L2, i.e. there effectively is
no defined memory region for L2, but it has the unfortunate side
effect of making KVM think the GFN is a MMIO page, thus triggering
emulation. The failure occurred with in-development code that
deliberately exposed a private memslot to L2, which L2 accessed
with an instruction that is not emulated by KVM.
Fixes: 95b3cf69bdf8 ("KVM: x86: let reexecute_instruction work for tdp")
Fixes: 1cb3f3ae5a38 ("KVM: x86: retry non-page-table writing instructions")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Cc: Jim Mattson <jmattson@google.com>
Cc: Krish Sadhukhan <krish.sadhukhan@oracle.com>
Cc: Xiao Guangrong <xiaoguangrong@tencent.com>
Cc: stable@vger.kernel.org
Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
2018-08-24 04:56:51 +08:00
|
|
|
* faulting on the non-existent MMIO address. Retrying an instruction
|
|
|
|
* from a nested guest is also pointless and dangerous as we are only
|
|
|
|
* explicitly shadowing L1's page tables, i.e. unprotecting something
|
|
|
|
* for L1 isn't going to magically fix whatever issue cause L2 to fail.
|
2018-08-24 04:56:50 +08:00
|
|
|
*/
|
2019-12-07 07:57:14 +08:00
|
|
|
if (!mmio_info_in_cache(vcpu, cr2_or_gpa, direct) && !is_guest_mode(vcpu))
|
2020-02-19 07:03:08 +08:00
|
|
|
emulation_type |= EMULTYPE_ALLOW_RETRY_PF;
|
2016-02-22 16:23:41 +08:00
|
|
|
emulate:
|
2017-12-05 00:57:40 +08:00
|
|
|
/*
|
|
|
|
* On AMD platforms, under certain conditions insn_len may be zero on #NPF.
|
|
|
|
* This can happen if a guest gets a page-fault on data access but the HW
|
|
|
|
* table walker is not able to read the instruction page (e.g instruction
|
|
|
|
* page is not present in memory). In those cases we simply restart the
|
2019-02-16 01:24:12 +08:00
|
|
|
* guest, with the exception of AMD Erratum 1096 which is unrecoverable.
|
2017-12-05 00:57:40 +08:00
|
|
|
*/
|
2019-02-16 01:24:12 +08:00
|
|
|
if (unlikely(insn && !insn_len)) {
|
2020-03-22 04:26:00 +08:00
|
|
|
if (!kvm_x86_ops.need_emulation_on_page_fault(vcpu))
|
2019-02-16 01:24:12 +08:00
|
|
|
return 1;
|
|
|
|
}
|
2017-12-05 00:57:40 +08:00
|
|
|
|
2019-12-07 07:57:14 +08:00
|
|
|
return x86_emulate_instruction(vcpu, cr2_or_gpa, emulation_type, insn,
|
KVM: x86: Remove emulation_result enums, EMULATE_{DONE,FAIL,USER_EXIT}
Deferring emulation failure handling (in some cases) to the caller of
x86_emulate_instruction() has proven fragile, e.g. multiple instances of
KVM not setting run->exit_reason on EMULATE_FAIL, largely due to it
being difficult to discern what emulation types can return what result,
and which combination of types and results are handled where.
Now that x86_emulate_instruction() always handles emulation failure,
i.e. EMULATION_FAIL is only referenced in callers, remove the
emulation_result enums entirely. Per KVM's existing exit handling
conventions, return '0' and '1' for "exit to userspace" and "resume
guest" respectively. Doing so cleans up many callers, e.g. they can
return kvm_emulate_instruction() directly instead of having to interpret
its result.
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-08-28 05:40:38 +08:00
|
|
|
insn_len);
|
2007-10-29 00:48:59 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_page_fault);
|
|
|
|
|
2020-03-24 08:42:57 +08:00
|
|
|
void kvm_mmu_invalidate_gva(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
|
|
|
|
gva_t gva, hpa_t root_hpa)
|
2008-09-24 00:18:35 +08:00
|
|
|
{
|
2018-06-28 05:59:20 +08:00
|
|
|
int i;
|
2018-06-28 05:59:16 +08:00
|
|
|
|
2020-03-24 08:42:57 +08:00
|
|
|
/* It's actually a GPA for vcpu->arch.guest_mmu. */
|
|
|
|
if (mmu != &vcpu->arch.guest_mmu) {
|
|
|
|
/* INVLPG on a non-canonical address is a NOP according to the SDM. */
|
|
|
|
if (is_noncanonical_address(gva, vcpu))
|
|
|
|
return;
|
|
|
|
|
|
|
|
kvm_x86_ops.tlb_flush_gva(vcpu, gva);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!mmu->invlpg)
|
2018-06-30 04:10:05 +08:00
|
|
|
return;
|
|
|
|
|
2020-03-24 08:42:57 +08:00
|
|
|
if (root_hpa == INVALID_PAGE) {
|
|
|
|
mmu->invlpg(vcpu, gva, mmu->root_hpa);
|
2018-06-28 05:59:18 +08:00
|
|
|
|
2020-03-24 08:42:57 +08:00
|
|
|
/*
|
|
|
|
* INVLPG is required to invalidate any global mappings for the VA,
|
|
|
|
* irrespective of PCID. Since it would take us roughly similar amount
|
|
|
|
* of work to determine whether any of the prev_root mappings of the VA
|
|
|
|
* is marked global, or to just sync it blindly, so we might as well
|
|
|
|
* just always sync it.
|
|
|
|
*
|
|
|
|
* Mappings not reachable via the current cr3 or the prev_roots will be
|
|
|
|
* synced when switching to that cr3, so nothing needs to be done here
|
|
|
|
* for them.
|
|
|
|
*/
|
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
|
|
|
|
if (VALID_PAGE(mmu->prev_roots[i].hpa))
|
|
|
|
mmu->invlpg(vcpu, gva, mmu->prev_roots[i].hpa);
|
|
|
|
} else {
|
|
|
|
mmu->invlpg(vcpu, gva, root_hpa);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_invalidate_gva);
|
2018-06-28 05:59:18 +08:00
|
|
|
|
2020-03-24 08:42:57 +08:00
|
|
|
void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
|
|
|
|
{
|
|
|
|
kvm_mmu_invalidate_gva(vcpu, vcpu->arch.mmu, gva, INVALID_PAGE);
|
2008-09-24 00:18:35 +08:00
|
|
|
++vcpu->stat.invlpg;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_invlpg);
|
|
|
|
|
2020-03-24 08:42:57 +08:00
|
|
|
|
2018-06-28 05:59:14 +08:00
|
|
|
void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid)
|
|
|
|
{
|
2018-10-09 03:28:05 +08:00
|
|
|
struct kvm_mmu *mmu = vcpu->arch.mmu;
|
2018-06-30 04:10:05 +08:00
|
|
|
bool tlb_flush = false;
|
2018-06-28 05:59:20 +08:00
|
|
|
uint i;
|
2018-06-28 05:59:14 +08:00
|
|
|
|
|
|
|
if (pcid == kvm_get_active_pcid(vcpu)) {
|
2018-06-28 05:59:16 +08:00
|
|
|
mmu->invlpg(vcpu, gva, mmu->root_hpa);
|
2018-06-30 04:10:05 +08:00
|
|
|
tlb_flush = true;
|
2018-06-28 05:59:14 +08:00
|
|
|
}
|
|
|
|
|
2018-06-28 05:59:20 +08:00
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) {
|
|
|
|
if (VALID_PAGE(mmu->prev_roots[i].hpa) &&
|
2020-03-21 05:28:32 +08:00
|
|
|
pcid == kvm_get_pcid(vcpu, mmu->prev_roots[i].pgd)) {
|
2018-06-28 05:59:20 +08:00
|
|
|
mmu->invlpg(vcpu, gva, mmu->prev_roots[i].hpa);
|
|
|
|
tlb_flush = true;
|
|
|
|
}
|
2018-06-28 05:59:18 +08:00
|
|
|
}
|
2018-06-28 05:59:15 +08:00
|
|
|
|
2018-06-30 04:10:05 +08:00
|
|
|
if (tlb_flush)
|
2020-03-22 04:26:00 +08:00
|
|
|
kvm_x86_ops.tlb_flush_gva(vcpu, gva);
|
2018-06-30 04:10:05 +08:00
|
|
|
|
2018-06-28 05:59:14 +08:00
|
|
|
++vcpu->stat.invlpg;
|
|
|
|
|
|
|
|
/*
|
2018-06-28 05:59:20 +08:00
|
|
|
* Mappings not reachable via the current cr3 or the prev_roots will be
|
|
|
|
* synced when switching to that cr3, so nothing needs to be done here
|
|
|
|
* for them.
|
2018-06-28 05:59:14 +08:00
|
|
|
*/
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_invpcid_gva);
|
|
|
|
|
2020-03-03 07:57:03 +08:00
|
|
|
void kvm_configure_mmu(bool enable_tdp, int tdp_page_level)
|
2008-02-07 20:47:41 +08:00
|
|
|
{
|
2020-03-03 07:57:02 +08:00
|
|
|
tdp_enabled = enable_tdp;
|
2020-03-03 07:57:03 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* max_page_level reflects the capabilities of KVM's MMU irrespective
|
|
|
|
* of kernel support, e.g. KVM may be capable of using 1GB pages when
|
|
|
|
* the kernel is not. But, KVM never creates a page size greater than
|
|
|
|
* what is used by the kernel for any given HVA, i.e. the kernel's
|
|
|
|
* capabilities are ultimately consulted by kvm_mmu_hugepage_adjust().
|
|
|
|
*/
|
|
|
|
if (tdp_enabled)
|
|
|
|
max_page_level = tdp_page_level;
|
|
|
|
else if (boot_cpu_has(X86_FEATURE_GBPAGES))
|
2020-04-28 08:54:22 +08:00
|
|
|
max_page_level = PG_LEVEL_1G;
|
2020-03-03 07:57:03 +08:00
|
|
|
else
|
2020-04-28 08:54:22 +08:00
|
|
|
max_page_level = PG_LEVEL_2M;
|
2008-02-07 20:47:41 +08:00
|
|
|
}
|
2020-03-03 07:57:02 +08:00
|
|
|
EXPORT_SYMBOL_GPL(kvm_configure_mmu);
|
2019-02-06 05:01:19 +08:00
|
|
|
|
|
|
|
/* The return value indicates if tlb flush on all vcpus is needed. */
|
|
|
|
typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head);
|
|
|
|
|
|
|
|
/* The caller should hold mmu-lock before calling this function. */
|
|
|
|
static __always_inline bool
|
|
|
|
slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot,
|
|
|
|
slot_level_handler fn, int start_level, int end_level,
|
|
|
|
gfn_t start_gfn, gfn_t end_gfn, bool lock_flush_tlb)
|
|
|
|
{
|
|
|
|
struct slot_rmap_walk_iterator iterator;
|
|
|
|
bool flush = false;
|
|
|
|
|
|
|
|
for_each_slot_rmap_range(memslot, start_level, end_level, start_gfn,
|
|
|
|
end_gfn, &iterator) {
|
|
|
|
if (iterator.rmap)
|
|
|
|
flush |= fn(kvm, iterator.rmap);
|
|
|
|
|
|
|
|
if (need_resched() || spin_needbreak(&kvm->mmu_lock)) {
|
|
|
|
if (flush && lock_flush_tlb) {
|
2019-03-13 02:45:59 +08:00
|
|
|
kvm_flush_remote_tlbs_with_address(kvm,
|
|
|
|
start_gfn,
|
|
|
|
iterator.gfn - start_gfn + 1);
|
2019-02-06 05:01:19 +08:00
|
|
|
flush = false;
|
|
|
|
}
|
|
|
|
cond_resched_lock(&kvm->mmu_lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (flush && lock_flush_tlb) {
|
2019-03-13 02:45:59 +08:00
|
|
|
kvm_flush_remote_tlbs_with_address(kvm, start_gfn,
|
|
|
|
end_gfn - start_gfn + 1);
|
2019-02-06 05:01:19 +08:00
|
|
|
flush = false;
|
|
|
|
}
|
|
|
|
|
|
|
|
return flush;
|
|
|
|
}
|
|
|
|
|
|
|
|
static __always_inline bool
|
|
|
|
slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot,
|
|
|
|
slot_level_handler fn, int start_level, int end_level,
|
|
|
|
bool lock_flush_tlb)
|
|
|
|
{
|
|
|
|
return slot_handle_level_range(kvm, memslot, fn, start_level,
|
|
|
|
end_level, memslot->base_gfn,
|
|
|
|
memslot->base_gfn + memslot->npages - 1,
|
|
|
|
lock_flush_tlb);
|
|
|
|
}
|
|
|
|
|
|
|
|
static __always_inline bool
|
|
|
|
slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot,
|
|
|
|
slot_level_handler fn, bool lock_flush_tlb)
|
|
|
|
{
|
2020-04-28 08:54:22 +08:00
|
|
|
return slot_handle_level(kvm, memslot, fn, PG_LEVEL_4K,
|
2020-04-28 08:54:21 +08:00
|
|
|
KVM_MAX_HUGEPAGE_LEVEL, lock_flush_tlb);
|
2019-02-06 05:01:19 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static __always_inline bool
|
|
|
|
slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot,
|
|
|
|
slot_level_handler fn, bool lock_flush_tlb)
|
|
|
|
{
|
2020-04-28 08:54:22 +08:00
|
|
|
return slot_handle_level(kvm, memslot, fn, PG_LEVEL_4K + 1,
|
2020-04-28 08:54:21 +08:00
|
|
|
KVM_MAX_HUGEPAGE_LEVEL, lock_flush_tlb);
|
2019-02-06 05:01:19 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static __always_inline bool
|
|
|
|
slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot,
|
|
|
|
slot_level_handler fn, bool lock_flush_tlb)
|
|
|
|
{
|
2020-04-28 08:54:22 +08:00
|
|
|
return slot_handle_level(kvm, memslot, fn, PG_LEVEL_4K,
|
|
|
|
PG_LEVEL_4K, lock_flush_tlb);
|
2019-02-06 05:01:19 +08:00
|
|
|
}
|
|
|
|
|
2019-06-23 01:42:04 +08:00
|
|
|
static void free_mmu_pages(struct kvm_mmu *mmu)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2019-06-23 01:42:04 +08:00
|
|
|
free_page((unsigned long)mmu->pae_root);
|
|
|
|
free_page((unsigned long)mmu->lm_root);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2019-06-23 01:42:04 +08:00
|
|
|
static int alloc_mmu_pages(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2007-01-06 08:36:40 +08:00
|
|
|
struct page *page;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
int i;
|
|
|
|
|
2007-01-06 08:36:40 +08:00
|
|
|
/*
|
2019-06-14 01:22:23 +08:00
|
|
|
* When using PAE paging, the four PDPTEs are treated as 'root' pages,
|
|
|
|
* while the PDP table is a per-vCPU construct that's allocated at MMU
|
|
|
|
* creation. When emulating 32-bit mode, cr3 is only 32 bits even on
|
|
|
|
* x86_64. Therefore we need to allocate the PDP table in the first
|
|
|
|
* 4GB of memory, which happens to fit the DMA32 zone. Except for
|
|
|
|
* SVM's 32-bit NPT support, TDP paging doesn't use PAE paging and can
|
|
|
|
* skip allocating the PDP table.
|
2007-01-06 08:36:40 +08:00
|
|
|
*/
|
2020-05-02 12:32:34 +08:00
|
|
|
if (tdp_enabled && vcpu->arch.tdp_level > PT32E_ROOT_LEVEL)
|
2019-06-14 01:22:23 +08:00
|
|
|
return 0;
|
|
|
|
|
2019-02-12 03:02:50 +08:00
|
|
|
page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_DMA32);
|
2007-01-06 08:36:40 +08:00
|
|
|
if (!page)
|
2010-01-22 16:55:05 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
|
2019-06-23 01:42:04 +08:00
|
|
|
mmu->pae_root = page_address(page);
|
2007-01-06 08:36:40 +08:00
|
|
|
for (i = 0; i < 4; ++i)
|
2019-06-23 01:42:04 +08:00
|
|
|
mmu->pae_root[i] = INVALID_PAGE;
|
2007-01-06 08:36:40 +08:00
|
|
|
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-12-30 08:50:01 +08:00
|
|
|
int kvm_mmu_create(struct kvm_vcpu *vcpu)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2018-06-28 05:59:20 +08:00
|
|
|
uint i;
|
2019-06-23 01:42:04 +08:00
|
|
|
int ret;
|
2018-06-28 05:59:20 +08:00
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.mmu = &vcpu->arch.root_mmu;
|
|
|
|
vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.root_mmu.root_hpa = INVALID_PAGE;
|
2020-03-21 05:28:32 +08:00
|
|
|
vcpu->arch.root_mmu.root_pgd = 0;
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.root_mmu.translate_gpa = translate_gpa;
|
2018-06-28 05:59:20 +08:00
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
|
2018-10-09 03:28:05 +08:00
|
|
|
vcpu->arch.root_mmu.prev_roots[i] = KVM_MMU_ROOT_INFO_INVALID;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2018-10-09 03:28:08 +08:00
|
|
|
vcpu->arch.guest_mmu.root_hpa = INVALID_PAGE;
|
2020-03-21 05:28:32 +08:00
|
|
|
vcpu->arch.guest_mmu.root_pgd = 0;
|
2018-10-09 03:28:08 +08:00
|
|
|
vcpu->arch.guest_mmu.translate_gpa = translate_gpa;
|
|
|
|
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
|
|
|
|
vcpu->arch.guest_mmu.prev_roots[i] = KVM_MMU_ROOT_INFO_INVALID;
|
2006-12-22 17:05:28 +08:00
|
|
|
|
2018-10-09 03:28:08 +08:00
|
|
|
vcpu->arch.nested_mmu.translate_gpa = translate_nested_gpa;
|
2019-06-23 01:42:04 +08:00
|
|
|
|
|
|
|
ret = alloc_mmu_pages(vcpu, &vcpu->arch.guest_mmu);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
ret = alloc_mmu_pages(vcpu, &vcpu->arch.root_mmu);
|
|
|
|
if (ret)
|
|
|
|
goto fail_allocate_root;
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
fail_allocate_root:
|
|
|
|
free_mmu_pages(&vcpu->arch.guest_mmu);
|
|
|
|
return ret;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
|
|
|
|
2019-09-13 10:46:07 +08:00
|
|
|
#define BATCH_ZAP_PAGES 10
|
2019-09-13 10:46:02 +08:00
|
|
|
static void kvm_zap_obsolete_pages(struct kvm *kvm)
|
|
|
|
{
|
|
|
|
struct kvm_mmu_page *sp, *node;
|
2019-09-13 10:46:07 +08:00
|
|
|
int nr_zapped, batch = 0;
|
2019-09-13 10:46:02 +08:00
|
|
|
|
|
|
|
restart:
|
|
|
|
list_for_each_entry_safe_reverse(sp, node,
|
|
|
|
&kvm->arch.active_mmu_pages, link) {
|
|
|
|
/*
|
|
|
|
* No obsolete valid page exists before a newly created page
|
|
|
|
* since active_mmu_pages is a FIFO list.
|
|
|
|
*/
|
|
|
|
if (!is_obsolete_sp(kvm, sp))
|
|
|
|
break;
|
|
|
|
|
|
|
|
/*
|
2020-06-24 03:35:39 +08:00
|
|
|
* Invalid pages should never land back on the list of active
|
|
|
|
* pages. Skip the bogus page, otherwise we'll get stuck in an
|
|
|
|
* infinite loop if the page gets put back on the list (again).
|
2019-09-13 10:46:02 +08:00
|
|
|
*/
|
2020-06-24 03:35:39 +08:00
|
|
|
if (WARN_ON(sp->role.invalid))
|
2019-09-13 10:46:02 +08:00
|
|
|
continue;
|
|
|
|
|
2019-09-13 10:46:08 +08:00
|
|
|
/*
|
|
|
|
* No need to flush the TLB since we're only zapping shadow
|
|
|
|
* pages with an obsolete generation number and all vCPUS have
|
|
|
|
* loaded a new root, i.e. the shadow pages being zapped cannot
|
|
|
|
* be in active use by the guest.
|
|
|
|
*/
|
2019-09-13 10:46:07 +08:00
|
|
|
if (batch >= BATCH_ZAP_PAGES &&
|
2019-09-13 10:46:08 +08:00
|
|
|
cond_resched_lock(&kvm->mmu_lock)) {
|
2019-09-13 10:46:07 +08:00
|
|
|
batch = 0;
|
2019-09-13 10:46:02 +08:00
|
|
|
goto restart;
|
|
|
|
}
|
|
|
|
|
2019-09-13 10:46:10 +08:00
|
|
|
if (__kvm_mmu_prepare_zap_page(kvm, sp,
|
|
|
|
&kvm->arch.zapped_obsolete_pages, &nr_zapped)) {
|
2019-09-13 10:46:07 +08:00
|
|
|
batch += nr_zapped;
|
2019-09-13 10:46:02 +08:00
|
|
|
goto restart;
|
2019-09-13 10:46:07 +08:00
|
|
|
}
|
2019-09-13 10:46:02 +08:00
|
|
|
}
|
|
|
|
|
2019-09-13 10:46:08 +08:00
|
|
|
/*
|
|
|
|
* Trigger a remote TLB flush before freeing the page tables to ensure
|
|
|
|
* KVM is not in the middle of a lockless shadow page table walk, which
|
|
|
|
* may reference the pages.
|
|
|
|
*/
|
2019-09-13 10:46:10 +08:00
|
|
|
kvm_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages);
|
2019-09-13 10:46:02 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Fast invalidate all shadow pages and use lock-break technique
|
|
|
|
* to zap obsolete pages.
|
|
|
|
*
|
|
|
|
* It's required when memslot is being deleted or VM is being
|
|
|
|
* destroyed, in these cases, we should ensure that KVM MMU does
|
|
|
|
* not use any resource of the being-deleted slot or all slots
|
|
|
|
* after calling the function.
|
|
|
|
*/
|
|
|
|
static void kvm_mmu_zap_all_fast(struct kvm *kvm)
|
|
|
|
{
|
2019-09-13 10:46:11 +08:00
|
|
|
lockdep_assert_held(&kvm->slots_lock);
|
|
|
|
|
2019-09-13 10:46:02 +08:00
|
|
|
spin_lock(&kvm->mmu_lock);
|
2019-09-13 10:46:06 +08:00
|
|
|
trace_kvm_mmu_zap_all_fast(kvm);
|
2019-09-13 10:46:11 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Toggle mmu_valid_gen between '0' and '1'. Because slots_lock is
|
|
|
|
* held for the entire duration of zapping obsolete pages, it's
|
|
|
|
* impossible for there to be multiple invalid generations associated
|
|
|
|
* with *valid* shadow pages at any given time, i.e. there is exactly
|
|
|
|
* one valid generation and (at most) one invalid generation.
|
|
|
|
*/
|
|
|
|
kvm->arch.mmu_valid_gen = kvm->arch.mmu_valid_gen ? 0 : 1;
|
2019-09-13 10:46:02 +08:00
|
|
|
|
2019-09-13 10:46:08 +08:00
|
|
|
/*
|
|
|
|
* Notify all vcpus to reload its shadow page table and flush TLB.
|
|
|
|
* Then all vcpus will switch to new shadow page table with the new
|
|
|
|
* mmu_valid_gen.
|
|
|
|
*
|
|
|
|
* Note: we need to do this under the protection of mmu_lock,
|
|
|
|
* otherwise, vcpu would purge shadow page but miss tlb flush.
|
|
|
|
*/
|
|
|
|
kvm_reload_remote_mmus(kvm);
|
|
|
|
|
2019-09-13 10:46:02 +08:00
|
|
|
kvm_zap_obsolete_pages(kvm);
|
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
}
|
|
|
|
|
2019-09-13 10:46:10 +08:00
|
|
|
static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
|
|
|
|
{
|
|
|
|
return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages));
|
|
|
|
}
|
|
|
|
|
2016-10-09 15:41:44 +08:00
|
|
|
static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm,
|
2016-10-25 15:50:42 +08:00
|
|
|
struct kvm_memory_slot *slot,
|
|
|
|
struct kvm_page_track_notifier_node *node)
|
2016-10-09 15:41:44 +08:00
|
|
|
{
|
2019-09-13 10:46:02 +08:00
|
|
|
kvm_mmu_zap_all_fast(kvm);
|
2015-05-13 14:42:23 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:16 +08:00
|
|
|
void kvm_mmu_init_vm(struct kvm *kvm)
|
2015-05-13 14:42:23 +08:00
|
|
|
{
|
2016-02-24 17:51:16 +08:00
|
|
|
struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
|
2015-05-13 14:42:23 +08:00
|
|
|
|
2016-02-24 17:51:16 +08:00
|
|
|
node->track_write = kvm_mmu_pte_write;
|
2016-10-09 15:41:44 +08:00
|
|
|
node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
|
2016-02-24 17:51:16 +08:00
|
|
|
kvm_page_track_register_notifier(kvm, node);
|
2015-05-13 14:42:23 +08:00
|
|
|
}
|
|
|
|
|
2016-02-24 17:51:16 +08:00
|
|
|
void kvm_mmu_uninit_vm(struct kvm *kvm)
|
2015-05-13 14:42:23 +08:00
|
|
|
{
|
2016-02-24 17:51:16 +08:00
|
|
|
struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
|
2015-05-13 14:42:23 +08:00
|
|
|
|
2016-02-24 17:51:16 +08:00
|
|
|
kvm_page_track_unregister_notifier(kvm, node);
|
2015-05-13 14:42:23 +08:00
|
|
|
}
|
|
|
|
|
2015-05-13 14:42:27 +08:00
|
|
|
void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
|
|
|
|
{
|
|
|
|
struct kvm_memslots *slots;
|
|
|
|
struct kvm_memory_slot *memslot;
|
2015-05-18 19:33:16 +08:00
|
|
|
int i;
|
2015-05-13 14:42:27 +08:00
|
|
|
|
|
|
|
spin_lock(&kvm->mmu_lock);
|
2015-05-18 19:33:16 +08:00
|
|
|
for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
|
|
|
|
slots = __kvm_memslots(kvm, i);
|
|
|
|
kvm_for_each_memslot(memslot, slots) {
|
|
|
|
gfn_t start, end;
|
|
|
|
|
|
|
|
start = max(gfn_start, memslot->base_gfn);
|
|
|
|
end = min(gfn_end, memslot->base_gfn + memslot->npages);
|
|
|
|
if (start >= end)
|
|
|
|
continue;
|
2015-05-13 14:42:27 +08:00
|
|
|
|
2019-03-13 02:45:58 +08:00
|
|
|
slot_handle_level_range(kvm, memslot, kvm_zap_rmapp,
|
2020-04-28 08:54:22 +08:00
|
|
|
PG_LEVEL_4K,
|
2020-04-28 08:54:21 +08:00
|
|
|
KVM_MAX_HUGEPAGE_LEVEL,
|
2019-03-13 02:45:58 +08:00
|
|
|
start, end - 1, true);
|
2015-05-18 19:33:16 +08:00
|
|
|
}
|
2015-05-13 14:42:27 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
}
|
|
|
|
|
2015-11-20 16:41:28 +08:00
|
|
|
static bool slot_rmap_write_protect(struct kvm *kvm,
|
|
|
|
struct kvm_rmap_head *rmap_head)
|
2015-05-13 14:42:24 +08:00
|
|
|
{
|
2015-11-20 16:41:28 +08:00
|
|
|
return __rmap_write_protect(kvm, rmap_head, false);
|
2015-05-13 14:42:24 +08:00
|
|
|
}
|
|
|
|
|
2015-01-28 10:54:26 +08:00
|
|
|
void kvm_mmu_slot_remove_write_access(struct kvm *kvm,
|
2020-02-27 09:32:27 +08:00
|
|
|
struct kvm_memory_slot *memslot,
|
|
|
|
int start_level)
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
{
|
2015-05-13 14:42:24 +08:00
|
|
|
bool flush;
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
|
2013-01-08 18:46:48 +08:00
|
|
|
spin_lock(&kvm->mmu_lock);
|
2020-02-27 09:32:27 +08:00
|
|
|
flush = slot_handle_level(kvm, memslot, slot_rmap_write_protect,
|
2020-04-28 08:54:21 +08:00
|
|
|
start_level, KVM_MAX_HUGEPAGE_LEVEL, false);
|
2013-01-08 18:46:48 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
2014-04-17 17:06:16 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We can flush all the TLBs out of the mmu lock without TLB
|
|
|
|
* corruption since we just change the spte from writable to
|
|
|
|
* readonly so that we only need to care the case of changing
|
|
|
|
* spte from present to present (changing the spte from present
|
|
|
|
* to nonpresent will flush all the TLBs immediately), in other
|
|
|
|
* words, the only case we care is mmu_spte_update() where we
|
2018-11-05 14:45:03 +08:00
|
|
|
* have checked SPTE_HOST_WRITEABLE | SPTE_MMU_WRITEABLE
|
2014-04-17 17:06:16 +08:00
|
|
|
* instead of PT_WRITABLE_MASK, that means it does not depend
|
|
|
|
* on PT_WRITABLE_MASK anymore.
|
|
|
|
*/
|
2015-01-12 15:28:54 +08:00
|
|
|
if (flush)
|
2020-02-19 05:07:36 +08:00
|
|
|
kvm_arch_flush_remote_tlbs_memslot(kvm, memslot);
|
[PATCH] kvm: userspace interface
web site: http://kvm.sourceforge.net
mailing list: kvm-devel@lists.sourceforge.net
(http://lists.sourceforge.net/lists/listinfo/kvm-devel)
The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture. The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace. Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.
Using this driver, one can start multiple virtual machines on a host.
Each virtual machine is a process on the host; a virtual cpu is a thread in
that process. kill(1), nice(1), top(1) work as expected. In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode. Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm). Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.
The driver supports i386 and x86_64 hosts and guests. All combinations are
allowed except x86_64 guest on i386 host. For i386 guests and hosts, both pae
and non-pae paging modes are supported.
SMP hosts and UP guests are supported. At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.
Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch. We plan to address this in two ways:
- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables
Currently a virtual desktop is responsive but consumes a lot of CPU. Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization. Linux/X is slower, probably due
to X being in a separate process.
In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.
Caveats (akpm: might no longer be true):
- The Windows install currently bluescreens due to a problem with the
virtual APIC. We are working on a fix. A temporary workaround is to
use an existing image or install through qemu
- Windows 64-bit does not work. That's also true for qemu, so it's
probably a problem with the device model.
[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-10 18:21:36 +08:00
|
|
|
}
|
2007-01-06 08:36:56 +08:00
|
|
|
|
2015-04-03 15:40:25 +08:00
|
|
|
static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
|
2015-11-20 16:41:28 +08:00
|
|
|
struct kvm_rmap_head *rmap_head)
|
2015-04-03 15:40:25 +08:00
|
|
|
{
|
|
|
|
u64 *sptep;
|
|
|
|
struct rmap_iterator iter;
|
|
|
|
int need_tlb_flush = 0;
|
kvm: rename pfn_t to kvm_pfn_t
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persistent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 (also discussed at kernel summit) to flag pages that are owned
and dynamically mapped by a device driver. The pmem driver, after
mapping a persistent memory range into the system memmap via
devm_memremap_pages(), arranges for DAX to distinguish pfn-only versus
page-backed pmem-pfns via flags in the new pfn_t type.
The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn, flags the
resulting pte(s) inserted into the process page tables with a new
_PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it keys
off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Finally, this need for "struct page" for persistent memory requires
memory capacity to store the memmap array. Given the memmap array for a
large pool of persistent may exhaust available DRAM introduce a
mechanism to allocate the memmap from persistent memory. The new
"struct vmem_altmap *" parameter to devm_memremap_pages() enables
arch_add_memory() to use reserved pmem capacity rather than the page
allocator.
This patch (of 18):
The core has developed a need for a "pfn_t" type [1]. Move the existing
pfn_t in KVM to kvm_pfn_t [2].
[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002199.html
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-September/002218.html
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-16 08:56:11 +08:00
|
|
|
kvm_pfn_t pfn;
|
2015-04-03 15:40:25 +08:00
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
|
2015-05-13 14:42:20 +08:00
|
|
|
restart:
|
2015-11-20 16:41:28 +08:00
|
|
|
for_each_rmap_spte(rmap_head, &iter, sptep) {
|
2020-06-23 04:20:33 +08:00
|
|
|
sp = sptep_to_sp(sptep);
|
2015-04-03 15:40:25 +08:00
|
|
|
pfn = spte_to_pfn(*sptep);
|
|
|
|
|
|
|
|
/*
|
2015-04-14 12:04:10 +08:00
|
|
|
* We cannot do huge page mapping for indirect shadow pages,
|
|
|
|
* which are found on the last rmap (level = 1) when not using
|
|
|
|
* tdp; such shadow pages are synced with the page table in
|
|
|
|
* the guest, and the guest page table is using 4K page size
|
|
|
|
* mapping if the indirect sp has level = 1.
|
2015-04-03 15:40:25 +08:00
|
|
|
*/
|
2019-11-12 06:12:27 +08:00
|
|
|
if (sp->role.direct && !kvm_is_reserved_pfn(pfn) &&
|
2020-01-09 04:24:48 +08:00
|
|
|
(kvm_is_zone_device_pfn(pfn) ||
|
|
|
|
PageCompound(pfn_to_page(pfn)))) {
|
2018-10-04 10:04:23 +08:00
|
|
|
pte_list_remove(rmap_head, sptep);
|
2018-12-06 21:21:08 +08:00
|
|
|
|
|
|
|
if (kvm_available_flush_tlb_with_range())
|
|
|
|
kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
|
|
|
|
KVM_PAGES_PER_HPAGE(sp->role.level));
|
|
|
|
else
|
|
|
|
need_tlb_flush = 1;
|
|
|
|
|
2015-05-13 14:42:20 +08:00
|
|
|
goto restart;
|
|
|
|
}
|
2015-04-03 15:40:25 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return need_tlb_flush;
|
|
|
|
}
|
|
|
|
|
|
|
|
void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
|
2015-05-18 19:20:23 +08:00
|
|
|
const struct kvm_memory_slot *memslot)
|
2015-04-03 15:40:25 +08:00
|
|
|
{
|
2015-05-18 19:20:23 +08:00
|
|
|
/* FIXME: const-ify all uses of struct kvm_memory_slot. */
|
2015-04-03 15:40:25 +08:00
|
|
|
spin_lock(&kvm->mmu_lock);
|
2015-05-18 19:20:23 +08:00
|
|
|
slot_handle_leaf(kvm, (struct kvm_memory_slot *)memslot,
|
|
|
|
kvm_mmu_zap_collapsible_spte, true);
|
2015-04-03 15:40:25 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
}
|
|
|
|
|
2020-02-19 05:07:34 +08:00
|
|
|
void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
|
|
|
|
struct kvm_memory_slot *memslot)
|
|
|
|
{
|
|
|
|
/*
|
2020-02-19 05:07:36 +08:00
|
|
|
* All current use cases for flushing the TLBs for a specific memslot
|
|
|
|
* are related to dirty logging, and do the TLB flush out of mmu_lock.
|
|
|
|
* The interaction between the various operations on memslot must be
|
|
|
|
* serialized by slots_locks to ensure the TLB flush from one operation
|
|
|
|
* is observed by any other operation on the same memslot.
|
2020-02-19 05:07:34 +08:00
|
|
|
*/
|
|
|
|
lockdep_assert_held(&kvm->slots_lock);
|
2020-02-19 05:07:35 +08:00
|
|
|
kvm_flush_remote_tlbs_with_address(kvm, memslot->base_gfn,
|
|
|
|
memslot->npages);
|
2020-02-19 05:07:34 +08:00
|
|
|
}
|
|
|
|
|
2015-01-28 10:54:24 +08:00
|
|
|
void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
|
|
|
|
struct kvm_memory_slot *memslot)
|
|
|
|
{
|
2015-05-13 14:42:24 +08:00
|
|
|
bool flush;
|
2015-01-28 10:54:24 +08:00
|
|
|
|
|
|
|
spin_lock(&kvm->mmu_lock);
|
2015-05-13 14:42:24 +08:00
|
|
|
flush = slot_handle_leaf(kvm, memslot, __rmap_clear_dirty, false);
|
2015-01-28 10:54:24 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* It's also safe to flush TLBs out of mmu lock here as currently this
|
|
|
|
* function is only used for dirty logging, in which case flushing TLB
|
|
|
|
* out of mmu lock also guarantees no dirty pages will be lost in
|
|
|
|
* dirty_bitmap.
|
|
|
|
*/
|
|
|
|
if (flush)
|
2020-02-19 05:07:36 +08:00
|
|
|
kvm_arch_flush_remote_tlbs_memslot(kvm, memslot);
|
2015-01-28 10:54:24 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_slot_leaf_clear_dirty);
|
|
|
|
|
|
|
|
void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm,
|
|
|
|
struct kvm_memory_slot *memslot)
|
|
|
|
{
|
2015-05-13 14:42:24 +08:00
|
|
|
bool flush;
|
2015-01-28 10:54:24 +08:00
|
|
|
|
|
|
|
spin_lock(&kvm->mmu_lock);
|
2015-05-13 14:42:24 +08:00
|
|
|
flush = slot_handle_large_level(kvm, memslot, slot_rmap_write_protect,
|
|
|
|
false);
|
2015-01-28 10:54:24 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
|
|
|
|
if (flush)
|
2020-02-19 05:07:36 +08:00
|
|
|
kvm_arch_flush_remote_tlbs_memslot(kvm, memslot);
|
2015-01-28 10:54:24 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_slot_largepage_remove_write_access);
|
|
|
|
|
|
|
|
void kvm_mmu_slot_set_dirty(struct kvm *kvm,
|
|
|
|
struct kvm_memory_slot *memslot)
|
|
|
|
{
|
2015-05-13 14:42:24 +08:00
|
|
|
bool flush;
|
2015-01-28 10:54:24 +08:00
|
|
|
|
|
|
|
spin_lock(&kvm->mmu_lock);
|
2015-05-13 14:42:24 +08:00
|
|
|
flush = slot_handle_all_level(kvm, memslot, __rmap_set_dirty, false);
|
2015-01-28 10:54:24 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
|
|
|
|
if (flush)
|
2020-02-19 05:07:36 +08:00
|
|
|
kvm_arch_flush_remote_tlbs_memslot(kvm, memslot);
|
2015-01-28 10:54:24 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(kvm_mmu_slot_set_dirty);
|
|
|
|
|
2019-09-13 10:46:04 +08:00
|
|
|
void kvm_mmu_zap_all(struct kvm *kvm)
|
2013-05-31 08:36:22 +08:00
|
|
|
{
|
|
|
|
struct kvm_mmu_page *sp, *node;
|
2019-02-06 05:01:31 +08:00
|
|
|
LIST_HEAD(invalid_list);
|
KVM: x86/mmu: Differentiate between nr zapped and list unstable
The return value of kvm_mmu_prepare_zap_page() has evolved to become
overloaded to convey two separate pieces of information. 1) was at
least one page zapped and 2) has the list of MMU pages become unstable.
In it's original incarnation (as kvm_mmu_zap_page()), there was no
return value at all. Commit 0738541396be ("KVM: MMU: awareness of new
kvm_mmu_zap_page behaviour") added a return value in preparation for
commit 4731d4c7a077 ("KVM: MMU: out of sync shadow core"). Although
the return value was of type 'int', it was actually used as a boolean
to indicate whether or not active_mmu_pages may have become unstable due
to zapping children. Walking a list with list_for_each_entry_safe()
only protects against deleting/moving the current entry, i.e. zapping a
child page would break iteration due to modifying any number of entries.
Later, commit 60c8aec6e2c9 ("KVM: MMU: use page array in unsync walk")
modified mmu_zap_unsync_children() to return an approximation of the
number of children zapped. This was not intentional, it was simply a
side effect of how the code was written.
The unintented side affect was then morphed into an actual feature by
commit 77662e0028c7 ("KVM: MMU: fix kvm_mmu_zap_page() and its calling
path"), which modified kvm_mmu_change_mmu_pages() to use the number of
zapped pages when determining the number of MMU pages in use by the VM.
Finally, commit 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed") added the initial page to the
return value to make its behavior more consistent with what most users
would expect. Incorporating the initial parent page in the return value
of kvm_mmu_zap_page() breaks the original usage of restarting a list
walk on a non-zero return value to handle a potentially unstable list,
i.e. walks will unnecessarily restart when any page is zapped.
Fix this by restoring the original behavior of kvm_mmu_zap_page(), i.e.
return a boolean to indicate that the list may be unstable and move the
number of zapped children to a dedicated parameter. Since the majority
of callers to kvm_mmu_prepare_zap_page() don't care about either return
value, preserve the current definition of kvm_mmu_prepare_zap_page() by
making it a wrapper of a new helper, __kvm_mmu_prepare_zap_page(). This
avoids having to update every call site and also provides cleaner code
for functions that only care about the number of pages zapped.
Fixes: 54a4f0239f2e ("KVM: MMU: make kvm_mmu_zap_page() return
the number of pages it actually freed")
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:35 +08:00
|
|
|
int ign;
|
2013-05-31 08:36:22 +08:00
|
|
|
|
2019-02-06 05:01:31 +08:00
|
|
|
spin_lock(&kvm->mmu_lock);
|
2013-05-31 08:36:22 +08:00
|
|
|
restart:
|
2019-02-06 05:01:32 +08:00
|
|
|
list_for_each_entry_safe(sp, node, &kvm->arch.active_mmu_pages, link) {
|
2020-06-24 03:35:39 +08:00
|
|
|
if (WARN_ON(sp->role.invalid))
|
2019-02-06 05:01:23 +08:00
|
|
|
continue;
|
2019-09-13 10:46:04 +08:00
|
|
|
if (__kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list, &ign))
|
2013-05-31 08:36:22 +08:00
|
|
|
goto restart;
|
KVM: x86/mmu: WARN if zapping a MMIO spte results in zapping children
Paolo expressed a concern that kvm_mmu_zap_mmio_sptes() could have a
quadratic runtime[1], i.e. restarting the spte walk while zapping only
MMIO sptes could result in re-walking large portions of the list over
and over due to the non-MMIO sptes encountered before the restart not
being removed.
At the time, the concern was legitimate as the walk was restarted when
any spte was zapped. But that is no longer the case as the walk is now
restarted iff one or more children have been zapped, which is necessary
because zapping children makes the active_mmu_pages list unstable.
Furthermore, it should be impossible for an MMIO spte to have children,
i.e. zapping an MMIO spte should never result in zapping children. In
other words, kvm_mmu_zap_mmio_sptes() should never restart its walk, and
so should always execute in linear time. WARN if this assertion fails.
Although it should never be needed, leave the restart logic in place.
In normal operation, the cost is at worst an extra CMP+Jcc, and if for
some reason the list does become unstable, not restarting would likely
crash KVM, or worse, the kernel.
[1] https://patchwork.kernel.org/patch/10756589/#22452085
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:36 +08:00
|
|
|
if (cond_resched_lock(&kvm->mmu_lock))
|
2013-05-31 08:36:22 +08:00
|
|
|
goto restart;
|
|
|
|
}
|
|
|
|
|
2019-02-06 05:01:23 +08:00
|
|
|
kvm_mmu_commit_zap_page(kvm, &invalid_list);
|
2013-05-31 08:36:22 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
}
|
|
|
|
|
2019-02-06 04:54:17 +08:00
|
|
|
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
|
2013-06-07 16:51:26 +08:00
|
|
|
{
|
2019-02-06 05:01:18 +08:00
|
|
|
WARN_ON(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS);
|
2019-02-06 05:01:12 +08:00
|
|
|
|
2019-02-06 05:01:18 +08:00
|
|
|
gen &= MMIO_SPTE_GEN_MASK;
|
2019-02-06 05:01:12 +08:00
|
|
|
|
2013-06-07 16:51:26 +08:00
|
|
|
/*
|
2019-02-06 05:01:12 +08:00
|
|
|
* Generation numbers are incremented in multiples of the number of
|
|
|
|
* address spaces in order to provide unique generations across all
|
|
|
|
* address spaces. Strip what is effectively the address space
|
|
|
|
* modifier prior to checking for a wrap of the MMIO generation so
|
|
|
|
* that a wrap in any address space is detected.
|
|
|
|
*/
|
|
|
|
gen &= ~((u64)KVM_ADDRESS_SPACE_NUM - 1);
|
|
|
|
|
2013-06-07 16:51:26 +08:00
|
|
|
/*
|
2019-02-06 05:01:12 +08:00
|
|
|
* The very rare case: if the MMIO generation number has wrapped,
|
2013-06-07 16:51:26 +08:00
|
|
|
* zap all shadow pages.
|
|
|
|
*/
|
2019-02-06 05:01:12 +08:00
|
|
|
if (unlikely(gen == 0)) {
|
2016-11-15 14:36:18 +08:00
|
|
|
kvm_debug_ratelimited("kvm: zapping shadow pages for mmio generation wraparound\n");
|
2019-09-13 10:46:04 +08:00
|
|
|
kvm_mmu_zap_all_fast(kvm);
|
2013-06-21 00:34:31 +08:00
|
|
|
}
|
2013-06-07 16:51:26 +08:00
|
|
|
}
|
|
|
|
|
2013-08-28 08:18:14 +08:00
|
|
|
static unsigned long
|
|
|
|
mmu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
|
2008-03-30 20:17:21 +08:00
|
|
|
{
|
|
|
|
struct kvm *kvm;
|
2011-05-25 08:12:27 +08:00
|
|
|
int nr_to_scan = sc->nr_to_scan;
|
2013-08-28 08:18:14 +08:00
|
|
|
unsigned long freed = 0;
|
2008-03-30 20:17:21 +08:00
|
|
|
|
2019-01-04 09:14:28 +08:00
|
|
|
mutex_lock(&kvm_lock);
|
2008-03-30 20:17:21 +08:00
|
|
|
|
|
|
|
list_for_each_entry(kvm, &vm_list, vm_list) {
|
2011-12-03 01:35:24 +08:00
|
|
|
int idx;
|
2010-06-04 21:55:29 +08:00
|
|
|
LIST_HEAD(invalid_list);
|
2008-03-30 20:17:21 +08:00
|
|
|
|
2012-08-20 17:35:39 +08:00
|
|
|
/*
|
|
|
|
* Never scan more than sc->nr_to_scan VM instances.
|
|
|
|
* Will not hit this condition practically since we do not try
|
|
|
|
* to shrink more than one VM and it is very unlikely to see
|
|
|
|
* !n_used_mmu_pages so many times.
|
|
|
|
*/
|
|
|
|
if (!nr_to_scan--)
|
|
|
|
break;
|
2012-06-04 19:53:23 +08:00
|
|
|
/*
|
|
|
|
* n_used_mmu_pages is accessed without holding kvm->mmu_lock
|
|
|
|
* here. We may skip a VM instance errorneosly, but we do not
|
|
|
|
* want to shrink a VM that only started to populate its MMU
|
|
|
|
* anyway.
|
|
|
|
*/
|
2019-09-13 10:46:10 +08:00
|
|
|
if (!kvm->arch.n_used_mmu_pages &&
|
|
|
|
!kvm_has_zapped_obsolete_pages(kvm))
|
2012-06-04 19:53:23 +08:00
|
|
|
continue;
|
|
|
|
|
2009-12-24 00:35:25 +08:00
|
|
|
idx = srcu_read_lock(&kvm->srcu);
|
2008-03-30 20:17:21 +08:00
|
|
|
spin_lock(&kvm->mmu_lock);
|
|
|
|
|
2019-09-13 10:46:10 +08:00
|
|
|
if (kvm_has_zapped_obsolete_pages(kvm)) {
|
|
|
|
kvm_mmu_commit_zap_page(kvm,
|
|
|
|
&kvm->arch.zapped_obsolete_pages);
|
|
|
|
goto unlock;
|
|
|
|
}
|
|
|
|
|
2020-06-24 03:35:41 +08:00
|
|
|
freed = kvm_mmu_zap_oldest_mmu_pages(kvm, sc->nr_to_scan);
|
2012-06-04 19:53:23 +08:00
|
|
|
|
2019-09-13 10:46:10 +08:00
|
|
|
unlock:
|
2008-03-30 20:17:21 +08:00
|
|
|
spin_unlock(&kvm->mmu_lock);
|
2009-12-24 00:35:25 +08:00
|
|
|
srcu_read_unlock(&kvm->srcu, idx);
|
2012-06-04 19:53:23 +08:00
|
|
|
|
2013-08-28 08:18:14 +08:00
|
|
|
/*
|
|
|
|
* unfair on small ones
|
|
|
|
* per-vm shrinkers cry out
|
|
|
|
* sadness comes quickly
|
|
|
|
*/
|
2012-06-04 19:53:23 +08:00
|
|
|
list_move_tail(&kvm->vm_list, &vm_list);
|
|
|
|
break;
|
2008-03-30 20:17:21 +08:00
|
|
|
}
|
|
|
|
|
2019-01-04 09:14:28 +08:00
|
|
|
mutex_unlock(&kvm_lock);
|
2013-08-28 08:18:14 +08:00
|
|
|
return freed;
|
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned long
|
|
|
|
mmu_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
|
|
|
|
{
|
KVM: create aggregate kvm_total_used_mmu_pages value
Of slab shrinkers, the VM code says:
* Note that 'shrink' will be passed nr_to_scan == 0 when the VM is
* querying the cache size, so a fastpath for that case is appropriate.
and it *means* it. Look at how it calls the shrinkers:
nr_before = (*shrinker->shrink)(0, gfp_mask);
shrink_ret = (*shrinker->shrink)(this_scan, gfp_mask);
So, if you do anything stupid in your shrinker, the VM will doubly
punish you.
The mmu_shrink() function takes the global kvm_lock, then acquires
every VM's kvm->mmu_lock in sequence. If we have 100 VMs, then
we're going to take 101 locks. We do it twice, so each call takes
202 locks. If we're under memory pressure, we can have each cpu
trying to do this. It can get really hairy, and we've seen lock
spinning in mmu_shrink() be the dominant entry in profiles.
This is guaranteed to optimize at least half of those lock
aquisitions away. It removes the need to take any of the locks
when simply trying to count objects.
A 'percpu_counter' can be a large object, but we only have one
of these for the entire system. There are not any better
alternatives at the moment, especially ones that handle CPU
hotplug.
Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
Signed-off-by: Tim Pepper <lnxninja@linux.vnet.ibm.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
2010-08-20 09:11:37 +08:00
|
|
|
return percpu_counter_read_positive(&kvm_total_used_mmu_pages);
|
2008-03-30 20:17:21 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static struct shrinker mmu_shrinker = {
|
2013-08-28 08:18:14 +08:00
|
|
|
.count_objects = mmu_shrink_count,
|
|
|
|
.scan_objects = mmu_shrink_scan,
|
2008-03-30 20:17:21 +08:00
|
|
|
.seeks = DEFAULT_SEEKS * 10,
|
|
|
|
};
|
|
|
|
|
2008-05-22 16:37:48 +08:00
|
|
|
static void mmu_destroy_caches(void)
|
2007-04-15 21:31:09 +08:00
|
|
|
{
|
2017-10-08 11:15:23 +08:00
|
|
|
kmem_cache_destroy(pte_list_desc_cache);
|
|
|
|
kmem_cache_destroy(mmu_page_header_cache);
|
2007-04-15 21:31:09 +08:00
|
|
|
}
|
|
|
|
|
2019-05-03 18:08:52 +08:00
|
|
|
static void kvm_set_mmio_spte_mask(void)
|
|
|
|
{
|
|
|
|
u64 mask;
|
|
|
|
|
|
|
|
/*
|
2020-05-27 16:49:09 +08:00
|
|
|
* Set a reserved PA bit in MMIO SPTEs to generate page faults with
|
|
|
|
* PFEC.RSVD=1 on MMIO accesses. 64-bit PTEs (PAE, x86-64, and EPT
|
|
|
|
* paging) support a maximum of 52 bits of PA, i.e. if the CPU supports
|
|
|
|
* 52-bit physical addresses then there are no reserved PA bits in the
|
|
|
|
* PTEs and so the reserved PA approach must be disabled.
|
2019-05-03 18:08:52 +08:00
|
|
|
*/
|
2020-05-27 16:49:09 +08:00
|
|
|
if (shadow_phys_bits < 52)
|
|
|
|
mask = BIT_ULL(51) | PT_PRESENT_MASK;
|
|
|
|
else
|
|
|
|
mask = 0;
|
2019-05-03 18:08:52 +08:00
|
|
|
|
2020-05-19 17:04:49 +08:00
|
|
|
kvm_mmu_set_mmio_spte_mask(mask, ACC_WRITE_MASK | ACC_USER_MASK);
|
2019-05-03 18:08:52 +08:00
|
|
|
}
|
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
static bool get_nx_auto_mode(void)
|
|
|
|
{
|
|
|
|
/* Return true when CPU has the bug, and mitigations are ON */
|
|
|
|
return boot_cpu_has_bug(X86_BUG_ITLB_MULTIHIT) && !cpu_mitigations_off();
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __set_nx_huge_pages(bool val)
|
|
|
|
{
|
|
|
|
nx_huge_pages = itlb_multihit_kvm_mitigation = val;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int set_nx_huge_pages(const char *val, const struct kernel_param *kp)
|
|
|
|
{
|
|
|
|
bool old_val = nx_huge_pages;
|
|
|
|
bool new_val;
|
|
|
|
|
|
|
|
/* In "auto" mode deploy workaround only if CPU has the bug. */
|
|
|
|
if (sysfs_streq(val, "off"))
|
|
|
|
new_val = 0;
|
|
|
|
else if (sysfs_streq(val, "force"))
|
|
|
|
new_val = 1;
|
|
|
|
else if (sysfs_streq(val, "auto"))
|
|
|
|
new_val = get_nx_auto_mode();
|
|
|
|
else if (strtobool(val, &new_val) < 0)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
__set_nx_huge_pages(new_val);
|
|
|
|
|
|
|
|
if (new_val != old_val) {
|
|
|
|
struct kvm *kvm;
|
|
|
|
|
|
|
|
mutex_lock(&kvm_lock);
|
|
|
|
|
|
|
|
list_for_each_entry(kvm, &vm_list, vm_list) {
|
KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast()
Acquire the per-VM slots_lock when zapping all shadow pages as part of
toggling nx_huge_pages. The fast zap algorithm relies on exclusivity
(via slots_lock) to identify obsolete vs. valid shadow pages, because it
uses a single bit for its generation number. Holding slots_lock also
obviates the need to acquire a read lock on the VM's srcu.
Failing to take slots_lock when toggling nx_huge_pages allows multiple
instances of kvm_mmu_zap_all_fast() to run concurrently, as the other
user, KVM_SET_USER_MEMORY_REGION, does not take the global kvm_lock.
(kvm_mmu_zap_all_fast() does take kvm->mmu_lock, but it can be
temporarily dropped by kvm_zap_obsolete_pages(), so it is not enough
to enforce exclusivity).
Concurrent fast zap instances causes obsolete shadow pages to be
incorrectly identified as valid due to the single bit generation number
wrapping, which results in stale shadow pages being left in KVM's MMU
and leads to all sorts of undesirable behavior.
The bug is easily confirmed by running with CONFIG_PROVE_LOCKING and
toggling nx_huge_pages via its module param.
Note, until commit 4ae5acbc4936 ("KVM: x86/mmu: Take slots_lock when
using kvm_mmu_zap_all_fast()", 2019-11-13) the fast zap algorithm used
an ulong-sized generation instead of relying on exclusivity for
correctness, but all callers except the recently added set_nx_huge_pages()
needed to hold slots_lock anyways. Therefore, this patch does not have
to be backported to stable kernels.
Given that toggling nx_huge_pages is by no means a fast path, force it
to conform to the current approach instead of reintroducing the previous
generation count.
Fixes: b8e8c8303ff28 ("kvm: mmu: ITLB_MULTIHIT mitigation", but NOT FOR STABLE)
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-11-14 03:30:32 +08:00
|
|
|
mutex_lock(&kvm->slots_lock);
|
2019-11-04 19:22:02 +08:00
|
|
|
kvm_mmu_zap_all_fast(kvm);
|
KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast()
Acquire the per-VM slots_lock when zapping all shadow pages as part of
toggling nx_huge_pages. The fast zap algorithm relies on exclusivity
(via slots_lock) to identify obsolete vs. valid shadow pages, because it
uses a single bit for its generation number. Holding slots_lock also
obviates the need to acquire a read lock on the VM's srcu.
Failing to take slots_lock when toggling nx_huge_pages allows multiple
instances of kvm_mmu_zap_all_fast() to run concurrently, as the other
user, KVM_SET_USER_MEMORY_REGION, does not take the global kvm_lock.
(kvm_mmu_zap_all_fast() does take kvm->mmu_lock, but it can be
temporarily dropped by kvm_zap_obsolete_pages(), so it is not enough
to enforce exclusivity).
Concurrent fast zap instances causes obsolete shadow pages to be
incorrectly identified as valid due to the single bit generation number
wrapping, which results in stale shadow pages being left in KVM's MMU
and leads to all sorts of undesirable behavior.
The bug is easily confirmed by running with CONFIG_PROVE_LOCKING and
toggling nx_huge_pages via its module param.
Note, until commit 4ae5acbc4936 ("KVM: x86/mmu: Take slots_lock when
using kvm_mmu_zap_all_fast()", 2019-11-13) the fast zap algorithm used
an ulong-sized generation instead of relying on exclusivity for
correctness, but all callers except the recently added set_nx_huge_pages()
needed to hold slots_lock anyways. Therefore, this patch does not have
to be backported to stable kernels.
Given that toggling nx_huge_pages is by no means a fast path, force it
to conform to the current approach instead of reintroducing the previous
generation count.
Fixes: b8e8c8303ff28 ("kvm: mmu: ITLB_MULTIHIT mitigation", but NOT FOR STABLE)
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-11-14 03:30:32 +08:00
|
|
|
mutex_unlock(&kvm->slots_lock);
|
2019-11-05 03:26:00 +08:00
|
|
|
|
|
|
|
wake_up_process(kvm->arch.nx_lpage_recovery_thread);
|
2019-11-04 19:22:02 +08:00
|
|
|
}
|
|
|
|
mutex_unlock(&kvm_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2007-04-15 21:31:09 +08:00
|
|
|
int kvm_mmu_module_init(void)
|
|
|
|
{
|
2018-01-11 00:26:59 +08:00
|
|
|
int ret = -ENOMEM;
|
|
|
|
|
2019-11-04 19:22:02 +08:00
|
|
|
if (nx_huge_pages == -1)
|
|
|
|
__set_nx_huge_pages(get_nx_auto_mode());
|
|
|
|
|
2018-10-09 03:28:10 +08:00
|
|
|
/*
|
|
|
|
* MMU roles use union aliasing which is, generally speaking, an
|
|
|
|
* undefined behavior. However, we supposedly know how compilers behave
|
|
|
|
* and the current status quo is unlikely to change. Guardians below are
|
|
|
|
* supposed to let us know if the assumption becomes false.
|
|
|
|
*/
|
|
|
|
BUILD_BUG_ON(sizeof(union kvm_mmu_page_role) != sizeof(u32));
|
|
|
|
BUILD_BUG_ON(sizeof(union kvm_mmu_extended_role) != sizeof(u32));
|
|
|
|
BUILD_BUG_ON(sizeof(union kvm_mmu_role) != sizeof(u64));
|
|
|
|
|
2018-08-15 01:15:34 +08:00
|
|
|
kvm_mmu_reset_all_pte_masks();
|
2016-12-07 08:46:16 +08:00
|
|
|
|
2019-05-03 18:08:52 +08:00
|
|
|
kvm_set_mmio_spte_mask();
|
|
|
|
|
2011-05-15 23:26:20 +08:00
|
|
|
pte_list_desc_cache = kmem_cache_create("pte_list_desc",
|
|
|
|
sizeof(struct pte_list_desc),
|
2017-10-06 09:07:24 +08:00
|
|
|
0, SLAB_ACCOUNT, NULL);
|
2011-05-15 23:26:20 +08:00
|
|
|
if (!pte_list_desc_cache)
|
2018-01-11 00:26:59 +08:00
|
|
|
goto out;
|
2007-04-15 21:31:09 +08:00
|
|
|
|
2007-05-30 17:34:53 +08:00
|
|
|
mmu_page_header_cache = kmem_cache_create("kvm_mmu_page_header",
|
|
|
|
sizeof(struct kvm_mmu_page),
|
2017-10-06 09:07:24 +08:00
|
|
|
0, SLAB_ACCOUNT, NULL);
|
2007-05-30 17:34:53 +08:00
|
|
|
if (!mmu_page_header_cache)
|
2018-01-11 00:26:59 +08:00
|
|
|
goto out;
|
2007-05-30 17:34:53 +08:00
|
|
|
|
2014-09-08 08:51:29 +08:00
|
|
|
if (percpu_counter_init(&kvm_total_used_mmu_pages, 0, GFP_KERNEL))
|
2018-01-11 00:26:59 +08:00
|
|
|
goto out;
|
2010-08-23 16:13:15 +08:00
|
|
|
|
2018-01-11 00:26:59 +08:00
|
|
|
ret = register_shrinker(&mmu_shrinker);
|
|
|
|
if (ret)
|
|
|
|
goto out;
|
2008-03-30 20:17:21 +08:00
|
|
|
|
2007-04-15 21:31:09 +08:00
|
|
|
return 0;
|
|
|
|
|
2018-01-11 00:26:59 +08:00
|
|
|
out:
|
2008-03-30 20:17:21 +08:00
|
|
|
mmu_destroy_caches();
|
2018-01-11 00:26:59 +08:00
|
|
|
return ret;
|
2007-04-15 21:31:09 +08:00
|
|
|
}
|
|
|
|
|
2007-11-20 13:11:38 +08:00
|
|
|
/*
|
2018-10-04 23:45:00 +08:00
|
|
|
* Calculate mmu pages needed for kvm.
|
2007-11-20 13:11:38 +08:00
|
|
|
*/
|
2019-04-09 02:07:30 +08:00
|
|
|
unsigned long kvm_mmu_calculate_default_mmu_pages(struct kvm *kvm)
|
2007-11-20 13:11:38 +08:00
|
|
|
{
|
2019-04-09 02:07:30 +08:00
|
|
|
unsigned long nr_mmu_pages;
|
|
|
|
unsigned long nr_pages = 0;
|
2009-12-24 00:35:21 +08:00
|
|
|
struct kvm_memslots *slots;
|
2011-11-24 17:39:18 +08:00
|
|
|
struct kvm_memory_slot *memslot;
|
2015-05-18 19:33:16 +08:00
|
|
|
int i;
|
2007-11-20 13:11:38 +08:00
|
|
|
|
2015-05-18 19:33:16 +08:00
|
|
|
for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
|
|
|
|
slots = __kvm_memslots(kvm, i);
|
2010-04-19 17:41:23 +08:00
|
|
|
|
2015-05-18 19:33:16 +08:00
|
|
|
kvm_for_each_memslot(memslot, slots)
|
|
|
|
nr_pages += memslot->npages;
|
|
|
|
}
|
2007-11-20 13:11:38 +08:00
|
|
|
|
|
|
|
nr_mmu_pages = nr_pages * KVM_PERMILLE_MMU_PAGES / 1000;
|
2019-04-09 02:07:30 +08:00
|
|
|
nr_mmu_pages = max(nr_mmu_pages, KVM_MIN_ALLOC_MMU_PAGES);
|
2007-11-20 13:11:38 +08:00
|
|
|
|
|
|
|
return nr_mmu_pages;
|
|
|
|
}
|
|
|
|
|
2010-09-27 18:07:07 +08:00
|
|
|
void kvm_mmu_destroy(struct kvm_vcpu *vcpu)
|
|
|
|
{
|
2013-10-02 22:56:12 +08:00
|
|
|
kvm_mmu_unload(vcpu);
|
2019-06-23 01:42:04 +08:00
|
|
|
free_mmu_pages(&vcpu->arch.root_mmu);
|
|
|
|
free_mmu_pages(&vcpu->arch.guest_mmu);
|
2010-09-27 18:07:07 +08:00
|
|
|
mmu_free_memory_caches(vcpu);
|
2010-12-23 16:08:35 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void kvm_mmu_module_exit(void)
|
|
|
|
{
|
|
|
|
mmu_destroy_caches();
|
|
|
|
percpu_counter_destroy(&kvm_total_used_mmu_pages);
|
|
|
|
unregister_shrinker(&mmu_shrinker);
|
2010-09-27 18:07:07 +08:00
|
|
|
mmu_audit_disable();
|
|
|
|
}
|
2019-11-05 03:26:00 +08:00
|
|
|
|
|
|
|
static int set_nx_huge_pages_recovery_ratio(const char *val, const struct kernel_param *kp)
|
|
|
|
{
|
|
|
|
unsigned int old_val;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
old_val = nx_huge_pages_recovery_ratio;
|
|
|
|
err = param_set_uint(val, kp);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
if (READ_ONCE(nx_huge_pages) &&
|
|
|
|
!old_val && nx_huge_pages_recovery_ratio) {
|
|
|
|
struct kvm *kvm;
|
|
|
|
|
|
|
|
mutex_lock(&kvm_lock);
|
|
|
|
|
|
|
|
list_for_each_entry(kvm, &vm_list, vm_list)
|
|
|
|
wake_up_process(kvm->arch.nx_lpage_recovery_thread);
|
|
|
|
|
|
|
|
mutex_unlock(&kvm_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void kvm_recover_nx_lpages(struct kvm *kvm)
|
|
|
|
{
|
|
|
|
int rcu_idx;
|
|
|
|
struct kvm_mmu_page *sp;
|
|
|
|
unsigned int ratio;
|
|
|
|
LIST_HEAD(invalid_list);
|
|
|
|
ulong to_zap;
|
|
|
|
|
|
|
|
rcu_idx = srcu_read_lock(&kvm->srcu);
|
|
|
|
spin_lock(&kvm->mmu_lock);
|
|
|
|
|
|
|
|
ratio = READ_ONCE(nx_huge_pages_recovery_ratio);
|
|
|
|
to_zap = ratio ? DIV_ROUND_UP(kvm->stat.nx_lpage_splits, ratio) : 0;
|
|
|
|
while (to_zap && !list_empty(&kvm->arch.lpage_disallowed_mmu_pages)) {
|
|
|
|
/*
|
|
|
|
* We use a separate list instead of just using active_mmu_pages
|
|
|
|
* because the number of lpage_disallowed pages is expected to
|
|
|
|
* be relatively small compared to the total.
|
|
|
|
*/
|
|
|
|
sp = list_first_entry(&kvm->arch.lpage_disallowed_mmu_pages,
|
|
|
|
struct kvm_mmu_page,
|
|
|
|
lpage_disallowed_link);
|
|
|
|
WARN_ON_ONCE(!sp->lpage_disallowed);
|
|
|
|
kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
|
|
|
|
WARN_ON_ONCE(sp->lpage_disallowed);
|
|
|
|
|
|
|
|
if (!--to_zap || need_resched() || spin_needbreak(&kvm->mmu_lock)) {
|
|
|
|
kvm_mmu_commit_zap_page(kvm, &invalid_list);
|
|
|
|
if (to_zap)
|
|
|
|
cond_resched_lock(&kvm->mmu_lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
spin_unlock(&kvm->mmu_lock);
|
|
|
|
srcu_read_unlock(&kvm->srcu, rcu_idx);
|
|
|
|
}
|
|
|
|
|
|
|
|
static long get_nx_lpage_recovery_timeout(u64 start_time)
|
|
|
|
{
|
|
|
|
return READ_ONCE(nx_huge_pages) && READ_ONCE(nx_huge_pages_recovery_ratio)
|
|
|
|
? start_time + 60 * HZ - get_jiffies_64()
|
|
|
|
: MAX_SCHEDULE_TIMEOUT;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int kvm_nx_lpage_recovery_worker(struct kvm *kvm, uintptr_t data)
|
|
|
|
{
|
|
|
|
u64 start_time;
|
|
|
|
long remaining_time;
|
|
|
|
|
|
|
|
while (true) {
|
|
|
|
start_time = get_jiffies_64();
|
|
|
|
remaining_time = get_nx_lpage_recovery_timeout(start_time);
|
|
|
|
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
while (!kthread_should_stop() && remaining_time > 0) {
|
|
|
|
schedule_timeout(remaining_time);
|
|
|
|
remaining_time = get_nx_lpage_recovery_timeout(start_time);
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
}
|
|
|
|
|
|
|
|
set_current_state(TASK_RUNNING);
|
|
|
|
|
|
|
|
if (kthread_should_stop())
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
kvm_recover_nx_lpages(kvm);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
int kvm_mmu_post_init_vm(struct kvm *kvm)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
|
|
|
|
err = kvm_vm_create_worker_thread(kvm, kvm_nx_lpage_recovery_worker, 0,
|
|
|
|
"kvm-nx-lpage-recovery",
|
|
|
|
&kvm->arch.nx_lpage_recovery_thread);
|
|
|
|
if (!err)
|
|
|
|
kthread_unpark(kvm->arch.nx_lpage_recovery_thread);
|
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
|
|
|
|
{
|
|
|
|
if (kvm->arch.nx_lpage_recovery_thread)
|
|
|
|
kthread_stop(kvm->arch.nx_lpage_recovery_thread);
|
|
|
|
}
|