License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 22:07:57 +08:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* linux/arch/x86_64/entry.S
|
|
|
|
*
|
|
|
|
* Copyright (C) 1991, 1992 Linus Torvalds
|
|
|
|
* Copyright (C) 2000, 2001, 2002 Andi Kleen SuSE Labs
|
|
|
|
* Copyright (C) 2000 Pavel Machek <pavel@suse.cz>
|
2015-06-09 02:43:07 +08:00
|
|
|
*
|
2005-04-17 06:20:36 +08:00
|
|
|
* entry.S contains the system-call and fault low-level handling routines.
|
|
|
|
*
|
2011-06-06 01:50:18 +08:00
|
|
|
* Some of this is documented in Documentation/x86/entry_64.txt
|
|
|
|
*
|
2008-11-16 22:29:00 +08:00
|
|
|
* A note on terminology:
|
2015-06-09 02:43:07 +08:00
|
|
|
* - iret frame: Architecture defined interrupt frame from SS to RIP
|
|
|
|
* at the top of the kernel process stack.
|
2006-09-26 16:52:29 +08:00
|
|
|
*
|
|
|
|
* Some macro usage:
|
2015-06-09 02:43:07 +08:00
|
|
|
* - ENTRY/END: Define functions in the symbol table.
|
|
|
|
* - TRACE_IRQ_*: Trace hardirq state for lock debugging.
|
|
|
|
* - idtentry: Define exception entry points.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
|
|
|
#include <linux/linkage.h>
|
|
|
|
#include <asm/segment.h>
|
|
|
|
#include <asm/cache.h>
|
|
|
|
#include <asm/errno.h>
|
2005-09-10 03:28:48 +08:00
|
|
|
#include <asm/asm-offsets.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <asm/msr.h>
|
|
|
|
#include <asm/unistd.h>
|
|
|
|
#include <asm/thread_info.h>
|
|
|
|
#include <asm/hw_irq.h>
|
2009-02-14 03:14:01 +08:00
|
|
|
#include <asm/page_types.h>
|
2006-07-03 15:24:45 +08:00
|
|
|
#include <asm/irqflags.h>
|
2008-01-30 20:32:08 +08:00
|
|
|
#include <asm/paravirt.h>
|
2009-01-13 19:41:35 +08:00
|
|
|
#include <asm/percpu.h>
|
2012-04-21 03:19:50 +08:00
|
|
|
#include <asm/asm.h>
|
2012-09-22 03:43:12 +08:00
|
|
|
#include <asm/smap.h>
|
x86-64, espfix: Don't leak bits 31:16 of %esp returning to 16-bit stack
The IRET instruction, when returning to a 16-bit segment, only
restores the bottom 16 bits of the user space stack pointer. This
causes some 16-bit software to break, but it also leaks kernel state
to user space. We have a software workaround for that ("espfix") for
the 32-bit kernel, but it relies on a nonzero stack segment base which
is not available in 64-bit mode.
In checkin:
b3b42ac2cbae x86-64, modify_ldt: Ban 16-bit segments on 64-bit kernels
we "solved" this by forbidding 16-bit segments on 64-bit kernels, with
the logic that 16-bit support is crippled on 64-bit kernels anyway (no
V86 support), but it turns out that people are doing stuff like
running old Win16 binaries under Wine and expect it to work.
This works around this by creating percpu "ministacks", each of which
is mapped 2^16 times 64K apart. When we detect that the return SS is
on the LDT, we copy the IRET frame to the ministack and use the
relevant alias to return to userspace. The ministacks are mapped
readonly, so if IRET faults we promote #GP to #DF which is an IST
vector and thus has its own stack; we then do the fixup in the #DF
handler.
(Making #GP an IST exception would make the msr_safe functions unsafe
in NMI/MC context, and quite possibly have other effects.)
Special thanks to:
- Andy Lutomirski, for the suggestion of using very small stack slots
and copy (as opposed to map) the IRET frame there, and for the
suggestion to mark them readonly and let the fault promote to #DF.
- Konrad Wilk for paravirt fixup and testing.
- Borislav Petkov for testing help and useful comments.
Reported-by: Brian Gerst <brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Link: http://lkml.kernel.org/r/1398816946-3351-1-git-send-email-hpa@linux.intel.com
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Andrew Lutomriski <amluto@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Dirk Hohndel <dirk@hohndel.org>
Cc: Arjan van de Ven <arjan.van.de.ven@intel.com>
Cc: comex <comexk@gmail.com>
Cc: Alexander van Heukelum <heukelum@fastmail.fm>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: <stable@vger.kernel.org> # consider after upstream merge
2014-04-30 07:46:09 +08:00
|
|
|
#include <asm/pgtable_types.h>
|
2016-01-12 00:04:34 +08:00
|
|
|
#include <asm/export.h>
|
2017-07-11 23:33:44 +08:00
|
|
|
#include <asm/frame.h>
|
2018-01-12 05:46:28 +08:00
|
|
|
#include <asm/nospec-branch.h>
|
2012-01-04 03:23:06 +08:00
|
|
|
#include <linux/err.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2017-12-04 22:07:59 +08:00
|
|
|
#include "calling.h"
|
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
.code64
|
|
|
|
.section .entry.text, "ax"
|
2008-05-13 03:20:42 +08:00
|
|
|
|
2008-01-30 20:32:08 +08:00
|
|
|
#ifdef CONFIG_PARAVIRT
|
2008-06-25 12:19:28 +08:00
|
|
|
ENTRY(native_usergs_sysret64)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_EMPTY
|
2008-01-30 20:32:08 +08:00
|
|
|
swapgs
|
|
|
|
sysretq
|
2017-07-11 23:33:44 +08:00
|
|
|
END(native_usergs_sysret64)
|
2008-01-30 20:32:08 +08:00
|
|
|
#endif /* CONFIG_PARAVIRT */
|
|
|
|
|
2017-11-23 12:39:16 +08:00
|
|
|
.macro TRACE_IRQS_FLAGS flags:req
|
2006-07-03 15:24:45 +08:00
|
|
|
#ifdef CONFIG_TRACE_IRQFLAGS
|
2018-02-26 19:11:21 +08:00
|
|
|
btl $9, \flags /* interrupts off? */
|
2015-06-09 02:43:07 +08:00
|
|
|
jnc 1f
|
2006-07-03 15:24:45 +08:00
|
|
|
TRACE_IRQS_ON
|
|
|
|
1:
|
|
|
|
#endif
|
|
|
|
.endm
|
|
|
|
|
2017-11-23 12:39:16 +08:00
|
|
|
.macro TRACE_IRQS_IRETQ
|
|
|
|
TRACE_IRQS_FLAGS EFLAGS(%rsp)
|
|
|
|
.endm
|
|
|
|
|
2012-05-30 23:54:53 +08:00
|
|
|
/*
|
|
|
|
* When dynamic function tracer is enabled it will add a breakpoint
|
|
|
|
* to all locations that it is about to modify, sync CPUs, update
|
|
|
|
* all the code, sync CPUs, then remove the breakpoints. In this time
|
|
|
|
* if lockdep is enabled, it might jump back into the debug handler
|
|
|
|
* outside the updating of the IST protection. (TRACE_IRQS_ON/OFF).
|
|
|
|
*
|
|
|
|
* We need to change the IDT table before calling TRACE_IRQS_ON/OFF to
|
|
|
|
* make sure the stack pointer does not get reset back to the top
|
|
|
|
* of the debug stack, and instead just reuses the current stack.
|
|
|
|
*/
|
|
|
|
#if defined(CONFIG_DYNAMIC_FTRACE) && defined(CONFIG_TRACE_IRQFLAGS)
|
|
|
|
|
|
|
|
.macro TRACE_IRQS_OFF_DEBUG
|
2015-06-09 02:43:07 +08:00
|
|
|
call debug_stack_set_zero
|
2012-05-30 23:54:53 +08:00
|
|
|
TRACE_IRQS_OFF
|
2015-06-09 02:43:07 +08:00
|
|
|
call debug_stack_reset
|
2012-05-30 23:54:53 +08:00
|
|
|
.endm
|
|
|
|
|
|
|
|
.macro TRACE_IRQS_ON_DEBUG
|
2015-06-09 02:43:07 +08:00
|
|
|
call debug_stack_set_zero
|
2012-05-30 23:54:53 +08:00
|
|
|
TRACE_IRQS_ON
|
2015-06-09 02:43:07 +08:00
|
|
|
call debug_stack_reset
|
2012-05-30 23:54:53 +08:00
|
|
|
.endm
|
|
|
|
|
2015-02-27 06:40:30 +08:00
|
|
|
.macro TRACE_IRQS_IRETQ_DEBUG
|
2018-07-02 18:47:57 +08:00
|
|
|
btl $9, EFLAGS(%rsp) /* interrupts off? */
|
2015-06-09 02:43:07 +08:00
|
|
|
jnc 1f
|
2012-05-30 23:54:53 +08:00
|
|
|
TRACE_IRQS_ON_DEBUG
|
|
|
|
1:
|
|
|
|
.endm
|
|
|
|
|
|
|
|
#else
|
2015-06-09 02:43:07 +08:00
|
|
|
# define TRACE_IRQS_OFF_DEBUG TRACE_IRQS_OFF
|
|
|
|
# define TRACE_IRQS_ON_DEBUG TRACE_IRQS_ON
|
|
|
|
# define TRACE_IRQS_IRETQ_DEBUG TRACE_IRQS_IRETQ
|
2012-05-30 23:54:53 +08:00
|
|
|
#endif
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
2015-06-09 02:43:07 +08:00
|
|
|
* 64-bit SYSCALL instruction entry. Up to 6 arguments in registers.
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
2016-03-10 11:00:35 +08:00
|
|
|
* This is the only entry point used for 64-bit system calls. The
|
|
|
|
* hardware interface is reasonably well designed and the register to
|
|
|
|
* argument mapping Linux uses fits well with the registers that are
|
|
|
|
* available when SYSCALL is used.
|
|
|
|
*
|
|
|
|
* SYSCALL instructions can be found inlined in libc implementations as
|
|
|
|
* well as some other programs and libraries. There are also a handful
|
|
|
|
* of SYSCALL instructions in the vDSO used, for example, as a
|
|
|
|
* clock_gettimeofday fallback.
|
|
|
|
*
|
2015-06-09 02:43:07 +08:00
|
|
|
* 64-bit SYSCALL saves rip to rcx, clears rflags.RF, then saves rflags to r11,
|
2015-02-27 06:40:32 +08:00
|
|
|
* then loads new ss, cs, and rip from previously programmed MSRs.
|
|
|
|
* rflags gets masked by a value from another MSR (so CLD and CLAC
|
|
|
|
* are not needed). SYSCALL does not save anything on the stack
|
|
|
|
* and does not change rsp.
|
|
|
|
*
|
|
|
|
* Registers on entry:
|
2005-04-17 06:20:36 +08:00
|
|
|
* rax system call number
|
2015-02-27 06:40:32 +08:00
|
|
|
* rcx return address
|
|
|
|
* r11 saved rflags (note: r11 is callee-clobbered register in C ABI)
|
2005-04-17 06:20:36 +08:00
|
|
|
* rdi arg0
|
|
|
|
* rsi arg1
|
2008-11-16 22:29:00 +08:00
|
|
|
* rdx arg2
|
2015-02-27 06:40:32 +08:00
|
|
|
* r10 arg3 (needs to be moved to rcx to conform to C ABI)
|
2005-04-17 06:20:36 +08:00
|
|
|
* r8 arg4
|
|
|
|
* r9 arg5
|
2015-06-09 02:43:07 +08:00
|
|
|
* (note: r12-r15, rbp, rbx are callee-preserved in C ABI)
|
2008-11-16 22:29:00 +08:00
|
|
|
*
|
2005-04-17 06:20:36 +08:00
|
|
|
* Only called from user space.
|
|
|
|
*
|
2015-03-17 21:42:59 +08:00
|
|
|
* When user can change pt_regs->foo always force IRET. That is because
|
2006-04-08 01:50:00 +08:00
|
|
|
* it deals with uncanonical addresses better. SYSRET has trouble
|
|
|
|
* with them due to bugs in both AMD and Intel CPUs.
|
2008-11-16 22:29:00 +08:00
|
|
|
*/
|
2005-04-17 06:20:36 +08:00
|
|
|
|
x86/entry/64: Create a per-CPU SYSCALL entry trampoline
Handling SYSCALL is tricky: the SYSCALL handler is entered with every
single register (except FLAGS), including RSP, live. It somehow needs
to set RSP to point to a valid stack, which means it needs to save the
user RSP somewhere and find its own stack pointer. The canonical way
to do this is with SWAPGS, which lets us access percpu data using the
%gs prefix.
With PAGE_TABLE_ISOLATION-like pagetable switching, this is
problematic. Without a scratch register, switching CR3 is impossible, so
%gs-based percpu memory would need to be mapped in the user pagetables.
Doing that without information leaks is difficult or impossible.
Instead, use a different sneaky trick. Map a copy of the first part
of the SYSCALL asm at a different address for each CPU. Now RIP
varies depending on the CPU, so we can use RIP-relative memory access
to access percpu memory. By putting the relevant information (one
scratch slot and the stack address) at a constant offset relative to
RIP, we can make SYSCALL work without relying on %gs.
A nice thing about this approach is that we can easily switch it on
and off if we want pagetable switching to be configurable.
The compat variant of SYSCALL doesn't have this problem in the first
place -- there are plenty of scratch registers, since we don't care
about preserving r8-r15. This patch therefore doesn't touch SYSCALL32
at all.
This patch actually seems to be a small speedup. With this patch,
SYSCALL touches an extra cache line and an extra virtual page, but
the pipeline no longer stalls waiting for SWAPGS. It seems that, at
least in a tight loop, the latter outweights the former.
Thanks to David Laight for an optimization tip.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Borislav Petkov <bpetkov@suse.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Laight <David.Laight@aculab.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Eduardo Valentin <eduval@amazon.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: aliguori@amazon.com
Cc: daniel.gruss@iaik.tugraz.at
Cc: hughd@google.com
Cc: keescook@google.com
Link: https://lkml.kernel.org/r/20171204150606.403607157@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-12-04 22:07:25 +08:00
|
|
|
.pushsection .entry_trampoline, "ax"
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The code in here gets remapped into cpu_entry_area's trampoline. This means
|
|
|
|
* that the assembler and linker have the wrong idea as to where this code
|
|
|
|
* lives (and, in fact, it's mapped more than once, so it's not even at a
|
|
|
|
* fixed address). So we can't reference any symbols outside the entry
|
|
|
|
* trampoline and expect it to work.
|
|
|
|
*
|
|
|
|
* Instead, we carefully abuse %rip-relative addressing.
|
|
|
|
* _entry_trampoline(%rip) refers to the start of the remapped) entry
|
|
|
|
* trampoline. We can thus find cpu_entry_area with this macro:
|
|
|
|
*/
|
|
|
|
|
|
|
|
#define CPU_ENTRY_AREA \
|
|
|
|
_entry_trampoline - CPU_ENTRY_AREA_entry_trampoline(%rip)
|
|
|
|
|
|
|
|
/* The top word of the SYSENTER stack is hot and is usable as scratch space. */
|
2017-12-05 09:25:07 +08:00
|
|
|
#define RSP_SCRATCH CPU_ENTRY_AREA_entry_stack + \
|
|
|
|
SIZEOF_entry_stack - 8 + CPU_ENTRY_AREA
|
x86/entry/64: Create a per-CPU SYSCALL entry trampoline
Handling SYSCALL is tricky: the SYSCALL handler is entered with every
single register (except FLAGS), including RSP, live. It somehow needs
to set RSP to point to a valid stack, which means it needs to save the
user RSP somewhere and find its own stack pointer. The canonical way
to do this is with SWAPGS, which lets us access percpu data using the
%gs prefix.
With PAGE_TABLE_ISOLATION-like pagetable switching, this is
problematic. Without a scratch register, switching CR3 is impossible, so
%gs-based percpu memory would need to be mapped in the user pagetables.
Doing that without information leaks is difficult or impossible.
Instead, use a different sneaky trick. Map a copy of the first part
of the SYSCALL asm at a different address for each CPU. Now RIP
varies depending on the CPU, so we can use RIP-relative memory access
to access percpu memory. By putting the relevant information (one
scratch slot and the stack address) at a constant offset relative to
RIP, we can make SYSCALL work without relying on %gs.
A nice thing about this approach is that we can easily switch it on
and off if we want pagetable switching to be configurable.
The compat variant of SYSCALL doesn't have this problem in the first
place -- there are plenty of scratch registers, since we don't care
about preserving r8-r15. This patch therefore doesn't touch SYSCALL32
at all.
This patch actually seems to be a small speedup. With this patch,
SYSCALL touches an extra cache line and an extra virtual page, but
the pipeline no longer stalls waiting for SWAPGS. It seems that, at
least in a tight loop, the latter outweights the former.
Thanks to David Laight for an optimization tip.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Borislav Petkov <bpetkov@suse.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Laight <David.Laight@aculab.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Eduardo Valentin <eduval@amazon.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: aliguori@amazon.com
Cc: daniel.gruss@iaik.tugraz.at
Cc: hughd@google.com
Cc: keescook@google.com
Link: https://lkml.kernel.org/r/20171204150606.403607157@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-12-04 22:07:25 +08:00
|
|
|
|
|
|
|
ENTRY(entry_SYSCALL_64_trampoline)
|
|
|
|
UNWIND_HINT_EMPTY
|
|
|
|
swapgs
|
|
|
|
|
|
|
|
/* Stash the user RSP. */
|
|
|
|
movq %rsp, RSP_SCRATCH
|
|
|
|
|
2017-12-04 22:07:35 +08:00
|
|
|
/* Note: using %rsp as a scratch reg. */
|
|
|
|
SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
|
|
|
|
|
x86/entry/64: Create a per-CPU SYSCALL entry trampoline
Handling SYSCALL is tricky: the SYSCALL handler is entered with every
single register (except FLAGS), including RSP, live. It somehow needs
to set RSP to point to a valid stack, which means it needs to save the
user RSP somewhere and find its own stack pointer. The canonical way
to do this is with SWAPGS, which lets us access percpu data using the
%gs prefix.
With PAGE_TABLE_ISOLATION-like pagetable switching, this is
problematic. Without a scratch register, switching CR3 is impossible, so
%gs-based percpu memory would need to be mapped in the user pagetables.
Doing that without information leaks is difficult or impossible.
Instead, use a different sneaky trick. Map a copy of the first part
of the SYSCALL asm at a different address for each CPU. Now RIP
varies depending on the CPU, so we can use RIP-relative memory access
to access percpu memory. By putting the relevant information (one
scratch slot and the stack address) at a constant offset relative to
RIP, we can make SYSCALL work without relying on %gs.
A nice thing about this approach is that we can easily switch it on
and off if we want pagetable switching to be configurable.
The compat variant of SYSCALL doesn't have this problem in the first
place -- there are plenty of scratch registers, since we don't care
about preserving r8-r15. This patch therefore doesn't touch SYSCALL32
at all.
This patch actually seems to be a small speedup. With this patch,
SYSCALL touches an extra cache line and an extra virtual page, but
the pipeline no longer stalls waiting for SWAPGS. It seems that, at
least in a tight loop, the latter outweights the former.
Thanks to David Laight for an optimization tip.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Borislav Petkov <bpetkov@suse.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Laight <David.Laight@aculab.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Eduardo Valentin <eduval@amazon.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: aliguori@amazon.com
Cc: daniel.gruss@iaik.tugraz.at
Cc: hughd@google.com
Cc: keescook@google.com
Link: https://lkml.kernel.org/r/20171204150606.403607157@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-12-04 22:07:25 +08:00
|
|
|
/* Load the top of the task stack into RSP */
|
|
|
|
movq CPU_ENTRY_AREA_tss + TSS_sp1 + CPU_ENTRY_AREA, %rsp
|
|
|
|
|
|
|
|
/* Start building the simulated IRET frame. */
|
|
|
|
pushq $__USER_DS /* pt_regs->ss */
|
|
|
|
pushq RSP_SCRATCH /* pt_regs->sp */
|
|
|
|
pushq %r11 /* pt_regs->flags */
|
|
|
|
pushq $__USER_CS /* pt_regs->cs */
|
|
|
|
pushq %rcx /* pt_regs->ip */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* x86 lacks a near absolute jump, and we can't jump to the real
|
|
|
|
* entry text with a relative jump. We could push the target
|
|
|
|
* address and then use retq, but this destroys the pipeline on
|
|
|
|
* many CPUs (wasting over 20 cycles on Sandy Bridge). Instead,
|
|
|
|
* spill RDI and restore it in a second-stage trampoline.
|
|
|
|
*/
|
|
|
|
pushq %rdi
|
|
|
|
movq $entry_SYSCALL_64_stage2, %rdi
|
2018-01-12 05:46:28 +08:00
|
|
|
JMP_NOSPEC %rdi
|
x86/entry/64: Create a per-CPU SYSCALL entry trampoline
Handling SYSCALL is tricky: the SYSCALL handler is entered with every
single register (except FLAGS), including RSP, live. It somehow needs
to set RSP to point to a valid stack, which means it needs to save the
user RSP somewhere and find its own stack pointer. The canonical way
to do this is with SWAPGS, which lets us access percpu data using the
%gs prefix.
With PAGE_TABLE_ISOLATION-like pagetable switching, this is
problematic. Without a scratch register, switching CR3 is impossible, so
%gs-based percpu memory would need to be mapped in the user pagetables.
Doing that without information leaks is difficult or impossible.
Instead, use a different sneaky trick. Map a copy of the first part
of the SYSCALL asm at a different address for each CPU. Now RIP
varies depending on the CPU, so we can use RIP-relative memory access
to access percpu memory. By putting the relevant information (one
scratch slot and the stack address) at a constant offset relative to
RIP, we can make SYSCALL work without relying on %gs.
A nice thing about this approach is that we can easily switch it on
and off if we want pagetable switching to be configurable.
The compat variant of SYSCALL doesn't have this problem in the first
place -- there are plenty of scratch registers, since we don't care
about preserving r8-r15. This patch therefore doesn't touch SYSCALL32
at all.
This patch actually seems to be a small speedup. With this patch,
SYSCALL touches an extra cache line and an extra virtual page, but
the pipeline no longer stalls waiting for SWAPGS. It seems that, at
least in a tight loop, the latter outweights the former.
Thanks to David Laight for an optimization tip.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Borislav Petkov <bpetkov@suse.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Laight <David.Laight@aculab.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Eduardo Valentin <eduval@amazon.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: aliguori@amazon.com
Cc: daniel.gruss@iaik.tugraz.at
Cc: hughd@google.com
Cc: keescook@google.com
Link: https://lkml.kernel.org/r/20171204150606.403607157@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-12-04 22:07:25 +08:00
|
|
|
END(entry_SYSCALL_64_trampoline)
|
|
|
|
|
|
|
|
.popsection
|
|
|
|
|
|
|
|
ENTRY(entry_SYSCALL_64_stage2)
|
|
|
|
UNWIND_HINT_EMPTY
|
|
|
|
popq %rdi
|
|
|
|
jmp entry_SYSCALL_64_after_hwframe
|
|
|
|
END(entry_SYSCALL_64_stage2)
|
|
|
|
|
2015-06-08 14:42:03 +08:00
|
|
|
ENTRY(entry_SYSCALL_64)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_EMPTY
|
2015-03-20 01:17:47 +08:00
|
|
|
/*
|
|
|
|
* Interrupts are off on entry.
|
|
|
|
* We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON,
|
|
|
|
* it is too small to ever cause noticeable irq latency.
|
|
|
|
*/
|
2008-01-30 20:32:08 +08:00
|
|
|
|
2017-08-08 11:59:21 +08:00
|
|
|
swapgs
|
2017-12-04 22:07:35 +08:00
|
|
|
/*
|
2018-02-10 01:06:38 +08:00
|
|
|
* This path is only taken when PAGE_TABLE_ISOLATION is disabled so it
|
2017-12-04 22:07:35 +08:00
|
|
|
* is not required to switch CR3.
|
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rsp, PER_CPU_VAR(rsp_scratch)
|
|
|
|
movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp
|
2015-03-20 01:17:47 +08:00
|
|
|
|
|
|
|
/* Construct struct pt_regs on stack */
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq $__USER_DS /* pt_regs->ss */
|
|
|
|
pushq PER_CPU_VAR(rsp_scratch) /* pt_regs->sp */
|
|
|
|
pushq %r11 /* pt_regs->flags */
|
|
|
|
pushq $__USER_CS /* pt_regs->cs */
|
|
|
|
pushq %rcx /* pt_regs->ip */
|
2017-08-08 11:59:21 +08:00
|
|
|
GLOBAL(entry_SYSCALL_64_after_hwframe)
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq %rax /* pt_regs->orig_ax */
|
2018-02-11 18:49:46 +08:00
|
|
|
|
|
|
|
PUSH_AND_CLEAR_REGS rax=$-ENOSYS
|
2015-06-09 02:43:07 +08:00
|
|
|
|
2017-11-22 12:43:56 +08:00
|
|
|
TRACE_IRQS_OFF
|
|
|
|
|
2016-01-29 07:11:28 +08:00
|
|
|
/* IRQs are off. */
|
2018-04-05 17:53:00 +08:00
|
|
|
movq %rax, %rdi
|
|
|
|
movq %rsp, %rsi
|
2016-01-29 07:11:28 +08:00
|
|
|
call do_syscall_64 /* returns with IRQs disabled */
|
|
|
|
|
2015-07-04 03:44:28 +08:00
|
|
|
TRACE_IRQS_IRETQ /* we're about to change IF */
|
2015-04-03 00:46:59 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Try to use SYSRET instead of IRET if we're returning to
|
2017-11-02 15:59:00 +08:00
|
|
|
* a completely clean 64-bit userspace context. If we're not,
|
|
|
|
* go to the slow exit path.
|
2015-04-03 00:46:59 +08:00
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
movq RCX(%rsp), %rcx
|
|
|
|
movq RIP(%rsp), %r11
|
2017-11-02 15:59:00 +08:00
|
|
|
|
|
|
|
cmpq %rcx, %r11 /* SYSRET requires RCX == RIP */
|
|
|
|
jne swapgs_restore_regs_and_return_to_usermode
|
2015-04-03 00:46:59 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* On Intel CPUs, SYSRET with non-canonical RCX/RIP will #GP
|
|
|
|
* in kernel space. This essentially lets the user take over
|
x86/asm/entry/64: Implement better check for canonical addresses
This change makes the check exact (no more false positives
on "negative" addresses).
Andy explains:
"Canonical addresses either start with 17 zeros or 17 ones.
In the old code, we checked that the top (64-47) = 17 bits were all
zero. We did this by shifting right by 47 bits and making sure that
nothing was left.
In the new code, we're shifting left by (64 - 48) = 16 bits and then
signed shifting right by the same amount, this propagating the 17th
highest bit to all positions to its left. If we get the same value we
started with, then we're good to go."
While it isn't really important to be fully correct here -
almost all addresses we'll ever see will be userspace ones,
but OTOH it looks to be cheap enough: the new code uses
two more ALU ops but preserves %rcx, allowing to not
reload it from pt_regs->cx again.
On disassembly level, the changes are:
cmp %rcx,0x80(%rsp) -> mov 0x80(%rsp),%r11; cmp %rcx,%r11
shr $0x2f,%rcx -> shl $0x10,%rcx; sar $0x10,%rcx; cmp %rcx,%r11
mov 0x58(%rsp),%rcx -> (eliminated)
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1429633649-20169-1-git-send-email-dvlasenk@redhat.com
[ Changelog massage. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-04-22 00:27:29 +08:00
|
|
|
* the kernel, since userspace controls RSP.
|
2015-04-03 00:46:59 +08:00
|
|
|
*
|
x86/asm/entry/64: Implement better check for canonical addresses
This change makes the check exact (no more false positives
on "negative" addresses).
Andy explains:
"Canonical addresses either start with 17 zeros or 17 ones.
In the old code, we checked that the top (64-47) = 17 bits were all
zero. We did this by shifting right by 47 bits and making sure that
nothing was left.
In the new code, we're shifting left by (64 - 48) = 16 bits and then
signed shifting right by the same amount, this propagating the 17th
highest bit to all positions to its left. If we get the same value we
started with, then we're good to go."
While it isn't really important to be fully correct here -
almost all addresses we'll ever see will be userspace ones,
but OTOH it looks to be cheap enough: the new code uses
two more ALU ops but preserves %rcx, allowing to not
reload it from pt_regs->cx again.
On disassembly level, the changes are:
cmp %rcx,0x80(%rsp) -> mov 0x80(%rsp),%r11; cmp %rcx,%r11
shr $0x2f,%rcx -> shl $0x10,%rcx; sar $0x10,%rcx; cmp %rcx,%r11
mov 0x58(%rsp),%rcx -> (eliminated)
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1429633649-20169-1-git-send-email-dvlasenk@redhat.com
[ Changelog massage. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-04-22 00:27:29 +08:00
|
|
|
* If width of "canonical tail" ever becomes variable, this will need
|
2015-04-03 00:46:59 +08:00
|
|
|
* to be updated to remain correct on both old and new CPUs.
|
2017-03-30 16:07:26 +08:00
|
|
|
*
|
2017-06-06 19:31:21 +08:00
|
|
|
* Change top bits to match most significant bit (47th or 56th bit
|
|
|
|
* depending on paging mode) in the address.
|
2015-04-03 00:46:59 +08:00
|
|
|
*/
|
2018-02-14 19:16:55 +08:00
|
|
|
#ifdef CONFIG_X86_5LEVEL
|
x86/mm: Optimize boot-time paging mode switching cost
By this point we have functioning boot-time switching between 4- and
5-level paging mode. But naive approach comes with cost.
Numbers below are for kernel build, allmodconfig, 5 times.
CONFIG_X86_5LEVEL=n:
Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):
17308719.892691 task-clock:u (msec) # 26.772 CPUs utilized ( +- 0.11% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
331,993,164 page-faults:u # 0.019 M/sec ( +- 0.01% )
43,614,978,867,455 cycles:u # 2.520 GHz ( +- 0.01% )
39,371,534,575,126 stalled-cycles-frontend:u # 90.27% frontend cycles idle ( +- 0.09% )
28,363,350,152,428 instructions:u # 0.65 insn per cycle
# 1.39 stalled cycles per insn ( +- 0.00% )
6,316,784,066,413 branches:u # 364.948 M/sec ( +- 0.00% )
250,808,144,781 branch-misses:u # 3.97% of all branches ( +- 0.01% )
646.531974142 seconds time elapsed ( +- 1.15% )
CONFIG_X86_5LEVEL=y:
Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):
17411536.780625 task-clock:u (msec) # 26.426 CPUs utilized ( +- 0.10% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
331,868,663 page-faults:u # 0.019 M/sec ( +- 0.01% )
43,865,909,056,301 cycles:u # 2.519 GHz ( +- 0.01% )
39,740,130,365,581 stalled-cycles-frontend:u # 90.59% frontend cycles idle ( +- 0.05% )
28,363,358,997,959 instructions:u # 0.65 insn per cycle
# 1.40 stalled cycles per insn ( +- 0.00% )
6,316,784,937,460 branches:u # 362.793 M/sec ( +- 0.00% )
251,531,919,485 branch-misses:u # 3.98% of all branches ( +- 0.00% )
658.886307752 seconds time elapsed ( +- 0.92% )
The patch tries to fix the performance regression by using
cpu_feature_enabled(X86_FEATURE_LA57) instead of pgtable_l5_enabled in
all hot code paths. These will statically patch the target code for
additional performance.
CONFIG_X86_5LEVEL=y + the patch:
Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):
17381990.268506 task-clock:u (msec) # 26.907 CPUs utilized ( +- 0.19% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
331,862,625 page-faults:u # 0.019 M/sec ( +- 0.01% )
43,697,726,320,051 cycles:u # 2.514 GHz ( +- 0.03% )
39,480,408,690,401 stalled-cycles-frontend:u # 90.35% frontend cycles idle ( +- 0.05% )
28,363,394,221,388 instructions:u # 0.65 insn per cycle
# 1.39 stalled cycles per insn ( +- 0.00% )
6,316,794,985,573 branches:u # 363.410 M/sec ( +- 0.00% )
251,013,232,547 branch-misses:u # 3.97% of all branches ( +- 0.01% )
645.991174661 seconds time elapsed ( +- 1.19% )
Unfortunately, this approach doesn't help with text size:
vmlinux.before .text size: 8190319
vmlinux.after .text size: 8200623
The .text section is increased by about 4k. Not sure if we can do anything
about this.
Signed-off-by: Kirill A. Shuemov <kirill.shutemov@linux.intel.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Borislav Petkov <bp@suse.de>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/20180216114948.68868-4-kirill.shutemov@linux.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-02-16 19:49:48 +08:00
|
|
|
ALTERNATIVE "shl $(64 - 48), %rcx; sar $(64 - 48), %rcx", \
|
|
|
|
"shl $(64 - 57), %rcx; sar $(64 - 57), %rcx", X86_FEATURE_LA57
|
2018-02-14 19:16:55 +08:00
|
|
|
#else
|
x86/asm/entry/64: Implement better check for canonical addresses
This change makes the check exact (no more false positives
on "negative" addresses).
Andy explains:
"Canonical addresses either start with 17 zeros or 17 ones.
In the old code, we checked that the top (64-47) = 17 bits were all
zero. We did this by shifting right by 47 bits and making sure that
nothing was left.
In the new code, we're shifting left by (64 - 48) = 16 bits and then
signed shifting right by the same amount, this propagating the 17th
highest bit to all positions to its left. If we get the same value we
started with, then we're good to go."
While it isn't really important to be fully correct here -
almost all addresses we'll ever see will be userspace ones,
but OTOH it looks to be cheap enough: the new code uses
two more ALU ops but preserves %rcx, allowing to not
reload it from pt_regs->cx again.
On disassembly level, the changes are:
cmp %rcx,0x80(%rsp) -> mov 0x80(%rsp),%r11; cmp %rcx,%r11
shr $0x2f,%rcx -> shl $0x10,%rcx; sar $0x10,%rcx; cmp %rcx,%r11
mov 0x58(%rsp),%rcx -> (eliminated)
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1429633649-20169-1-git-send-email-dvlasenk@redhat.com
[ Changelog massage. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-04-22 00:27:29 +08:00
|
|
|
shl $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
|
|
|
|
sar $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
|
2018-02-14 19:16:55 +08:00
|
|
|
#endif
|
2015-06-09 02:43:07 +08:00
|
|
|
|
x86/asm/entry/64: Implement better check for canonical addresses
This change makes the check exact (no more false positives
on "negative" addresses).
Andy explains:
"Canonical addresses either start with 17 zeros or 17 ones.
In the old code, we checked that the top (64-47) = 17 bits were all
zero. We did this by shifting right by 47 bits and making sure that
nothing was left.
In the new code, we're shifting left by (64 - 48) = 16 bits and then
signed shifting right by the same amount, this propagating the 17th
highest bit to all positions to its left. If we get the same value we
started with, then we're good to go."
While it isn't really important to be fully correct here -
almost all addresses we'll ever see will be userspace ones,
but OTOH it looks to be cheap enough: the new code uses
two more ALU ops but preserves %rcx, allowing to not
reload it from pt_regs->cx again.
On disassembly level, the changes are:
cmp %rcx,0x80(%rsp) -> mov 0x80(%rsp),%r11; cmp %rcx,%r11
shr $0x2f,%rcx -> shl $0x10,%rcx; sar $0x10,%rcx; cmp %rcx,%r11
mov 0x58(%rsp),%rcx -> (eliminated)
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1429633649-20169-1-git-send-email-dvlasenk@redhat.com
[ Changelog massage. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-04-22 00:27:29 +08:00
|
|
|
/* If this changed %rcx, it was not canonical */
|
|
|
|
cmpq %rcx, %r11
|
2017-11-02 15:59:00 +08:00
|
|
|
jne swapgs_restore_regs_and_return_to_usermode
|
2015-04-03 00:46:59 +08:00
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
cmpq $__USER_CS, CS(%rsp) /* CS must match SYSRET */
|
2017-11-02 15:59:00 +08:00
|
|
|
jne swapgs_restore_regs_and_return_to_usermode
|
2015-04-03 00:46:59 +08:00
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
movq R11(%rsp), %r11
|
|
|
|
cmpq %r11, EFLAGS(%rsp) /* R11 == RFLAGS */
|
2017-11-02 15:59:00 +08:00
|
|
|
jne swapgs_restore_regs_and_return_to_usermode
|
2015-04-03 00:46:59 +08:00
|
|
|
|
|
|
|
/*
|
2016-08-04 01:14:29 +08:00
|
|
|
* SYSCALL clears RF when it saves RFLAGS in R11 and SYSRET cannot
|
|
|
|
* restore RF properly. If the slowpath sets it for whatever reason, we
|
|
|
|
* need to restore it correctly.
|
|
|
|
*
|
|
|
|
* SYSRET can restore TF, but unlike IRET, restoring TF results in a
|
|
|
|
* trap from userspace immediately after SYSRET. This would cause an
|
|
|
|
* infinite loop whenever #DB happens with register state that satisfies
|
|
|
|
* the opportunistic SYSRET conditions. For example, single-stepping
|
|
|
|
* this user code:
|
2015-04-03 00:46:59 +08:00
|
|
|
*
|
2015-06-09 02:43:07 +08:00
|
|
|
* movq $stuck_here, %rcx
|
2015-04-03 00:46:59 +08:00
|
|
|
* pushfq
|
|
|
|
* popq %r11
|
|
|
|
* stuck_here:
|
|
|
|
*
|
|
|
|
* would never get past 'stuck_here'.
|
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
testq $(X86_EFLAGS_RF|X86_EFLAGS_TF), %r11
|
2017-11-02 15:59:00 +08:00
|
|
|
jnz swapgs_restore_regs_and_return_to_usermode
|
2015-04-03 00:46:59 +08:00
|
|
|
|
|
|
|
/* nothing to check for RSP */
|
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
cmpq $__USER_DS, SS(%rsp) /* SS must match SYSRET */
|
2017-11-02 15:59:00 +08:00
|
|
|
jne swapgs_restore_regs_and_return_to_usermode
|
2015-04-03 00:46:59 +08:00
|
|
|
|
|
|
|
/*
|
2015-06-09 02:43:07 +08:00
|
|
|
* We win! This label is here just for ease of understanding
|
|
|
|
* perf profiles. Nothing jumps here.
|
2015-04-03 00:46:59 +08:00
|
|
|
*/
|
|
|
|
syscall_return_via_sysret:
|
x86/asm/entry/64: Implement better check for canonical addresses
This change makes the check exact (no more false positives
on "negative" addresses).
Andy explains:
"Canonical addresses either start with 17 zeros or 17 ones.
In the old code, we checked that the top (64-47) = 17 bits were all
zero. We did this by shifting right by 47 bits and making sure that
nothing was left.
In the new code, we're shifting left by (64 - 48) = 16 bits and then
signed shifting right by the same amount, this propagating the 17th
highest bit to all positions to its left. If we get the same value we
started with, then we're good to go."
While it isn't really important to be fully correct here -
almost all addresses we'll ever see will be userspace ones,
but OTOH it looks to be cheap enough: the new code uses
two more ALU ops but preserves %rcx, allowing to not
reload it from pt_regs->cx again.
On disassembly level, the changes are:
cmp %rcx,0x80(%rsp) -> mov 0x80(%rsp),%r11; cmp %rcx,%r11
shr $0x2f,%rcx -> shl $0x10,%rcx; sar $0x10,%rcx; cmp %rcx,%r11
mov 0x58(%rsp),%rcx -> (eliminated)
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1429633649-20169-1-git-send-email-dvlasenk@redhat.com
[ Changelog massage. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-04-22 00:27:29 +08:00
|
|
|
/* rcx and r11 are already restored (see code above) */
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_EMPTY
|
2018-02-11 18:49:43 +08:00
|
|
|
POP_REGS pop_rdi=0 skip_r11rcx=1
|
2017-12-04 22:07:24 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Now all regs are restored except RSP and RDI.
|
|
|
|
* Save old stack pointer and switch to trampoline stack.
|
|
|
|
*/
|
|
|
|
movq %rsp, %rdi
|
2017-12-04 22:07:29 +08:00
|
|
|
movq PER_CPU_VAR(cpu_tss_rw + TSS_sp0), %rsp
|
2017-12-04 22:07:24 +08:00
|
|
|
|
|
|
|
pushq RSP-RDI(%rdi) /* RSP */
|
|
|
|
pushq (%rdi) /* RDI */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We are on the trampoline stack. All regs except RDI are live.
|
|
|
|
* We can do future final exit work right here.
|
|
|
|
*/
|
2017-12-04 22:07:59 +08:00
|
|
|
SWITCH_TO_USER_CR3_STACK scratch_reg=%rdi
|
2017-12-04 22:07:24 +08:00
|
|
|
|
2017-11-02 15:59:03 +08:00
|
|
|
popq %rdi
|
2017-12-04 22:07:24 +08:00
|
|
|
popq %rsp
|
2015-04-03 00:46:59 +08:00
|
|
|
USERGS_SYSRET64
|
2015-06-08 14:42:03 +08:00
|
|
|
END(entry_SYSCALL_64)
|
2008-11-16 22:29:00 +08:00
|
|
|
|
2016-08-14 00:38:19 +08:00
|
|
|
/*
|
|
|
|
* %rdi: prev task
|
|
|
|
* %rsi: next task
|
|
|
|
*/
|
|
|
|
ENTRY(__switch_to_asm)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_FUNC
|
2016-08-14 00:38:19 +08:00
|
|
|
/*
|
|
|
|
* Save callee-saved registers
|
|
|
|
* This must match the order in inactive_task_frame
|
|
|
|
*/
|
|
|
|
pushq %rbp
|
|
|
|
pushq %rbx
|
|
|
|
pushq %r12
|
|
|
|
pushq %r13
|
|
|
|
pushq %r14
|
|
|
|
pushq %r15
|
|
|
|
|
|
|
|
/* switch stack */
|
|
|
|
movq %rsp, TASK_threadsp(%rdi)
|
|
|
|
movq TASK_threadsp(%rsi), %rsp
|
|
|
|
|
Kbuild: rename CC_STACKPROTECTOR[_STRONG] config variables
The changes to automatically test for working stack protector compiler
support in the Kconfig files removed the special STACKPROTECTOR_AUTO
option that picked the strongest stack protector that the compiler
supported.
That was all a nice cleanup - it makes no sense to have the AUTO case
now that the Kconfig phase can just determine the compiler support
directly.
HOWEVER.
It also meant that doing "make oldconfig" would now _disable_ the strong
stackprotector if you had AUTO enabled, because in a legacy config file,
the sane stack protector configuration would look like
CONFIG_HAVE_CC_STACKPROTECTOR=y
# CONFIG_CC_STACKPROTECTOR_NONE is not set
# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
CONFIG_CC_STACKPROTECTOR_AUTO=y
and when you ran this through "make oldconfig" with the Kbuild changes,
it would ask you about the regular CONFIG_CC_STACKPROTECTOR (that had
been renamed from CONFIG_CC_STACKPROTECTOR_REGULAR to just
CONFIG_CC_STACKPROTECTOR), but it would think that the STRONG version
used to be disabled (because it was really enabled by AUTO), and would
disable it in the new config, resulting in:
CONFIG_HAVE_CC_STACKPROTECTOR=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_CC_STACKPROTECTOR=y
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
That's dangerously subtle - people could suddenly find themselves with
the weaker stack protector setup without even realizing.
The solution here is to just rename not just the old RECULAR stack
protector option, but also the strong one. This does that by just
removing the CC_ prefix entirely for the user choices, because it really
is not about the compiler support (the compiler support now instead
automatially impacts _visibility_ of the options to users).
This results in "make oldconfig" actually asking the user for their
choice, so that we don't have any silent subtle security model changes.
The end result would generally look like this:
CONFIG_HAVE_CC_STACKPROTECTOR=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_STACKPROTECTOR=y
CONFIG_STACKPROTECTOR_STRONG=y
CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
where the "CC_" versions really are about internal compiler
infrastructure, not the user selections.
Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-06-14 11:21:18 +08:00
|
|
|
#ifdef CONFIG_STACKPROTECTOR
|
2016-08-14 00:38:19 +08:00
|
|
|
movq TASK_stack_canary(%rsi), %rbx
|
|
|
|
movq %rbx, PER_CPU_VAR(irq_stack_union)+stack_canary_offset
|
|
|
|
#endif
|
|
|
|
|
2018-01-13 01:49:25 +08:00
|
|
|
#ifdef CONFIG_RETPOLINE
|
|
|
|
/*
|
|
|
|
* When switching from a shallower to a deeper call stack
|
|
|
|
* the RSB may either underflow or use entries populated
|
|
|
|
* with userspace addresses. On CPUs where those concerns
|
|
|
|
* exist, overwrite the RSB with entries which capture
|
|
|
|
* speculative execution to prevent attack.
|
|
|
|
*/
|
2018-02-19 18:50:56 +08:00
|
|
|
FILL_RETURN_BUFFER %r12, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW
|
2018-01-13 01:49:25 +08:00
|
|
|
#endif
|
|
|
|
|
2016-08-14 00:38:19 +08:00
|
|
|
/* restore callee-saved registers */
|
|
|
|
popq %r15
|
|
|
|
popq %r14
|
|
|
|
popq %r13
|
|
|
|
popq %r12
|
|
|
|
popq %rbx
|
|
|
|
popq %rbp
|
|
|
|
|
|
|
|
jmp __switch_to
|
|
|
|
END(__switch_to_asm)
|
|
|
|
|
2015-02-27 06:40:33 +08:00
|
|
|
/*
|
|
|
|
* A newly forked process directly context switches into this address.
|
|
|
|
*
|
2016-08-14 00:38:19 +08:00
|
|
|
* rax: prev task we switched from
|
2016-08-14 00:38:20 +08:00
|
|
|
* rbx: kernel thread func (NULL for user thread)
|
|
|
|
* r12: kernel thread arg
|
2015-02-27 06:40:33 +08:00
|
|
|
*/
|
|
|
|
ENTRY(ret_from_fork)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_EMPTY
|
2016-08-14 00:38:19 +08:00
|
|
|
movq %rax, %rdi
|
2017-05-23 23:37:29 +08:00
|
|
|
call schedule_tail /* rdi: 'prev' task parameter */
|
2015-02-27 06:40:33 +08:00
|
|
|
|
2017-05-23 23:37:29 +08:00
|
|
|
testq %rbx, %rbx /* from kernel_thread? */
|
|
|
|
jnz 1f /* kernel threads are uncommon */
|
2016-01-29 07:11:27 +08:00
|
|
|
|
2016-08-14 00:38:20 +08:00
|
|
|
2:
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_REGS
|
2017-05-23 23:37:29 +08:00
|
|
|
movq %rsp, %rdi
|
2016-01-29 07:11:27 +08:00
|
|
|
call syscall_return_slowpath /* returns with IRQs disabled */
|
|
|
|
TRACE_IRQS_ON /* user mode is traced as IRQS on */
|
2017-11-02 15:59:00 +08:00
|
|
|
jmp swapgs_restore_regs_and_return_to_usermode
|
2016-08-14 00:38:20 +08:00
|
|
|
|
|
|
|
1:
|
|
|
|
/* kernel thread */
|
2018-05-18 14:47:12 +08:00
|
|
|
UNWIND_HINT_EMPTY
|
2016-08-14 00:38:20 +08:00
|
|
|
movq %r12, %rdi
|
2018-01-12 05:46:28 +08:00
|
|
|
CALL_NOSPEC %rbx
|
2016-08-14 00:38:20 +08:00
|
|
|
/*
|
|
|
|
* A kernel thread is allowed to return here after successfully
|
|
|
|
* calling do_execve(). Exit to userspace to complete the execve()
|
|
|
|
* syscall.
|
|
|
|
*/
|
|
|
|
movq $0, RAX(%rsp)
|
|
|
|
jmp 2b
|
2015-02-27 06:40:33 +08:00
|
|
|
END(ret_from_fork)
|
|
|
|
|
2008-11-12 05:51:52 +08:00
|
|
|
/*
|
2015-04-04 03:49:13 +08:00
|
|
|
* Build the entry stubs with some assembler magic.
|
|
|
|
* We pack 1 stub into every 8-byte block.
|
2008-11-12 05:51:52 +08:00
|
|
|
*/
|
2015-04-04 03:49:13 +08:00
|
|
|
.align 8
|
2008-11-12 05:51:52 +08:00
|
|
|
ENTRY(irq_entries_start)
|
2015-04-04 03:49:13 +08:00
|
|
|
vector=FIRST_EXTERNAL_VECTOR
|
|
|
|
.rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq $(~vector+0x80) /* Note: always in signed byte range */
|
2015-04-04 03:49:13 +08:00
|
|
|
jmp common_interrupt
|
|
|
|
.align 8
|
2017-07-11 23:33:44 +08:00
|
|
|
vector=vector+1
|
2015-04-04 03:49:13 +08:00
|
|
|
.endr
|
2008-11-12 05:51:52 +08:00
|
|
|
END(irq_entries_start)
|
|
|
|
|
2017-07-11 23:33:38 +08:00
|
|
|
.macro DEBUG_ENTRY_ASSERT_IRQS_OFF
|
|
|
|
#ifdef CONFIG_DEBUG_ENTRY
|
2017-12-04 22:07:07 +08:00
|
|
|
pushq %rax
|
|
|
|
SAVE_FLAGS(CLBR_RAX)
|
|
|
|
testl $X86_EFLAGS_IF, %eax
|
2017-07-11 23:33:38 +08:00
|
|
|
jz .Lokay_\@
|
|
|
|
ud2
|
|
|
|
.Lokay_\@:
|
2017-12-04 22:07:07 +08:00
|
|
|
popq %rax
|
2017-07-11 23:33:38 +08:00
|
|
|
#endif
|
|
|
|
.endm
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Enters the IRQ stack if we're not already using it. NMI-safe. Clobbers
|
|
|
|
* flags and puts old RSP into old_rsp, and leaves all other GPRs alone.
|
|
|
|
* Requires kernel GSBASE.
|
|
|
|
*
|
|
|
|
* The invariant is that, if irq_count != -1, then the IRQ stack is in use.
|
|
|
|
*/
|
2018-02-21 05:01:09 +08:00
|
|
|
.macro ENTER_IRQ_STACK regs=1 old_rsp save_ret=0
|
2017-07-11 23:33:38 +08:00
|
|
|
DEBUG_ENTRY_ASSERT_IRQS_OFF
|
2018-02-21 05:01:09 +08:00
|
|
|
|
|
|
|
.if \save_ret
|
|
|
|
/*
|
|
|
|
* If save_ret is set, the original stack contains one additional
|
|
|
|
* entry -- the return address. Therefore, move the address one
|
|
|
|
* entry below %rsp to \old_rsp.
|
|
|
|
*/
|
|
|
|
leaq 8(%rsp), \old_rsp
|
|
|
|
.else
|
2017-07-11 23:33:38 +08:00
|
|
|
movq %rsp, \old_rsp
|
2018-02-21 05:01:09 +08:00
|
|
|
.endif
|
2017-07-11 23:33:44 +08:00
|
|
|
|
|
|
|
.if \regs
|
|
|
|
UNWIND_HINT_REGS base=\old_rsp
|
|
|
|
.endif
|
|
|
|
|
2017-07-11 23:33:38 +08:00
|
|
|
incl PER_CPU_VAR(irq_count)
|
2017-07-11 23:33:39 +08:00
|
|
|
jnz .Lirq_stack_push_old_rsp_\@
|
2017-07-11 23:33:38 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Right now, if we just incremented irq_count to zero, we've
|
|
|
|
* claimed the IRQ stack but we haven't switched to it yet.
|
|
|
|
*
|
|
|
|
* If anything is added that can interrupt us here without using IST,
|
|
|
|
* it must be *extremely* careful to limit its stack usage. This
|
|
|
|
* could include kprobes and a hypothetical future IST-less #DB
|
|
|
|
* handler.
|
2017-07-11 23:33:39 +08:00
|
|
|
*
|
|
|
|
* The OOPS unwinder relies on the word at the top of the IRQ
|
|
|
|
* stack linking back to the previous RSP for the entire time we're
|
|
|
|
* on the IRQ stack. For this to work reliably, we need to write
|
|
|
|
* it before we actually move ourselves to the IRQ stack.
|
|
|
|
*/
|
|
|
|
|
|
|
|
movq \old_rsp, PER_CPU_VAR(irq_stack_union + IRQ_STACK_SIZE - 8)
|
|
|
|
movq PER_CPU_VAR(irq_stack_ptr), %rsp
|
|
|
|
|
|
|
|
#ifdef CONFIG_DEBUG_ENTRY
|
|
|
|
/*
|
|
|
|
* If the first movq above becomes wrong due to IRQ stack layout
|
|
|
|
* changes, the only way we'll notice is if we try to unwind right
|
|
|
|
* here. Assert that we set up the stack right to catch this type
|
|
|
|
* of bug quickly.
|
2017-07-11 23:33:38 +08:00
|
|
|
*/
|
2017-07-11 23:33:39 +08:00
|
|
|
cmpq -8(%rsp), \old_rsp
|
|
|
|
je .Lirq_stack_okay\@
|
|
|
|
ud2
|
|
|
|
.Lirq_stack_okay\@:
|
|
|
|
#endif
|
2017-07-11 23:33:38 +08:00
|
|
|
|
2017-07-11 23:33:39 +08:00
|
|
|
.Lirq_stack_push_old_rsp_\@:
|
2017-07-11 23:33:38 +08:00
|
|
|
pushq \old_rsp
|
2017-07-11 23:33:44 +08:00
|
|
|
|
|
|
|
.if \regs
|
|
|
|
UNWIND_HINT_REGS indirect=1
|
|
|
|
.endif
|
2018-02-21 05:01:09 +08:00
|
|
|
|
|
|
|
.if \save_ret
|
|
|
|
/*
|
|
|
|
* Push the return address to the stack. This return address can
|
|
|
|
* be found at the "real" original RSP, which was offset by 8 at
|
|
|
|
* the beginning of this macro.
|
|
|
|
*/
|
|
|
|
pushq -8(\old_rsp)
|
|
|
|
.endif
|
2017-07-11 23:33:38 +08:00
|
|
|
.endm
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Undoes ENTER_IRQ_STACK.
|
|
|
|
*/
|
2017-07-11 23:33:44 +08:00
|
|
|
.macro LEAVE_IRQ_STACK regs=1
|
2017-07-11 23:33:38 +08:00
|
|
|
DEBUG_ENTRY_ASSERT_IRQS_OFF
|
|
|
|
/* We need to be off the IRQ stack before decrementing irq_count. */
|
|
|
|
popq %rsp
|
|
|
|
|
2017-07-11 23:33:44 +08:00
|
|
|
.if \regs
|
|
|
|
UNWIND_HINT_REGS
|
|
|
|
.endif
|
|
|
|
|
2017-07-11 23:33:38 +08:00
|
|
|
/*
|
|
|
|
* As in ENTER_IRQ_STACK, irq_count == 0, we are still claiming
|
|
|
|
* the irq stack but we're not on it.
|
|
|
|
*/
|
|
|
|
|
|
|
|
decl PER_CPU_VAR(irq_count)
|
|
|
|
.endm
|
|
|
|
|
x86: move entry_64.S register saving out of the macros
Here is a combined patch that moves "save_args" out-of-line for
the interrupt macro and moves "error_entry" mostly out-of-line
for the zeroentry and errorentry macros.
The save_args function becomes really straightforward and easy
to understand, with the possible exception of the stack switch
code, which now needs to copy the return address of to the
calling function. Normal interrupts arrive with ((~vector)-0x80)
on the stack, which gets adjusted in common_interrupt:
<common_interrupt>:
(5) addq $0xffffffffffffff80,(%rsp) /* -> ~(vector) */
(4) sub $0x50,%rsp /* space for registers */
(5) callq ffffffff80211290 <save_args>
(5) callq ffffffff80214290 <do_IRQ>
<ret_from_intr>:
...
An apic interrupt stub now look like this:
<thermal_interrupt>:
(5) pushq $0xffffffffffffff05 /* ~(vector) */
(4) sub $0x50,%rsp /* space for registers */
(5) callq ffffffff80211290 <save_args>
(5) callq ffffffff80212b8f <smp_thermal_interrupt>
(5) jmpq ffffffff80211f93 <ret_from_intr>
Similarly the exception handler register saving function becomes
simpler, without the need of any parameter shuffling. The stub
for an exception without errorcode looks like this:
<overflow>:
(6) callq *0x1cad12(%rip) # ffffffff803dd448 <pv_irq_ops+0x38>
(2) pushq $0xffffffffffffffff /* no syscall */
(4) sub $0x78,%rsp /* space for registers */
(5) callq ffffffff8030e3b0 <error_entry>
(3) mov %rsp,%rdi /* pt_regs pointer */
(2) xor %esi,%esi /* no error code */
(5) callq ffffffff80213446 <do_overflow>
(5) jmpq ffffffff8030e460 <error_exit>
And one for an exception with errorcode like this:
<segment_not_present>:
(6) callq *0x1cab92(%rip) # ffffffff803dd448 <pv_irq_ops+0x38>
(4) sub $0x78,%rsp /* space for registers */
(5) callq ffffffff8030e3b0 <error_entry>
(3) mov %rsp,%rdi /* pt_regs pointer */
(5) mov 0x78(%rsp),%rsi /* load error code */
(9) movq $0xffffffffffffffff,0x78(%rsp) /* no syscall */
(5) callq ffffffff80213209 <do_segment_not_present>
(5) jmpq ffffffff8030e460 <error_exit>
Unfortunately, this last type is more than 32 bytes. But the total space
savings due to this patch is about 2500 bytes on an smp-configuration,
and I think the code is clearer than it was before. The tested kernels
were non-paravirt ones (i.e., without the indirect call at the top of
the exception handlers).
Anyhow, I tested this patch on top of a recent -tip. The machine
was an 2x4-core Xeon at 2333MHz. Measured where the delays between
(almost-)adjacent rdtsc instructions. The graphs show how much
time is spent outside of the program as a function of the measured
delay. The area under the graph represents the total time spent
outside the program. Eight instances of the rdtsctest were
started, each pinned to a single cpu. The histogams are added.
For each kernel two measurements were done: one in mostly idle
condition, the other while running "bonnie++ -f", bound to cpu 0.
Each measurement took 40 minutes runtime. See the attached graphs
for the results. The graphs overlap almost everywhere, but there
are small differences.
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-11-19 08:18:11 +08:00
|
|
|
/*
|
2018-02-21 05:01:13 +08:00
|
|
|
* Interrupt entry helper function.
|
x86: move entry_64.S register saving out of the macros
Here is a combined patch that moves "save_args" out-of-line for
the interrupt macro and moves "error_entry" mostly out-of-line
for the zeroentry and errorentry macros.
The save_args function becomes really straightforward and easy
to understand, with the possible exception of the stack switch
code, which now needs to copy the return address of to the
calling function. Normal interrupts arrive with ((~vector)-0x80)
on the stack, which gets adjusted in common_interrupt:
<common_interrupt>:
(5) addq $0xffffffffffffff80,(%rsp) /* -> ~(vector) */
(4) sub $0x50,%rsp /* space for registers */
(5) callq ffffffff80211290 <save_args>
(5) callq ffffffff80214290 <do_IRQ>
<ret_from_intr>:
...
An apic interrupt stub now look like this:
<thermal_interrupt>:
(5) pushq $0xffffffffffffff05 /* ~(vector) */
(4) sub $0x50,%rsp /* space for registers */
(5) callq ffffffff80211290 <save_args>
(5) callq ffffffff80212b8f <smp_thermal_interrupt>
(5) jmpq ffffffff80211f93 <ret_from_intr>
Similarly the exception handler register saving function becomes
simpler, without the need of any parameter shuffling. The stub
for an exception without errorcode looks like this:
<overflow>:
(6) callq *0x1cad12(%rip) # ffffffff803dd448 <pv_irq_ops+0x38>
(2) pushq $0xffffffffffffffff /* no syscall */
(4) sub $0x78,%rsp /* space for registers */
(5) callq ffffffff8030e3b0 <error_entry>
(3) mov %rsp,%rdi /* pt_regs pointer */
(2) xor %esi,%esi /* no error code */
(5) callq ffffffff80213446 <do_overflow>
(5) jmpq ffffffff8030e460 <error_exit>
And one for an exception with errorcode like this:
<segment_not_present>:
(6) callq *0x1cab92(%rip) # ffffffff803dd448 <pv_irq_ops+0x38>
(4) sub $0x78,%rsp /* space for registers */
(5) callq ffffffff8030e3b0 <error_entry>
(3) mov %rsp,%rdi /* pt_regs pointer */
(5) mov 0x78(%rsp),%rsi /* load error code */
(9) movq $0xffffffffffffffff,0x78(%rsp) /* no syscall */
(5) callq ffffffff80213209 <do_segment_not_present>
(5) jmpq ffffffff8030e460 <error_exit>
Unfortunately, this last type is more than 32 bytes. But the total space
savings due to this patch is about 2500 bytes on an smp-configuration,
and I think the code is clearer than it was before. The tested kernels
were non-paravirt ones (i.e., without the indirect call at the top of
the exception handlers).
Anyhow, I tested this patch on top of a recent -tip. The machine
was an 2x4-core Xeon at 2333MHz. Measured where the delays between
(almost-)adjacent rdtsc instructions. The graphs show how much
time is spent outside of the program as a function of the measured
delay. The area under the graph represents the total time spent
outside the program. Eight instances of the rdtsctest were
started, each pinned to a single cpu. The histogams are added.
For each kernel two measurements were done: one in mostly idle
condition, the other while running "bonnie++ -f", bound to cpu 0.
Each measurement took 40 minutes runtime. See the attached graphs
for the results. The graphs overlap almost everywhere, but there
are small differences.
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-11-19 08:18:11 +08:00
|
|
|
*
|
2018-02-21 05:01:13 +08:00
|
|
|
* Entry runs with interrupts off. Stack layout at entry:
|
|
|
|
* +----------------------------------------------------+
|
|
|
|
* | regs->ss |
|
|
|
|
* | regs->rsp |
|
|
|
|
* | regs->eflags |
|
|
|
|
* | regs->cs |
|
|
|
|
* | regs->ip |
|
|
|
|
* +----------------------------------------------------+
|
|
|
|
* | regs->orig_ax = ~(interrupt number) |
|
|
|
|
* +----------------------------------------------------+
|
|
|
|
* | return address |
|
|
|
|
* +----------------------------------------------------+
|
x86: move entry_64.S register saving out of the macros
Here is a combined patch that moves "save_args" out-of-line for
the interrupt macro and moves "error_entry" mostly out-of-line
for the zeroentry and errorentry macros.
The save_args function becomes really straightforward and easy
to understand, with the possible exception of the stack switch
code, which now needs to copy the return address of to the
calling function. Normal interrupts arrive with ((~vector)-0x80)
on the stack, which gets adjusted in common_interrupt:
<common_interrupt>:
(5) addq $0xffffffffffffff80,(%rsp) /* -> ~(vector) */
(4) sub $0x50,%rsp /* space for registers */
(5) callq ffffffff80211290 <save_args>
(5) callq ffffffff80214290 <do_IRQ>
<ret_from_intr>:
...
An apic interrupt stub now look like this:
<thermal_interrupt>:
(5) pushq $0xffffffffffffff05 /* ~(vector) */
(4) sub $0x50,%rsp /* space for registers */
(5) callq ffffffff80211290 <save_args>
(5) callq ffffffff80212b8f <smp_thermal_interrupt>
(5) jmpq ffffffff80211f93 <ret_from_intr>
Similarly the exception handler register saving function becomes
simpler, without the need of any parameter shuffling. The stub
for an exception without errorcode looks like this:
<overflow>:
(6) callq *0x1cad12(%rip) # ffffffff803dd448 <pv_irq_ops+0x38>
(2) pushq $0xffffffffffffffff /* no syscall */
(4) sub $0x78,%rsp /* space for registers */
(5) callq ffffffff8030e3b0 <error_entry>
(3) mov %rsp,%rdi /* pt_regs pointer */
(2) xor %esi,%esi /* no error code */
(5) callq ffffffff80213446 <do_overflow>
(5) jmpq ffffffff8030e460 <error_exit>
And one for an exception with errorcode like this:
<segment_not_present>:
(6) callq *0x1cab92(%rip) # ffffffff803dd448 <pv_irq_ops+0x38>
(4) sub $0x78,%rsp /* space for registers */
(5) callq ffffffff8030e3b0 <error_entry>
(3) mov %rsp,%rdi /* pt_regs pointer */
(5) mov 0x78(%rsp),%rsi /* load error code */
(9) movq $0xffffffffffffffff,0x78(%rsp) /* no syscall */
(5) callq ffffffff80213209 <do_segment_not_present>
(5) jmpq ffffffff8030e460 <error_exit>
Unfortunately, this last type is more than 32 bytes. But the total space
savings due to this patch is about 2500 bytes on an smp-configuration,
and I think the code is clearer than it was before. The tested kernels
were non-paravirt ones (i.e., without the indirect call at the top of
the exception handlers).
Anyhow, I tested this patch on top of a recent -tip. The machine
was an 2x4-core Xeon at 2333MHz. Measured where the delays between
(almost-)adjacent rdtsc instructions. The graphs show how much
time is spent outside of the program as a function of the measured
delay. The area under the graph represents the total time spent
outside the program. Eight instances of the rdtsctest were
started, each pinned to a single cpu. The histogams are added.
For each kernel two measurements were done: one in mostly idle
condition, the other while running "bonnie++ -f", bound to cpu 0.
Each measurement took 40 minutes runtime. See the attached graphs
for the results. The graphs overlap almost everywhere, but there
are small differences.
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-11-19 08:18:11 +08:00
|
|
|
*/
|
2018-02-21 05:01:13 +08:00
|
|
|
ENTRY(interrupt_entry)
|
|
|
|
UNWIND_HINT_FUNC
|
|
|
|
ASM_CLAC
|
2015-01-09 00:25:15 +08:00
|
|
|
cld
|
2017-12-04 22:07:23 +08:00
|
|
|
|
2018-02-21 05:01:13 +08:00
|
|
|
testb $3, CS-ORIG_RAX+8(%rsp)
|
2017-12-04 22:07:23 +08:00
|
|
|
jz 1f
|
|
|
|
SWAPGS
|
2018-02-21 05:01:13 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Switch to the thread stack. The IRET frame and orig_ax are
|
|
|
|
* on the stack, as well as the return address. RDI..R12 are
|
|
|
|
* not (yet) on the stack and space has not (yet) been
|
|
|
|
* allocated for them.
|
|
|
|
*/
|
2018-02-21 05:01:10 +08:00
|
|
|
pushq %rdi
|
2018-02-21 05:01:13 +08:00
|
|
|
|
2018-02-21 05:01:10 +08:00
|
|
|
/* Need to switch before accessing the thread stack. */
|
|
|
|
SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi
|
|
|
|
movq %rsp, %rdi
|
|
|
|
movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp
|
2018-02-21 05:01:13 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We have RDI, return address, and orig_ax on the stack on
|
|
|
|
* top of the IRET frame. That means offset=24
|
|
|
|
*/
|
|
|
|
UNWIND_HINT_IRET_REGS base=%rdi offset=24
|
2018-02-21 05:01:10 +08:00
|
|
|
|
|
|
|
pushq 7*8(%rdi) /* regs->ss */
|
|
|
|
pushq 6*8(%rdi) /* regs->rsp */
|
|
|
|
pushq 5*8(%rdi) /* regs->eflags */
|
|
|
|
pushq 4*8(%rdi) /* regs->cs */
|
|
|
|
pushq 3*8(%rdi) /* regs->ip */
|
|
|
|
pushq 2*8(%rdi) /* regs->orig_ax */
|
|
|
|
pushq 8(%rdi) /* return address */
|
|
|
|
UNWIND_HINT_FUNC
|
|
|
|
|
|
|
|
movq (%rdi), %rdi
|
2017-12-04 22:07:23 +08:00
|
|
|
1:
|
|
|
|
|
2018-02-21 05:01:08 +08:00
|
|
|
PUSH_AND_CLEAR_REGS save_ret=1
|
|
|
|
ENCODE_FRAME_POINTER 8
|
x86/asm/entry/64: Always allocate a complete "struct pt_regs" on the kernel stack
The 64-bit entry code was using six stack slots less by not
saving/restoring registers which are callee-preserved according
to the C ABI, and was not allocating space for them.
Only when syscalls needed a complete "struct pt_regs" was
the complete area allocated and filled in.
As an additional twist, on interrupt entry a "slightly less
truncated pt_regs" trick is used, to make nested interrupt
stacks easier to unwind.
This proved to be a source of significant obfuscation and subtle
bugs. For example, 'stub_fork' had to pop the return address,
extend the struct, save registers, and push return address back.
Ugly. 'ia32_ptregs_common' pops return address and "returns" via
jmp insn, throwing a wrench into CPU return stack cache.
This patch changes the code to always allocate a complete
"struct pt_regs" on the kernel stack. The saving of registers
is still done lazily.
"Partial pt_regs" trick on interrupt stack is retained.
Macros which manipulate "struct pt_regs" on stack are reworked:
- ALLOC_PT_GPREGS_ON_STACK allocates the structure.
- SAVE_C_REGS saves to it those registers which are clobbered
by C code.
- SAVE_EXTRA_REGS saves to it all other registers.
- Corresponding RESTORE_* and REMOVE_PT_GPREGS_FROM_STACK macros
reverse it.
'ia32_ptregs_common', 'stub_fork' and friends lost their ugly dance
with the return pointer.
LOAD_ARGS32 in ia32entry.S now uses symbolic stack offsets
instead of magic numbers.
'error_entry' and 'save_paranoid' now use SAVE_C_REGS +
SAVE_EXTRA_REGS instead of having it open-coded yet again.
Patch was run-tested: 64-bit executables, 32-bit executables,
strace works.
Timing tests did not show measurable difference in 32-bit
and 64-bit syscalls.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1423778052-21038-2-git-send-email-dvlasenk@redhat.com
Link: http://lkml.kernel.org/r/b89763d354aa23e670b9bdf3a40ae320320a7c2e.1424989793.git.luto@amacapital.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-02-27 06:40:27 +08:00
|
|
|
|
2018-02-21 05:01:09 +08:00
|
|
|
testb $3, CS+8(%rsp)
|
2015-04-27 21:21:51 +08:00
|
|
|
jz 1f
|
2015-07-04 03:44:31 +08:00
|
|
|
|
|
|
|
/*
|
2017-12-04 22:07:23 +08:00
|
|
|
* IRQ from user mode.
|
|
|
|
*
|
2015-11-13 04:59:00 +08:00
|
|
|
* We need to tell lockdep that IRQs are off. We can't do this until
|
|
|
|
* we fix gsbase, and we should do it before enter_from_user_mode
|
2018-02-21 05:01:13 +08:00
|
|
|
* (which can take locks). Since TRACE_IRQS_OFF is idempotent,
|
2015-11-13 04:59:00 +08:00
|
|
|
* the simplest way to handle it is to just call it twice if
|
|
|
|
* we enter from user mode. There's no reason to optimize this since
|
|
|
|
* TRACE_IRQS_OFF is a no-op if lockdep is off.
|
|
|
|
*/
|
|
|
|
TRACE_IRQS_OFF
|
|
|
|
|
2015-11-13 04:59:04 +08:00
|
|
|
CALL_enter_from_user_mode
|
2015-07-04 03:44:31 +08:00
|
|
|
|
x86/asm/entry/64: Always allocate a complete "struct pt_regs" on the kernel stack
The 64-bit entry code was using six stack slots less by not
saving/restoring registers which are callee-preserved according
to the C ABI, and was not allocating space for them.
Only when syscalls needed a complete "struct pt_regs" was
the complete area allocated and filled in.
As an additional twist, on interrupt entry a "slightly less
truncated pt_regs" trick is used, to make nested interrupt
stacks easier to unwind.
This proved to be a source of significant obfuscation and subtle
bugs. For example, 'stub_fork' had to pop the return address,
extend the struct, save registers, and push return address back.
Ugly. 'ia32_ptregs_common' pops return address and "returns" via
jmp insn, throwing a wrench into CPU return stack cache.
This patch changes the code to always allocate a complete
"struct pt_regs" on the kernel stack. The saving of registers
is still done lazily.
"Partial pt_regs" trick on interrupt stack is retained.
Macros which manipulate "struct pt_regs" on stack are reworked:
- ALLOC_PT_GPREGS_ON_STACK allocates the structure.
- SAVE_C_REGS saves to it those registers which are clobbered
by C code.
- SAVE_EXTRA_REGS saves to it all other registers.
- Corresponding RESTORE_* and REMOVE_PT_GPREGS_FROM_STACK macros
reverse it.
'ia32_ptregs_common', 'stub_fork' and friends lost their ugly dance
with the return pointer.
LOAD_ARGS32 in ia32entry.S now uses symbolic stack offsets
instead of magic numbers.
'error_entry' and 'save_paranoid' now use SAVE_C_REGS +
SAVE_EXTRA_REGS instead of having it open-coded yet again.
Patch was run-tested: 64-bit executables, 32-bit executables,
strace works.
Timing tests did not show measurable difference in 32-bit
and 64-bit syscalls.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1423778052-21038-2-git-send-email-dvlasenk@redhat.com
Link: http://lkml.kernel.org/r/b89763d354aa23e670b9bdf3a40ae320320a7c2e.1424989793.git.luto@amacapital.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-02-27 06:40:27 +08:00
|
|
|
1:
|
2018-02-21 05:01:09 +08:00
|
|
|
ENTER_IRQ_STACK old_rsp=%rdi save_ret=1
|
2015-01-09 00:25:15 +08:00
|
|
|
/* We entered an interrupt context - irqs are off: */
|
|
|
|
TRACE_IRQS_OFF
|
|
|
|
|
2018-02-21 05:01:09 +08:00
|
|
|
ret
|
|
|
|
END(interrupt_entry)
|
|
|
|
|
2018-02-21 05:01:13 +08:00
|
|
|
|
|
|
|
/* Interrupt entry/exit. */
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-11-13 20:50:20 +08:00
|
|
|
/*
|
|
|
|
* The interrupt stubs push (~vector+0x80) onto the stack and
|
|
|
|
* then jump to common_interrupt.
|
|
|
|
*/
|
2008-11-12 05:51:52 +08:00
|
|
|
.p2align CONFIG_X86_L1_CACHE_SHIFT
|
|
|
|
common_interrupt:
|
2015-06-09 02:43:07 +08:00
|
|
|
addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */
|
2018-02-21 05:01:11 +08:00
|
|
|
call interrupt_entry
|
|
|
|
UNWIND_HINT_REGS indirect=1
|
|
|
|
call do_IRQ /* rdi points to pt_regs */
|
2015-03-23 21:03:59 +08:00
|
|
|
/* 0(%rsp): old RSP */
|
2005-09-13 00:49:24 +08:00
|
|
|
ret_from_intr:
|
2017-02-03 17:03:25 +08:00
|
|
|
DISABLE_INTERRUPTS(CLBR_ANY)
|
2006-07-03 15:24:45 +08:00
|
|
|
TRACE_IRQS_OFF
|
2011-01-06 22:22:47 +08:00
|
|
|
|
2017-07-11 23:33:38 +08:00
|
|
|
LEAVE_IRQ_STACK
|
2011-01-06 22:22:47 +08:00
|
|
|
|
x86/asm/entry/64: Clean up usage of TEST insns
By the nature of TEST operation, it is often possible
to test a narrower part of the operand:
"testl $3, mem" -> "testb $3, mem"
This results in shorter insns, because TEST insn has no
sign-entending byte-immediate forms unlike other ALU ops.
text data bss dec hex filename
11674 0 0 11674 2d9a entry_64.o.before
11658 0 0 11658 2d8a entry_64.o
Changes in object code:
- f7 84 24 88 00 00 00 03 00 00 00 testl $0x3,0x88(%rsp)
+ f6 84 24 88 00 00 00 03 testb $0x3,0x88(%rsp)
- f7 44 24 68 03 00 00 00 testl $0x3,0x68(%rsp)
+ f6 44 24 68 03 testb $0x3,0x68(%rsp)
- f7 84 24 90 00 00 00 03 00 00 00 testl $0x3,0x90(%rsp)
+ f6 84 24 90 00 00 00 03 testb $0x3,0x90(%rsp)
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1430140912-7960-2-git-send-email-dvlasenk@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-04-27 21:21:52 +08:00
|
|
|
testb $3, CS(%rsp)
|
2015-04-27 21:21:51 +08:00
|
|
|
jz retint_kernel
|
2015-06-09 02:43:07 +08:00
|
|
|
|
2015-07-04 03:44:31 +08:00
|
|
|
/* Interrupt came from user space */
|
|
|
|
GLOBAL(retint_user)
|
|
|
|
mov %rsp,%rdi
|
|
|
|
call prepare_exit_to_usermode
|
2006-07-03 15:24:45 +08:00
|
|
|
TRACE_IRQS_IRETQ
|
2017-11-02 15:58:59 +08:00
|
|
|
|
2017-11-02 15:59:00 +08:00
|
|
|
GLOBAL(swapgs_restore_regs_and_return_to_usermode)
|
2017-11-02 15:58:59 +08:00
|
|
|
#ifdef CONFIG_DEBUG_ENTRY
|
|
|
|
/* Assert that pt_regs indicates user mode. */
|
2017-11-02 20:09:26 +08:00
|
|
|
testb $3, CS(%rsp)
|
2017-11-02 15:58:59 +08:00
|
|
|
jnz 1f
|
|
|
|
ud2
|
|
|
|
1:
|
|
|
|
#endif
|
2018-02-11 18:49:43 +08:00
|
|
|
POP_REGS pop_rdi=0
|
2017-12-04 22:07:24 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS.
|
|
|
|
* Save old stack pointer and switch to trampoline stack.
|
|
|
|
*/
|
|
|
|
movq %rsp, %rdi
|
2017-12-04 22:07:29 +08:00
|
|
|
movq PER_CPU_VAR(cpu_tss_rw + TSS_sp0), %rsp
|
2017-12-04 22:07:24 +08:00
|
|
|
|
|
|
|
/* Copy the IRET frame to the trampoline stack. */
|
|
|
|
pushq 6*8(%rdi) /* SS */
|
|
|
|
pushq 5*8(%rdi) /* RSP */
|
|
|
|
pushq 4*8(%rdi) /* EFLAGS */
|
|
|
|
pushq 3*8(%rdi) /* CS */
|
|
|
|
pushq 2*8(%rdi) /* RIP */
|
|
|
|
|
|
|
|
/* Push user RDI on the trampoline stack. */
|
|
|
|
pushq (%rdi)
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We are on the trampoline stack. All regs except RDI are live.
|
|
|
|
* We can do future final exit work right here.
|
|
|
|
*/
|
|
|
|
|
2017-12-04 22:07:59 +08:00
|
|
|
SWITCH_TO_USER_CR3_STACK scratch_reg=%rdi
|
2017-12-04 22:07:35 +08:00
|
|
|
|
2017-12-04 22:07:24 +08:00
|
|
|
/* Restore RDI. */
|
|
|
|
popq %rdi
|
|
|
|
SWAPGS
|
2017-11-02 15:58:59 +08:00
|
|
|
INTERRUPT_RETURN
|
|
|
|
|
2006-07-03 15:24:45 +08:00
|
|
|
|
2015-03-31 02:09:31 +08:00
|
|
|
/* Returning to kernel space */
|
2015-04-01 01:00:05 +08:00
|
|
|
retint_kernel:
|
2015-03-31 02:09:31 +08:00
|
|
|
#ifdef CONFIG_PREEMPT
|
|
|
|
/* Interrupts are off */
|
|
|
|
/* Check if we need preemption */
|
2018-07-02 18:47:57 +08:00
|
|
|
btl $9, EFLAGS(%rsp) /* were interrupts off? */
|
2015-04-01 01:00:05 +08:00
|
|
|
jnc 1f
|
2015-06-09 02:43:07 +08:00
|
|
|
0: cmpl $0, PER_CPU_VAR(__preempt_count)
|
2015-04-01 01:00:07 +08:00
|
|
|
jnz 1f
|
2015-03-31 02:09:31 +08:00
|
|
|
call preempt_schedule_irq
|
2015-04-01 01:00:07 +08:00
|
|
|
jmp 0b
|
2015-04-01 01:00:05 +08:00
|
|
|
1:
|
2015-03-31 02:09:31 +08:00
|
|
|
#endif
|
2006-07-03 15:24:45 +08:00
|
|
|
/*
|
|
|
|
* The iretq could re-enable interrupts:
|
|
|
|
*/
|
|
|
|
TRACE_IRQS_IRETQ
|
2015-04-03 00:46:59 +08:00
|
|
|
|
2017-11-02 15:58:59 +08:00
|
|
|
GLOBAL(restore_regs_and_return_to_kernel)
|
|
|
|
#ifdef CONFIG_DEBUG_ENTRY
|
|
|
|
/* Assert that pt_regs indicates kernel mode. */
|
2017-11-02 20:09:26 +08:00
|
|
|
testb $3, CS(%rsp)
|
2017-11-02 15:58:59 +08:00
|
|
|
jz 1f
|
|
|
|
ud2
|
|
|
|
1:
|
|
|
|
#endif
|
2018-02-11 18:49:43 +08:00
|
|
|
POP_REGS
|
2017-11-02 15:59:01 +08:00
|
|
|
addq $8, %rsp /* skip regs->orig_ax */
|
membarrier/x86: Provide core serializing command
There are two places where core serialization is needed by membarrier:
1) When returning from the membarrier IPI,
2) After scheduler updates curr to a thread with a different mm, before
going back to user-space, since the curr->mm is used by membarrier to
check whether it needs to send an IPI to that CPU.
x86-32 uses IRET as return from interrupt, and both IRET and SYSEXIT to go
back to user-space. The IRET instruction is core serializing, but not
SYSEXIT.
x86-64 uses IRET as return from interrupt, which takes care of the IPI.
However, it can return to user-space through either SYSRETL (compat
code), SYSRETQ, or IRET. Given that SYSRET{L,Q} is not core serializing,
we rely instead on write_cr3() performed by switch_mm() to provide core
serialization after changing the current mm, and deal with the special
case of kthread -> uthread (temporarily keeping current mm into
active_mm) by adding a sync_core() in that specific case.
Use the new sync_core_before_usermode() to guarantee this.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrea Parri <parri.andrea@gmail.com>
Cc: Andrew Hunter <ahh@google.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Avi Kivity <avi@scylladb.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Dave Watson <davejwatson@fb.com>
Cc: David Sehr <sehr@google.com>
Cc: Greg Hackmann <ghackmann@google.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Maged Michael <maged.michael@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Will Deacon <will.deacon@arm.com>
Cc: linux-api@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Link: http://lkml.kernel.org/r/20180129202020.8515-10-mathieu.desnoyers@efficios.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 04:20:18 +08:00
|
|
|
/*
|
|
|
|
* ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization
|
|
|
|
* when returning from IPI handler.
|
|
|
|
*/
|
2014-07-23 23:34:11 +08:00
|
|
|
INTERRUPT_RETURN
|
|
|
|
|
|
|
|
ENTRY(native_iret)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
x86-64, espfix: Don't leak bits 31:16 of %esp returning to 16-bit stack
The IRET instruction, when returning to a 16-bit segment, only
restores the bottom 16 bits of the user space stack pointer. This
causes some 16-bit software to break, but it also leaks kernel state
to user space. We have a software workaround for that ("espfix") for
the 32-bit kernel, but it relies on a nonzero stack segment base which
is not available in 64-bit mode.
In checkin:
b3b42ac2cbae x86-64, modify_ldt: Ban 16-bit segments on 64-bit kernels
we "solved" this by forbidding 16-bit segments on 64-bit kernels, with
the logic that 16-bit support is crippled on 64-bit kernels anyway (no
V86 support), but it turns out that people are doing stuff like
running old Win16 binaries under Wine and expect it to work.
This works around this by creating percpu "ministacks", each of which
is mapped 2^16 times 64K apart. When we detect that the return SS is
on the LDT, we copy the IRET frame to the ministack and use the
relevant alias to return to userspace. The ministacks are mapped
readonly, so if IRET faults we promote #GP to #DF which is an IST
vector and thus has its own stack; we then do the fixup in the #DF
handler.
(Making #GP an IST exception would make the msr_safe functions unsafe
in NMI/MC context, and quite possibly have other effects.)
Special thanks to:
- Andy Lutomirski, for the suggestion of using very small stack slots
and copy (as opposed to map) the IRET frame there, and for the
suggestion to mark them readonly and let the fault promote to #DF.
- Konrad Wilk for paravirt fixup and testing.
- Borislav Petkov for testing help and useful comments.
Reported-by: Brian Gerst <brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Link: http://lkml.kernel.org/r/1398816946-3351-1-git-send-email-hpa@linux.intel.com
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Andrew Lutomriski <amluto@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Dirk Hohndel <dirk@hohndel.org>
Cc: Arjan van de Ven <arjan.van.de.ven@intel.com>
Cc: comex <comexk@gmail.com>
Cc: Alexander van Heukelum <heukelum@fastmail.fm>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: <stable@vger.kernel.org> # consider after upstream merge
2014-04-30 07:46:09 +08:00
|
|
|
/*
|
|
|
|
* Are we returning to a stack segment from the LDT? Note: in
|
|
|
|
* 64-bit mode SS:RSP on the exception stack is always valid.
|
|
|
|
*/
|
2014-05-05 01:36:22 +08:00
|
|
|
#ifdef CONFIG_X86_ESPFIX64
|
2015-06-09 02:43:07 +08:00
|
|
|
testb $4, (SS-RIP)(%rsp)
|
|
|
|
jnz native_irq_return_ldt
|
2014-05-05 01:36:22 +08:00
|
|
|
#endif
|
x86-64, espfix: Don't leak bits 31:16 of %esp returning to 16-bit stack
The IRET instruction, when returning to a 16-bit segment, only
restores the bottom 16 bits of the user space stack pointer. This
causes some 16-bit software to break, but it also leaks kernel state
to user space. We have a software workaround for that ("espfix") for
the 32-bit kernel, but it relies on a nonzero stack segment base which
is not available in 64-bit mode.
In checkin:
b3b42ac2cbae x86-64, modify_ldt: Ban 16-bit segments on 64-bit kernels
we "solved" this by forbidding 16-bit segments on 64-bit kernels, with
the logic that 16-bit support is crippled on 64-bit kernels anyway (no
V86 support), but it turns out that people are doing stuff like
running old Win16 binaries under Wine and expect it to work.
This works around this by creating percpu "ministacks", each of which
is mapped 2^16 times 64K apart. When we detect that the return SS is
on the LDT, we copy the IRET frame to the ministack and use the
relevant alias to return to userspace. The ministacks are mapped
readonly, so if IRET faults we promote #GP to #DF which is an IST
vector and thus has its own stack; we then do the fixup in the #DF
handler.
(Making #GP an IST exception would make the msr_safe functions unsafe
in NMI/MC context, and quite possibly have other effects.)
Special thanks to:
- Andy Lutomirski, for the suggestion of using very small stack slots
and copy (as opposed to map) the IRET frame there, and for the
suggestion to mark them readonly and let the fault promote to #DF.
- Konrad Wilk for paravirt fixup and testing.
- Borislav Petkov for testing help and useful comments.
Reported-by: Brian Gerst <brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Link: http://lkml.kernel.org/r/1398816946-3351-1-git-send-email-hpa@linux.intel.com
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Andrew Lutomriski <amluto@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Dirk Hohndel <dirk@hohndel.org>
Cc: Arjan van de Ven <arjan.van.de.ven@intel.com>
Cc: comex <comexk@gmail.com>
Cc: Alexander van Heukelum <heukelum@fastmail.fm>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: <stable@vger.kernel.org> # consider after upstream merge
2014-04-30 07:46:09 +08:00
|
|
|
|
2014-11-23 10:00:31 +08:00
|
|
|
.global native_irq_return_iret
|
2014-07-23 23:34:11 +08:00
|
|
|
native_irq_return_iret:
|
x86_64, traps: Rework bad_iret
It's possible for iretq to userspace to fail. This can happen because
of a bad CS, SS, or RIP.
Historically, we've handled it by fixing up an exception from iretq to
land at bad_iret, which pretends that the failed iret frame was really
the hardware part of #GP(0) from userspace. To make this work, there's
an extra fixup to fudge the gs base into a usable state.
This is suboptimal because it loses the original exception. It's also
buggy because there's no guarantee that we were on the kernel stack to
begin with. For example, if the failing iret happened on return from an
NMI, then we'll end up executing general_protection on the NMI stack.
This is bad for several reasons, the most immediate of which is that
general_protection, as a non-paranoid idtentry, will try to deliver
signals and/or schedule from the wrong stack.
This patch throws out bad_iret entirely. As a replacement, it augments
the existing swapgs fudge into a full-blown iret fixup, mostly written
in C. It's should be clearer and more correct.
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-11-23 10:00:33 +08:00
|
|
|
/*
|
|
|
|
* This may fault. Non-paranoid faults on return to userspace are
|
|
|
|
* handled by fixup_bad_iret. These include #SS, #GP, and #NP.
|
|
|
|
* Double-faults due to espfix64 are handled in do_double_fault.
|
|
|
|
* Other faults here are fatal.
|
|
|
|
*/
|
2005-04-17 06:20:36 +08:00
|
|
|
iretq
|
2008-02-10 06:24:08 +08:00
|
|
|
|
2014-05-05 01:36:22 +08:00
|
|
|
#ifdef CONFIG_X86_ESPFIX64
|
2014-07-23 23:34:11 +08:00
|
|
|
native_irq_return_ldt:
|
2016-09-13 06:05:51 +08:00
|
|
|
/*
|
|
|
|
* We are running with user GSBASE. All GPRs contain their user
|
|
|
|
* values. We have a percpu ESPFIX stack that is eight slots
|
|
|
|
* long (see ESPFIX_STACK_SIZE). espfix_waddr points to the bottom
|
|
|
|
* of the ESPFIX stack.
|
|
|
|
*
|
|
|
|
* We clobber RAX and RDI in this code. We stash RDI on the
|
|
|
|
* normal stack and RAX on the ESPFIX stack.
|
|
|
|
*
|
|
|
|
* The ESPFIX stack layout we set up looks like this:
|
|
|
|
*
|
|
|
|
* --- top of ESPFIX stack ---
|
|
|
|
* SS
|
|
|
|
* RSP
|
|
|
|
* RFLAGS
|
|
|
|
* CS
|
|
|
|
* RIP <-- RSP points here when we're done
|
|
|
|
* RAX <-- espfix_waddr points here
|
|
|
|
* --- bottom of ESPFIX stack ---
|
|
|
|
*/
|
|
|
|
|
|
|
|
pushq %rdi /* Stash user RDI */
|
2017-12-04 22:07:35 +08:00
|
|
|
SWAPGS /* to kernel GS */
|
|
|
|
SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi /* to kernel CR3 */
|
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
movq PER_CPU_VAR(espfix_waddr), %rdi
|
2016-09-13 06:05:51 +08:00
|
|
|
movq %rax, (0*8)(%rdi) /* user RAX */
|
|
|
|
movq (1*8)(%rsp), %rax /* user RIP */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rax, (1*8)(%rdi)
|
2016-09-13 06:05:51 +08:00
|
|
|
movq (2*8)(%rsp), %rax /* user CS */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rax, (2*8)(%rdi)
|
2016-09-13 06:05:51 +08:00
|
|
|
movq (3*8)(%rsp), %rax /* user RFLAGS */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rax, (3*8)(%rdi)
|
2016-09-13 06:05:51 +08:00
|
|
|
movq (5*8)(%rsp), %rax /* user SS */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rax, (5*8)(%rdi)
|
2016-09-13 06:05:51 +08:00
|
|
|
movq (4*8)(%rsp), %rax /* user RSP */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rax, (4*8)(%rdi)
|
2016-09-13 06:05:51 +08:00
|
|
|
/* Now RAX == RSP. */
|
|
|
|
|
|
|
|
andl $0xffff0000, %eax /* RAX = (RSP & 0xffff0000) */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* espfix_stack[31:16] == 0. The page tables are set up such that
|
|
|
|
* (espfix_stack | (X & 0xffff0000)) points to a read-only alias of
|
|
|
|
* espfix_waddr for any X. That is, there are 65536 RO aliases of
|
|
|
|
* the same page. Set up RSP so that RSP[31:16] contains the
|
|
|
|
* respective 16 bits of the /userspace/ RSP and RSP nonetheless
|
|
|
|
* still points to an RO alias of the ESPFIX stack.
|
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
orq PER_CPU_VAR(espfix_stack), %rax
|
2017-12-04 22:07:35 +08:00
|
|
|
|
2017-12-04 22:07:59 +08:00
|
|
|
SWITCH_TO_USER_CR3_STACK scratch_reg=%rdi
|
2017-12-04 22:07:35 +08:00
|
|
|
SWAPGS /* to user GS */
|
|
|
|
popq %rdi /* Restore user RDI */
|
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rax, %rsp
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS offset=8
|
2016-09-13 06:05:51 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* At this point, we cannot write to the stack any more, but we can
|
|
|
|
* still read.
|
|
|
|
*/
|
|
|
|
popq %rax /* Restore user RAX */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* RSP now points to an ordinary IRET frame, except that the page
|
|
|
|
* is read-only and RSP[31:16] are preloaded with the userspace
|
|
|
|
* values. We can now IRET back to userspace.
|
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp native_irq_return_iret
|
2014-05-05 01:36:22 +08:00
|
|
|
#endif
|
2006-06-26 19:56:55 +08:00
|
|
|
END(common_interrupt)
|
x86-64, espfix: Don't leak bits 31:16 of %esp returning to 16-bit stack
The IRET instruction, when returning to a 16-bit segment, only
restores the bottom 16 bits of the user space stack pointer. This
causes some 16-bit software to break, but it also leaks kernel state
to user space. We have a software workaround for that ("espfix") for
the 32-bit kernel, but it relies on a nonzero stack segment base which
is not available in 64-bit mode.
In checkin:
b3b42ac2cbae x86-64, modify_ldt: Ban 16-bit segments on 64-bit kernels
we "solved" this by forbidding 16-bit segments on 64-bit kernels, with
the logic that 16-bit support is crippled on 64-bit kernels anyway (no
V86 support), but it turns out that people are doing stuff like
running old Win16 binaries under Wine and expect it to work.
This works around this by creating percpu "ministacks", each of which
is mapped 2^16 times 64K apart. When we detect that the return SS is
on the LDT, we copy the IRET frame to the ministack and use the
relevant alias to return to userspace. The ministacks are mapped
readonly, so if IRET faults we promote #GP to #DF which is an IST
vector and thus has its own stack; we then do the fixup in the #DF
handler.
(Making #GP an IST exception would make the msr_safe functions unsafe
in NMI/MC context, and quite possibly have other effects.)
Special thanks to:
- Andy Lutomirski, for the suggestion of using very small stack slots
and copy (as opposed to map) the IRET frame there, and for the
suggestion to mark them readonly and let the fault promote to #DF.
- Konrad Wilk for paravirt fixup and testing.
- Borislav Petkov for testing help and useful comments.
Reported-by: Brian Gerst <brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Link: http://lkml.kernel.org/r/1398816946-3351-1-git-send-email-hpa@linux.intel.com
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Andrew Lutomriski <amluto@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Dirk Hohndel <dirk@hohndel.org>
Cc: Arjan van de Ven <arjan.van.de.ven@intel.com>
Cc: comex <comexk@gmail.com>
Cc: Alexander van Heukelum <heukelum@fastmail.fm>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: <stable@vger.kernel.org> # consider after upstream merge
2014-04-30 07:46:09 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* APIC interrupts.
|
2008-11-16 22:29:00 +08:00
|
|
|
*/
|
x86, trace: Add irq vector tracepoints
[Purpose of this patch]
As Vaibhav explained in the thread below, tracepoints for irq vectors
are useful.
http://www.spinics.net/lists/mm-commits/msg85707.html
<snip>
The current interrupt traces from irq_handler_entry and irq_handler_exit
provide when an interrupt is handled. They provide good data about when
the system has switched to kernel space and how it affects the currently
running processes.
There are some IRQ vectors which trigger the system into kernel space,
which are not handled in generic IRQ handlers. Tracing such events gives
us the information about IRQ interaction with other system events.
The trace also tells where the system is spending its time. We want to
know which cores are handling interrupts and how they are affecting other
processes in the system. Also, the trace provides information about when
the cores are idle and which interrupts are changing that state.
<snip>
On the other hand, my usecase is tracing just local timer event and
getting a value of instruction pointer.
I suggested to add an argument local timer event to get instruction pointer before.
But there is another way to get it with external module like systemtap.
So, I don't need to add any argument to irq vector tracepoints now.
[Patch Description]
Vaibhav's patch shared a trace point ,irq_vector_entry/irq_vector_exit, in all events.
But there is an above use case to trace specific irq_vector rather than tracing all events.
In this case, we are concerned about overhead due to unwanted events.
So, add following tracepoints instead of introducing irq_vector_entry/exit.
so that we can enable them independently.
- local_timer_vector
- reschedule_vector
- call_function_vector
- call_function_single_vector
- irq_work_entry_vector
- error_apic_vector
- thermal_apic_vector
- threshold_apic_vector
- spurious_apic_vector
- x86_platform_ipi_vector
Also, introduce a logic switching IDT at enabling/disabling time so that a time penalty
makes a zero when tracepoints are disabled. Detailed explanations are as follows.
- Create trace irq handlers with entering_irq()/exiting_irq().
- Create a new IDT, trace_idt_table, at boot time by adding a logic to
_set_gate(). It is just a copy of original idt table.
- Register the new handlers for tracpoints to the new IDT by introducing
macros to alloc_intr_gate() called at registering time of irq_vector handlers.
- Add checking, whether irq vector tracing is on/off, into load_current_idt().
This has to be done below debug checking for these reasons.
- Switching to debug IDT may be kicked while tracing is enabled.
- On the other hands, switching to trace IDT is kicked only when debugging
is disabled.
In addition, the new IDT is created only when CONFIG_TRACING is enabled to avoid being
used for other purposes.
Signed-off-by: Seiji Aguchi <seiji.aguchi@hds.com>
Link: http://lkml.kernel.org/r/51C323ED.5050708@hds.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
2013-06-20 23:46:53 +08:00
|
|
|
.macro apicinterrupt3 num sym do_sym
|
2008-11-23 17:08:28 +08:00
|
|
|
ENTRY(\sym)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq $~(\num)
|
2011-11-29 19:03:46 +08:00
|
|
|
.Lcommon_\sym:
|
2018-02-21 05:01:11 +08:00
|
|
|
call interrupt_entry
|
|
|
|
UNWIND_HINT_REGS indirect=1
|
|
|
|
call \do_sym /* rdi points to pt_regs */
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp ret_from_intr
|
2008-11-23 17:08:28 +08:00
|
|
|
END(\sym)
|
|
|
|
.endm
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2016-07-15 17:42:43 +08:00
|
|
|
/* Make sure APIC interrupt handlers end up in the irqentry section: */
|
2017-08-03 10:38:21 +08:00
|
|
|
#define PUSH_SECTION_IRQENTRY .pushsection .irqentry.text, "ax"
|
|
|
|
#define POP_SECTION_IRQENTRY .popsection
|
2016-07-15 17:42:43 +08:00
|
|
|
|
x86, trace: Add irq vector tracepoints
[Purpose of this patch]
As Vaibhav explained in the thread below, tracepoints for irq vectors
are useful.
http://www.spinics.net/lists/mm-commits/msg85707.html
<snip>
The current interrupt traces from irq_handler_entry and irq_handler_exit
provide when an interrupt is handled. They provide good data about when
the system has switched to kernel space and how it affects the currently
running processes.
There are some IRQ vectors which trigger the system into kernel space,
which are not handled in generic IRQ handlers. Tracing such events gives
us the information about IRQ interaction with other system events.
The trace also tells where the system is spending its time. We want to
know which cores are handling interrupts and how they are affecting other
processes in the system. Also, the trace provides information about when
the cores are idle and which interrupts are changing that state.
<snip>
On the other hand, my usecase is tracing just local timer event and
getting a value of instruction pointer.
I suggested to add an argument local timer event to get instruction pointer before.
But there is another way to get it with external module like systemtap.
So, I don't need to add any argument to irq vector tracepoints now.
[Patch Description]
Vaibhav's patch shared a trace point ,irq_vector_entry/irq_vector_exit, in all events.
But there is an above use case to trace specific irq_vector rather than tracing all events.
In this case, we are concerned about overhead due to unwanted events.
So, add following tracepoints instead of introducing irq_vector_entry/exit.
so that we can enable them independently.
- local_timer_vector
- reschedule_vector
- call_function_vector
- call_function_single_vector
- irq_work_entry_vector
- error_apic_vector
- thermal_apic_vector
- threshold_apic_vector
- spurious_apic_vector
- x86_platform_ipi_vector
Also, introduce a logic switching IDT at enabling/disabling time so that a time penalty
makes a zero when tracepoints are disabled. Detailed explanations are as follows.
- Create trace irq handlers with entering_irq()/exiting_irq().
- Create a new IDT, trace_idt_table, at boot time by adding a logic to
_set_gate(). It is just a copy of original idt table.
- Register the new handlers for tracpoints to the new IDT by introducing
macros to alloc_intr_gate() called at registering time of irq_vector handlers.
- Add checking, whether irq vector tracing is on/off, into load_current_idt().
This has to be done below debug checking for these reasons.
- Switching to debug IDT may be kicked while tracing is enabled.
- On the other hands, switching to trace IDT is kicked only when debugging
is disabled.
In addition, the new IDT is created only when CONFIG_TRACING is enabled to avoid being
used for other purposes.
Signed-off-by: Seiji Aguchi <seiji.aguchi@hds.com>
Link: http://lkml.kernel.org/r/51C323ED.5050708@hds.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
2013-06-20 23:46:53 +08:00
|
|
|
.macro apicinterrupt num sym do_sym
|
2016-07-15 17:42:43 +08:00
|
|
|
PUSH_SECTION_IRQENTRY
|
x86, trace: Add irq vector tracepoints
[Purpose of this patch]
As Vaibhav explained in the thread below, tracepoints for irq vectors
are useful.
http://www.spinics.net/lists/mm-commits/msg85707.html
<snip>
The current interrupt traces from irq_handler_entry and irq_handler_exit
provide when an interrupt is handled. They provide good data about when
the system has switched to kernel space and how it affects the currently
running processes.
There are some IRQ vectors which trigger the system into kernel space,
which are not handled in generic IRQ handlers. Tracing such events gives
us the information about IRQ interaction with other system events.
The trace also tells where the system is spending its time. We want to
know which cores are handling interrupts and how they are affecting other
processes in the system. Also, the trace provides information about when
the cores are idle and which interrupts are changing that state.
<snip>
On the other hand, my usecase is tracing just local timer event and
getting a value of instruction pointer.
I suggested to add an argument local timer event to get instruction pointer before.
But there is another way to get it with external module like systemtap.
So, I don't need to add any argument to irq vector tracepoints now.
[Patch Description]
Vaibhav's patch shared a trace point ,irq_vector_entry/irq_vector_exit, in all events.
But there is an above use case to trace specific irq_vector rather than tracing all events.
In this case, we are concerned about overhead due to unwanted events.
So, add following tracepoints instead of introducing irq_vector_entry/exit.
so that we can enable them independently.
- local_timer_vector
- reschedule_vector
- call_function_vector
- call_function_single_vector
- irq_work_entry_vector
- error_apic_vector
- thermal_apic_vector
- threshold_apic_vector
- spurious_apic_vector
- x86_platform_ipi_vector
Also, introduce a logic switching IDT at enabling/disabling time so that a time penalty
makes a zero when tracepoints are disabled. Detailed explanations are as follows.
- Create trace irq handlers with entering_irq()/exiting_irq().
- Create a new IDT, trace_idt_table, at boot time by adding a logic to
_set_gate(). It is just a copy of original idt table.
- Register the new handlers for tracpoints to the new IDT by introducing
macros to alloc_intr_gate() called at registering time of irq_vector handlers.
- Add checking, whether irq vector tracing is on/off, into load_current_idt().
This has to be done below debug checking for these reasons.
- Switching to debug IDT may be kicked while tracing is enabled.
- On the other hands, switching to trace IDT is kicked only when debugging
is disabled.
In addition, the new IDT is created only when CONFIG_TRACING is enabled to avoid being
used for other purposes.
Signed-off-by: Seiji Aguchi <seiji.aguchi@hds.com>
Link: http://lkml.kernel.org/r/51C323ED.5050708@hds.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
2013-06-20 23:46:53 +08:00
|
|
|
apicinterrupt3 \num \sym \do_sym
|
2016-07-15 17:42:43 +08:00
|
|
|
POP_SECTION_IRQENTRY
|
x86, trace: Add irq vector tracepoints
[Purpose of this patch]
As Vaibhav explained in the thread below, tracepoints for irq vectors
are useful.
http://www.spinics.net/lists/mm-commits/msg85707.html
<snip>
The current interrupt traces from irq_handler_entry and irq_handler_exit
provide when an interrupt is handled. They provide good data about when
the system has switched to kernel space and how it affects the currently
running processes.
There are some IRQ vectors which trigger the system into kernel space,
which are not handled in generic IRQ handlers. Tracing such events gives
us the information about IRQ interaction with other system events.
The trace also tells where the system is spending its time. We want to
know which cores are handling interrupts and how they are affecting other
processes in the system. Also, the trace provides information about when
the cores are idle and which interrupts are changing that state.
<snip>
On the other hand, my usecase is tracing just local timer event and
getting a value of instruction pointer.
I suggested to add an argument local timer event to get instruction pointer before.
But there is another way to get it with external module like systemtap.
So, I don't need to add any argument to irq vector tracepoints now.
[Patch Description]
Vaibhav's patch shared a trace point ,irq_vector_entry/irq_vector_exit, in all events.
But there is an above use case to trace specific irq_vector rather than tracing all events.
In this case, we are concerned about overhead due to unwanted events.
So, add following tracepoints instead of introducing irq_vector_entry/exit.
so that we can enable them independently.
- local_timer_vector
- reschedule_vector
- call_function_vector
- call_function_single_vector
- irq_work_entry_vector
- error_apic_vector
- thermal_apic_vector
- threshold_apic_vector
- spurious_apic_vector
- x86_platform_ipi_vector
Also, introduce a logic switching IDT at enabling/disabling time so that a time penalty
makes a zero when tracepoints are disabled. Detailed explanations are as follows.
- Create trace irq handlers with entering_irq()/exiting_irq().
- Create a new IDT, trace_idt_table, at boot time by adding a logic to
_set_gate(). It is just a copy of original idt table.
- Register the new handlers for tracpoints to the new IDT by introducing
macros to alloc_intr_gate() called at registering time of irq_vector handlers.
- Add checking, whether irq vector tracing is on/off, into load_current_idt().
This has to be done below debug checking for these reasons.
- Switching to debug IDT may be kicked while tracing is enabled.
- On the other hands, switching to trace IDT is kicked only when debugging
is disabled.
In addition, the new IDT is created only when CONFIG_TRACING is enabled to avoid being
used for other purposes.
Signed-off-by: Seiji Aguchi <seiji.aguchi@hds.com>
Link: http://lkml.kernel.org/r/51C323ED.5050708@hds.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
2013-06-20 23:46:53 +08:00
|
|
|
.endm
|
|
|
|
|
2008-11-23 17:08:28 +08:00
|
|
|
#ifdef CONFIG_SMP
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt3 IRQ_MOVE_CLEANUP_VECTOR irq_move_cleanup_interrupt smp_irq_move_cleanup_interrupt
|
|
|
|
apicinterrupt3 REBOOT_VECTOR reboot_interrupt smp_reboot_interrupt
|
2008-11-23 17:08:28 +08:00
|
|
|
#endif
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2009-01-20 11:36:04 +08:00
|
|
|
#ifdef CONFIG_X86_UV
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt
|
2009-01-20 11:36:04 +08:00
|
|
|
#endif
|
2015-06-09 02:43:07 +08:00
|
|
|
|
|
|
|
apicinterrupt LOCAL_TIMER_VECTOR apic_timer_interrupt smp_apic_timer_interrupt
|
|
|
|
apicinterrupt X86_PLATFORM_IPI_VECTOR x86_platform_ipi smp_x86_platform_ipi
|
2005-11-06 00:25:53 +08:00
|
|
|
|
2013-04-11 19:25:11 +08:00
|
|
|
#ifdef CONFIG_HAVE_KVM
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi
|
|
|
|
apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_posted_intr_wakeup_ipi
|
2017-04-28 13:13:58 +08:00
|
|
|
apicinterrupt3 POSTED_INTR_NESTED_VECTOR kvm_posted_intr_nested_ipi smp_kvm_posted_intr_nested_ipi
|
2013-04-11 19:25:11 +08:00
|
|
|
#endif
|
|
|
|
|
2013-06-22 19:33:30 +08:00
|
|
|
#ifdef CONFIG_X86_MCE_THRESHOLD
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt THRESHOLD_APIC_VECTOR threshold_interrupt smp_threshold_interrupt
|
2013-06-22 19:33:30 +08:00
|
|
|
#endif
|
|
|
|
|
2015-05-06 19:58:56 +08:00
|
|
|
#ifdef CONFIG_X86_MCE_AMD
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt DEFERRED_ERROR_VECTOR deferred_error_interrupt smp_deferred_error_interrupt
|
2015-05-06 19:58:56 +08:00
|
|
|
#endif
|
|
|
|
|
2013-06-22 19:33:30 +08:00
|
|
|
#ifdef CONFIG_X86_THERMAL_VECTOR
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt THERMAL_APIC_VECTOR thermal_interrupt smp_thermal_interrupt
|
2013-06-22 19:33:30 +08:00
|
|
|
#endif
|
2008-06-02 21:56:14 +08:00
|
|
|
|
2008-11-23 17:08:28 +08:00
|
|
|
#ifdef CONFIG_SMP
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt CALL_FUNCTION_SINGLE_VECTOR call_function_single_interrupt smp_call_function_single_interrupt
|
|
|
|
apicinterrupt CALL_FUNCTION_VECTOR call_function_interrupt smp_call_function_interrupt
|
|
|
|
apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt
|
2008-11-23 17:08:28 +08:00
|
|
|
#endif
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt ERROR_APIC_VECTOR error_interrupt smp_error_interrupt
|
|
|
|
apicinterrupt SPURIOUS_APIC_VECTOR spurious_interrupt smp_spurious_interrupt
|
2008-11-16 22:29:00 +08:00
|
|
|
|
2010-10-14 14:01:34 +08:00
|
|
|
#ifdef CONFIG_IRQ_WORK
|
2015-06-09 02:43:07 +08:00
|
|
|
apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt
|
2008-12-03 17:39:53 +08:00
|
|
|
#endif
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* Exception entry points.
|
2008-11-16 22:29:00 +08:00
|
|
|
*/
|
2017-12-04 22:07:29 +08:00
|
|
|
#define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8)
|
2014-05-22 06:07:09 +08:00
|
|
|
|
|
|
|
.macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1
|
2008-11-23 17:08:28 +08:00
|
|
|
ENTRY(\sym)
|
2017-10-21 00:21:33 +08:00
|
|
|
UNWIND_HINT_IRET_REGS offset=\has_error_code*8
|
2017-07-11 23:33:44 +08:00
|
|
|
|
2014-05-22 06:07:09 +08:00
|
|
|
/* Sanity check */
|
|
|
|
.if \shift_ist != -1 && \paranoid == 0
|
|
|
|
.error "using shift_ist requires paranoid=1"
|
|
|
|
.endif
|
|
|
|
|
2012-11-02 19:18:39 +08:00
|
|
|
ASM_CLAC
|
2014-05-22 06:07:08 +08:00
|
|
|
|
2017-10-21 00:21:35 +08:00
|
|
|
.if \has_error_code == 0
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq $-1 /* ORIG_RAX: no syscall to restart */
|
2014-05-22 06:07:08 +08:00
|
|
|
.endif
|
|
|
|
|
2018-04-04 06:39:26 +08:00
|
|
|
.if \paranoid == 1
|
2018-02-15 01:59:23 +08:00
|
|
|
testb $3, CS-ORIG_RAX(%rsp) /* If coming from userspace, switch stacks */
|
2017-12-04 22:07:23 +08:00
|
|
|
jnz .Lfrom_usermode_switch_stack_\@
|
2014-11-12 04:49:41 +08:00
|
|
|
.endif
|
2017-12-04 22:07:23 +08:00
|
|
|
|
|
|
|
.if \paranoid
|
2015-06-09 02:43:07 +08:00
|
|
|
call paranoid_entry
|
2014-05-22 06:07:08 +08:00
|
|
|
.else
|
2015-06-09 02:43:07 +08:00
|
|
|
call error_entry
|
2014-05-22 06:07:08 +08:00
|
|
|
.endif
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_REGS
|
2015-02-27 06:40:34 +08:00
|
|
|
/* returned flag: ebx=0: need swapgs on exit, ebx=1: don't need it */
|
2014-05-22 06:07:08 +08:00
|
|
|
|
|
|
|
.if \paranoid
|
2014-05-22 06:07:09 +08:00
|
|
|
.if \shift_ist != -1
|
2015-06-09 02:43:07 +08:00
|
|
|
TRACE_IRQS_OFF_DEBUG /* reload IDT in case of recursion */
|
2014-05-22 06:07:09 +08:00
|
|
|
.else
|
2008-11-21 23:44:28 +08:00
|
|
|
TRACE_IRQS_OFF
|
2014-05-22 06:07:08 +08:00
|
|
|
.endif
|
2014-05-22 06:07:09 +08:00
|
|
|
.endif
|
2014-05-22 06:07:08 +08:00
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rsp, %rdi /* pt_regs pointer */
|
2014-05-22 06:07:08 +08:00
|
|
|
|
|
|
|
.if \has_error_code
|
2015-06-09 02:43:07 +08:00
|
|
|
movq ORIG_RAX(%rsp), %rsi /* get error code */
|
|
|
|
movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
|
2014-05-22 06:07:08 +08:00
|
|
|
.else
|
2015-06-09 02:43:07 +08:00
|
|
|
xorl %esi, %esi /* no error code */
|
2014-05-22 06:07:08 +08:00
|
|
|
.endif
|
|
|
|
|
2014-05-22 06:07:09 +08:00
|
|
|
.if \shift_ist != -1
|
2015-06-09 02:43:07 +08:00
|
|
|
subq $EXCEPTION_STKSZ, CPU_TSS_IST(\shift_ist)
|
2014-05-22 06:07:09 +08:00
|
|
|
.endif
|
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
call \do_sym
|
2014-05-22 06:07:08 +08:00
|
|
|
|
2014-05-22 06:07:09 +08:00
|
|
|
.if \shift_ist != -1
|
2015-06-09 02:43:07 +08:00
|
|
|
addq $EXCEPTION_STKSZ, CPU_TSS_IST(\shift_ist)
|
2014-05-22 06:07:09 +08:00
|
|
|
.endif
|
|
|
|
|
2015-02-27 06:40:34 +08:00
|
|
|
/* these procedures expect "no swapgs" flag in ebx */
|
2014-05-22 06:07:08 +08:00
|
|
|
.if \paranoid
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp paranoid_exit
|
2014-05-22 06:07:08 +08:00
|
|
|
.else
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp error_exit
|
2014-05-22 06:07:08 +08:00
|
|
|
.endif
|
|
|
|
|
2018-04-04 06:39:26 +08:00
|
|
|
.if \paranoid == 1
|
2014-11-12 04:49:41 +08:00
|
|
|
/*
|
2017-12-04 22:07:23 +08:00
|
|
|
* Entry from userspace. Switch stacks and treat it
|
2014-11-12 04:49:41 +08:00
|
|
|
* as a normal entry. This means that paranoid handlers
|
|
|
|
* run in real process context if user_mode(regs).
|
|
|
|
*/
|
2017-12-04 22:07:23 +08:00
|
|
|
.Lfrom_usermode_switch_stack_\@:
|
2015-06-09 02:43:07 +08:00
|
|
|
call error_entry
|
2014-11-12 04:49:41 +08:00
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rsp, %rdi /* pt_regs pointer */
|
2014-11-12 04:49:41 +08:00
|
|
|
|
|
|
|
.if \has_error_code
|
2015-06-09 02:43:07 +08:00
|
|
|
movq ORIG_RAX(%rsp), %rsi /* get error code */
|
|
|
|
movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
|
2014-11-12 04:49:41 +08:00
|
|
|
.else
|
2015-06-09 02:43:07 +08:00
|
|
|
xorl %esi, %esi /* no error code */
|
2014-11-12 04:49:41 +08:00
|
|
|
.endif
|
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
call \do_sym
|
2014-11-12 04:49:41 +08:00
|
|
|
|
x86/entry/64: Remove %ebx handling from error_entry/exit
error_entry and error_exit communicate the user vs. kernel status of
the frame using %ebx. This is unnecessary -- the information is in
regs->cs. Just use regs->cs.
This makes error_entry simpler and makes error_exit more robust.
It also fixes a nasty bug. Before all the Spectre nonsense, the
xen_failsafe_callback entry point returned like this:
ALLOC_PT_GPREGS_ON_STACK
SAVE_C_REGS
SAVE_EXTRA_REGS
ENCODE_FRAME_POINTER
jmp error_exit
And it did not go through error_entry. This was bogus: RBX
contained garbage, and error_exit expected a flag in RBX.
Fortunately, it generally contained *nonzero* garbage, so the
correct code path was used. As part of the Spectre fixes, code was
added to clear RBX to mitigate certain speculation attacks. Now,
depending on kernel configuration, RBX got zeroed and, when running
some Wine workloads, the kernel crashes. This was introduced by:
commit 3ac6d8c787b8 ("x86/entry/64: Clear registers for exceptions/interrupts, to reduce speculation attack surface")
With this patch applied, RBX is no longer needed as a flag, and the
problem goes away.
I suspect that malicious userspace could use this bug to crash the
kernel even without the offending patch applied, though.
[ Historical note: I wrote this patch as a cleanup before I was aware
of the bug it fixed. ]
[ Note to stable maintainers: this should probably get applied to all
kernels. If you're nervous about that, a more conservative fix to
add xorl %ebx,%ebx; incl %ebx before the jump to error_exit should
also fix the problem. ]
Reported-and-tested-by: M. Vefa Bicakci <m.v.b@runbox.com>
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Cc: xen-devel@lists.xenproject.org
Fixes: 3ac6d8c787b8 ("x86/entry/64: Clear registers for exceptions/interrupts, to reduce speculation attack surface")
Link: http://lkml.kernel.org/r/b5010a090d3586b2d6e06c7ad3ec5542d1241c45.1532282627.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-07-23 02:05:09 +08:00
|
|
|
jmp error_exit
|
2014-11-12 04:49:41 +08:00
|
|
|
.endif
|
2008-11-24 20:24:28 +08:00
|
|
|
END(\sym)
|
2008-11-23 17:08:28 +08:00
|
|
|
.endm
|
2008-11-21 23:44:28 +08:00
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
idtentry divide_error do_divide_error has_error_code=0
|
|
|
|
idtentry overflow do_overflow has_error_code=0
|
|
|
|
idtentry bounds do_bounds has_error_code=0
|
|
|
|
idtentry invalid_op do_invalid_op has_error_code=0
|
|
|
|
idtentry device_not_available do_device_not_available has_error_code=0
|
|
|
|
idtentry double_fault do_double_fault has_error_code=1 paranoid=2
|
|
|
|
idtentry coprocessor_segment_overrun do_coprocessor_segment_overrun has_error_code=0
|
|
|
|
idtentry invalid_TSS do_invalid_TSS has_error_code=1
|
|
|
|
idtentry segment_not_present do_segment_not_present has_error_code=1
|
|
|
|
idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0
|
|
|
|
idtentry coprocessor_error do_coprocessor_error has_error_code=0
|
|
|
|
idtentry alignment_check do_alignment_check has_error_code=1
|
|
|
|
idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reload gs selector with exception handling
|
|
|
|
* edi: new selector
|
|
|
|
*/
|
2008-06-25 12:19:32 +08:00
|
|
|
ENTRY(native_load_gs_index)
|
2017-07-11 23:33:44 +08:00
|
|
|
FRAME_BEGIN
|
x86/debug: Remove perpetually broken, unmaintainable dwarf annotations
So the dwarf2 annotations in low level assembly code have
become an increasing hindrance: unreadable, messy macros
mixed into some of the most security sensitive code paths
of the Linux kernel.
These debug info annotations don't even buy the upstream
kernel anything: dwarf driven stack unwinding has caused
problems in the past so it's out of tree, and the upstream
kernel only uses the much more robust framepointers based
stack unwinding method.
In addition to that there's a steady, slow bitrot going
on with these annotations, requiring frequent fixups.
There's no tooling and no functionality upstream that
keeps it correct.
So burn down the sick forest, allowing new, healthier growth:
27 files changed, 350 insertions(+), 1101 deletions(-)
Someone who has the willingness and time to do this
properly can attempt to reintroduce dwarf debuginfo in x86
assembly code plus dwarf unwinding from first principles,
with the following conditions:
- it should be maximally readable, and maximally low-key to
'ordinary' code reading and maintenance.
- find a build time method to insert dwarf annotations
automatically in the most common cases, for pop/push
instructions that manipulate the stack pointer. This could
be done for example via a preprocessing step that just
looks for common patterns - plus special annotations for
the few cases where we want to depart from the default.
We have hundreds of CFI annotations, so automating most of
that makes sense.
- it should come with build tooling checks that ensure that
CFI annotations are sensible. We've seen such efforts from
the framepointer side, and there's no reason it couldn't be
done on the dwarf side.
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Jan Beulich <JBeulich@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-05-28 18:21:47 +08:00
|
|
|
pushfq
|
2009-01-29 06:35:03 +08:00
|
|
|
DISABLE_INTERRUPTS(CLBR_ANY & ~CLBR_RDI)
|
2017-11-23 12:39:16 +08:00
|
|
|
TRACE_IRQS_OFF
|
2008-11-28 02:10:08 +08:00
|
|
|
SWAPGS
|
2016-04-08 08:31:50 +08:00
|
|
|
.Lgs_change:
|
2015-06-09 02:43:07 +08:00
|
|
|
movl %edi, %gs
|
2016-04-08 08:31:49 +08:00
|
|
|
2: ALTERNATIVE "", "mfence", X86_BUG_SWAPGS_FENCE
|
2008-01-30 20:32:08 +08:00
|
|
|
SWAPGS
|
2017-11-23 12:39:16 +08:00
|
|
|
TRACE_IRQS_FLAGS (%rsp)
|
x86/debug: Remove perpetually broken, unmaintainable dwarf annotations
So the dwarf2 annotations in low level assembly code have
become an increasing hindrance: unreadable, messy macros
mixed into some of the most security sensitive code paths
of the Linux kernel.
These debug info annotations don't even buy the upstream
kernel anything: dwarf driven stack unwinding has caused
problems in the past so it's out of tree, and the upstream
kernel only uses the much more robust framepointers based
stack unwinding method.
In addition to that there's a steady, slow bitrot going
on with these annotations, requiring frequent fixups.
There's no tooling and no functionality upstream that
keeps it correct.
So burn down the sick forest, allowing new, healthier growth:
27 files changed, 350 insertions(+), 1101 deletions(-)
Someone who has the willingness and time to do this
properly can attempt to reintroduce dwarf debuginfo in x86
assembly code plus dwarf unwinding from first principles,
with the following conditions:
- it should be maximally readable, and maximally low-key to
'ordinary' code reading and maintenance.
- find a build time method to insert dwarf annotations
automatically in the most common cases, for pop/push
instructions that manipulate the stack pointer. This could
be done for example via a preprocessing step that just
looks for common patterns - plus special annotations for
the few cases where we want to depart from the default.
We have hundreds of CFI annotations, so automating most of
that makes sense.
- it should come with build tooling checks that ensure that
CFI annotations are sensible. We've seen such efforts from
the framepointer side, and there's no reason it couldn't be
done on the dwarf side.
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Jan Beulich <JBeulich@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-05-28 18:21:47 +08:00
|
|
|
popfq
|
2017-07-11 23:33:44 +08:00
|
|
|
FRAME_END
|
2008-11-28 02:10:08 +08:00
|
|
|
ret
|
2017-07-11 23:33:44 +08:00
|
|
|
ENDPROC(native_load_gs_index)
|
2016-01-12 00:04:34 +08:00
|
|
|
EXPORT_SYMBOL(native_load_gs_index)
|
2008-11-16 22:29:00 +08:00
|
|
|
|
2016-04-08 08:31:50 +08:00
|
|
|
_ASM_EXTABLE(.Lgs_change, bad_gs)
|
2015-06-09 02:43:07 +08:00
|
|
|
.section .fixup, "ax"
|
2005-04-17 06:20:36 +08:00
|
|
|
/* running with kernelgs */
|
2008-11-16 22:29:00 +08:00
|
|
|
bad_gs:
|
2015-06-09 02:43:07 +08:00
|
|
|
SWAPGS /* switch back to user gs */
|
2016-04-27 03:23:27 +08:00
|
|
|
.macro ZAP_GS
|
|
|
|
/* This can't be a string because the preprocessor needs to see it. */
|
|
|
|
movl $__USER_DS, %eax
|
|
|
|
movl %eax, %gs
|
|
|
|
.endm
|
|
|
|
ALTERNATIVE "", "ZAP_GS", X86_BUG_NULL_SEG
|
2015-06-09 02:43:07 +08:00
|
|
|
xorl %eax, %eax
|
|
|
|
movl %eax, %gs
|
|
|
|
jmp 2b
|
2008-11-28 02:10:08 +08:00
|
|
|
.previous
|
2008-11-16 22:29:00 +08:00
|
|
|
|
2006-08-03 04:37:28 +08:00
|
|
|
/* Call softirq on interrupt stack. Interrupts are off. */
|
2013-09-05 21:49:45 +08:00
|
|
|
ENTRY(do_softirq_own_stack)
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq %rbp
|
|
|
|
mov %rsp, %rbp
|
2017-07-11 23:33:44 +08:00
|
|
|
ENTER_IRQ_STACK regs=0 old_rsp=%r11
|
2015-06-09 02:43:07 +08:00
|
|
|
call __do_softirq
|
2017-07-11 23:33:44 +08:00
|
|
|
LEAVE_IRQ_STACK regs=0
|
2006-08-03 04:37:28 +08:00
|
|
|
leaveq
|
2005-07-29 12:15:49 +08:00
|
|
|
ret
|
2017-07-11 23:33:44 +08:00
|
|
|
ENDPROC(do_softirq_own_stack)
|
2007-06-23 08:29:25 +08:00
|
|
|
|
2008-07-09 06:06:49 +08:00
|
|
|
#ifdef CONFIG_XEN
|
2017-09-01 01:42:49 +08:00
|
|
|
idtentry hypervisor_callback xen_do_hypervisor_callback has_error_code=0
|
2008-07-09 06:06:49 +08:00
|
|
|
|
|
|
|
/*
|
2008-11-28 02:10:08 +08:00
|
|
|
* A note on the "critical region" in our callback handler.
|
|
|
|
* We want to avoid stacking callback handlers due to events occurring
|
|
|
|
* during handling of the last event. To do this, we keep events disabled
|
|
|
|
* until we've done all processing. HOWEVER, we must enable events before
|
|
|
|
* popping the stack frame (can't be done atomically) and so it would still
|
|
|
|
* be possible to get enough handler activations to overflow the stack.
|
|
|
|
* Although unlikely, bugs of that kind are hard to track down, so we'd
|
|
|
|
* like to avoid the possibility.
|
|
|
|
* So, on entry to the handler we detect whether we interrupted an
|
|
|
|
* existing activation in its critical region -- if so, we pop the current
|
|
|
|
* activation and restart the handler using the previous one.
|
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
ENTRY(xen_do_hypervisor_callback) /* do_hypervisor_callback(struct *pt_regs) */
|
|
|
|
|
2008-11-28 02:10:08 +08:00
|
|
|
/*
|
|
|
|
* Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
|
|
|
|
* see the correct pointer to the pt_regs
|
|
|
|
*/
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_FUNC
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rdi, %rsp /* we don't return, adjust the stack frame */
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_REGS
|
2017-07-11 23:33:38 +08:00
|
|
|
|
|
|
|
ENTER_IRQ_STACK old_rsp=%r10
|
2015-06-09 02:43:07 +08:00
|
|
|
call xen_evtchn_do_upcall
|
2017-07-11 23:33:38 +08:00
|
|
|
LEAVE_IRQ_STACK
|
|
|
|
|
2015-02-19 23:23:17 +08:00
|
|
|
#ifndef CONFIG_PREEMPT
|
2015-06-09 02:43:07 +08:00
|
|
|
call xen_maybe_preempt_hcall
|
2015-02-19 23:23:17 +08:00
|
|
|
#endif
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp error_exit
|
x86, binutils, xen: Fix another wrong size directive
The latest binutils (2.21.0.20110302/Ubuntu) breaks the build
yet another time, under CONFIG_XEN=y due to a .size directive that
refers to a slightly differently named (hence, to the now very
strict and unforgiving assembler, non-existent) symbol.
[ mingo:
This unnecessary build breakage caused by new binutils
version 2.21 gets escallated back several kernel releases spanning
several years of Linux history, affecting over 130,000 upstream
kernel commits (!), on CONFIG_XEN=y 64-bit kernels (i.e. essentially
affecting all major Linux distro kernel configs).
Git annotate tells us that this slight debug symbol code mismatch
bug has been introduced in 2008 in commit 3d75e1b8:
3d75e1b8 (Jeremy Fitzhardinge 2008-07-08 15:06:49 -0700 1231) ENTRY(xen_do_hypervisor_callback) # do_hypervisor_callback(struct *pt_regs)
The 'bug' is just a slight assymetry in ENTRY()/END()
debug-symbols sequences, with lots of assembly code between the
ENTRY() and the END():
ENTRY(xen_do_hypervisor_callback) # do_hypervisor_callback(struct *pt_regs)
...
END(do_hypervisor_callback)
Human reviewers almost never catch such small mismatches, and binutils
never even warned about it either.
This new binutils version thus breaks the Xen build on all upstream kernels
since v2.6.27, out of the blue.
This makes a straightforward Git bisection of all 64-bit Xen-enabled kernels
impossible on such binutils, for a bisection window of over hundred
thousand historic commits. (!)
This is a major fail on the side of binutils and binutils needs to turn
this show-stopper build failure into a warning ASAP. ]
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Cc: Jeremy Fitzhardinge <jeremy@goop.org>
Cc: Jan Beulich <jbeulich@novell.com>
Cc: H.J. Lu <hjl.tools@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Kees Cook <kees.cook@canonical.com>
LKML-Reference: <1299877178-26063-1-git-send-email-heukelum@fastmail.fm>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2011-03-12 04:59:38 +08:00
|
|
|
END(xen_do_hypervisor_callback)
|
2008-07-09 06:06:49 +08:00
|
|
|
|
|
|
|
/*
|
2008-11-28 02:10:08 +08:00
|
|
|
* Hypervisor uses this for application faults while it executes.
|
|
|
|
* We get here for two reasons:
|
|
|
|
* 1. Fault while reloading DS, ES, FS or GS
|
|
|
|
* 2. Fault while executing IRET
|
|
|
|
* Category 1 we do not need to fix up as Xen has already reloaded all segment
|
|
|
|
* registers that could be reloaded and zeroed the others.
|
|
|
|
* Category 2 we fix up by killing the current process. We cannot use the
|
|
|
|
* normal Linux return path in this case because if we use the IRET hypercall
|
|
|
|
* to pop the stack frame we end up in an infinite loop of failsafe callbacks.
|
|
|
|
* We distinguish between categories by comparing each saved segment register
|
|
|
|
* with its current contents: any discrepancy means we in category 1.
|
|
|
|
*/
|
2008-07-09 06:06:49 +08:00
|
|
|
ENTRY(xen_failsafe_callback)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_EMPTY
|
2015-06-09 02:43:07 +08:00
|
|
|
movl %ds, %ecx
|
|
|
|
cmpw %cx, 0x10(%rsp)
|
|
|
|
jne 1f
|
|
|
|
movl %es, %ecx
|
|
|
|
cmpw %cx, 0x18(%rsp)
|
|
|
|
jne 1f
|
|
|
|
movl %fs, %ecx
|
|
|
|
cmpw %cx, 0x20(%rsp)
|
|
|
|
jne 1f
|
|
|
|
movl %gs, %ecx
|
|
|
|
cmpw %cx, 0x28(%rsp)
|
|
|
|
jne 1f
|
2008-07-09 06:06:49 +08:00
|
|
|
/* All segments match their saved values => Category 2 (Bad IRET). */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq (%rsp), %rcx
|
|
|
|
movq 8(%rsp), %r11
|
|
|
|
addq $0x30, %rsp
|
|
|
|
pushq $0 /* RIP */
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS offset=8
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp general_protection
|
2008-07-09 06:06:49 +08:00
|
|
|
1: /* Segment mismatch => Category 1 (Bad segment). Retry the IRET. */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq (%rsp), %rcx
|
|
|
|
movq 8(%rsp), %r11
|
|
|
|
addq $0x30, %rsp
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq $-1 /* orig_ax = -1 => not a system call */
|
2018-02-11 18:49:45 +08:00
|
|
|
PUSH_AND_CLEAR_REGS
|
2016-10-21 00:34:40 +08:00
|
|
|
ENCODE_FRAME_POINTER
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp error_exit
|
2008-07-09 06:06:49 +08:00
|
|
|
END(xen_failsafe_callback)
|
|
|
|
|
x86, trace: Add irq vector tracepoints
[Purpose of this patch]
As Vaibhav explained in the thread below, tracepoints for irq vectors
are useful.
http://www.spinics.net/lists/mm-commits/msg85707.html
<snip>
The current interrupt traces from irq_handler_entry and irq_handler_exit
provide when an interrupt is handled. They provide good data about when
the system has switched to kernel space and how it affects the currently
running processes.
There are some IRQ vectors which trigger the system into kernel space,
which are not handled in generic IRQ handlers. Tracing such events gives
us the information about IRQ interaction with other system events.
The trace also tells where the system is spending its time. We want to
know which cores are handling interrupts and how they are affecting other
processes in the system. Also, the trace provides information about when
the cores are idle and which interrupts are changing that state.
<snip>
On the other hand, my usecase is tracing just local timer event and
getting a value of instruction pointer.
I suggested to add an argument local timer event to get instruction pointer before.
But there is another way to get it with external module like systemtap.
So, I don't need to add any argument to irq vector tracepoints now.
[Patch Description]
Vaibhav's patch shared a trace point ,irq_vector_entry/irq_vector_exit, in all events.
But there is an above use case to trace specific irq_vector rather than tracing all events.
In this case, we are concerned about overhead due to unwanted events.
So, add following tracepoints instead of introducing irq_vector_entry/exit.
so that we can enable them independently.
- local_timer_vector
- reschedule_vector
- call_function_vector
- call_function_single_vector
- irq_work_entry_vector
- error_apic_vector
- thermal_apic_vector
- threshold_apic_vector
- spurious_apic_vector
- x86_platform_ipi_vector
Also, introduce a logic switching IDT at enabling/disabling time so that a time penalty
makes a zero when tracepoints are disabled. Detailed explanations are as follows.
- Create trace irq handlers with entering_irq()/exiting_irq().
- Create a new IDT, trace_idt_table, at boot time by adding a logic to
_set_gate(). It is just a copy of original idt table.
- Register the new handlers for tracpoints to the new IDT by introducing
macros to alloc_intr_gate() called at registering time of irq_vector handlers.
- Add checking, whether irq vector tracing is on/off, into load_current_idt().
This has to be done below debug checking for these reasons.
- Switching to debug IDT may be kicked while tracing is enabled.
- On the other hands, switching to trace IDT is kicked only when debugging
is disabled.
In addition, the new IDT is created only when CONFIG_TRACING is enabled to avoid being
used for other purposes.
Signed-off-by: Seiji Aguchi <seiji.aguchi@hds.com>
Link: http://lkml.kernel.org/r/51C323ED.5050708@hds.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
2013-06-20 23:46:53 +08:00
|
|
|
apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
|
2010-05-14 19:40:51 +08:00
|
|
|
xen_hvm_callback_vector xen_evtchn_do_upcall
|
|
|
|
|
2008-07-09 06:06:49 +08:00
|
|
|
#endif /* CONFIG_XEN */
|
2008-11-24 20:24:28 +08:00
|
|
|
|
2013-02-04 09:22:39 +08:00
|
|
|
#if IS_ENABLED(CONFIG_HYPERV)
|
x86, trace: Add irq vector tracepoints
[Purpose of this patch]
As Vaibhav explained in the thread below, tracepoints for irq vectors
are useful.
http://www.spinics.net/lists/mm-commits/msg85707.html
<snip>
The current interrupt traces from irq_handler_entry and irq_handler_exit
provide when an interrupt is handled. They provide good data about when
the system has switched to kernel space and how it affects the currently
running processes.
There are some IRQ vectors which trigger the system into kernel space,
which are not handled in generic IRQ handlers. Tracing such events gives
us the information about IRQ interaction with other system events.
The trace also tells where the system is spending its time. We want to
know which cores are handling interrupts and how they are affecting other
processes in the system. Also, the trace provides information about when
the cores are idle and which interrupts are changing that state.
<snip>
On the other hand, my usecase is tracing just local timer event and
getting a value of instruction pointer.
I suggested to add an argument local timer event to get instruction pointer before.
But there is another way to get it with external module like systemtap.
So, I don't need to add any argument to irq vector tracepoints now.
[Patch Description]
Vaibhav's patch shared a trace point ,irq_vector_entry/irq_vector_exit, in all events.
But there is an above use case to trace specific irq_vector rather than tracing all events.
In this case, we are concerned about overhead due to unwanted events.
So, add following tracepoints instead of introducing irq_vector_entry/exit.
so that we can enable them independently.
- local_timer_vector
- reschedule_vector
- call_function_vector
- call_function_single_vector
- irq_work_entry_vector
- error_apic_vector
- thermal_apic_vector
- threshold_apic_vector
- spurious_apic_vector
- x86_platform_ipi_vector
Also, introduce a logic switching IDT at enabling/disabling time so that a time penalty
makes a zero when tracepoints are disabled. Detailed explanations are as follows.
- Create trace irq handlers with entering_irq()/exiting_irq().
- Create a new IDT, trace_idt_table, at boot time by adding a logic to
_set_gate(). It is just a copy of original idt table.
- Register the new handlers for tracpoints to the new IDT by introducing
macros to alloc_intr_gate() called at registering time of irq_vector handlers.
- Add checking, whether irq vector tracing is on/off, into load_current_idt().
This has to be done below debug checking for these reasons.
- Switching to debug IDT may be kicked while tracing is enabled.
- On the other hands, switching to trace IDT is kicked only when debugging
is disabled.
In addition, the new IDT is created only when CONFIG_TRACING is enabled to avoid being
used for other purposes.
Signed-off-by: Seiji Aguchi <seiji.aguchi@hds.com>
Link: http://lkml.kernel.org/r/51C323ED.5050708@hds.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
2013-06-20 23:46:53 +08:00
|
|
|
apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
|
2013-02-04 09:22:39 +08:00
|
|
|
hyperv_callback_vector hyperv_vector_handler
|
2018-01-24 21:23:33 +08:00
|
|
|
|
|
|
|
apicinterrupt3 HYPERV_REENLIGHTENMENT_VECTOR \
|
|
|
|
hyperv_reenlightenment_vector hyperv_reenlightenment_intr
|
2018-03-05 13:17:18 +08:00
|
|
|
|
|
|
|
apicinterrupt3 HYPERV_STIMER0_VECTOR \
|
|
|
|
hv_stimer0_callback_vector hv_stimer0_vector_handler
|
2013-02-04 09:22:39 +08:00
|
|
|
#endif /* CONFIG_HYPERV */
|
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
idtentry debug do_debug has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK
|
2015-07-24 06:37:48 +08:00
|
|
|
idtentry int3 do_int3 has_error_code=0
|
2015-06-09 02:43:07 +08:00
|
|
|
idtentry stack_segment do_stack_segment has_error_code=1
|
|
|
|
|
2009-03-30 10:56:29 +08:00
|
|
|
#ifdef CONFIG_XEN
|
2017-11-02 15:59:07 +08:00
|
|
|
idtentry xennmi do_nmi has_error_code=0
|
2017-09-01 01:42:49 +08:00
|
|
|
idtentry xendebug do_debug has_error_code=0
|
|
|
|
idtentry xenint3 do_int3 has_error_code=0
|
2009-03-30 10:56:29 +08:00
|
|
|
#endif
|
2015-06-09 02:43:07 +08:00
|
|
|
|
|
|
|
idtentry general_protection do_general_protection has_error_code=1
|
2017-08-28 14:47:22 +08:00
|
|
|
idtentry page_fault do_page_fault has_error_code=1
|
2015-06-09 02:43:07 +08:00
|
|
|
|
2010-10-14 17:22:52 +08:00
|
|
|
#ifdef CONFIG_KVM_GUEST
|
2015-06-09 02:43:07 +08:00
|
|
|
idtentry async_page_fault do_async_page_fault has_error_code=1
|
2010-10-14 17:22:52 +08:00
|
|
|
#endif
|
2015-06-09 02:43:07 +08:00
|
|
|
|
2008-11-24 20:24:28 +08:00
|
|
|
#ifdef CONFIG_X86_MCE
|
2018-01-18 23:28:26 +08:00
|
|
|
idtentry machine_check do_mce has_error_code=0 paranoid=1
|
2008-11-24 20:24:28 +08:00
|
|
|
#endif
|
|
|
|
|
2015-02-27 06:40:34 +08:00
|
|
|
/*
|
2018-02-15 01:59:23 +08:00
|
|
|
* Save all registers in pt_regs, and switch gs if needed.
|
2015-02-27 06:40:34 +08:00
|
|
|
* Use slow, but surefire "are we in kernel?" check.
|
|
|
|
* Return: ebx=0: need swapgs on exit, ebx=1: otherwise
|
|
|
|
*/
|
|
|
|
ENTRY(paranoid_entry)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_FUNC
|
2015-02-27 06:40:33 +08:00
|
|
|
cld
|
2018-02-15 01:59:23 +08:00
|
|
|
PUSH_AND_CLEAR_REGS save_ret=1
|
|
|
|
ENCODE_FRAME_POINTER 8
|
2015-06-09 02:43:07 +08:00
|
|
|
movl $1, %ebx
|
|
|
|
movl $MSR_GS_BASE, %ecx
|
2015-02-27 06:40:33 +08:00
|
|
|
rdmsr
|
2015-06-09 02:43:07 +08:00
|
|
|
testl %edx, %edx
|
|
|
|
js 1f /* negative -> in kernel */
|
2015-02-27 06:40:33 +08:00
|
|
|
SWAPGS
|
2015-06-09 02:43:07 +08:00
|
|
|
xorl %ebx, %ebx
|
2017-12-04 22:07:35 +08:00
|
|
|
|
|
|
|
1:
|
|
|
|
SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
|
|
|
|
|
|
|
|
ret
|
2015-02-27 06:40:34 +08:00
|
|
|
END(paranoid_entry)
|
2008-11-24 20:24:28 +08:00
|
|
|
|
2015-02-27 06:40:34 +08:00
|
|
|
/*
|
|
|
|
* "Paranoid" exit path from exception stack. This is invoked
|
|
|
|
* only on return from non-NMI IST interrupts that came
|
|
|
|
* from kernel space.
|
|
|
|
*
|
|
|
|
* We may be returning to very strange contexts (e.g. very early
|
|
|
|
* in syscall entry), so checking for preemption here would
|
|
|
|
* be complicated. Fortunately, we there's no good reason
|
|
|
|
* to try to handle preemption here.
|
2015-06-09 02:43:07 +08:00
|
|
|
*
|
|
|
|
* On entry, ebx is "no swapgs" flag (1: don't need swapgs, 0: need it)
|
2015-02-27 06:40:34 +08:00
|
|
|
*/
|
2008-11-24 20:24:28 +08:00
|
|
|
ENTRY(paranoid_exit)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_REGS
|
2017-02-03 17:03:25 +08:00
|
|
|
DISABLE_INTERRUPTS(CLBR_ANY)
|
2012-05-30 23:54:53 +08:00
|
|
|
TRACE_IRQS_OFF_DEBUG
|
2015-06-09 02:43:07 +08:00
|
|
|
testl %ebx, %ebx /* swapgs needed? */
|
2017-11-02 15:59:02 +08:00
|
|
|
jnz .Lparanoid_exit_no_swapgs
|
2015-02-27 06:40:30 +08:00
|
|
|
TRACE_IRQS_IRETQ
|
2017-12-04 22:08:00 +08:00
|
|
|
RESTORE_CR3 scratch_reg=%rbx save_reg=%r14
|
2008-11-24 20:24:28 +08:00
|
|
|
SWAPGS_UNSAFE_STACK
|
2017-11-02 15:59:02 +08:00
|
|
|
jmp .Lparanoid_exit_restore
|
|
|
|
.Lparanoid_exit_no_swapgs:
|
2015-02-27 06:40:30 +08:00
|
|
|
TRACE_IRQS_IRETQ_DEBUG
|
2018-02-14 15:39:11 +08:00
|
|
|
RESTORE_CR3 scratch_reg=%rbx save_reg=%r14
|
2017-11-02 15:59:02 +08:00
|
|
|
.Lparanoid_exit_restore:
|
|
|
|
jmp restore_regs_and_return_to_kernel
|
2008-11-24 20:24:28 +08:00
|
|
|
END(paranoid_exit)
|
|
|
|
|
|
|
|
/*
|
2018-02-15 01:59:23 +08:00
|
|
|
* Save all registers in pt_regs, and switch GS if needed.
|
2008-11-24 20:24:28 +08:00
|
|
|
*/
|
|
|
|
ENTRY(error_entry)
|
2018-02-15 01:59:23 +08:00
|
|
|
UNWIND_HINT_FUNC
|
2008-11-24 20:24:28 +08:00
|
|
|
cld
|
2018-02-15 01:59:23 +08:00
|
|
|
PUSH_AND_CLEAR_REGS save_ret=1
|
|
|
|
ENCODE_FRAME_POINTER 8
|
x86/asm/entry/64: Clean up usage of TEST insns
By the nature of TEST operation, it is often possible
to test a narrower part of the operand:
"testl $3, mem" -> "testb $3, mem"
This results in shorter insns, because TEST insn has no
sign-entending byte-immediate forms unlike other ALU ops.
text data bss dec hex filename
11674 0 0 11674 2d9a entry_64.o.before
11658 0 0 11658 2d8a entry_64.o
Changes in object code:
- f7 84 24 88 00 00 00 03 00 00 00 testl $0x3,0x88(%rsp)
+ f6 84 24 88 00 00 00 03 testb $0x3,0x88(%rsp)
- f7 44 24 68 03 00 00 00 testl $0x3,0x68(%rsp)
+ f6 44 24 68 03 testb $0x3,0x68(%rsp)
- f7 84 24 90 00 00 00 03 00 00 00 testl $0x3,0x90(%rsp)
+ f6 84 24 90 00 00 00 03 testb $0x3,0x90(%rsp)
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1430140912-7960-2-git-send-email-dvlasenk@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-04-27 21:21:52 +08:00
|
|
|
testb $3, CS+8(%rsp)
|
2015-07-04 03:44:27 +08:00
|
|
|
jz .Lerror_kernelspace
|
2015-06-10 03:36:01 +08:00
|
|
|
|
2015-07-04 03:44:27 +08:00
|
|
|
/*
|
|
|
|
* We entered from user mode or we're pretending to have entered
|
|
|
|
* from user mode due to an IRET fault.
|
|
|
|
*/
|
2008-11-24 20:24:28 +08:00
|
|
|
SWAPGS
|
2017-12-04 22:07:35 +08:00
|
|
|
/* We have user CR3. Change to kernel CR3. */
|
|
|
|
SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
|
2015-06-10 03:36:01 +08:00
|
|
|
|
2015-07-04 03:44:27 +08:00
|
|
|
.Lerror_entry_from_usermode_after_swapgs:
|
2017-12-04 22:07:23 +08:00
|
|
|
/* Put us onto the real thread stack. */
|
|
|
|
popq %r12 /* save return addr in %12 */
|
|
|
|
movq %rsp, %rdi /* arg0 = pt_regs pointer */
|
|
|
|
call sync_regs
|
|
|
|
movq %rax, %rsp /* switch stack */
|
|
|
|
ENCODE_FRAME_POINTER
|
|
|
|
pushq %r12
|
|
|
|
|
2015-11-13 04:59:00 +08:00
|
|
|
/*
|
|
|
|
* We need to tell lockdep that IRQs are off. We can't do this until
|
|
|
|
* we fix gsbase, and we should do it before enter_from_user_mode
|
|
|
|
* (which can take locks).
|
|
|
|
*/
|
|
|
|
TRACE_IRQS_OFF
|
2015-11-13 04:59:04 +08:00
|
|
|
CALL_enter_from_user_mode
|
2015-11-13 04:59:00 +08:00
|
|
|
ret
|
2015-07-04 03:44:31 +08:00
|
|
|
|
2015-07-04 03:44:27 +08:00
|
|
|
.Lerror_entry_done:
|
2008-11-24 20:24:28 +08:00
|
|
|
TRACE_IRQS_OFF
|
|
|
|
ret
|
|
|
|
|
2015-02-27 06:40:34 +08:00
|
|
|
/*
|
|
|
|
* There are two places in the kernel that can potentially fault with
|
|
|
|
* usergs. Handle them here. B stepping K8s sometimes report a
|
|
|
|
* truncated RIP for IRET exceptions returning to compat mode. Check
|
|
|
|
* for these here too.
|
|
|
|
*/
|
2015-07-04 03:44:27 +08:00
|
|
|
.Lerror_kernelspace:
|
2015-06-09 02:43:07 +08:00
|
|
|
leaq native_irq_return_iret(%rip), %rcx
|
|
|
|
cmpq %rcx, RIP+8(%rsp)
|
2015-07-04 03:44:27 +08:00
|
|
|
je .Lerror_bad_iret
|
2015-06-09 02:43:07 +08:00
|
|
|
movl %ecx, %eax /* zero extend */
|
|
|
|
cmpq %rax, RIP+8(%rsp)
|
2015-07-04 03:44:27 +08:00
|
|
|
je .Lbstep_iret
|
2016-04-08 08:31:50 +08:00
|
|
|
cmpq $.Lgs_change, RIP+8(%rsp)
|
2015-07-04 03:44:27 +08:00
|
|
|
jne .Lerror_entry_done
|
2015-06-10 03:36:01 +08:00
|
|
|
|
|
|
|
/*
|
2016-04-08 08:31:50 +08:00
|
|
|
* hack: .Lgs_change can fail with user gsbase. If this happens, fix up
|
2015-06-10 03:36:01 +08:00
|
|
|
* gsbase and proceed. We'll fix up the exception and land in
|
2016-04-08 08:31:50 +08:00
|
|
|
* .Lgs_change's error handler with kernel gsbase.
|
2015-06-10 03:36:01 +08:00
|
|
|
*/
|
2016-09-30 09:01:06 +08:00
|
|
|
SWAPGS
|
2017-12-04 22:07:35 +08:00
|
|
|
SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
|
2016-09-30 09:01:06 +08:00
|
|
|
jmp .Lerror_entry_done
|
2009-10-12 22:18:23 +08:00
|
|
|
|
2015-07-04 03:44:27 +08:00
|
|
|
.Lbstep_iret:
|
2009-10-12 22:18:23 +08:00
|
|
|
/* Fix truncated RIP */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rcx, RIP+8(%rsp)
|
x86_64, traps: Rework bad_iret
It's possible for iretq to userspace to fail. This can happen because
of a bad CS, SS, or RIP.
Historically, we've handled it by fixing up an exception from iretq to
land at bad_iret, which pretends that the failed iret frame was really
the hardware part of #GP(0) from userspace. To make this work, there's
an extra fixup to fudge the gs base into a usable state.
This is suboptimal because it loses the original exception. It's also
buggy because there's no guarantee that we were on the kernel stack to
begin with. For example, if the failing iret happened on return from an
NMI, then we'll end up executing general_protection on the NMI stack.
This is bad for several reasons, the most immediate of which is that
general_protection, as a non-paranoid idtentry, will try to deliver
signals and/or schedule from the wrong stack.
This patch throws out bad_iret entirely. As a replacement, it augments
the existing swapgs fudge into a full-blown iret fixup, mostly written
in C. It's should be clearer and more correct.
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-11-23 10:00:33 +08:00
|
|
|
/* fall through */
|
|
|
|
|
2015-07-04 03:44:27 +08:00
|
|
|
.Lerror_bad_iret:
|
2015-06-10 03:36:01 +08:00
|
|
|
/*
|
2017-12-04 22:07:35 +08:00
|
|
|
* We came from an IRET to user mode, so we have user
|
|
|
|
* gsbase and CR3. Switch to kernel gsbase and CR3:
|
2015-06-10 03:36:01 +08:00
|
|
|
*/
|
x86_64, traps: Rework bad_iret
It's possible for iretq to userspace to fail. This can happen because
of a bad CS, SS, or RIP.
Historically, we've handled it by fixing up an exception from iretq to
land at bad_iret, which pretends that the failed iret frame was really
the hardware part of #GP(0) from userspace. To make this work, there's
an extra fixup to fudge the gs base into a usable state.
This is suboptimal because it loses the original exception. It's also
buggy because there's no guarantee that we were on the kernel stack to
begin with. For example, if the failing iret happened on return from an
NMI, then we'll end up executing general_protection on the NMI stack.
This is bad for several reasons, the most immediate of which is that
general_protection, as a non-paranoid idtentry, will try to deliver
signals and/or schedule from the wrong stack.
This patch throws out bad_iret entirely. As a replacement, it augments
the existing swapgs fudge into a full-blown iret fixup, mostly written
in C. It's should be clearer and more correct.
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-11-23 10:00:33 +08:00
|
|
|
SWAPGS
|
2017-12-04 22:07:35 +08:00
|
|
|
SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
|
2015-06-10 03:36:01 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Pretend that the exception came from user mode: set up pt_regs
|
x86/entry/64: Remove %ebx handling from error_entry/exit
error_entry and error_exit communicate the user vs. kernel status of
the frame using %ebx. This is unnecessary -- the information is in
regs->cs. Just use regs->cs.
This makes error_entry simpler and makes error_exit more robust.
It also fixes a nasty bug. Before all the Spectre nonsense, the
xen_failsafe_callback entry point returned like this:
ALLOC_PT_GPREGS_ON_STACK
SAVE_C_REGS
SAVE_EXTRA_REGS
ENCODE_FRAME_POINTER
jmp error_exit
And it did not go through error_entry. This was bogus: RBX
contained garbage, and error_exit expected a flag in RBX.
Fortunately, it generally contained *nonzero* garbage, so the
correct code path was used. As part of the Spectre fixes, code was
added to clear RBX to mitigate certain speculation attacks. Now,
depending on kernel configuration, RBX got zeroed and, when running
some Wine workloads, the kernel crashes. This was introduced by:
commit 3ac6d8c787b8 ("x86/entry/64: Clear registers for exceptions/interrupts, to reduce speculation attack surface")
With this patch applied, RBX is no longer needed as a flag, and the
problem goes away.
I suspect that malicious userspace could use this bug to crash the
kernel even without the offending patch applied, though.
[ Historical note: I wrote this patch as a cleanup before I was aware
of the bug it fixed. ]
[ Note to stable maintainers: this should probably get applied to all
kernels. If you're nervous about that, a more conservative fix to
add xorl %ebx,%ebx; incl %ebx before the jump to error_exit should
also fix the problem. ]
Reported-and-tested-by: M. Vefa Bicakci <m.v.b@runbox.com>
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Cc: xen-devel@lists.xenproject.org
Fixes: 3ac6d8c787b8 ("x86/entry/64: Clear registers for exceptions/interrupts, to reduce speculation attack surface")
Link: http://lkml.kernel.org/r/b5010a090d3586b2d6e06c7ad3ec5542d1241c45.1532282627.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-07-23 02:05:09 +08:00
|
|
|
* as if we faulted immediately after IRET.
|
2015-06-10 03:36:01 +08:00
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
mov %rsp, %rdi
|
|
|
|
call fixup_bad_iret
|
|
|
|
mov %rax, %rsp
|
2015-07-04 03:44:27 +08:00
|
|
|
jmp .Lerror_entry_from_usermode_after_swapgs
|
2008-11-24 20:24:28 +08:00
|
|
|
END(error_entry)
|
|
|
|
|
|
|
|
ENTRY(error_exit)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_REGS
|
2017-02-03 17:03:25 +08:00
|
|
|
DISABLE_INTERRUPTS(CLBR_ANY)
|
2008-11-24 20:24:28 +08:00
|
|
|
TRACE_IRQS_OFF
|
x86/entry/64: Remove %ebx handling from error_entry/exit
error_entry and error_exit communicate the user vs. kernel status of
the frame using %ebx. This is unnecessary -- the information is in
regs->cs. Just use regs->cs.
This makes error_entry simpler and makes error_exit more robust.
It also fixes a nasty bug. Before all the Spectre nonsense, the
xen_failsafe_callback entry point returned like this:
ALLOC_PT_GPREGS_ON_STACK
SAVE_C_REGS
SAVE_EXTRA_REGS
ENCODE_FRAME_POINTER
jmp error_exit
And it did not go through error_entry. This was bogus: RBX
contained garbage, and error_exit expected a flag in RBX.
Fortunately, it generally contained *nonzero* garbage, so the
correct code path was used. As part of the Spectre fixes, code was
added to clear RBX to mitigate certain speculation attacks. Now,
depending on kernel configuration, RBX got zeroed and, when running
some Wine workloads, the kernel crashes. This was introduced by:
commit 3ac6d8c787b8 ("x86/entry/64: Clear registers for exceptions/interrupts, to reduce speculation attack surface")
With this patch applied, RBX is no longer needed as a flag, and the
problem goes away.
I suspect that malicious userspace could use this bug to crash the
kernel even without the offending patch applied, though.
[ Historical note: I wrote this patch as a cleanup before I was aware
of the bug it fixed. ]
[ Note to stable maintainers: this should probably get applied to all
kernels. If you're nervous about that, a more conservative fix to
add xorl %ebx,%ebx; incl %ebx before the jump to error_exit should
also fix the problem. ]
Reported-and-tested-by: M. Vefa Bicakci <m.v.b@runbox.com>
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Cc: xen-devel@lists.xenproject.org
Fixes: 3ac6d8c787b8 ("x86/entry/64: Clear registers for exceptions/interrupts, to reduce speculation attack surface")
Link: http://lkml.kernel.org/r/b5010a090d3586b2d6e06c7ad3ec5542d1241c45.1532282627.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-07-23 02:05:09 +08:00
|
|
|
testb $3, CS(%rsp)
|
|
|
|
jz retint_kernel
|
2015-06-09 02:43:07 +08:00
|
|
|
jmp retint_user
|
2008-11-24 20:24:28 +08:00
|
|
|
END(error_exit)
|
|
|
|
|
2017-11-02 15:59:08 +08:00
|
|
|
/*
|
|
|
|
* Runs on exception stack. Xen PV does not go through this path at all,
|
|
|
|
* so we can use real assembly here.
|
2017-12-04 22:07:35 +08:00
|
|
|
*
|
|
|
|
* Registers:
|
|
|
|
* %r14: Used to save/restore the CR3 of the interrupted context
|
|
|
|
* when PAGE_TABLE_ISOLATION is in use. Do not clobber.
|
2017-11-02 15:59:08 +08:00
|
|
|
*/
|
2008-11-24 20:24:28 +08:00
|
|
|
ENTRY(nmi)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2017-11-02 15:59:08 +08:00
|
|
|
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
/*
|
|
|
|
* We allow breakpoints in NMIs. If a breakpoint occurs, then
|
|
|
|
* the iretq it performs will take us out of NMI context.
|
|
|
|
* This means that we can have nested NMIs where the next
|
|
|
|
* NMI is using the top of the stack of the previous NMI. We
|
|
|
|
* can't let it execute because the nested NMI will corrupt the
|
|
|
|
* stack of the previous NMI. NMI handlers are not re-entrant
|
|
|
|
* anyway.
|
|
|
|
*
|
|
|
|
* To handle this case we do the following:
|
|
|
|
* Check the a special location on the stack that contains
|
|
|
|
* a variable that is set when NMIs are executing.
|
|
|
|
* The interrupted task's stack is also checked to see if it
|
|
|
|
* is an NMI stack.
|
|
|
|
* If the variable is not set and the stack is not the NMI
|
|
|
|
* stack then:
|
|
|
|
* o Set the special variable on the stack
|
2015-07-16 01:29:36 +08:00
|
|
|
* o Copy the interrupt frame into an "outermost" location on the
|
|
|
|
* stack
|
|
|
|
* o Copy the interrupt frame into an "iret" location on the stack
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
* o Continue processing the NMI
|
|
|
|
* If the variable is set or the previous stack is the NMI stack:
|
2015-07-16 01:29:36 +08:00
|
|
|
* o Modify the "iret" location to jump to the repeat_nmi
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
* o return back to the first NMI
|
|
|
|
*
|
|
|
|
* Now on exit of the first NMI, we first clear the stack variable
|
|
|
|
* The NMI stack will tell any nested NMIs at that point that it is
|
|
|
|
* nested. Then we pop the stack normally with iret, and if there was
|
|
|
|
* a nested NMI that updated the copy interrupt stack frame, a
|
|
|
|
* jump will be made to the repeat_nmi code that will handle the second
|
|
|
|
* NMI.
|
2015-07-16 01:29:35 +08:00
|
|
|
*
|
|
|
|
* However, espfix prevents us from directly returning to userspace
|
|
|
|
* with a single IRET instruction. Similarly, IRET to user mode
|
|
|
|
* can fault. We therefore handle NMIs from user space like
|
|
|
|
* other IST entries.
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
*/
|
|
|
|
|
2017-08-08 10:43:13 +08:00
|
|
|
ASM_CLAC
|
|
|
|
|
2015-03-26 01:18:13 +08:00
|
|
|
/* Use %rdx as our temp variable throughout */
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq %rdx
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
2015-07-16 01:29:35 +08:00
|
|
|
testb $3, CS-RIP+8(%rsp)
|
|
|
|
jz .Lnmi_from_kernel
|
|
|
|
|
|
|
|
/*
|
|
|
|
* NMI from user mode. We need to run on the thread stack, but we
|
|
|
|
* can't go through the normal entry paths: NMIs are masked, and
|
|
|
|
* we don't want to enable interrupts, because then we'll end
|
|
|
|
* up in an awkward situation in which IRQs are on but NMIs
|
|
|
|
* are off.
|
2015-09-21 07:32:05 +08:00
|
|
|
*
|
|
|
|
* We also must not push anything to the stack before switching
|
|
|
|
* stacks lest we corrupt the "NMI executing" variable.
|
2015-07-16 01:29:35 +08:00
|
|
|
*/
|
|
|
|
|
2017-11-02 15:59:08 +08:00
|
|
|
swapgs
|
2015-07-16 01:29:35 +08:00
|
|
|
cld
|
2017-12-04 22:07:35 +08:00
|
|
|
SWITCH_TO_KERNEL_CR3 scratch_reg=%rdx
|
2015-07-16 01:29:35 +08:00
|
|
|
movq %rsp, %rdx
|
|
|
|
movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS base=%rdx offset=8
|
2015-07-16 01:29:35 +08:00
|
|
|
pushq 5*8(%rdx) /* pt_regs->ss */
|
|
|
|
pushq 4*8(%rdx) /* pt_regs->rsp */
|
|
|
|
pushq 3*8(%rdx) /* pt_regs->flags */
|
|
|
|
pushq 2*8(%rdx) /* pt_regs->cs */
|
|
|
|
pushq 1*8(%rdx) /* pt_regs->rip */
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2015-07-16 01:29:35 +08:00
|
|
|
pushq $-1 /* pt_regs->orig_ax */
|
2018-02-11 18:49:46 +08:00
|
|
|
PUSH_AND_CLEAR_REGS rdx=(%rdx)
|
2016-10-21 00:34:40 +08:00
|
|
|
ENCODE_FRAME_POINTER
|
2015-07-16 01:29:35 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* At this point we no longer need to worry about stack damage
|
|
|
|
* due to nesting -- we're on the normal thread stack and we're
|
|
|
|
* done with the NMI stack.
|
|
|
|
*/
|
|
|
|
|
|
|
|
movq %rsp, %rdi
|
|
|
|
movq $-1, %rsi
|
|
|
|
call do_nmi
|
|
|
|
|
2012-02-20 05:43:37 +08:00
|
|
|
/*
|
2015-07-16 01:29:35 +08:00
|
|
|
* Return back to user mode. We must *not* do the normal exit
|
2016-10-21 00:34:40 +08:00
|
|
|
* work, because we don't want to enable interrupts.
|
2012-02-20 05:43:37 +08:00
|
|
|
*/
|
2017-11-02 15:59:00 +08:00
|
|
|
jmp swapgs_restore_regs_and_return_to_usermode
|
2012-02-20 05:43:37 +08:00
|
|
|
|
2015-07-16 01:29:35 +08:00
|
|
|
.Lnmi_from_kernel:
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
/*
|
2015-07-16 01:29:36 +08:00
|
|
|
* Here's what our stack frame will look like:
|
|
|
|
* +---------------------------------------------------------+
|
|
|
|
* | original SS |
|
|
|
|
* | original Return RSP |
|
|
|
|
* | original RFLAGS |
|
|
|
|
* | original CS |
|
|
|
|
* | original RIP |
|
|
|
|
* +---------------------------------------------------------+
|
|
|
|
* | temp storage for rdx |
|
|
|
|
* +---------------------------------------------------------+
|
|
|
|
* | "NMI executing" variable |
|
|
|
|
* +---------------------------------------------------------+
|
|
|
|
* | iret SS } Copied from "outermost" frame |
|
|
|
|
* | iret Return RSP } on each loop iteration; overwritten |
|
|
|
|
* | iret RFLAGS } by a nested NMI to force another |
|
|
|
|
* | iret CS } iteration if needed. |
|
|
|
|
* | iret RIP } |
|
|
|
|
* +---------------------------------------------------------+
|
|
|
|
* | outermost SS } initialized in first_nmi; |
|
|
|
|
* | outermost Return RSP } will not be changed before |
|
|
|
|
* | outermost RFLAGS } NMI processing is done. |
|
|
|
|
* | outermost CS } Copied to "iret" frame on each |
|
|
|
|
* | outermost RIP } iteration. |
|
|
|
|
* +---------------------------------------------------------+
|
|
|
|
* | pt_regs |
|
|
|
|
* +---------------------------------------------------------+
|
|
|
|
*
|
|
|
|
* The "original" frame is used by hardware. Before re-enabling
|
|
|
|
* NMIs, we need to be done with it, and we need to leave enough
|
|
|
|
* space for the asm code here.
|
|
|
|
*
|
|
|
|
* We return by executing IRET while RSP points to the "iret" frame.
|
|
|
|
* That will either return for real or it will loop back into NMI
|
|
|
|
* processing.
|
|
|
|
*
|
|
|
|
* The "outermost" frame is copied to the "iret" frame on each
|
|
|
|
* iteration of the loop, so each iteration starts with the "iret"
|
|
|
|
* frame pointing to the final return target.
|
|
|
|
*/
|
|
|
|
|
2012-02-20 05:43:37 +08:00
|
|
|
/*
|
2015-07-16 01:29:36 +08:00
|
|
|
* Determine whether we're a nested NMI.
|
|
|
|
*
|
2015-07-16 01:29:37 +08:00
|
|
|
* If we interrupted kernel code between repeat_nmi and
|
|
|
|
* end_repeat_nmi, then we are a nested NMI. We must not
|
|
|
|
* modify the "iret" frame because it's being written by
|
|
|
|
* the outer NMI. That's okay; the outer NMI handler is
|
|
|
|
* about to about to call do_nmi anyway, so we can just
|
|
|
|
* resume the outer NMI.
|
2012-02-20 05:43:37 +08:00
|
|
|
*/
|
2015-07-16 01:29:37 +08:00
|
|
|
|
|
|
|
movq $repeat_nmi, %rdx
|
|
|
|
cmpq 8(%rsp), %rdx
|
|
|
|
ja 1f
|
|
|
|
movq $end_repeat_nmi, %rdx
|
|
|
|
cmpq 8(%rsp), %rdx
|
|
|
|
ja nested_nmi_out
|
|
|
|
1:
|
2012-02-20 05:43:37 +08:00
|
|
|
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
/*
|
2015-07-16 01:29:37 +08:00
|
|
|
* Now check "NMI executing". If it's set, then we're nested.
|
2015-07-16 01:29:36 +08:00
|
|
|
* This will not detect if we interrupted an outer NMI just
|
|
|
|
* before IRET.
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
cmpl $1, -8(%rsp)
|
|
|
|
je nested_nmi
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
|
|
|
/*
|
2015-07-16 01:29:36 +08:00
|
|
|
* Now test if the previous stack was an NMI stack. This covers
|
|
|
|
* the case where we interrupt an outer NMI after it clears
|
2015-07-16 01:29:38 +08:00
|
|
|
* "NMI executing" but before IRET. We need to be careful, though:
|
|
|
|
* there is one case in which RSP could point to the NMI stack
|
|
|
|
* despite there being no NMI active: naughty userspace controls
|
|
|
|
* RSP at the very beginning of the SYSCALL targets. We can
|
|
|
|
* pull a fast one on naughty userspace, though: we program
|
|
|
|
* SYSCALL to mask DF, so userspace cannot cause DF to be set
|
|
|
|
* if it controls the kernel's RSP. We set DF before we clear
|
|
|
|
* "NMI executing".
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
*/
|
2015-04-01 22:50:57 +08:00
|
|
|
lea 6*8(%rsp), %rdx
|
|
|
|
/* Compare the NMI stack (rdx) with the stack we came from (4*8(%rsp)) */
|
|
|
|
cmpq %rdx, 4*8(%rsp)
|
|
|
|
/* If the stack pointer is above the NMI stack, this is a normal NMI */
|
|
|
|
ja first_nmi
|
2015-06-09 02:43:07 +08:00
|
|
|
|
2015-04-01 22:50:57 +08:00
|
|
|
subq $EXCEPTION_STKSZ, %rdx
|
|
|
|
cmpq %rdx, 4*8(%rsp)
|
|
|
|
/* If it is below the NMI stack, it is a normal NMI */
|
|
|
|
jb first_nmi
|
2015-07-16 01:29:38 +08:00
|
|
|
|
|
|
|
/* Ah, it is within the NMI stack. */
|
|
|
|
|
|
|
|
testb $(X86_EFLAGS_DF >> 8), (3*8 + 1)(%rsp)
|
|
|
|
jz first_nmi /* RSP was user controlled. */
|
|
|
|
|
|
|
|
/* This is a nested NMI. */
|
2015-04-01 22:50:57 +08:00
|
|
|
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
nested_nmi:
|
|
|
|
/*
|
2015-07-16 01:29:36 +08:00
|
|
|
* Modify the "iret" frame to point to repeat_nmi, forcing another
|
|
|
|
* iteration of NMI handling.
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
*/
|
2015-07-16 01:29:39 +08:00
|
|
|
subq $8, %rsp
|
2015-06-09 02:43:07 +08:00
|
|
|
leaq -10*8(%rsp), %rdx
|
|
|
|
pushq $__KERNEL_DS
|
|
|
|
pushq %rdx
|
x86/debug: Remove perpetually broken, unmaintainable dwarf annotations
So the dwarf2 annotations in low level assembly code have
become an increasing hindrance: unreadable, messy macros
mixed into some of the most security sensitive code paths
of the Linux kernel.
These debug info annotations don't even buy the upstream
kernel anything: dwarf driven stack unwinding has caused
problems in the past so it's out of tree, and the upstream
kernel only uses the much more robust framepointers based
stack unwinding method.
In addition to that there's a steady, slow bitrot going
on with these annotations, requiring frequent fixups.
There's no tooling and no functionality upstream that
keeps it correct.
So burn down the sick forest, allowing new, healthier growth:
27 files changed, 350 insertions(+), 1101 deletions(-)
Someone who has the willingness and time to do this
properly can attempt to reintroduce dwarf debuginfo in x86
assembly code plus dwarf unwinding from first principles,
with the following conditions:
- it should be maximally readable, and maximally low-key to
'ordinary' code reading and maintenance.
- find a build time method to insert dwarf annotations
automatically in the most common cases, for pop/push
instructions that manipulate the stack pointer. This could
be done for example via a preprocessing step that just
looks for common patterns - plus special annotations for
the few cases where we want to depart from the default.
We have hundreds of CFI annotations, so automating most of
that makes sense.
- it should come with build tooling checks that ensure that
CFI annotations are sensible. We've seen such efforts from
the framepointer side, and there's no reason it couldn't be
done on the dwarf side.
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Jan Beulich <JBeulich@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-05-28 18:21:47 +08:00
|
|
|
pushfq
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq $__KERNEL_CS
|
|
|
|
pushq $repeat_nmi
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
|
|
|
/* Put stack back */
|
2015-06-09 02:43:07 +08:00
|
|
|
addq $(6*8), %rsp
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
|
|
|
nested_nmi_out:
|
2015-06-09 02:43:07 +08:00
|
|
|
popq %rdx
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
2015-07-16 01:29:36 +08:00
|
|
|
/* We are returning to kernel mode, so this cannot result in a fault. */
|
2017-11-02 15:59:08 +08:00
|
|
|
iretq
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
|
|
|
first_nmi:
|
2015-07-16 01:29:36 +08:00
|
|
|
/* Restore rdx. */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq (%rsp), %rdx
|
2012-02-24 22:54:37 +08:00
|
|
|
|
2015-07-16 01:29:40 +08:00
|
|
|
/* Make room for "NMI executing". */
|
|
|
|
pushq $0
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
2015-07-16 01:29:36 +08:00
|
|
|
/* Leave room for the "iret" frame */
|
2015-06-09 02:43:07 +08:00
|
|
|
subq $(5*8), %rsp
|
2012-10-02 08:29:25 +08:00
|
|
|
|
2015-07-16 01:29:36 +08:00
|
|
|
/* Copy the "original" frame to the "outermost" frame */
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
.rept 5
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq 11*8(%rsp)
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
.endr
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2012-02-24 22:54:37 +08:00
|
|
|
|
2012-02-25 04:55:13 +08:00
|
|
|
/* Everything up to here is safe from nested NMIs */
|
|
|
|
|
2015-07-16 01:29:41 +08:00
|
|
|
#ifdef CONFIG_DEBUG_ENTRY
|
|
|
|
/*
|
|
|
|
* For ease of testing, unmask NMIs right away. Disabled by
|
|
|
|
* default because IRET is very expensive.
|
|
|
|
*/
|
|
|
|
pushq $0 /* SS */
|
|
|
|
pushq %rsp /* RSP (minus 8 because of the previous push) */
|
|
|
|
addq $8, (%rsp) /* Fix up RSP */
|
|
|
|
pushfq /* RFLAGS */
|
|
|
|
pushq $__KERNEL_CS /* CS */
|
|
|
|
pushq $1f /* RIP */
|
2017-11-02 15:59:08 +08:00
|
|
|
iretq /* continues at repeat_nmi below */
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2015-07-16 01:29:41 +08:00
|
|
|
1:
|
|
|
|
#endif
|
|
|
|
|
2015-07-16 01:29:36 +08:00
|
|
|
repeat_nmi:
|
2012-02-24 22:54:37 +08:00
|
|
|
/*
|
|
|
|
* If there was a nested NMI, the first NMI's iret will return
|
|
|
|
* here. But NMIs are still enabled and we can take another
|
|
|
|
* nested NMI. The nested NMI checks the interrupted RIP to see
|
|
|
|
* if it is between repeat_nmi and end_repeat_nmi, and if so
|
|
|
|
* it will just return, as we are about to repeat an NMI anyway.
|
|
|
|
* This makes it safe to copy to the stack frame that a nested
|
|
|
|
* NMI will update.
|
2015-07-16 01:29:36 +08:00
|
|
|
*
|
|
|
|
* RSP is pointing to "outermost RIP". gsbase is unknown, but, if
|
|
|
|
* we're repeating an NMI, gsbase has the same value that it had on
|
|
|
|
* the first iteration. paranoid_entry will load the kernel
|
2015-07-16 01:29:40 +08:00
|
|
|
* gsbase if needed before we call do_nmi. "NMI executing"
|
|
|
|
* is zero.
|
2012-02-24 22:54:37 +08:00
|
|
|
*/
|
2015-07-16 01:29:40 +08:00
|
|
|
movq $1, 10*8(%rsp) /* Set "NMI executing". */
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
2012-02-24 22:54:37 +08:00
|
|
|
/*
|
2015-07-16 01:29:36 +08:00
|
|
|
* Copy the "outermost" frame to the "iret" frame. NMIs that nest
|
|
|
|
* here must not modify the "iret" frame while we're writing to
|
|
|
|
* it or it will end up containing garbage.
|
2012-02-24 22:54:37 +08:00
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
addq $(10*8), %rsp
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
.rept 5
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq -6*8(%rsp)
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
.endr
|
2015-06-09 02:43:07 +08:00
|
|
|
subq $(5*8), %rsp
|
2012-02-24 22:54:37 +08:00
|
|
|
end_repeat_nmi:
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
|
|
|
|
/*
|
2015-07-16 01:29:36 +08:00
|
|
|
* Everything below this point can be preempted by a nested NMI.
|
|
|
|
* If this happens, then the inner NMI will change the "iret"
|
|
|
|
* frame to point back to repeat_nmi.
|
x86: Add workaround to NMI iret woes
In x86, when an NMI goes off, the CPU goes into an NMI context that
prevents other NMIs to trigger on that CPU. If an NMI is suppose to
trigger, it has to wait till the previous NMI leaves NMI context.
At that time, the next NMI can trigger (note, only one more NMI will
trigger, as only one can be latched at a time).
The way x86 gets out of NMI context is by calling iret. The problem
with this is that this causes problems if the NMI handle either
triggers an exception, or a breakpoint. Both the exception and the
breakpoint handlers will finish with an iret. If this happens while
in NMI context, the CPU will leave NMI context and a new NMI may come
in. As NMI handlers are not made to be re-entrant, this can cause
havoc with the system, not to mention, the nested NMI will write
all over the previous NMI's stack.
Linus Torvalds proposed the following workaround to this problem:
https://lkml.org/lkml/2010/7/14/264
"In fact, I wonder if we couldn't just do a software NMI disable
instead? Hav ea per-cpu variable (in the _core_ percpu areas that get
allocated statically) that points to the NMI stack frame, and just
make the NMI code itself do something like
NMI entry:
- load percpu NMI stack frame pointer
- if non-zero we know we're nested, and should ignore this NMI:
- we're returning to kernel mode, so return immediately by using
"popf/ret", which also keeps NMI's disabled in the hardware until the
"real" NMI iret happens.
- before the popf/iret, use the NMI stack pointer to make the NMI
return stack be invalid and cause a fault
- set the NMI stack pointer to the current stack pointer
NMI exit (not the above "immediate exit because we nested"):
clear the percpu NMI stack pointer
Just do the iret.
Now, the thing is, now the "iret" is atomic. If we had a nested NMI,
we'll take a fault, and that re-does our "delayed" NMI - and NMI's
will stay masked.
And if we didn't have a nested NMI, that iret will now unmask NMI's,
and everything is happy."
I first tried to follow this advice but as I started implementing this
code, a few gotchas showed up.
One, is accessing per-cpu variables in the NMI handler.
The problem is that per-cpu variables use the %gs register to get the
variable for the given CPU. But as the NMI may happen in userspace,
we must first perform a SWAPGS to get to it. The NMI handler already
does this later in the code, but its too late as we have saved off
all the registers and we don't want to do that for a disabled NMI.
Peter Zijlstra suggested to keep all variables on the stack. This
simplifies things greatly and it has the added benefit of cache locality.
Two, faulting on the iret.
I really wanted to make this work, but it was becoming very hacky, and
I never got it to be stable. The iret already had a fault handler for
userspace faulting with bad segment registers, and getting NMI to trigger
a fault and detect it was very tricky. But for strange reasons, the system
would usually take a double fault and crash. I never figured out why
and decided to go with a simple "jmp" approach. The new approach I took
also simplified things.
Finally, the last problem with Linus's approach was to have the nested
NMI handler do a ret instead of an iret to give the first NMI NMI-context
again.
The problem is that ret is much more limited than an iret. I couldn't figure
out how to get the stack back where it belonged. I could have copied the
current stack, pushed the return onto it, but my fear here is that there
may be some place that writes data below the stack pointer. I know that
is not something code should depend on, but I don't want to chance it.
I may add this feature later, but for now, an NMI handler that loses NMI
context will not get it back.
Here's what is done:
When an NMI comes in, the HW pushes the interrupt stack frame onto the
per cpu NMI stack that is selected by the IST.
A special location on the NMI stack holds a variable that is set when
the first NMI handler runs. If this variable is set then we know that
this is a nested NMI and we process the nested NMI code.
There is still a race when this variable is cleared and an NMI comes
in just before the first NMI does the return. For this case, if the
variable is cleared, we also check if the interrupted stack is the
NMI stack. If it is, then we process the nested NMI code.
Why the two tests and not just test the interrupted stack?
If the first NMI hits a breakpoint and loses NMI context, and then it
hits another breakpoint and while processing that breakpoint we get a
nested NMI. When processing a breakpoint, the stack changes to the
breakpoint stack. If another NMI comes in here we can't rely on the
interrupted stack to be the NMI stack.
If the variable is not set and the interrupted task's stack is not the
NMI stack, then we know this is the first NMI and we can process things
normally. But in order to do so, we need to do a few things first.
1) Set the stack variable that tells us that we are in an NMI handler
2) Make two copies of the interrupt stack frame.
One copy is used to return on iret
The other is used to restore the first one if we have a nested NMI.
This is what the stack will look like:
+-------------------------+
| original SS |
| original Return RSP |
| original RFLAGS |
| original CS |
| original RIP |
+-------------------------+
| temp storage for rdx |
+-------------------------+
| NMI executing variable |
+-------------------------+
| Saved SS |
| Saved Return RSP |
| Saved RFLAGS |
| Saved CS |
| Saved RIP |
+-------------------------+
| copied SS |
| copied Return RSP |
| copied RFLAGS |
| copied CS |
| copied RIP |
+-------------------------+
| pt_regs |
+-------------------------+
The original stack frame contains what the HW put in when we entered
the NMI.
We store %rdx as a temp variable to use. Both the original HW stack
frame and this %rdx storage will be clobbered by nested NMIs so we
can not rely on them later in the first NMI handler.
The next item is the special stack variable that is set when we execute
the rest of the NMI handler.
Then we have two copies of the interrupt stack. The second copy is
modified by any nested NMIs to let the first NMI know that we triggered
a second NMI (latched) and that we should repeat the NMI handler.
If the first NMI hits an exception or breakpoint that takes it out of
NMI context, if a second NMI comes in before the first one finishes,
it will update the copied interrupt stack to point to a fix up location
to trigger another NMI.
When the first NMI calls iret, it will instead jump to the fix up
location. This fix up location will copy the saved interrupt stack back
to the copy and execute the nmi handler again.
Note, the nested NMI knows enough to check if it preempted a previous
NMI handler while it is in the fixup location. If it has, it will not
modify the copied interrupt stack and will just leave as if nothing
happened. As the NMI handle is about to execute again, there's no reason
to latch now.
To test all this, I forced the NMI handler to call iret and take itself
out of NMI context. I also added assemble code to write to the serial to
make sure that it hits the nested path as well as the fix up path.
Everything seems to be working fine.
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul Turner <pjt@google.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-12-09 01:36:23 +08:00
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
pushq $-1 /* ORIG_RAX: no syscall to restart */
|
x86/asm/entry/64: Always allocate a complete "struct pt_regs" on the kernel stack
The 64-bit entry code was using six stack slots less by not
saving/restoring registers which are callee-preserved according
to the C ABI, and was not allocating space for them.
Only when syscalls needed a complete "struct pt_regs" was
the complete area allocated and filled in.
As an additional twist, on interrupt entry a "slightly less
truncated pt_regs" trick is used, to make nested interrupt
stacks easier to unwind.
This proved to be a source of significant obfuscation and subtle
bugs. For example, 'stub_fork' had to pop the return address,
extend the struct, save registers, and push return address back.
Ugly. 'ia32_ptregs_common' pops return address and "returns" via
jmp insn, throwing a wrench into CPU return stack cache.
This patch changes the code to always allocate a complete
"struct pt_regs" on the kernel stack. The saving of registers
is still done lazily.
"Partial pt_regs" trick on interrupt stack is retained.
Macros which manipulate "struct pt_regs" on stack are reworked:
- ALLOC_PT_GPREGS_ON_STACK allocates the structure.
- SAVE_C_REGS saves to it those registers which are clobbered
by C code.
- SAVE_EXTRA_REGS saves to it all other registers.
- Corresponding RESTORE_* and REMOVE_PT_GPREGS_FROM_STACK macros
reverse it.
'ia32_ptregs_common', 'stub_fork' and friends lost their ugly dance
with the return pointer.
LOAD_ARGS32 in ia32entry.S now uses symbolic stack offsets
instead of magic numbers.
'error_entry' and 'save_paranoid' now use SAVE_C_REGS +
SAVE_EXTRA_REGS instead of having it open-coded yet again.
Patch was run-tested: 64-bit executables, 32-bit executables,
strace works.
Timing tests did not show measurable difference in 32-bit
and 64-bit syscalls.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Will Drewry <wad@chromium.org>
Link: http://lkml.kernel.org/r/1423778052-21038-2-git-send-email-dvlasenk@redhat.com
Link: http://lkml.kernel.org/r/b89763d354aa23e670b9bdf3a40ae320320a7c2e.1424989793.git.luto@amacapital.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-02-27 06:40:27 +08:00
|
|
|
|
2011-12-09 01:32:27 +08:00
|
|
|
/*
|
2015-02-27 06:40:34 +08:00
|
|
|
* Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit
|
2011-12-09 01:32:27 +08:00
|
|
|
* as we should not be calling schedule in NMI context.
|
|
|
|
* Even with normal interrupts enabled. An NMI should not be
|
|
|
|
* setting NEED_RESCHED or anything that normal interrupts and
|
|
|
|
* exceptions might do.
|
|
|
|
*/
|
2015-06-09 02:43:07 +08:00
|
|
|
call paranoid_entry
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_REGS
|
2012-06-07 22:21:21 +08:00
|
|
|
|
2008-11-24 20:24:28 +08:00
|
|
|
/* paranoidentry do_nmi, 0; without TRACE_IRQS_OFF */
|
2015-06-09 02:43:07 +08:00
|
|
|
movq %rsp, %rdi
|
|
|
|
movq $-1, %rsi
|
|
|
|
call do_nmi
|
2012-06-07 22:21:21 +08:00
|
|
|
|
2017-12-04 22:08:00 +08:00
|
|
|
RESTORE_CR3 scratch_reg=%r15 save_reg=%r14
|
2017-12-04 22:07:35 +08:00
|
|
|
|
2015-06-09 02:43:07 +08:00
|
|
|
testl %ebx, %ebx /* swapgs needed? */
|
|
|
|
jnz nmi_restore
|
2008-11-24 20:24:28 +08:00
|
|
|
nmi_swapgs:
|
|
|
|
SWAPGS_UNSAFE_STACK
|
|
|
|
nmi_restore:
|
2018-02-11 18:49:43 +08:00
|
|
|
POP_REGS
|
2015-07-16 01:29:36 +08:00
|
|
|
|
2017-11-02 15:59:05 +08:00
|
|
|
/*
|
|
|
|
* Skip orig_ax and the "outermost" frame to point RSP at the "iret"
|
|
|
|
* at the "iret" frame.
|
|
|
|
*/
|
|
|
|
addq $6*8, %rsp
|
2012-10-02 08:29:25 +08:00
|
|
|
|
2015-07-16 01:29:38 +08:00
|
|
|
/*
|
|
|
|
* Clear "NMI executing". Set DF first so that we can easily
|
|
|
|
* distinguish the remaining code between here and IRET from
|
2017-11-02 15:59:08 +08:00
|
|
|
* the SYSCALL entry and exit paths.
|
|
|
|
*
|
|
|
|
* We arguably should just inspect RIP instead, but I (Andy) wrote
|
|
|
|
* this code when I had the misapprehension that Xen PV supported
|
|
|
|
* NMIs, and Xen PV would break that approach.
|
2015-07-16 01:29:38 +08:00
|
|
|
*/
|
|
|
|
std
|
|
|
|
movq $0, 5*8(%rsp) /* clear "NMI executing" */
|
2015-07-16 01:29:36 +08:00
|
|
|
|
|
|
|
/*
|
2017-11-02 15:59:08 +08:00
|
|
|
* iretq reads the "iret" frame and exits the NMI stack in a
|
|
|
|
* single instruction. We are returning to kernel mode, so this
|
|
|
|
* cannot result in a fault. Similarly, we don't need to worry
|
|
|
|
* about espfix64 on the way back to kernel mode.
|
2015-07-16 01:29:36 +08:00
|
|
|
*/
|
2017-11-02 15:59:08 +08:00
|
|
|
iretq
|
2008-11-24 20:24:28 +08:00
|
|
|
END(nmi)
|
|
|
|
|
|
|
|
ENTRY(ignore_sysret)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_EMPTY
|
2015-06-09 02:43:07 +08:00
|
|
|
mov $-ENOSYS, %eax
|
2008-11-24 20:24:28 +08:00
|
|
|
sysret
|
|
|
|
END(ignore_sysret)
|
2016-07-15 04:22:55 +08:00
|
|
|
|
|
|
|
ENTRY(rewind_stack_do_exit)
|
2017-07-11 23:33:44 +08:00
|
|
|
UNWIND_HINT_FUNC
|
2016-07-15 04:22:55 +08:00
|
|
|
/* Prevent any naive code from trying to unwind to our caller. */
|
|
|
|
xorl %ebp, %ebp
|
|
|
|
|
|
|
|
movq PER_CPU_VAR(cpu_current_top_of_stack), %rax
|
2017-07-11 23:33:44 +08:00
|
|
|
leaq -PTREGS_SIZE(%rax), %rsp
|
|
|
|
UNWIND_HINT_FUNC sp_offset=PTREGS_SIZE
|
2016-07-15 04:22:55 +08:00
|
|
|
|
|
|
|
call do_exit
|
|
|
|
END(rewind_stack_do_exit)
|