Currently with -mindirect-branch=thunk and -mfunction-return=thunk compiler
options expoline thunks are put into individual COMDAT group sections. s390
is the only architecture which has group sections and it has implications
for kpatch and objtool tools support.
Using -mindirect-branch=thunk-extern and -mfunction-return=thunk-extern
is an alternative, which comes with a need to generate all required
expoline thunks manually. Unfortunately modules area is too far away from
the kernel image, and expolines from the kernel image cannon be used.
But since all new distributions (except Debian) build kernels for machine
generations newer than z10, where "exrl" instruction is available, that
leaves only 16 expolines thunks possible.
Provide an option to build the kernel with
-mindirect-branch=thunk-extern and -mfunction-return=thunk-extern for
z10 or newer. This also requires to postlink expoline thunks into all
modules explicitly. Currently modules already contain most expolines
anyhow.
Unfortunately -mindirect-branch=thunk-extern and
-mfunction-return=thunk-extern options support is broken in gcc <= 11.2.
Additional compile test is required to verify proper gcc support.
Acked-by: Ilya Leoshkevich <iii@linux.ibm.com>
Co-developed-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Currently assembler generated expoline thunks are always in a form
__s390_indirect_jump_rXuse_rX even when exrl instruction is available
and no additional register is utilized.
Generate __s390_indirect_jump_rX versions using a single register if the
kernel is built for z10 or newer machine, which have exrl instruction
available. Thunks generated are identical to the ones generated by the
compiler.
This helps to reduce the number of thunks for newer machines generations.
Acked-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Acked-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Commit c1e18c17bd ("s390/pci: add zpci_set_irq()/zpci_clear_irq()")
made zpci_set_irq()/zpci_clear_irq() non-static in preparation for using
them in zpci_hot_reset_device(). The version of zpci_hot_reset_device()
that was finally merged however exploits the fact that IRQs and DMA is
implicitly disabled by clp_disable_fh() so the call to zpci_clear_irq()
was never added. There are no other calls outside pci_irq.c so lets make
both functions static.
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
This reverts commit 6deaa3bbca ("s390: extend expoline to BC
instructions"). Expolines to BC instructions were added to be utilized
by commit de5cb6eb51 ("s390: use expoline thunks in the BPF JIT"). But
corresponding code has been removed by commit e1cf4befa2 ("bpf, s390x:
remove ld_abs/ld_ind"). And compiler does not generate such expolines as
well.
Compared to regular expolines, expolines to BC instructions contain
displacement and all possible variations cannot be generated in advance,
making kpatch support more complicated. So, remove those to avoid
future usages.
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Acked-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Change struct ext_code to contain a union which allows to simply
assign the int_code instead of using a cast.
In order to keep the patch small the anonymous union is embedded
within the existing struct instead of changing the struct ext_code to
a union.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Add a proper union in lowcore to reflect architecture and get rid of a
"magic" cast in order to read the full per code.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Add a proper union in lowcore to reflect architecture and get rid of a
"magic" cast in order to read the full program interruption code.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Fix the following build warning with W=1
arch/s390/lib/test_unwind.c:172:21: warning: variable 'fops' set but not used [-Wunused-but-set-variable]
struct ftrace_ops *fops;
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
The only user is gone. Remove the section.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Instead of decoding the instruction that faulted to get the register
which needs to be zeroed, simply encode its number into the extable
entries during code generation. This allows to get rid of a bit of
code, and is also what other architectures are doing.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Reviewed-by: Ilya Leoshkevich <iii@linux.ibm.com>
Tested-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
This is more or less a combination of commit 2e77a62cb3 ("arm64:
extable: add a dedicated uaccess handler") and commit 4b5305decc
("x86/extable: Extend extable functionality").
To describe the problem that needs to solved let's cite the full arm64
commit message:
------
For inline assembly, we place exception fixups out-of-line in the
`.fixup` section such that these are out of the way of the fast path.
This has a few drawbacks:
* Since the fixup code is anonymous, backtraces will symbolize fixups
as offsets from the nearest prior symbol, currently
`__entry_tramp_text_end`. This is confusing, and painful to debug
without access to the relevant vmlinux.
* Since the exception handler adjusts the PC to execute the fixup, and
the fixup uses a direct branch back into the function it fixes,
backtraces of fixups miss the original function. This is confusing,
and violates requirements for RELIABLE_STACKTRACE (and therefore
LIVEPATCH).
* Inline assembly and associated fixups are generated from templates,
and we have many copies of logically identical fixups which only
differ in which specific registers are written to and which address
is branched to at the end of the fixup. This is potentially wasteful
of I-cache resources, and makes it hard to add additional logic to
fixups without significant bloat.
This patch address all three concerns for inline uaccess fixups by
adding a dedicated exception handler which updates registers in
exception context and subsequent returns back into the function which
faulted, removing the need for fixups specialized to each faulting
instruction.
Other than backtracing, there should be no functional change as a result
of this patch.
------
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Follow arm64, riscv, and x86 and change extable layout to common
"relative table with data". This allows to get rid of s390 specific
code in sorttable.c.
The main difference to before is that extable entries do not contain a
relative function pointer anymore. Instead data and type fields are
added.
The type field is used to indicate which exception handler needs to be
called, while the data field is currently unused.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Add and use fixup_exception helper function in order to remove the
duplicated exception handler fixup code at several places.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Pass pt_regs to early program check handler like it is done for every
other interrupt and exception handler.
Also the passed pt_regs can be changed by the called function and the
changes register contents and psw contents will be taken into account
when returning. In addition the return psw will not be copied to the
program check old psw in lowcore, but to the usual return psw
location, like it is also done by the regular program check handler.
This allows also to get rid of the code that disabled lowcore
protection when changing the return address.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Just like arm64, riscv, and x86 move extable related functions to
mm/extable.c. This is currently only one function, but this will
change with subsequent changes.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Follow arm64 and riscv and move the EX_TABLE define to asm-extable.h
which is a lot less generic than the current linkage.h.
Also make sure that all files which contain EX_TABLE usages actually
include the new header file. This should make sure that the files
always compile and there won't be any random compile breakage due to
other header file dependencies.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
It is very unlikely that an exception happens within the amode31 text
section, therefore safe a couple of cycles for the common case, and
search the amode31 extable last.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
The early program check handler is active before the amode31 extable
is sorted. Therefore in case a program check happens early within the
amode31 code the extable entry might not be found.
Fix this by sorting the amode31 extable early.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
This patch adds CEX8 exploitation support for the AP bus code,
the zcrypt device driver zoo and the vfio device driver.
Signed-off-by: Harald Freudenberger <freude@linux.ibm.com>
Reviewed-by: Jürgen Christ <jchrist@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Disallow constructs like this:
pte_val(*pte) = __pa(addr) | prot;
which would directly write into a page table. Users are supposed to
use the set_pte()/set_pXd() primitives, which guarantee block
concurrent (aka atomic) writes.
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert pgtable code so pte_val()/pXd_val() aren't used as lvalue
anymore. This allows in later step to convert pte_val()/pXd_val() to
functions, which in turn makes it impossible to use these macros to
modify page table entries like they have been used before.
Therefore a construct like this:
pte_val(*pte) = __pa(addr) | prot;
which would directly write into a page table, isn't possible anymore
with the last step of this series.
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert pgtable code so pte_val()/pXd_val() aren't used as lvalue
anymore. This allows in later step to convert pte_val()/pXd_val() to
functions, which in turn makes it impossible to use these macros to
modify page table entries like they have been used before.
Therefore a construct like this:
pte_val(*pte) = __pa(addr) | prot;
which would directly write into a page table, isn't possible anymore
with the last step of this series.
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert pgtable code so pte_val()/pXd_val() aren't used as lvalue
anymore. This allows in later step to convert pte_val()/pXd_val() to
functions, which in turn makes it impossible to use these macros to
modify page table entries like they have been used before.
Therefore a construct like this:
pte_val(*pte) = __pa(addr) | prot;
which would directly write into a page table, isn't possible anymore
with the last step of this series.
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert pgtable code so pte_val()/pXd_val() aren't used as lvalue
anymore. This allows in later step to convert pte_val()/pXd_val() to
functions, which in turn makes it impossible to use these macros to
modify page table entries like they have been used before.
Therefore a construct like this:
pte_val(*pte) = __pa(addr) | prot;
which would directly write into a page table, isn't possible anymore
with the last step of this series.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use the new set_pXd()/set_pte() helper functions at all places where
page table entries are modified.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Add set_pte_bit()/clear_pte_bit() and set_pXd_bit()/clear_pXd_bit
helper functions which are supposed to be used if bits within
ptes/pXds are set/cleared.
The only point of these helper functions is to get more readable
code. This is quite similar to what arm64 has.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Add set_pXd()/set_pte() helper functions which must be used to update
page table entries. The new helpers use WRITE_ONCE() to make sure that
a page table entry is written to only once.
Without this the compiler could otherwise generate code which writes
several times to a page table entry when updating its contents from
invalid to valid, which could lead to surprising results especially
for multithreaded processes...
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Finally use epsw to create a complete psw mask within pt_regs. Without
this only some bits are correct, while other bits are (incorrectly)
always zero.
The epsw instruction is quite heavy weight, however given that this
only effects ftrace_regs_caller this seems to be the right thing, so
we finally get a complete psw mask for ftrace kprobed functions.
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Acked-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Remove opencoded offsetof and use offsetof instead.
The generated code is identical before/after this change.
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
With commit 5789284710 ("s390/smp: reallocate IPL CPU lowcore")
virtual addresses are wrongly passed to memblock_free_late() and
SPX instructions on IPL CPU reinitialization.
Note: this does not fix a bug currently, since virtual and
physical addresses are identical.
Fixes: 5789284710 ("s390/smp: reallocate IPL CPU lowcore")
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Running kprobe test on a kernel built with clang 14 didn't actually
trigger pgm_pre_handler() and no unwinder code was called. Even though
do_report_trap() is a global symbol, clang inlined it in several local
functions including illegal_op() handler, so that kprobbing a global
symbol didn't have a desired effect.
To achieve the same test result (unwinding from a program check
handler) introduce a local function and probe an instruction in the
middle, so that kprobe doesn't take KPROBE_ON_FTRACE path.
While at it, add another test for KPROBE_ON_FTRACE.
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
By default no backtraces are printed when a test succeeds, but sometimes
it is useful to spot issues automated test doesn't cover. Add "backtrace"
module parameter to force it.
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
-mpacked-stack option has been supported by both minimum
gcc and clang versions for a while. With commit e2bc3e91d9
("scripts/min-tool-version.sh: Raise minimum clang version to 13.0.0
for s390") minimum clang version now also supports a combination
of flags -mpacked-stack -mbackchain -pg -mfentry and fulfills
all requirements to always enable the packed stack layout.
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
This helps to avoid several merge conflicts later.
* fixes:
s390/extable: fix exception table sorting
s390/ftrace: fix arch_ftrace_get_regs implementation
s390/ftrace: fix ftrace_caller/ftrace_regs_caller generation
s390/setup: preserve memory at OLDMEM_BASE and OLDMEM_SIZE
s390/cio: verify the driver availability for path_event call
s390/module: fix building test_modules_helpers.o with clang
MAINTAINERS: downgrade myself to Reviewer for s390
MAINTAINERS: add Alexander Gordeev as maintainer for s390
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
s390 has a swap_ex_entry_fixup function, however it is not being used
since common code expects a swap_ex_entry_fixup define. If it is not
defined the default implementation will be used. So fix this by adding
a proper define.
However also the implementation of the function must be fixed, since a
NULL value for handler has a special meaning and must not be adjusted.
Luckily all of this doesn't fix a real bug currently: the main extable
is correctly sorted during build time, and for runtime sorting there
is currently no case where the handler field is not NULL.
Fixes: 05a68e892e ("s390/kernel: expand exception table logic to allow new handling options")
Acked-by: Ilya Leoshkevich <iii@linux.ibm.com>
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
arch_ftrace_get_regs is supposed to return a struct pt_regs pointer
only if the pt_regs structure contains all register contents, which
means it must have been populated when created via ftrace_regs_caller.
If it was populated via ftrace_caller the contents are not complete
(the psw mask part is missing), and therefore a NULL pointer needs be
returned.
The current code incorrectly always returns a struct pt_regs pointer.
Fix this by adding another pt_regs flag which indicates if the
contents are complete, and fix arch_ftrace_get_regs accordingly.
Fixes: 894979689d ("s390/ftrace: provide separate ftrace_caller/ftrace_regs_caller implementations")
Reported-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Reported-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Acked-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
ftrace_caller was used for both ftrace_caller and ftrace_regs_caller,
which means that the target address of the hotpatch trampoline was
never updated.
With commit 894979689d ("s390/ftrace: provide separate
ftrace_caller/ftrace_regs_caller implementations") a separate
ftrace_regs_caller entry point was implemeted, however it was
forgotten to implement the necessary changes for ftrace_modify_call
and ftrace_make_call, where the branch target has to be modified
accordingly.
Therefore add the missing code now.
Fixes: 894979689d ("s390/ftrace: provide separate ftrace_caller/ftrace_regs_caller implementations")
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Acked-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
We need to preserve the values at OLDMEM_BASE and OLDMEM_SIZE which are
used by zgetdump in case when kdump crashes. In that case zgetdump will
attempt to read OLDMEM_BASE and OLDMEM_SIZE in order to find out where
the memory range [0 - OLDMEM_SIZE] belonging to the production kernel is.
Fixes: f1a5469474 ("s390/setup: don't reserve memory that occupied decompressor's head")
Cc: stable@vger.kernel.org # 5.15+
Signed-off-by: Alexander Egorenkov <egorenar@linux.ibm.com>
Acked-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use CRST_ALLOC_ORDER to make it more obvious what the order means,
and also to be consistent with other code, e.g. the vmemmap code.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
There is a confusion with regard to the source address of
memcpy_real() and calling functions. While the declared
type for a source assumes a virtual address, in fact it
always called with physical address of the source.
This confusion led to bugs in copy_oldmem_kernel() and
copy_oldmem_user() functions, where __pa() macro applied
mistakenly to physical addresses. It does not lead to a
real issue, since virtual and physical addresses are
currently the same.
Fix both the bugs and memcpy_real() prototype by making
type of source address consistent to the function name
and the way it actually used.
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Virtual addresses of vmcore_info and os_info members are
wrongly passed to copy_oldmem_kernel(), while the function
expects physical address of the source. Instead, __pa()
macro should have been applied.
Yet, use of __pa() macro could be somehow confusing, since
copy_oldmem_kernel() may treat the source as an offset, not
as a direct physical address (that depens from the oldmem
availability and location).
Fix the virtual vs physical address confusion and make the
way the old lowcore is read consistent across all sources.
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
It is quite pointless to use memcpy to copy two bytes, besides that
this construct will also partially remove type and size sanity checks.
Therefore simply use an assignment.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Due to historical reasons os_info handling functions misuse
the notion of physical vs virtual addresses difference.
Note: this does not fix a bug currently, since virtual
and physical addresses are identical.
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Due to historical reasons memcpy_absolute() and friend functions
misuse the notion of physical vs virtual addresses difference.
Note: this does not fix a bug currently, since virtual and physical
addresses are identical.
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
