linux/arch/riscv/kernel
Palmer Dabbelt 77eea559ba
Merge patch series "bpf, riscv: use BPF prog pack allocator in BPF JIT"
Puranjay Mohan <puranjay12@gmail.com> says:

Here is some data to prove the V2 fixes the problem:

Without this series:
root@rv-selftester:~/src/kselftest/bpf# time ./test_tag
test_tag: OK (40945 tests)

real    7m47.562s
user    0m24.145s
sys     6m37.064s

With this series applied:
root@rv-selftester:~/src/selftest/bpf# time ./test_tag
test_tag: OK (40945 tests)

real    7m29.472s
user    0m25.865s
sys     6m18.401s

BPF programs currently consume a page each on RISCV. For systems with many BPF
programs, this adds significant pressure to instruction TLB. High iTLB pressure
usually causes slow down for the whole system.

Song Liu introduced the BPF prog pack allocator[1] to mitigate the above issue.
It packs multiple BPF programs into a single huge page. It is currently only
enabled for the x86_64 BPF JIT.

I enabled this allocator on the ARM64 BPF JIT[2]. It is being reviewed now.

This patch series enables the BPF prog pack allocator for the RISCV BPF JIT.

======================================================
Performance Analysis of prog pack allocator on RISCV64
======================================================

Test setup:
===========

Host machine: Debian GNU/Linux 11 (bullseye)
Qemu Version: QEMU emulator version 8.0.3 (Debian 1:8.0.3+dfsg-1)
u-boot-qemu Version: 2023.07+dfsg-1
opensbi Version: 1.3-1

To test the performance of the BPF prog pack allocator on RV, a stresser
tool[4] linked below was built. This tool loads 8 BPF programs on the system and
triggers 5 of them in an infinite loop by doing system calls.

The runner script starts 20 instances of the above which loads 8*20=160 BPF
programs on the system, 5*20=100 of which are being constantly triggered.
The script is passed a command which would be run in the above environment.

The script was run with following perf command:
./run.sh "perf stat -a \
        -e iTLB-load-misses \
        -e dTLB-load-misses  \
        -e dTLB-store-misses \
        -e instructions \
        --timeout 60000"

The output of the above command is discussed below before and after enabling the
BPF prog pack allocator.

The tests were run on qemu-system-riscv64 with 8 cpus, 16G memory. The rootfs
was created using Bjorn's riscv-cross-builder[5] docker container linked below.

Results
=======

Before enabling prog pack allocator:
------------------------------------

Performance counter stats for 'system wide':

           4939048      iTLB-load-misses
           5468689      dTLB-load-misses
            465234      dTLB-store-misses
     1441082097998      instructions

      60.045791200 seconds time elapsed

After enabling prog pack allocator:
-----------------------------------

Performance counter stats for 'system wide':

           3430035      iTLB-load-misses
           5008745      dTLB-load-misses
            409944      dTLB-store-misses
     1441535637988      instructions

      60.046296600 seconds time elapsed

Improvements in metrics
=======================

It was expected that the iTLB-load-misses would decrease as now a single huge
page is used to keep all the BPF programs compared to a single page for each
program earlier.

--------------------------------------------
The improvement in iTLB-load-misses: -30.5 %
--------------------------------------------

I repeated this expriment more than 100 times in different setups and the
improvement was always greater than 30%.

This patch series is boot tested on the Starfive VisionFive 2 board[6].
The performance analysis was not done on the board because it doesn't
expose iTLB-load-misses, etc. The stresser program was run on the board to test
the loading and unloading of BPF programs

[1] https://lore.kernel.org/bpf/20220204185742.271030-1-song@kernel.org/
[2] https://lore.kernel.org/all/20230626085811.3192402-1-puranjay12@gmail.com/
[3] https://lore.kernel.org/all/20230626085811.3192402-2-puranjay12@gmail.com/
[4] https://github.com/puranjaymohan/BPF-Allocator-Bench
[5] https://github.com/bjoto/riscv-cross-builder
[6] https://www.starfivetech.com/en/site/boards

* b4-shazam-merge:
  bpf, riscv: use prog pack allocator in the BPF JIT
  riscv: implement a memset like function for text
  riscv: extend patch_text_nosync() for multiple pages
  bpf: make bpf_prog_pack allocator portable

Link: https://lore.kernel.org/r/20230831131229.497941-1-puranjay12@gmail.com
Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-09-08 11:25:25 -07:00
..
compat_vdso riscv: Handle zicsr/zifencei issue between gcc and binutils 2023-08-16 07:39:38 -07:00
pi riscv: Introduce virtual kernel mapping KASLR 2023-09-05 19:49:27 -07:00
probes riscv: kprobes: simulate c.beqz and c.bnez 2023-08-16 07:48:40 -07:00
vdso riscv: replace deprecated scall with ecall 2023-06-20 09:02:09 -07:00
.gitignore .gitignore: add SPDX License Identifier 2020-03-25 11:50:48 +01:00
acpi.c RISC-V: ACPI: Fix acpi_os_ioremap to return iomem address 2023-08-02 13:49:43 -07:00
alternative.c Merge patch series "Add non-coherent DMA support for AX45MP" 2023-09-08 11:24:34 -07:00
asm-offsets.c RISC-V: Add arch functions to support hibernation/suspend-to-disk 2023-04-29 11:25:13 -07:00
cacheinfo.c RISC-V Patches for the 6.4 Merge Window, Part 1 2023-04-28 16:55:39 -07:00
cfi.c riscv: Add CFI error handling 2023-08-23 14:16:39 -07:00
compat_signal.c riscv: compat: signal: Add rt_frame implementation 2022-05-17 16:37:21 -07:00
compat_syscall_table.c riscv: Implement syscall wrappers 2023-08-23 14:16:36 -07:00
copy-unaligned.h RISC-V: Probe for unaligned access speed 2023-09-01 09:06:25 -07:00
copy-unaligned.S RISC-V: Probe for unaligned access speed 2023-09-01 09:06:25 -07:00
cpu_ops_sbi.c riscv: cpu_ops_sbi: Add 64bit hartid support on RV64 2022-07-19 16:38:58 -07:00
cpu_ops_spinwait.c RISC-V: cpu_ops_spinwait.c should include head.h 2022-08-11 13:24:16 -07:00
cpu_ops.c RISC-V: Align SBI probe implementation with spec 2023-04-29 13:04:50 -07:00
cpu-hotplug.c riscv: Switch to hotplug core state synchronization 2023-05-15 13:44:59 +02:00
cpu.c RISC-V Patches for the 6.6 Merge Window, Part 1 2023-09-01 08:09:48 -07:00
cpufeature.c Merge patch series "RISC-V: Probe for misaligned access speed" 2023-09-08 11:24:12 -07:00
crash_core.c riscv: Export va_kernel_pa_offset in vmcoreinfo 2023-08-02 13:50:31 -07:00
crash_dump.c vmcore: convert copy_oldmem_page() to take an iov_iter 2022-04-29 14:37:59 -07:00
crash_save_regs.S RISC-V: Fixup get incorrect user mode PC for kernel mode regs 2022-08-11 08:54:40 -07:00
efi-header.S riscv: Prepare EFI header for relocatable kernels 2023-04-19 07:46:28 -07:00
efi.c efi: Discover BTI support in runtime services regions 2023-02-04 09:19:02 +01:00
elf_kexec.c - An extensive rework of kexec and crash Kconfig from Eric DeVolder 2023-08-29 14:53:51 -07:00
entry.S riscv: replace deprecated scall with ecall 2023-06-20 09:02:09 -07:00
fpu.S riscv: abstract out CSR names for supervisor vs machine mode 2019-11-05 09:20:42 -08:00
ftrace.c RISC-V: Don't check text_mutex during stop_machine 2023-03-09 14:58:51 -08:00
head.h riscv: entry: Convert to generic entry 2023-03-23 08:47:00 -07:00
head.S riscv: remove redundant mv instructions 2023-08-31 00:18:25 -07:00
hibernate-asm.S riscv: hibernation: Remove duplicate call of suspend_restore_csrs 2023-06-19 09:27:57 -07:00
hibernate.c riscv: hibernate: remove WARN_ON in save_processor_state 2023-06-23 10:06:22 -07:00
image-vars.h riscv: libstub: Implement KASLR by using generic functions 2023-09-05 19:49:31 -07:00
irq.c riscv: stack: Fixup independent softirq stack for CONFIG_FRAME_POINTER=n 2023-08-16 07:26:29 -07:00
jump_label.c jump_label: make initial NOP patching the special case 2022-06-24 09:48:55 +02:00
kexec_relocate.S riscv: Don't use va_pa_offset on kdump 2022-01-09 10:38:06 -08:00
kgdb.c RISC-V: rename parse_asm.h to insn.h 2022-12-29 06:59:47 -08:00
machine_kexec_file.c RISC-V: Add kexec_file support 2022-05-19 12:14:18 -07:00
machine_kexec.c riscv: kexec: Fixup crash_smp_send_stop without multi cores 2022-11-29 21:50:59 -08:00
Makefile Merge patch series "RISC-V: Probe for misaligned access speed" 2023-09-08 11:24:12 -07:00
mcount-dyn.S riscv: entry: Consolidate general regs saving/restoring 2023-03-23 08:47:03 -07:00
mcount.S riscv: Add ftrace_stub_graph 2023-08-23 14:16:38 -07:00
module-sections.c riscv: add missing header file includes 2019-10-28 00:46:01 -07:00
module.c riscv: module: Add ADD16 and SUB16 rela types 2023-01-31 23:29:40 -08:00
patch.c riscv: implement a memset like function for text 2023-09-06 06:26:06 -07:00
perf_callchain.c riscv: Fix fill_callchain return value 2022-03-30 23:01:42 -07:00
perf_regs.c perf/arch: Remove perf_sample_data::regs_user_copy 2020-11-09 18:12:34 +01:00
process.c riscv: Add prctl controls for userspace vector management 2023-06-08 07:16:53 -07:00
ptrace.c RISC-V: Add ptrace support for vectors 2023-09-01 13:05:38 -07:00
reset.c riscv: Use do_kernel_power_off() 2022-05-19 19:30:30 +02:00
riscv_ksyms.c RISC-V: add infrastructure to allow different str* implementations 2023-01-31 11:43:23 -08:00
sbi-ipi.c RISC-V: Allow marking IPIs as suitable for remote FENCEs 2023-04-08 11:26:24 +01:00
sbi.c RISC-V: Align SBI probe implementation with spec 2023-04-29 13:04:50 -07:00
setup.c Merge patch series "riscv: Introduce KASLR" 2023-09-08 11:25:13 -07:00
signal.c riscv: signal: validate altstack to reflect Vector 2023-06-08 07:16:48 -07:00
smp.c riscv: Fix CPU feature detection with SMP disabled 2023-08-08 15:28:25 -07:00
smpboot.c RISC-V: alternative: Remove feature_probe_func 2023-09-01 09:06:26 -07:00
soc.c riscv: Fix builtin DTB handling 2021-01-07 19:00:50 -08:00
stacktrace.c riscv: Use READ_ONCE_NOCHECK in imprecise unwinding stack mode 2023-03-09 14:50:35 -08:00
suspend_entry.S riscv: Add types to indirectly called assembly functions 2023-08-23 14:16:37 -07:00
suspend.c RISC-V: Change suspend_save_csrs and suspend_restore_csrs to public function 2023-04-29 11:25:10 -07:00
sys_riscv.c riscv: Implement syscall wrappers 2023-08-23 14:16:36 -07:00
syscall_table.c riscv: Implement syscall wrappers 2023-08-23 14:16:36 -07:00
time.c RISC-V: time.c: Add ACPI support for time_init() 2023-06-01 08:45:13 -07:00
traps_misaligned.c riscv: traps_misaligned: do not duplicate stringify 2022-08-11 08:56:53 -07:00
traps.c RISC-V Patches for the 6.6 Merge Window, Part 1 2023-09-01 08:09:48 -07:00
vdso.c riscv: vdso: include vdso/vsyscall.h for vdso_data 2023-07-04 07:54:41 -07:00
vector.c riscv: vector: clear V-reg in the first-use trap 2023-07-01 07:38:21 -07:00
vmlinux-xip.lds.S riscv: vmlinux-xip.lds.S: remove .alternative section 2023-06-25 16:24:03 -07:00
vmlinux.lds.S riscv: enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION 2023-06-25 16:24:05 -07:00