korg/linux
0
0
Fork 0
mirror of https://mirrors.bfsu.edu.cn/git/linux.git synced 2024-11-15 00:04:15 +08:00
Code Issues Packages Projects Releases Wiki Activity
edd898181e
linux/mm
History
Uladzislau Rezki (Sony) edd898181e mm: vmalloc: avoid calling __find_vmap_area() twice in __vunmap() 
Currently the __vunmap() path calls __find_vmap_area() twice.  Once on
entry to check that the area exists, then inside the remove_vm_area()
function which also performs a new search for the VA.

In order to improvie it from a performance point of view we split
remove_vm_area() into two new parts:
  - find_unlink_vmap_area() that does a search and unlink from tree;
  - __remove_vm_area() that removes without searching.

In this case there is no any functional change for remove_vm_area()
whereas vm_remove_mappings(), where a second search happens, switches to
the __remove_vm_area() variant where the already detached VA is passed as
a parameter, so there is no need to find it again.

Performance wise, i use test_vmalloc.sh with 32 threads doing alloc
free on a 64-CPUs-x86_64-box:

perf without this patch:
-   31.41%     0.50%  vmalloc_test/10  [kernel.vmlinux]    [k] __vunmap
   - 30.92% __vunmap
      - 17.67% _raw_spin_lock
           native_queued_spin_lock_slowpath
      - 12.33% remove_vm_area
         - 11.79% free_vmap_area_noflush
            - 11.18% _raw_spin_lock
                 native_queued_spin_lock_slowpath
        0.76% free_unref_page

perf with this patch:
-   11.35%     0.13%  vmalloc_test/14  [kernel.vmlinux]    [k] __vunmap
   - 11.23% __vunmap
      - 8.28% find_unlink_vmap_area
         - 7.95% _raw_spin_lock
              7.44% native_queued_spin_lock_slowpath
      - 1.93% free_vmap_area_noflush
         - 0.56% _raw_spin_lock
              0.53% native_queued_spin_lock_slowpath
        0.60% __vunmap_range_noflush

__vunmap() consumes around ~20% less CPU cycles on this test.

Also, switch from find_vmap_area() to find_unlink_vmap_area() to prevent a
double access to the vmap_area_lock: one for finding area, second time is
for unlinking from a tree.

[urezki@gmail.com: switch to find_unlink_vmap_area() in vm_unmap_ram()]
  Link: https://lkml.kernel.org/r/20221222190022.134380-2-urezki@gmail.com
Link: https://lkml.kernel.org/r/20221222190022.134380-1-urezki@gmail.com
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reported-by: Roman Gushchin <roman.gushchin@linux.dev>
Reviewed-by: Lorenzo Stoakes <lstoakes@gmail.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Oleksiy Avramchenko <oleksiy.avramchenko@sony.com>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-01-18 17:12:48 -08:00
..
damon mm/swap: convert deactivate_page() to folio_deactivate() 2023-01-18 17:12:47 -08:00
kasan kasan: allow sampling page_alloc allocations for HW_TAGS 2023-01-18 17:12:45 -08:00
kfence hardening updates for v6.2-rc1 2022-12-14 12:20:00 -08:00
kmsan kmsan: export kmsan_handle_urb 2022-12-21 14:31:52 -08:00
backing-dev.c mm: add /sys/class/bdi/<bdi>/min_ratio_fine knob 2022-11-30 15:59:06 -08:00
balloon_compaction.c mm: Convert all PageMovable users to movable_operations 2022-08-02 12:34:03 -04:00
bootmem_info.c bootmem: remove the vmemmap pages from kmemleak in put_page_bootmem 2022-08-28 14:02:45 -07:00
cma_debug.c mm/cma_debug: show complete cma name in debugfs directories 2022-09-11 20:25:50 -07:00
cma_sysfs.c
cma.c cma: tracing: print alloc result in trace_cma_alloc_finish 2023-01-18 17:12:41 -08:00
cma.h mm/cma: provide option to opt out from exposing pages on activation failure 2022-03-22 15:57:09 -07:00
compaction.c mm, compaction: fix fast_isolate_around() to stay within boundaries 2022-11-30 15:59:07 -08:00
debug_page_ref.c
debug_vm_pgtable.c mm: remove unused savedwrite infrastructure 2022-11-30 15:58:49 -08:00
debug.c mm,thp,rmap: simplify compound page mapcount handling 2022-11-30 15:58:46 -08:00
dmapool.c mm/dmapool.c: revert "make dma pool to use kmalloc_node" 2022-01-15 16:30:28 +02:00
early_ioremap.c mm/early_ioremap: declare early_memremap_pgprot_adjust() 2022-03-22 15:57:11 -07:00
fadvise.c mm/fadvise: use LLONG_MAX instead of -1 for eof 2022-12-11 18:12:11 -08:00
failslab.c mm: fix unexpected changes to {failslab|fail_page_alloc}.attr 2022-11-22 18:50:44 -08:00
filemap.c filemap: convert replace_page_cache_page() to replace_page_cache_folio() 2022-12-11 18:12:12 -08:00
folio-compat.c folio-compat: remove lru_cache_add() 2022-12-11 18:12:13 -08:00
frontswap.c frontswap: don't call ->init if no ops are registered 2022-09-26 12:14:34 -07:00
gup_test.c mm/gup_test: free memory allocated via kvcalloc() using kvfree() 2022-12-15 16:37:48 -08:00
gup_test.h mm/gup_test: start/stop/read functionality for PIN LONGTERM test 2022-11-08 17:37:15 -08:00
gup.c New Feature: 2022-12-17 14:06:53 -06:00
highmem.c highmem: fix kmap_to_page() for kmap_local_page() addresses 2022-10-12 18:51:51 -07:00
hmm.c mm/hugetlb: make walk_hugetlb_range() safe to pmd unshare 2023-01-18 17:12:39 -08:00
huge_memory.c mm: swap: convert mark_page_lazyfree() to folio_mark_lazyfree() 2023-01-18 17:12:42 -08:00
hugetlb_cgroup.c mm/hugeltb_cgroup: convert hugetlb_cgroup_commit_charge*() to folios 2022-11-30 15:58:43 -08:00
hugetlb_vmemmap.c mm/hugetlb_vmemmap: remap head page to newly allocated page 2022-11-30 15:58:47 -08:00
hugetlb_vmemmap.h mm: hugetlb_vmemmap: improve hugetlb_vmemmap code readability 2022-08-08 18:06:43 -07:00
hugetlb.c hugetlb: initialize variable to avoid compiler warning 2023-01-18 17:12:42 -08:00
hwpoison-inject.c mm/hwpoison: add __init/__exit annotations to module init/exit funcs 2022-10-03 14:03:05 -07:00
init-mm.c mm: remove rb tree. 2022-09-26 19:46:16 -07:00
internal.h mm: move folio_set_compound_order() to mm/internal.h 2023-01-18 17:12:36 -08:00
interval_tree.c
io-mapping.c
ioremap.c mm: ioremap: Add ioremap/iounmap_allowed() 2022-06-27 12:22:31 +01:00
Kconfig New Feature: 2022-12-17 14:06:53 -06:00
Kconfig.debug mm, slub: add CONFIG_SLUB_TINY 2022-11-27 23:38:02 +01:00
khugepaged.c mm/MADV_COLLAPSE: don't expand collapse when vm_end is past requested end 2023-01-11 16:14:20 -08:00
kmemleak.c mm/kmemleak: use %pK to display kernel pointers in backtrace 2022-12-15 16:37:49 -08:00
ksm.c mm/ksm: convert break_ksm() to use walk_page_range_vma() 2022-12-11 18:12:09 -08:00
list_lru.c mm: kmem: make mem_cgroup_from_obj() vmalloc()-safe 2022-06-16 19:48:31 -07:00
maccess.c maccess: Fix writing offset in case of fault in strncpy_from_kernel_nofault() 2022-11-11 11:44:46 -08:00
madvise.c mm/swap: convert deactivate_page() to folio_deactivate() 2023-01-18 17:12:47 -08:00
Makefile mm: memcontrol: drop dead CONFIG_MEMCG_SWAP config symbol 2022-10-03 14:03:36 -07:00
mapping_dirty_helpers.c mm: Rename pmd_read_atomic() 2022-12-15 10:37:27 -08:00
memblock.c mm: Always release pages to the buddy allocator in memblock_free_late(). 2023-01-08 18:49:33 +02:00
memcontrol.c mm: memcontrol: deprecate charge moving 2023-01-18 17:12:42 -08:00
memfd.c mm/memfd: add write seals when apply SEAL_EXEC to executable memfd 2023-01-18 17:12:37 -08:00
memory_hotplug.c mm: add pageblock_aligned() macro 2022-10-03 14:03:04 -07:00
memory-failure.c mm/memory-failure.c: cleanup in unpoison_memory 2022-11-30 15:59:08 -08:00
memory-tiers.c mm/demotion: fix NULL vs IS_ERR checking in memory_tier_init 2022-11-30 15:58:54 -08:00
memory.c mm/memory: add vm_normal_folio() 2023-01-18 17:12:47 -08:00
mempolicy.c mm/mempolicy: do not duplicate policy if it is not applicable for set_mempolicy_home_node 2023-01-18 17:12:41 -08:00
mempool.c mempool: do not use ksize() for poisoning 2022-11-30 15:58:41 -08:00
memremap.c mm/memremap.c: map FS_DAX device memory as decrypted 2022-11-08 15:57:23 -08:00
memtest.c
migrate_device.c mm/migrate_device: return number of migrating pages in args->cpages 2022-11-22 18:50:43 -08:00
migrate.c mm/hugetlb: move swap entry handling into vma lock when faulted 2023-01-18 17:12:38 -08:00
mincore.c mm: convert find_get_incore_page() to filemap_get_incore_folio() 2022-11-08 17:37:18 -08:00
mlock.c mm/mlock: drop dead code in count_mm_mlocked_page_nr() 2022-09-26 19:46:27 -07:00
mm_init.c memory: move hotplug memory notifier priority to same file for easy sorting 2022-11-08 17:37:17 -08:00
mm_slot.h mm: introduce common struct mm_slot 2022-10-03 14:02:43 -07:00
mmap_lock.c
mmap.c mm: update mmap_sem comments to refer to mmap_lock 2023-01-11 16:14:22 -08:00
mmu_gather.c mm: mmu_gather: allow more than one batch of delayed rmaps 2022-12-11 18:12:21 -08:00
mmu_notifier.c mm/mmu_notifier.c: fix race in mmu_interval_notifier_remove() 2022-04-21 20:01:10 -07:00
mmzone.c mm: multi-gen LRU: groundwork 2022-09-26 19:46:09 -07:00
mprotect.c mm/uffd: always wr-protect pte in pte|pmd_mkuffd_wp() 2023-01-18 17:12:37 -08:00
mremap.c mm, mremap: fix mremap() expanding vma with addr inside vma 2022-12-21 14:31:51 -08:00
msync.c mm/msync: use vma_find() instead of vma linked list 2022-09-26 19:46:25 -07:00
nommu.c nommu: fix split_vma() map_count error 2023-01-11 16:14:23 -08:00
oom_kill.c mm: reduce noise in show_mem for lowmem allocations 2022-09-26 19:46:29 -07:00
page_alloc.c kasan: allow sampling page_alloc allocations for HW_TAGS 2023-01-18 17:12:45 -08:00
page_counter.c mm: page_counter: remove unneeded atomic ops for low/min 2022-09-11 20:26:01 -07:00
page_ext.c Merge branch 'mm-hotfixes-stable' into mm-stable 2022-11-30 14:58:42 -08:00
page_idle.c mm: don't be stuck to rmap lock on reclaim path 2022-05-19 14:08:54 -07:00
page_io.c page_io: remove buffer_head include 2023-01-18 17:12:40 -08:00
page_isolation.c mm/page_isolation: fix clang deadcode warning 2022-10-28 13:37:22 -07:00
page_owner.c mm: reuse pageblock_start/end_pfn() macro 2022-10-03 14:03:03 -07:00
page_poison.c
page_reporting.c mm/page_reporting: Add checks for page_reporting_order param 2022-11-28 16:48:19 +00:00
page_reporting.h
page_table_check.c mm: use kstrtobool() instead of strtobool() 2022-11-30 15:58:45 -08:00
page_vma_mapped.c mm/hugetlb: introduce hugetlb_walk() 2023-01-18 17:12:39 -08:00
page-writeback.c mm/swap: convert deactivate_page() to folio_deactivate() 2023-01-18 17:12:47 -08:00
pagewalk.c mm/hugetlb: introduce hugetlb_walk() 2023-01-18 17:12:39 -08:00
percpu-internal.h percpu: improve percpu_alloc_percpu event trace 2022-05-13 07:20:18 -07:00
percpu-km.c
percpu-stats.c mm: use vmalloc_array and vcalloc for array allocations 2022-03-08 09:30:46 -05:00
percpu-vm.c
percpu.c mm/percpu.c: remove the lcm code since block size is fixed at page size 2022-11-07 22:59:18 -08:00
pgalloc-track.h
pgtable-generic.c mm: avoid unnecessary flush on change_huge_pmd() 2022-05-13 07:20:05 -07:00
process_vm_access.c use less confusing names for iov_iter direction initializers 2022-11-25 13:01:55 -05:00
ptdump.c mm: pagewalk: Fix race between unmap and page walker 2022-09-03 10:13:13 -07:00
readahead.c mm: add PSI accounting around ->read_folio and ->readahead calls 2022-09-20 08:24:38 -06:00
rmap.c mm: rmap: remove lock_page_memcg() 2023-01-18 17:12:42 -08:00
rodata_test.c mm/rodata_test: use PAGE_ALIGNED() helper 2022-10-03 14:03:05 -07:00
secretmem.c mm/secretmem: remove reduntant return value 2022-10-03 14:03:36 -07:00
shmem.c swap: avoid holding swap reference in swap_cache_get_folio 2023-01-18 17:12:45 -08:00
shrinker_debug.c mm: shrinkers: fix double kfree on shrinker name 2022-07-29 18:07:13 -07:00
shuffle.c mm/shuffle: convert module_param_call to module_param_cb 2022-10-03 14:03:07 -07:00
shuffle.h
slab_common.c hardening updates for v6.2-rc1 2022-12-14 12:20:00 -08:00
slab.c Merge branch 'slab/for-6.2/kmalloc_redzone' into slab/for-next 2022-11-21 10:36:09 +01:00
slab.h Merge branch 'slub-tiny-v1r6' into slab/for-next 2022-12-01 00:14:00 +01:00
slob.c Merge branch 'slab/for-6.1/kmalloc_size_roundup' into slab/for-next 2022-09-29 11:30:55 +02:00
slub.c MM patches for 6.2-rc1. 2022-12-13 19:29:45 -08:00
sparse-vmemmap.c mm/sparse-vmemmap: generalise vmemmap_populate_hugepages() 2022-12-11 18:12:12 -08:00
sparse.c mm/hwpoison: introduce per-memory_block hwpoison counter 2022-11-08 17:37:22 -08:00
swap_cgroup.c mm: memcontrol: don't allocate cgroup swap arrays when memcg is disabled 2022-10-03 14:03:36 -07:00
swap_slots.c mm/swap: convert put_swap_page() to put_swap_folio() 2022-10-03 14:02:46 -07:00
swap_state.c swap: avoid holding swap reference in swap_cache_get_folio 2023-01-18 17:12:45 -08:00
swap.c mm/swap: convert deactivate_page() to folio_deactivate() 2023-01-18 17:12:47 -08:00
swap.h mm: convert find_get_incore_page() to filemap_get_incore_folio() 2022-11-08 17:37:18 -08:00
swapfile.c swapfile: get rid of volatile and avoid redundant read 2023-01-18 17:12:44 -08:00
truncate.c folio-compat: remove lru_cache_add() 2022-12-11 18:12:13 -08:00
usercopy.c mm: use kstrtobool() instead of strtobool() 2022-11-30 15:58:45 -08:00
userfaultfd.c mm/uffd: always wr-protect pte in pte|pmd_mkuffd_wp() 2023-01-18 17:12:37 -08:00
util.c mm: new primitive kvmemdup() 2023-01-18 17:12:47 -08:00
vmalloc.c mm: vmalloc: avoid calling __find_vmap_area() twice in __vunmap() 2023-01-18 17:12:48 -08:00
vmpressure.c
vmscan.c mm/swap: convert deactivate_page() to folio_deactivate() 2023-01-18 17:12:47 -08:00
vmstat.c mm: vmscan: split khugepaged stats from direct reclaim stats 2022-11-30 15:58:41 -08:00
workingset.c folio-compat: remove lru_cache_add() 2022-12-11 18:12:13 -08:00
z3fold.c zpool: clean out dead code 2022-12-11 18:12:10 -08:00
zbud.c zpool: clean out dead code 2022-12-11 18:12:10 -08:00
zpool.c zpool: clean out dead code 2022-12-11 18:12:10 -08:00
zsmalloc.c zsmalloc: implement writeback mechanism for zsmalloc 2022-12-11 18:12:10 -08:00
zswap.c zswap: fix writeback lock ordering for zsmalloc 2022-12-11 18:12:09 -08:00