korg/linux
0
0
Fork 0
mirror of https://mirrors.bfsu.edu.cn/git/linux.git synced 2025-01-25 23:34:47 +08:00
Code Issues Packages Projects Releases Wiki Activity
46c22e7b09
linux/mm
History
Yang Shi 46c22e7b09 mm: gup: fix the fast GUP race against THP collapse 
commit 70cbc3cc78 upstream.

Since general RCU GUP fast was introduced in commit 2667f50e8b ("mm:
introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer
sufficient to handle concurrent GUP-fast in all cases, it only handles
traditional IPI-based GUP-fast correctly.  On architectures that send an
IPI broadcast on TLB flush, it works as expected.  But on the
architectures that do not use IPI to broadcast TLB flush, it may have the
below race:

   CPU A                                          CPU B
THP collapse                                     fast GUP
                                              gup_pmd_range() <-- see valid pmd
                                                  gup_pte_range() <-- work on pte
pmdp_collapse_flush() <-- clear pmd and flush
__collapse_huge_page_isolate()
    check page pinned <-- before GUP bump refcount
                                                      pin the page
                                                      check PTE <-- no change
__collapse_huge_page_copy()
    copy data to huge page
    ptep_clear()
install huge pmd for the huge page
                                                      return the stale page
discard the stale page

The race can be fixed by checking whether PMD is changed or not after
taking the page pin in fast GUP, just like what it does for PTE.  If the
PMD is changed it means there may be parallel THP collapse, so GUP should
back off.

Also update the stale comment about serializing against fast GUP in
khugepaged.

Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com
Fixes: 2667f50e8b ("mm: introduce a general RCU get_user_pages_fast()")
Acked-by: David Hildenbrand <david@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Yang Shi <shy828301@gmail.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2022-10-12 09:53:26 +02:00
..
damon mm/damon/dbgfs: fix memory leak when using debugfs_lookup() 2022-10-05 10:39:37 +02:00
kasan kasan: prevent cpu_quarantine corruption when CPU offline and cache shrink occur at same time 2022-05-09 09:14:41 +02:00
kfence mm/kfence: reset PG_slab and memcg_data before freeing __kfence_pool 2022-05-25 09:57:23 +02:00
backing-dev.c writeback: avoid use-after-free after removing device 2022-08-31 17:16:47 +02:00
balloon_compaction.c mm: fix typos in comments 2021-05-07 00:26:35 -07:00
bootmem_info.c bootmem: remove the vmemmap pages from kmemleak in put_page_bootmem 2022-08-31 17:16:48 +02:00
cleancache.c
cma_debug.c mm/cma: change cma mutex to irq safe spinlock 2021-05-05 11:27:21 -07:00
cma_sysfs.c mm: cma: support sysfs 2021-05-05 11:27:24 -07:00
cma.c Revert "mm/cma.c: remove redundant cma_mutex lock" 2022-06-09 10:23:27 +02:00
cma.h mm: cma: support sysfs 2021-05-05 11:27:24 -07:00
compaction.c mm, compaction: fast_find_migrateblock() should return pfn in the target zone 2022-06-09 10:23:21 +02:00
debug_page_ref.c
debug_vm_pgtable.c mm/debug_vm_pgtable: remove pte entry from the page table 2022-02-08 18:34:05 +01:00
debug.c mm/debug: sync up latest migrate_reason to migrate_reason_names 2021-09-24 16:13:35 -07:00
dmapool.c mm/dmapool: use DEVICE_ATTR_RO macro 2021-06-29 10:53:52 -07:00
early_ioremap.c mm/early_ioremap.c: remove redundant early_ioremap_shutdown() 2021-09-08 11:50:24 -07:00
fadvise.c
failslab.c
filemap.c mm/filemap: fix UAF in find_lock_entries 2022-07-12 16:34:47 +02:00
frontswap.c mm/mempool: minor coding style tweaks 2021-05-05 11:27:27 -07:00
gup_test.c selftests/vm: gup_test: test faulting in kernel, and verify pinnable pages 2021-05-05 11:27:26 -07:00
gup_test.h selftests/vm: gup_test: fix test flag 2021-05-05 11:27:26 -07:00
gup.c mm: gup: fix the fast GUP race against THP collapse 2022-10-12 09:53:26 +02:00
highmem.c highmem: fix checks in __kmap_local_sched_{in,out} 2022-04-13 20:59:21 +02:00
hmm.c mm/hmm: fault non-owner device private entries 2022-08-03 12:03:54 +02:00
huge_memory.c mm/huge_memory: do not overkill when splitting huge_zero_page 2022-05-18 10:26:55 +02:00
hugetlb_cgroup.c hugetlb: make free_huge_page irq safe 2021-05-05 11:27:22 -07:00
hugetlb_vmemmap.c mm: hugetlb: introduce CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON 2021-06-30 20:47:26 -07:00
hugetlb_vmemmap.h mm: hugetlb: introduce nr_free_vmemmap_pages in the struct hstate 2021-06-30 20:47:25 -07:00
hugetlb.c mm/hugetlb: avoid corrupting page->mapping in hugetlb_mcopy_atomic_pte 2022-09-05 10:30:04 +02:00
hwpoison-inject.c mm/hwpoison: avoid the impact of hwpoison_filter() return value on mce handler 2022-07-12 16:35:05 +02:00
init-mm.c mm: add setup_initial_init_mm() helper 2021-07-08 11:48:21 -07:00
internal.h mm/numa: automatically generate node migration order 2021-09-03 09:58:16 -07:00
interval_tree.c mm/interval_tree: add comments to improve code readability 2021-04-30 11:20:38 -07:00
io-mapping.c mm: add a io_mapping_map_user helper 2021-04-30 11:20:39 -07:00
ioremap.c mm: move ioremap_page_range to vmalloc.c 2021-09-08 11:50:24 -07:00
Kconfig kmap_local: don't assume kmap PTEs are linear arrays in memory 2021-11-25 09:48:43 +01:00
Kconfig.debug
khugepaged.c mm: gup: fix the fast GUP race against THP collapse 2022-10-12 09:53:26 +02:00
kmemleak.c Revert "mm: kmemleak: take a full lowmem check in kmemleak_*_phys()" 2022-09-15 11:30:00 +02:00
ksm.c mm/ksm: remove old GCC 4.9+ check 2021-09-13 10:18:28 -07:00
list_lru.c mm: vmscan: consolidate shrinker_maps handling code 2021-05-05 11:27:23 -07:00
maccess.c ARM: 9115/1: mm/maccess: fix unaligned copy_{from,to}_kernel_nofault 2021-08-20 11:39:25 +01:00
madvise.c mm: fix madivse_pageout mishandling on non-LRU page 2022-10-05 10:39:39 +02:00
Makefile mm: introduce Data Access MONitor (DAMON) 2021-09-08 11:50:24 -07:00
mapping_dirty_helpers.c mm/mapping_dirty_helpers: remove double Note in kerneldoc 2021-07-01 11:06:02 -07:00
memblock.c memblock: use kfree() to release kmalloced memblock regions 2022-03-02 11:48:10 +01:00
memcontrol.c memcg: sync flush only if periodic flush is delayed 2022-04-27 14:38:57 +02:00
memfd.c memfd: fix F_SEAL_WRITE after shmem huge page allocated 2022-03-08 19:12:48 +01:00
memory_hotplug.c Merge branch 'akpm' (patches from Andrew) 2021-09-08 12:55:35 -07:00
memory-failure.c mm,hwpoison: check mm when killing accessing process 2022-10-05 10:39:39 +02:00
memory.c mm: fix page leak with multiple threads mapping the same page 2022-08-03 12:03:42 +02:00
mempolicy.c mm/mempolicy: fix get_nodes out of bound access 2022-08-17 14:23:47 +02:00
mempool.c kasan: use separate (un)poison implementation for integrated init 2021-06-04 19:32:21 +01:00
memremap.c mm/memremap: fix memunmap_pages() race with get_dev_pagemap() 2022-08-17 14:23:44 +02:00
memtest.c
migrate.c mm/migrate_device.c: flush TLB while holding PTL 2022-10-05 10:39:39 +02:00
mincore.c inode: make init and permission helpers idmapped mount aware 2021-01-24 14:27:16 +01:00
mlock.c mm/mlock: fix potential imbalanced rlimit ucounts adjustment 2022-05-15 20:18:53 +02:00
mm_init.c include/linux/page-flags-layout.h: cleanups 2021-04-30 11:20:42 -07:00
mmap_lock.c mm: mmap_lock: fix disabling preemption directly 2021-07-23 17:43:28 -07:00
mmap.c mm: Fix TLB flush for not-first PFNMAP mappings in unmap_region() 2022-09-20 12:39:45 +02:00
mmu_gather.c mm: eliminate "expecting prototype" kernel-doc warnings 2021-04-16 16:10:36 -07:00
mmu_notifier.c mm/mmu_notifier.c: fix race in mmu_interval_notifier_remove() 2022-04-27 14:38:58 +02:00
mmzone.c
mprotect.c mm: don't try to NUMA-migrate COW pages that have other uses 2022-02-23 12:03:03 +01:00
mremap.c mmmremap.c: avoid pointless invalidate_range_start/end on mremap(old_size=0) 2022-04-13 20:59:22 +02:00
msync.c mm/msync: exit early when the flags is an MS_ASYNC and start < vm_start 2021-04-30 11:20:37 -07:00
nommu.c Merge tag 'denywrite-for-5.15' of git://github.com/davidhildenbrand/linux 2021-09-04 11:35:47 -07:00
oom_kill.c oom_kill.c: futex: delay the OOM reaper to allow time for proper futex cleanup 2022-04-27 14:38:58 +02:00
page_alloc.c mm: prevent page_frag_alloc() from corrupting the memory 2022-10-05 10:39:39 +02:00
page_counter.c mm: page_counter: mitigate consequences of a page_counter underflow 2021-04-30 11:20:38 -07:00
page_ext.c mm/migrate: add CPU hotplug to demotion #ifdef 2021-10-18 20:22:02 -10:00
page_idle.c mm/idle_page_tracking: make PG_idle reusable 2021-09-08 11:50:24 -07:00
page_io.c mm: fix unexpected zeroed page mapping with zram swap 2022-04-20 09:34:18 +02:00
page_isolation.c Merge branch 'akpm' (patches from Andrew) 2021-09-08 12:55:35 -07:00
page_owner.c mm: remove pfn_valid_within() and CONFIG_HOLES_IN_ZONE 2021-09-08 11:50:22 -07:00
page_poison.c mm: page_poison: print page info when corruption is caught 2021-04-30 11:20:36 -07:00
page_reporting.c mm/page_reporting: allow driver to specify reporting order 2021-06-29 10:53:47 -07:00
page_reporting.h mm/page_reporting: export reporting order as module parameter 2021-06-29 10:53:47 -07:00
page_vma_mapped.c mm: device exclusive memory access 2021-07-01 11:06:03 -07:00
page-writeback.c writeback: avoid use-after-free after removing device 2022-08-31 17:16:47 +02:00
pagewalk.c mm: pagewalk: Fix race between unmap and page walker 2022-09-08 12:28:05 +02:00
percpu-internal.h Merge branch 'for-5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/dennis/percpu 2021-07-01 17:17:24 -07:00
percpu-km.c percpu: flush tlb in pcpu_reclaim_populated() 2021-07-04 18:30:17 +00:00
percpu-stats.c percpu: rework memcg accounting 2021-06-05 20:43:15 +00:00
percpu-vm.c percpu: flush tlb in pcpu_reclaim_populated() 2021-07-04 18:30:17 +00:00
percpu.c Merge branch 'akpm' (patches from Andrew) 2021-09-08 12:55:35 -07:00
pgalloc-track.h mm: fix typos in comments 2021-05-07 00:26:35 -07:00
pgtable-generic.c mm/thp: fix __split_huge_pmd_locked() on shmem migration entry 2021-06-16 09:24:42 -07:00
process_vm_access.c mm/process_vm_access.c: remove duplicate include 2021-05-05 11:27:27 -07:00
ptdump.c mm: pagewalk: Fix race between unmap and page walker 2022-09-08 12:28:05 +02:00
readahead.c mm: Protect operations adding pages to page cache with invalidate_lock 2021-07-13 13:14:27 +02:00
rmap.c mm/rmap: Fix anon_vma->degree ambiguity leading to double-reuse 2022-09-05 10:30:07 +02:00
rodata_test.c
secretmem.c mm: fix dereferencing possible ERR_PTR 2022-10-05 10:39:39 +02:00
shmem.c mm: shmem: fix missing cache flush in shmem_mfill_atomic_pte() 2022-05-15 20:18:53 +02:00
shuffle.c mm: eliminate "expecting prototype" kernel-doc warnings 2021-04-16 16:10:36 -07:00
shuffle.h mm/shuffle: fix section mismatch warning 2021-05-22 15:09:07 -10:00
slab_common.c mm, kfence: support kmem_dump_obj() for KFENCE objects 2022-04-27 14:38:51 +02:00
slab.c mm, kfence: support kmem_dump_obj() for KFENCE objects 2022-04-27 14:38:51 +02:00
slab.h mm, kfence: support kmem_dump_obj() for KFENCE objects 2022-04-27 14:38:51 +02:00
slob.c mm, kfence: support kmem_dump_obj() for KFENCE objects 2022-04-27 14:38:51 +02:00
slub.c mm: slub: fix flush_cpu_slab()/__free_slab() invocations in task context. 2022-09-28 11:11:44 +02:00
sparse-vmemmap.c mm: sparsemem: split the huge PMD mapping of vmemmap pages 2021-06-30 20:47:26 -07:00
sparse.c mm: introduce memmap_alloc() to unify memory map allocation 2021-09-03 09:58:15 -07:00
swap_cgroup.c
swap_slots.c mm: Replace deprecated CPU-hotplug functions. 2021-08-28 01:46:17 +02:00
swap_state.c mm: swap: get rid of livelock in swapin readahead 2022-03-23 09:16:41 +01:00
swap.c mm: fs: invalidate bh_lrus for only cold path 2021-09-24 16:13:35 -07:00
swapfile.c mm, memcg: inline swap-related functions to improve disabled memcg config 2021-09-03 09:58:12 -07:00
truncate.c Merge branch 'akpm' (patches from Andrew) 2021-09-03 10:08:28 -07:00
usercopy.c mm/usercopy: return 1 from hardened_usercopy __setup() handler 2022-04-08 14:24:14 +02:00
userfaultfd.c mm: userfaultfd: fix UFFDIO_CONTINUE on fallocated shmem pages 2022-07-21 21:24:11 +02:00
util.c mm: vmalloc: introduce array allocation functions 2022-07-12 16:35:01 +02:00
vmacache.c
vmalloc.c mm: defer kmemleak object creation of module_alloc() 2022-03-08 19:12:38 +01:00
vmpressure.c mm/vmpressure: replace vmpressure_to_css() with vmpressure_to_memcg() 2021-09-03 09:58:17 -07:00
vmscan.c mm,vmscan: fix divide by zero in get_scan_count 2021-09-08 18:45:53 -07:00
vmstat.c mm/vmstat: protect per cpu variables with preempt disable on RT 2021-09-08 15:32:34 -07:00
workingset.c memcg: sync flush only if periodic flush is delayed 2022-04-27 14:38:57 +02:00
z3fold.c mm/z3fold: add kerneldoc fields for z3fold_pool 2021-07-01 11:06:03 -07:00
zbud.c mm/zbud: add kerneldoc fields for zbud_pool 2021-07-01 11:06:03 -07:00
zpool.c mm: fix typos in comments 2021-05-07 00:26:35 -07:00
zsmalloc.c zsmalloc: fix races between asynchronous zspage free and page migration 2022-06-06 08:43:39 +02:00
zswap.c mm/zswap.c: fix two bugs in zswap_writeback_entry() 2021-06-30 20:47:31 -07:00