Johannes Weiner 55779ec759 mm: fix vm-scalability regression in cgroup-aware workingset code ... Commit 23047a96d7 ("mm: workingset: per-cgroup cache thrash detection") added a page->mem_cgroup lookup to the cache eviction, refault, and activation paths, as well as locking to the activation path, and the vm-scalability tests showed a regression of -23%. While the test in question is an artificial worst-case scenario that doesn't occur in real workloads - reading two sparse files in parallel at full CPU speed just to hammer the LRU paths - there is still some optimizations that can be done in those paths. Inline the lookup functions to eliminate calls. Also, page->mem_cgroup doesn't need to be stabilized when counting an activation; we merely need to hold the RCU lock to prevent the memcg from being freed. This cuts down on overhead quite a bit: 23047a96d7 063f6715e77a7be5770d6081fe ---------------- -------------------------- %stddev %change %stddev \ | \ 21621405 +- 0% +11.3% 24069657 +- 2% vm-scalability.throughput [linux@roeck-us.net: drop unnecessary include file] [hannes@cmpxchg.org: add WARN_ON_ONCE()s] Link: http://lkml.kernel.org/r/20160707194024.GA26580@cmpxchg.org Link: http://lkml.kernel.org/r/20160624175101.GA3024@cmpxchg.org Reported-by: Ye Xiaolong <xiaolong.ye@intel.com> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Michal Hocko <mhocko@suse.com> Cc: Vladimir Davydov <vdavydov@virtuozzo.com> Signed-off-by: Guenter Roeck <linux@roeck-us.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2016-07-28 16:07:41 -07:00
..
kasan	kasan/quarantine: fix bugs on qlist_move_cache()	2016-07-15 14:54:27 +09:00
backing-dev.c	mm: throttle on IO only when there are too many dirty and writeback pages	2016-05-20 17:58:30 -07:00
balloon_compaction.c	mm: balloon: use general non-lru movable page feature	2016-07-26 16:19:19 -07:00
bootmem.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
cleancache.c	cleancache: constify cleancache_ops structure	2016-01-27 09:09:57 -05:00
cma_debug.c	mm/cma_debug: correct size input to bitmap function	2015-07-17 16:39:54 -07:00
cma.c	mm/cma: silence warnings due to max() usage	2016-05-27 14:49:37 -07:00
cma.h	mm: cma: mark cma_bitmap_maxno() inline in header	2015-08-14 15:56:32 -07:00
compaction.c	mm/page_alloc: introduce post allocation processing on page allocator	2016-07-26 16:19:19 -07:00
debug_page_ref.c	mm/page_ref: add tracepoint to track down page reference manipulation	2016-03-17 15:09:34 -07:00
debug.c	mm: introduce page reference manipulation functions	2016-03-17 15:09:34 -07:00
dmapool.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
early_ioremap.c	mm/early_ioremap: use offset_in_page macro	2015-11-05 19:34:48 -08:00
fadvise.c	mm/fadvise.c: do not discard partial pages with POSIX_FADV_DONTNEED	2016-06-09 14:23:11 -07:00
failslab.c	mm: fault-inject take over bootstrap kmem_cache check	2016-03-15 16:55:16 -07:00
filemap.c	shmem: add huge pages support	2016-07-26 16:19:19 -07:00
frame_vector.c	mm/gup: Switch all callers of get_user_pages() to not pass tsk/mm	2016-02-16 10:11:12 +01:00
frontswap.c	mm, frontswap: convert frontswap_enabled to static key	2016-07-26 16:19:19 -07:00
gup.c	thp: file pages support for split_huge_page()	2016-07-26 16:19:19 -07:00
highmem.c	mm/highmem: make nr_free_highpages() handles all highmem zones by itself	2016-05-19 19:12:14 -07:00
huge_memory.c	thp: fix comments of __pmd_trans_huge_lock()	2016-07-26 16:19:19 -07:00
hugetlb_cgroup.c	mm, hugetlb_cgroup: round limit_in_bytes down to hugepage size	2016-05-20 17:58:30 -07:00
hugetlb.c	Merge branch 'akpm' (patches from Andrew)	2016-07-26 19:55:54 -07:00
hwpoison-inject.c	hwpoison: use page_cgroup_ino for filtering by memcg	2015-09-10 13:29:01 -07:00
init-mm.c
internal.h	mm: introduce fault_env	2016-07-26 16:19:19 -07:00
interval_tree.c	mm: replace vma->sharead.linear with vma->shared	2015-02-10 14:30:31 -08:00
Kconfig	thp: introduce CONFIG_TRANSPARENT_HUGE_PAGECACHE	2016-07-26 16:19:19 -07:00
Kconfig.debug	mm/page_ref: add tracepoint to track down page reference manipulation	2016-03-17 15:09:34 -07:00
khugepaged.c	mm, thp: convert from optimistic swapin collapsing to conservative	2016-07-26 16:19:19 -07:00
kmemcheck.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
kmemleak-test.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
kmemleak.c	mm: prevent KASAN false positives in kmemleak	2016-06-24 17:23:52 -07:00
ksm.c	mm: do not pass mm_struct into handle_mm_fault	2016-07-26 16:19:19 -07:00
list_lru.c	mm: memcontrol: move kmem accounting code to CONFIG_MEMCG	2016-01-20 17:09:18 -08:00
maccess.c	x86: remove more uaccess_32.h complexity	2016-05-22 17:21:27 -07:00
madvise.c	mm: make mmap_sem for write waits killable for mm syscalls	2016-05-23 17:04:14 -07:00
Makefile	thp: extract khugepaged from mm/huge_memory.c	2016-07-26 16:19:19 -07:00
memblock.c	mm/memblock.c:memblock_add_range(): if nr_new is 0 just return	2016-07-26 16:19:19 -07:00
memcontrol.c	mm: fix vm-scalability regression in cgroup-aware workingset code	2016-07-28 16:07:41 -07:00
memory_hotplug.c	memory-hotplug: more general validation of zone during online	2016-07-26 16:19:19 -07:00
memory-failure.c	mm/memory-failure.c: replace "MCE" with "Memory failure"	2016-05-20 17:58:30 -07:00
memory.c	thp: introduce CONFIG_TRANSPARENT_HUGE_PAGECACHE	2016-07-26 16:19:19 -07:00
mempolicy.c	shmem: add huge pages support	2016-07-26 16:19:19 -07:00
mempool.c	mm: mempool: kasan: don't poot mempool objects in quarantine	2016-06-24 17:23:52 -07:00
memtest.c	memtest: remove unused header files	2015-09-08 15:35:28 -07:00
migrate.c	mm: introduce do_set_pmd()	2016-07-26 16:19:19 -07:00
mincore.c	mm, fs: remove remaining PAGE_CACHE_* and page_cache_{get,release} usage	2016-04-04 10:41:08 -07:00
mlock.c	mm: make mmap_sem for write waits killable for mm syscalls	2016-05-23 17:04:14 -07:00
mm_init.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
mmap.c	shmem: get_unmapped_area align huge page	2016-07-26 16:19:19 -07:00
mmu_context.c	mm/mmu_context, sched/core: Fix mmu_context.h assumption	2016-04-28 11:44:19 +02:00
mmu_notifier.c	fix Christoph's email addresses	2016-03-17 15:09:34 -07:00
mmzone.c	mm, page_alloc: inline the fast path of the zonelist iterator	2016-05-19 19:12:14 -07:00
mprotect.c	mm: thp: check pmd_trans_unstable() after split_huge_pmd()	2016-07-26 16:19:19 -07:00
mremap.c	mm: thp: check pmd_trans_unstable() after split_huge_pmd()	2016-07-26 16:19:19 -07:00
msync.c	mm/msync: use offset_in_page macro	2015-11-05 19:34:48 -08:00
nobootmem.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
nommu.c	mm: introduce fault_env	2016-07-26 16:19:19 -07:00
oom_kill.c	mm, oom: tighten task_will_free_mem() locking	2016-07-28 16:07:41 -07:00
page_alloc.c	mm: update the comment in __isolate_free_page	2016-07-28 16:07:41 -07:00
page_counter.c	mm: page_counter: let page_counter_try_charge() return bool	2015-11-05 19:34:48 -08:00
page_ext.c	mm: use early_pfn_to_nid in page_ext_init	2016-05-27 14:49:37 -07:00
page_idle.c	mm: add page_check_address_transhuge() helper	2016-01-15 17:56:32 -08:00
page_io.c	block, fs, mm, drivers: use bio set/get op accessors	2016-06-07 13:41:38 -06:00
page_isolation.c	mm/page_isolation: clean up confused code	2016-07-26 16:19:19 -07:00
page_owner.c	mm/page_owner: use stackdepot to store stacktrace	2016-07-26 16:19:19 -07:00
page_poison.c	mm: check the return value of lookup_page_ext for all call sites	2016-06-03 15:06:22 -07:00
page-writeback.c	shmem: add huge pages support	2016-07-26 16:19:19 -07:00
pagewalk.c	thp: rename split_huge_page_pmd() to split_huge_pmd()	2016-01-15 17:56:32 -08:00
percpu-km.c	mm: percpu: use pr_fmt to prefix output	2016-03-17 15:09:34 -07:00
percpu-vm.c	percpu: move region iterations out of pcpu_[de]populate_chunk()	2014-09-02 14:46:02 -04:00
percpu.c	percpu: fix synchronization between synchronous map extension and chunk destruction	2016-05-25 11:48:25 -04:00
pgtable-generic.c	mm/thp/migration: switch from flush_tlb_range to flush_pmd_tlb_range	2016-03-17 15:09:34 -07:00
process_vm_access.c	mm/gup: Introduce get_user_pages_remote()	2016-02-16 10:04:09 +01:00
quicklist.c	fix Christoph's email addresses	2016-03-17 15:09:34 -07:00
readahead.c	mm, memcg: use consistent gfp flags during readahead	2016-07-26 16:19:19 -07:00
rmap.c	mm, rmap: account shmem thp pages	2016-07-26 16:19:19 -07:00
shmem.c	shmem: split huge pages beyond i_size under memory pressure	2016-07-26 16:19:19 -07:00
slab_common.c	mm: charge/uncharge kmemcg from generic page allocator paths	2016-07-26 16:19:19 -07:00
slab.c	mm/slab: use list_move instead of list_del/list_add	2016-07-26 16:19:19 -07:00
slab.h	mm: memcontrol: cleanup kmem charge functions	2016-07-26 16:19:19 -07:00
slob.c	mm: slab: free kmem_cache_node after destroy sysfs file	2016-02-18 16:23:24 -08:00
slub.c	mm: charge/uncharge kmemcg from generic page allocator paths	2016-07-26 16:19:19 -07:00
sparse-vmemmap.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
sparse.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
swap_cgroup.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
swap_state.c	mm: thp: broken page count after commit aa88b68c3b	2016-06-09 14:23:11 -07:00
swap.c	shmem: add huge pages support	2016-07-26 16:19:19 -07:00
swapfile.c	mm, frontswap: convert frontswap_enabled to static key	2016-07-26 16:19:19 -07:00
truncate.c	truncate: handle file thp	2016-07-26 16:19:19 -07:00
userfaultfd.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
util.c	rmap: support file thp	2016-07-26 16:19:19 -07:00
vmacache.c	mm/vmacache: inline vmacache_valid_mm()	2015-11-05 19:34:48 -08:00
vmalloc.c	mm: charge/uncharge kmemcg from generic page allocator paths	2016-07-26 16:19:19 -07:00
vmpressure.c	mm/vmpressure.c: fix subtree pressure detection	2016-02-03 08:28:43 -08:00
vmscan.c	vmscan: split file huge pages before paging them out	2016-07-26 16:19:19 -07:00
vmstat.c	mm, rmap: account shmem thp pages	2016-07-26 16:19:19 -07:00
workingset.c	mm: fix vm-scalability regression in cgroup-aware workingset code	2016-07-28 16:07:41 -07:00
z3fold.c	mm/z3fold.c: avoid modifying HEADLESS page and minor cleanup	2016-06-03 16:02:55 -07:00
zbud.c	mm/zbud.c: use list_last_entry() instead of list_tail_entry()	2016-01-15 11:40:52 -08:00
zpool.c	mm: zsmalloc: constify struct zs_pool name	2015-11-06 17:50:42 -08:00
zsmalloc.c	mm: fix build warnings in <linux/compaction.h>	2016-07-26 16:19:19 -07:00
zswap.c	mm/zswap: use workqueue to destroy pool	2016-05-20 17:58:30 -07:00