linux/net/core
Jesper Dangaard Brouer be5dba25b4 net: page_pool: use alloc_pages_bulk in refill code path
There are cases where the page_pool need to refill with pages from the
page allocator.  Some workloads cause the page_pool to release pages
instead of recycling these pages.

For these workload it can improve performance to bulk alloc pages from the
page-allocator to refill the alloc cache.

For XDP-redirect workload with 100G mlx5 driver (that use page_pool)
redirecting xdp_frame packets into a veth, that does XDP_PASS to create an
SKB from the xdp_frame, which then cannot return the page to the
page_pool.

Performance results under GitHub xdp-project[1]:
 [1] https://github.com/xdp-project/xdp-project/blob/master/areas/mem/page_pool06_alloc_pages_bulk.org

Mel: The patch "net: page_pool: convert to use alloc_pages_bulk_array
variant" was squashed with this patch. From the test page, the array
variant was superior with one of the test results as follows.

	Kernel		XDP stats       CPU     pps           Delta
	Baseline	XDP-RX CPU      total   3,771,046       n/a
	List		XDP-RX CPU      total   3,940,242    +4.49%
	Array		XDP-RX CPU      total   4,249,224   +12.68%

Link: https://lkml.kernel.org/r/20210325114228.27719-10-mgorman@techsingularity.net
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Reviewed-by: Alexander Lobakin <alobakin@pm.me>
Cc: Alexander Duyck <alexander.duyck@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Chuck Lever <chuck.lever@oracle.com>
Cc: David Miller <davem@davemloft.net>
Cc: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-04-30 11:20:43 -07:00
..
bpf_sk_storage.c bpf: Prevent deadlock from recursive bpf_task_storage_[get|delete] 2021-02-26 11:51:48 -08:00
datagram.c udp: fix skb_copy_and_csum_datagram with odd segment sizes 2021-02-04 18:56:56 -08:00
datagram.h
dev_addr_lists.c net: core: Correct function name dev_uc_flush() in the kerneldoc 2021-03-28 17:56:56 -07:00
dev_ioctl.c net: fix dev_ifsioc_locked() race condition 2021-02-11 18:14:19 -08:00
dev.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-04-26 12:00:00 -07:00
devlink.c devlink: Extend SF port attributes to have external attribute 2021-04-24 00:58:53 -07:00
drop_monitor.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-03-25 15:31:22 -07:00
dst_cache.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
dst.c net, bpf: Fix ip6ip6 crash with collect_md populated skbs 2021-03-10 12:24:18 -08:00
failover.c
fib_notifier.c net: fib_notifier: propagate extack down to the notifier block callback 2019-10-04 11:10:56 -07:00
fib_rules.c treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
filter.c bpf: selftests: Add kfunc_call test 2021-03-26 20:41:52 -07:00
flow_dissector.c flow_dissector: Fix out-of-bounds warning in __skb_flow_bpf_to_target() 2021-04-16 17:02:27 -07:00
flow_offload.c net: flow_offload: Fix memory leak for indirect flow block 2020-12-09 16:08:33 -08:00
gen_estimator.c net_sched: gen_estimator: support large ewma log 2021-01-15 18:11:06 -08:00
gen_stats.c docs: networking: convert gen_stats.txt to ReST 2020-04-28 14:39:46 -07:00
gro_cells.c gro_cells: reduce number of synchronize_net() calls 2020-11-25 11:28:12 -08:00
hwbm.c net: hwbm: Make the hwbm_pool lock a mutex 2019-06-09 19:40:10 -07:00
link_watch.c net: Add IF_OPER_TESTING 2020-04-20 12:43:24 -07:00
lwt_bpf.c lwt_bpf: Replace preempt_disable() with migrate_disable() 2020-12-07 11:53:40 -08:00
lwtunnel.c net: ipv6: add rpl sr tunnel 2020-03-29 22:30:57 -07:00
Makefile net: selftest: fix build issue if INET is disabled 2021-04-28 14:06:45 -07:00
neighbour.c neighbour: Prevent Race condition in neighbour subsytem 2021-04-21 14:47:43 -07:00
net_namespace.c net: initialize net->net_cookie at netns setup 2021-02-11 14:10:07 -08:00
net-procfs.c net: move the ptype_all and ptype_base declarations to include/linux/netdevice.h 2021-03-22 13:14:45 -07:00
net-sysfs.c net-sysfs: remove possible sleep from an RCU read-side critical section 2021-03-22 13:28:13 -07:00
net-sysfs.h net-sysfs: add netdev_change_owner() 2020-02-26 20:07:25 -08:00
net-traces.c page_pool: add tracepoints for page_pool with details need by XDP 2019-06-19 11:23:13 -04:00
netclassid_cgroup.c net: Remove the err argument from sock_from_file 2020-12-04 22:32:40 +01:00
netevent.c net: core: Correct function name netevent_unregister_notifier() in the kerneldoc 2021-03-28 17:56:56 -07:00
netpoll.c Revert "net: Have netpoll bring-up DSA management interface" 2021-02-06 14:42:57 -08:00
netprio_cgroup.c net: Remove the err argument from sock_from_file 2020-12-04 22:32:40 +01:00
page_pool.c net: page_pool: use alloc_pages_bulk in refill code path 2021-04-30 11:20:43 -07:00
pktgen.c pktgen: fix misuse of BUG_ON() in pktgen_thread_worker() 2021-01-27 16:46:37 -08:00
ptp_classifier.c ptp: Add generic ptp v2 header parsing function 2020-08-19 16:07:49 -07:00
request_sock.c tcp: add rcu protection around tp->fastopen_rsk 2019-10-13 10:13:08 -07:00
rtnetlink.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-04-09 20:48:35 -07:00
scm.c scm: fix a typo in put_cmsg() 2021-04-16 11:41:07 -07:00
secure_seq.c crypto: lib/sha1 - remove unnecessary includes of linux/cryptohash.h 2020-05-08 15:32:17 +10:00
selftests.c net: add generic selftest support 2021-04-20 16:08:02 -07:00
skbuff.c Networking changes for 5.13. 2021-04-29 11:57:23 -07:00
skmsg.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-04-09 20:48:35 -07:00
sock_diag.c bpf, net: Rework cookie generator as per-cpu one 2020-09-30 11:50:35 -07:00
sock_map.c sock_map: Fix a potential use-after-free in sock_map_close() 2021-04-12 17:35:26 +02:00
sock_reuseport.c udp: Prevent reuseport_select_sock from reading uninitialized socks 2021-01-08 19:15:40 -08:00
sock.c net: sock: remove the unnecessary check in proto_register 2021-04-23 13:10:03 -07:00
stream.c tcp: make sure EPOLLOUT wont be missed 2019-08-19 13:07:43 -07:00
sysctl_net_core.c net: change netdev_unregister_timeout_secs min value to 1 2021-03-25 17:24:06 -07:00
timestamping.c net: Introduce a new MII time stamping interface. 2019-12-25 19:51:33 -08:00
tso.c net: tso: add UDP segmentation support 2020-06-18 20:46:23 -07:00
utils.c net: Fix skb->csum update in inet_proto_csum_replace16(). 2020-01-24 20:54:30 +01:00
xdp.c xdp: fix xdp_return_frame() kernel BUG throw for page_pool memory model 2021-03-31 15:15:23 -07:00