korg/linux
0
0
Fork 0
mirror of https://mirrors.bfsu.edu.cn/git/linux.git synced 2024-12-13 22:14:20 +08:00
Code Issues Packages Projects Releases Wiki Activity
6db5dd2bf4
linux/net/ipv4
History
Arjun Roy 7a7f094635 tcp: Use per-vma locking for receive zerocopy 
Per-VMA locking allows us to lock a struct vm_area_struct without
taking the process-wide mmap lock in read mode.

Consider a process workload where the mmap lock is taken constantly in
write mode. In this scenario, all zerocopy receives are periodically
blocked during that period of time - though in principle, the memory
ranges being used by TCP are not touched by the operations that need
the mmap write lock. This results in performance degradation.

Now consider another workload where the mmap lock is never taken in
write mode, but there are many TCP connections using receive zerocopy
that are concurrently receiving. These connections all take the mmap
lock in read mode, but this does induce a lot of contention and atomic
ops for this process-wide lock. This results in additional CPU
overhead caused by contending on the cache line for this lock.

However, with per-vma locking, both of these problems can be avoided.

As a test, I ran an RPC-style request/response workload with 4KB
payloads and receive zerocopy enabled, with 100 simultaneous TCP
connections. I measured perf cycles within the
find_tcp_vma/mmap_read_lock/mmap_read_unlock codepath, with and
without per-vma locking enabled.

When using process-wide mmap semaphore read locking, about 1% of
measured perf cycles were within this path. With per-VMA locking, this
value dropped to about 0.45%.

Signed-off-by: Arjun Roy <arjunroy@google.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-06-18 11:16:00 +01:00
..
bpfilter
netfilter xtables: move icmp/icmpv6 logic to xt_tcpudp 2023-03-22 21:48:59 +01:00
af_inet.c net: ioctl: Use kernel memory on protocol ioctl callbacks 2023-06-15 22:33:26 -07:00
ah4.c net: ipv4: Remove completion function scaffolding 2023-02-13 18:35:15 +08:00
arp.c neighbour: annotate lockless accesses to n->nud_state 2023-03-15 00:37:32 -07:00
bpf_tcp_ca.c bpf: Remove unused arguments from btf_struct_access(). 2023-04-04 16:57:10 -07:00
cipso_ipv4.c cipso_ipv4: use iph_set_totlen in skbuff_setattr 2023-02-01 20:54:27 -08:00
datagram.c Networking fixes for 6.1-rc2, including fixes from netfilter 2022-10-20 17:24:59 -07:00
devinet.c net: ipv4: Allow changing IPv4 address protocol 2023-03-23 08:32:52 +00:00
esp4_offload.c net: move gso declarations and functions to their own files 2023-06-10 00:11:41 -07:00
esp4.c net: ipv4: Remove completion function scaffolding 2023-02-13 18:35:15 +08:00
fib_frontend.c ipv4: Fix incorrect table ID in IOCTL path 2023-03-16 17:26:31 -07:00
fib_lookup.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2022-02-17 11:44:20 -08:00
fib_notifier.c net: ipv4: remove superfluous header files from fib_notifier.c 2021-09-28 17:32:56 -07:00
fib_rules.c ipv4: remove unnecessary type castings 2022-04-30 15:12:58 +01:00
fib_semantics.c neighbour: switch to standard rcu, instead of rcu_bh 2023-03-21 21:32:18 -07:00
fib_trie.c ipv4: Fix error return code in fib_table_insert() 2022-11-22 20:18:20 -08:00
fou_bpf.c bpf,fou: Add bpf_skb_{set,get}_fou_encap kfuncs 2023-04-12 16:40:39 -07:00
fou_core.c bpf,fou: Add bpf_skb_{set,get}_fou_encap kfuncs 2023-04-12 16:40:39 -07:00
fou_nl.c net: ynl: prefix uAPI header include with uapi/ 2023-05-26 10:30:14 +01:00
fou_nl.h net: ynl: prefix uAPI header include with uapi/ 2023-05-26 10:30:14 +01:00
gre_demux.c
gre_offload.c net: move gso declarations and functions to their own files 2023-06-10 00:11:41 -07:00
icmp.c icmp: guard against too small mtu 2023-03-31 21:37:06 -07:00
igmp.c ipv4: constify ip_mc_sf_allow() socket argument 2023-03-17 08:56:37 +00:00
inet_connection_sock.c ipv4: Set correct scope in inet_csk_route_*(). 2023-06-06 21:13:03 -07:00
inet_diag.c net: inet: Retire port only listening_hash 2022-05-12 16:52:18 -07:00
inet_fragment.c net: dropreason: add SKB_DROP_REASON_FRAG_REASM_TIMEOUT 2022-10-31 20:14:27 -07:00
inet_hashtables.c ipv6: Remove in6addr_any alternatives. 2023-03-29 08:22:52 +01:00
inet_timewait_sock.c net: no longer support SOCK_REFCNT_DEBUG feature 2023-02-15 10:25:21 +00:00
inetpeer.c inetpeer: Fix data-races around sysctl. 2022-07-08 12:10:33 +01:00
ip_forward.c ip: Fix data-races around sysctl_ip_fwd_update_priority. 2022-07-15 11:49:55 +01:00
ip_fragment.c Revert "net: Remove low_thresh in ip defrag" 2023-05-16 20:46:30 -07:00
ip_gre.c ip_gre: clean up some inconsistent indenting 2023-06-02 10:44:46 +01:00
ip_input.c net: add support for ipv4 big tcp 2023-02-01 20:54:27 -08:00
ip_options.c ipv4: drop fragmentation code from ip_options_build() 2022-01-29 17:53:07 +00:00
ip_output.c ip, ip6: Fix splice to raw and ping sockets 2023-06-16 11:45:16 -07:00
ip_sockglue.c ipv{4,6}/raw: fix output xfrm lookup wrt protocol 2023-05-23 15:38:59 +02:00
ip_tunnel_core.c net: Add helper function to parse netlink msg of ip_tunnel_parm 2022-10-03 07:59:06 +01:00
ip_tunnel.c bpf-next-for-netdev 2023-04-13 16:43:38 -07:00
ip_vti.c ipv4: tunnels: use DEV_STATS_INC() 2022-11-16 12:48:44 +00:00
ipcomp.c xfrm: ipcomp: add extack to ipcomp{4,6}_init_state 2022-09-29 07:18:00 +02:00
ipconfig.c net: ipconfig: move ic_nameservers_fallback into #ifdef block 2023-05-22 11:17:55 +01:00
ipip.c ipip,ip_tunnel,sit: Add FOU support for externally controlled ipip devices 2023-04-12 16:40:39 -07:00
ipmr_base.c ipmr: adopt rcu_read_lock() in mr_dump() 2022-06-24 11:34:38 +01:00
ipmr.c net: ioctl: Use kernel memory on protocol ioctl callbacks 2023-06-15 22:33:26 -07:00
Kconfig tcp: configurable source port perturb table size 2022-11-16 13:02:04 +00:00
Makefile bpf,fou: Add bpf_skb_{set,get}_fou_encap kfuncs 2023-04-12 16:40:39 -07:00
metrics.c ipv4: prevent potential spectre v1 gadget in ip_metrics_convert() 2023-01-23 21:37:25 -08:00
netfilter.c netfilter: Use l3mdev flow key when re-routing mangled packets 2022-05-16 13:03:29 +02:00
netlink.c
nexthop.c neighbour: switch to standard rcu, instead of rcu_bh 2023-03-21 21:32:18 -07:00
ping.c ping: Stop using RTO_ONLINK. 2023-05-24 08:22:06 +01:00
proc.c icmp: Add counters for rate limits 2023-01-26 10:52:18 +01:00
protocol.c
raw_diag.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2023-04-06 12:01:20 -07:00
raw.c net: ioctl: Use kernel memory on protocol ioctl callbacks 2023-06-15 22:33:26 -07:00
route.c net: dst: fix missing initialization of rt_uncached 2023-04-21 20:26:56 -07:00
syncookies.c tcp: Set route scope properly in cookie_v4_check(). 2023-06-06 21:13:03 -07:00
sysctl_net_ipv4.c tcp: enforce receive buffer memory limits by allowing the tcp window to shrink 2023-06-17 09:53:53 +01:00
tcp_bbr.c bpf: Add __bpf_kfunc tag to all kfuncs 2023-02-02 00:25:14 +01:00
tcp_bic.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_bpf.c tcp_bpf: Make tcp_bpf_sendpage() go through tcp_bpf_sendmsg(MSG_SPLICE_PAGES) 2023-06-12 21:13:23 -07:00
tcp_cdg.c Random number generator fixes for Linux 6.1-rc1. 2022-10-16 15:27:07 -07:00
tcp_cong.c net: Update an existing TCP congestion control algorithm. 2023-03-22 22:53:00 -07:00
tcp_cubic.c bpf: Add __bpf_kfunc tag to all kfuncs 2023-02-02 00:25:14 +01:00
tcp_dctcp.c bpf: Add __bpf_kfunc tag to all kfuncs 2023-02-02 00:25:14 +01:00
tcp_dctcp.h
tcp_diag.c tcp: Access &tcp_hashinfo via net. 2022-09-20 10:21:49 -07:00
tcp_fastopen.c tcp: Make SYN ACK RTO tunable by BPF programs with TFO 2022-08-17 10:19:22 +01:00
tcp_highspeed.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_htcp.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_hybla.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_illinois.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_input.c tcp: fix mishandling when the sack compression is deferred. 2023-06-01 13:15:12 +02:00
tcp_ipv4.c tcp: enforce receive buffer memory limits by allowing the tcp window to shrink 2023-06-17 09:53:53 +01:00
tcp_lp.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_metrics.c genetlink: start to validate reserved header bytes 2022-08-29 12:47:15 +01:00
tcp_minisocks.c net: tcp: make the txhash available in TIME_WAIT sockets for IPv4 too 2023-05-25 13:20:45 +02:00
tcp_nv.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_offload.c net: move gso declarations and functions to their own files 2023-06-10 00:11:41 -07:00
tcp_output.c tcp: enforce receive buffer memory limits by allowing the tcp window to shrink 2023-06-17 09:53:53 +01:00
tcp_plb.c prandom: remove prandom_u32_max() 2022-12-20 03:13:45 +01:00
tcp_rate.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2022-04-28 13:02:01 -07:00
tcp_recovery.c tcp: preserve const qualifier in tcp_sk() 2023-03-18 12:23:34 +00:00
tcp_scalable.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_timer.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2023-06-01 15:38:26 -07:00
tcp_ulp.c net/ulp: use consistent error code when blocking ULP 2023-01-19 09:26:16 -08:00
tcp_vegas.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_vegas.h
tcp_veno.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_westwood.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp_yeah.c tcp: add accessors to read/set tp->snd_cwnd 2022-04-06 12:05:41 -07:00
tcp.c tcp: Use per-vma locking for receive zerocopy 2023-06-18 11:16:00 +01:00
tunnel4.c
udp_bpf.c bpf, sockmap: Fix an infinite loop error when len is 0 in tcp_bpf_recvmsg_parser() 2023-03-03 17:25:15 +01:00
udp_diag.c udp: Access &udp_table via net. 2022-11-16 09:43:35 +00:00
udp_impl.h net: remove noblock parameter from recvmsg() entities 2022-04-12 15:00:25 +02:00
udp_offload.c net: move gso declarations and functions to their own files 2023-06-10 00:11:41 -07:00
udp_tunnel_core.c net/tunnel: wait until all sk_user_data reader finish before releasing the sock 2022-12-12 09:51:52 +00:00
udp_tunnel_nic.c udp_tunnel: Add checks for nla_nest_start() in __udp_tunnel_nic_dump_write() 2022-11-29 08:44:24 -08:00
udp_tunnel_stub.c
udp.c net: ioctl: Use kernel memory on protocol ioctl callbacks 2023-06-15 22:33:26 -07:00
udplite.c udplite: Print deprecation notice. 2023-06-15 15:08:58 -07:00
xfrm4_input.c
xfrm4_output.c
xfrm4_policy.c net: dst: fix missing initialization of rt_uncached 2023-04-21 20:26:56 -07:00
xfrm4_protocol.c net: xfrm: unexport __init-annotated xfrm4_protocol_init() 2022-06-08 10:10:13 -07:00
xfrm4_state.c
xfrm4_tunnel.c xfrm: tunnel: add extack to ipip_init_state, xfrm6_tunnel_init_state 2022-09-29 07:18:00 +02:00