korg/linux
0
0
Fork 0
mirror of https://mirrors.bfsu.edu.cn/git/linux.git synced 2024-11-11 12:28:41 +08:00
Code Issues Packages Projects Releases Wiki Activity

net-zerocopy: Introduce short-circuit small reads.

Browse Source 
Sometimes, we may call tcp receive zerocopy when inq is 0,
or inq < PAGE_SIZE, or inq is generally small enough that
it is cheaper to copy rather than remap pages.

In these cases, we may want to either return early (inq=0) or
attempt to use the provided copy buffer to simply copy
the received data.

This allows us to save both system call overhead and
the latency of acquiring mmap_sem in read mode for cases where
it would be useless to do so.

This patchset enables this behaviour by:
1. Returning quickly if inq is 0.
2. Attempting to perform a regular copy if a hybrid copybuffer is
   provided and it is large enough to absorb all available bytes.
3. Return quickly if no such buffer was provided and there are less
   than PAGE_SIZE bytes available.

For small RPC ping-pong workloads, normally we would have
1 getsockopt(), 1 recvmsg() and 1 sendmsg() call per RPC. With this
change, we remove the recvmsg() call entirely, reducing the syscall
overhead by about 33%. In testing with small (hundreds of bytes)
RPC traffic, this yields a syscall reduction of about 33% and
an efficiency gain of about 3-5% when defined as QPS/CPU Util.

Signed-off-by: Arjun Roy <arjunroy@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
This commit is contained in:
Arjun Roy 2020-12-02 14:53:47 -08:00 committed by Jakub Kicinski
parent 936ced4157
commit f21a3c4803
1 changed files with 36 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
net/ipv4/tcp.c
View File

@ -1800,6 +1800,39 @@ static int find_next_mappable_frag(const skb_frag_t *frag,
return offset;
}
static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len,
int nonblock, int flags,
struct scm_timestamping_internal *tss,
int *cmsg_flags);
static int receive_fallback_to_copy(struct sock *sk,
struct tcp_zerocopy_receive *zc, int inq)
{
unsigned long copy_address = (unsigned long)zc->copybuf_address;
struct scm_timestamping_internal tss_unused;
int err, cmsg_flags_unused;
struct msghdr msg = {};
struct iovec iov;
zc->length = 0;
zc->recv_skip_hint = 0;
if (copy_address != zc->copybuf_address)
return -EINVAL;
err = import_single_range(READ, (void __user *)copy_address,
inq, &iov, &msg.msg_iter);
if (err)
return err;
err = tcp_recvmsg_locked(sk, &msg, inq, /*nonblock=*/1, /*flags=*/0,
&tss_unused, &cmsg_flags_unused);
if (err < 0)
return err;
zc->copybuf_len = err;
return 0;
}
static int tcp_copy_straggler_data(struct tcp_zerocopy_receive *zc,
struct sk_buff *skb, u32 copylen,
u32 *offset, u32 *seq)
@ -1904,6 +1937,9 @@ static int tcp_zerocopy_receive(struct sock *sk,
sock_rps_record_flow(sk);
if (inq && inq <= copybuf_len)
return receive_fallback_to_copy(sk, zc, inq);
if (inq < PAGE_SIZE) {
zc->length = 0;
zc->recv_skip_hint = inq;