Jens Axboe 5eff57fa9f io_uring/uring_cmd: defer SQE copying until it's needed ... The previous commit turned on async data for uring_cmd, and did the basic conversion of setting everything up on the prep side. However, for a lot of use cases, -EIOCBQUEUED will get returned on issue, as the operation got successfully queued. For that case, a persistent SQE isn't needed, as it's just used for issue. Unless execution goes async immediately, defer copying the double SQE until it's necessary. This greatly reduces the overhead of such commands, as evidenced by a perf diff from before and after this change: 10.60% -8.58% [kernel.vmlinux] [k] io_uring_cmd_prep where the prep side drops from 10.60% to ~2%, which is more expected. Performance also rises from ~113M IOPS to ~122M IOPS, bringing us back to where it was before the async command prep. Tested-by: Anuj Gupta <anuj20.g@samsung.com> Reviewed-by: Anuj Gupta <anuj20.g@samsung.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>		2024-04-15 08:10:25 -06:00
..
advise.c	io_uring: always go async for unsupported fadvise flags	2023-01-29 15:18:26 -07:00
advise.h
alloc_cache.h	io_uring/alloc_cache: shrink default max entries from 512 to 128	2024-04-15 08:10:25 -06:00
cancel.c	io_uring/cancel: don't default to setting req->work.cancel_seq	2024-02-08 13:27:06 -07:00
cancel.h	io_uring/cancel: don't default to setting req->work.cancel_seq	2024-02-08 13:27:06 -07:00
epoll.c	io_uring: undeprecate epoll_ctl support	2023-05-26 20:22:41 -06:00
epoll.h
fdinfo.c	io_uring: Fix sqpoll utilization check racing with dying sqpoll	2024-03-09 07:27:09 -07:00
fdinfo.h
filetable.c	io_uring: drop any code related to SCM_RIGHTS	2023-12-19 12:36:34 -07:00
filetable.h	io_uring: expand main struct io_kiocb flags to 64-bits	2024-02-08 13:27:03 -07:00
fs.c	io_uring/fs: consider link->flags when getting path for LINKAT	2023-11-20 09:01:42 -07:00
fs.h
futex.c	io_uring/futex: always remove futex entry for cancel all	2024-03-15 15:37:15 -06:00
futex.h	io_uring: add support for vectored futex waits	2023-09-29 02:37:08 -06:00
io_uring.c	io_uring/uring_cmd: switch to always allocating async data	2024-04-15 08:10:25 -06:00
io_uring.h	io_uring: clean up io_lockdep_assert_cq_locked	2024-04-15 08:10:24 -06:00
io-wq.c	io-wq: fully initialize wqe before calling cpuhp_state_add_instance_nocalls()	2023-10-05 14:11:18 -06:00
io-wq.h	io_uring: break out of iowq iopoll on teardown	2023-09-07 09:02:27 -06:00
kbuf.c	io_uring/kbuf: hold io_buffer_list reference over mmap	2024-04-02 19:03:27 -06:00
kbuf.h	io_uring/kbuf: hold io_buffer_list reference over mmap	2024-04-02 19:03:27 -06:00
Makefile	io-uring: add napi busy poll support	2024-02-09 11:54:19 -07:00
msg_ring.c	io_uring: use io_file_from_index in io_msg_grab_file	2023-06-20 09:36:22 -06:00
msg_ring.h	io_uring: get rid of double locking	2022-12-07 06:47:13 -07:00
napi.c	io_uring/napi: enable even with a timeout of 0	2024-02-15 15:37:28 -07:00
napi.h	io_uring: add register/unregister napi function	2024-02-09 11:54:32 -07:00
net.c	io_uring/net: move connect to always using async data	2024-04-15 08:10:25 -06:00
net.h	io_uring/net: move connect to always using async data	2024-04-15 08:10:25 -06:00
nop.c
nop.h
notif.c	io_uring/notif: add constant for ubuf_info flags	2023-04-15 14:21:04 -06:00
notif.h	io_uring/notif: add constant for ubuf_info flags	2023-04-15 14:21:04 -06:00
opdef.c	io_uring/uring_cmd: switch to always allocating async data	2024-04-15 08:10:25 -06:00
opdef.h	io_uring/rw: mark readv/writev as vectored in the opcode definition	2023-09-21 12:00:46 -06:00
openclose.c	io_uring: enable audit and restrict cred override for IORING_OP_FIXED_FD_INSTALL	2024-01-23 15:25:14 -07:00
openclose.h	io_uring/openclose: add support for IORING_OP_FIXED_FD_INSTALL	2023-12-12 07:42:57 -07:00
poll.c	io_uring: refactor io_fill_cqe_req_aux	2024-04-15 08:10:24 -06:00
poll.h	io_uring/rw: ensure poll based multishot read retries appropriately	2024-01-28 20:37:11 -07:00
refs.h
register.c	io_uring: add register/unregister napi function	2024-02-09 11:54:32 -07:00
register.h	io_uring/register: move io_uring_register(2) related code to register.c	2023-12-19 08:54:20 -07:00
rsrc.c	io_uring: drop any code related to SCM_RIGHTS	2023-12-19 12:36:34 -07:00
rsrc.h	io_uring: Don't include af_unix.h.	2024-02-12 19:02:11 -07:00
rw.c	io_uring/rw: add iovec recycling	2024-04-15 08:10:25 -06:00
rw.h	io_uring/rw: add iovec recycling	2024-04-15 08:10:25 -06:00
slist.h	io_uring: silence variable ‘prev’ set but not used warning	2023-03-09 10:10:58 -07:00
splice.c	splice: return type ssize_t from all helpers	2023-12-12 16:19:59 +01:00
splice.h
sqpoll.c	io_uring/sqpoll: early exit thread if task_context wasn't allocated	2024-03-18 20:22:42 -06:00
sqpoll.h	io_uring/sqpoll: statistics of the true utilization of sq threads	2024-03-01 06:28:19 -07:00
statx.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
statx.h
sync.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
sync.h
tctx.c	io_uring: Add io_uring_setup flag to pre-register ring fd and never install it	2023-05-16 08:06:00 -06:00
tctx.h	io_uring: simplify __io_uring_add_tctx_node	2022-10-07 12:25:30 -06:00
timeout.c	io_uring: refactor io_fill_cqe_req_aux	2024-04-15 08:10:24 -06:00
timeout.h	io_uring: remove unused return from io_disarm_next	2022-09-21 13:15:01 -06:00
truncate.c	io_uring: add support for ftruncate	2024-02-09 09:04:39 -07:00
truncate.h	io_uring: add support for ftruncate	2024-02-09 09:04:39 -07:00
uring_cmd.c	io_uring/uring_cmd: defer SQE copying until it's needed	2024-04-15 08:10:25 -06:00
uring_cmd.h	io_uring/uring_cmd: switch to always allocating async data	2024-04-15 08:10:25 -06:00
waitid.c	io_uring: remove struct io_tw_state::locked	2024-04-15 08:10:24 -06:00
waitid.h	io_uring: add IORING_OP_WAITID support	2023-09-21 12:04:45 -06:00
xattr.c	io_uring: use file_mnt_idmap helper	2024-02-06 19:55:14 -07:00
xattr.h