Pavel Begunkov 8836df0212 io_uring: adjust defer tw counting ... [ Upstream commit dc12d1799c ] The UINT_MAX work item counting bias in io_req_local_work_add() in case of !IOU_F_TWQ_LAZY_WAKE works in a sense that we will not miss a wake up, however it's still eerie. In particular, if we add a lazy work item after a non-lazy one, we'll increment it and get nr_tw==0, and subsequent adds may try to unnecessarily wake up the task, which is though not so likely to happen in real workloads. Half the bias, it's still large enough to be larger than any valid ->cq_wait_nr, which is limited by IORING_MAX_CQ_ENTRIES, but further have a good enough of space before it overflows. Fixes: 8751d15426 ("io_uring: reduce scheduling due to tw") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/108b971e958deaf7048342930c341ba90f75d806.1705438669.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <sashal@kernel.org>		2024-01-25 15:36:00 -08:00
..
advise.c	io_uring: always go async for unsupported fadvise flags	2023-01-29 15:18:26 -07:00
advise.h
alloc_cache.h	io_uring/rsrc: consolidate node caching	2023-04-12 12:09:41 -06:00
cancel.c	io_uring: use fget/fput consistently	2024-01-20 11:51:38 +01:00
cancel.h	io_uring/cancel: support opcode based lookup and cancelation	2023-07-17 10:05:48 -06:00
epoll.c	io_uring: undeprecate epoll_ctl support	2023-05-26 20:22:41 -06:00
epoll.h
fdinfo.c	io_uring/fdinfo: remove need for sqpoll lock for thread/pid retrieval	2023-11-28 17:19:52 +00:00
fdinfo.h
filetable.c	io_uring: add helpers to decode the fixed file file_ptr	2023-06-20 09:36:22 -06:00
filetable.h	io_uring: add helpers to decode the fixed file file_ptr	2023-06-20 09:36:22 -06:00
fs.c	io_uring/fs: consider link->flags when getting path for LINKAT	2023-12-03 07:33:07 +01:00
fs.h
io_uring.c	io_uring: adjust defer tw counting	2024-01-25 15:36:00 -08:00
io_uring.h	io_uring: enable io_mem_alloc/free to be used in other parts	2023-12-08 08:52:20 +01:00
io-wq.c	io-wq: fully initialize wqe before calling cpuhp_state_add_instance_nocalls()	2023-10-05 14:11:18 -06:00
io-wq.h	io_uring: break out of iowq iopoll on teardown	2023-09-07 09:02:27 -06:00
kbuf.c	io_uring/kbuf: check for buffer list readiness after NULL check	2023-12-13 18:45:17 +01:00
kbuf.h	io_uring/kbuf: defer release of mapped buffer rings	2023-12-08 08:52:20 +01:00
Makefile
msg_ring.c	io_uring: use io_file_from_index in io_msg_grab_file	2023-06-20 09:36:22 -06:00
msg_ring.h	io_uring: get rid of double locking	2022-12-07 06:47:13 -07:00
net.c	io_uring/net: ensure socket is marked connected on connect retry	2023-11-20 11:59:38 +01:00
net.h	io_uring: Add KASAN support for alloc_caches	2023-04-03 07:16:14 -06:00
nop.c
nop.h
notif.c	io_uring/notif: add constant for ubuf_info flags	2023-04-15 14:21:04 -06:00
notif.h	io_uring/notif: add constant for ubuf_info flags	2023-04-15 14:21:04 -06:00
opdef.c	io_uring: Pass whole sqe to commands	2023-05-04 08:19:05 -06:00
opdef.h	io_uring: Split io_issue_def struct	2023-01-29 15:17:41 -07:00
openclose.c	io_uring: correct check for O_TMPFILE	2023-08-07 12:34:23 -06:00
openclose.h
poll.c	io_uring: never overflow io_aux_cqe	2023-08-11 10:42:57 -06:00
poll.h	io_uring: avoid indirect function calls for the hottest task_work	2023-06-02 08:55:37 -06:00
refs.h
rsrc.c	io_uring: fix off-by one bvec index	2023-12-03 07:33:07 +01:00
rsrc.h	io_uring/af_unix: disable sending io_uring over sockets	2023-12-13 18:45:20 +01:00
rw.c	io_uring/rw: ensure io->bytes_done is always initialized	2024-01-25 15:35:45 -08:00
rw.h	io_uring: avoid indirect function calls for the hottest task_work	2023-06-02 08:55:37 -06:00
slist.h	io_uring: silence variable ‘prev’ set but not used warning	2023-03-09 10:10:58 -07:00
splice.c	io_uring/splice: use fput() directly	2023-08-10 10:24:25 -06:00
splice.h
sqpoll.c	io_uring/fdinfo: remove need for sqpoll lock for thread/pid retrieval	2023-11-28 17:19:52 +00:00
sqpoll.h	io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used	2023-08-16 13:40:28 -06:00
statx.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
statx.h
sync.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
sync.h
tctx.c	io_uring: Add io_uring_setup flag to pre-register ring fd and never install it	2023-05-16 08:06:00 -06:00
tctx.h	io_uring: simplify __io_uring_add_tctx_node	2022-10-07 12:25:30 -06:00
timeout.c	io_uring: never overflow io_aux_cqe	2023-08-11 10:42:57 -06:00
timeout.h	io_uring: remove unused return from io_disarm_next	2022-09-21 13:15:01 -06:00
uring_cmd.c	io_uring/cmd: fix breakage in SOCKET_URING_OP_SIOC* implementation	2023-12-20 17:01:52 +01:00
uring_cmd.h	io_uring: Remove unnecessary BUILD_BUG_ON	2023-05-04 08:19:05 -06:00
xattr.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
xattr.h