Jens Axboe 28aabffae6 io_uring/sqpoll: close race on waiting for sqring entries ... When an application uses SQPOLL, it must wait for the SQPOLL thread to consume SQE entries, if it fails to get an sqe when calling io_uring_get_sqe(). It can do so by calling io_uring_enter(2) with the flag value of IORING_ENTER_SQ_WAIT. In liburing, this is generally done with io_uring_sqring_wait(). There's a natural expectation that once this call returns, a new SQE entry can be retrieved, filled out, and submitted. However, the kernel uses the cached sq head to determine if the SQRING is full or not. If the SQPOLL thread is currently in the process of submitting SQE entries, it may have updated the cached sq head, but not yet committed it to the SQ ring. Hence the kernel may find that there are SQE entries ready to be consumed, and return successfully to the application. If the SQPOLL thread hasn't yet committed the SQ ring entries by the time the application returns to userspace and attempts to get a new SQE, it will fail getting a new SQE. Fix this by having io_sqring_full() always use the user visible SQ ring head entry, rather than the internally cached one. Cc: stable@vger.kernel.org # 5.10+ Link: https://github.com/axboe/liburing/discussions/1267 Reported-by: Benedek Thaler <thaler@thaler.hu> Signed-off-by: Jens Axboe <axboe@kernel.dk>		2024-10-15 09:13:51 -06:00
..
advise.c	io_uring/advise: support 64-bit lengths	2024-06-16 14:54:55 -06:00
advise.h
alloc_cache.h	io_uring/alloc_cache: switch to array based caching	2024-04-15 08:10:25 -06:00
cancel.c	io_uring: fix warnings on shadow variables	2024-04-15 08:10:26 -06:00
cancel.h	io_uring: fix cancellation overwriting req->flags	2024-06-13 19:25:28 -06:00
epoll.c	io_uring: undeprecate epoll_ctl support	2023-05-26 20:22:41 -06:00
epoll.h
eventfd.c	io_uring/eventfd: move refs to refcount_t	2024-09-08 16:43:57 -06:00
eventfd.h	io_uring/eventfd: move eventfd handling to separate file	2024-06-16 14:54:55 -06:00
fdinfo.c	io_uring/rsrc: change ubuf->ubuf_end to length tracking	2024-09-15 09:15:22 -06:00
fdinfo.h
filetable.c	io_uring/filetable: don't unnecessarily clear/reset bitmap	2024-05-08 08:27:45 -06:00
filetable.h	io_uring: expand main struct io_kiocb flags to 64-bits	2024-02-08 13:27:03 -07:00
fs.c	io_uring/fs: consider link->flags when getting path for LINKAT	2023-11-20 09:01:42 -07:00
fs.h
futex.c	io_uring/alloc_cache: switch to array based caching	2024-04-15 08:10:25 -06:00
futex.h	io_uring/alloc_cache: switch to array based caching	2024-04-15 08:10:25 -06:00
io_uring.c	io_uring: fix casts to io_req_flags_t	2024-09-24 13:31:04 -06:00
io_uring.h	io_uring/sqpoll: close race on waiting for sqring entries	2024-10-15 09:13:51 -06:00
io-wq.c	io_uring/io-wq: inherit cpuset of cgroup in io worker	2024-09-11 07:27:56 -06:00
io-wq.h	io_uring/io-wq: make io_wq_work flags atomic	2024-06-16 14:54:55 -06:00
kbuf.c	for-6.12/io_uring-20240913	2024-09-16 13:29:00 +02:00
kbuf.h	io_uring/kbuf: add support for incremental buffer consumption	2024-08-29 08:44:58 -06:00
Makefile	io_uring: add GCOV_PROFILE_URING Kconfig option	2024-08-30 10:52:02 -06:00
memmap.c	io_uring: don't attempt to mmap larger than what the user asks for	2024-05-29 09:53:14 -06:00
memmap.h	io_uring: move mapping/allocation helpers to a separate file	2024-04-15 08:10:26 -06:00
msg_ring.c	io_uring/msg_ring: fix uninitialized use of target_req->flags	2024-07-25 08:41:35 -06:00
msg_ring.h	io_uring/msg_ring: add an alloc cache for io_kiocb entries	2024-06-24 08:39:55 -06:00
napi.c	io_uring: user registered clockid for wait timeouts	2024-08-25 08:27:01 -06:00
napi.h	io_uring/napi: postpone napi timeout adjustment	2024-08-25 08:27:01 -06:00
net.c	io_uring/net: harden multishot termination case for recv	2024-09-30 08:26:59 -06:00
net.h	io_uring: Introduce IORING_OP_LISTEN	2024-06-19 07:57:21 -06:00
nop.c	io_uring: support to inject result for NOP	2024-05-10 06:09:45 -06:00
nop.h
notif.c	io_uring/notif: disable LAZY_WAKE for linked notifs	2024-04-30 13:06:27 -06:00
notif.h	io_uring/notif: implement notification stacking	2024-04-22 19:31:18 -06:00
opdef.c	io_uring: Fix probe of disabled operations	2024-06-19 08:58:00 -06:00
opdef.h	io_uring: Fix probe of disabled operations	2024-06-19 08:58:00 -06:00
openclose.c	io_uring: enable audit and restrict cred override for IORING_OP_FIXED_FD_INSTALL	2024-01-23 15:25:14 -07:00
openclose.h	io_uring/openclose: add support for IORING_OP_FIXED_FD_INSTALL	2023-12-12 07:42:57 -07:00
poll.c	io_uring: keep multishot request NAPI timeout current	2024-07-30 06:18:58 -06:00
poll.h	io_uring/poll: shrink alloc cache size to 32	2024-04-15 08:10:25 -06:00
refs.h	io_uring: kill dead code in io_req_complete_post	2024-04-15 08:10:26 -06:00
register.c	io_uring: clean up a type in io_uring_register_get_file()	2024-09-16 12:04:10 -06:00
register.h	io_uring: clean up a type in io_uring_register_get_file()	2024-09-16 12:04:10 -06:00
rsrc.c	for-6.12/io_uring-20240922	2024-09-24 11:11:38 -07:00
rsrc.h	io_uring/rsrc: change ubuf->ubuf_end to length tracking	2024-09-15 09:15:22 -06:00
rw.c	io_uring/rw: allow pollable non-blocking attempts for !FMODE_NOWAIT	2024-10-06 20:58:53 -06:00
rw.h	io_uring/alloc_cache: switch to array based caching	2024-04-15 08:10:25 -06:00
slist.h	io_uring: silence variable ‘prev’ set but not used warning	2023-03-09 10:10:58 -07:00
splice.c	splice: return type ssize_t from all helpers	2023-12-12 16:19:59 +01:00
splice.h
sqpoll.c	for-6.12/io_uring-20240922	2024-09-24 11:11:38 -07:00
sqpoll.h	io_uring/sqpoll: statistics of the true utilization of sq threads	2024-03-01 06:28:19 -07:00
statx.c	vfs: retire user_path_at_empty and drop empty arg from getname_flags	2024-06-05 17:03:57 +02:00
statx.h
sync.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
sync.h
tctx.c	io_uring: Add io_uring_setup flag to pre-register ring fd and never install it	2023-05-16 08:06:00 -06:00
tctx.h	io_uring: simplify __io_uring_add_tctx_node	2022-10-07 12:25:30 -06:00
timeout.c	io_uring: fix io_match_task must_hold	2024-07-24 08:01:49 -06:00
timeout.h	io_uring: remove unused return from io_disarm_next	2022-09-21 13:15:01 -06:00
truncate.c	io_uring: add support for ftruncate	2024-02-09 09:04:39 -07:00
truncate.h	io_uring: add support for ftruncate	2024-02-09 09:04:39 -07:00
uring_cmd.c	io_uring/cmd: expose iowq to cmds	2024-09-11 10:44:10 -06:00
uring_cmd.h	io_uring/alloc_cache: switch to array based caching	2024-04-15 08:10:25 -06:00
waitid.c	io_uring: remove struct io_tw_state::locked	2024-04-15 08:10:24 -06:00
waitid.h	io_uring: add IORING_OP_WAITID support	2023-09-21 12:04:45 -06:00
xattr.c	vfs: retire user_path_at_empty and drop empty arg from getname_flags	2024-06-05 17:03:57 +02:00
xattr.h