Pavel Begunkov e94f68527a block: kill extra rcu lock/unlock in queue enter ... blk_try_enter_queue() already takes rcu_read_lock/unlock, so we can avoid the second pair in percpu_ref_tryget_live(), use a newly added percpu_ref_tryget_live_rcu(). As rcu_read_lock/unlock imply barrier()s, it's pretty noticeable, especially for for !CONFIG_PREEMPT_RCU (default for some distributions), where __rcu_read_lock/unlock() are not inlined. 3.20% io_uring [kernel.vmlinux] [k] __rcu_read_unlock 3.05% io_uring [kernel.vmlinux] [k] __rcu_read_lock 2.52% io_uring [kernel.vmlinux] [k] __rcu_read_unlock 2.28% io_uring [kernel.vmlinux] [k] __rcu_read_lock Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/6b11c67ea495ed9d44f067622d852de4a510ce65.1634822969.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>		2021-10-21 08:37:26 -06:00
..
partitions	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
badblocks.c	treewide: Use fallthrough pseudo-keyword	2020-08-23 17:36:59 -05:00
bdev.c	fs: bdev: fix conflicting comment from lookup_bdev	2021-10-21 08:24:14 -06:00
bfq-cgroup.c	block, bfq: fix UAF problem in bfqg_stats_init()	2021-10-19 15:18:30 -06:00
bfq-iosched.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
bfq-iosched.h	block, bfq: cleanup the repeated declaration	2021-08-25 06:45:33 -06:00
bfq-wf2q.c	block: Introduce IOPRIO_NR_LEVELS	2021-08-18 07:21:12 -06:00
bio-integrity.c	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
bio.c	block: inline a part of bio_release_pages()	2021-10-20 08:08:07 -06:00
blk-cgroup-rwstat.c	blk-cgroup: Fix the recursive blkg rwstat	2021-03-05 11:32:15 -07:00
blk-cgroup-rwstat.h	blk-cgroup: separate out blkg_rwstat under CONFIG_BLK_CGROUP_RWSTAT	2019-11-07 12:28:13 -07:00
blk-cgroup.c	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
blk-core.c	block: kill extra rcu lock/unlock in queue enter	2021-10-21 08:37:26 -06:00
blk-crypto-fallback.c	block: move struct request to blk-mq.h	2021-10-18 06:17:02 -06:00
blk-crypto-internal.h	block: move struct request to blk-mq.h	2021-10-18 06:17:02 -06:00
blk-crypto.c	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
blk-exec.c	block: add a struct io_comp_batch argument to fops->iopoll()	2021-10-18 14:40:40 -06:00
blk-flush.c	blk-mq: don't handle non-flush requests in blk_insert_flush	2021-10-19 11:10:09 -06:00
blk-integrity.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00
blk-ioc.c	block: remove retry loop in ioc_release_fn()	2020-07-16 10:22:15 -06:00
blk-iocost.c	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
blk-iolatency.c	mm: don't include <linux/blk-cgroup.h> in <linux/backing-dev.h>	2021-10-18 06:17:01 -06:00
blk-ioprio.c	block: Introduce the ioprio rq-qos policy	2021-06-21 15:03:40 -06:00
blk-ioprio.h	block: Introduce the ioprio rq-qos policy	2021-06-21 15:03:40 -06:00
blk-lib.c	block: export blk_next_bio()	2021-06-17 15:51:20 +02:00
blk-map.c	Merge branch 'akpm' (patches from Andrew)	2021-09-03 10:08:28 -07:00
blk-merge.c	block: convert leftovers to bdev_get_queue	2021-10-20 08:08:03 -06:00
blk-mq-cpumap.c	blk-mq: remove the calling of local_memory_node()	2020-10-20 07:08:17 -06:00
blk-mq-debugfs-zoned.c	block: Cleanup license notice	2019-01-17 21:21:40 -07:00
blk-mq-debugfs.c	block: remove some blk_mq_hw_ctx debugfs entries	2021-10-18 14:40:33 -06:00
blk-mq-debugfs.h	blk-mq: no need to check return value of debugfs_create functions	2019-06-13 03:00:30 -06:00
blk-mq-pci.c	block: Fix blk_mq_*_map_queues() kernel-doc headers	2019-05-31 15:12:34 -06:00
blk-mq-rdma.c	block: Fix blk_mq_*_map_queues() kernel-doc headers	2019-05-31 15:12:34 -06:00
blk-mq-sched.c	block: clean up blk_mq_submit_bio() merging	2021-10-21 08:27:17 -06:00
blk-mq-sched.h	block: clean up blk_mq_submit_bio() merging	2021-10-21 08:27:17 -06:00
blk-mq-sysfs.c	block: remove blk-mq-sysfs dead code	2021-08-02 13:37:29 -06:00
blk-mq-tag.c	blk-mq: Fix blk_mq_tagset_busy_iter() for shared tags	2021-10-21 08:21:52 -06:00
blk-mq-tag.h	block: move blk_mq_tag_to_rq() inline	2021-10-19 05:55:41 -06:00
blk-mq-virtio.c	blk-mq: Fix typo in comment	2020-03-17 20:55:21 +01:00
blk-mq.c	block: clean up blk_mq_submit_bio() merging	2021-10-21 08:27:17 -06:00
blk-mq.h	blk-mq: move blk_mq_flush_plug_list to block/blk-mq.h	2021-10-20 09:56:11 -06:00
blk-pm.c	scsi: block: Fix a race in the runtime power management code	2020-12-09 11:41:41 -05:00
blk-pm.h	block: Remove unused blk_pm_*() function definitions	2021-02-22 06:33:48 -07:00
blk-rq-qos.c	rq-qos: fix missed wake-ups in rq_qos_throttle try two	2021-06-08 15:12:57 -06:00
blk-rq-qos.h	block: only mark bio as tracked if it really is tracked	2021-10-18 08:50:47 -06:00
blk-settings.c	block: add an explicit ->disk backpointer to the request_queue	2021-08-23 12:54:31 -06:00
blk-stat.c	blk-stat: make q->stats->lock irqsafe	2020-09-01 16:48:46 -06:00
blk-stat.h	block: deactivate blk_stat timer in wbt_disable_default()	2018-12-12 06:47:51 -07:00
blk-sysfs.c	block: don't allow writing to the poll queue attribute	2021-10-18 06:17:36 -06:00
blk-throttle.c	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
blk-throttle.h	block: move blk-throtl fast path inline	2021-10-18 06:17:03 -06:00
blk-timeout.c	block: blk-timeout: delete duplicated word	2020-07-31 16:29:47 -06:00
blk-wbt.c	blk-wbt: prevent NULL pointer dereference in wb_timer_fn	2021-10-19 06:13:41 -06:00
blk-wbt.h	blk-wbt: introduce a new disable state to prevent false positive by rwb_enabled()	2021-06-21 15:03:41 -06:00
blk-zoned.c	blk-zoned: allow BLKREPORTZONE without CAP_SYS_ADMIN	2021-08-24 10:12:36 -06:00
blk.h	blk-mq: don't handle non-flush requests in blk_insert_flush	2021-10-19 11:10:09 -06:00
bounce.c	mm: don't include <linux/blk-cgroup.h> in <linux/backing-dev.h>	2021-10-18 06:17:01 -06:00
bsg-lib.c	scsi: bsg-lib: Fix commands without data transfer in bsg_transport_sg_io_fn()	2021-08-01 13:21:40 -04:00
bsg.c	scsi: bsg: Fix device unregistration	2021-09-14 00:22:15 -04:00
disk-events.c	block: return errors from disk_alloc_events	2021-08-23 12:55:45 -06:00
elevator.c	blk-mq: Change shared sbitmap naming to shared tags	2021-10-18 06:17:03 -06:00
elevator.h	block: move elevator.h to block/	2021-10-18 06:17:01 -06:00
fops.c	block: convert fops.c magic constants to SHIFT_SECTOR	2021-10-21 08:27:17 -06:00
genhd.c	block: move bdev_read_only() into the header	2021-10-19 05:53:22 -06:00
holder.c	block: drop unused includes in <linux/genhd.h>	2021-10-18 06:17:02 -06:00
ioctl.c	block: merge block_ioctl into blkdev_ioctl	2021-10-18 06:17:34 -06:00
ioprio.c	block: fix default IO priority handling	2021-08-18 07:23:15 -06:00
Kconfig	block: move menu "Partition type" to block/partitions/Kconfig	2021-10-18 06:17:35 -06:00
Kconfig.iosched	block: simplify Kconfig files	2021-10-18 06:17:35 -06:00
keyslot-manager.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00
kyber-iosched.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
Makefile	block: move CONFIG_BLOCK guard to top Makefile	2021-10-18 06:17:35 -06:00
mq-deadline.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
opal_proto.h	block: sed-opal: Change the check condition for regular session validity	2020-03-12 08:00:10 -06:00
sed-opal.c	block: sed-opal: Change the check condition for regular session validity	2020-03-12 08:00:10 -06:00
t10-pi.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00