Jens Axboe d38a9c04c0 block: only check previous entry for plug merge attempt ... Currently we scan the entire plug list, which is potentially very expensive. In an IOPS bound workload, we can drive about 5.6M IOPS with merging enabled, and profiling shows that the plug merge check is the (by far) most expensive thing we're doing: Overhead Command Shared Object Symbol + 20.89% io_uring [kernel.vmlinux] [k] blk_attempt_plug_merge + 4.98% io_uring [kernel.vmlinux] [k] io_submit_sqes + 4.78% io_uring [kernel.vmlinux] [k] blkdev_direct_IO + 4.61% io_uring [kernel.vmlinux] [k] blk_mq_submit_bio Instead of browsing the whole list, just check the previously inserted entry. That is enough for a naive merge check and will catch most cases, and for devices that need full merging, the IO scheduler attached to such devices will do that anyway. The plug merge is meant to be an inexpensive check to avoid getting a request, but if we repeatedly scan the list for every single insert, it is very much not a cheap check. With this patch, the workload instead runs at ~7.0M IOPS, providing a 25% improvement. Disabling merging entirely yields another 5% improvement. Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>		2021-10-18 06:17:35 -06:00
..
partitions	block: move menu "Partition type" to block/partitions/Kconfig	2021-10-18 06:17:35 -06:00
badblocks.c	treewide: Use fallthrough pseudo-keyword	2020-08-23 17:36:59 -05:00
bdev.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00
bfq-cgroup.c	block: move elevator.h to block/	2021-10-18 06:17:01 -06:00
bfq-iosched.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
bfq-iosched.h	block, bfq: cleanup the repeated declaration	2021-08-25 06:45:33 -06:00
bfq-wf2q.c	block: Introduce IOPRIO_NR_LEVELS	2021-08-18 07:21:12 -06:00
bio-integrity.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00
bio.c	block: mark bio_truncate static	2021-10-18 06:17:35 -06:00
blk-cgroup-rwstat.c	blk-cgroup: Fix the recursive blkg rwstat	2021-03-05 11:32:15 -07:00
blk-cgroup-rwstat.h	blk-cgroup: separate out blkg_rwstat under CONFIG_BLK_CGROUP_RWSTAT	2019-11-07 12:28:13 -07:00
blk-cgroup.c	block: move blk-throtl fast path inline	2021-10-18 06:17:03 -06:00
blk-core.c	block: inline hot paths of blk_account_io_*()	2021-10-18 06:17:34 -06:00
blk-crypto-fallback.c	block: move struct request to blk-mq.h	2021-10-18 06:17:02 -06:00
blk-crypto-internal.h	block: move struct request to blk-mq.h	2021-10-18 06:17:02 -06:00
blk-crypto.c	blk-crypto: fix check for too-large dun_bytes	2021-08-25 06:45:00 -06:00
blk-exec.c	block: return errors from blk_execute_rq()	2021-06-30 15:35:45 -06:00
blk-flush.c	blk-mq: fix is_flush_rq	2021-08-17 20:17:34 -06:00
blk-integrity.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00
blk-ioc.c	block: remove retry loop in ioc_release_fn()	2020-07-16 10:22:15 -06:00
blk-iocost.c	for-5.15/block-2021-08-30	2021-08-30 18:52:11 -07:00
blk-iolatency.c	mm: don't include <linux/blk-cgroup.h> in <linux/backing-dev.h>	2021-10-18 06:17:01 -06:00
blk-ioprio.c	block: Introduce the ioprio rq-qos policy	2021-06-21 15:03:40 -06:00
blk-ioprio.h	block: Introduce the ioprio rq-qos policy	2021-06-21 15:03:40 -06:00
blk-lib.c	block: export blk_next_bio()	2021-06-17 15:51:20 +02:00
blk-map.c	Merge branch 'akpm' (patches from Andrew)	2021-09-03 10:08:28 -07:00
blk-merge.c	block: only check previous entry for plug merge attempt	2021-10-18 06:17:35 -06:00
blk-mq-cpumap.c	blk-mq: remove the calling of local_memory_node()	2020-10-20 07:08:17 -06:00
blk-mq-debugfs-zoned.c	block: Cleanup license notice	2019-01-17 21:21:40 -07:00
blk-mq-debugfs.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
blk-mq-debugfs.h	blk-mq: no need to check return value of debugfs_create functions	2019-06-13 03:00:30 -06:00
blk-mq-pci.c	block: Fix blk_mq_*_map_queues() kernel-doc headers	2019-05-31 15:12:34 -06:00
blk-mq-rdma.c	block: Fix blk_mq_*_map_queues() kernel-doc headers	2019-05-31 15:12:34 -06:00
blk-mq-sched.c	blk-mq: inline hot part of __blk_mq_sched_restart	2021-10-18 06:17:34 -06:00
blk-mq-sched.h	block: move bio_mergeable out of bio.h	2021-10-18 06:17:35 -06:00
blk-mq-sysfs.c	block: remove blk-mq-sysfs dead code	2021-08-02 13:37:29 -06:00
blk-mq-tag.c	block: improve batched tag allocation	2021-10-18 06:17:35 -06:00
blk-mq-tag.h	block: improve batched tag allocation	2021-10-18 06:17:35 -06:00
blk-mq-virtio.c	blk-mq: Fix typo in comment	2020-03-17 20:55:21 +01:00
blk-mq.c	block: improve batched tag allocation	2021-10-18 06:17:35 -06:00
blk-mq.h	block: pre-allocate requests if plug is started and is a batch	2021-10-18 06:17:03 -06:00
blk-pm.c	scsi: block: Fix a race in the runtime power management code	2020-12-09 11:41:41 -05:00
blk-pm.h	block: Remove unused blk_pm_*() function definitions	2021-02-22 06:33:48 -07:00
blk-rq-qos.c	rq-qos: fix missed wake-ups in rq_qos_throttle try two	2021-06-08 15:12:57 -06:00
blk-rq-qos.h	block: Introduce the ioprio rq-qos policy	2021-06-21 15:03:40 -06:00
blk-settings.c	block: add an explicit ->disk backpointer to the request_queue	2021-08-23 12:54:31 -06:00
blk-stat.c	blk-stat: make q->stats->lock irqsafe	2020-09-01 16:48:46 -06:00
blk-stat.h	block: deactivate blk_stat timer in wbt_disable_default()	2018-12-12 06:47:51 -07:00
blk-sysfs.c	block: move blk-throtl fast path inline	2021-10-18 06:17:03 -06:00
blk-throttle.c	block: move blk-throtl fast path inline	2021-10-18 06:17:03 -06:00
blk-throttle.h	block: move blk-throtl fast path inline	2021-10-18 06:17:03 -06:00
blk-timeout.c	block: blk-timeout: delete duplicated word	2020-07-31 16:29:47 -06:00
blk-wbt.c	block: add an explicit ->disk backpointer to the request_queue	2021-08-23 12:54:31 -06:00
blk-wbt.h	blk-wbt: introduce a new disable state to prevent false positive by rwb_enabled()	2021-06-21 15:03:41 -06:00
blk-zoned.c	blk-zoned: allow BLKREPORTZONE without CAP_SYS_ADMIN	2021-08-24 10:12:36 -06:00
blk.h	block: inline hot paths of blk_account_io_*()	2021-10-18 06:17:34 -06:00
bounce.c	mm: don't include <linux/blk-cgroup.h> in <linux/backing-dev.h>	2021-10-18 06:17:01 -06:00
bsg-lib.c	scsi: bsg-lib: Fix commands without data transfer in bsg_transport_sg_io_fn()	2021-08-01 13:21:40 -04:00
bsg.c	scsi: bsg: Fix device unregistration	2021-09-14 00:22:15 -04:00
disk-events.c	block: return errors from disk_alloc_events	2021-08-23 12:55:45 -06:00
elevator.c	blk-mq: Change shared sbitmap naming to shared tags	2021-10-18 06:17:03 -06:00
elevator.h	block: move elevator.h to block/	2021-10-18 06:17:01 -06:00
fops.c	block: merge block_ioctl into blkdev_ioctl	2021-10-18 06:17:34 -06:00
genhd.c	block: drop unused includes in <linux/genhd.h>	2021-10-18 06:17:02 -06:00
holder.c	block: drop unused includes in <linux/genhd.h>	2021-10-18 06:17:02 -06:00
ioctl.c	block: merge block_ioctl into blkdev_ioctl	2021-10-18 06:17:34 -06:00
ioprio.c	block: fix default IO priority handling	2021-08-18 07:23:15 -06:00
Kconfig	block: move menu "Partition type" to block/partitions/Kconfig	2021-10-18 06:17:35 -06:00
Kconfig.iosched	block: simplify Kconfig files	2021-10-18 06:17:35 -06:00
keyslot-manager.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00
kyber-iosched.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
Makefile	block: move CONFIG_BLOCK guard to top Makefile	2021-10-18 06:17:35 -06:00
mq-deadline.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
opal_proto.h	block: sed-opal: Change the check condition for regular session validity	2020-03-12 08:00:10 -06:00
sed-opal.c	block: sed-opal: Change the check condition for regular session validity	2020-03-12 08:00:10 -06:00
t10-pi.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00