Jens Axboe 4e5dff41be blk-mq: improve heavily contended tag case ... Even with a number of waitqueues, we can get into a situation where we are heavily contended on the waitqueue lock. I got a report on spc1 where we're spending seconds doing this. Arguably the use case is nasty, I reproduce it with one device and 1000 threads banging on the device. But that doesn't mean we shouldn't be handling it better. What ends up happening is that a thread will fail to get a tag, add itself to the waitqueue, and subsequently get woken up when a tag is freed - only to find itself going back to sleep on the waitqueue. Instead of waking all threads, use an exclusive wait and wake up our sbitmap batch count instead. This seems to work well for me (massive improvement for this use case), and it survives basic testing. But I haven't fully verified it yet. An additional improvement is running the queue and checking for a new tag BEFORE needing to add ourselves to the waitqueue. Signed-off-by: Jens Axboe <axboe@kernel.dk>		2017-12-22 11:09:37 -07:00
..
partitions	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
badblocks.c	badblocks: fix wrong return value in badblocks_set if badblocks are disabled	2017-11-03 11:29:50 -07:00
bfq-cgroup.c	block, bfq: move debug blkio stats behind CONFIG_DEBUG_BLK_CGROUP	2017-11-14 20:13:33 -07:00
bfq-iosched.c	block, bfq: move debug blkio stats behind CONFIG_DEBUG_BLK_CGROUP	2017-11-14 20:13:33 -07:00
bfq-iosched.h	block, bfq: move debug blkio stats behind CONFIG_DEBUG_BLK_CGROUP	2017-11-14 20:13:33 -07:00
bfq-wf2q.c	block, bfq: update blkio stats outside the scheduler lock	2017-11-14 20:13:33 -07:00
bio-integrity.c	block: remove unnecessary NULL checks in bioset_integrity_free()	2017-10-06 13:03:12 -06:00
bio.c	block: remove useless assignment in bio_split	2017-11-22 11:26:05 -07:00
blk-cgroup.c	blkcg: add sanity check for blkcg policy operations	2017-11-04 12:31:15 -06:00
blk-core.c	block/laptop_mode: Convert timers to use timer_setup()	2017-11-21 15:46:44 -08:00
blk-exec.c	block: introduce new block status code type	2017-06-09 09:27:32 -06:00
blk-flush.c	blk-mq: don't allocate driver tag upfront for flush rq	2017-11-04 12:40:13 -06:00
blk-integrity.c	block: switch bios to blk_status_t	2017-06-09 09:27:32 -06:00
blk-ioc.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-lib.c	Merge branch 'for-4.15/block' of git://git.kernel.dk/linux-block	2017-11-14 15:32:19 -08:00
blk-map.c	Merge branch 'work.iov_iter' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2017-11-17 12:08:18 -08:00
blk-merge.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-mq-cpumap.c	blk-mq: map queues to all present CPUs	2017-07-24 10:01:31 -06:00
blk-mq-debugfs.c	Merge branch 'for-4.15/block' of git://git.kernel.dk/linux-block	2017-11-14 15:32:19 -08:00
blk-mq-debugfs.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-mq-pci.c	blk-mq-pci: add a fallback when pci_irq_get_affinity returns NULL	2017-08-18 08:08:14 -06:00
blk-mq-rdma.c	block: Add rdma affinity based queue mapping helper	2017-08-08 14:58:03 -04:00
blk-mq-sched.c	blk-mq: only run the hardware queue if IO is pending	2017-11-10 19:55:57 -07:00
blk-mq-sched.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-mq-sysfs.c	blk-mq: untangle debugfs and sysfs	2017-05-04 08:24:13 -06:00
blk-mq-tag.c	blk-mq: improve heavily contended tag case	2017-12-22 11:09:37 -07:00
blk-mq-tag.h	Merge branch 'for-4.15/block' of git://git.kernel.dk/linux-block	2017-11-14 15:32:19 -08:00
blk-mq-virtio.c	blk-mq: provide a default queue mapping for virtio device	2017-02-27 20:54:05 +02:00
blk-mq.c	block/blk-mq.c: use kmalloc_array_node()	2017-11-15 18:21:02 -08:00
blk-mq.h	Merge branch 'for-4.15/block' of git://git.kernel.dk/linux-block	2017-11-14 15:32:19 -08:00
blk-settings.c	block: remove __bio_kmap_atomic	2017-11-10 19:53:25 -07:00
blk-softirq.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-stat.c	treewide: setup_timer() -> timer_setup()	2017-11-21 15:57:07 -08:00
blk-stat.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-sysfs.c	blk-sysfs: remove NULL pointer checking in queue_wb_lat_store	2017-11-23 22:00:17 -07:00
blk-tag.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-throttle.c	treewide: setup_timer() -> timer_setup()	2017-11-21 15:57:07 -08:00
blk-timeout.c	block: Fix a race between blk_cleanup_queue() and timeout handling	2017-10-30 13:28:10 -06:00
blk-wbt.c	blk-wbt: fix comments typo	2017-11-23 22:00:20 -07:00
blk-wbt.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
blk-zoned.c	block: replace bi_bdev with a gendisk pointer and partitions index	2017-08-23 12:49:55 -06:00
blk.h	Merge branch 'for-4.15/block' of git://git.kernel.dk/linux-block	2017-11-14 15:32:19 -08:00
bounce.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
bsg-lib.c	bsg-lib: fix use-after-free under memory-pressure	2017-10-04 08:35:04 -06:00
bsg.c	block: pass full fmode_t to blk_verify_command	2017-11-10 19:53:25 -07:00
cfq-iosched.c	block/cfq: cache rightmost rb_node	2017-09-08 18:26:49 -07:00
cmdline-parser.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
compat_ioctl.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
deadline-iosched.c	block, scheduler: convert xxx_var_store to void	2017-08-28 10:01:08 -06:00
elevator.c	elevator: allow name aliases	2017-10-25 12:36:50 -06:00
genhd.c	block: genhd.c: fix message typo	2017-11-19 11:02:19 -07:00
ioctl.c	block: move CAP_SYS_ADMIN check in blkdev_roset()	2017-10-25 12:25:00 -06:00
ioprio.c	block: Add fallthrough markers to switch statements	2017-06-21 11:46:07 -06:00
Kconfig	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
Kconfig.iosched	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
kyber-iosched.c	block: kyber: check if there are requests in ctx in kyber_has_work()	2017-11-01 08:20:02 -06:00
Makefile	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mq-deadline.c	mq-deadline: add 'deadline' as a name alias	2017-10-25 12:36:55 -06:00
noop-iosched.c	block: move existing elevator ops to union	2017-01-17 10:03:33 -07:00
opal_proto.h	block: sed-opal: Set MBRDone on S3 resume path if TPER is MBREnabled	2017-09-11 09:45:52 -06:00
partition-generic.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
scsi_ioctl.c	block: pass full fmode_t to blk_verify_command	2017-11-10 19:53:25 -07:00
sed-opal.c	block: sed-opal: Set MBRDone on S3 resume path if TPER is MBREnabled	2017-09-11 09:45:52 -06:00
t10-pi.c	t10-pi: Move opencoded contants to common header	2017-07-03 16:56:25 -06:00