linux/lib
Jens Axboe 4e5dff41be blk-mq: improve heavily contended tag case
Even with a number of waitqueues, we can get into a situation where we
are heavily contended on the waitqueue lock. I got a report on spc1
where we're spending seconds doing this. Arguably the use case is nasty,
I reproduce it with one device and 1000 threads banging on the device.
But that doesn't mean we shouldn't be handling it better.

What ends up happening is that a thread will fail to get a tag, add
itself to the waitqueue, and subsequently get woken up when a tag is
freed - only to find itself going back to sleep on the waitqueue.

Instead of waking all threads, use an exclusive wait and wake up our
sbitmap batch count instead. This seems to work well for me (massive
improvement for this use case), and it survives basic testing. But I
haven't fully verified it yet.

An additional improvement is running the queue and checking for a new
tag BEFORE needing to add ourselves to the waitqueue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-12-22 11:09:37 -07:00
..
842 License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
fonts License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lz4 lib/lz4: make arrays static const, reduces object code size 2017-10-03 17:54:25 -07:00
lzo License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
mpi Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 2017-11-14 10:52:09 -08:00
raid6 License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
reed_solomon
xz lib/xz: Add fall-through comments to a switch statement 2017-10-12 15:10:12 +02:00
zlib_deflate
zlib_inflate lib/zlib_inflate/inftrees.c: fix potential buffer overflow 2017-05-08 17:15:12 -07:00
zstd lib: Add zstd modules 2017-08-15 09:02:08 -07:00
.gitignore
argv_split.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ashldi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
ashrdi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
asn1_decoder.c ASN.1: check for error from ASN1_OP_END__ACT actions 2017-12-08 15:13:27 +00:00
assoc_array.c Merge branch 'linus' into locking/core, to resolve conflicts 2017-11-07 10:32:44 +01:00
atomic64_test.c lib/atomic64_test.c: add a test that atomic64_inc_not_zero() returns an int 2017-07-14 15:05:13 -07:00
atomic64.c locking/atomic: Implement atomic{,64,_long}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() 2016-06-16 10:48:32 +02:00
audit.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
bcd.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
bch.c
bitmap.c documentation: kernel-api: add more info on bitmap functions 2017-10-19 13:01:40 -06:00
bitrev.c
bsearch.c lib/bsearch.c: micro-optimize pivot position calculation 2017-07-10 16:32:35 -07:00
btree.c
bug.c bug: define the "cut here" string in a single place 2017-11-17 16:10:01 -08:00
build_OID_registry
bust_spinlocks.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
chacha20.c random: replace non-blocking pool with a Chacha20-based CRNG 2016-07-03 00:57:23 -04:00
check_signature.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
checksum.c
clz_ctz.c
clz_tab.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
cmdline.c lib/cmdline.c: remove meaningless comment 2017-09-08 18:26:49 -07:00
cmpdi2.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
compat_audit.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
cordic.c
cpu_rmap.c
cpumask.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
crc4.c docs: clean up and add rest of CRC functions to kernel-api.rst 2017-09-26 15:01:20 -06:00
crc7.c
crc8.c docs: clean up and add rest of CRC functions to kernel-api.rst 2017-09-26 15:01:20 -06:00
crc16.c
crc32.c docs: clean up and add rest of CRC functions to kernel-api.rst 2017-09-26 15:01:20 -06:00
crc32defs.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
crc32test.c lib: add module support to crc32 tests 2017-02-24 17:46:57 -08:00
crc-ccitt.c
crc-itu-t.c
crc-t10dif.c
ctype.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
debug_info.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
debug_locks.c
debugobjects.c debugobjects: Make kmemleak ignore debug objects 2017-08-14 16:51:01 +02:00
dec_and_lock.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
decompress_bunzip2.c
decompress_inflate.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
decompress_unlz4.c lib/decompress_unlz4: change module to work with new LZ4 module version 2017-02-24 17:46:57 -08:00
decompress_unlzma.c
decompress_unlzo.c
decompress_unxz.c
decompress.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
devres.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
digsig.c lib/digsig: fix dereference of NULL user_key_payload 2017-10-12 17:16:40 +01:00
div64.c A relatively calm cycle for the docs tree again. 2017-11-13 08:25:06 -08:00
dma-debug.c lib/dma-debug.c: fix incorrect pfn calculation 2017-11-17 16:10:00 -08:00
dma-noop.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dma-virt.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dump_stack.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dynamic_debug.c dynamic-debug-howto: fix optional/omitted ending line number to be LARGE instead of 0 2017-11-17 16:10:01 -08:00
dynamic_queue_limits.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next 2017-11-15 11:56:19 -08:00
earlycpio.c lib/cpio: Make find_cpio_data()'s offset arg optional 2016-06-08 11:04:19 +02:00
errseq.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
extable.c lib/extable.c: use bsearch() library function in search_extable() 2017-07-10 16:32:35 -07:00
fault-inject.c fault-inject: fix wrong should_fail() decision in task context 2017-08-10 15:54:06 -07:00
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
fdt.c
find_bit.c lib/find_bit.c: micro-optimise find_next_*_bit 2017-02-24 17:46:57 -08:00
flex_array.c
flex_proportions.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
gcd.c lib/gcd: add kernel-doc notation 2017-10-07 10:45:14 -06:00
gen_crc32table.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
genalloc.c lib/genalloc.c: make the avail variable an atomic_long_t 2017-11-17 16:10:02 -08:00
glob.c lib: add module support to glob tests 2017-02-24 17:46:57 -08:00
globtest.c lib: add module support to glob tests 2017-02-24 17:46:57 -08:00
hexdump.c lib/hexdump.c: return -EINVAL in case of error in hex2bin() 2017-09-08 18:26:49 -07:00
hweight.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
idr.c mm, truncate: do not check mapping for every page being truncated 2017-11-15 18:21:06 -08:00
inflate.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
int_sqrt.c lib/int_sqrt: adjust comments 2017-11-17 16:10:01 -08:00
interval_tree_test.c lib/rbtree-test: lower default params 2017-11-17 16:10:02 -08:00
interval_tree.c
iomap_copy.c
iomap.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
iommu-common.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
iommu-helper.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ioremap.c License cleanup: add SPDX license identifiers to some files 2017-11-02 10:04:46 -07:00
iov_iter.c new primitive: iov_iter_for_each_range() 2017-10-11 22:36:54 -04:00
irq_poll.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
irq_regs.c
is_single_threaded.c sched/headers: Prepare to move 'init_task' and 'init_thread_union' from <linux/sched.h> to <linux/sched/task.h> 2017-03-02 08:42:38 +01:00
jedec_ddr_data.c
kasprintf.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
Kconfig This pull request contains the following core changes: 2017-11-22 20:46:06 -10:00
Kconfig.debug locking/lockdep: Remove the cross-release locking checks 2017-12-12 12:38:51 +01:00
Kconfig.kasan mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB 2016-07-28 16:07:41 -07:00
Kconfig.kgdb lib: update location of kgdb documentation 2017-05-16 08:44:22 -03:00
Kconfig.ubsan Kconfig: lib/Kconfig.ubsan fix reference to ubsan documentation 2016-12-14 16:04:08 -08:00
kfifo.c
klist.c
kobject_uevent.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2017-10-05 18:19:22 -07:00
kobject.c kobject: Export kobject_get_unless_zero() 2017-03-22 20:11:35 -06:00
kstrtox.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
kstrtox.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lcm.c
libcrc32c.c crypto: Work around deallocated stack frame reference gcc bug on sparc. 2017-06-08 17:36:03 +08:00
list_debug.c bug: switch data corruption check to __must_check 2017-02-24 17:46:56 -08:00
list_sort.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
llist.c locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE() 2017-10-25 11:01:08 +02:00
locking-selftest-hardirq.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-mutex.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-rsem.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-rtmutex.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-softirq.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-wsem.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lockref.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lru_cache.c
lshrdi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
Makefile lib: test module for find_*_bit() functions 2017-11-17 16:10:02 -08:00
memory-notifier-error-inject.c
memweight.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
muldi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
net_utils.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
netdev-notifier-error-inject.c
nlattr.c netlink: Relax attr validation for fixed length types 2017-12-07 14:00:57 -05:00
nmi_backtrace.c lib/nmi_backtrace.c: fix kernel text address leak 2017-11-17 16:10:02 -08:00
nodemask.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
notifier-error-inject.c
notifier-error-inject.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
of-reconfig-notifier-error-inject.c
oid_registry.c 509: fix printing uninitialized stack memory when OID is empty 2017-12-08 15:13:28 +00:00
once.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2017-11-04 09:26:51 +09:00
parman.c lib: Introduce priority array area manager 2017-02-03 16:35:42 -05:00
parser.c parser: add u64 number parser 2016-12-06 10:17:03 +02:00
pci_iomap.c
percpu_counter.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
percpu_ida.c sched/headers: Prepare to remove the <linux/gfp.h> include from <linux/sched.h> 2017-03-02 08:42:34 +01:00
percpu_test.c
percpu-refcount.c percpu-refcount: support synchronous switch to atomic mode. 2017-03-22 19:18:43 -07:00
plist.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/clock.h> 2017-03-02 08:42:27 +01:00
pm-notifier-error-inject.c
prime_numbers.c lib/prime_numbers: Suppress warn on kmalloc failure 2017-01-23 09:17:12 +01:00
radix-tree.c mm, truncate: do not check mapping for every page being truncated 2017-11-15 18:21:06 -08:00
random32.c treewide: Switch DEFINE_TIMER callbacks to struct timer_list * 2017-11-21 15:57:05 -08:00
ratelimit.c lib/ratelimit.c: use deferred printk() version 2017-10-03 17:54:26 -07:00
rational.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
rbtree_test.c lib/rbtree-test: lower default params 2017-11-17 16:10:02 -08:00
rbtree.c lib/rbtree,drm/mm: add rbtree_replace_node_cached() 2017-12-14 16:00:48 -08:00
reciprocal_div.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
refcount.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
rhashtable.c rhashtable: Documentation tweak 2017-09-19 15:18:33 -07:00
sbitmap.c blk-mq: improve heavily contended tag case 2017-12-22 11:09:37 -07:00
scatterlist.c lib/scatterlist: Introduce and export __sg_alloc_table_from_pages 2017-09-07 10:48:29 +01:00
seq_buf.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
sg_pool.c
sg_split.c
sha1.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
show_mem.c lib/show_mem.c: teach show_mem to work with the given nodemask 2017-02-22 16:41:30 -08:00
siphash.c siphash: implement HalfSipHash1-3 for hash tables 2017-01-09 13:58:57 -05:00
smp_processor_id.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
sort.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
stackdepot.c lib/stackdepot: export save/fetch stack for drivers 2016-11-11 08:12:37 -08:00
stmp_device.c
string_helpers.c mm: treewide: remove GFP_TEMPORARY allocation flag 2017-09-13 18:53:16 -07:00
string.c lib: add module support to string tests 2017-11-17 16:10:01 -08:00
strncpy_from_user.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
strnlen_user.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
swiotlb.c x86/mm: Add DMA support for SEV memory encryption 2017-11-07 15:35:58 +01:00
syscall.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
test_bitmap.c lib/test_bitmap.c: use ULL suffix for 64-bit constants 2017-09-13 18:53:15 -07:00
test_bpf.c bpf: add BPF_J{LT,LE,SLT,SLE} instructions 2017-08-09 16:53:56 -07:00
test_debug_virtual.c lib: add test module for CONFIG_DEBUG_VIRTUAL 2017-09-08 18:26:49 -07:00
test_find_bit.c lib: test module for find_*_bit() functions 2017-11-17 16:10:02 -08:00
test_firmware.c test_firmware: add batched firmware tests 2017-08-10 13:58:41 -07:00
test_hash.c lib/test_hash.c: fix warning in preprocessor symbol evaluation 2016-09-01 17:52:01 -07:00
test_hexdump.c
test_kasan.c lib/test: delete five error messages for failed memory allocations 2017-11-17 16:10:01 -08:00
test_kmod.c lib/test: delete five error messages for failed memory allocations 2017-11-17 16:10:01 -08:00
test_list_sort.c lib/test: delete five error messages for failed memory allocations 2017-11-17 16:10:01 -08:00
test_module.c
test_parman.c lib: fix spelling mistake: "actualy" -> "actually" 2017-02-26 11:03:38 -05:00
test_printf.c printk: hash addresses printed with %p 2017-11-29 12:09:02 +11:00
test_rhashtable.c test_rhashtable: remove initdata annotation 2017-09-21 20:40:55 -07:00
test_siphash.c siphash: implement HalfSipHash1-3 for hash tables 2017-01-09 13:58:57 -05:00
test_sort.c Revert "lib/test_sort.c: make it explicitly non-modular" 2017-05-08 17:15:10 -07:00
test_static_key_base.c
test_static_keys.c
test_string.c lib: add module support to string tests 2017-11-17 16:10:01 -08:00
test_sysctl.c test_sysctl: test against int proc_dointvec() array support 2017-07-12 16:26:00 -07:00
test_user_copy.c lib: remove check for AVR32 arch in test_user_copy 2017-05-01 09:36:30 +02:00
test_uuid.c uuid: fix incorrect uuid_equal conversion in test_uuid_test 2017-07-21 09:38:30 +02:00
test-kstrtox.c
test-string_helpers.c
textsearch.c
timerqueue.c timerqueue: Use rb_entry_safe() instead of open-coding it 2017-01-20 08:03:42 +01:00
ts_bm.c
ts_fsm.c textsearch: fix typos in library helpers 2017-10-22 03:14:07 +01:00
ts_kmp.c textsearch: fix typos in library helpers 2017-10-22 03:14:07 +01:00
ubsan.c UBSAN: fix typo in format string 2016-08-02 17:31:41 -04:00
ubsan.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ucmpdi2.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
ucs2_string.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
usercopy.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
uuid.c uuid: hoist uuid_is_null() helper from libnvdimm 2017-06-05 16:59:05 +02:00
vsprintf.c vsprintf: don't use 'restricted_pointer()' when not restricting 2017-11-29 11:28:09 -08:00
win_minmax.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
xxhash.c lib: Add xxhash module 2017-08-15 09:02:07 -07:00