linux/block
Tejun Heo f733164829 blk-cgroup: reimplement basic IO stats using cgroup rstat
blk-cgroup has been using blkg_rwstat to track basic IO stats.
Unfortunately, reading recursive stats scales badly as itinvolves
walking all descendants.  On systems with a huge number of cgroups
(dead or alive), this can lead to substantial CPU cost when reading IO
stats.

This patch reimplements basic IO stats using cgroup rstat which uses
more memory but makes recursive stat reading O(# descendants which
have been active since last reading) instead of O(# descendants).

* blk-cgroup core no longer uses sync/async stats.  Introduce new stat
  enums - BLKG_IOSTAT_{READ|WRITE|DISCARD}.

* Add blkg_iostat[_set] which encapsulates byte and io stats, last
  values for propagation delta calculation and u64_stats_sync for
  correctness on 32bit archs.

* Update the new percpu stat counters directly and implement
  blkcg_rstat_flush() to implement propagation.

* blkg_print_stat() can now bring the stats up to date by calling
  cgroup_rstat_flush() and print them instead of directly summing up
  all descendants.

* It now allocates 96 bytes per cpu.  It used to be 40 bytes.

Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Dan Schatzberg <dschatzberg@fb.com>
Cc: Daniel Xu <dlxu@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-11-07 12:28:13 -07:00
..
partitions docs: admin-guide: add a series of orphaned documents 2019-07-15 11:03:02 -03:00
badblocks.c block: switch all files cleared marked as GPLv2 to SPDX tags 2019-04-30 16:11:57 -06:00
bfq-cgroup.c bfq-iosched: stop using blkg->stat_bytes and ->stat_ios 2019-11-07 12:28:13 -07:00
bfq-iosched.c bfq-iosched: stop using blkg->stat_bytes and ->stat_ios 2019-11-07 12:28:13 -07:00
bfq-iosched.h bfq-iosched: stop using blkg->stat_bytes and ->stat_ios 2019-11-07 12:28:13 -07:00
bfq-wf2q.c bfq: Fix the missing barrier in __bfq_entity_update_weight_prio 2019-09-06 14:33:48 -06:00
bio-integrity.c block/bio-integrity: fix a memory leak bug 2019-07-11 20:01:21 -06:00
bio.c block: move same page handling from __bio_add_pc_page to the callers 2019-08-22 07:14:39 -06:00
blk-cgroup.c blk-cgroup: reimplement basic IO stats using cgroup rstat 2019-11-07 12:28:13 -07:00
blk-core.c block: add zone open, close and finish operations 2019-11-07 06:31:48 -07:00
blk-exec.c block: account statistics for passthrough requests 2019-10-10 17:52:31 -06:00
blk-flush.c block: fix null pointer dereference in blk_mq_rq_timed_out() 2019-09-27 07:01:25 -06:00
blk-integrity.c block: centralize PI remapping logic to the block layer 2019-09-17 20:03:49 -06:00
blk-ioc.c block: remove the queue_lock indirection 2018-11-15 12:17:28 -07:00
blk-iocost.c iocost: don't nest spin_lock_irq in ioc_weight_write() 2019-10-31 11:40:57 -06:00
blk-iolatency.c blkcg: s/RQ_QOS_CGROUP/RQ_QOS_LATENCY/ 2019-08-28 21:17:08 -06:00
blk-lib.c block: fix 32 bit overflow in __blkdev_issue_discard() 2018-11-14 08:17:18 -07:00
blk-map.c block: remove the bi_phys_segments field in struct bio 2019-06-20 10:29:22 -06:00
blk-merge.c block: avoid blk_bio_segment_split for small I/O operations 2019-11-04 17:13:54 -07:00
blk-mq-cpumap.c blk-mq: balance mapping between present CPUs and queues 2019-08-04 21:43:12 -06:00
blk-mq-debugfs-zoned.c block: Cleanup license notice 2019-01-17 21:21:40 -07:00
blk-mq-debugfs.c for-5.3/block-20190708 2019-07-09 10:45:06 -07:00
blk-mq-debugfs.h blk-mq: no need to check return value of debugfs_create functions 2019-06-13 03:00:30 -06:00
blk-mq-pci.c block: Fix blk_mq_*_map_queues() kernel-doc headers 2019-05-31 15:12:34 -06:00
blk-mq-rdma.c block: Fix blk_mq_*_map_queues() kernel-doc headers 2019-05-31 15:12:34 -06:00
blk-mq-sched.c blk-mq: move lockdep_assert_held() into elevator_exit 2019-09-26 00:45:05 -06:00
blk-mq-sched.h block: blk-mq: Remove blk_mq_sched_started_request and started_request 2019-07-23 07:25:09 -06:00
blk-mq-sysfs.c blk-mq: make sure that line break can be printed 2019-11-04 07:14:10 -07:00
blk-mq-tag.c blk-mq: introduce blk_mq_tagset_wait_completed_request() 2019-08-04 21:41:29 -06:00
blk-mq-tag.h Merge branch 'for-4.15/block' of git://git.kernel.dk/linux-block 2017-11-14 15:32:19 -08:00
blk-mq-virtio.c block: Fix blk_mq_*_map_queues() kernel-doc headers 2019-05-31 15:12:34 -06:00
blk-mq.c blk-mq: Make blk_mq_run_hw_queue() return void 2019-11-01 08:42:41 -06:00
blk-mq.h blk-mq: Inline status checkers 2019-10-07 08:31:59 -06:00
blk-pm.c block: bypass blk_set_runtime_active for uninitialized q->dev 2019-09-12 07:11:56 -06:00
blk-pm.h block: remove the queue_lock indirection 2018-11-15 12:17:28 -07:00
blk-rq-qos.c blk-wbt: fix performance regression in wbt scale_up/scale_down 2019-10-06 09:26:41 -06:00
blk-rq-qos.h blk-rq-qos: fix first node deletion of rq_qos_del() 2019-10-15 10:13:13 -06:00
blk-settings.c dma-mapping updates for 5.4: 2019-09-19 13:27:23 -07:00
blk-softirq.c block: remove a few unused exports 2018-11-15 12:13:25 -07:00
blk-stat.c blk-stat: Optimise blk_stat_add() 2019-10-07 21:19:10 -06:00
blk-stat.h block: deactivate blk_stat timer in wbt_disable_default() 2018-12-12 06:47:51 -07:00
blk-sysfs.c block: Remove "dying" checks from sysfs callbacks 2019-10-07 08:31:59 -06:00
blk-throttle.c blk-throtl: stop using blkg->stat_bytes and ->stat_ios 2019-11-07 12:28:13 -07:00
blk-timeout.c block: add SPDX tags to block layer files missing licensing information 2019-04-30 16:12:03 -06:00
blk-wbt.c blk-wbt: fix performance regression in wbt scale_up/scale_down 2019-10-06 09:26:41 -06:00
blk-wbt.h block/rq_qos: implement rq_qos_ops->queue_depth_changed() 2019-08-28 21:17:07 -06:00
blk-zoned.c block: add zone open, close and finish ioctl support 2019-11-07 06:31:50 -07:00
blk.h block: account statistics for passthrough requests 2019-10-10 17:52:31 -06:00
bounce.c block: remove the i argument to bio_for_each_segment_all 2019-04-30 09:26:13 -06:00
bsg-lib.c block: drop device references in bsg_queue_rq() 2019-09-23 11:17:24 -06:00
bsg.c block: switch all files cleared marked as GPLv2 to SPDX tags 2019-04-30 16:11:57 -06:00
cmdline-parser.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
compat_ioctl.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
elevator.c Merge branch 'for-linus' into for-5.5/block 2019-11-07 12:27:19 -07:00
genhd.c block: Delay default elevator initialization 2019-09-05 19:52:34 -06:00
ioctl.c block: add zone open, close and finish ioctl support 2019-11-07 06:31:50 -07:00
ioprio.c docs: block: convert to ReST 2019-07-15 09:20:27 -03:00
Kconfig blkcg: implement blk-iocost 2019-08-28 21:17:12 -06:00
Kconfig.iosched docs: block: convert to ReST 2019-07-15 09:20:27 -03:00
kyber-iosched.c blk-mq: remove blk_mq_put_ctx() 2019-07-02 21:03:27 -06:00
Makefile blkcg: implement blk-iocost 2019-08-28 21:17:12 -06:00
mq-deadline.c block: Introduce elevator features 2019-09-05 19:52:33 -06:00
opal_proto.h block: sed-opal: Introduce Opal Datastore UID 2019-11-04 07:11:32 -07:00
partition-generic.c block: fix use-after-free on gendisk 2019-04-22 09:48:12 -06:00
scsi_ioctl.c block: switch all files cleared marked as GPLv2 to SPDX tags 2019-04-30 16:11:57 -06:00
sed-opal.c block: sed-opal: Introduce Opal Datastore UID 2019-11-04 07:11:32 -07:00
t10-pi.c block: Fix three kernel-doc warnings 2019-10-07 08:31:59 -06:00