renegade-project/linux-next
2
0
Fork 0
mirror of https://github.com/edk2-porting/linux-next.git synced 2024-12-30 08:04:13 +08:00
Code
108a42358a
linux-next/fs/xfs
History
Dave Chinner 108a42358a xfs: Lower CIL flush limit for large logs 
The current CIL size aggregation limit is 1/8th the log size. This
means for large logs we might be aggregating at least 250MB of dirty objects
in memory before the CIL is flushed to the journal. With CIL shadow
buffers sitting around, this means the CIL is often consuming >500MB
of temporary memory that is all allocated under GFP_NOFS conditions.

Flushing the CIL can take some time to do if there is other IO
ongoing, and can introduce substantial log force latency by itself.
It also pins the memory until the objects are in the AIL and can be
written back and reclaimed by shrinkers. Hence this threshold also
tends to determine the minimum amount of memory XFS can operate in
under heavy modification without triggering the OOM killer.

Modify the CIL space limit to prevent such huge amounts of pinned
metadata from aggregating. We can have 2MB of log IO in flight at
once, so limit aggregation to 16x this size. This threshold was
chosen as it little impact on performance (on 16-way fsmark) or log
traffic but pins a lot less memory on large logs especially under
heavy memory pressure.  An aggregation limit of 8x had 5-10%
performance degradation and a 50% increase in log throughput for
the same workload, so clearly that was too small for highly
concurrent workloads on large logs.

This was found via trace analysis of AIL behaviour. e.g. insertion
from a single CIL flush:

xfs_ail_insert: old lsn 0/0 new lsn 1/3033090 type XFS_LI_INODE flags IN_AIL

$ grep xfs_ail_insert /mnt/scratch/s.t |grep "new lsn 1/3033090" |wc -l
1721823
$

So there were 1.7 million objects inserted into the AIL from this
CIL checkpoint, the first at 2323.392108, the last at 2325.667566 which
was the end of the trace (i.e. it hadn't finished). Clearly a major
problem.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Allison Collins <allison.henderson@oracle.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
2020-03-27 08:32:54 -07:00
..
libxfs xfs: validate the realtime geometry in xfs_validate_sb_common 2020-03-27 08:32:53 -07:00
scrub xfs: prohibit fs freezing when using empty transactions 2020-03-26 08:19:24 -07:00
Kconfig treewide: Add SPDX license identifier - Makefile/Kconfig 2019-05-21 10:50:46 +02:00
kmem.c xfs: Correct comment tyops -> typos 2019-11-10 10:21:57 -08:00
kmem.h xfs: Remove kmem_zone_free() wrapper 2019-11-18 08:40:44 -08:00
Makefile xfs: introduce fake roots for ag-rooted btrees 2020-03-18 08:12:23 -07:00
mrlock.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_acl.c xfs: only allocate the buffer size actually needed in __xfs_set_acl 2020-03-02 20:55:55 -08:00
xfs_acl.h xfs: improve xfs_forget_acl 2020-03-02 20:55:55 -08:00
xfs_aops.c xfs: ratelimit xfs_discard_page messages 2020-03-02 20:55:51 -08:00
xfs_aops.h xfs: add a xfs_inode_buftarg helper 2019-10-28 08:37:54 -07:00
xfs_attr_inactive.c xfs: add a function to deal with corrupt buffers post-verifiers 2020-03-12 07:58:12 -07:00
xfs_attr_list.c xfs: add a function to deal with corrupt buffers post-verifiers 2020-03-12 07:58:12 -07:00
xfs_bio_io.c xfs: chain bios the right way around in xfs_rw_bdev 2019-07-10 10:04:16 -07:00
xfs_bmap_item.c xfs: Remove kmem_zone_free() wrapper 2019-11-18 08:40:44 -08:00
xfs_bmap_item.h xfs: merge xfs_bud_init into xfs_trans_get_bud 2019-06-28 19:27:36 -07:00
xfs_bmap_util.c xfs: remove the di_version field from struct icdinode 2020-03-19 08:48:47 -07:00
xfs_bmap_util.h xfs: simplify xfs_iomap_eof_align_last_fsb 2019-11-03 10:22:30 -08:00
xfs_buf_item.c xfs: add a new xfs_sb_version_has_v3inode helper 2020-03-19 08:47:34 -07:00
xfs_buf_item.h xfs: check log iovec size to make sure it's plausibly a buffer log format 2020-01-16 08:07:24 -08:00
xfs_buf.c xfs: xfs_buf_corruption_error should take __this_address 2020-03-12 07:58:12 -07:00
xfs_buf.h xfs: add a function to deal with corrupt buffers post-verifiers 2020-03-12 07:58:12 -07:00
xfs_dir2_readdir.c xfs: fix regression in "cleanup xfs_dir2_block_getdents" 2020-03-13 10:37:13 -07:00
xfs_discard.c xfs: remove XFS_BUF_TO_AGF 2020-03-11 09:11:39 -07:00
xfs_discard.h
xfs_dquot_item.c xfs: fix unmount hang and memory leak on shutdown during quotaoff 2020-03-18 08:12:23 -07:00
xfs_dquot_item.h xfs: factor out quotaoff intent AIL removal and memory free 2020-03-18 08:12:23 -07:00
xfs_dquot.c xfs: remove the kuid/kgid conversion wrappers 2020-03-02 20:55:50 -08:00
xfs_dquot.h xfs: remove the xfs_dq_logitem_t typedef 2019-11-13 18:22:26 -08:00
xfs_error.c xfs: xfs_buf_corruption_error should take __this_address 2020-03-12 07:58:12 -07:00
xfs_error.h xfs: xfs_buf_corruption_error should take __this_address 2020-03-12 07:58:12 -07:00
xfs_export.c xfs: remove unused header files 2019-06-28 19:30:43 -07:00
xfs_export.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_extent_busy.c xfs: cleanup use of the XFS_ALLOC_ flags 2019-11-03 10:22:31 -08:00
xfs_extent_busy.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_extfree_item.c xfs: Remove kmem_zone_free() wrapper 2019-11-18 08:40:44 -08:00
xfs_extfree_item.h xfs: merge xfs_efd_init into xfs_trans_get_efd 2019-06-28 19:27:35 -07:00
xfs_file.c xfs: fix IOCB_NOWAIT handling in xfs_file_dio_aio_read 2020-01-15 22:13:11 -08:00
xfs_filestream.c xfs: make xfs_*read_agf return EAGAIN to ALLOC_FLAG_TRYLOCK callers 2020-01-26 14:32:26 -08:00
xfs_filestream.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_fsmap.c xfs: prohibit fs freezing when using empty transactions 2020-03-26 08:19:24 -07:00
xfs_fsmap.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_fsops.c xfs: remove unused header files 2019-06-28 19:30:43 -07:00
xfs_fsops.h xfs: change some error-less functions to void types 2019-05-01 20:26:30 -07:00
xfs_globals.c xfs: multithreaded iwalk implementation 2019-07-03 07:33:26 -07:00
xfs_health.c xfs: introduce new v5 bulkstat structure 2019-07-03 20:36:26 -07:00
xfs_icache.c xfs: ensure that the inode uid/gid match values match the icdinode ones 2020-03-02 20:55:50 -08:00
xfs_icache.h xfs: rename the speculative block allocation reclaim toggle functions 2019-04-26 12:28:55 -07:00
xfs_icreate_item.c xfs: Remove kmem_zone_free() wrapper 2019-11-18 08:40:44 -08:00
xfs_icreate_item.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_inode_item.c xfs: remove the di_version field from struct icdinode 2020-03-19 08:48:47 -07:00
xfs_inode_item.h xfs: remove the xfs_log_item_t typedef 2019-06-28 19:27:33 -07:00
xfs_inode.c xfs: remove the di_version field from struct icdinode 2020-03-19 08:48:47 -07:00
xfs_inode.h xfs: merge the projid fields in struct xfs_icdinode 2019-11-13 11:13:45 -08:00
xfs_ioctl32.c xfs: lift cursor copy in/out into xfs_ioc_attr_list 2020-03-02 20:55:54 -08:00
xfs_ioctl32.h xfs: rename compat_time_t to old_time32_t 2020-01-06 08:57:36 -08:00
xfs_ioctl.c xfs: remove the di_version field from struct icdinode 2020-03-19 08:48:47 -07:00
xfs_ioctl.h xfs: embedded the attrlist cursor into struct xfs_attr_list_context 2020-03-02 20:55:55 -08:00
xfs_iomap.c xfs: change return value of xfs_inode_need_cow to int 2020-01-20 14:34:47 -08:00
xfs_iomap.h xfs: simplify the xfs_iomap_write_direct calling 2019-11-03 10:22:30 -08:00
xfs_iops.c xfs: remove the di_version field from struct icdinode 2020-03-19 08:48:47 -07:00
xfs_iops.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_itable.c xfs: remove the di_version field from struct icdinode 2020-03-19 08:48:47 -07:00
xfs_itable.h xfs: remove all *_ITER_ABORT values 2019-08-29 21:22:41 -07:00
xfs_iwalk.c xfs: kill the XFS_WANT_CORRUPT_* macros 2019-11-12 17:19:02 -08:00
xfs_iwalk.h xfs: remove all *_ITER_CONTINUE values 2019-08-30 22:43:56 -07:00
xfs_linux.h xfs: clean up the attr flag confusion 2020-03-02 20:55:55 -08:00
xfs_log_cil.c xfs: merge xlog_commit_record with xlog_write_done 2020-03-27 08:32:54 -07:00
xfs_log_priv.h xfs: Lower CIL flush limit for large logs 2020-03-27 08:32:54 -07:00
xfs_log_recover.c xfs: remove the di_version field from struct icdinode 2020-03-19 08:48:47 -07:00
xfs_log.c xfs: remove some stale comments from the log code 2020-03-27 08:32:54 -07:00
xfs_log.h xfs: refactor and split xfs_log_done() 2020-03-27 08:32:53 -07:00
xfs_message.c xfs: make the assertion message functions take a mount parameter 2019-11-05 08:28:27 -08:00
xfs_message.h xfs: make the assertion message functions take a mount parameter 2019-11-05 08:28:27 -08:00
xfs_mount.c xfs: remove XFS_BUF_TO_SBP 2020-03-11 09:11:39 -07:00
xfs_mount.h xfs: remove unused structure members & simple typedefs 2019-11-13 18:22:41 -08:00
xfs_mru_cache.c fs: xfs: Remove KM_NOSLEEP and KM_SLEEP. 2019-08-26 12:06:22 -07:00
xfs_mru_cache.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_ondisk.h xfs: make struct xfs_buf_log_format have a consistent size 2020-01-16 08:07:23 -08:00
xfs_pnfs.c xfs: use super s_id instead of struct xfs_mount m_fsname 2019-11-05 08:28:25 -08:00
xfs_pnfs.h
xfs_pwork.c xfs: poll waiting for quotacheck 2019-07-03 08:21:58 -07:00
xfs_pwork.h xfs: poll waiting for quotacheck 2019-07-03 08:21:58 -07:00
xfs_qm_bhv.c xfs: remove the xfs_disk_dquot_t and xfs_dquot_t 2019-11-13 11:13:45 -08:00
xfs_qm_syscalls.c xfs: fix unmount hang and memory leak on shutdown during quotaoff 2020-03-18 08:12:23 -07:00
xfs_qm.c xfs: preserve default grace interval during quotacheck 2020-03-26 08:19:24 -07:00
xfs_qm.h xfs: quota: move to time64_t interfaces 2020-01-06 08:57:37 -08:00
xfs_quota.h xfs: remove the icdinode di_uid/di_gid members 2020-03-02 20:55:50 -08:00
xfs_quotaops.c xfs: quota: move to time64_t interfaces 2020-01-06 08:57:37 -08:00
xfs_refcount_item.c xfs: Remove kmem_zone_free() wrapper 2019-11-18 08:40:44 -08:00
xfs_refcount_item.h xfs: merge xfs_cud_init into xfs_trans_get_cud 2019-06-28 19:27:35 -07:00
xfs_reflink.c xfs: remove unnecessary null pointer checks from _read_agf callers 2020-01-26 14:32:27 -08:00
xfs_reflink.h xfs: change return value of xfs_inode_need_cow to int 2020-01-20 14:34:47 -08:00
xfs_rmap_item.c xfs: Remove kmem_zone_free() wrapper 2019-11-18 08:40:44 -08:00
xfs_rmap_item.h xfs: merge xfs_rud_init into xfs_trans_get_rud 2019-06-28 19:27:36 -07:00
xfs_rtalloc.c xfs: make xfs_trans_get_buf return an error code 2020-01-26 14:32:26 -08:00
xfs_rtalloc.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_stats.c xfs: Use scnprintf() for avoiding potential buffer overflow 2020-03-12 07:58:13 -07:00
xfs_stats.h xfs: use offsetof() in place of offset macros for __xfsstats 2018-10-18 17:21:39 +11:00
xfs_super.c Merge branch 'merge.nfs-fs_parse.1' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2020-02-08 13:26:41 -08:00
xfs_super.h xfs: include QUOTA, FATAL ASSERT build options in XFS_BUILD_OPTIONS 2019-10-21 09:04:57 -07:00
xfs_symlink.c xfs: only check the superblock version for dinode size calculation 2020-03-19 08:48:47 -07:00
xfs_symlink.h xfs: Correct comment tyops -> typos 2019-11-10 10:21:57 -08:00
xfs_sysctl.c xfs: remove unused header files 2019-06-28 19:30:43 -07:00
xfs_sysctl.h xfs: multithreaded iwalk implementation 2019-07-03 07:33:26 -07:00
xfs_sysfs.c xfs: avoid unused to_mp() function warning 2019-09-24 09:40:19 -07:00
xfs_sysfs.h xfs: convert to SPDX license tags 2018-06-06 14:17:53 -07:00
xfs_trace.c xfs: support bulk loading of staged btrees 2020-03-18 08:12:23 -07:00
xfs_trace.h xfs: split xlog_ticket_done 2020-03-27 08:32:53 -07:00
xfs_trans_ail.c xfs: clear PF_MEMALLOC before exiting xfsaild thread 2020-03-11 09:11:38 -07:00
xfs_trans_buf.c xfs: fix xfs_buf_ioerror_alert location reporting 2020-01-26 14:32:27 -08:00
xfs_trans_dquot.c xfs: quota: move to time64_t interfaces 2020-01-06 08:57:37 -08:00
xfs_trans_priv.h xfs: don't use xfs_trans_free_items in the commit path 2019-06-28 19:27:31 -07:00
xfs_trans.c xfs: split xlog_ticket_done 2020-03-27 08:32:53 -07:00
xfs_trans.h xfs: make xfs_trans_get_buf return an error code 2020-01-26 14:32:26 -08:00
xfs_xattr.c xfs: embedded the attrlist cursor into struct xfs_attr_list_context 2020-03-02 20:55:55 -08:00
xfs.h xfs: remove b_last_holder & associated macros 2018-08-12 08:37:31 -07:00