There's a report that newly added --rootdir print too many warnings for
hardlinks, which is maybe not that uncommon. We still want to let the
user know about that so print it just once and count how many were
found:
$ mkfs.btrfs --rootdir ...
WARNING: '/tmp/btrfs-progs-mkfs-rootdir-hardlinks.7RcdfR/rootdir/inside_link' has extra hardlinks, they will be converted into new inodes
WARNING: 12 hardlinks were detected in /tmp/btrfs-progs-mkfs-rootdir-hardlinks.7RcdfR/rootdir, all converted to new inodes
Link: https://github.com/kdave/btrfs-progs/pull/872#issuecomment-2289096125
Signed-off-by: David Sterba <dsterba@suse.com>
The way the CRC32C checksum used for btrfs-send differs from the way
it's used elsewhere in btrfs. Without making the distinction, it's easy
to make the flawed assumption that CRC32C always refers to the same, and
end up with code that produces the wrong checksums.
This small note should guide the reader to the right function.
The best notes on the protocol I found are here:
https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive.html
The crc32c might be used in two meanings and this could be confusing
when implementing the send stream protocol.
Rust code describing the algorithm for the crc crate that worked for me:
pub const CRC_32_BTRFS_SEND: crc::Algorithm<u32> = crc::Algorithm {
width: 32, poly: 0x1edc6f41, init: 0, refin: true, refout: true,
xorout: 0, check: 0xe3069283, residue: 0xb798b438
};
(it's a slight variation on the one used in ISCSI)
Note: Documentation/dev/dev-send-stream.rst briefly mentions that
Pull-request: #794
Author: rhn <gihu.rhn@porcupinefactory.org>
[ rephrase changelog and copy text from pull request and add link to
developer documentation of the send stream ]
Signed-off-by: David Sterba <dsterba@suse.com>
process_clone() only searches the received_uuid, but could exist in an
earlier uuid that isn't the received_uuid. Mirror what process_snapshot
does and search both the received_uuid and if that fails look up by
normal uuid.
Fixes: https://github.com/kdave/btrfs-progs/issues/606
Issue: #606
Pull-request: #643
Pull-request: #862
Signed-off-by: Arsenii Skvortsov <ettavolt@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Copy linux.git/include/linux/container_of.h definition of container_of
and the const variant (currently unused)
Signed-off-by: David Sterba <dsterba@suse.com>
Sync up with kernel and fix warnings reported by -Wcast-qual. eg.
Most of the change is due to extent_buffer::data, which is a direct
struct member, unlike in kernel where it's an array of pages. The
const qualifier cannot be used the same way so it's dropped in affected
herlpers.
Signed-off-by: David Sterba <dsterba@suse.com>
There are no functional changes, only cleanup of header files. This
could lead to build failures in case the headers were used as a
convenience outside of scope of libbtrfs just because of the kernel
compatibility.
- removed various definitions of variables, types, helpers and macros
from kerncompat.h that are neither used nor needed for libbtrfs code
- file list.h no longer shipped
- file rbtree.h no longer shipped
Signed-off-by: David Sterba <dsterba@suse.com>
None of the public API uses the rb-tree code besides definitions, so
change the includes in ctree.h and drop rbtree.h, this is used only by
internal implementation in send-utils.c. We could remove it in the
future but last time it was not possible due to 3rd party code depending
on it.
Removed in 83ab92512e ("libbtrfs: remove the support for fs without
uuid tree") and reverted again in f9b0da8e78 ("Revert "libbtrfs:
remove the support for fs without uuid tree"")
Signed-off-by: David Sterba <dsterba@suse.com>
The list_head is used in struct definitions but otherwise not at all as
it was copied from kernel code. For ctree.h add stub definition that
won't change the containing structure size.
Drop list.h from libbtrfs. This may break some builds if they used the
header, though this was never meant to be exported.
Signed-off-by: David Sterba <dsterba@suse.com>
The BUILD_ASSERT macro checks what _Static_assert can do. Remove it as
it's not really used in ioctl.h as it defines a stub. The assertions
still remain in the code outside of libbtrfs, we can delete it here as
the API is frozen and won't be changed.
Signed-off-by: David Sterba <dsterba@suse.com>
The stack trace and BUG_ON related reporting was inherited from the
tools, this should not be part of libbtrfs and was not intended to be
exported.
BUG() is still in used in ctree.h and send-utils.c so replace it with a
bare error report and remove the rest.
Keep __always_inline as it's needed for Musl.
Signed-off-by: David Sterba <dsterba@suse.com>
Commit bf0f3db765 ("btrfs-progs: introduce UASSERT() for purely
userspace code") added UASERT to distinguish ASSERT macro from user
space code. This was wrongly added to libbtrfs/ and pulled the
common/messages.h include too.
Signed-off-by: David Sterba <dsterba@suse.com>
Remove BUG_ONs that seem to be sanity checks that are done in other
places. We want to remove them from public header so BUG_ON can be
removed completely.
Signed-off-by: David Sterba <dsterba@suse.com>
There are many definitions of types etc that are not used in libbtrfs
but used to be in other code before the separation. Remove it as it's
not meant to be exported.
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 14ac1a6051 ("btrfs-progs: mkfs: add support for squota")
mistakenly added ctree.h from libbtrfs/ but this is not supposed to be
used outside of the library. Moreover the correct ctree.h was already
there.
Signed-off-by: David Sterba <dsterba@suse.com>
The new test case creates a special layout like this:
rootdir/ (fs1 ino=256)
|- dir1/ (fs1 ino=257)
| |- dir1/ (fs2 ino=257)
| |- dir2/ (fs2 ino=258)
| |- file1 (fs2 ino=259)
| |- file2 (fs2 ino=260)
|- dir2/ (fs1 ino=258)
|- file1 (fs1 ino=259)
|- file2 (fs2 ino=259)
This layout intentionally creates inode number conflicts, which will
make the old "mkfs.btrfs --rootdir" to fail.
But newer reworked one will successfully handle them, just leave a test
case to avoid to hit the old bugs.
Signed-off-by: Qu Wenruo <wqu@suse.com>
The test case will create the following directory layout:
.
|- rootdir/
| |- inside_link
|- outside_link
Both inside_link and outside_link are hard links of each other.
And use rootdir/ as the rootdir for mkfs.
This is to ensure the nlink of inside_link is correctly set to 1.
Inspired by the recent rework which fixes the handling of hard links.
Signed-off-by: Qu Wenruo <wqu@suse.com>
The recent rework changes how we detect hard links.
[OLD BEHAVIOR]
We trusted st_nlink and st_ino, reuse them without extra sanity
checks.
That behavior has problems handling cross mount-point or hard links out
of the rootdir cases.
[NEW BEHAVIOR]
The new refactored code will treat every inode, no matter if it's a
hardlink, as a new inode.
This means we will break the hard link detection, and every hard link
will be created as a different inode.
For the most common use case, like populating a rootfs, it's totally
fine.
[EXTRA WARNING]
But for cases where the user have extra hard links inside the rootdir,
output a warning just to inform the end user.
This will not cause any content difference, just breaking the hard links
into new inodes.
Signed-off-by: Qu Wenruo <wqu@suse.com>
[PITFALLS]
There are several hidden pitfalls of the existing traverse_directory():
- Hand written preorder traversal
There is already a better written standard library function, nftw()
doing exactly what we need.
- Over-designed path list
To properly handle the directory change, we have structure
directory_name_entry, to record every inode until rootdir.
But it has two string members, dir_name and path, which is a little
confusing and overkilled.
As for preorder traversal, we will never need to read the parent's
filename, just its btrfs inode number.
And it's exported while no one utilizes it out of mkfs/rootdir.c.
- Weird inode numbers
We use the inode number from st->st_ino, with an extra offset.
This by itself is not safe, if the rootdir has child directories in
another filesystem.
And this results very weird inode numbers, e.g:
item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
item 6 key (88347519 INODE_ITEM 0) itemoff 15815 itemsize 160
item 16 key (88347520 INODE_ITEM 0) itemoff 15363 itemsize 160
item 20 key (88347521 INODE_ITEM 0) itemoff 15119 itemsize 160
item 24 key (88347522 INODE_ITEM 0) itemoff 14875 itemsize 160
item 26 key (88347523 INODE_ITEM 0) itemoff 14700 itemsize 160
item 28 key (88347524 INODE_ITEM 0) itemoff 14525 itemsize 160
item 30 key (88347557 INODE_ITEM 0) itemoff 14350 itemsize 160
item 32 key (88347566 INODE_ITEM 0) itemoff 14175 itemsize 160
Which is far from a regular fs created by copying the data.
- Weird directory inode size calculation
Unlike kernel, which updated the directory inode size every time new
child inodes are added, we calculate the directory inode size by
searching all its children first, then later new inodes linked to this
directory won't touch the inode size.
- Bad hard link detection and cross mount point handling
The hard link detection is purely based on the st_ino returned from
the host filesystem, this means we do not have extra checks whether
the inode is even inside the same fs.
And we directly reuse st_nlink from the host filesystem, if there
is a hard link out of rootdir, the st_nlink will be incorrect and
cause a corrupted fs.
Enhance all these points by:
- Use nftw() to do the preorder traversal
It also provides the extra level detection, which is pretty handy.
- Use a simple local inode_entry to record each parent
The only value is a u64 to record the inode number.
And one simple rootdir_path structure to record the list of
inode_entry, alone with the current level.
This rootdir_path structure along with two helpers,
rootdir_path_push() and rootdir_path_pop(), along with the
preorder traversal provided by nftw(), are enough for us to record
all the parent directories until the rootdir.
- Grab new inode number properly
Just call btrfs_get_free_objectid() to grab a proper inode number,
other than using some weird calculated value.
- Treat every inode as a new one
This means we will have no hard link support for now.
But I still believe it's a good trade-off, especially considering the
old handling is buggy for several corner cases.
- Use btrfs_insert_inode() and btrfs_add_link() to update directory
inode automatically
With all the refactoring, the code is shorter and easier to read.
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Qu Wenruo <wqu@suse.com>
We use the UASSERT() wrapper instead of the plain assert() as this can
be tuned to print the stack trace too if supported.
Signed-off-by: David Sterba <dsterba@suse.com>
All issues have been fixed in latest master, enable the checks for devel
too. It takes about 17m. Also rename the file, drop the "ci-" prefix.
Signed-off-by: David Sterba <dsterba@suse.com>
Fixes so 'python3 -m build' works and package can be uploaded to pypi
(https://pypi.org/project/btrfsutil/).
- setup.py is still used for local build (make)
- for pypi it must be done by 'python3 -m build' that is build in a
temporary directory
- btrfsutilpy.h must be also distributed
- version is set manually (the git VERSION file is not accessible)
- the project page metadata is empty, the README.md should be added
Issue: #310
Signed-off-by: David Sterba <dsterba@suse.com>
Make it more visible what the result of snapshotted subvolume is. This
partially duplicates the other section.
[ci skip]
Issue: #644
Signed-off-by: David Sterba <dsterba@suse.com>
It is possible to create swapfile on a multi-device filesystem but it's
not reliable. The check that verifies that in kernel:
10698 } else if (device != map->stripes[0].dev) {
10699 btrfs_warn(fs_info, "swapfile must be on one device");
10700 ret = -EINVAL;
10701 goto out;
10702 }
This does not count devices but rather the actual placement of the
swapfile extents, so multi-device filesystem with single profile can
create it as long as there's enough space and the allocator decides to
place it properly.
[ci skip]
Pull-request: #839
Signed-off-by: David Sterba <dsterba@suse.com>
The thread sanitizer finds race conditions and in the past did find
some bugs. There's not much threaded code, it's namely the progress
tracking in btrfs-convert so the coverage is slightly redundant. Add it
just in case.
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
ASAN test fails at misc/055 with the following leak:
Qgroupid Referenced Exclusive Path
-------- ---------- --------- ----
0/5 16.00KiB 16.00KiB <toplevel>
0/256 16.00KiB 16.00KiB <stale>
====== RUN CHECK /home/runner/work/btrfs-progs/btrfs-progs/btrfs qgroup clear-stale /home/runner/work/btrfs-progs/btrfs-progs/tests/mnt
=================================================================
==102571==ERROR: LeakSanitizer: detected memory leaks
Indirect leak of 4096 byte(s) in 1 object(s) allocated from:
#0 0x7fd1c98fbb37 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x55aa2f8953f8 in btrfs_util_subvolume_path_fd libbtrfsutil/subvolume.c:178
#2 0x55aa2f8fa2a6 in get_or_add_qgroup cmds/qgroup.c:837
#3 0x55aa2f8fa7e9 in update_qgroup_info cmds/qgroup.c:883
#4 0x55aa2f8fd912 in __qgroups_search cmds/qgroup.c:1385
#5 0x55aa2f8fe196 in qgroups_search_all cmds/qgroup.c:1453
#6 0x55aa2f902a7c in cmd_qgroup_clear_stale cmds/qgroup.c:2281
#7 0x55aa2f73425b in cmd_execute cmds/commands.h:126
#8 0x55aa2f734bcc in handle_command_group /home/runner/work/btrfs-progs/btrfs-progs/btrfs.c:177
#9 0x55aa2f73425b in cmd_execute cmds/commands.h:126
#10 0x55aa2f735a96 in main /home/runner/work/btrfs-progs/btrfs-progs/btrfs.c:518
#11 0x7fd1c942a1c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
#12 0x7fd1c942a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
#13 0x55aa2f734144 in _start (/home/runner/work/btrfs-progs/btrfs-progs/btrfs+0x84144) (BuildId: 56f3dd838e1ae189c142c5d27fac025cd46deddb)
Indirect leak of 432 byte(s) in 2 object(s) allocated from:
#0 0x7fd1c98fb4d0 in calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
#1 0x55aa2f8fa1a1 in get_or_add_qgroup cmds/qgroup.c:822
#2 0x55aa2f8fa7e9 in update_qgroup_info cmds/qgroup.c:883
#3 0x55aa2f8fd912 in __qgroups_search cmds/qgroup.c:1385
#4 0x55aa2f8fe196 in qgroups_search_all cmds/qgroup.c:1453
#5 0x55aa2f902a7c in cmd_qgroup_clear_stale cmds/qgroup.c:2281
#6 0x55aa2f73425b in cmd_execute cmds/commands.h:126
#7 0x55aa2f734bcc in handle_command_group /home/runner/work/btrfs-progs/btrfs-progs/btrfs.c:177
#8 0x55aa2f73425b in cmd_execute cmds/commands.h:126
#9 0x55aa2f735a96 in main /home/runner/work/btrfs-progs/btrfs-progs/btrfs.c:518
#10 0x7fd1c942a1c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
#11 0x7fd1c942a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
#12 0x55aa2f734144 in _start (/home/runner/work/btrfs-progs/btrfs-progs/btrfs+0x84144) (BuildId: 56f3dd838e1ae189c142c5d27fac025cd46deddb)
[CAUSE]
Above leaks are caused by two btrfs_qgroup structures and one path for
toplevel qgroup.
It's caused by the fact that we called qgroups_search_all() but didn't
do any cleanup.
[FIX]
Call __free_all_qgroups() inside cmd_qgroup_clear_stale() to properly
free the qgroups.
Fixes: 701ab151c2 ("btrfs-progs: qgroup: new command to delete stale qgroups")
Signed-off-by: Qu Wenruo <wqu@suse.com>
Commit d7492ec59e ("btrfs-progs: use on-stack buffer in
__ino_to_path_fd") was supposed to switch path buffer from dynamic
allocation to on-stack but it was done wrong. The btrfs_data_container
is a flexible array so it needs to be explicitly allocated to the right
size.
The conversion turned it to an array. Gcc 13.x started to warn about
access to fspath->val[i] being out of bounds. Fortunately overall size
was 65536 and used only first 4096 bytes.
cmds/inspect.c: In function ‘__ino_to_path_fd’:
cmds/inspect.c:86:35: warning: array subscript i is outside array bounds of ‘__u64[]’ {aka ‘long long unsigned int[]’} [-Warray-bounds=]
86 | ptr += fspath->val[i];
| ~~~~~~~~~~~^~~
In file included from ./kernel-shared/accessors.h:11,
from cmds/inspect.c:35:
./kernel-shared/uapi/btrfs.h:724:17: note: while referencing ‘val’
724 | __u64 val[]; /* out */
Add an on-stack buffer and map it over fspath, similar to the previous
dynamic array.
Signed-off-by: David Sterba <dsterba@suse.com>
mkfs_main() is a main-like function, meaning that return and exit are
equivalent. Deduplicate our cleanup code by moving the error label.
Signed-off-by: Mark Harmstone <maharmstone@fb.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Recent changes to the python code reworked path_converter() so that it
does not use fd_converter() anymore. Assuming it may be used in the
future again comment it out.
Signed-off-by: David Sterba <dsterba@suse.com>
Currently mkfs uses its own create_uuid_tree(), but that function is
only handling FS_TREE. This means for btrfs-convert we do not generate
the uuid tree, nor add the UUID of the image subvolume. This can be a
problem if we're going to support multiple subvolumes during mkfs time.
To address this, introduce a new helper, btrfs_rebuild_uuid_tree():
- Create a new uuid tree if there is not one
- Remove all the existing items from uuid tree
- Iterate through all subvolumes
* If the subvolume has no valid UUID, regenerate one
* Add the uuid entry for the subvolume UUID
* If the subvolume has received UUID, also add it to UUID tree
By this, this new helper can handle all the uuid tree generation needs for:
- Current mkfs
Only one uuid entry for FS_TREE
- Current btrfs-convert
Only FS_TREE and the image subvolume
- Future multi-subvolume mkfs
As we do the scan for all subvolumes.
- Future "btrfs rescue rebuild-uuid-tree"
Signed-off-by: Qu Wenruo <wqu@suse.com>
The modification is minimal:
- Replace WARN_ON() with UASSERT()
- Remove the @trans parameter for btrfs_extend_item() and
btrfs_mark_buffer_dirty()
As progs version doesn't need a transaction handler.
- Remove the btrfs_uuid_tree_add() in mkfs/main.c
Signed-off-by: Qu Wenruo <wqu@suse.com>
Currently we already have a kernel-shared/uuid-tree.c, which is mostly
shared with kernel.
Kernel also has a uuid-tree.h, but we are still using ctree.h for the
header.
Move all the uuid-tree related definitions to kernel-shared/uuid-tree.h,
making future code sync easier.
Signed-off-by: Qu Wenruo <wqu@suse.com>
btrfs_insert_dir_item wasn't setting the transid field in
btrfs_dir_item. Set it to the current transaction ID rather than writing
uninitialized memory to disk.
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Mark Harmstone <maharmstone@fb.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Python 3.13, currently in beta, removed the internal
_PyObject_LookupSpecial function. The libbtrfsutil Python bindings use
it in the path_converter() function because it was based on internal
path_converter() function in CPython [1]. This is causing build failures
on Fedora Rawhide [2] and Gentoo [3]. Replace path_converter() with a
version that only uses public functions based on the one in drgn [4].
1: d9efa45d74/Modules/posixmodule.c (L1253)
2: https://bugzilla.redhat.com/show_bug.cgi?id=2245650
3: https://github.com/kdave/btrfs-progs/issues/838
4: 9ad29fd864/libdrgn/python/util.c (L81)
Issue: #838
Reported-by: Neal Gompa <neal@gompa.dev>
Reported-by: Sam James <sam@gentoo.org>
Reviewed-by: Neal Gompa <neal@gompa.dev>
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: David Sterba <dsterba@suse.com>
With the new feature description output in "btrfs --version" there is no
need to do the config.h hack to determine if we have certain feature.
This provides a more reliable way to detect features.
Signed-off-by: Qu Wenruo <wqu@suse.com>
For "btrfstune --csum", currently we do the following operations in just
one transaction for each:
- Delete old data csums
- Change new data csums objectid
Both operation can modify up to GiB or even TiB level of metadata, doing
them in just one transaction is definitely going to cause problems.
This patch adds a leaf number based threshold (32 leaves), after
modifying/deleting this many leaves, we commit a transaction to avoid
huge amount of dirty leaves piling up.
Signed-off-by: Qu Wenruo <wqu@suse.com>
