Right now fuse kernel serializes direct writes on the
same file. This serialization is good for such FUSE
implementations which rely on the inode lock to
avoid any data inconsistency issues but it hurts badly
such FUSE implementations which have their own mechanism
of dealing with cache/data integrity and can handle
parallel direct writes on the same file.
This patch allows parallel direct writes on the same file to be
enabled with the help of a flag FOPEN_PARALLEL_DIRECT_WRITES.
FUSE implementations which want to use this feature can
set this flag during fuse init. Default behaviour remains
same i.e no parallel direct writes on the same file.
Corresponding fuse kernel patch(Merged).
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.2&id=153524053bbb0d27bb2e0be36d1b46862e9ce74c
Such as for the xfstest-dev's generic/684 test case it will clear
suid and sgid if the fallocate request is commited by an unprivileged
user.
The kernel fuse passed the ATTR_KILL_SUID/ATTR_KILL_SGID flags to
userspace but it will be dropped.
Signed-off-by: Xiubo Li <xiubli@redhat.com>
This addresses https://github.com/libfuse/libfuse/issues/729
commit db35a37def introduced a public
config.h (rename to fuse_config.h to avoid conflicts) that
was installed with the package and included by libfuse users
through fuse_common.h. Probablem is that this file does not have
unique defines so that they are unique to libfuse - on including
the file conflicts with libfuse users came up.
In principle all defines could be prefixed, but then most of them
are internal for libfuse compilation only. So this splits out
publically required defines to a new file 'libfuse_config.h'
and changes back to include of "fuse_config.h" only when
HAVE_LIBFUSE_PRIVATE_CONFIG_H is defined.
This also renames HAVE_LIBC_VERSIONED_SYMBOLS to
LIBFUSE_BUILT_WITH_VERSIONED_SYMBOLS, as it actually
better explains for libfuse users what that variable
is for.
This addresses: https://github.com/libfuse/libfuse/issues/724
HAVE_LIBC_VERSIONED_SYMBOLS configures the library if to use
versioned symbols and is set at meson configuration time.
External filesystems (the main target, actually)
include fuse headers and the preprocessor
then acts on HAVE_LIBC_VERSIONED_SYMBOLS. Problem was now that
'config.h' was not distributed with libfuse and so
HAVE_LIBC_VERSIONED_SYMBOLS was never defined with external
tools and the preprocessor did the wrong decision.
This commit also increases the the minimal meson version,
as this depends on meson feature only available in 0.50
<quote 'meson' >
WARNING: Project specifies a minimum meson_
version '>= 0.42' but uses features which were added
in newer versions:
* 0.50.0: {'install arg in configure_file'}
</quote>
Additionally the config file has been renamed to "fuse_config.h"
to avoid clashes - 'config.h' is not very specific.
when -D_FILE_OFFSET_BITS=64 is defined, the off_t type is 64 bits wide
already. the fuse_common.h header already checks for this, and errors
when it is not, so be consistent with all the other uses of off_t.
some libcs like musl do not have a 32-bit off_t type, and don't define
__off64_t.
The io for FUSE requests and responses can now be further customized by allowing to write custom functions for reading/writing the responses. This includes overriding the splice io.
The reason for this addition is that having a custom file descriptor is not sufficient to allow custom io. Different types of file descriptor require different mechanisms of io interaction. For example, some file descriptor communication has boundaries (SOCK_DGRAM, EOF, etc...), while other types of fd:s might be unbounded (SOCK_STREAMS, ...). For unbounded communication, you have to read the header of the FUSE request first, and then read the remaining packet data. Furthermore, the one read call does not necessarily return all the data expected, requiring further
calls in a loop.
fuse_loop_mt and fuse_new had not been defined when
HAVE_LIBC_VERSIONED_SYMBOLS had not been set and additionally,
fuse_new_31 was missing in the version script and was therefore
an unusable symbol.
This also adds a test for unset HAVE_LIBC_VERSIONED_SYMBOLS.
In fact only gnu-libc fully supports symbol versioning, so it is
better to have a generic macro for it. This also allows to manually
disable symbol version and allows to run tests with that
configuration on gnu-libc. That testing will still not catch compat
issues, but least ensures the code can compile.
Testing for __APPLE__ and __ULIBC__ is now done by meson. More of such
checks can be added by people using other libcs.
On benchmarking metadata operations with a single threaded bonnie++
and "max_idle_threads" limited to 1, 'top' was showing suspicious
160% cpu usage.
Profiling the system with flame graphs showed that an astonishing
amount of CPU time was spent in thread creation and destruction.
After verifying the code it turned out that fuse_do_work() was
creating a new thread every time all existing idle threads
were already busy. And then just a few lines later after processing
the current request it noticed that it had created too many threads
and destructed the current thread. I.e. there was a thread
creation/destruction ping-pong.
Code is changed to only create new threads if the max number of
threads is not reached.
Furthermore, thread destruction is disabled, as creation/destruction
is expensive in general.
With this change cpu usage of passthrough_hp went from ~160% to
~80% (with different values of max_idle_threads). And bonnie
values got approximately faster by 90%. This is a with single
threaded bonnie++
bonnie++ -x 4 -q -s0 -d <path> -n 30:1:1:10 -r 0
Without this patch, using the default max_idle_threads=10 and just
a single bonnie++ the thread creation/destruction code path is not
triggered. Just one libfuse and one application thread is just
a corner case - the requirement for the issue was just
n-application-threads >= max_idle_threads.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
struct fuse_loop_config was passed as a plain struct, without any
version identifer. This had two implications
1) Any addition of new parameters required a FUSE_SYMVER for
fuse_session_loop_mt() and fuse_loop_mt() as otherwise a read
beyond end-of previous struct size might have happened.
2) Filesystems also might have been recompiled and the developer
might not have noticed the struct extensions and unexpected for
the developer (or people recomliling the code) uninitialized
parameters would have been passed.
Code is updated to have struct fuse_loop_config as an opaque/private
data type for file systems that want version 312
(FUSE_MAKE_VERSION(3, 12)). The deprecated fuse_loop_config_v1
is visible, but should not be used outside of internal
conversion functions
File systems that want version >= 32 < 312 get the previous
struct (through ifdefs) and the #define of fuse_loop_mt
and fuse_session_loop_mt ensures that these recompiled file
systems call into the previous API, which then converts
the struct. This is similar to existing compiled applications
when just libfuse updated, but binaries it is solved with
the FUSE_SYMVER ABI compact declarations.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
In fuse kernel, 'commit 53db28933e95 ("fuse: extend init flags")'
made the changes to handle flags going beyond 32 bits but i think
changes were not done in libfuse to handle the same.
This patch prepares the ground in libfuse for incoming FUSE kernel
patches (Atomic open + lookup) where flags went beyond 32 bits.
It makes struct same as in fuse kernel resulting in name change of
few fields.
Allow requesting from kernel to avoid flush on close at file open
time. If kernel does not support FOPEN_NOFLUSH flag, the request
will be ignored.
For passthrough_hp example, request to avoid flush on close when
writeback cache is disabled and file is opened O_RDONLY.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
When directories with open handles are removed, the releasedir and
fsyncdir operations might be called with a NULL path. That is because
there is no hiding behavior like for regular files and the nodes get
removed immediately.
We currently do not pass anything in PREDEFINED and that means
FUSE_USE_VERSION is undefined.
Add that definition so that Doxygen built-in C pre-processor can use
FUSE_USE_VERSION value to find the correct comment to parse.
This commit defines a new capability called `FUSE_CAP_CACHE_SYMLINKS`.
It is off by default but you can now enable it by setting this flag in
in the `want` field of the `fuse_conn_info` structure.
When enabled, the kernel will save symlinks in its page cache,
by making use of the feature introduced in kernel 4.20:
5571f1e654
On uclibc and MacOS we don't use versioned symbols. Hence,
there's no definition for fuse_session_loop_mt on those cases
and the linker won't be able to resolve calls to fuse_session_loop_mt()
Signed-off-by: Asaf Kahlon <asafka7@gmail.com>
* Add fallocate to list of operations that may omit path.
* earlier versions is 10+ years old.
Document is not obvious how old it is. Add it.
* Update manpage link
copy_file_range was first implemented with copy-based emulation in
glibc 2.27, but the emulation was subsequently removed again because
correct emulation depends on why the application attempted to make a
copy. Therefore, file systems cannot rely on low-level userspace
performing emulation.
Define FUSE_USE_VERSION < 35 to get old ioctl prototype
with int commands; define FUSE_USE_VERSION >= 35 to get
new ioctl prototype with unsigned int commands.
Fixes#463.
Applications may wish to call fuse_log() for unified logging. This way
they don't need to define their own wrappers to invoke the log message
handler function installed by fuse_set_log_func().
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Introduce an API for custom log handler functions. This allows libfuse
applications to send messages to syslog(3) or other logging systems.
See include/fuse_log.h for details.
Convert libfuse from fprintf(stderr, ...) to log_fuse(level, ...). Most
messages are error messages with FUSE_LOG_ERR log level. There are also
some debug messages which now use the FUSE_LOG_DEBUG log level.
Note that lib/mount_util.c is used by both libfuse and fusermount3.
Since fusermount3 does not link against libfuse, we cannot call
fuse_log() from lib/mount_util.c. This file will continue to use
fprintf(stderr, ...) until someone figures out how to split it up.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
