Optimised implementations for single and double precision, Advanced
SIMD and SVE, copied from Arm Optimized Routines.
As previously, data tables are used via a barrier to prevent
overly aggressive constant inlining. Special-case handlers are
marked NOINLINE to avoid incurring the penalty of switching call
standards unnecessarily.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Optimised implementations for single and double precision, Advanced
SIMD and SVE, copied from Arm Optimized Routines. Log lookup table
added as HIDDEN symbol to allow it to be shared between AdvSIMD and
SVE variants.
As previously, data tables are used via a barrier to prevent
overly aggressive constant inlining. Special-case handlers are
marked NOINLINE to avoid incurring the penalty of switching call
standards unnecessarily.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Optimised implementations for single and double precision, Advanced
SIMD and SVE, copied from Arm Optimized Routines.
As previously, data tables are used via a barrier to prevent
overly aggressive constant inlining. Special-case handlers are
marked NOINLINE to avoid incurring the penalty of switching call
standards unnecessarily.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Replace the loop-over-scalar placeholder routines with optimised
implementations from Arm Optimized Routines (AOR).
Also add some headers containing utilities for aarch64 libmvec
routines, and update libm-test-ulps.
Data tables for new routines are used via a pointer with a
barrier on it, in order to prevent overly aggressive constant
inlining in GCC. This allows a single adrp, combined with offset
loads, to be used for every constant in the table.
Special-case handlers are marked NOINLINE in order to confine the
save/restore overhead of switching from vector to normal calling
standard. This way we only incur the extra memory access in the
exceptional cases. NOINLINE definitions have been moved to
math_private.h in order to reduce duplication.
AOR exposes a config option, WANT_SIMD_EXCEPT, to enable
selective masking (and later fixing up) of invalid lanes, in
order to trigger fp exceptions correctly (AdvSIMD only). This is
tested and maintained in AOR, however it is configured off at
source level here for performance reasons. We keep the
WANT_SIMD_EXCEPT blocks in routine sources to greatly simplify
the upstreaming process from AOR to glibc.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Add --disable-encoding to makeinfo flags so that it does not generate
unicode quote glyphs.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Linux 6.4 adds the riscv_hwprobe syscall on riscv and enables
memfd_secret on s390. Update syscall-names.list and regenerate the
arch-syscall.h headers with build-many-glibcs.py update-syscalls.
Tested with build-many-glibcs.py.
Trying to mount procfs can fail due multiples reasons: proc is locked
due the container configuration, mount syscall is filtered by a
Linux Secuirty Module, or any other security or hardening mechanism
that Linux might eventually add.
The tests does require a new procfs without binding to parent, and
to fully fix it would require to change how the container was created
(which is out of the scope of the test itself). Instead of trying to
foresee any possible scenario, if procfs can not be mount fail with
unsupported.
Checked on aarch64-linux-gnu.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The tst-ttyname-direct.c checks the ttyname with procfs mounted in
bind mode (MS_BIND|MS_REC), while tst-ttyname-namespace.c checks
with procfs mount with MS_NOSUID|MS_NOEXEC|MS_NODEV in a new
namespace.
Checked on x86_64-linux-gnu and aarch64-linux-gnu.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
This patch improves tests-clean Makefile target to reliably clean
test artifacts from a build directory. Before this patch tests-clean
missed around 3k (out of total 9k) .out and .test-result files.
Signed-off-by: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
These files could be useful to any port that wants to use ld.so.cache.
Signed-off-by: Sergey Bugaev <bugaevc@gmail.com>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
ldconfig was allocating PATH_MAX bytes on the stack for the library file
name. The issues with PATH_MAX usage are well documented [0][1]; even if
a program does not rely on paths being limited to PATH_MAX bytes,
allocating 4096 bytes on the stack for paths that are typically rather
short (strlen ("/lib64/libc.so.6") is 16) is wasteful and dangerous.
[0]: https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html
[1]: https://eklitzke.org/path-max-is-tricky
Instead, make use of asprintf to dynamically allocate memory of just the
right size on the heap.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Signed-off-by: Sergey Bugaev <bugaevc@gmail.com>
In documentation, call strings like "CST" time zone abbreviations, not
time zone names. This terminology is more precise, and is what tzdb uses.
A string like "CST" is ambiguous and does not fully name a time zone.
Few tests needed to properly check for asprintf and system calls return
values with _FORTIFY_SOURCE enabled.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The fread routine return value needs to be checked when fortification
is enabled, hence use xfread helper.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The declaration and definition of these routines aren't consistent.
Make the definition of __readlink_chk and __readlinkat_chk match the
declaration of the routines they fortify. While there are no problems
today this avoids any future potential problems related to the mismatch.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
This will enable __REDIRECT_FORTIFY* macros to be used when _FORTIFY_SOURCE
is set.
Routine declarations that were in bits/wchar2.h are moved into the
bits/wchar2-decl.h file.
The file is now included into include/wchar.h irrespectively from
fortification.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Few tests using swprintf are passing incorrect maxlen parameter.
This triggers an abort when _FORTIFY_SOURCE is enabled.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
On i386 and x86_64, for libc.a specifically, __mempcpy_chk calls
mempcpy which leads POSIX routines to call non-POSIX mempcpy indirectly.
This leads the linknamespace test to fail when glibc is built with
__FORTIFY_SOURCE=3.
Since calling mempcpy doesn't bring any benefit for libc.a, directly
call __mempcpy instead.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Replace alloca with a scratch_buffer to avoid potential stack overflows.
Checked on i686-gnu and x86_64-linux-gnu
Message-Id: <20230619144334.2902429-1-josimmon@redhat.com>
There is a potential memory leak for large writes due to writev being a
"shall occur" cancellation point. Add back the cleanup handler removed
in cf30aa43a5.
Checked on i686-gnu and x86_64-linux-gnu.
Message-Id: <20230619143842.2901522-1-josimmon@redhat.com>
ISO C2x defines scanf %b for input of binary integers (with an
optional 0b or 0B prefix). Implement such support, along with the
corresponding SCNb* macros in <inttypes.h>. Unlike the support for
binary integers with 0b or 0B prefix with scanf %i, this is supported
in all versions of scanf (independent of the standards mode used for
compilation), because there are no backwards compatibility concerns
(%b wasn't previously a supported format) the way there were for %i.
Tested for x86_64 and x86.
ISO C2x defines printf length modifiers wN (for intN_t / int_leastN_t
/ uintN_t / uint_leastN_t) and wfN (for int_fastN_t / uint_fastN_t).
Add support for those length modifiers (such a feature was previously
requested in bug 24466). scanf support is to be added separately.
GCC 13 has format checking support for these modifiers.
When used with the support for registering format specifiers, these
modifiers are translated to existing flags in struct printf_info,
rather than trying to add some way of distinguishing them without
breaking the printf_info ABI. C2x requires an error to be returned
for unsupported values of N; this is implemented for printf-family
functions, but the parse_printf_format interface doesn't support error
returns, so such an error gets discarded by that function.
Tested for x86_64 and x86.
With fortification enabled, system calls return result needs to be checked,
has it gets the __wur macro enabled.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
With fortification enabled, read calls return result needs to be checked,
has it gets the __wur macro enabled.
Note on read call removal from sysdeps/pthread/tst-cancel20.c and
sysdeps/pthread/tst-cancel21.c:
It is assumed that this second read call was there to overcome the race
condition between pipe closure and thread cancellation that could happen
in the original code. Since this race condition got fixed by
d0e3ffb7a5 the second call seems
superfluous. Hence, instead of checking for the return value of read, it
looks reasonable to simply remove it.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Use a scratch_buffer rather than alloca to avoid potential stack
overflows.
Checked on i686-gnu and x86_64-linux-gnu
Message-Id: <20230608155844.976554-1-josimmon@redhat.com>
For strerror, this fixes commit 28aff04781 ("string:
Implement strerror in terms of strerror_l"). This commit avoids
returning NULL for strerror_l as well, although POSIX allows this
behavior for strerror_l.
Reviewed-by: Arjun Shankar <arjun@redhat.com>
These functions are about to be added to POSIX, under Austin Group
issue 986.
The fortified strlcat implementation does not raise SIGABRT if the
destination buffer does not contain a null terminator, it just
inherits the non-failing regular strlcat behavior.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
With fortification enabled, fgets calls return result needs to be checked,
has it gets the __wur macro enabled.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
With fortification enabled, fread calls return result needs to be checked,
has it gets the __wur macro enabled.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The tst-mallocfork2 and tst-mallocfork3 create large number of
subprocesss, around 11k for former and 20k for latter, to check
for malloc async-signal-safeness on both fork and _Fork. However
they do not really exercise allocation patterns different than
other tests fro malloc itself, and the spawned process just exit
without any extra computation.
The tst-malloc-tcache-leak is similar, but creates 100k threads
and already checks the resulting with mallinfo.
These tests are also very sensitive to system load (since they
estresss heavy the kernel resource allocation), and adding them
on THP tunable and mcheck tests increase the pressure even more.
For THP the fork tests do not add any more coverage than other
tests. The mcheck is also not enable for tst-malloc-tcache-leak.
Checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
There is no fork detection on current arc4random implementation, so
use lower subprocess on fork tests. The tests now run on 0.1s
instead of 8s on a Ryzen9 5900X.
Checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The getdate testcases all expect successful results. Add support for
negative testcases and testcases where a full date and time are not
supplied by skipping the tm checks in the test. Add a testcase that
would catch a use-after-free that was recently found.
Reviewed-by: Arjun Shankar <arjun@redhat.com>
Different systems prefer a different divisors.
From benchmarks[1] so far the following divisors have been found:
ICX : 2
SKX : 2
BWD : 8
For Intel, we are generalizing that BWD and older prefers 8 as a
divisor, and SKL and newer prefers 2. This number can be further tuned
as benchmarks are run.
[1]: https://github.com/goldsteinn/memcpy-nt-benchmarks
Reviewed-by: DJ Delorie <dj@redhat.com>
This patch should have no affect on existing functionality.
The current code, which has a single switch for model detection and
setting prefered features, is difficult to follow/extend. The cases
use magic numbers and many microarchitectures are missing. This makes
it difficult to reason about what is implemented so far and/or
how/where to add support for new features.
This patch splits the model detection and preference setting stages so
that CPU preferences can be set based on a complete list of available
microarchitectures, rather than based on model magic numbers.
Reviewed-by: DJ Delorie <dj@redhat.com>
Current `non_temporal_threshold` set to roughly '3/4 * sizeof_L3 /
ncores_per_socket'. This patch updates that value to roughly
'sizeof_L3 / 4`
The original value (specifically dividing the `ncores_per_socket`) was
done to limit the amount of other threads' data a `memcpy`/`memset`
could evict.
Dividing by 'ncores_per_socket', however leads to exceedingly low
non-temporal thresholds and leads to using non-temporal stores in
cases where REP MOVSB is multiple times faster.
Furthermore, non-temporal stores are written directly to main memory
so using it at a size much smaller than L3 can place soon to be
accessed data much further away than it otherwise could be. As well,
modern machines are able to detect streaming patterns (especially if
REP MOVSB is used) and provide LRU hints to the memory subsystem. This
in affect caps the total amount of eviction at 1/cache_associativity,
far below meaningfully thrashing the entire cache.
As best I can tell, the benchmarks that lead this small threshold
where done comparing non-temporal stores versus standard cacheable
stores. A better comparison (linked below) is to be REP MOVSB which,
on the measure systems, is nearly 2x faster than non-temporal stores
at the low-end of the previous threshold, and within 10% for over
100MB copies (well past even the current threshold). In cases with a
low number of threads competing for bandwidth, REP MOVSB is ~2x faster
up to `sizeof_L3`.
The divisor of `4` is a somewhat arbitrary value. From benchmarks it
seems Skylake and Icelake both prefer a divisor of `2`, but older CPUs
such as Broadwell prefer something closer to `8`. This patch is meant
to be followed up by another one to make the divisor cpu-specific, but
in the meantime (and for easier backporting), this patch settles on
`4` as a middle-ground.
Benchmarks comparing non-temporal stores, REP MOVSB, and cacheable
stores where done using:
https://github.com/goldsteinn/memcpy-nt-benchmarks
Sheets results (also available in pdf on the github):
https://docs.google.com/spreadsheets/d/e/2PACX-1vS183r0rW_jRX6tG_E90m9qVuFiMbRIJvi5VAE8yYOvEOIEEc3aSNuEsrFbuXw5c3nGboxMmrupZD7K/pubhtml
Reviewed-by: DJ Delorie <dj@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
