Commit Graph

1180 Commits

Author SHA1 Message Date
Pádraig Brady
99960eeab9 chroot: don't chdir() if not changing root
This allows chroot to be used as a light weight tool
to change user identification for a command,
while not changing the current working directory.
It also makes `chroot / true` consistently succeed on
all platforms for non root users.

* src/chroot.c (main): If the same root is specified. i.e. '/'
then don't change the current working directory, and avoid the
overhead of the other redundant calls.
* tests/misc/chroot-fail.sh: Remove failure guard previously
needed on some systems.  Also add an explicit case to ensure
we don't change directory.
* NEWS: Mention the change in behavior.
2014-05-16 22:01:31 +01:00
Pádraig Brady
2dc5d044a8 df: also deduplicate virtual file systems
* src/df.c (filter_mountlist): Remove the constraint that
a '/' needs to be in the device name for a mount entry to
be considered for deduplication.  Virtual file systems also
have storage associated with them (like tmpfs for example),
and thus need to be deduplicated since they will be shown
in the default df output and subject to --total processing also.
* test/df/skip-duplicates.sh: Add a test to ensure we deduplicate
all entries, even for virtual file systems.  Also avoid possible
length operations on many remote file systems in the initial
check of df operation.  Also avoid the assumption that "/root"
is on the same file system as "/".
* NEWS: Mention the change in behavior.
2014-05-13 23:23:30 +01:00
Paul Eggert
3974c0932d dd: fix conv=ascii, conv=ebcdic, conv=ibm to match POSIX
Problem reported by Don Baggett in <http:/bugs.gnu.org/17422>.
* NEWS:
* doc/coreutils.texi (dd invocation): Document this.
* src/dd.c (conversions): conv=ascii implies conv=unblock.
conv=ebcdic and conv=ibm imply conv=block.
(ascii_to_ebcdic, ebcdic_to_ascii): Correct to match
POSIX 1003.1-2013.
* tests/dd/ascii.sh: New file.
* tests/local.mk (all_tests): Add it.
2014-05-06 18:38:46 -07:00
Pádraig Brady
0b04ff22ed mv,cp: preserve symlink xattrs when copying across file systems
* src/copy.c (copy_internal): Include the copy_attr() call for symlinks.
This should not dereference symlinks, since llistxattr() is used
in attr_copy_file() in libattr, and so should copy all but the filtered
extended attributes.  Note we don't just move the copy_attr() call
before the set_owner() call, as that would break capabilities
for non symlinks.
* tests/cp/cp-mv-enotsup-xattr.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/16131
2014-05-06 10:26:50 +01:00
Pádraig Brady
be9f38bf6c numfmt: improve processing throughput by 800%
The devmsg() calls that took quote_n() arguments,
didn't normally output anything, but still incurred
the overhead of those quote_n() calls.

* src/numfmt.c (devmsg): Move the inline function
with _internal_ enablement check to...
* src/system.h: ...here as a variadic macro, with
the enablement check at the outer level.
* src/factor.c: As per numfmt.c but there is no
performance change in this case.
* NEWS: Mention the significant performance improvement.
2014-05-02 02:02:20 +01:00
Pádraig Brady
10a9652435 numfmt: support zero padding using --format="%010f"
* src/numfmt.c (setup_padding_buffer): Simplify the code by not
explicitly dealing with heap exhaustion.
(parse_format_string): Likewise.  Handle multiple grouping
modifiers as does the standard printf.  Handle the new leading
zero --format modifier.
(double_to_human): Use more defensive coding against overwriting
stack buffers.  Honor the leading zeros width.
(usage): Mention the leading zero --format modifier.
(main): Allow --padding in combo with a --format (width),
as the number of leading zeros are useful independent of
the main field width.
* doc/coreutils.texi (numfmt invocation): Likewise.
* tests/misc/numfmt.pl: Add new test cases.
* NEWS: Mention the improvement.
2014-05-02 01:59:31 +01:00
Pádraig Brady
f940fece04 ptx: fix whitespace trimming with multiple files
This issue was identified by running the test suite with
http://code.google.com/p/address-sanitizer/
which is included in GCC 4.8 and enabled with -fsanitize=address

This was checked on Fedora 20 with GCC 4.8 as follows:

  $ yum install libasan  # http://bugzilla.redhat.com/991003
  $ rm -f src/ptx.o
  $ make check AM_CFLAGS='-fsanitize=address' SUBDIRS=. VERBOSE=yes
  $ failure identified in tests/test-suite.log

To see this particular failure triggered with multiple files:

  $ src/ptx <(echo a) <(echo a) 2>&1 | asan_symbolize.py -d

=================================================================
==32178==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x60200000e74f at pc 0x435442 bp 0x7fffe8a1b290 sp 0x7fffe8a1b288
READ of size 1 at 0x60200000e74f thread T0
    #0 0x435441 in define_all_fields coreutils/src/ptx.c:1425
    #1 0x7fa206d31d64 in __libc_start_main ??:?
    #2 0x42f77c in _start ??:?
0x60200000e74f is located 1 bytes to the left of 3-byte region
[0x60200000e750,0x60200000e753) allocated by thread T0 here:
    #0 0x421809 in realloc ??:?
    #1 0x439b4e in fread_file coreutils/lib/read-file.c:97
Shadow bytes around the buggy address:
  0x0c047fff9c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9cc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9cd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fd fd
=>0x0c047fff9ce0: fa fa 03 fa fa fa fd fd fa[fa]03 fa fa fa 00 00
  0x0c047fff9cf0: fa fa 04 fa fa fa 04 fa fa fa fd fa fa fa fd fa
  0x0c047fff9d00: fa fa 00 fa fa fa fd fa fa fa 00 fa fa fa 00 fa
  0x0c047fff9d10: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c047fff9d20: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c047fff9d30: fa fa fd fa fa fa 00 fa fa fa 00 fa fa fa 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:     fa
  Heap right redzone:    fb
  Freed heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==32178==ABORTING

The initial report and high level analysis were from Jim Meyering...

  "The underlying problem is that swallow_file_in_memory()
  is setting the contents of the global text_buffer for the first file,
  then updating it (clobbering old value) for the second file.
  Yet, some pointers to the initial buffer have been squirreled away
  and later, one of them (keyafter) is presumed to point into
  the new "text_buffer", which it does not.  The subsequent
  SKIP_WHITE_BACKWARDS use backs up "cursor" and goes out of bounds."

* src/ptx.c (text_buffers): Maintain references for the limits of each
buffer corresponding to each file, rather than just the last processed.
(struct OCCURS): Add a member to map back to the corresponding file.
Note normally this could be computed from the "reference" member
rather than needing the extra storage, however this is not possible
when in --references mode.
(find_occurs_in_text): Reference the array rather than a single entry.
(define_all_fields): Likewise.  Also avoid computing the file index
since this is now stored directly.
(main): Update text_buffers[] array rather than a single text_buffer.
* tests/misc/ptx-overrun.sh: Even though this issue is already triggered
with AddressSanitizer, add a new case to demonstrate the whitespace
trimming issue, and to trigger without AddressSanitizer.
Fixes https://bugs.gnu.org/16171
2014-04-29 09:50:58 +01:00
Guilherme de Almeida Suckevicz
08783f100f ls: don't output colors with unknown TERM env variable
--colors controls whether to output colors depending on
whether we're connected to a terminal or not, while this
change gives control over which terminals we output colors to.

* NEWS: Mention the change in behavior.
* src/ls.c (known_term_type): A new function to search the static
list from dircolors.h
(parse_ls_colors): Honor the TERM when both LS_COLORS and COLORTERM
are non empty.
* tests/ls/color-term.sh: A new test.
* tests/local.mk: Reference the new test.
Fixes http://bugs.gnu.org/15992
2014-04-17 02:16:04 +01:00
Pádraig Brady
d2980924ac maint: document timeout --preserve-status in v8.21 NEWS
* NEWS: Add the omitted new feature.
* cfk.mk (old_NEWS_hash): Adjust accordingly.
2014-04-09 22:13:52 +01:00
Pádraig Brady
30acfcab50 doc: clarify in --help that -Z doesn't take an argument
* NEWS: Fix a confusing old entry.
* cfg.mk (old_NEWS_hash): Adjust accordingly.
* src/cp.c (usage): Separate the -Z and --context descriptions.
* src/install.c: Likewise.
* src/mkdir.c: Likewise.
* src/mkfifo.c: Likewise.
* src/mknod.c: Likewise.
Fixes http://bugs.gnu.org/17220
2014-04-09 14:17:06 +01:00
Pádraig Brady
b85eb8d683 shred: overwrite inode storage used by some file systems
* doc/coreutils.texi (shred invocation): Mention some reasons
why clearing slack space might be useful.
* src/shred.c (do_wipefd): Add initial writes for each pass
for small regular files in case the storage for those is
in the inode, and thus a larger write up to a block size would
bypass that.  Move the direct I/O control to...
(dopass): ... here so we can avoid enabling it for these small
initial writes.  It's better to retry direct I/O for each pass
anyway to handle the case where direct I/O is disabled for only
the last portion of a file when the size is not a multiple of
the block size.  Note we don't avoid the sync for the initial
write as it will be small but more importantly could be on a
different part of the disk and so worth doing independently
to ensure the write is not discarded.
* tests/misc/shred-exact.sh: Check some more direct I/O cases.
* NEWS: Mention the improvements.
The inode storage issue was mentioned by Paul Eggert.
2014-04-05 01:56:07 +01:00
Pádraig Brady
217618e8bf shred: avoid a data pass on empty files
* src/shred.c (do_wipefd): Don't increase the size written
for an empty file up to a full block.  Also increase the size
to OFF_T_MAX in the edge case where we do overflow.
* NEWS: Mention the shred improvements from recent changes.
* tests/misc/shred-passes.sh: Adjust as we no longer
write a BLKSIZE of data for empty files.
2014-04-04 05:32:41 +01:00
Paul Eggert
9111dd2499 head: port to Darwin and use simpler seeks
This removes an unportable assumption that if lseek succeeds, the
file is capable of seeking.  See: http://bugs.gnu.org/17145
* src/head.c (elseek): New function, for consistency in reporting
lseek failures.
(elide_tail_bytes_file, elide_tail_lines_seekable)
(elide_tail_lines_file, head_lines, head): Use it.
(elide_tail_bytes_file, elide_tail_lines_file):
New args CURRENT_POS and SIZE.  All uses changed.  Don't bother
invoking lseek, since we know the file's pos and size now.
(elide_tail_bytes_file): Change a local from uintmax_t to off_t,
since it fits.
(head): Use lseek only on regular files, since its behavior on
unseekable devices is implementation-defined.
* NEWS: Document this.
2014-04-02 07:45:58 -07:00
Paul Eggert
d08381bc26 head: fix bug with head -c-N when stdin is not at start
* src/head.c (elide_tail_bytes_file): Fix typo in lseek invocation.
* tests/misc/head-c.sh: Add test for this bug.
* NEWS: Document this.
2014-03-31 10:24:07 -07:00
Bernhard Voelker
5de5d8a5c0 ptx: avoid --format long option falling through into --help
* src/ptx.c (main): Add a 'break' after the --format handling case.
Otherwise it would fall through into the usage case.
* tests/misc/ptx.pl: Add test cases for --format=tex and --format=roff.
* NEWS (Bug fixes): Mention the fix.
Bug introduced in 1999-04-04 commit, SH-UTILS-1_16f-269-gd815c15.
Spotted by coverity (MISSING_BREAK).
2014-03-26 11:29:53 +01:00
Pádraig Brady
5f9a5b3f03 split: avoid unnecessary input buffering
Input buffering is best avoided because it introduces
delayed processing of output for intermittent input,
especially when the output size is less than that of
the input buffer.  This is significant when output
is being further processed which could happen if split
is writing to precreated fifos, or through --filter.

If input is arriving quickly from a pipe then this will
already be buffered before we read it, so fast arriving
input shouldn't be a performance issue.

* src/split.c (lines_split, lines_bytes_split, bytes_split,
lines_chunk_split, bytes_chunk_extract): s/full_read/safe_read/.
* THANKS.in: Mention the reporter.
* NEWS: Mention the improvement.
2014-03-24 09:10:20 +00:00
Dylan Simon
09eda9ed9d chmod: fix erroneous warnings with -R --changes
For files with "special" bits set, we would stat the relative
file name in the wrong directory, giving an erroneous ENOENT diagnostic.
This issue was introduced with commit v5.92-653-gc1994c1
which changed fts to not change directory on traversal.

* src/chmod.c (mode_changed): Use fts->fts_cwd_fd with fstatat rather
than stat.  All callers changed.
* tests/chmod/c-option.sh: Add a test case.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/17035
2014-03-19 02:33:44 +00:00
Jim Meyering
0093ac8d57 ln: with -sr, don't segfault for a TARGET of ''
Prior to this change, "ln -sr '' F" would segfault, attempting
to read path2[1] in relpath.c's path_common_prefix function.
This problem arises whenever canonicalize_filename_mode returns
NULL.
* src/ln.c (convert_abs_rel): Call relpath only when
both canonicalize_filename_mode calls return non-NULL.
* tests/ln/relative.sh: Add a test to trigger this failure.
* THANKS.in: List reporter's name/address.
* NEWS (Bug fixes): Mention it.
Reported by Erik Bernstein in 739752@bugs.debian.org.
Fixes http://bugs.gnu.org/17010.
2014-03-13 20:05:10 -07:00
Pádraig Brady
e972be3c4b chroot: improve --userspec and --groups look-up
- Support arbitrary numbers in --groups, consistent with
  what is already done for --userspec
- Avoid look-ups entirely for --groups items with a leading '+'
- Support names that are actually numbers in --groups
- Ignore an empty --groups="" option for consistency with --userspec
- Look up both inside and outside the chroot with inside taking
  precedence.  The look-up outside may load required libraries
  to complete the look-up inside the chroot.  This can happen for
  example with a 32 bit chroot on a 64 bit system, where the
  32 bit NSS plugins within the chroot fail to load.

* src/chroot.c (parse_additional_groups): A new function refactored
from set_addition_groups(), to just do the parsing.  The actual
setgroups() call is separated out for calling from the chroot later.
(main): Call parse_user_spec() and parse_additional_groups()
both outside and inside the chroot for the reasons outlined above.
* tests/misc/chroot-credentials.sh: Ensure arbitrary numeric IDs
can be specified without causing look-up errors.
* NEWS: Mention the improvements.
* THANKS.in: Add Norihiro Kamae who initially reported the issue
with a proposed patch.
Also thanks to Dmitry V. Levin for his diagnosis and sample patch.
2014-03-13 14:07:45 +00:00
Pádraig Brady
a4faa6a0a3 date: fix crash or infinite loop when parsing a malformed TZ=""
* NEWS: Mention the fix.
* gnulib: Update to incorporate the fix.
This is the only change in this gnulib update.
* tests/misc/date.pl: Add a test for this case.

Fixes http://bugs.gnu.org/16872
2014-02-27 23:56:28 +00:00
Bernhard Voelker
5475e6083f shuf: convert error diagnostic to lowercase
* src/shuf.c (main): s/No/no/, introduced by commit v8.22-25-g9f60f37.
* NEWS: Also adjust the NEWS for that recent commit to make it
clear this was new bug rather than a regression.

Prompted by the syntax-check rule sc_error_message_uppercase
2014-02-26 11:26:24 +00:00
Paul Eggert
9f60f37a28 shuf: with -r, don't dump core if the input is empty
Problem reported by valiant xiao in <http://bugs.gnu.org/16855>.
* NEWS: Document this.
* src/shuf.c (main): With -r, report an error if the input is empty.
* tests/misc/shuf.sh: Test for the bug.
2014-02-23 15:35:27 -08:00
Алексей Шилин
476ce37019 head: fix --lines=-0 outputting nothing if no newline at EOF
* src/head.c (elide_tail_lines_pipe): Just output all input in
this case to avoid the issue and also avoid redundant '\n' processing.
(elide_tail_lines_seekable): Likewise.
* tests/misc/head-elide-tail.pl: Add tests for no '\n' at EOF.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/16329
2014-02-09 20:22:33 +00:00
Niels Möller
b370924c03 od: add an --endian option to control byte swapping
* src/od.c (main): Handle the new --endian option,
taking "little" and "big" as parameters.
(usage): Describe the new option.
(PRINT_FIELDS): Adjust to swap bytes if required.
* tests/misc/od-endian.sh: A new test to verify
the byte swapping operations for hex (ints) and floats
for all sizes between 1 and 16 inclusive.
* test/local.mk: Reference the new test.
* doc/coreutils.texi (od invocation): Describe the new option.
* NEWS: Mention the new feature.
2014-02-09 11:29:24 +00:00
Pádraig Brady
b938b6e289 stat,tail: improve support for HFS+ and HFSX
* src/stat.c (human_fstype): Add new file system ID definitions.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/16336
2014-01-29 01:17:53 +00:00
Pádraig Brady
7fa1641db0 ln: fix replacing symbolic links whose targets can't exist
* src/ln.c (errno_nonexisting): A new function to determine if
the errno implies that a file doesn't or can't (currently) exist.
(target_directory_operand): Use the new function to expand the
set of errors we handle.
* tests/ln/sf-1.sh: Add test cases for the newly handled errors.
* THANKS.in: Mention the reporter.
* NEWS: Mention the bug fix.
2014-01-17 11:52:42 +00:00
Nicolas Iooss
d718331e59 copy: fix a segfault in SELinux context copying code
* src/selinux.c (restorecon_private): On ArchLinux the
`fakeroot cp -a file1 file2` command segfaulted due
to getfscreatecon() returning a NULL context.
So map this to the sometimes ignored ENODATA error,
rather than crashing.
* tests/cp/no-ctx.sh: Add a new test case.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/16335
2014-01-13 23:11:37 +00:00
Pádraig Brady
243128dbf0 copy: fix SELinux context preservation for existing directories
* src/copy.c (copy_internal): Use the global process context
to set the context of existing directories before they're populated.
This is more consistent with the new directory case, and fixes
a bug for existing directories where we erroneously set the
context to the last copied descendent, rather than to that of
the source directory itself.
* tests/cp/cp-a-selinux.sh: Add a test for this case.
* NEWS: Mention the fix.
* THANKS.in: Add reporter Michal Trunecka.
2014-01-13 12:52:26 +00:00
Bernhard Voelker
275c078fb4 maint: update all copyright year number ranges
Run "make update-copyright", but then also run this,
  perl -pi -e 's/2\d\d\d-//' tests/sample-test
to make that one script use the single most recent year number.
2014-01-02 22:19:59 +01:00
Pádraig Brady
c3e9f60a49 maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2013-12-13 15:20:31 +00:00
Pádraig Brady
3384461026 version 8.22
* NEWS: Record release date.
2013-12-13 15:03:45 +00:00
Pádraig Brady
ee7512cf00 stat,tail: improve support for hostfs and smackfs
hostfs is provided by the Linux UML subsystem.
smackfs is provided by the Linux Smack security module.

* src/stat.c (human_fstype): Add new file system ID definitions.
* NEWS: Mention the improvement, and adjust for the fact that
SNFS is a remote file system.
2013-12-13 14:05:53 +00:00
Pádraig Brady
cbdb4d77d9 build: default to avoiding openssl system libraries
* configure.ac: Don't change the gnulib default of 'no' for
whether to link with openssl system libraries if available.
Distributions can explicitly enable this as their policy allows.
* NEWS: Adjust accordingly.
2013-12-13 02:28:11 +00:00
Paul Eggert
e1b753b3e3 shuf: --repeat, not --repetitions; default --head-count is infinity
Original problem reported by Philipp Thomas in
<http://bugs.gnu.org/16061>.
* NEWS: shuf --repeat, not shuf --repetitions.
* doc/coreutils.texi (shuf invocation):
* src/shuf.c (usage, long_opts, main):
* tests/misc/shuf.sh:
Likewise.  Also, the default head-count is infinity.
2013-12-06 09:55:55 -08:00
Bernhard Voelker
34582a1aed doc: clarify that df now generally processes special files correctly
Since v8.21-172-g33660b4, df not only treats symbolic link arguments
differently, as stated there, but now generally processes special file
arguments in a non-canonicalized form correctly:

  $ cd /dev && df-old sdb
  Filesystem     1K-blocks  Used Available Use% Mounted on
  devtmpfs         1014572    48   1014524   1% /dev

  $ cd /dev && df-new sdb
  Filesystem     1K-blocks    Used Available Use% Mounted on
  /dev/sdb        10190136 6039532   3609932  63% /home

Document df's new behavior.

* doc/coreutils.texi (df invocation): In the paragraph describing
df's behavior regarding special file arguments, relax the condition
for such special files from "... is an absolute name of ..." to
"... resolves to ...".
* NEWS (Bug fixes): Mention the new behavior also here.
2013-12-06 00:59:36 +01:00
Pádraig Brady
b53b0fd940 md5sum, sha*sum: use libcrypto where available
libcrypto is generally available and has well optimized
crypto hash routines particular to various systems.
For example, testing sha1sum with openssl-1.0.0j
on an i3-2310M, gives a performance boost of about 40%:

$ time sha1sum.old --tag ~/test.iso
SHA1 (/home/padraig/test.iso) = 3c27f7ed01965fd2b89e22128fd62dc51a3bef30
real    0m4.692s
user    0m4.499s
sys     0m0.162s

$ time sha1sum.new --tag ~/test.iso
SHA1 (/home/padraig/test.iso) = 3c27f7ed01965fd2b89e22128fd62dc51a3bef30
real    0m2.685s
user    0m2.512s
sys     0m0.170s

* configure.ac: By default, enable use of libcrypto if available.
* src/local.mk: Link with libcrypto.
* NEWS: Mention the md5sum and sha*sum improvements.
2013-12-04 13:18:56 +00:00
Pádraig Brady
2091f44993 df: handle bind mounts when processing device nodes
* src/df.c (get_disk): Use the same heuristic used in
get_point() to select the shortest file system mount point,
in an attempt to show the base mounted file system.
* NEWS: Mention the bug fix.
2013-12-04 13:17:17 +00:00
Pádraig Brady
33660b4973 df: dereference symlinks to disk device nodes
This is so the matching for the device is done on the canonical name
of the disk node, rather than on the path of the symlink.
In any case the user will generally want to use the symlink target.

* src/df.c (get_disk): Canonicalize the passed file,
before matching against the list of mounted file system devices.
Note we pass the original symlink name to the "file" output field,
as the symlink target is usually available through the "source" field.
* tests/df/df-symlink.sh: Test the dereferencing operation.
* tests/local.mk: Mention the new test.
* NEWS: Mention the fix.
Reported by Ondrej Oprala
2013-12-04 13:16:27 +00:00
Bernhard Voelker
64520354bb df: remove long-obsolescent --megabytes option
This option has been undocumented for 12 years [1], and warned
about for a year [2].

[1] commit FILEUTILS-4_1_4-23-gd177203
[2] commit v8.17-43-g453ce92

* src/df.c (MEGABYTES_OPTION): Remove.
(long_options): Remove "megabytes" element.
(main): In the option parsing loop, remove the MEGABYTES_OPTION case.
* NEWS: Mention the change.
2013-11-29 13:07:47 +01:00
Pádraig Brady
7958a4a4fe selinux: adjust utils to run restorecon with -Z
cp, mv, install, mkdir, mkfifo, mknod are adjusted so that:
 -Z no longer accepts an argument.
 -Z or --context without an argument do not warn without SELinux.
 --context with an argument will warn without SELinux.

* src/local.mk: Reference the new selinux module where required.
* src/system.h: Make the argument to --context optional.
* src/mkdir.c: Likewise.  Also handle the SMACK case for --context.
Note we currently silently ignore -Z with SMACK.
* src/mkfifo.c: Likewise.
* src/mknod.c: Likewise.
* src/install.c: Likewise.  Note install(1) by default already
set the context for target files to their system default,
albeit with an older method.  Use the -Z option to select between
the old and new context restoration behavior, and document
the differences and details for how context restoration
is done in new and old methods, with a view disabling the
old method entirely in future.
* src/cp.c: Make the argument to --context optional.
Note -Z implies --no-preserve=context.  I.E. -Z overrides
that aspect of -a no matter what order specified.
(struct cp_options): Document the context handling options.
(main): Check/adjust option combinations after all
options are processed, to both simplify processing
and to make handling independent of order of options
on the command line.  Also improve the diagnostics
from a failed call to setfscreatecon().
(set_process_security_ctx): A new function,
refactored to set the default context from the source file,
or with the type adjusted as per the system default for
the destination path.
(set_file_security_ctx): A new function refactored to
set the security context of an existing file, either based on
the process context or the default system context for a path.
(copy_internal): Use the refactored functions to simplify
error handling and consistently fail or warn as needed.
(copy_reg): Likewise.
(copy_internal): With --preserve=context, also copy
context from non regular files.  Note for directories this may
impact the copying of subsequent files to that directory?
(copy_attr): If we're handling SELinux explicitly,
then exclude to avoid the redudant copy with --preserve=context,
and the problematic copy with -Z.  Note SELinux attribute exclusion
also now honors cp -a --no-preserve=context.  Note there was a
very small window over 10 years ago, where attr_copy_file was
available, while attr_copy_check_permissions was not, so we
don't bother adding an explicit m4 check for the latter function.
* src/mv.c: Support --context, but don't allow specifying an argument.
* src/chcon.c: Adjust a comment to be specific to SELinux.
* src/runcon.c: Likewise.
* src/copy.c: Honor the context settings to "restorecon" as appropriate.
* src/copy.h: Add a new setting to select "restorecon" functionality.
* tests/mkdir/selinux.sh: s/-Z/--context=/
* tests/cp/cp-a-selinux.sh: Augment this test with cases
testing basic -Z functionality, and also test the various
invalid option combinations and option precedence.
* tests/mkdir/restorecon.sh: Add a new test for the
more involved mkdir -Z handling, since the directory changing
and non existent directories need to be specially handled.
Also check the similar but simpler handling of -Z by mk{nod,fifo}.
* tests/local.mk: Reference the new test.
* doc/coreutils.texi (cp invocation): Update as per interface changes.
(mv invocation): Likewise.
(install invocation): Likewise.
(mkfifo invocation): Likewise.
(mknod invocation): Likewise.
(mkdir invocation): Likewise.
* NEWS: Mention the new feature and change in behavior.
2013-11-27 14:19:32 +00:00
Pádraig Brady
569b4edd18 shred: provide --remove methods to avoid excessive syncing
A sync operation is very often expensive.  For illustration
I timed the following python script which indicated that
each ext4 dir sync was taking about 2ms and 12ms, on an
SSD and traditional disk respectively.

  import os
  d=os.open(".", os.O_DIRECTORY|os.O_RDONLY)
  for i in range(1000):
     os.fdatasync(d)

So syncing for each character for each file can result
in significant delays.  Often this overhead is redundant,
as only the data is sensitive and not the file name.
Even if the names are sensitive, your file system may
employ synchronous metadata updates, which also makes
explicit syncing redundant.

* tests/misc/shred-remove.sh: Ensure all the new parameters
actually unlink the file.
* doc/coreutils.texi (shred invocation): Describe the new
parameters to the --remove option.
* src/shred.c (Usage): Likewise.
(main): Parse the new options.
(wipename): Inspect the new enum to see which of
the now optional tasks to perform.
* NEWS: Mention the new feature.
* THANKS.in: Add reporter Joseph D. Wagner
2013-11-27 01:43:12 +00:00
Pádraig Brady
ba6582e95c tail: improve inotify handling of symlinks
Previous behavior failed to read contents of a (re)appearing file,
when symlinked by tail's watched file.  Also we now diagnose other
edge cases when running in inotify mode, where an initially
missing or regular file changes to a symlink.

* src/tail.c (main): If any arg is a symlink, use polling mode.
(recheck): Diagnose the edge case where a symlink appears during
inotify processing.
* tests/tail-2/symlink.sh: Test the fix. Mention the edge cases.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
Reported by: Ondrej Oprala
2013-11-27 01:40:08 +00:00
Pádraig Brady
799e10f361 df: add --output=file to directly output specified arguments
* src/df.c (usage): Document the new 'file' --output field.
(get_dev): Add a new parameter to pass the specified
argument from the command line through.  Use '-' if a
command line parameter is not being used.
* doc/coreutils.texi (df invocation): Describe the new 'file' field.
* tests/df/df-output.sh: Adjust all fields test, and
add a specific test for --output=file.
* NEWS: Mention the new feature.
2013-11-27 01:39:11 +00:00
Colin Leitner
4e37fc642f stty: add support for mark/space parity
This adds support for using a constant or "stick" parity bit.

* src/stty.c (usage): Mention the new flag.
* tests/misc/stty.sh: Adjust for the new flag.
* NEWS: Mention the improvement.
* docs/coreutils.texi (stty invocation): Mention the new flag.
2013-11-17 12:26:12 +00:00
Pádraig Brady
9120845375 base64: improve encoding I/O efficiency
Since the I/O overhead is significant to the relatively
simple processing done by this utility, use fputs() rather
than fputc() to output '\n'.
Time to process a 100MiB file was measured to
decrease from 0.417s to 0.383s, i.e. an 8% improvement.

Related to these changes, is a processing improvement in
gnulib, which increases throughput by 60% when processing
full buffers, which improves processing of a 100MiB file
with standard wrapped output, down to 0.256s.
http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=commit;h=43fd1e7b

Also increase the encoding buffer size from 3 to 30KiB.
This was seen to give a further 8% improvement, taking
processing time down to 0.235s in the wrapped output case.
The decoding size buffer is not adjusted,
due to the noted caveat with --ignore-garbage.

* src/base64.c (BLOCKSIZE): Split into ENC_ and DEC_ variants,
with the former increased from 3KiB to 30KiB.
(wrap_write): Use the simpler fputc() rather than fputs()
to output the '\n' character.  Also check against EOF
rather than < 0 for errors.
(do_encode): Likewise.
* NEWS: Mention the large increase in performance, which
with the I/O improvements in coreutils and the processing
improvement in gnulib, amount to about a 60% throughput increase.
2013-11-11 23:30:39 +00:00
Pádraig Brady
c2eaef2f57 shred: increase I/O block size for periodic pattern case
* src/shred.c (dopass): In the periodic pattern case increase the
I/O block size from 12KiB to 60KiB (also a multiple of 3 and 4096).
* NEWS: Adjust accordingly.
2013-11-08 13:39:35 +00:00
Pádraig Brady
c93d5b4aa7 shred: fix direct I/O failures for last write to file
Since direct I/O is now enabled with commit v8.21-139-gebaf961
we must handle the case where we write an odd size at the
end of a file (with --exact), or we specify an odd --size that
is larger than 64KiB, or in the very unlikely case of a device
with an odd size.  This issue was present since direct I/O
support was first added in v5.3.0, but latent since v6.0.
Theoretically this could have also been an issue after that on
systems which didn't have alignment constraints, but did have
size constraints for direct I/O.

* src/shred.c (dopass): On the first pass for a file, always
retry a write that fails with EINVAL, so we handle direct I/O
failure at either the start or end of the file.  Adjust the comment
as the original case is out of date and implicitly handled
by this more general fix.
* tests/misc/shred-exact.sh: Add a test case.
* NEWS: Add a "bug fix" entry for shred since there are
two related issues now fixed.
2013-11-08 13:33:50 +00:00
Gian Piero Carrubba
bf6bf52dce cp: fix --link regarding the dereferencing of symbolic links
* src/copy.c (create_hard_link): Add a bool 'dereference' parameter,
and pass AT_SYMLINK_FOLLOW as 'flags' to linkat() when dereference
is true.
(should_dereference): Add new 'bool' function to determine if a
file should be dereferenced or not.
(copy_internal): Use the above new should_dereference() and remember
its return value in a new local bool 'dereference' variable.  Use that
in all three calls to create_hard_link().
* src/cp.c (main): after parsing the options, if x.dereference is
still DEFEF_UNDEFINED and the x.recursive is true, then only set
x.dereference to DEREF_NEVER iff --link was not specified.
* doc/coreutils.texi (cp invocation): Mention that cp(1) does not
follow symbolic links in the source when --link is specified.
Likewise in the description of the -R option when used together with
that option.
* tests/cp/same-file.sh: Adapt the expected results for the -fl,
the -bl and the -bfl tests.
* tests/cp/link-deref.sh: Add a new test.
* tests/local.mk (all_tests): Reference the above new test.
* NEWS (Changes in behavior): Mention the change.

This fixes http://bugs.gnu.org/15173

Co-authored-by: Bernhard Voelker <mail@bernhard-voelker.de>
2013-11-07 23:45:22 +01:00
Pádraig Brady
2c6736f92f shred: write larger chunks when possible
* src/shred.c (dopass): When not needing to write periodic patterns,
use a 64KiB block size to reduce the number of write system calls.
2013-11-06 17:43:13 +00:00
Pádraig Brady
ebaf961f3a shred: enable direct I/O when possible
Commit v5.92-1057-g43d487b introduced a regression
in coreutils 6.0 where it removed the page alignment
of the buffer to write, thus disabling direct I/O.
We want to use direct I/O when possible to avoid
impacting the page cache at least, as we know we don't
want to cache the data we're writing.

* src/shred.c (dopass): Allocate the buffer on the heap,
while using a more general calculation to allow to have
the output size independent from the fillpattern() size
constraint of a multiple of 3.  Also we dispense with the
union as it's no longer needed given we're aligning on
a page boundary and thus don't need to explicitly handle
uint32_t alignment.
2013-11-05 13:43:20 +00:00
Sergio Durigan Junior
5ee7d8f55b rm: with -I, prompt before deleting a write protected file
This regression was introduced in commit v6.7-71-g0928c24

* src/rm.c (main): Make the -I option behave like --interactive=once.
* tests/rm/interactive-once.sh: Add cases for single and multiple files.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/9308
2013-09-26 11:25:05 +01:00
Pádraig Brady
7415d8d9f8 id: support specifying the user by user ID
* src/id.c (usage): Remove 'name' from the synopsis,
implying that one can also specify by user ID.
(main): Like chown(1), call parse_user_spec() to implement
user name or ID lookup with appropriate precedence.
* doc/coreutils.texi (id invocation): Mention that
a user ID is supported and how '+' affects lookup order.
* tests/misc/id-groups.sh: Remove test now subsumed into...
* tests/misc/id-uid.sh: New test covering new interface.
* tests/local.mk: Rename the test.
* NEWS: Mention the new feature.
Addresses http://bugs.gnu.org/15421
2013-09-22 03:27:17 +01:00
Bernhard Voelker
2a0d241f69 id: add -z, --zero option
* src/group-list.h (print_group_list): Add a parameter for the
delimiter of type char.
* src/group-list.c (print_group_list): Likewise, and use it instead
of a white space character to delimit the group entries.
* src/groups.c (main): Pass white space character to print_group_list().
* src/id.c (longopts):  Add array element for the new long option.
(usage): Document the new option.  While at it, fix the alignment
of the descriptions to match that of HELP_OPTION_DESCRIPTION.
(main): Define the bool flag opt_zero indicating the use of the
new option.  In the getopt_long loop, handle it.
Output an error diagnostic in the case the --zero option has been
specified together with the default format.
In the case of -gG, pass either a NUL or a white space character to
print_group_list() - depending on the above new flag.
Likewise change the printing of the final newline character: output
a NUL instead if the --zero option has been specified.
* doc/coreutils.texi (id invocation): Document the new option.
While at it, move the @exitstatus macro down after the macro
@primaryAndSupplementaryGroups in order to be consistent with
other texinfo documents.
(groups invocation): Move @exitstatus down after the macro
@primaryAndSupplementaryGroups here, too.
* tests/misc/id-zero.sh: Add new test exercising the new option.
* tests/local.mk (all_tests): Reference it.
* NEWS (New features): Mention the new option.
Fixes http://bugs.gnu.org/9987
2013-09-21 14:15:45 +02:00
FUJIWARA Katsunori
b86437e126 ls: fix possible incorrect exit status when recursing directories
If there is an error reading a directory that was referenced
through recursion, rather than directly on the command line,
then exit with the "less serious" exit code, rather than the
"serious" exit code reserved for command line arguments.
This issue was introduced in commit v5.2.1-1908-gb58dea5

* src/ls.c (print_dir): Ensure that the command_line_arg param
is false for directories being recursed into.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/15249
2013-09-05 02:50:11 +01:00
Bernhard Voelker
333dc83d52 du: add --inodes option
This new option can be used to find directories with a huge
amount of files.  The GNU find utility has the printf format
"%h" which prints the number of entries in a directory, but
this is non-cumulative and doesn't handle hard links.

* src/du.c (struct duinfo): Add new member for counting inodes.
(duinfo_init): Initialize inodes member with Zero.
(duinfo_set): Set inodes counter to 1.
(duinfo_add): Sum up the 2 given inodes counters.
(opt_inodes): Add new boolean flag to remember if the --inodes
option has been specified.
(INODES_OPTION): Add new enum value to be used ...
(long_options): ... here.
(usage): Add description of the new option.
(print_size): Pass inodes counter or size to print_only_size,
depending on the inodes mode.
(process_file): Adapt threshold handling: with --inodes, print or
elide the entries according to the struct member inodes.
(main): Add a case for accepting the new INODES_OPTION.
Print a warning diagnostic when --inodes is used together with the
option --apparent-size or -b.
Reset the output_block_size to 1 ... and thus ignoring the
options -m and -k.
* tests/du/inodes.sh: Add a new test.
* tests/local.mk (all_tests): Mention it.
* doc/coreutils.texi (du invocation): Document the new option.
* NEWS: Mention the new option.
2013-07-27 14:25:28 +02:00
Ken Booth
2bdb74ec1a mv: replace empty directories in cross file system move
src/copy.c (copy_internal): Use rmdir() rather than unlink()
when the source is a directory, so that empty directories
are replaced in the destination as per POSIX.
* tests/mv/part-rename.sh: Augment with various combinations.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/14763
2013-07-25 16:45:02 +01:00
Pádraig Brady
e6fc265b75 dd: make status=none suppress all diagnostics
* src/dd.c (STATUS_NONE): Simplify the enum so that
it's more general than just suppressing transfer counts.
Then test this in all locations where non fatal diagnostics
are output.
* tests/dd/misc.sh: Ensure the diagnostic about
being unable to skip past the end of input is suppressed.
* NEWS: Mention the change in behavior.
Fixes http://bugs.gnu.org/14897
2013-07-19 11:23:12 +01:00
Assaf Gordon
3a84293987 shuf: add --repetition to support repetition in output
main(): Process new option.  Replace input_numbers_option_used()
with a local variable.  Re-organize argument processing.
usage(): Describe the new option.
(write_random_numbers): A new function to generate a
permutation of the specified input range with repetition.
(write_random_lines): Likewise for stdin and --echo.
(write_permuted_numbers):  New function refactored from
write_permuted_output().
(write_permuted_lines): Likewise.
* tests/misc/shuf.sh: Add tests for --repetitions option.
* doc/coreutils.texi: Mention --repetitions, add examples.
* TODO: Mention an optimization to avoid needing to
read all of the input into memory with --repetitions.
* NEWS: Mention new shuf option.
2013-07-11 01:17:31 +01:00
Pádraig Brady
616fa9c7b3 df: fix mount list processing with unstatable mount dirs
* src/df.c (filter_mount_list): Initialize devlist->dev_num correctly
when unable to stat() a mount point.  This will avoid possible invalid
deduplication done on the list due to use of uninitialized memory.
* tests/df/skip-duplicates.sh: Ensure this code path is exercised.
Also refactor the test to be table driven.
* NEWS: Mention the bug fix.
2013-07-09 02:36:27 +01:00
Jarkko Sakkinen
7d5976f668 mkdir,mkfifo,mknod: with -Z, create SMACK security context
Enable creation of SMACK security context with -Z command-line switch
if SMACK is enabled.

* mkdir.c (main): Set process security context to given SMACK label.
* mkfifo.c (main): Likewise.
* mknod.c (main): Likewise.
* src/local.mk: link mk{dir, fifo, nod} with libsmack.
* NEWS: Mention the new feature.
2013-07-01 14:33:05 +01:00
Jarkko Sakkinen
cd510a676d ls: with -Z, show SMACK security context
Enable showing of file SMACK security with '-Z' command-line switch
if SMACK is enabled.  Showing SMACK context of a file does not strictly
require SMACK to be enabled but this required to make choice whether to
show SELinux or SMACK security context.

* src/ls.c (getfilecon_cache): Retrieve SMACK context if available.
(gobble_file): Handle SMACK context similarly to SELinux context.
* src/local.mk: Link lsl with libsmack.
* NEWS: Mention the new feature.
* .mailmap: Merge the Author's 2 email addresses.
2013-06-24 21:00:14 +01:00
Pádraig Brady
bc70095df2 stdbuf: make it mandatory to specify a buffering option
This is consistent with the documented interface and
avoids any ambiguity in a user thinking that stdbuf without options
might reset to a "standard" buffering setup.

* src/stdbuf.c (set_libstdbuf_options): Indicate with the return value
whether any env variables were actually set.
(main): Fail unless some env variables were set.
* tests/misc/stdbuf.sh: Ensure this constraint is enforced.
* NEWS: Mention the small change in behavior.
2013-06-22 04:02:19 +01:00
Eric Blake
74cd098d2e doc: mention prior stat change
For a file of size 1234 bytes, commit ca9aa759 had the side effect
of changing 'stat -c "%'s" file' from outputting "?s" to the nicer
"1,234", depending on locale.  This is worth mentioning in the NEWS.

Resolves part of http://bugs.gnu.org/14556.

* NEWS: Mention 8.7 improvement in stat.
* cfg.mk (old_NEWS_hash): Adjust accordingly.
2013-06-05 14:22:25 -06:00
Jim Meyering
51a8f70713 od: -wN, N>64K, avoid misbehavior on systems with 32-bit size_t
* src/od.c (PRINT_FIELDS): Declare "i" to be of type uintmax_t, so that
the numerator in the expression for "next_pad" does not overflow.
(print_named_ascii): Likewise.
(print_ascii): Likewise.
Bug introduced via commit v6.12-42-g20c0b87.
* tests/misc/od.pl: Exercise each of the three affected code paths.
* NEWS (Bug fixes): Mention it.
Reported by Rich Burridge.
2013-06-02 09:25:38 -07:00
Pádraig Brady
fec363cebf split: with --line-bytes only allocate memory as needed
* src/split.c (line_bytes_split): Rewrite to only buffer
when necessary.  I.E. only increase the buffer when we've
already lines output in a split and we encounter a line
larger than the input buffer size, in which case a hold
buffer will be increased in increments of the input buffer size.
(lines_rr): Use the more abstract xalloc_die() just like
we did in line_bytes_split(), rather than explicitly
printing the "memory exhausted" message and exiting.
* tests/split/line-bytes.sh: Add a new test for this
function which previously had no test coverage.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/13537
2013-05-23 11:13:24 +01:00
Paul Eggert
e7198a67b8 mkdir: don't assume umask equals POSIX default ACL mask
This fixes Bug#14371, reported by Killer Bassist.
* NEWS: Document this.
* src/mkdir.c (struct mkdir_options): Remove member ancestor_mode.
New member umask_value.  All uses changed.
* src/mkdir.c (make_ancestor): Fix umask assumption.
* src/mkdir.c, src/mkfifo.c, src/mknod.c (main):
Leave umask alone.  This requires invoking lchmod after creating
the file, which introduces a race condition, but this can't be
avoided on hosts with "POSIX" default ACLs, and there's no easy
way with network file systems to tell what kind of host the
directory is on.
* tests/local.mk (all_tests): Add tests/mkdir/p-acl.sh.
* tests/mkdir/p-acl.sh: New file.
2013-05-11 19:22:54 -07:00
Jarkko Sakkinen
ca181728ee id: with -Z, show SMACK security context
Adds an optional dependency on libsmack.

* m4/jm-macros.m4: Look for the smack library/header.
* src/id.c (main): Output the smack context if available.
* src/local.mk: Link with libsmack if available.
* NEWS: Mention the new feature.
2013-05-08 11:55:08 +01:00
Pádraig Brady
e414ff4c4c stat,tail: improve support for snfs
The StorNext distributed file system was previously known as CVFS.

* src/stat.c (human_fstype): Add new file system ID definition.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/14251
2013-04-28 03:43:40 +01:00
Bernhard Voelker
d461bfd274 tail: let -f --retry wait for inaccessible files
The --retry option is indeed useful for both following modes
by name and by file descriptor.  The difference is that in the
latter case, it is effective only during the initial open.

As a regression of the implementation of the inotify support,
tail -f --retry would immediately exit if the given file is
inaccessible.

* src/tail.c (usage): Change the description of the --retry option:
remove the note that this option would mainly be useful when
following by name.
(main): Change diagnosing dubios uses of --retry option:
when the --retry option is used without following, then issue
a warning that this option is ignored; when it is used together
with --follow=descriptor, then issue a warning that it is only
effective for the initial open.
Disable inotify also in the case when the initial open in tail_file()
failed (which is the actual bug fix).
* init.cfg (retry_delay_): Pass excess arguments to the test function.
* tests/tail-2/retry.sh: Add new tests.
* tests/local.mk (all_tests): Mention it.
* doc/coreutils.texi (tail invocation): Enhance the documentation
of the --retry option.  Clarify the difference in tail's behavior
regarding the --retry option when combined with the following modes
name versus descriptor.
* NEWS (Bug fixes): Mention the fix.

Reported by Noel Morrison in:
http://lists.gnu.org/archive/html/coreutils/2013-04/msg00003.html
2013-04-20 16:33:06 +02:00
Assaf Gordon
4114c93af3 csplit: add the --suppress-matched option
With --suppress-matched, the lines that match the pattern will not be
printed in the output files.  I.E. the first line from the second
and subsequent splits will be suppressed.

* src/csplit.c: process_regexp(),process_line_count(): Don't output the
matched lines.  Since csplit includes "up to but not including" matched
lines in each split, the first line (in the next group) is the matched
line - so just skip it.
main(): Handle new option.
usage(): Mention new option.
* doc/coreutils.texi (csplit invocation): Mention new option, examples.
* tests/misc/csplit-suppress-matched.pl: New test script.
* tests/local.mk: Reference the new test.
* NEWS: Mention new feature.
2013-04-10 14:34:52 +01:00
Rémy Lefevre
e52293aa7f ln: --relative: fix updating of existing symlinks
Don't dereference an existing symlink being replaced.
I.E. generate the symlink relative to the symlink's containing dir,
rather than to some arbitrary place it points to.

* src/ln.c (convert_abs_rel): Don't consider the final component
of the symlink name when canonicalizing, as we want to avoid
dereferencing the final component.
* tests/ln/relative.sh: Add a test case.
* NEWS: Mention the fix.
Resolves http://bugs.gnu.org/14116
2013-04-04 02:46:35 +01:00
Assaf Gordon
20d7bce0f7 shuf: use reservoir-sampling for large or unknown sized inputs
Reservoir sampling optimizes selecting K random lines from large or
unknown-sized input: http://en.wikipedia.org/wiki/Reservoir_sampling
Note this also avoids reading any input when -n0 is specified.

* src/shuf.c (main): Use reservoir-sampling when the number of output
lines is known, and the input size is large or unknown.
(input_size): A new function to get the input size for regular files.
(read_input_reservoir_sampling): New function to read lines from input,
keeping only K lines in memory, replacing lines with decreasing prob.
(write_permuted_output_reservoir): New function to output reservoir.
* tests/misc/shuf-reservoir.sh: An expensive_ test using valgrind to
exercise the reservoir-sampling code.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
2013-03-25 20:07:14 +00:00
Pádraig Brady
4c49dc823f stat,tail: improve support for efivarfs, exofs, f2fs and ubifs
* src/stat.c (human_fstype): Add new file system ID definitions.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/14020
2013-03-24 02:30:20 +00:00
Torbjörn Granlund
9fe7c5b6eb build: fix factor build failure on aarch64
* src/longlong.h (__aarch64__): Make add_ssaaaa and sub_ddmmss work.
* NEWS: Mention the build fix.
Reported at https://bugzilla.redhat.com/917735
2013-03-04 18:09:49 +00:00
Assaf Gordon
374f569579 uniq: add the --group option
* src/uniq.c (usage): Summarize the new option,
and adjust the --all-repeated option to be more consistent.
(check_file): Merge the --group functionality into
the core loop for the default uniq operation since
it's very similar and can output lines immediately upon reading.
(main): Handle the new --group option and make it
mutually exclusive with other selection options.
* tests/misc/uniq.pl: Add tests.
* NEWS: Mention the new feature.
* doc/coreutils.texi (uniq invocation): Describe --group.
2013-02-28 18:20:30 +00:00
Assaf Gordon
6eb51ce29e join: Add the -z, --zero-terminated option
* NEWS: Mention join's new option: --zero-terminated (-z).
* src/join.c: Add new option, --zero-terminated (-z), to make
join use the NUL byte as separator/delimiter rather than newline.
(get_line): Use readlinebuffer_delim in place of readlinebuffer.
(main): Handle the new option.
(usage): Describe new option the same way sort does.
* doc/coreutils.texi (join invocation): Describe the new option.
* tests/misc/join.pl: add tests for -z option.
2013-02-28 01:49:56 +00:00
Ondrej Oprala
3a20f68885 install: cleanup properly if the strip program failed for any reason
* src/install.c (strip): Indicate failure with a return code instead
of terminating the program.
(install_file_in_file): Handle strip's return code and unlink the
created file if necessary.
* tests/install/strip-program.sh: Add a test to cover the changes.
* NEWS (Bug fixes): Mention the fix.
Reported by John Reiser in http://bugzilla.redhat.com/632444.
2013-02-22 21:31:32 +01:00
Pádraig Brady
44715883e1 maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2013-02-14 16:23:45 +00:00
Pádraig Brady
1cb8e6392c version 8.21
* NEWS: Record release date.
2013-02-14 15:52:18 +00:00
Ondřej Vašík
4b84062d66 stty: add support for DTR/DSR hardware control flow
Originally requested in Red Hat bugzilla #445213.

* src/stty.c (mode_info): Add support for DTR/DSR hardware flow control,
if available.
* doc/coreutils.texi: Document it.
* tests/misc/stty.sh: Add it to the list of serial options to avoid.
* NEWS: Mention the improvement.
2013-02-05 16:05:54 +00:00
Assaf Gordon
8a303a8963 numfmt: a new command to format numbers
* AUTHORS: Add my name.
* NEWS: Mention the new program.
* README: Reference the new program.
* src/numfmt.c: New file.
* src/.gitignore: Ignore the new binary.
* build-aux/gen-lists-of-programs.sh: Update.
* scripts/git-hooks/commit-msg: Allow numfmt: commit prefix.
* po/POTFILES.in: Add new c file.
* tests/misc/numfmt.pl: A new test file giving >93% coverage.
* tests/local.mk: Reference the new test.
* man/.gitignore: Ignore the new man page.
* man/local.mk: Reference the new man page.
* man/numfmt.x: A new template.
* doc/coreutils.texi: Document the new command.
2013-02-04 23:40:32 +00:00
Bernhard Voelker
f25519d688 df: do not treat rootfs specially
Like any other pseudo file system, df should show rootfs only
when the -a option is specified, i.e. specifying -trootfs alone
is not sufficient.  As the rootfs entry is now elided by the
general deduplication in filter_mount_list (commit v8.20-103-gbb116d3),
all other references to rootfs can be removed again.

* src/df.c (show_rootfs): Remove global variable.
(ROOTFS): Remove constant.
(filter_mount_list): Remove case to handle rootfs specially.
(main): In the case for handling the -t option, remove setting
of the show_rootfs variable.
* tests/df/skip-rootfs.sh: Adapt the test case "df -t rootfs":
the rootfs file system must not be printed (because no -a).
* doc/coreutils.texi (df invocation): Correct the documentation
about eliding mount entries: it is not the first occurrence of
the the device which wins, but now rather the entry with the
shortest mount point name.  Also adapt the description about
eliding pseudo file system types like rootfs.
* NEWS (Changes in behavior): Adapt entry.
2013-01-28 14:56:44 +01:00
Stephan Krempel
f8c0258d46 timeout: ensure a blocked SIGALRM from the parent is unblocked
* src/timeout.c (unblock_signal): A new function to unblock a
specified signal, or warn if not possible.
(set_timeout): Ensure SIGALRM is unblocked before we setup the timer.
* tests/misc/timeout-blocked.pl: A new test for the issue.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
Fixes: http://bugs.gnu.org/13535
2013-01-26 02:38:52 +00:00
Pádraig Brady
2238ab5741 seq: fix to always honor the step value
* src/seq.c (main): With 3 positive integer args we were
checking the end value was == "1", rather than the step value.
* tests/misc/seq.pl: Add tests for this case.
Reported by Marcel Böhme in http://bugs.gnu.org/13525
2013-01-26 02:37:13 +00:00
Pádraig Brady
326e5855bc seq: fix misaligment with -w when no precision for start value
* src/seq.c (get_default_format): Also account for the case where '.'
is auto added to the start value, which is significant when the
number sequence narrows.
* tests/misc/seq.pl: Add two new tests for the failing cases.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/13394
2013-01-26 02:35:58 +00:00
Pádraig Brady
51ce0bf844 cut: with -f, process each line independently
Previously line N+1 was inspected before line N was fully output,
which causes output ordering issues at the terminal or delays
from intermittent sources like tail -f.

* src/cut.c (cut_fields): Adjust so that we record the
previous output character so we can use that info to
determine wether to output a '\n' or not.
* tests/misc/cut.pl: Add tests to ensure existing
functionality isn't broken.
* NEWS: Mention the fix.
Fixes bug http://bugs.gnu.org/13498
2013-01-26 02:31:53 +00:00
Jakob Truelsen
f8afbb424c du: add -t SIZE, --threshold=SIZE option
* src/du.c (opt_threshold): Add variable to hold the value of
the --threshold option specified by the user.
(long_options): Add a required_argument entry for the new
--threshold option.
(usage): Add --threshold option.
(process_file): Elide printing the entry if its size does not
meet the value specified by the --threshold option.
(main): In the argument parsing loop, add a case for the new
-t option. Convert the given argument by permitting the
well-known suffixes for megabyte, gigabytes, etc.
Handle the special case "-0": give an error as this value is
not permitted.
* doc/coreutils.texi (du invocation): Add documentation for the
above new option.
* tests/du/threshold.sh: Add new test to exercise the new option.
* tests/local.mk (all_tests): Mention the above test.

Co-authored-by: Bernhard Voelker <mail@bernhard-voelker.de>
2013-01-21 11:38:54 +01:00
Pádraig Brady
05382299cd factor: apply a more general fix to enable correct assembly
In addition to the previous 64 bit guards we've placed in longlong.h
there are additional _LP64 guards required for mips with -mcpu >= 3,
to avoid a build failure (http://bugs.gnu.org/13353) and on sparc
with -mcpu >= v9 in 32 bit mode where for example,
`factor 2123123123123123123123` would go into an infinite loop.

Since factor.c currently operates on uintmax_t, we restrict the use
of the assembly in longlong.h to when 'long' has the same width, to
provide a more general guard for this code.

* src/factor.c: Restrict the use of longlong.h assembly code,
to when the width of intmax_t == long.
* src/longlong.h: Remove the previous _LP64 guards to avoid
divergence from GMP's longlong.h
* NEWS: Adjust the info on build and runtime fixes.
2013-01-07 02:11:36 +00:00
Jim Meyering
77da73c754 maint: update all copyright year number ranges
Run "make update-copyright", but then also run this,
  perl -pi -e 's/2\d\d\d-//' tests/sample-test
to make that one script use the single most recent year number.
2013-01-01 04:51:20 +01:00
Jim Meyering
a4d3a44d27 maint: adjust NEWS entry wording
* NEWS: Adjust wording in a few entries.
2012-12-31 15:05:55 +01:00
Mike Frysinger
b6b24ad49d factor: disable x86_64 assembly code for x32 builds
The current x86_64 asm code does not work for x32 (__ILP32__) ABIs,
so disable it.  Note simply deleting the q suffix is not enough.

* src/longlong.h: Restrict x86_64 assembly to _LP64 targets,
which is consistent with other checks in longlong.h and
avoids this code on x32.
* NEWS: Mention the build fix.
2012-12-21 15:57:50 +00:00
Pádraig Brady
76f606a954 seq: fix newline output when -s specified
This regression was introduced in commit v8.19-132-g3786fb6.

* src/seq.c (seq_fast): Don't use puts() to output the first number,
and instead insert it into the buffer as for other numbers.
Also output the terminator unconditionally.
* tests/misc/seq.pl: Add some basic tests for the -s option.
* NEWS: Mention the fix.
* THANKS.in: Reported by Philipp Gortan.
2012-12-21 15:55:19 +00:00
Pádraig Brady
a05a326b0c readlink: support multiple command line arguments
This allows efficient processing of multiple files,
while also increasing compatibility with BSD's readlink(1).
We also add the -z, --zero option to delimit output items
with the NUL character which disambiguates output in the
presence of '\n' characters.

* src/readlink.c (usage): Add the --zero description,
and also adjust the description of --no-newline accordingly.
(main): Handle the -z option and iterate over multiple arguments.
Also as in commit v8.15-24-g9d46b25 we use fputs() and putchar()
rather than printf() for performance reasons.
* doc/coreutils.texi (readlink invocation): Document the
new --zero option, adjust the --no-newline description, and
tweak the general info to indicate multiple files are supported.
* tests/readlink/multi.sh: A new test for the new functionality.
* tests/local.mk: Reference the new test.
* man/readlink.x: Adjust the summary and also reference realpath.
* NEWS: Mention the improvement.
* THANKS.in: Suggested by Aaron Davies.
2012-12-17 21:08:16 +00:00
Bernhard Voelker
c321f7450f tail,stat: improve support for the ceph file system
Teach tail -f that it must use polling on ceph file systems, and
let stat -f --format=%T report the file system type name, "ceph".
Website: http://ceph.com/

* src/stat.c (human_fstype): Add a case: ceph, 0x00C36400, remote.
* NEWS (Improvements): Mention it.
* THANKS.in: Update.
Reported by Konrad Wróblewski in http://bugs.gnu.org/13172.
2012-12-15 00:15:25 +01:00
Jim Meyering
4f62d2fe28 tests: add test case and note that last week's cut change is a bug fix
* tests/misc/cut-huge-to-eol-range.sh: New test, showing that
the change in v8.20-51-g7d03466 is a bug fix after all.
* tests/local.mk (all_tests): Add it.
* NEWS (Bug fixes): Mention it.
2012-12-09 15:08:36 -08:00
Bernhard Voelker
62543570d7 cp: fix --no-preserve=mode to not exit 1
cp --no-preserve=mode exited 1 unconditionally.  Furthermore,
the tests which would have detected this error - namely
link-preserve.sh and reserve-mode.sh - failed to test
cp's exit code.

* src/copy.c (copy_reg): In the case x->explicit_no_preserve_mode,
do only set return_val to false iff the previous set_acl ()
failed.
* tests/cp/link-preserve.sh: Check cp's exit code.
* tests/cp/link-symlink.sh: Likewise.
* tests/cp/preserve-mode.sh: Likewise.
* NEWS: Mention the fix.

Bug introduced in commit v8.19-145-g24ebca6.

Reported by Florian Pritz in http://bugs.gnu.org/13119.
2012-12-08 19:09:19 +01:00
Ondrej Oprala
10d35b438e df: do not print duplicate entries and rootfs by default
* src/df.c (struct devlist): Add new struct for storing already-
examined device numbers.
(devlist_head): Add new store of the above type.
(show_rootfs): Add new global boolean to not skip rootfs.
(dev_examined): Add new function to check if the device has
already been traversed.
(get_dev): Filter out rootfs unless "-t rootfs" or the -a
option is specified. Filter out duplicate entries by calling
the above new dev_examined unless the -a option is specified.
(main): Set the show_rootfs variable appropriately when the -t
option is specified for rootfs. Free device list (guarded by
IF_LINT).
* tests/df/skip-duplicates.sh: Add test to exercise the skipping
of duplicate entries.
* tests/df/skip-rootfs.sh: Add test to exercise the skipping
of the rootfs pseudo file system.
* tests/local.mk: Add the above new tests.
* NEWS (Changes in behavior): Mention the changes.
* doc/coreutils.texi (df invocation): Document df's behavior about
skipping rootfs and duplicate entries.

Co-authored-by: Bernhard Voelker.
2012-12-07 21:13:38 +01:00
Colin Watson
7d034663e4 factor: fix infinite loop on 32 bit powerpc
Both Debian and Ubuntu builds of coreutils 8.20 hang while running the
test suite on powerpc, which is reproducible using 'factor 122'.

This turns out to be somewhat related to http://bugs.gnu.org/12754,
but not quite the same.  uintmax_t is 64 bits, but the cntlzw
instruction takes 32-bit operands, and the cntlzd option is only
available on 64-bit hardware.

* src/longlong.h: Add an _LP64 check around the PPC64 code,
so that this falls back to the C implementations.
* NEWS: Mention the fix.
2012-12-04 12:28:42 +00:00
Jim Meyering
06aeeecb3f cut: do not print extraneous delimiters in some unusual cases
When printing output delimiters, and when a to-EOL range subsumes
at least one other range, cut would mistakenly print delimiters for
the subsumed range.  This bug was probably introduced via commit
v5.2.1-639-g847e066.
* src/cut.c (set_fields): Ignore any range that is subsumed by a
to-EOL range.  Also, move two declarations down.
* tests/misc/cut.pl: Add tests to exercise this.
* NEWS (Bug fixes): Mention it.
Reported by Marcel Böhme in http://bugs.gnu.org/12966
2012-11-24 15:23:31 -08:00
Jim Meyering
1b874511b6 cut: treat -b2-,3- like -b2-, not like -b3-
* src/cut.c (set_fields): When two right-open-ended ranges are
specified, don't blindly let the latter one take precedence over
the former.  Instead, use the union of the ranges.
* tests/misc/cut.pl: Add tests to exercise this.
* NEWS (Bug fixes): Mention it.
Reported by Marcel Böhme in http://bugs.gnu.org/12966
Thanks to Berhard Voelker for catching log and NEWS typos.
2012-11-24 15:23:28 -08:00