Memoizing these initialization functions saves some memory for
long fetches which require scanning many unwanted revisions
before any wanted revisions happen.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This should help me track down errors in git-svn more easily:
write .git/Git_XXXXXX: Bad file descriptor
at /usr/lib/perl5/SVN/Ra.pm line 623
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This memoization appears unneeded as the check_cherry_pick2 cache is
in front of it does enough.
With this change applied, importing from local svn+ssh and http copies
of the R repo[1] takes only 2:00 (2 hours) on my system and the git-svn
process never uses more than 60MB RSS on my x86-64 GNU/Linux system[2].
This 60M measurement is only for the git-svn Perl process itself and
does not include memory used by git subprocesses accessing large packs
(subprocess memory usage _is_ measured by my time(1) tool).
Before this change, an import took longer (2:20) on svn+ssh:// but
git-svn used around 240MB during the imports. Worse yet, git-svn
ballooned to over 400M when writing out the cache to the filesystem.
I also tried removing memoization for `has_no_changes', too, but a
local copy of the R repository(*) was not close to finishing within
10 hours on my system.
[1] http://svn.r-project.org/R
[2] file:// repos causes libsvn to use more memory internally
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
There is no reason to keep entries in the %revs hash after we're
done processing a revision, so allow entries become freed as
processing continues.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This override was probably never necessary, but most likely a no-op
as it does not appear to do anything in SVN::Ra itself.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Despite attempting to use local memory pools everywhere we can,
(including our call to SVN::Ra::do_update and all subsequent reporter
calls), there does not appear to be a way to force the Git::SVN::Fetcher
callbacks to use a pool other than the per-SVN::Ra pool.
Git::SVN::Fetcher ends up using the main RA pool which grows
monotonically in size for the lifetime of the RA object.
Thus the only way to free that memory appears to be to destroy and
recreate the RA connection for at every --log-window-size interval.
This reduces memory usage over the course of fetching 10K revisions
using a test repository created with the script at the end of this
commit message.
As reported by time(1) on my x86-64 system:
before: 54024k
after: 28680k
Unfortunately, there remains some yet-to-be-tracked-down slow memory
growth which would be evident as the `nr' parameter increases in
the repository generation script:
-----------------------------8<------------------------------
set -e
tmp=$(mktemp -d svntestrepo-XXXXXXXX)
svnadmin create "$tmp"
repo=file://"$(cd $tmp && pwd)"
svn co "$repo" "$tmp/wd"
cd "$tmp/wd"
if ! test -f a
then
> a
svn add a
svn commit -m 'A'
fi
nr=10000
while test $nr -gt 0
do
echo $nr > a
svn commit -q -m A
nr=$((nr - 1))
done
echo "repository created in $repo"
-----------------------------8<------------------------------
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Advertise that the svn-remote.<name>.pushurl config key allows specifying
the commit URL for the entire SVN repository in the documentation of the
git svn dcommit command.
Signed-off-by: Sveinung Kvilhaugsvik <sveinung84@users.sourceforge.net>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This should further reduce memory usage from the new mergeinfo
speedups without hurting performance too much, assuming
reasonable latency to the SVN server.
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This should reduce excessive memory usage from the new mergeinfo
caches without hurting performance too much, assuming reasonable
latency to the SVN server.
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
We do not need to store entire lists of commits, only the
number of incomplete and the first commit for reference.
This reduces the amount of data we need to store in memory
and on disk stores.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Subversion can put mergeinfo on any sub-directory to track cherry-picks.
Since cherry-picks are not represented explicitly in git, git-svn should
just ignore it.
Signed-off-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
In a Subversion repository where many feature branches are merged into a
trunk, the svn:mergeinfo property can grow very large. This severely
slows down git-svn's make_log_entry() because it is checking all
mergeinfo entries every time the property changes.
In most cases, the additions to svn:mergeinfo since the last commit are
pretty small, and there is nothing to gain by checking merges that were
already checked for the last commit in the branch.
Add a mergeinfo_changes() function which computes the set of interesting
changes to svn:mergeinfo since the last commit. Filter out merged
branches whose ranges haven't changed, and remove a common prefix of
ranges from other merged branches.
This speeds up "git svn fetch" by several orders of magnitude on a large
repository where thousands of feature branches have been merged.
Signed-off-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Code clean-up.
* sb/plug-transport-leak:
.mailmap: add Stefan Bellers corporate mail address
transport: free leaking head in transport_print_push_status()
Allow us build with NO_PTHREADS=NoThanks compilation option.
* eb/no-pthreads:
Handle atexit list internaly for unthreaded builds
pack-objects: set number of threads before checking and warning
index-pack: fix compilation with NO_PTHREADS
Add managed "env" array to child_process to clarify the lifetime
rules.
* rs/run-command-env-array:
use env_array member of struct child_process
run-command: add env_array, an optional argv_array for env
Splitting pack-objects output into multiple packs is incompatible
with the use of reachability bitmap.
* jk/pack-objects-no-bitmap-when-splitting:
pack-objects: turn off bitmaps when we split packs
push --signed promises to take user.signingkey as the signing key but
fails to read the config.
Make it do so.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some internal error messages leaked out of the bash completion when
typing "git cmd <TAB>" and the machinery tried to complete
refnames.
* js/completion-hide-not-a-repo:
completion: silence "fatal: Not a git repository" error
Newer versions of 'meld' breaks the auto-detection we use to see if
they are new enough to support the `--output` option.
* da/mergetool-meld:
mergetools/meld: make usage of `--output` configurable and more robust
Allow a temporary directory specified to be used while running "git
mergetool" backend.
* da/mergetool-temporary-directory:
t7610-mergetool: add test cases for mergetool.writeToTemp
mergetool: add an option for writing to a temporary directory
Allow "git mergetool --help" to run outside a Git repository.
* da/mergetool-tool-help:
difftool: don't assume that default sh is sane
mergetool: don't require a work tree for --tool-help
git-sh-setup: move GIT_DIR initialization into a function
mergetool: use more conservative temporary filenames
test-lib-functions: adjust style to match CodingGuidelines
t7610-mergetool: prefer test_config over git config
Tweak the names of the three throw-away files "git mergetool" comes
up with to feed the merge tool backend, so that a file with a
single dot in its name in the original (e.g. "hello.c") will have
only one dot in these variants (e.g. "hello_BASE_4321.c").
* da/mergetool-temporary-filename:
mergetool: use more conservative temporary filenames
The clean-up of this test script was long overdue and is a very
welcome change.
* da/mergetool-tests:
test-lib-functions: adjust style to match CodingGuidelines
t7610-mergetool: use test_config to isolate tests
t7610-mergetool: add missing && and remove commented-out code
t7610-mergetool: use tabs instead of a mix of tabs and spaces
The API to update refs have been restructured to allow introducing
a true transactional updates later. We would even allow storing
refs in backends other than the traditional filesystem-based one.
* rs/ref-transaction: (25 commits)
ref_transaction_commit: bail out on failure to remove a ref
lockfile: remove unable_to_lock_error
refs.c: do not permit err == NULL
remote rm/prune: print a message when writing packed-refs fails
for-each-ref: skip and warn about broken ref names
refs.c: allow listing and deleting badly named refs
test: put tests for handling of bad ref names in one place
packed-ref cache: forbid dot-components in refnames
branch -d: simplify by using RESOLVE_REF_READING
branch -d: avoid repeated symref resolution
reflog test: test interaction with detached HEAD
refs.c: change resolve_ref_unsafe reading argument to be a flags field
refs.c: make write_ref_sha1 static
fetch.c: change s_update_ref to use a ref transaction
refs.c: ref_transaction_commit: distinguish name conflicts from other errors
refs.c: pass a list of names to skip to is_refname_available
refs.c: call lock_ref_sha1_basic directly from commit
refs.c: refuse to lock badly named refs in lock_ref_sha1_basic
rename_ref: don't ask read_ref_full where the ref came from
refs.c: pass the ref log message to _create/delete/update instead of _commit
...
This "stk" shadows the first declaration at the top. There's currently
no bad effect. But let's avoid it.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Note that despite the private address being first and primary,
Google owns the copyright on this patch as any other patch I'll be
sending signed off by the sbeller@google.com address.
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A new filter to programatically edit the tail end of the commit log
messages.
* cc/interpret-trailers:
Documentation: add documentation for 'git interpret-trailers'
trailer: add tests for commands in config file
trailer: execute command from 'trailer.<name>.command'
trailer: add tests for "git interpret-trailers"
trailer: add interpret-trailers command
trailer: put all the processing together and print
trailer: parse trailers from file or stdin
trailer: process command line trailer arguments
trailer: read and process config information
trailer: process trailers from input message and arguments
trailer: add data structures and basic functions
Test scripts were taught to notice "-x" option to show shell trace,
as if the tests were run under "sh -x".
* jk/test-shell-trace:
test-lib.sh: support -x option for shell-tracing
t5304: use helper to report failure of "test foo = bar"
t5304: use test_path_is_* instead of "test -f"
Formatting nitpicks to help a (pickier) reimplementation of
AsciiDoc to grok our documentation.
* bc/asciidoc:
Documentation: fix mismatched delimiters in git-imap-send
Documentation: adjust document title underlining
This reverts commit 10f343ea81, whose
output is no longer bit-for-bit equivalent from the older versions
of Git, which the infrastructure to (pretend to) upload tarballs
kernel.org uses depends on.
core.filemode is set automatically when a repo is created.
But when a repo is exported via CIFS or cygwin is mixed with Git for Windows
or Eclipse core.filemode may better be set manually to false.
Update and improve the documentation
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Correct backtick quoting for some of the modification states to give
consistent web rendering.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Avoid:
# ./t1304-default-acl.sh
ok 1 - checking for a working acl setup
ok 2 - Setup test repo
not ok 3 - Objects creation does not break ACLs with restrictive umask
#
# # SHA1 for empty blob
# check_perms_and_acl .git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391
#
not ok 4 - git gc does not break ACLs with restrictive umask
#
# git gc &&
# check_perms_and_acl .git/objects/pack/*.pack
#
# failed 2 among 4 test(s)
1..4
on systems where USER isn't set. It's usually set by the login
process, but it isn't set when launching some Docker images. For
example:
$ docker run --rm debian env
HOME=/
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b2dfdfe797ed
'id -u -n' has been in POSIX from Issue 2 through 2013 [1], so I don't
expect compatibility issues.
[1]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/id.html
Signed-off-by: W. Trevor King <wking@tremily.us>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Wrap atexit()s calls on unthreaded builds to handle callback list
internally.
This is needed because on unthreaded builds, asyncs inherits parent's
atexit() list, that gets run as soon as the async exit()s (and again at
the end of async's parent process). That led to remove temporary files
too early.
Also remove a by-atexit-callback guard against this kind of issue in
clone.c, as this patch makes it redundant.
Fixes test 5537 (temporary shallow file vanished before unpack-objects
could open it)
BTW remove an unused variable in shallow.c.
Helped-by: Duy Nguyen <pclouds@gmail.com>
Helped-by: Andreas Schwab <schwab@linux-m68k.org>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Etienne Buira <etienne.buira@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Convert users of struct child_process to using the managed env_array for
specifying environment variables instead of supplying an array on the
stack or bringing their own argv_array. This shortens and simplifies
the code and ensures automatically that the allocated memory is freed
after use.
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
