git/t/t5539-fetch-http-shallow.sh

158 lines
4.1 KiB
Bash
Raw Normal View History

#!/bin/sh
test_description='fetch/clone from a shallow clone over http'
GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
tests: mark tests relying on the current default for `init.defaultBranch` In addition to the manual adjustment to let the `linux-gcc` CI job run the test suite with `master` and then with `main`, this patch makes sure that GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME is set in all test scripts that currently rely on the initial branch name being `master by default. To determine which test scripts to mark up, the first step was to force-set the default branch name to `master` in - all test scripts that contain the keyword `master`, - t4211, which expects `t/t4211/history.export` with a hard-coded ref to initialize the default branch, - t5560 because it sources `t/t556x_common` which uses `master`, - t8002 and t8012 because both source `t/annotate-tests.sh` which also uses `master`) This trick was performed by this command: $ sed -i '/^ *\. \.\/\(test-lib\|lib-\(bash\|cvs\|git-svn\)\|gitweb-lib\)\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' $(git grep -l master t/t[0-9]*.sh) \ t/t4211*.sh t/t5560*.sh t/t8002*.sh t/t8012*.sh After that, careful, manual inspection revealed that some of the test scripts containing the needle `master` do not actually rely on a specific default branch name: either they mention `master` only in a comment, or they initialize that branch specificially, or they do not actually refer to the current default branch. Therefore, the aforementioned modification was undone in those test scripts thusly: $ git checkout HEAD -- \ t/t0027-auto-crlf.sh t/t0060-path-utils.sh \ t/t1011-read-tree-sparse-checkout.sh \ t/t1305-config-include.sh t/t1309-early-config.sh \ t/t1402-check-ref-format.sh t/t1450-fsck.sh \ t/t2024-checkout-dwim.sh \ t/t2106-update-index-assume-unchanged.sh \ t/t3040-subprojects-basic.sh t/t3301-notes.sh \ t/t3308-notes-merge.sh t/t3423-rebase-reword.sh \ t/t3436-rebase-more-options.sh \ t/t4015-diff-whitespace.sh t/t4257-am-interactive.sh \ t/t5323-pack-redundant.sh t/t5401-update-hooks.sh \ t/t5511-refspec.sh t/t5526-fetch-submodules.sh \ t/t5529-push-errors.sh t/t5530-upload-pack-error.sh \ t/t5548-push-porcelain.sh \ t/t5552-skipping-fetch-negotiator.sh \ t/t5572-pull-submodule.sh t/t5608-clone-2gb.sh \ t/t5614-clone-submodules-shallow.sh \ t/t7508-status.sh t/t7606-merge-custom.sh \ t/t9302-fast-import-unpack-limit.sh We excluded one set of test scripts in these commands, though: the range of `git p4` tests. The reason? `git p4` stores the (foreign) remote branch in the branch called `p4/master`, which is obviously not the default branch. Manual analysis revealed that only five of these tests actually require a specific default branch name to pass; They were modified thusly: $ sed -i '/^ *\. \.\/lib-git-p4\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' t/t980[0167]*.sh t/t9811*.sh Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-19 07:44:19 +08:00
export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
. ./test-lib.sh
. "$TEST_DIRECTORY"/lib-httpd.sh
start_httpd
commit() {
echo "$1" >tracked &&
git add tracked &&
t5539: make timestamp requirements more explicit The test for "no shallow lines after receiving ACK ready" is very sensitive to the timestamps of the commits we create. It's looking for the fetch negotiation to send a "ready", which in turn depends on the order in which we traverse commits during the negotiation. It works reliably now because the base commit "7" is created without test_commit, and thus gets a commit time matching the current system clock. Whereas the new commits created in this test do use test_commit, and get the usual test_tick time from 2005. So the fetch into the "clone" repository results in a commit graph like this (I omitted some of the "unrelated" commits for clarity; they're all just a sequence of test_ticks): $ git log --graph --format='%ct %s %d' * 1112912953 new (origin/master, origin/HEAD) * 1594322236 7 (grafted, master) * 1112912893 unrelated15 (origin/unrelated15, unrelated15) [...] * 1112912053 unrelated1 (origin/unrelated1, unrelated1) * 1112911993 new-too (HEAD -> newnew, tag: new-too) The important things to see are: - "7" is way in the future compared to the other commits - "new-too" in the fetching repo is older than "new" (and its "unrelated" ancestors) in the shallow repo If we change our "setup shallow clone" step to use test_tick, too (and get rid of the dependency on the system clock), then the test will fail. The resulting graph looks like this: $ git log --graph --format='%ct %s %d' * 1112913373 new (origin/master, origin/HEAD) * 1112912353 7 (grafted, master) * 1112913313 unrelated15 (origin/unrelated15, unrelated15) [...] * 1112912473 unrelated1 (origin/unrelated1, unrelated1) * 1112912413 new-too (HEAD -> newnew, tag: new-too) Our "new-too" is still older than "new" and "unrelated", but now "7" is older than all of them (because it advanced test_tick, which the other tests built on top of). In the original, we advertised "7" as the first "have" before anything else, but now "new-too" is more recent. You'd see the same thing in the unlikely event that the system clock was set before our test_tick default in 2005. Let's make the timing requirements more explicit. The important thing is that the client advertise all of its shared commits first, before presenting its unique "new-too" commit. We can do that and get rid of the system clock dependency at the same time by creating all of the shared commits around time X (using test_tick), and then creating "new-too" with some time long before X. The resulting graph looks like this: $ git log --graph --format='%ct %s %d' * 1500001380 new (origin/master, origin/HEAD) * 1500000420 7 (grafted, master) * 1500001320 unrelated15 (origin/unrelated15, unrelated15) [...] * 1500000480 unrelated1 (origin/unrelated1, unrelated1) * 1400000060 new-too (HEAD -> newnew, tag: new-too) That also lets us get rid of the hacky test_tick added by f0e802ca20 (t5539: update a flaky test, 2014-07-14). That was clearly dancing around the same problem, but only addressed the relationship between commits created in the two subshells (which did use test_tick, but overlapped because increments of test_tick in subshells are lost). Now that we're using consistent and well-placed times for both lines of history, we don't have to care about a one-tick difference between the two sides. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-10 04:39:09 +08:00
test_tick &&
git commit -m "$1"
}
test_expect_success 'setup shallow clone' '
t5539: make timestamp requirements more explicit The test for "no shallow lines after receiving ACK ready" is very sensitive to the timestamps of the commits we create. It's looking for the fetch negotiation to send a "ready", which in turn depends on the order in which we traverse commits during the negotiation. It works reliably now because the base commit "7" is created without test_commit, and thus gets a commit time matching the current system clock. Whereas the new commits created in this test do use test_commit, and get the usual test_tick time from 2005. So the fetch into the "clone" repository results in a commit graph like this (I omitted some of the "unrelated" commits for clarity; they're all just a sequence of test_ticks): $ git log --graph --format='%ct %s %d' * 1112912953 new (origin/master, origin/HEAD) * 1594322236 7 (grafted, master) * 1112912893 unrelated15 (origin/unrelated15, unrelated15) [...] * 1112912053 unrelated1 (origin/unrelated1, unrelated1) * 1112911993 new-too (HEAD -> newnew, tag: new-too) The important things to see are: - "7" is way in the future compared to the other commits - "new-too" in the fetching repo is older than "new" (and its "unrelated" ancestors) in the shallow repo If we change our "setup shallow clone" step to use test_tick, too (and get rid of the dependency on the system clock), then the test will fail. The resulting graph looks like this: $ git log --graph --format='%ct %s %d' * 1112913373 new (origin/master, origin/HEAD) * 1112912353 7 (grafted, master) * 1112913313 unrelated15 (origin/unrelated15, unrelated15) [...] * 1112912473 unrelated1 (origin/unrelated1, unrelated1) * 1112912413 new-too (HEAD -> newnew, tag: new-too) Our "new-too" is still older than "new" and "unrelated", but now "7" is older than all of them (because it advanced test_tick, which the other tests built on top of). In the original, we advertised "7" as the first "have" before anything else, but now "new-too" is more recent. You'd see the same thing in the unlikely event that the system clock was set before our test_tick default in 2005. Let's make the timing requirements more explicit. The important thing is that the client advertise all of its shared commits first, before presenting its unique "new-too" commit. We can do that and get rid of the system clock dependency at the same time by creating all of the shared commits around time X (using test_tick), and then creating "new-too" with some time long before X. The resulting graph looks like this: $ git log --graph --format='%ct %s %d' * 1500001380 new (origin/master, origin/HEAD) * 1500000420 7 (grafted, master) * 1500001320 unrelated15 (origin/unrelated15, unrelated15) [...] * 1500000480 unrelated1 (origin/unrelated1, unrelated1) * 1400000060 new-too (HEAD -> newnew, tag: new-too) That also lets us get rid of the hacky test_tick added by f0e802ca20 (t5539: update a flaky test, 2014-07-14). That was clearly dancing around the same problem, but only addressed the relationship between commits created in the two subshells (which did use test_tick, but overlapped because increments of test_tick in subshells are lost). Now that we're using consistent and well-placed times for both lines of history, we don't have to care about a one-tick difference between the two sides. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-10 04:39:09 +08:00
test_tick=1500000000 &&
commit 1 &&
commit 2 &&
commit 3 &&
commit 4 &&
commit 5 &&
commit 6 &&
commit 7 &&
git clone --no-local --depth=5 .git shallow &&
git config --global transfer.fsckObjects true
'
test_expect_success 'clone http repository' '
git clone --bare --no-local shallow "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
git clone $HTTPD_URL/smart/repo.git clone &&
(
cd clone &&
git fsck &&
git log --format=%s origin/main >actual &&
cat <<EOF >expect &&
7
6
5
4
3
EOF
test_cmp expect actual
)
'
# This test is tricky. We need large enough "have"s that fetch-pack
# will put pkt-flush in between. Then we need a "have" the server
# does not have, it'll send "ACK %s ready"
test_expect_success 'no shallow lines after receiving ACK ready' '
(
cd shallow &&
for i in $(test_seq 15)
do
git checkout --orphan unrelated$i &&
test_commit unrelated$i &&
git push -q "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" \
refs/heads/unrelated$i:refs/heads/unrelated$i &&
git push -q ../clone/.git \
refs/heads/unrelated$i:refs/heads/unrelated$i ||
exit 1
done &&
git checkout main &&
test_commit new &&
git push "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" main
) &&
(
cd clone &&
git checkout --orphan newnew &&
t5539: make timestamp requirements more explicit The test for "no shallow lines after receiving ACK ready" is very sensitive to the timestamps of the commits we create. It's looking for the fetch negotiation to send a "ready", which in turn depends on the order in which we traverse commits during the negotiation. It works reliably now because the base commit "7" is created without test_commit, and thus gets a commit time matching the current system clock. Whereas the new commits created in this test do use test_commit, and get the usual test_tick time from 2005. So the fetch into the "clone" repository results in a commit graph like this (I omitted some of the "unrelated" commits for clarity; they're all just a sequence of test_ticks): $ git log --graph --format='%ct %s %d' * 1112912953 new (origin/master, origin/HEAD) * 1594322236 7 (grafted, master) * 1112912893 unrelated15 (origin/unrelated15, unrelated15) [...] * 1112912053 unrelated1 (origin/unrelated1, unrelated1) * 1112911993 new-too (HEAD -> newnew, tag: new-too) The important things to see are: - "7" is way in the future compared to the other commits - "new-too" in the fetching repo is older than "new" (and its "unrelated" ancestors) in the shallow repo If we change our "setup shallow clone" step to use test_tick, too (and get rid of the dependency on the system clock), then the test will fail. The resulting graph looks like this: $ git log --graph --format='%ct %s %d' * 1112913373 new (origin/master, origin/HEAD) * 1112912353 7 (grafted, master) * 1112913313 unrelated15 (origin/unrelated15, unrelated15) [...] * 1112912473 unrelated1 (origin/unrelated1, unrelated1) * 1112912413 new-too (HEAD -> newnew, tag: new-too) Our "new-too" is still older than "new" and "unrelated", but now "7" is older than all of them (because it advanced test_tick, which the other tests built on top of). In the original, we advertised "7" as the first "have" before anything else, but now "new-too" is more recent. You'd see the same thing in the unlikely event that the system clock was set before our test_tick default in 2005. Let's make the timing requirements more explicit. The important thing is that the client advertise all of its shared commits first, before presenting its unique "new-too" commit. We can do that and get rid of the system clock dependency at the same time by creating all of the shared commits around time X (using test_tick), and then creating "new-too" with some time long before X. The resulting graph looks like this: $ git log --graph --format='%ct %s %d' * 1500001380 new (origin/master, origin/HEAD) * 1500000420 7 (grafted, master) * 1500001320 unrelated15 (origin/unrelated15, unrelated15) [...] * 1500000480 unrelated1 (origin/unrelated1, unrelated1) * 1400000060 new-too (HEAD -> newnew, tag: new-too) That also lets us get rid of the hacky test_tick added by f0e802ca20 (t5539: update a flaky test, 2014-07-14). That was clearly dancing around the same problem, but only addressed the relationship between commits created in the two subshells (which did use test_tick, but overlapped because increments of test_tick in subshells are lost). Now that we're using consistent and well-placed times for both lines of history, we don't have to care about a one-tick difference between the two sides. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-10 04:39:09 +08:00
test_tick=1400000000 &&
test_commit new-too &&
# NEEDSWORK: If the overspecification of the expected result is reduced, we
# might be able to run this test in all protocol versions.
GIT_TRACE_PACKET="$TRASH_DIRECTORY/trace" GIT_TEST_PROTOCOL_VERSION=0 \
git fetch --depth=2 &&
grep "fetch-pack< ACK .* ready" ../trace &&
! grep "fetch-pack> done" ../trace
)
'
test_expect_success 'clone shallow since ...' '
test_create_repo shallow-since &&
(
cd shallow-since &&
GIT_COMMITTER_DATE="100000000 +0700" git commit --allow-empty -m one &&
GIT_COMMITTER_DATE="200000000 +0700" git commit --allow-empty -m two &&
GIT_COMMITTER_DATE="300000000 +0700" git commit --allow-empty -m three &&
mv .git "$HTTPD_DOCUMENT_ROOT_PATH/shallow-since.git" &&
git clone --shallow-since "300000000 +0700" $HTTPD_URL/smart/shallow-since.git ../shallow11 &&
git -C ../shallow11 log --pretty=tformat:%s HEAD >actual &&
echo three >expected &&
test_cmp expected actual
)
'
test_expect_success 'fetch shallow since ...' '
git -C shallow11 fetch --shallow-since "200000000 +0700" origin &&
git -C shallow11 log --pretty=tformat:%s origin/main >actual &&
cat >expected <<-\EOF &&
three
two
EOF
test_cmp expected actual
'
test_expect_success 'shallow clone exclude tag two' '
test_create_repo shallow-exclude &&
(
cd shallow-exclude &&
test_commit one &&
test_commit two &&
test_commit three &&
mv .git "$HTTPD_DOCUMENT_ROOT_PATH/shallow-exclude.git" &&
git clone --shallow-exclude two $HTTPD_URL/smart/shallow-exclude.git ../shallow12 &&
git -C ../shallow12 log --pretty=tformat:%s HEAD >actual &&
echo three >expected &&
test_cmp expected actual
)
'
test_expect_success 'fetch exclude tag one' '
git -C shallow12 fetch --shallow-exclude one origin &&
git -C shallow12 log --pretty=tformat:%s origin/main >actual &&
test_write_lines three two >expected &&
test_cmp expected actual
'
fetch, upload-pack: --deepen=N extends shallow boundary by N commits In git-fetch, --depth argument is always relative with the latest remote refs. This makes it a bit difficult to cover this use case, where the user wants to make the shallow history, say 3 levels deeper. It would work if remote refs have not moved yet, but nobody can guarantee that, especially when that use case is performed a couple months after the last clone or "git fetch --depth". Also, modifying shallow boundary using --depth does not work well with clones created by --since or --not. This patch fixes that. A new argument --deepen=<N> will add <N> more (*) parent commits to the current history regardless of where remote refs are. Have/Want negotiation is still respected. So if remote refs move, the server will send two chunks: one between "have" and "want" and another to extend shallow history. In theory, the client could send no "want"s in order to get the second chunk only. But the protocol does not allow that. Either you send no want lines, which means ls-remote; or you have to send at least one want line that carries deep-relative to the server.. The main work was done by Dongcan Jiang. I fixed it up here and there. And of course all the bugs belong to me. (*) We could even support --deepen=<N> where <N> is negative. In that case we can cut some history from the shallow clone. This operation (and --depth=<shorter depth>) does not require interaction with remote side (and more complicated to implement as a result). Helped-by: Duy Nguyen <pclouds@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Dongcan Jiang <dongcan.jiang@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-12 18:54:09 +08:00
test_expect_success 'fetching deepen' '
test_create_repo shallow-deepen &&
(
cd shallow-deepen &&
test_commit one &&
test_commit two &&
test_commit three &&
mv .git "$HTTPD_DOCUMENT_ROOT_PATH/shallow-deepen.git" &&
git clone --depth 1 $HTTPD_URL/smart/shallow-deepen.git deepen &&
mv "$HTTPD_DOCUMENT_ROOT_PATH/shallow-deepen.git" .git &&
test_commit four &&
git -C deepen log --pretty=tformat:%s main >actual &&
fetch, upload-pack: --deepen=N extends shallow boundary by N commits In git-fetch, --depth argument is always relative with the latest remote refs. This makes it a bit difficult to cover this use case, where the user wants to make the shallow history, say 3 levels deeper. It would work if remote refs have not moved yet, but nobody can guarantee that, especially when that use case is performed a couple months after the last clone or "git fetch --depth". Also, modifying shallow boundary using --depth does not work well with clones created by --since or --not. This patch fixes that. A new argument --deepen=<N> will add <N> more (*) parent commits to the current history regardless of where remote refs are. Have/Want negotiation is still respected. So if remote refs move, the server will send two chunks: one between "have" and "want" and another to extend shallow history. In theory, the client could send no "want"s in order to get the second chunk only. But the protocol does not allow that. Either you send no want lines, which means ls-remote; or you have to send at least one want line that carries deep-relative to the server.. The main work was done by Dongcan Jiang. I fixed it up here and there. And of course all the bugs belong to me. (*) We could even support --deepen=<N> where <N> is negative. In that case we can cut some history from the shallow clone. This operation (and --depth=<shorter depth>) does not require interaction with remote side (and more complicated to implement as a result). Helped-by: Duy Nguyen <pclouds@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Dongcan Jiang <dongcan.jiang@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-12 18:54:09 +08:00
echo three >expected &&
test_cmp expected actual &&
mv .git "$HTTPD_DOCUMENT_ROOT_PATH/shallow-deepen.git" &&
git -C deepen fetch --deepen=1 &&
git -C deepen log --pretty=tformat:%s origin/main >actual &&
fetch, upload-pack: --deepen=N extends shallow boundary by N commits In git-fetch, --depth argument is always relative with the latest remote refs. This makes it a bit difficult to cover this use case, where the user wants to make the shallow history, say 3 levels deeper. It would work if remote refs have not moved yet, but nobody can guarantee that, especially when that use case is performed a couple months after the last clone or "git fetch --depth". Also, modifying shallow boundary using --depth does not work well with clones created by --since or --not. This patch fixes that. A new argument --deepen=<N> will add <N> more (*) parent commits to the current history regardless of where remote refs are. Have/Want negotiation is still respected. So if remote refs move, the server will send two chunks: one between "have" and "want" and another to extend shallow history. In theory, the client could send no "want"s in order to get the second chunk only. But the protocol does not allow that. Either you send no want lines, which means ls-remote; or you have to send at least one want line that carries deep-relative to the server.. The main work was done by Dongcan Jiang. I fixed it up here and there. And of course all the bugs belong to me. (*) We could even support --deepen=<N> where <N> is negative. In that case we can cut some history from the shallow clone. This operation (and --depth=<shorter depth>) does not require interaction with remote side (and more complicated to implement as a result). Helped-by: Duy Nguyen <pclouds@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Dongcan Jiang <dongcan.jiang@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-12 18:54:09 +08:00
cat >expected <<-\EOF &&
four
three
two
EOF
test_cmp expected actual
)
'
test_done