From 9fbd3d85745fdf32dac06355e9c524270ae034bb Mon Sep 17 00:00:00 2001 From: "Yann E. MORIN" Date: Thu, 11 Jan 2024 11:38:17 +0100 Subject: [PATCH] Revert "support/download: generate even more reproducible tarballs" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commit 768f9f80f62c (support/download: generate even more reproducible tarballs) causes non-reproducibility in tarballs we previousy generated, especially the archives for two cargo-vendored packages, ripgrep and sentry-cli. The cause is that those two pakcages eventually vendor a file that has the u+x bit set, but is otehrwise go-x. With 768f9f80f62c, the files are now go+x, so the hash for those generated archives has changed. Besides, that commit was wrong: it did not account for the 'r' bit for go part, leaving some non-reproducibility still unaccounted for. So, to generate really reproducible archives, we would need to fix that read bit as well, and that has the potential to affect all the archives we generated so far. If we wanted to do so, we'd need a way to version all generated archives, like we do for git and svn, but now for all the different CVSes, as well as for all the vendoring post-processes. For 768f9f80f62c, all that was of conern was the working copies of CVSes (i.e. git, svn, cvs...) that we cache in the Buildroot download dir, not the temporary files during post-processing. Indeed, in that latter case, the user has virtually no way to mangle with the mode of the intermediate extract before repack. And we do have a big fat warning that users should not attempt to meddle with the git tree that Buildroot caches. As 768f9f80f62c however demonstrates, is that it took quite a long time between the introduction of the git caching, and the time someone eventually discovered they could meddle in there. This shows that the issue it not actually critical in most setups. Also, the tar manual [0] hints at a better solution to handle reproducibility, which even avoids touching the files on disk which is even nicer: ‘--mode='go+u,go-w'’ Omit irrelevant information about file permissions. If we were to actually handle the mode bit for reproducibility, we'd need to: - introduce archive versioning for all download backends and prost-processing - use the tar officially suggested method So, revert that change, as it was incomplete, was not really fixing much issues, and causes actual issues. This reverts commit 768f9f80f62c1da6e298c680f0f4bfa887f38c78. [0] https://www.gnu.org/software/tar/manual/tar.html#Reproducibility Thanks to Vincent and Arnout for pointing at the tar manual. Reported-by: Antoine Coutant Reported-by: Thomas Petazzoni Signed-off-by: Yann E. MORIN Cc: Vincent Fazio Cc: Arnout Vandecappelle (Essensium/Mind) Tested-by: Romain Naour Signed-off-by: Antoine Coutant --- support/download/helpers | 3 --- 1 file changed, 3 deletions(-) diff --git a/support/download/helpers b/support/download/helpers index 265685eff5..90a7d6c1ec 100755 --- a/support/download/helpers +++ b/support/download/helpers @@ -53,9 +53,6 @@ mk_tar_gz() { tmp="$(mktemp --tmpdir="$(pwd)")" pushd "${in_dir}" >/dev/null - # Enforce group/others mode bits - chmod -R go-wx+X . - # Establish list find . -not -type d -and -not \( -false "${find_opts[@]}" \) >"${tmp}.list" # Sort list for reproducibility