Building perl reproducibly: __DATE__ and __TIME__ usage in perl.c #14860
Comments
From @ntyniOn Sun, Jul 12, 2015 at 10:59:27PM +0300, Niko Tyni wrote:
This probably got accidentally overlooked; opening a bug report about it. Re-attaching the two alternative patches. |
From @ntyni0001-Allow-overriding-the-compile-time-in-perl-V-output.patchFrom f5b9f09daee93032d66c1e574d972a977ce0ac9a Mon Sep 17 00:00:00 2001
From: Niko Tyni <ntyni@debian.org>
Date: Sun, 12 Jul 2015 22:23:00 +0300
Subject: [PATCH] Allow overriding the compile time in "perl -V" output
The C preprocessor macros __DATE__ and __TIME__ embed the compile time
into the binary for the purposes of "perl -V" output. This makes the
build unreproducible: compiling the same source with the same toolchain
cannot be made to yield bitwise identical binaries and other generated
files.
The compile time can now be overridden with the PERL_BUILD_DATE macro.
---
perl.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/perl.c b/perl.c
index cbb66e0..f4ceadb 100644
--- a/perl.c
+++ b/perl.c
@@ -1795,15 +1795,20 @@ S_Internals_V(pTHX_ CV *cv)
PUSHs(Perl_newSVpvn_flags(aTHX_ non_bincompat_options,
sizeof(non_bincompat_options) - 1, SVs_TEMP));
+#ifndef PERL_BUILD_DATE
#ifdef __DATE__
# ifdef __TIME__
- PUSHs(Perl_newSVpvn_flags(aTHX_
- STR_WITH_LEN("Compiled at " __DATE__ " " __TIME__),
- SVs_TEMP));
+# define PERL_BUILD_DATE __DATE__ " " __TIME__
# else
- PUSHs(Perl_newSVpvn_flags(aTHX_ STR_WITH_LEN("Compiled on " __DATE__),
- SVs_TEMP));
+# define PERL_BUILD_DATE __DATE__
# endif
+#endif
+#endif
+
+#ifdef PERL_BUILD_DATE
+ PUSHs(Perl_newSVpvn_flags(aTHX_
+ STR_WITH_LEN("Compiled at " PERL_BUILD_DATE),
+ SVs_TEMP));
#else
PUSHs(&PL_sv_undef);
#endif
--
2.1.4
|
From @ntyni0001-Compile-Config-cf_time-into-the-perl-binary-instead-.patchFrom 821d98ec9cd0611c50f0fb078d12546834ab2685 Mon Sep 17 00:00:00 2001
From: Niko Tyni <ntyni@debian.org>
Date: Sun, 5 Jul 2015 18:41:10 +0300
Subject: [PATCH] Compile $Config{cf_time} into the perl binary instead of
__DATE__/__TIME__
Unlike the C preprocessor macros __DATE__ and __TIME__, the cf_time
value is overridable via config.over mechanism. This can be used
to make the build reproducible, so that compiling the same source
with the same toolchain always gives bitwise identical binaries
and other generated files.
---
config_h.SH | 2 ++
perl.c | 11 +----------
2 files changed, 3 insertions(+), 10 deletions(-)
diff --git a/config_h.SH b/config_h.SH
index 0d4a409..40aa3a3 100755
--- a/config_h.SH
+++ b/config_h.SH
@@ -5244,6 +5244,8 @@ sed <<!GROK!THIS! >$CONFIG_H -e 's!^#undef\(.*/\)\*!/\*#define\1 \*!' -e 's!^#un
#$usesocks USE_SOCKS /**/
#endif
+#define PERL_CF_TIME "$cf_time"
+
#endif
!GROK!THIS!
;;
diff --git a/perl.c b/perl.c
index cbb66e0..bf2b5ea 100644
--- a/perl.c
+++ b/perl.c
@@ -1795,18 +1795,9 @@ S_Internals_V(pTHX_ CV *cv)
PUSHs(Perl_newSVpvn_flags(aTHX_ non_bincompat_options,
sizeof(non_bincompat_options) - 1, SVs_TEMP));
-#ifdef __DATE__
-# ifdef __TIME__
PUSHs(Perl_newSVpvn_flags(aTHX_
- STR_WITH_LEN("Compiled at " __DATE__ " " __TIME__),
+ STR_WITH_LEN("Configured at " PERL_CF_TIME),
SVs_TEMP));
-# else
- PUSHs(Perl_newSVpvn_flags(aTHX_ STR_WITH_LEN("Compiled on " __DATE__),
- SVs_TEMP));
-# endif
-#else
- PUSHs(&PL_sv_undef);
-#endif
for (i = 1; i <= local_patch_count; i++) {
/* This will be an undef, if PL_localpatches[i] is NULL. */
--
2.1.4
|
From @tonycozOn Mon Aug 17 09:29:14 2015, ntyni@debian.org wrote:
My preference would be "Allow overriding the compile time in "perl -V" output" since it only changes the behaviour of the build whe explicitly requested. The macro definitions will need reindentation. Anyone else have an opinion? Tony |
|
The RT System itself - Status changed from 'new' to 'open' |
From @bulk88On Mon Aug 17 23:34:55 2015, tonyc wrote:
I'd also go with ""Allow overriding the compile time in "perl -V" output", for some reason I think that $cf_time is not reproducible unless they say config.sh is part of Perl's source code and they ship config.sh in Debian, instead of it being a build product as P5P says it is. --- a/perl.c +#ifndef PERL_BUILD_DATE<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<FIRST ONE this needs an #elif, not a #ifdef then a #ifndef on the same define -- |
From @ntyniOn Tue, Aug 18, 2015 at 07:59:24AM -0700, bulk88 via RT wrote:
Thanks for looking at this. The point is that cf_time can be overridden via config.over to make it I'm happy with PERL_BUILD_DATE (i.e. the first patch) if somebody I guess we could patch cflags.SH, but at that point we could just as |
From @ntyniOn Mon, Aug 17, 2015 at 11:34:57PM -0700, Tony Cook via RT wrote:
[...]
We only now noticed that the cf_time patch causes a test failure in The easiest fix is to amend the patch not to change the wording. Updated |
From @ntyni0001-Compile-Config-cf_time-into-the-perl-binary-instead-.patchFrom aad30acda02cb94fb1e8d718c7b86416fa4d7d4d Mon Sep 17 00:00:00 2001
From: Niko Tyni <ntyni@debian.org>
Date: Sun, 5 Jul 2015 18:41:10 +0300
Subject: [PATCH] Compile $Config{cf_time} into the perl binary instead of
__DATE__/__TIME__
Unlike the C preprocessor macros __DATE__ and __TIME__, the cf_time
value is overridable via config.over mechanism. This can be used
to make the build reproducible, so that compiling the same source
with the same toolchain always gives bitwise identical binaries
and other generated files.
Bug: https://rt.perl.org/Ticket/Display.html?id=125830
Bug-Debian: https://bugs.debian.org/774422
Patch-Name: debian/do-not-record-build-date.diff
---
config_h.SH | 2 ++
perl.c | 11 +----------
2 files changed, 3 insertions(+), 10 deletions(-)
diff --git a/config_h.SH b/config_h.SH
index fb2224e..8d67775 100755
--- a/config_h.SH
+++ b/config_h.SH
@@ -5199,6 +5199,8 @@ sed <<!GROK!THIS! >$CONFIG_H -e 's!^#undef\(.*/\)\*!/\*#define\1 \*!' -e 's!^#un
#$usesocks USE_SOCKS /**/
#endif
+#define PERL_CF_TIME "$cf_time"
+
#endif
!GROK!THIS!
;;
diff --git a/perl.c b/perl.c
index e64f1f4..0909f59 100644
--- a/perl.c
+++ b/perl.c
@@ -1795,18 +1795,9 @@ S_Internals_V(pTHX_ CV *cv)
PUSHs(Perl_newSVpvn_flags(aTHX_ non_bincompat_options,
sizeof(non_bincompat_options) - 1, SVs_TEMP));
-#ifdef __DATE__
-# ifdef __TIME__
PUSHs(Perl_newSVpvn_flags(aTHX_
- STR_WITH_LEN("Compiled at " __DATE__ " " __TIME__),
+ STR_WITH_LEN("Compiled at " PERL_CF_TIME),
SVs_TEMP));
-# else
- PUSHs(Perl_newSVpvn_flags(aTHX_ STR_WITH_LEN("Compiled on " __DATE__),
- SVs_TEMP));
-# endif
-#else
- PUSHs(&PL_sv_undef);
-#endif
for (i = 1; i <= local_patch_count; i++) {
/* This will be an undef, if PL_localpatches[i] is NULL. */
--
2.1.4
|
From @TuxOn Tue, 18 Aug 2015 19:30:45 +0300, Niko Tyni <ntyni@debian.org> wrote:
That is on me. I have *no* problem with matching both,
-- |
From @jmdhOn Tue, Aug 18, 2015 at 07:30:45PM +0300, Niko Tyni wrote:
This is true, but it's then quite likely to be entirely false statement (Note that this discussion is largely independent of which basic patch Dominic. |
From @jmdh0003-Fix-cpan-Config-Perl-V-V.pm-to-match-perl-V.patchFrom 08894fe5363ce808df6c2d2f881f76430e54f450 Mon Sep 17 00:00:00 2001
From: Dominic Hargreaves <dom@earth.li>
Date: Tue, 18 Aug 2015 15:35:07 +0200
Subject: [PATCH 3/3] Fix cpan/Config-Perl-V/V.pm to match perl -V
perl -V has been patched to return 'Configured at' rather than
'Compiled at' as part of the reproducible builds effort. Reflect this
change in cpan/Config-Perl-V/V.pm too.
Patch-Name: debian/fix_config_perl_v_for_configured_at.diff
---
cpan/Config-Perl-V/V.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/cpan/Config-Perl-V/V.pm b/cpan/Config-Perl-V/V.pm
index def79bb..c860f97 100644
--- a/cpan/Config-Perl-V/V.pm
+++ b/cpan/Config-Perl-V/V.pm
@@ -274,7 +274,7 @@ sub plv2hash
my $build = { %empty_build };
- $pv =~ m{^\s+Compiled at\s+(.*)}m
+ $pv =~ m{^\s+(?:Compiled|Configured) at\s+(.*)}m
and $build->{stamp} = $1;
$pv =~ m{^\s+Locally applied patches:(?:\s+|\n)(.*?)(?:[\s\n]+Buil[td] under)}ms
and $build->{patches} = [ split m/\n+\s*/, $1 ];
@@ -338,6 +338,7 @@ sub myconfig
my $stamp = eval { Config::compile_date () };
if (defined $stamp) {
$stamp =~ s/^Compiled at //;
+ $stamp =~ s/^Configured at //;
$build->{osname} = $^O;
$build->{stamp} = $stamp;
$build->{patches} = [ Config::local_patches () ];
--
2.1.4
|
From @tonycozOn Tue Aug 18 09:14:15 2015, ntyni@debian.org wrote:
Passing the definition to all the the XS builds didn't particularly bother me, but I expected to have less trouble passing the -DPERL_BUILD_DATE=... to Configure: ./Configure ... -Accflags=-DPERL_BUILD_DATE='"some date"' (and many variations) lost the quotes each time, presumably due to all the shell manipulation of the variables (which might be fixable, but I'm not sure I want to jump down that hole.) Editing config.sh and rebuilding config.sh works though. Tony |
From @ntyniOn Tue, Aug 18, 2015 at 06:56:59PM -0700, Tony Cook via RT wrote:
I tried many variations of this as well and failed.
There's probably a typo there... I got it to work by appending '#define PERL_BUILD_DATE "..."' to config.h |
From @jmdhHere is the final version of this patch included in Debian, with the reindentation requested by Tony. I believe this should be suitable for inclusion in blead (and indeed it applies cleanly). Thanks, |
From @jmdhdo-not-record-build-date.diffFrom 5915800fd10cbb59be3f1b3639ebb1a7ce5ea88c Mon Sep 17 00:00:00 2001
From: Niko Tyni <ntyni@debian.org>
Date: Sun, 12 Jul 2015 22:23:00 +0300
Subject: Allow overriding the compile time in "perl -V" output
The C preprocessor macros __DATE__ and __TIME__ embed the compile time
into the binary for the purposes of "perl -V" output. This makes the
build unreproducible: compiling the same source with the same toolchain
cannot be made to yield bitwise identical binaries and other generated
files.
The compile time can now be overridden with the PERL_BUILD_DATE macro.
Bug: https://rt.perl.org/Ticket/Display.html?id=125830
Bug-Debian: https://bugs.debian.org/774422
Patch-Name: debian/do-not-record-build-date.diff
---
perl.c | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
diff --git a/perl.c b/perl.c
index e64f1f4..4d835d2 100644
--- a/perl.c
+++ b/perl.c
@@ -1795,15 +1795,20 @@ S_Internals_V(pTHX_ CV *cv)
PUSHs(Perl_newSVpvn_flags(aTHX_ non_bincompat_options,
sizeof(non_bincompat_options) - 1, SVs_TEMP));
-#ifdef __DATE__
-# ifdef __TIME__
+#ifndef PERL_BUILD_DATE
+# ifdef __DATE__
+# ifdef __TIME__
+# define PERL_BUILD_DATE __DATE__ " " __TIME__
+# else
+# define PERL_BUILD_DATE __DATE__
+# endif
+# endif
+#endif
+
+#ifdef PERL_BUILD_DATE
PUSHs(Perl_newSVpvn_flags(aTHX_
- STR_WITH_LEN("Compiled at " __DATE__ " " __TIME__),
- SVs_TEMP));
-# else
- PUSHs(Perl_newSVpvn_flags(aTHX_ STR_WITH_LEN("Compiled on " __DATE__),
+ STR_WITH_LEN("Compiled at " PERL_BUILD_DATE),
SVs_TEMP));
-# endif
#else
PUSHs(&PL_sv_undef);
#endif
|
From @jimcSo, it looks now that the narrow goals of the ticket are addressed. Id like to extend the discussion. perl5.23.2-80-gcd03fd3-k<f00ball> ie including the -g<sha> shown in git describe this would simplify benchmarking, and potentially yield benchmark results Porting/bench /opt/perl5/perl5*-g*-k* Its clear to me there are many steps, to many parts of the build system, that said, I think attached is the 1st step: cf_hash=`grep -v '^#=' config.sh | md5sum` |
From @jimc0001-Configure-improve-config.sh-by-setting-up-for-no-SHA.patchFrom 1d671a29526bf7587a592aadb57b338a840e1a5b Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Tue, 1 Sep 2015 20:52:18 -0600
Subject: [PATCH] Configure: improve config.sh by setting up for no SHA1-noise.
the header info in config.sh currently has a number of fields whose
value is unconstrained ($cf_time, $cf_by, $myuname). For example:
# Package name : perl5
# Source directory : .
# Configuration time: Fri Aug 28 21:09:08 MDT 2015
# Configured by : jimc
# Target system : linux groucho.jimc.earth 4.2.0-rc7-x2a-00125-gd0b89bd #278 smp sat aug 22 13:35:17 mdt 2015 x86_64 x86_64 x86_64 gnulinux
The presence of these fields destroys any `md5sum config.sh` utility,
which otherwize would tell much about the config, SW and HW platform
perl was built on.
This patch tweaks Configure to markup the variant part of the header
so those 3 lines are easy to exclude from a grep, wo messing with the
uses of the varnames themselves.
grep -v '^#=' config.sh | grep -v | cksum
So this is a small step towards adding -K<md5-mumble> into the perl
executable's installed version, such that we could do:
Porting/bench /opt/perl/perl5.23.2-80-gcd03fd3-k*
and get benchmark results that have a chance of being comparable across
many porters' boxes.
---
Configure | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/Configure b/Configure
index 464737d..6d8d149 100755
--- a/Configure
+++ b/Configure
@@ -23937,11 +23937,11 @@ $startsh
# instead choose to run each of the .SH files by yourself, or "Configure -S".
#
-# Package name : $package
-# Source directory : $src
-# Configuration time: $cf_time
-# Configured by : $cf_by
-# Target system : $myuname
+#= Package name : $package
+#= Source directory : $src
+#= Configuration time: $cf_time
+#= Configured by : $cf_by
+#= Target system : $myuname
EOT
: Add in command line options if available
--
2.1.0
|
From @tonycozOn Tue Sep 01 20:59:32 2015, yoduh wrote:
This should be a new ticket. Tony |
From @tonycozOn Sat Aug 22 00:45:23 2015, ntyni@debian.org wrote:
I've held off on applying your patch because I really want to be able to control this from the Configure command-line, but I don't see a way to do so. Tony |
From @tonycozOn Sun Oct 18 16:30:20 2015, tonyc wrote:
This was fixed in 7bd8199, so I've applied this patch as 6baa8db, thanks for the patch. Tony |
|
@tonycoz - Status changed from 'open' to 'resolved' |
Migrated from rt.perl.org#125830 (status was 'resolved')
Searchable as RT125830$
The text was updated successfully, but these errors were encountered: