[PATCH] for perl static build for gcc+win32 #16469
Comments
From Vadim.Konovalov@dell.comCreated by vadim.konovalov@dell.comI happen to do: gmake -f GNUmakefile ALL_STATIC=define I get this error: .\dist\Unicode-Normalize\mkheader expects for the Next, "ALL_STATIC" means "BUILD_STATIC", so I've edited it to be so. Therefore I think the attached edit looks proper. TIA, Perl Info |
From Vadim.Konovalov@dell.comInline Patchdiff -rbu perl-5.27.9-orig/win32/GNUmakefile perl-5.27.9-32/win32/GNUmakefile
--- perl-5.27.9-orig/win32/GNUmakefile 2018-02-17 14:10:52.000000000 +0300
+++ perl-5.27.9-32/win32/GNUmakefile 2018-03-10 23:49:56.252280400 +0300
@@ -232,10 +232,11 @@
#
# in addition to BUILD_STATIC the option ALL_STATIC makes *every*
-# extension get statically built
+# extension get statically built.
# This will result in a very large perl executable, but the main purpose
# is to have proper linking set so as to be able to create miscellaneous
-# executables with different built-in extensions
+# executables with different built-in extensions.
+# Using ALL_STATIC automatically turns on BUILD_STATIC.
#
#ALL_STATIC := define
@@ -875,14 +876,11 @@
GLOBEXE = ..\perlglob.exe
CONFIGPM = ..\lib\Config.pm
GENUUDMAP = ..\generate_uudmap.exe
-ifeq ($(BUILD_STATIC),define)
-PERLSTATIC = static
-else
ifeq ($(ALL_STATIC),define)
-PERLSTATIC = static
-else
-PERLSTATIC =
+BUILD_STATIC = define
endif
+ifeq ($(BUILD_STATIC),define)
+PERLSTATIC = static
endif
# Unicode data files generated by mktables
@@ -1497,7 +1495,7 @@
ifeq ($(CCTYPE),GCC)
$(LIB32) $(LIB_FLAGS) $@ $(PERLDLL_OBJ)
if exist $(STATICDIR) rmdir /s /q $(STATICDIR)
- for %%i in ($(shell type Extensions_static)) do \
+ for %%i in ($(shell ..\perl -MConfig -we 'print map {m|([^/]+)$$|; "../lib/auto/$$_/$$1$$Config{_a} "} split " ",$$Config{static_ext}')) do \
@mkdir $(STATICDIR) && cd $(STATICDIR) && \
$(ARCHPREFIX)ar x ..\%%i && \
$(ARCHPREFIX)ar q ..\$@ *$(o) && \
@@ -1559,7 +1557,9 @@
$(PERLEXESTATIC): $(PERLSTATICLIB) $(CONFIGPM) $(PERLEXEST_OBJ) $(PERLEXE_RES)
ifeq ($(CCTYPE),GCC)
$(LINK32) -mconsole -o $@ $(BLINK_FLAGS) \
- $(PERLEXEST_OBJ) $(PERLEXE_RES) $(PERLSTATICLIB) $(LIBFILES)
+ $(PERLEXEST_OBJ) $(PERLEXE_RES) $(PERLSTATICLIB) $(shell type Extensions_static) \
+ $(LIBFILES)
+
else
$(LINK32) -out:$@ $(BLINK_FLAGS) \
$(PERLEXEST_OBJ) $(PERLEXE_RES) $(PERLSTATICLIB) $(LIBFILES) $(SETARGV_OBJ)
@@ -1610,7 +1610,7 @@
# be running in parallel like UNIDATAFILES, this target a placeholder for the
# future
ifeq ($(BUILD_STATIC),define)
-rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE) $(PERLEXESTATIC)
+rebasePE : Extensions_normalize Extensions $(PERLDLL) $(PERLEXE) $(PERLEXESTATIC)
else
rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE)
endif
diff -rbu perl-5.27.9-orig/win32/list_static_libs.pl perl-5.27.9-32/win32/list_static_libs.pl
--- perl-5.27.9-orig/win32/list_static_libs.pl 2018-02-17 14:10:52.000000000 +0300
+++ perl-5.27.9-32/win32/list_static_libs.pl 2018-03-10 23:42:10.438127100 +0300
@@ -7,11 +7,11 @@
my @statics = split /\s+/, $Config{static_ext};
-my %extralibs;
+my (@extralibs, %extralibs); # collect extralibs, preserving their order
for (@statics) {
my $file = "..\\lib\\auto\\$_\\extralibs.ld";
open my $fh, '<', $file or die "can't open $file for reading: $!";
- $extralibs{$_}++ for grep {/\S/} split /\s+/, join '', <$fh>;
+ push @extralibs, grep {++$extralibs{$_}==1 ? $_ : ()} grep {/\S/} split /\s+/, join '', <$fh>;
}
print map {s|/|\\|g;m|([^\\]+)$|;"..\\lib\\auto\\$_\\$1$Config{_a} "} @statics;
-print map {"$_ "} sort keys %extralibs;
+print map {"$_ "} @extralibs; |
From Vadim.Konovalov@dell.comHere it is On 03/14/2018 02:32 AM, Konovalov, Vadim wrote:
bug ticket. |
From @dur-randirI can confirm that static build currently fails when done using mingw. But there's another makefile which is also used when building with mingw, win32/makefile.mk, though it's used with dmake, not gmake. And for me, to make dmake-based all-static build work, the only required change was: Inline Patchdiff --git a/win32/makefile.mk b/win32/makefile.mk
index 80bd6e8544..d82ec57819 100644
--- a/win32/makefile.mk
+++ b/win32/makefile.mk
@@ -1542,7 +1542,7 @@ Extensions_realclean :
# be running in parallel like UNIDATAFILES, this target a placeholder for the
# future
.IF "$(BUILD_STATIC)"=="define"
-rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE) $(PERLEXESTATIC)
+rebasePE : Extensions_normalize Extensions $(PERLDLL) $(PERLEXE) $(PERLEXESTATIC)
.ELSE
rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE)
.ENDIF
I suppose that the same applies to gmake-based builds. |
|
The RT System itself - Status changed from 'new' to 'open' |
From Vadim.Konovalov@dell.com
thanks This is correct - to fix the build - all is needed to do - place Extensions_normalize sooner in the chain; In addition, my patch makes 'ALL_STATIC' to have also 'BUILD_STATIC', and yet fixes an error in helper script all in all, seems reasonable update even at 5.27.9, but 5.27.10 also good for it :) |
From @dur-randirOn Tue, 27 Mar 2018 22:53:19 -0700, Vadim.Konovalov@dell.com wrote:
What is the error there? How can I check that it was present before your patch and is absent after applying it? |
From Vadim.Konovalov@dell.com
The incorrect thing there is that objects are "sort"-ed, but there is no actual reason for them to be sorted; In simple cases - things work In more complicated cases - when order is significant - linker error happens there, sometimes; actually order should be preserved; PS; it was me who introduced that file. Why I placed the 'sort' there, or was it even me - IDK; :):) |
From @bulk88On Wed, 28 Mar 2018 05:46:10 -0700, Vadim.Konovalov@dell.com wrote:
I agree, left to right ordering of deps has no meaning for a makefile. Usually a maketool, on a NON-PARALLEL build will build from left to right. But during parallel building any dep is fair game to execute at any time if the tree says all sub-deps r completed. Testing each target individually is the only way to prove the dep tree works for it and parallelness also will work. Personally I suggest (and have done this before when editing those Win32 makefiles) putting longer running deps on the left so more cores have more work to do. So if there are 4 deps, one 10 minutes, and 3 2 minutes deps and 1 5 minute, lets call them A10, B2, C2, D2, E5. On a 2 core machine, you want the maketool to do Core 1 A10 = 10 mins not Core 1 E5 A10 = 15 mins If changing the left to order fixes something, it means there is a bug in the dep tree and the 2 deps that are written as siblings, 1 is parent and 1 is the child in reality and the MKF needs to be restructured to reflect that (or build products changed to not depend on each other, more parallelness). -- |
From Vadim.Konovalov@dell.com
At the mentioned by me place 'sort' was not on dependencies, rather it was part of " list_static_libs.pl" On the other hand, at this makefile goal: rebasePE : Extensions_normalize Extensions Extensions_normalize should be satisfied before - all other extensions; |
From @dur-randirI've split the original patch into a series of four which can be applied independently. I've tested them all using a dmake-based mingw toolchain. |
From @dur-randir0001-Fix-static-builds-with-MinGW.patchFrom 60915860b161b84edac990681f0adc7d8de46233 Mon Sep 17 00:00:00 2001
From: Sergey Aleynikov <sergey.aleynikov@gmail.com>
Date: Wed, 28 Mar 2018 22:35:05 +0300
Subject: [PATCH 1/4] Fix static builds with MinGW
Move Extensions_normalize target before Extensions
target to satisfy dependencies.
---
win32/GNUmakefile | 2 +-
win32/makefile.mk | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/win32/GNUmakefile b/win32/GNUmakefile
index 7e464fa3cb..5c28731c49 100644
--- a/win32/GNUmakefile
+++ b/win32/GNUmakefile
@@ -1610,7 +1610,7 @@ PostExt : ..\lib\Storable\Limit.pm
# be running in parallel like UNIDATAFILES, this target a placeholder for the
# future
ifeq ($(BUILD_STATIC),define)
-rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE) $(PERLEXESTATIC)
+rebasePE : Extensions_normalize Extensions $(PERLDLL) $(PERLEXE) $(PERLEXESTATIC)
else
rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE)
endif
diff --git a/win32/makefile.mk b/win32/makefile.mk
index 80bd6e8544..ba9c5bf5c0 100644
--- a/win32/makefile.mk
+++ b/win32/makefile.mk
@@ -1542,7 +1542,7 @@ Extensions_realclean :
# be running in parallel like UNIDATAFILES, this target a placeholder for the
# future
.IF "$(BUILD_STATIC)"=="define"
-rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE) $(PERLEXESTATIC)
+rebasePE : Extensions_normalize Extensions $(PERLDLL) $(PERLEXE) $(PERLEXESTATIC)
.ELSE
rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE)
.ENDIF
--
2.16.1.windows.4
|
From @dur-randir0002-Fix-spelling-in-windows-makefiles.patchFrom 94fdafacfb9d2ee4590b5390b9aac91bb936f844 Mon Sep 17 00:00:00 2001
From: Sergey Aleynikov <sergey.aleynikov@gmail.com>
Date: Wed, 28 Mar 2018 23:26:20 +0300
Subject: [PATCH 2/4] Fix spelling in windows makefiles
---
win32/GNUmakefile | 4 ++--
win32/Makefile | 4 ++--
win32/makefile.mk | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/win32/GNUmakefile b/win32/GNUmakefile
index 5c28731c49..c1d4ac63a5 100644
--- a/win32/GNUmakefile
+++ b/win32/GNUmakefile
@@ -232,10 +232,10 @@ DEFAULT_INC_EXCLUDES_DOT := define
#
# in addition to BUILD_STATIC the option ALL_STATIC makes *every*
-# extension get statically built
+# extension get statically built.
# This will result in a very large perl executable, but the main purpose
# is to have proper linking set so as to be able to create miscellaneous
-# executables with different built-in extensions
+# executables with different built-in extensions.
#
#ALL_STATIC := define
diff --git a/win32/Makefile b/win32/Makefile
index 83369d2054..eff48e1d32 100644
--- a/win32/Makefile
+++ b/win32/Makefile
@@ -199,10 +199,10 @@ CCTYPE = MSVC60
#
# in addition to BUILD_STATIC the option ALL_STATIC makes *every*
-# extension get statically built
+# extension get statically built.
# This will result in a very large perl executable, but the main purpose
# is to have proper linking set so as to be able to create miscellaneous
-# executables with different built-in extensions
+# executables with different built-in extensions.
#
#ALL_STATIC = define
diff --git a/win32/makefile.mk b/win32/makefile.mk
index ba9c5bf5c0..053a934746 100644
--- a/win32/makefile.mk
+++ b/win32/makefile.mk
@@ -227,10 +227,10 @@ DEFAULT_INC_EXCLUDES_DOT *= define
#
# in addition to BUILD_STATIC the option ALL_STATIC makes *every*
-# extension get statically built
+# extension get statically built.
# This will result in a very large perl executable, but the main purpose
# is to have proper linking set so as to be able to create miscellaneous
-# executables with different built-in extensions
+# executables with different built-in extensions.
#
#ALL_STATIC *= define
--
2.16.1.windows.4
|
From @dur-randir0003-Fix-ALL_STATIC-builds-with-MinGW.patchFrom 81551f7b954d9764af3b9ab060ff141a6ea3c387 Mon Sep 17 00:00:00 2001
From: Sergey Aleynikov <sergey.aleynikov@gmail.com>
Date: Wed, 28 Mar 2018 23:28:56 +0300
Subject: [PATCH 3/4] Fix ALL_STATIC builds with MinGW
ALL_STATIC required BUILD_STATIC set but that was not documented.
---
win32/GNUmakefile | 4 ++--
win32/makefile.mk | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/win32/GNUmakefile b/win32/GNUmakefile
index c1d4ac63a5..1704126816 100644
--- a/win32/GNUmakefile
+++ b/win32/GNUmakefile
@@ -235,7 +235,7 @@ DEFAULT_INC_EXCLUDES_DOT := define
# extension get statically built.
# This will result in a very large perl executable, but the main purpose
# is to have proper linking set so as to be able to create miscellaneous
-# executables with different built-in extensions.
+# executables with different built-in extensions. It implies BUILD_STATIC.
#
#ALL_STATIC := define
@@ -1609,7 +1609,7 @@ PostExt : ..\lib\Storable\Limit.pm
# all PE files need to be built by the time this target runs, PP files can still
# be running in parallel like UNIDATAFILES, this target a placeholder for the
# future
-ifeq ($(BUILD_STATIC),define)
+ifeq ($(PERLSTATIC),static)
rebasePE : Extensions_normalize Extensions $(PERLDLL) $(PERLEXE) $(PERLEXESTATIC)
else
rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE)
diff --git a/win32/makefile.mk b/win32/makefile.mk
index 053a934746..ed250c720e 100644
--- a/win32/makefile.mk
+++ b/win32/makefile.mk
@@ -230,7 +230,7 @@ DEFAULT_INC_EXCLUDES_DOT *= define
# extension get statically built.
# This will result in a very large perl executable, but the main purpose
# is to have proper linking set so as to be able to create miscellaneous
-# executables with different built-in extensions.
+# executables with different built-in extensions. It implies BUILD_STATIC.
#
#ALL_STATIC *= define
@@ -1541,7 +1541,7 @@ Extensions_realclean :
# all PE files need to be built by the time this target runs, PP files can still
# be running in parallel like UNIDATAFILES, this target a placeholder for the
# future
-.IF "$(BUILD_STATIC)"=="define"
+.IF "$(PERLSTATIC)"=="static"
rebasePE : Extensions_normalize Extensions $(PERLDLL) $(PERLEXE) $(PERLEXESTATIC)
.ELSE
rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE)
--
2.16.1.windows.4
|
From @dur-randir0004-Maintain-extralibs-order-for-linker.patchFrom 7cffb760e2744e69439c3e0f34f5721b2fcf8d2d Mon Sep 17 00:00:00 2001
From: Sergey Aleynikov <sergey.aleynikov@gmail.com>
Date: Wed, 28 Mar 2018 23:53:32 +0300
Subject: [PATCH 4/4] Maintain extralibs order for linker
As per discussion in RT # 132992
---
win32/list_static_libs.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/win32/list_static_libs.pl b/win32/list_static_libs.pl
index 4b63d90812..a86eea5273 100644
--- a/win32/list_static_libs.pl
+++ b/win32/list_static_libs.pl
@@ -7,11 +7,11 @@ use Config;
my @statics = split /\s+/, $Config{static_ext};
-my %extralibs;
+my (@extralibs, %extralibs); # collect extralibs, preserving their order
for (@statics) {
my $file = "..\\lib\\auto\\$_\\extralibs.ld";
open my $fh, '<', $file or die "can't open $file for reading: $!";
- $extralibs{$_}++ for grep {/\S/} split /\s+/, join '', <$fh>;
+ push @extralibs, grep {!$extralibs{$_}++} grep {/\S/} split /\s+/, join '', <$fh>;
}
print map {s|/|\\|g;m|([^\\]+)$|;"..\\lib\\auto\\$_\\$1$Config{_a} "} @statics;
-print map {"$_ "} sort keys %extralibs;
+print map {"$_ "} @extralibs;
--
2.16.1.windows.4
|
From Vadim.Konovalov@dell.com
Nice approach to chew the food in smaller pieces :) |
From @khwilliamsonOn 03/28/2018 03:00 PM, Sergey Aleynikov via RT wrote:
Thanks, applied as Can this ticket be closed? |
From Vadim.Konovalov@dell.com
|
From @bulk88On Thu, 29 Mar 2018 07:56:39 -0700, public@khwilliamson.com wrote:
I wouldn't have applied this. Left to right order can't be depended on. -- |
From Vadim.Konovalov@dell.com
This is a bug in GNUmakefile, (and also there is another bug, which I mentioned PATCH are good in a sense that they do not improve situation but it also does not do it worse. |
From @khwilliamsonOn 03/29/2018 09:51 AM, Konovalov, Vadim wrote:
Don't you mean that they DO improve the situation? Back when I had to deal with these kinds of things, on a 16-bit Unix |
From @bulk88There are multiple bugs here, or design problems. Normalize is a separate target because it is very long to build because of Im tempted to make a Extensions_normalize_dyn and Extensions_normalize_static target, and split Extensions_static into targets, parent (make_ext.pl) child (list_static_libs.pl) and child depends on the mega-build-everything make_ext.pl static target and the other Extensions_normalize_static target. That allows paralleling but all static build products feed in together. I've played with the MKF a bit but I am trying to avoid a "ifeq ($(ALL_STATIC),define)" near Extensions_static but I think thats the only way to control if Normalize is dyn or static. and which bigger target it feeds into. -- |
From Vadim.Konovalov@dell.com
Bulk88 has concern that reordering of dependancies in makefile goal is not a correct way of fixing things, and explains this point by thinking about parallel build From this POV - there was a bug, and there is a bug in case of parallel build; Still, his concern could be addressed later then, I would not mind another separate bug ticket for this. There is little point of not accepting improvement patches, which incrementally improve the satiation, by arguing that there is another bug nearby |
From Vadim.Konovalov@dell.com
IOW a person who introduced "Extensions_normalise" introduced a bug for parallelized build;
|
From @bulk88I think this ticket can now be closed since the patches in it, that were applied, that I found issues with, were fixed in https://perl5.git.perl.org/perl.git/commit/908f2cb56527d29c9176e478fa7eee0d02a7c77e -- |
|
@xsawyerx - Status changed from 'open' to 'pending release' |
From @khwilliamsonThank you for filing this report. You have helped make Perl better. With the release yesterday of Perl 5.28.0, this and 185 other issues have been Perl 5.28.0 may be downloaded via: If you find that the problem persists, feel free to reopen this ticket. |
|
@khwilliamson - Status changed from 'pending release' to 'resolved' |
Migrated from rt.perl.org#132992 (status was 'resolved')
Searchable as RT132992$
The text was updated successfully, but these errors were encountered: