Skip Menu |
Report information
Id: 123854
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: bulk88 <bulk88 [at] hotmail.com>
Cc:
AdminCc:

Operating System: mswin32
PatchStatus: (no value)
Severity: low
Type: core
Perl Version: 5.21.4
Fixed In: (no value)



Subject: parallel building with Win32 dmake problems
To: perlbug [...] perl.org
Date: Mon, 16 Feb 2015 23:49:07 -0500
From: bulk88 <bulk88 [...] hotmail.com>
Download (untitled) / with headers
text/plain 11.8k
This is a bug report for perl from bulk88@hotmail.com, generated with the help of perlbug 1.40 running under perl 5.21.4. ----------------------------------------------------------------- [Please describe your issue here] This really shouldn't be a ticket, but the mailing list server has a 250KB message size limit, and I dont want to send 22 posts to the ML. The log files will be in a different post for this ticket since that post will bounce from the ML server when RT forwards it to the ML server. I decided to implement parallel building with dmake since nmake's clone with parallelism, QT's jom make tool, has serious bugs, and I dont think it is worth my time to get them fixed when dmake parallelism doesn't have straight up "build will absolutely fail" bugs. Jom's specific bug IIRC is doesn't support having a dep/target with ".." in it. A target with ".." in its name is silently skipped. I am using dmake that came with strawberry perl 5.18. It calls itself dmake 4.12 ---------------------------------------------------------------- C:\perl521>dmake -V dmake - Version 4.12 (Windows / MinGW) Copyright (c) 1990,...,1997 by WTI Corp. Default Configuration: MAXLINELENGTH := 32766 MAXPROCESSLIMIT := 64 .IMPORT .IGNORE: DMAKEROOT .MAKEFILES : makefile.mk makefile .SOURCE : .NULL DMAKEROOT *= $(ABSMAKECMD:d)startup MAKESTARTUP := $(DMAKEROOT)\startup.mk Please read the NEWS file for the latest release notes. C:\perl521> ---------------------------------------------------------------- But dmake parallel ("dmake -P2 all") has very bad design in finding "next shell line to run". A "dmake -P2 all" does around "!unsatisfied_deps_of_target_all" IO stat/file test calls to MS CRT/kernel, for each recipe line. With procmon profiling, in 1 minute, it made 839447 IO calls (839447 lines, 99% of which are IO calls). I've attached the IO call log in the 2nd post to this ticket. It contains about 1 minute (and remember, each IO calls is slower because of the procmon profiler) of IO calls of dmake.exe. Yes it is 150 MB long. With file IO profiling off, and just process events turned on in procmon, it takes 3 SECONDS to run each "echo something >> config.h" line on the profilers. Without the procmon profiler, it takes 30-35 seconds to generate config.h with 1 core maxed out at nearly 100% kernel time for that core. So without procmon, it takes 350 ms to do each echo shell call. Below a console log of what is happening. C:\perl521\srcnewb4opt\win32>dmake -P2 all dmake: makefile.mk: line 1682: Warning: -- Macro `TESTPREPGCC' redefined afte r use if not exist ".\mini" mkdir ".\mini" if exist config.h del /f config.h copy config_H.vc config.h rem. > .\mini\.exists lib /def:ntdllundec.def /machine:x86 copy config.vc config.w32 Microsoft (R) Libra 1 file(s) copied. ry Manager Version 7.10.6030 Copyright (C) Microsoft Corporation. All rights reserved. Creating library ntdllundec.lib and object ntdllundec.exp lib /def:ntdlldec.def /machine:X86 ntdllundec.lib 1 file(s) copied. Microsoft (R) Library Manager Version 7.10.6030 Copyright (C) Microsoft Corporation. All rights reserved. LIB : warning LNK4001: no object files specified; libraries used Creating library ntdlldec.lib and object ntdlldec.exp copy runperl.c perlmain.c rc -i.. -DINCLUDE_MANIFEST perlexe.rc 1 file(s) copied. cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DN O_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_TEXTMODE_S CRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T -Foperlgl ob.obj -Fdperlglob.pdb perlglob.c perlglob.c link -nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpath:"c:\per l\lib\CORE" -machine:x86 oldnames.lib kernel32.lib user32.lib gdi32. lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib -out:..\perl glob.exe -subsystem:console \ perlglob.obj setargv.obj dmake: Warning: -- Target [..\perly.c] was made but the time stamp has not been updated. if exist ..\perlglob.exe.manifest mt -nologo -manifest ..\perlglob.exe.manifest -outputresource:..\perlglob.exe;1 && if exist ..\perlglob.exe.manif est del ..\perlglob.exe.manifest LINK : warning LNK4235: /LTCG specified but no code generation required; remove /LTCG from the link command line to improve linker performance setargv.obj : warning LNK4099: PDB 'libc.pdb' was not found with 'C:\Program Fil es\Microsoft Visual Studio .NET 2003\VC7\LIB\setargv.obj' or at 'C:\perl521\srcn ewb4opt\libc.pdb'; linking object as if no debug info ********everything above took 3 seconds*********************** *******30 to 35 second pause with no console output here, this is generating miniperl's config.h with many echo >> config.h shell calls ******************* cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\av.obj -Fdmini\av.pdb ..\av.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\caretx.obj -Fdmini\caretx.pdb ..\caretx.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\deb.obj -Fdmini\deb.pdb ..\deb.c av.c caretx.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\doio.obj -Fdmini\doio.pdb ..\doio.c deb.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\doop.obj -Fdmini\doop.pdb ..\doop.c doio.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\dump.obj -Fdmini\dump.pdb ..\dump.c doop.c cl -c -I..\ -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOL E -DNO_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_TE XTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T - Fo..\generate_uudmap.obj -Fd..\generate_uudmap.pdb ..\generate_uudmap.c dump.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\gv.obj -Fdmini\gv.pdb ..\gv.c generate_uudmap.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\mro.obj -Fdmini\mro.pdb ..\mro.c gv.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\hv.obj -Fdmini\hv.pdb ..\hv.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\locale.obj -Fdmini\locale.pdb ..\locale.c mro.c hv.c ******killed****** By my eye, "dmake -P2 ../mini/av.obj" is faster than "dmake -P2 ../miniperl.exe" is faster than "dmake -P2 Extensions" which is faster than "dmake -P2 all", all because of that delay before the first .obj for miniperl is produced. The narrower the build target, or the less total targets, the shorter that pause before the first miniperl .obj is compiled. I am thinking that the only way to hack fix the performance problem with the echos is turn all the echo calls in .\config.h target http://perl5.git.perl.org/perl.git/blob/HEAD:/win32/makefile.mk#l1054 into 1 very long shell line filled with &&s. The other solution is for miniperl.exe and miniperl's config.h, and buildcustomize.pl to be built from a 2nd dmake lang makefile, to break the IO file existence test chain tree up. Anyone disagree to consolidating all the echos into 1 shell call or have any other opinions? [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl 5.21.4: Configured by Owner at Thu Sep 18 12:08:58 2014. Summary of my perl5 (revision 5 version 21 subversion 4) configuration: Derived from: 7d2b2edb94ab56333b9049a3e26d15ea18445512 Ancestor: 19be3be6968e2337bcdfe480693fff795ecd1304 Platform: osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -DWIN32 -D_CONSOLE -DNO_STRICT -DPERL_TEXTMODE_SCRIPTS -DPERL_HASH_FUNC_ONE_AT_A_TIME -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T', optimize='-O1 -MD -Zi -DNDEBUG', cppflags='-DWIN32' ccversion='12.00.8168', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -libpath:"c:\perl521\lib\CORE" -machine:x86' libpth=C:\PROGRA~1\MIAF9D~1\VC98\lib libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl521.lib gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -libpath:"c:\perl521\lib\CORE" -machine:x86' Locally applied patches: uncommitted-changes a0fe7a7e75de29e59f1da0d6822dc06e5be658fe a261faffee83d0145642ab5d1d046c9f813bc497 6506ab86ad1602a9ca720fcd30446dce1461d23d 7d2b2edb94ab56333b9049a3e26d15ea18445512 --- @INC for perl 5.21.4: lib C:/perl521/srcnew/lib . --- Environment for perl 5.21.4: HOME (unset) LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH= PERL_BADLANG (unset) PERL_JSON_BACKEND=Cpanel::JSON::XS PERL_YAML_BACKEND=YAML SHELL (unset)
Download bad parallel dmake procs only.CSV
application/vnd.ms-excel 38.4k

Message body not shown because it is not plain text.

RT-Send-CC: perl5-porters [...] perl.org
detailed IO log mentioned in OP -- bulk88 ~ bulk88 at hotmail.com
Subject: bad parallel dmake procs+disk IO.7z
Download bad parallel dmake procs+disk IO.7z
application/octet-stream 3.9m

Message body not shown because it is not plain text.

Date: Tue, 17 Feb 2015 08:36:00 +0000
From: Steve Hay <steve.m.hay [...] googlemail.com>
CC: "bugs-bitbucket [...] rt.perl.org" <bugs-bitbucket [...] rt.perl.org>
Subject: Re: [perl #123854] parallel building with Win32 dmake problems
To: "perl5-porters [...] perl.org" <perl5-porters [...] perl.org>
Download (untitled) / with headers
text/plain 1.2k
[...] Show quoted text
> I am thinking that the only way to hack fix the performance problem with > the echos is turn all the echo calls in .\config.h target > http://perl5.git.perl.org/perl.git/blob/HEAD:/win32/makefile.mk#l1054 > into 1 very long shell line filled with &&s. The other solution is for > miniperl.exe and miniperl's config.h, and buildcustomize.pl to be built > from a 2nd dmake lang makefile, to break the IO file existence test > chain tree up. Anyone disagree to consolidating all the echos into 1 > shell call or have any other opinions? >
I haven't looked through the details, but I have no particular problem with your suggestion as long as the lines in the makefile.mk are still separate (i.e. using the \ continuation character) - one big long line in the makefile.mk would be gross! Another option might be to put all the echo commands into a batch file in win32/ and have the makefile.mk issue a single shell call to run that batch file (passing in the required macro values as arguments)? That would have the additional benefit that the echo commands could be shared with win32/Makefile, which has a very similar block of echoes in it. Or would that still cause trouble because the batch file is still doing separate echoes?
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 478b
On Tue Feb 17 00:36:33 2015, shay wrote: Show quoted text
> I haven't looked through the details, but I have no particular problem > with your suggestion as long as the lines in the makefile.mk are still > separate (i.e. using the \ continuation character) - one big long line > in the makefile.mk would be gross!
This idea was done in the patch in https://rt.perl.org/Ticket/Display.html?id=123867 so this ticket can be closed if nobody else has objections. -- bulk88 ~ bulk88 at hotmail.com
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 207b
On Tue Apr 28 12:40:02 2015, bulk88 wrote: Show quoted text
> This idea was done in the patch in > https://rt.perl.org/Ticket/Display.html?id=123867 so this ticket can > be closed if nobody else has objections.
Done. Tony


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org