Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parallel building with Win32 dmake problems #14518

Closed
p5pRT opened this issue Feb 17, 2015 · 9 comments
Closed

parallel building with Win32 dmake problems #14518

p5pRT opened this issue Feb 17, 2015 · 9 comments

Comments

@p5pRT
Copy link

p5pRT commented Feb 17, 2015

Migrated from rt.perl.org#123854 (status was 'resolved')

Searchable as RT123854$

@p5pRT
Copy link
Author

p5pRT commented Feb 17, 2015

From @bulk88

Created by @bulk88

This really shouldn't be a ticket, but the mailing list server has a
250KB message size limit, and I dont want to send 22 posts to the ML.
The log files will be in a different post for this ticket since that
post will bounce from the ML server when RT forwards it to the ML server.

I decided to implement parallel building with dmake since nmake's clone
with parallelism, QT's jom make tool, has serious bugs, and I dont think
it is worth my time to get them fixed when dmake parallelism doesn't
have straight up "build will absolutely fail" bugs. Jom's specific bug
IIRC is doesn't support having a dep/target with ".." in it. A target
with ".." in its name is silently skipped.

I am using dmake that came with strawberry perl 5.18. It calls itself
dmake 4.12
----------------------------------------------------------------
C​:\perl521>dmake -V
dmake - Version 4.12 (Windows / MinGW)
Copyright (c) 1990,...,1997 by WTI Corp.

Default Configuration​:
  MAXLINELENGTH := 32766
  MAXPROCESSLIMIT := 64
  .IMPORT .IGNORE​: DMAKEROOT
  .MAKEFILES : makefile.mk makefile
  .SOURCE : .NULL
  DMAKEROOT *= $(ABSMAKECMD​:d)startup
  MAKESTARTUP := $(DMAKEROOT)\startup.mk

Please read the NEWS file for the latest release notes.

C​:\perl521>
----------------------------------------------------------------

But dmake parallel ("dmake -P2 all") has very bad design in finding
"next shell line to run". A "dmake -P2 all" does around
"!unsatisfied_deps_of_target_all" IO stat/file test calls to MS
CRT/kernel, for each recipe line. With procmon profiling, in 1 minute,
it made 839447 IO calls (839447 lines, 99% of which are IO calls). I've
attached the IO call log in the 2nd post to this ticket. It contains
about 1 minute (and remember, each IO calls is slower because of the
procmon profiler) of IO calls of dmake.exe. Yes it is 150 MB long. With
file IO profiling off, and just process events turned on in procmon, it
takes 3 SECONDS to run each "echo something >> config.h" line on the
profilers. Without the procmon profiler, it takes 30-35 seconds to
generate config.h with 1 core maxed out at nearly 100% kernel time for
that core. So without procmon, it takes 350 ms to do each echo shell
call. Below a console log of what is happening.

C​:\perl521\srcnewb4opt\win32>dmake -P2 all
dmake​: makefile.mk​: line 1682​: Warning​: -- Macro `TESTPREPGCC'
redefined afte
r use
if not exist ".\mini" mkdir ".\mini"
if exist config.h del /f config.h
copy config_H.vc config.h
rem. > .\mini\.exists
lib /def​:ntdllundec.def /machine​:x86
copy config.vc config.w32
Microsoft (R) Libra 1 file(s) copied.
ry Manager Version 7.10.6030
Copyright (C) Microsoft Corporation. All rights reserved.

  Creating library ntdllundec.lib and object ntdllundec.exp
lib /def​:ntdlldec.def /machine​:X86 ntdllundec.lib
  1 file(s) copied.
Microsoft (R) Library Manager Version 7.10.6030
Copyright (C) Microsoft Corporation. All rights reserved.

LIB : warning LNK4001​: no object files specified; libraries used
  Creating library ntdlldec.lib and object ntdlldec.exp
copy runperl.c perlmain.c
rc -i.. -DINCLUDE_MANIFEST perlexe.rc
  1 file(s) copied.
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DN
O_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_TEXTMODE_S
CRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T
-Foperlgl
ob.obj -Fdperlglob.pdb perlglob.c
perlglob.c
link -nologo -nodefaultlib -debug -opt​:ref,icf -ltcg
-libpath​:"c​:\per
l\lib\CORE" -machine​:x86 oldnames.lib kernel32.lib
user32.lib gdi32.
lib winspool.lib comdlg32.lib advapi32.lib shell32.lib
ole32.lib
oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
-out​:..\perl
glob.exe -subsystem​:console \
  perlglob.obj setargv.obj
dmake​: Warning​: -- Target [..\perly.c] was made but the time stamp has
not been
  updated.
if exist ..\perlglob.exe.manifest mt -nologo -manifest
..\perlglob.exe.manifest
-outputresource​:..\perlglob.exe;1 && if exist
..\perlglob.exe.manif
est del ..\perlglob.exe.manifest
LINK : warning LNK4235​: /LTCG specified but no code generation required;
remove
/LTCG from the link command line to improve linker performance
setargv.obj : warning LNK4099​: PDB 'libc.pdb' was not found with
'C​:\Program Fil
es\Microsoft Visual Studio .NET 2003\VC7\LIB\setargv.obj' or at
'C​:\perl521\srcn
ewb4opt\libc.pdb'; linking object as if no debug info
********everything above took 3 seconds***********************
*******30 to 35 second pause with no console output here, this is
generating miniperl's config.h with many echo >> config.h shell calls
*******************
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\av.obj -Fdmini\av.pdb ..\av.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\caretx.obj -Fdmini\caretx.pdb ..\caretx.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\deb.obj -Fdmini\deb.pdb ..\deb.c
av.c
caretx.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\doio.obj -Fdmini\doio.pdb ..\doio.c
deb.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\doop.obj -Fdmini\doop.pdb ..\doop.c
doio.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\dump.obj -Fdmini\dump.pdb ..\dump.c
doop.c
cl -c -I..\ -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOL
E -DNO_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_TE
XTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-D_USE_32BIT_TIME_T -
Fo..\generate_uudmap.obj -Fd..\generate_uudmap.pdb ..\generate_uudmap.c
dump.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\gv.obj -Fdmini\gv.pdb ..\gv.c
generate_uudmap.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\mro.obj -Fdmini\mro.pdb ..\mro.c
gv.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\hv.obj -Fdmini\hv.pdb ..\hv.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\locale.obj -Fdmini\locale.pdb ..\locale.c
mro.c
hv.c
******killed******

By my eye, "dmake -P2 ../mini/av.obj" is faster than "dmake -P2
../miniperl.exe" is faster than "dmake -P2 Extensions" which is faster
than "dmake -P2 all", all because of that delay before the first .obj
for miniperl is produced. The narrower the build target, or the less
total targets, the shorter that pause before the first miniperl .obj is
compiled.

I am thinking that the only way to hack fix the performance problem with
the echos is turn all the echo calls in .\config.h target
http​://perl5.git.perl.org/perl.git/blob/HEAD​:/win32/makefile.mk#l1054
into 1 very long shell line filled with &&s. The other solution is for
miniperl.exe and miniperl's config.h, and buildcustomize.pl to be built
from a 2nd dmake lang makefile, to break the IO file existence test
chain tree up. Anyone disagree to consolidating all the echos into 1
shell call or have any other opinions?

Perl Info

Flags:
                category=core
                severity=low

Site configuration information for perl 5.21.4:

Configured by Owner at Thu Sep 18 12:08:58 2014.

Summary of my perl5 (revision 5 version 21 subversion 4) configuration:
              Derived from: 7d2b2edb94ab56333b9049a3e26d15ea18445512
              Ancestor: 19be3be6968e2337bcdfe480693fff795ecd1304
              Platform:
                osname=MSWin32, osvers=5.1,
archname=MSWin32-x86-multi-thread
                uname=''
                config_args='undef'
                hint=recommended, useposix=true, d_sigaction=undef
                useithreads=define, usemultiplicity=define
                use64bitint=undef, use64bitall=undef, uselongdouble=undef
                usemymalloc=n, bincompat5005=undef
              Compiler:
                cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG
-DWIN32
-D_CONSOLE -DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS
-DPERL_HASH_FUNC_ONE_AT_A_TIME -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T',
                optimize='-O1 -MD -Zi -DNDEBUG',
                cppflags='-DWIN32'
                ccversion='12.00.8168', gccversion='', gccosandvers=''
                intsize=4, longsize=4, ptrsize=4, doublesize=8,
byteorder=1234
                d_longlong=undef, longlongsize=8, d_longdbl=define,
longdblsize=8,
longdblkind=0
                ivtype='long', ivsize=4, nvtype='double', nvsize=8,
Off_t='__int64',
lseeksize=8
                alignbytes=8, prototype=define
              Linker and Libraries:
                ld='link', ldflags ='-nologo -nodefaultlib -debug
-opt:ref,icf
-libpath:"c:\perl521\lib\CORE"  -machine:x86'
                libpth=C:\PROGRA~1\MIAF9D~1\VC98\lib
                libs=oldnames.lib kernel32.lib user32.lib gdi32.lib
winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
                perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib
winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
                libc=msvcrt.lib, so=dll, useshrplib=true,
libperl=perl521.lib
                gnulibc_version=''
              Dynamic Linking:
                dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
                cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug
-opt:ref,icf  -libpath:"c:\perl521\lib\CORE"  -machine:x86'

Locally applied patches:
                uncommitted-changes
                a0fe7a7e75de29e59f1da0d6822dc06e5be658fe
                a261faffee83d0145642ab5d1d046c9f813bc497
                6506ab86ad1602a9ca720fcd30446dce1461d23d
                7d2b2edb94ab56333b9049a3e26d15ea18445512


@INC for perl 5.21.4:
                lib
                C:/perl521/srcnew/lib
                .


Environment for perl 5.21.4:
                HOME (unset)
                LANG (unset)
                LANGUAGE (unset)
                LD_LIBRARY_PATH (unset)
                LOGDIR (unset)
                PATH=
                PERL_BADLANG (unset)
                PERL_JSON_BACKEND=Cpanel::JSON::XS
                PERL_YAML_BACKEND=YAML
                SHELL (unset)























@p5pRT
Copy link
Author

p5pRT commented Feb 17, 2015

From @bulk88

[bad parallel dmake procs only.CSV](https://rt-archive.perl.org/perl5/Ticket/Attachment/1331679/711111/bad parallel dmake procs only.CSV)

@p5pRT
Copy link
Author

p5pRT commented Feb 17, 2015

From @bulk88

detailed IO log mentioned in OP

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Feb 17, 2015

From @bulk88

[bad parallel dmake procs+disk IO.7z](https://rt-archive.perl.org/perl5/Ticket/Attachment/1331687/711117/bad parallel dmake procs+disk IO.7z)

@p5pRT
Copy link
Author

p5pRT commented Feb 17, 2015

From @steve-m-hay

[...]

I am thinking that the only way to hack fix the performance problem with
the echos is turn all the echo calls in .\config.h target
http​://perl5.git.perl.org/perl.git/blob/HEAD​:/win32/makefile.mk#l1054
into 1 very long shell line filled with &&s. The other solution is for
miniperl.exe and miniperl's config.h, and buildcustomize.pl to be built
from a 2nd dmake lang makefile, to break the IO file existence test
chain tree up. Anyone disagree to consolidating all the echos into 1
shell call or have any other opinions?

I haven't looked through the details, but I have no particular problem
with your suggestion as long as the lines in the makefile.mk are still
separate (i.e. using the \ continuation character) - one big long line
in the makefile.mk would be gross!

Another option might be to put all the echo commands into a batch file
in win32/ and have the makefile.mk issue a single shell call to run
that batch file (passing in the required macro values as arguments)?
That would have the additional benefit that the echo commands could be
shared with win32/Makefile, which has a very similar block of echoes
in it. Or would that still cause trouble because the batch file is
still doing separate echoes?

@p5pRT
Copy link
Author

p5pRT commented Feb 17, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Apr 28, 2015

From @bulk88

On Tue Feb 17 00​:36​:33 2015, shay wrote​:

I haven't looked through the details, but I have no particular problem
with your suggestion as long as the lines in the makefile.mk are still
separate (i.e. using the \ continuation character) - one big long line
in the makefile.mk would be gross!

This idea was done in the patch in https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123867 so this ticket can be closed if nobody else has objections.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented May 12, 2015

From @tonycoz

On Tue Apr 28 12​:40​:02 2015, bulk88 wrote​:

This idea was done in the patch in
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123867 so this ticket can
be closed if nobody else has objections.

Done.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 12, 2015

@tonycoz - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant