Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PATCH] Fix missing build dependency for pods #16210

Closed
p5pRT opened this issue Oct 27, 2017 · 10 comments
Closed

[PATCH] Fix missing build dependency for pods #16210

p5pRT opened this issue Oct 27, 2017 · 10 comments

Comments

@p5pRT
Copy link

p5pRT commented Oct 27, 2017

Migrated from rt.perl.org#132360 (status was 'resolved')

Searchable as RT132360$

@p5pRT
Copy link
Author

p5pRT commented Oct 27, 2017

From dan.dedrick@gmail.com

Created by dan.dedrick@gmail.com

When building perl with highly parallel options (e.g. -j 32) on a machine
with many cores (e.g. 32) ocassionaly it fails with the following type
of error.
 
make[1]​: Entering directory '.../cpan/podlators'
Can't locate Getopt/Long.pm in @​INC (you may need to install the Getopt​::Long module) (@​INC contains​: .../cpan/AutoLoader/lib .../dist/Carp/lib .../dist/PathTools .../dist/PathTools/lib .../cpan/ExtUtils-Install/lib .../cpan/ExtUtils-MakeMaker/lib .../cpan/ExtUtils-Manifest/lib .../cpan/File-Path/lib .../ext/re .../dist/Term-ReadLine/lib .../dist/Exporter/lib .../ext/File-Find/lib .../cpan/Text-Tabs/lib .../dist/constant/lib .../cpan/version/lib .../lib ../../lib .) at .../cpan/ExtUtils-MakeMaker/lib/ExtUtils/Command/MM.pm line 109.
Makefile​:445​: recipe for target 'manifypods' failed
make[1]​: *** [manifypods] Error 2

The scripts pod2man, pod2text, podchecker, podselect, and pod2usage all use
Getopt-Long and since they are all part of nonxs modules this needs to be
added here to prevent these build races.

Perl Info

Flags:
    category=core
    severity=low
    Type=Patch
    PatchStatus=HasPatch

Site configuration information for perl 5.27.6:

Configured by ddedrick at Thu Oct 26 19:54:14 EDT 2017.

Summary of my perl5 (revision 5 version 27 subversion 6) configuration:
  Local Commit: 26cf06121b03ef60fd49af6f0f56e743b332f6b0
  Ancestor: ffaed89356842ad8f2641781afa31ead7349fb7a
  Platform:
    osname=linux
    osvers=4.12.13-300.fc26.x86_64
    archname=x86_64-linux
    uname='linux aeryn 4.12.13-300.fc26.x86_64 #1 smp thu sep 14 16:00:38 utc 2017 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Dprefix=/home/ddedrick/src/perl/ -Dusedevel'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='7.2.1 20170915 (Red Hat 7.2.1-2)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib /lib64 /usr/lib64 /usr/local/lib64
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.25.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.25'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    26cf06121b03ef60fd49af6f0f56e743b332f6b0


@INC for perl 5.27.6:
    lib
    /home/ddedrick/src/perl/lib/site_perl/5.27.6/x86_64-linux
    /home/ddedrick/src/perl/lib/site_perl/5.27.6
    /home/ddedrick/src/perl/lib/5.27.6/x86_64-linux
    /home/ddedrick/src/perl/lib/5.27.6


Environment for perl 5.27.6:
    HOME=/home/ddedrick
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/ddedrick/.local/bin:/home/ddedrick/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Oct 27, 2017

From dan.dedrick@gmail.com

0001-Fix-missing-build-dependency-for-pods.patch
From 26cf06121b03ef60fd49af6f0f56e743b332f6b0 Mon Sep 17 00:00:00 2001
From: Dan Dedrick <ddedrick@lexmark.com>
Date: Thu, 12 Oct 2017 21:49:39 -0400
Subject: [PATCH] Fix missing build dependency for pods

When building perl with highly parallel options (e.g. -j 32) on a machine
with many cores (e.g. 32) ocassionaly it fails with the following type
of error.

make[1]: Entering directory '.../cpan/podlators'
Can't locate Getopt/Long.pm in @INC (you may need to install the Getopt::Long module) (@INC contains: .../cpan/AutoLoader/lib .../dist/Carp/lib .../dist/PathTools .../dist/PathTools/lib .../cpan/ExtUtils-Install/lib .../cpan/ExtUtils-MakeMaker/lib .../cpan/ExtUtils-Manifest/lib .../cpan/File-Path/lib .../ext/re .../dist/Term-ReadLine/lib .../dist/Exporter/lib .../ext/File-Find/lib .../cpan/Text-Tabs/lib .../dist/constant/lib .../cpan/version/lib .../lib ../../lib .) at .../cpan/ExtUtils-MakeMaker/lib/ExtUtils/Command/MM.pm line 109.
Makefile:445: recipe for target 'manifypods' failed
make[1]: *** [manifypods] Error 2

The scripts pod2man, pod2text, podchecker, podselect, and pod2usage all use
Getopt-Long and since they are all part of nonxs modules this needs to be
added here to prevent these build races.
---
 write_buildcustomize.pl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/write_buildcustomize.pl b/write_buildcustomize.pl
index 8666a6bce2..7a55405029 100644
--- a/write_buildcustomize.pl
+++ b/write_buildcustomize.pl
@@ -39,6 +39,7 @@ my @toolchain = qw(cpan/AutoLoader/lib
                    cpan/Text-Tabs/lib
 		   dist/constant/lib
 		   cpan/version/lib
+		   cpan/Getopt-Long/lib
 		   );
 
 # Text-ParseWords used only in ExtUtils::Liblist::Kid::_win32_ext()
@@ -47,7 +48,6 @@ my @toolchain = qw(cpan/AutoLoader/lib
 push @toolchain, qw(
 	cpan/Text-ParseWords/lib
 	dist/ExtUtils-ParseXS/lib
-	cpan/Getopt-Long/lib
 	cpan/parent/lib
 	cpan/ExtUtils-Constant/lib
 ) if $^O eq 'MSWin32';
-- 
2.13.6

@p5pRT
Copy link
Author

p5pRT commented Oct 28, 2017

From @jkeenan

On Fri, 27 Oct 2017 00​:47​:58 GMT, dan.dedrick@​gmail.com wrote​:

This is a bug report for perl from dan.dedrick@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.27.6.

-----------------------------------------------------------------
[Please describe your issue here]

When building perl with highly parallel options (e.g. -j 32) on a
machine
with many cores (e.g. 32) ocassionaly it fails with the following type
of error.

make[1]​: Entering directory '.../cpan/podlators'
Can't locate Getopt/Long.pm in @​INC (you may need to install the
Getopt​::Long module) (@​INC contains​: .../cpan/AutoLoader/lib
.../dist/Carp/lib .../dist/PathTools .../dist/PathTools/lib
.../cpan/ExtUtils-Install/lib .../cpan/ExtUtils-MakeMaker/lib
.../cpan/ExtUtils-Manifest/lib .../cpan/File-Path/lib .../ext/re
.../dist/Term-ReadLine/lib .../dist/Exporter/lib .../ext/File-Find/lib
.../cpan/Text-Tabs/lib .../dist/constant/lib .../cpan/version/lib
.../lib ../../lib .) at .../cpan/ExtUtils-
MakeMaker/lib/ExtUtils/Command/MM.pm line 109.
Makefile​:445​: recipe for target 'manifypods' failed
make[1]​: *** [manifypods] Error 2

The scripts pod2man, pod2text, podchecker, podselect, and pod2usage
all use
Getopt-Long and since they are all part of nonxs modules this needs to
be
added here to prevent these build races.

Can you explain a bit more as to what you think is the sequence of events involved in the race condition?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Oct 28, 2017

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Oct 28, 2017

From dan.dedrick@gmail.com

On Fri, 27 Oct 2017 17​:28​:22 -0700, jkeenan wrote​:

On Fri, 27 Oct 2017 00​:47​:58 GMT, dan.dedrick@​gmail.com wrote​:

This is a bug report for perl from dan.dedrick@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.27.6.

-----------------------------------------------------------------
[Please describe your issue here]

When building perl with highly parallel options (e.g. -j 32) on a
machine
with many cores (e.g. 32) ocassionaly it fails with the following
type
of error.

make[1]​: Entering directory '.../cpan/podlators'
Can't locate Getopt/Long.pm in @​INC (you may need to install the
Getopt​::Long module) (@​INC contains​: .../cpan/AutoLoader/lib
.../dist/Carp/lib .../dist/PathTools .../dist/PathTools/lib
.../cpan/ExtUtils-Install/lib .../cpan/ExtUtils-MakeMaker/lib
.../cpan/ExtUtils-Manifest/lib .../cpan/File-Path/lib .../ext/re
.../dist/Term-ReadLine/lib .../dist/Exporter/lib .../ext/File-
Find/lib
.../cpan/Text-Tabs/lib .../dist/constant/lib .../cpan/version/lib
.../lib ../../lib .) at .../cpan/ExtUtils-
MakeMaker/lib/ExtUtils/Command/MM.pm line 109.
Makefile​:445​: recipe for target 'manifypods' failed
make[1]​: *** [manifypods] Error 2

The scripts pod2man, pod2text, podchecker, podselect, and pod2usage
all use
Getopt-Long and since they are all part of nonxs modules this needs
to
be
added here to prevent these build races.

Can you explain a bit more as to what you think is the sequence of
events involved in the race condition?

Thank you very much.

If cpan/podlators starts building before cpan/Getopt-Long has finished it will attempt to run the pod2man, which contains "require Getopt​::Long;" before Getopt-Long has been published to the lib dir. Therefore when it runs it searches its path and finds no GetOpt-Long and fails. This can naturally happen when running a very parallel make invocation (-j 32) but it is still fairly rare. It can be artificially induced by reordering the nonxs_ext variable such that cpan/podlators/pm_to_blib is before cpan/Getopt-Long/pm_to_blib (this needs to be done in a fresh workspace right after the Configure is run).

@p5pRT
Copy link
Author

p5pRT commented Jun 6, 2018

From patrickdepinguin@gmail.com

On Fri, 27 Oct 2017 18​:40​:10 -0700, dan.dedrick@​gmail.com wrote​:

On Fri, 27 Oct 2017 17​:28​:22 -0700, jkeenan wrote​:

On Fri, 27 Oct 2017 00​:47​:58 GMT, dan.dedrick@​gmail.com wrote​:

This is a bug report for perl from dan.dedrick@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.27.6.

-----------------------------------------------------------------
[Please describe your issue here]

When building perl with highly parallel options (e.g. -j 32) on a
machine
with many cores (e.g. 32) ocassionaly it fails with the following
type
of error.

make[1]​: Entering directory '.../cpan/podlators'
Can't locate Getopt/Long.pm in @​INC (you may need to install the
Getopt​::Long module) (@​INC contains​: .../cpan/AutoLoader/lib
.../dist/Carp/lib .../dist/PathTools .../dist/PathTools/lib
.../cpan/ExtUtils-Install/lib .../cpan/ExtUtils-MakeMaker/lib
.../cpan/ExtUtils-Manifest/lib .../cpan/File-Path/lib .../ext/re
.../dist/Term-ReadLine/lib .../dist/Exporter/lib .../ext/File-
Find/lib
.../cpan/Text-Tabs/lib .../dist/constant/lib .../cpan/version/lib
.../lib ../../lib .) at .../cpan/ExtUtils-
MakeMaker/lib/ExtUtils/Command/MM.pm line 109.
Makefile​:445​: recipe for target 'manifypods' failed
make[1]​: *** [manifypods] Error 2

The scripts pod2man, pod2text, podchecker, podselect, and pod2usage
all use
Getopt-Long and since they are all part of nonxs modules this needs
to
be
added here to prevent these build races.

Can you explain a bit more as to what you think is the sequence of
events involved in the race condition?

Thank you very much.

If cpan/podlators starts building before cpan/Getopt-Long has finished
it will attempt to run the pod2man, which contains "require
Getopt​::Long;" before Getopt-Long has been published to the lib dir.
Therefore when it runs it searches its path and finds no GetOpt-Long
and fails. This can naturally happen when running a very parallel make
invocation (-j 32) but it is still fairly rare. It can be artificially
induced by reordering the nonxs_ext variable such that
cpan/podlators/pm_to_blib is before cpan/Getopt-Long/pm_to_blib (this
needs to be done in a fresh workspace right after the Configure is
run).

I have also seen this problem for perl 5.26.1 on a parallel build with -j33 on a machine with 32 cores.
The issue can be reproduced easily by adding a delay to the handling of Getopt-Long after the Makefile has been generated by the configure step.

Inline Patch
--- output/build/host-perl-5.26.1/Makefile	2018-06-06 15:18:34.905011014 +0200
+++ /tmp/Makefile.host	2018-06-06 15:19:01.200009936 +0200
@@ -583,6 +583,11 @@
 	$(MINIPERL) make_ext.pl $@ $(MAKE_EXT_ARGS) MAKE="$(MAKE)" LIBPERL_A=$(LIBPERL) LINKTYPE=static $(STATIC_LDFLAGS)
 
 n_dummy $(nonxs_ext):	$(MINIPERL_EXE) lib/buildcustomize.pl preplibrary FORCE
+	@echo "------- makefile -$@-"
+	@if [ "$@" = "cpan/Getopt-Long/pm_to_blib" ]; then \
+		echo "---- getopt" ; \
+		sleep 60 ; \
+	fi
 	$(MINIPERL) make_ext.pl $@ $(MAKE_EXT_ARGS) MAKE="$(MAKE)" LIBPERL_A=$(LIBPERL)
 
 ext/Pod-Functions/pm_to_blib: cpan/Pod-Simple/pm_to_blib cpan/Pod-Escapes/pm_to_blib pod/perlfunc.pod


With this above reproduction hack, the problem can even be reproduced with '-j5'.

Using the patch provided by Dan Dedrick, the problem is solved.

Could this patch please be applied to official releases?

Thanks,
Thomas

@p5pRT
Copy link
Author

p5pRT commented Nov 14, 2018

@arc - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

p5pRT commented Nov 14, 2018

From @arc

Thomas De Schampheleire via RT <perlbug-followup@​perl.org> wrote​:

Using the patch provided by Dan Dedrick, the problem is solved.

Could this patch please be applied to official releases?

Thanks, applied (with an extra comment, and Dan Dedrick added to the
AUTHORS file) as bb6e76f

Welcome to the roster of Perl authors, Dan!

--
Aaron Crane

@p5pRT
Copy link
Author

p5pRT commented May 22, 2019

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Author

p5pRT commented May 22, 2019

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant