Bug #132403 for perl5: Long @INC gets replaced with executable

# Sun, 05 Nov 2017 22:13:20 -0800 Father Chrysostomos <sprout@cpan.org> - Ticket created [Reply] [Comment]

Date:	Sun, 5 Nov 2017 22:06:17 -0800
Subject:	Long @INC gets replaced with executable
To:	"perlbug [...] perl.org" <perlbug [...] perl.org>
From:	Father Chrysostomos <sprout [...] cpan.org>

Download (untitled) / with headers
text/plain 3.7k

The attached script sets PERL5LIB to a very long value, and then runs a perl one-liner with thousands of -I options. When I run it like this: $ ./perl -Ilib /path/to/02fail.t I get ./perl ./perl ./perl ./perl ... etc., one for every entry that should have been in @INC. Instead, @INC is filled with the executable name. Strangely, I can’t reproduce it if I put the PERL5LIB value and the -I switches in text files and run it like this: $ PERL5LIB="`cat /tmp/lib.text`" ./perl -Ilib `cat /tmp/inc.text` -e 'print "$_\n" for @INC' It gives me the correct @INC that I has expecting to see. My bash-fu is not very good. Maybe my command line is wrong. Maybe this is a heisenbug that depends on how stuff is laid out in memory. Maybe I should stop speculating. $ ./perl -Ilib -V Summary of my perl5 (revision 5 version 27 subversion 6) configuration: Local Commit: 6361bd97c31b3021d24a400ed9f6f3b7877d84ef Ancestor: 301bcfdaa73a979264d6ee1845c40043ddd464eb Platform: osname=darwin osvers=12.5.0 archname=darwin-2level uname='darwin pint.local 12.5.0 darwin kernel version 12.5.0: sun sep 29 13:33:47 pdt 2013; root:xnu-2050.48.12~1release_x86_64 x86_64 ' config_args='-Dusedevel -DDEBUGGING -Aoptimize=-O0 -des' hint=recommended useposix=true d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='cc' ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.8 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include -DPERL_USE_SAFE_PUTENV' optimize='-O3 -O0 -g' cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.8 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include' ccversion='' gccversion='4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.27)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='cc' ldflags =' -mmacosx-version-min=10.8 -fstack-protector -L/usr/local/lib -L/opt/local/lib' libpth=/usr/local/lib /usr/bin/../lib/clang/4.2/lib /usr/lib /opt/local/lib libs=-lpthread -ldbm -ldl -lm -lutil -lc perllibs=-lpthread -ldl -lm -lutil -lc libc= so=dylib useshrplib=false libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=bundle d_dlsymun=undef ccdlflags=' ' cccdlflags=' ' lddlflags=' -mmacosx-version-min=10.8 -bundle -undefined dynamic_lookup -L/usr/local/lib -L/opt/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF Locally applied patches: 6361bd97c31b3021d24a400ed9f6f3b7877d84ef Built under darwin Compiled at Nov 1 2017 13:14:28 @INC: lib /usr/local/lib/perl5/site_perl/5.27.6/darwin-2level /usr/local/lib/perl5/site_perl/5.27.6 /usr/local/lib/perl5/5.27.6/darwin-2level /usr/local/lib/perl5/5.27.6 /usr/local/lib/perl5/site_perl

Download 02fail.t
text/plain 197.5k

Message body is not shown because sender requested not to inline it.

# Sun, 05 Nov 2017 22:13:20 -0800 The RT System itself - Severity low added

# Sun, 05 Nov 2017 22:13:20 -0800 The RT System itself - Type unknown added

# Mon, 06 Nov 2017 06:13:10 -0800 James E Keenan <jkeenan@cpan.org> - Correspondence added [Reply] [Comment]

RT-Send-CC:

perl5-porters [...] perl.org

Download (untitled) / with headers
text/plain 1.1k

On Mon, 06 Nov 2017 06:13:20 GMT, sprout wrote: Show quoted text

> The attached script sets PERL5LIB to a very long value, and then runs > a perl one-liner with thousands of -I options. When I run it like > this: > > $ ./perl -Ilib /path/to/02fail.t > > I get > > ./perl > ./perl > ./perl > ./perl > ... > etc., one for every entry that should have been in @INC. Instead, > @INC is filled with the executable name. >

I get that result on Ubuntu Linux 16.04 LTS, where the default shell is dash. However, when I try that on FreeBSD-10.3, where my default shell is Bourne shell, I get a file with 3446 lines consisting solely of newlines. So there's a shell-specific aspect to this problem. Show quoted text

> Strangely, I can’t reproduce it if I put the PERL5LIB value and the -I > switches in text files and run it like this: > > $ PERL5LIB="`cat /tmp/lib.text`" ./perl -Ilib `cat /tmp/inc.text` -e > 'print "$_\n" for @INC' > > It gives me the correct @INC that I has expecting to see. My bash-fu > is not very good. Maybe my command line is wrong. Maybe this is a > heisenbug that depends on how stuff is laid out in memory. Maybe I > should stop speculating. >

-- James E Keenan (jkeenan@cpan.org)

# Mon, 06 Nov 2017 06:13:10 -0800 The RT System itself - Status changed from 'new' to 'open'

# Mon, 06 Nov 2017 14:29:33 -0800 Zefram <zefram@fysh.org> - Correspondence added [Reply] [Comment]

Date:	Mon, 6 Nov 2017 22:29:18 +0000
Subject:	Re: [perl #132403] Long @INC gets replaced with executable
To:	perl5-porters [...] perl.org
From:	Zefram <zefram [...] fysh.org>

Download (untitled) / with headers
text/plain 1.2k

Father Chrysostomos wrote: Show quoted text

>Strangely, I can't reproduce it if I put the PERL5LIB value and the -I >switches in text files and run it like this:

Your failing test has a shell quoting problem. Making @INC long, through PERL5LIB and -I options, is a red herring. The critical system() call fails its own. It can be readily reduced to a single failing line that doesn't involve @INC at all: system qq`$^X -e "warn qq|\$_\\n| for 0..2"`; If you change "system" to "print", you'll find that the shell command that this will execute is /usr/bin/perl -e "warn qq|$_\n| for 0..2" which has a bunch of active shell metacharacters. In particular, it's got $_ within double quotes, so you get $_ expanded *by the shell*. What this expands to is somewhat dependent on the shell that system() invokes, but is quite likely to be whatever it was in the environment for the outer perl process, which depends on the shell from which *that* was run: $ perl -lwe 'print $ENV{_}' /usr/bin/perl So after shell expansion, the command to execute for the inner perl becomes /usr/bin/perl -e "warn qq|/usr/bin/perl\n| for 0..2" so perl executes the program warn qq|/usr/bin/perl\n| for 0..2 which won't tell you much about the list being iterated over. PEBKAC. -zefram

# Mon, 06 Nov 2017 14:35:05 -0800 Father Chrysostomos <sprout@cpan.org> - Correspondence added [Reply] [Comment]

RT-Send-CC:

perl5-porters [...] perl.org

Download (untitled) / with headers
text/plain 1.5k

On Mon, 06 Nov 2017 14:29:33 -0800, zefram@fysh.org wrote: Show quoted text

> Father Chrysostomos wrote:

> >Strangely, I can't reproduce it if I put the PERL5LIB value and the -I > >switches in text files and run it like this:

> > Your failing test has a shell quoting problem. Making @INC long, through > PERL5LIB and -I options, is a red herring. The critical system() call > fails its own. It can be readily reduced to a single failing line that > doesn't involve @INC at all: > > system qq`$^X -e "warn qq|\$_\\n| for 0..2"`; > > If you change "system" to "print", you'll find that the shell command > that this will execute is > > /usr/bin/perl -e "warn qq|$_\n| for 0..2" > > which has a bunch of active shell metacharacters. In particular, it's > got $_ within double quotes, so you get $_ expanded *by the shell*. > What this expands to is somewhat dependent on the shell that system() > invokes, but is quite likely to be whatever it was in the environment > for the outer perl process, which depends on the shell from which *that* > was run: > > $ perl -lwe 'print $ENV{_}' > /usr/bin/perl > > So after shell expansion, the command to execute for the inner perl > becomes > > /usr/bin/perl -e "warn qq|/usr/bin/perl\n| for 0..2" > > so perl executes the program > > warn qq|/usr/bin/perl\n| for 0..2 > > which won't tell you much about the list being iterated over. PEBKAC.

Thank you. That means I have to go back to debugging why Test::Strict’s t/02fail.t was failing for me under ‘make test’ but not when run directly. :-( -- Father Chrysostomos

# Mon, 06 Nov 2017 14:35:05 -0800 Father Chrysostomos <sprout@cpan.org> - Status changed from 'open' to 'rejected'

# Mon, 06 Nov 2017 14:39:02 -0800 Zefram <zefram@fysh.org> - Correspondence added [Reply] [Comment]

From:	Zefram <zefram [...] fysh.org>
To:	perl5-porters [...] perl.org
Subject:	Re: [perl #132403] Long @INC gets replaced with executable
Date:	Mon, 6 Nov 2017 22:38:50 +0000

Download (untitled) / with headers
text/plain 440b

The @INC content is long enough that it may be close to the kernel's limit for the size of command line arguments and environment. Maybe under "make test" there are a couple more environment variables which push it over the threshold, and exec fails with E2BIG. Really, whatever's generating such long @INC content needs to find a way to package it into less environment space. It's not portable to have such a big environment. -zefram