Bug #87322 for perl5: Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7

# Wed, 30 Mar 2011 06:35:51 -0700 Christian Walde <mithaldu@yahoo.de> - Ticket created [Reply] [Comment]

Subject:	Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Wed, 30 Mar 2011 15:35:29 +0200
To:	perlbug [...] perl.org
From:	"Christian Walde" <mithaldu [...] yahoo.de>

Download (untitled) / with headers
text/plain 5.1k

This is a bug report for perl from mithaldu@yahoo.de, generated with the help of perlbug 1.39 running under perl 5.10.1. ----------------------------------------------------------------- [Please describe your issue here] Short: On Windows 7 environment variable values can be arbitrarily large. However the perl executable ignores $ENV{PERL5LIB} if it goes above 32kbyte, despite being able to read it properly. This behavior can be reproduced with this minimal test script (example run outputs included): https://gist.github.com/894331 Long: On Wed, 30 Mar 2011 10:54:57 +0200, Christian Walde <mithaldu@yahoo.de> wrote: Show quoted text

> On Wed, 30 Mar 2011 02:22:32 +0200, perl@0ne.us <perl@0ne.us> wrote: >

> > I just got this FAIL report today from CPANTesters: > > > > http://www.cpantesters.org/cpan/report/0ee8eb99-6c5d-1014-a630-0d8a02da96e1 > > > > As you can see, the prereq is correctly recognized by the toolchain > > and loaded - it's even in the lengthy @INC list, BUT when executing the > > test script it seems like @INC disappears somehow? Since I don't know > > the details of your setup I have to assume something is broken.

> > https://gist.github.com/893981 > > Of note: The test ( https://gist.github.com/893981#L225 ) clearly has a whole battery of directories in $ENV{PERL5LIB} ( https://gist.github.com/893981#L247 ) which the perl executable can obviously see, including the Params::Classify dir as the very first; since that's a Data::Dumper output of %ENV. However @INC remains unchanged by that. ( https://gist.github.com/893981#L579 ) > > This seems to indicate to me that the perl executable itself is, for whatever reason, flat out ignoring the PERL5LIB and/or failing to inject it into @INC. Other possibilities include Module::Build messing things up or the Windows Perl executable exclusively having issues.

Thanks to a hint from Brian Raven i managed to reduce this to a minimal case and compared it across WinXP, Win7 and a Linux box. Turns out that: - on Win XP perl dies if an ENV entry is >28kbyte - on Linux perl dies if an ENV entry is >128kbyte - on Win7 perl handles arbitrarily large values in ENV entries, but flat-out ignores any PERL5LIB values over 32kbyte (all with 5.10, tested with both ActivePerl and Strawberry Perl) Since this acts the same across both AP and Strawberry, it seems to be a genuine perl core bug. [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl 5.10.1: Configured by 1 at Thu Nov 4 19:37:29 2010. Summary of my perl5 (revision 5 version 10 subversion 1) configuration: Platform: osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread uname='Win32 strawberryperl 5.10.1.4 #1 Thu Nov 4 19:32:28 2010 i386' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags =' -s -O2 -DWIN32 -DHAVE_DES_FCRYPT -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fno-strict-aliasing -DPERL_MSVCRT_READFIX', optimize='-s -O2', cppflags='-DWIN32' ccversion='', gccversion='3.4.5', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='g++', ldflags ='-s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"' libpth=C:\strawberry\c\lib libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 libc=, so=dll, useshrplib=true, libperl=libperl510.a gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-mdll -s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"' Locally applied patches: --- @INC for perl 5.10.1: C:/strawberry/perl/lib C:/strawberry/perl/site/lib C:\strawberry\perl\vendor\lib . --- Environment for perl 5.10.1: CYGWIN=nodosfilewarning HOME (unset) LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=c:\cygwin\bin;C:\Program Files (x86)\Git\cmd;C:\Program Files (x86)\PHP\;C:\Perl\site\bin;C:\Perl\bin;C:\Program Files (x86)\ActiveState Komodo IDE 6\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\TortoiseSVN\bin;C:\Program Files (x86)\TortoiseGit\bin;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin PERL_BADLANG (unset) SHELL (unset)

# Wed, 30 Mar 2011 06:35:52 -0700 The RT System itself - Operating System mswin32 added

# Wed, 30 Mar 2011 06:35:52 -0700 The RT System itself - Severity low added

# Wed, 30 Mar 2011 06:35:52 -0700 The RT System itself - Type core added

# Wed, 30 Mar 2011 06:35:52 -0700 The RT System itself - Perl Version 5.10.1 added

# Wed, 30 Mar 2011 12:41:24 -0700 Jan Dubois <jand@activestate.com> - Correspondence added [Reply] [Comment]

Download (untitled) / with headers
text/plain 469b

The problem is the creation of subprocesses when the environment block is too large: "CreateProcessA() fails if the total size of the environment block for the process exceeds 32,767 characters." I don't consider this a bug in Perl, it is just a limitation of the OS. Using the Unicode version of the API (CreateProcessW) would avoid this particular problem, but could still run into other issues, where target programs can't deal with these huge environment blocks.

# Wed, 30 Mar 2011 12:41:24 -0700 The RT System itself - Status changed from 'new' to 'open'

# Wed, 30 Mar 2011 12:50:34 -0700 Christian Walde <mithaldu@yahoo.de> - Correspondence added [Reply] [Comment]

Download (untitled) / with headers
text/plain 502b

Jan, sorry to be blunt, but you missed something while reading: This is a problem specific to Windows 7 in a pretty unique manner. Windows 7 does not have that limit and can create the new process just fine. I was able to create processes with an $ENV{PERL5LIB} of 300 kilobyte, with the PERL5LIB value accessible inside the perl process via %ENV, complete and intact. Whichever part is responsible for @INC however just throws its hands up when it sees the ENV size and does absolutely nothing.

# Sun, 10 Jul 2011 17:37:52 -0700 Christian Walde <mithaldu@yahoo.de> - Correspondence added [Reply] [Comment]

RT-Send-CC:

perl5-porters [...] perl.org

Download (untitled) / with headers
text/plain 662b

On Wed Mar 30 06:35:51 2011, Mithaldu wrote: Show quoted text

> On Windows 7 environment variable values can be arbitrarily large. > However the perl executable ignores $ENV{PERL5LIB} if it goes above > 32kbyte, despite being able to read it properly. This behavior can > be reproduced with this minimal test script (example run outputs > included): > > https://gist.github.com/894331

Nobody seems to have taken an interest, and i sadly lack the knowledge and skill to poke at it myself. This is however still being an issue and is causing false negatives when smoking on Win7. So i have to ask: Could someone please take a look at whatever piece of code populates @INC? :)

# Mon, 11 Jul 2011 04:25:33 -0700 Robert May <robertmay@cpan.org> - Correspondence added [Reply] [Comment]

CC:	perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 11 Jul 2011 12:24:56 +0100
To:	perlbug-followup [...] perl.org
From:	Robert May <robertmay [...] cpan.org>

Download (untitled) / with headers
text/plain 1.6k

On 11 July 2011 01:37, Christian Walde via RT <perlbug-followup@perl.org> wrote: Show quoted text

> On Wed Mar 30 06:35:51 2011, Mithaldu wrote:

>> On Windows 7 environment variable values can be arbitrarily large. >> However the perl executable ignores $ENV{PERL5LIB} if it goes above >> 32kbyte, despite being able to read it properly. This behavior can >> be reproduced with this minimal test script (example run outputs >> included): >> >> https://gist.github.com/894331

> > Nobody seems to have taken an interest, and i sadly lack the knowledge > and skill to poke at it myself. This is however still being an issue and > is causing false negatives when smoking on Win7. > > So i have to ask: Could someone please take a look at whatever piece of > code populates @INC? :)

I did start to have a look at this, and quickly got out of my depth. It's almost certainly down to limits on the enviroment block size for the process. See: http://msdn.microsoft.com/en-us/library/ms682653(v=vs.85).aspx http://blogs.msdn.com/b/oldnewthing/archive/2010/02/03/9957320.aspx Which state a limit for the environment block at ~32k bytes. But it's actually a whole load more complicated than that, depending on version of the c runtime and whether unicode is the default for the OS or not (Windows 2000 onwards?) - as on more recent OS the environment block is always unicode, and calls to Get/SetEnvironmentVariable as converted to/from Unicode as required. It's also possible for the process creator to specify the size of the environment block allocated to a process (See the unicode variant of CreateProcess()). I'm not sure what perl can do about this, as it has to live with the ENVIRONMENT block it has ... Rob.

# Mon, 11 Jul 2011 09:08:02 -0700 "Christian Walde" <walde.christian@googlemail.com> - Correspondence added [Reply] [Comment]

CC:	perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 11 Jul 2011 18:06:49 +0200
To:	perlbug-followup [...] perl.org, "Robert May" <robertmay [...] cpan.org>
From:	"Christian Walde" <walde.christian [...] googlemail.com>

Download (untitled) / with headers
text/plain 1.2k

On Mon Jul 11 04:25:33 2011, ROBERTMAY wrote: Show quoted text

> I'm not sure what perl can do about this, as it has to live with the > ENVIRONMENT block it has ...

I am not sure where i am going wrong in communicating this. :( The environment block is there, unharmed, accessible and perfectly fine. The perl interpreter just chooses to ignore it on Windows 7. On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}` and see my entire PERL5LIB stuff, even if it's a megabyte of data. At the same time @INC is empty, despite the data obviously being available to the Perl interpreter. This is not a case of Perl being unable to get the ENV data. It gets the data without any sort of problem. It just looks at it, goes "welp" and completely ignores it when it's supposed to fill in @INC, despite everything being just fine, present and accessible. I'm fairly sure that this is caused by there being some sort of filter inside the function that populates @INC which bails out when `length $ENV{PERL5LIB} > $os_env_size`. Which would be fairly pointless, since in all cases i've observed the OS itself refuses to create the process in the first place. On poking of theorbtwo i've written a test to demonstrate. It is in the attachment. -- With regards, Christian Walde

Download perl5lib.t
text/plain 838b

Message body is not shown because sender requested not to inline it.

# Mon, 11 Jul 2011 10:34:40 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	[perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 11 Jul 2011 12:34:11 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download (untitled) / with headers
text/plain 1.6k

On Mon, Jul 11, 2011 at 11:06 AM, Christian Walde < walde.christian@googlemail.com> wrote: Show quoted text

> On Mon Jul 11 04:25:33 2011, ROBERTMAY wrote: > > I'm not sure what perl can do about this, as it has to live with the

>> ENVIRONMENT block it has ... >>

> > I am not sure where i am going wrong in communicating this. :( > > The environment block is there, unharmed, accessible and perfectly > fine. The perl interpreter just chooses to ignore it on Windows 7. > > On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}` > and see my entire PERL5LIB stuff, even if it's a megabyte of data. At > the same time @INC is empty, despite the data obviously being available > to the Perl interpreter. > > This is not a case of Perl being unable to get the ENV data. > > It gets the data without any sort of problem. > > It just looks at it, goes "welp" and completely ignores it when it's > supposed to fill in @INC, despite everything being just fine, present > and accessible. > > I'm fairly sure that this is caused by there being some sort of filter > inside the function that populates @INC which bails out when `length > $ENV{PERL5LIB} > $os_env_size`. Which would be fairly pointless, since > in all cases i've observed the OS itself refuses to create the process > in the first place. > > On poking of theorbtwo i've written a test to demonstrate. It is in the > attachment. >

Looking at your test file, and then at the S_incpush*() functions in perl.c, I'd say the test is faulty. If I read the code correctly, supplying non-existent directories in PERL5LIB *should* result in them being ignored when augmenting @INC. --Phil

# Mon, 11 Jul 2011 11:24:01 -0700 "Christian Walde" <walde.christian@googlemail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 11 Jul 2011 20:23:07 +0200
To:	"Philip Monsen" <philip.monsen [...] gmail.com>
From:	"Christian Walde" <walde.christian [...] googlemail.com>

Download (untitled) / with headers
text/plain 2.3k

On Mon, 11 Jul 2011 19:34:11 +0200, Philip Monsen <philip.monsen@gmail.com> wrote: Show quoted text

> On Mon, Jul 11, 2011 at 11:06 AM, Christian Walde < > walde.christian@googlemail.com> wrote: >

>> On Mon Jul 11 04:25:33 2011, ROBERTMAY wrote: >> >> I'm not sure what perl can do about this, as it has to live with the

>>> ENVIRONMENT block it has ... >>>

>> >> I am not sure where i am going wrong in communicating this. :( >> >> The environment block is there, unharmed, accessible and perfectly >> fine. The perl interpreter just chooses to ignore it on Windows 7. >> >> On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}` >> and see my entire PERL5LIB stuff, even if it's a megabyte of data. At >> the same time @INC is empty, despite the data obviously being available >> to the Perl interpreter. >> >> This is not a case of Perl being unable to get the ENV data. >> >> It gets the data without any sort of problem. >> >> It just looks at it, goes "welp" and completely ignores it when it's >> supposed to fill in @INC, despite everything being just fine, present >> and accessible. >> >> I'm fairly sure that this is caused by there being some sort of filter >> inside the function that populates @INC which bails out when `length >> $ENV{PERL5LIB} > $os_env_size`. Which would be fairly pointless, since >> in all cases i've observed the OS itself refuses to create the process >> in the first place. >> >> On poking of theorbtwo i've written a test to demonstrate. It is in the >> attachment. >>

> > Looking at your test file, and then at the S_incpush*() functions in perl.c, > I'd say the test is faulty. If I read the code correctly, supplying > non-existent directories in PERL5LIB *should* result in them being ignored > when augmenting @INC.

If that sort of filtering were to happen, then this test should fail: is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@INC (this should work everywhere)"; As it doesn't i don't think that filtering does happen. Furthermore: The test is a simplified version of things that happened when doing smoke runs on Windows 7 and did actually cause false negatives to be registered in the cpantesters database. I'm not just making this stuff up, it is a problem that has already exploded nastily in reality and the stuff i communicate is merely a way to condense it down to essentials. -- With regards, Christian Walde

# Mon, 11 Jul 2011 15:44:52 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 11 Jul 2011 17:44:21 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download perl5lib.t
text/plain 1k

Message body is not shown because sender requested not to inline it.

Download (untitled) / with headers
text/plain 2k

Download (untitled) / with headers
text/html 2.8k

On Mon, Jul 11, 2011 at 1:23 PM, Christian Walde <walde.christian@googlemail.com> wrote:
Show quoted text

On Mon, 11 Jul 2011 19:34:11 +0200, Philip Monsen <philip.monsen@gmail.com> wrote:

Looking at your test file, and then at the S_incpush*() functions in perl.c,
I'd say the test is faulty. If I read the code correctly, supplying
non-existent directories in PERL5LIB *should* result in them being ignored
when augmenting @INC.

If that sort of filtering were to happen, then this test should fail:

Show quoted text

is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@INC (this should work everywhere)";

After some further staring at perl.c and experimentation on the command line, I agree that my earlier assertion was dead wrong.

Show quoted text

As it doesn't i don't think that filtering does happen.

Furthermore: The test is a simplified version of things that happened when doing smoke runs on Windows 7 and did actually cause false negatives to be registered in the cpantesters database.

I'm not just making this stuff up, it is a problem that has already exploded nastily in reality and the stuff i communicate is merely a way to condense it down to essentials.

Understood.

However, your test code, as-is, failed both tests out of the box for me under 5.12.1 on HP-UX (with the only change being changing the path separator from ";" to ":"). This is mostly what led me to my earlier hasty conclusion. I was able to modify the code to pass both tests on my HP-UX system. The key there was making sure to compare apples to apples, i.e. getting the size of @INC as seen by the subshell-spawned perl, not as already set in the test script, in the event they're different.

Of course, that proved to just be a red herring for the Win7 issue. In a 64-bit Win7 environment I do see what you're seeing. Total size of the environment (%ENV key lengths + value lengths) in the subshell is as expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long, then @INC isn't augmented at all.

--Phil

# Mon, 11 Jul 2011 15:46:54 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 11 Jul 2011 17:46:27 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download (untitled) / with headers
text/plain 564b

On Mon, Jul 11, 2011 at 5:44 PM, Philip Monsen <philip.monsen@gmail.com>wrote: Show quoted text

> > Of course, that proved to just be a red herring for the Win7 issue. In a > 64-bit Win7 environment I do see what you're seeing. Total size of the > environment (%ENV key lengths + value lengths) in the subshell is as > expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long, then > @INC isn't augmented at all. >

Also -- Disregard the attachment in my previous message. It was just code I was toying with and wasn't meant to be significant for this discussion.

# Tue, 12 Jul 2011 00:08:58 -0700 "Christian Walde" <walde.christian@googlemail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 12 Jul 2011 09:08:20 +0200
To:	"Philip Monsen" <philip.monsen [...] gmail.com>
From:	"Christian Walde" <walde.christian [...] googlemail.com>

Download (untitled) / with headers
text/plain 741b

On Tue, 12 Jul 2011 00:44:21 +0200, Philip Monsen <philip.monsen@gmail.com> wrote: Show quoted text

> However, your test code, as-is, failed both tests out of the box for me > under 5.12.1 on HP-UX (with the only change being changing the path > separator from ";" to ":").

Ah, OS conventions. :( Show quoted text

> Of course, that proved to just be a red herring for the Win7 issue. In a > 64-bit Win7 environment I do see what you're seeing. Total size of the > environment (%ENV key lengths + value lengths) in the subshell is as > expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long, then > @INC isn't augmented at all.

Thanks a lot for looking into this! You're the first person to corroborate my findings. :) -- With regards, Christian Walde

# Tue, 12 Jul 2011 06:49:16 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 12 Jul 2011 08:48:46 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download (untitled) / with headers
text/plain 731b

On Tue, Jul 12, 2011 at 2:08 AM, Christian Walde < walde.christian@googlemail.com> wrote: Show quoted text

> On Tue, 12 Jul 2011 00:44:21 +0200, Philip Monsen <philip.monsen@gmail.com> > wrote: >

>> In a 64-bit Win7 environment I do see what you're seeing. Total size of >> the >> environment (%ENV key lengths + value lengths) in the subshell is as >> expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long, >> then >> @INC isn't augmented at all. >>

> > Thanks a lot for looking into this! You're the first person to corroborate > my findings. :) >

FWIW, I've confirmed the phenomenon also occurs on 64-bit Win 2008 (under Strawberry 64-bit). Next step is to do some instrumentation/debugging to try to zero in on why. --Phil

# Tue, 12 Jul 2011 07:43:28 -0700 "Craig Berry" <craig.a.berry@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, Robert May <robertmay [...] cpan.org>, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 12 Jul 2011 09:43:00 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	"Craig A. Berry" <craig.a.berry [...] gmail.com>

Download (untitled) / with headers
text/plain 1k

On Mon, Jul 11, 2011 at 11:06 AM, Christian Walde <walde.christian@googlemail.com> wrote: Show quoted text

> > On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}` > and see my entire PERL5LIB stuff, even if it's a megabyte of data. At > the same time @INC is empty, despite the data obviously being available > to the Perl interpreter.

I think what's in Perl's %ENV hash is irrelevant; when it loads values into @INC, it's not looking at that -- it's fetching a single value from the actual environment using PerlEnv_getenv(), which is usually a macro pointing to getenv(), which on Windows appears to be a macro pointing to win32_getenv(), which is implemented in terms of GetEnvironmentVariableA, which I believe is documented to only support 32K. As a workaround, you could try storing these absurdly long values in the registry. win32_getenv() does support fetching values from there when the name begins with "PERL". Sorry for all the caveats and wishy-washy assertions; these are just guesses based on a couple minutes of code inspection.

# Tue, 12 Jul 2011 07:47:29 -0700 Zefram <zefram@fysh.org> - Correspondence added [Reply] [Comment]

Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 12 Jul 2011 15:46:52 +0100
To:	Christian Walde <walde.christian [...] googlemail.com>, perl5-porters [...] perl.org
From:	Zefram <zefram [...] fysh.org>

Download (untitled) / with headers
text/plain 145b

Craig A. Berry wrote: Show quoted text

>GetEnvironmentVariableA, which I believe is documented to only support >32K.

So where is %ENV getting it from? -zefram

# Tue, 12 Jul 2011 08:01:23 -0700 "Christian Walde" <walde.christian@googlemail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, "Robert May" <robertmay [...] cpan.org>, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 12 Jul 2011 17:00:36 +0200
To:	"Craig A. Berry" <craig.a.berry [...] gmail.com>
From:	"Christian Walde" <walde.christian [...] googlemail.com>

Download (untitled) / with headers
text/plain 1k

Show quoted text

> GetEnvironmentVariableA, which I believe is documented to only support > 32K.

This is documented here: http://msdn.microsoft.com/en-us/library/ms682653(v=vs.85).aspx Specifically it says: Show quoted text

> Windows Server 2003 and Windows XP/2000: The maximum size of the environment block for the process is 32,767 characters. > The maximum size changed starting with Windows Vista and Windows Server 2008.

-- Show quoted text

> I think what's in Perl's %ENV hash is irrelevant; when it loads values > into @INC, it's not looking at that -- it's fetching a single value > from the actual environment using PerlEnv_getenv(), which is usually a > macro pointing to getenv(), which on Windows appears to be a macro > pointing to win32_getenv(), which is implemented in terms of > GetEnvironmentVariableA

As far as implementation details go, this is interesting information. The hypothesis would then not be that there is sort of filtering in the INC population, but that the ENV population uses a superior method of data retrieval than the INV population. The question is: What is it? -- With regards, Christian Walde

# Tue, 12 Jul 2011 08:03:34 -0700 "Craig Berry" <craig.a.berry@gmail.com> - Correspondence added [Reply] [Comment]

CC:	Christian Walde <walde.christian [...] googlemail.com>, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 12 Jul 2011 10:02:59 -0500
To:	Zefram <zefram [...] fysh.org>
From:	"Craig A. Berry" <craig.a.berry [...] gmail.com>

Download (untitled) / with headers
text/plain 332b

On Tue, Jul 12, 2011 at 9:46 AM, Zefram <zefram@fysh.org> wrote: Show quoted text

> Craig A. Berry wrote:

>>GetEnvironmentVariableA, which I believe is documented to only support >>32K.

> > So where is %ENV getting it from?

Probably directly from the CRT's environ array. There is a lot of magic here and there to handle %ENV, especially in hv.c.

# Tue, 12 Jul 2011 08:31:15 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	"Craig A. Berry" <craig.a.berry [...] gmail.com>, perlbug-followup [...] perl.org, Robert May <robertmay [...] cpan.org>, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 12 Jul 2011 10:30:43 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download (untitled) / with headers
text/plain 972b

On Tue, Jul 12, 2011 at 10:00 AM, Christian Walde < walde.christian@googlemail.com> wrote: Show quoted text

> GetEnvironmentVariableA, which I believe is documented to only support

>> 32K. >>

> > This is documented here: http://msdn.microsoft.com/en-** > us/library/ms682653(v=vs.85).**aspx<http://msdn.microsoft.com/en-us/library/ms682653%28v=vs.85%29.aspx> > > Specifically it says: > > Windows Server 2003 and Windows XP/2000: The maximum size of the

>> environment block for the process is 32,767 characters. >> The maximum size changed starting with Windows Vista and Windows Server >> 2008. >>

> >

After reading this a couple times it seems an approach that may work, since when populating @INC we only need PERL5LIB to be readonly, would be to call GetEnvironmentStrings and parse it manually to get the value of PERL5LIB. The doc also says: There is no technical limitation on the size of the environment block. It should be easy to check whether this would work. --Phil

# Thu, 14 Jul 2011 11:25:07 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Thu, 14 Jul 2011 13:24:39 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download (untitled) / with headers
text/plain 1.2k

On Tue, Jul 12, 2011 at 10:30 AM, Philip Monsen <philip.monsen@gmail.com>wrote: Show quoted text

> > After reading this a couple times it seems an approach that may work, since > when populating @INC we only need PERL5LIB to be readonly, would be to call > GetEnvironmentStrings and parse it manually to get the value of PERL5LIB. > The doc also says: > > There is no technical limitation on the size of the environment block. > > It should be easy to check whether this would work. > >

Result so far: within win32_getenv(), the GetEnvironmentValueA() call returns 0 when PERL5LIB is set to a ridiculously large value. Subsequent GetLastError() returns ERROR_NOT_ENOUGH_MEMORY, which implies that it found the value in the environment (we would get ERROR_ENVVAR_NOT_FOUND if it weren't in the env) but that it was simply too large to handle. Right now, as pointed out a while back in the thread, the only attempted action taken in the code is to do a registry lookup if the variable name starts as PERL. I'll next try making that a last-ditch retrieval, and wiring in a GetEnvironmentStrings() call + result traversal to see if we can get the value that way, if we see the ERROR_NOT_ENOUGH_MEMORY case. There is precedence in the code already for using GetEnvironmentString(). win32_clearenv() uses it. --Phil

# Thu, 14 Jul 2011 13:36:38 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Thu, 14 Jul 2011 13:36:00 -0700
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download (untitled) / with headers
text/plain 399b

On Thu, Jul 14, 2011 at 11:24 AM, Philip Monsen <philip.monsen@gmail.com>wrote: Show quoted text

> I'll next try making that a last-ditch retrieval, and wiring in a > GetEnvironmentStrings() call + result traversal to see if we can get the > value that way, if we see the ERROR_NOT_ENOUGH_MEMORY case. > > This works for 5.14.1, and existing tests in test suite are unaffected.

Patch against blead to come. --Phil

# Fri, 15 Jul 2011 08:17:48 -0700 "Eric Brine" <ikegami@adaelis.com> - Correspondence added [Reply] [Comment]

CC:	Christian Walde <walde.christian [...] googlemail.com>, perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Fri, 15 Jul 2011 11:17:16 -0400
To:	Philip Monsen <philip.monsen [...] gmail.com>
From:	Eric Brine <ikegami [...] adaelis.com>

Download (untitled) / with headers
text/plain 1.4k

On Thu, Jul 14, 2011 at 2:24 PM, Philip Monsen <philip.monsen@gmail.com>wrote: Show quoted text

> On Tue, Jul 12, 2011 at 10:30 AM, Philip Monsen <philip.monsen@gmail.com

> >wrote:

>

> > > > After reading this a couple times it seems an approach that may work,

> since

> > when populating @INC we only need PERL5LIB to be readonly, would be to

> call

> > GetEnvironmentStrings and parse it manually to get the value of PERL5LIB. > > The doc also says: > > > > There is no technical limitation on the size of the environment

> block.

> > > > It should be easy to check whether this would work. > > > >

> Result so far: within win32_getenv(), the GetEnvironmentValueA() call > returns 0 when PERL5LIB is set to a ridiculously large value. Subsequent > GetLastError() returns ERROR_NOT_ENOUGH_MEMORY, which implies that it found > the value in the environment (we would get ERROR_ENVVAR_NOT_FOUND if it > weren't in the env) but that it was simply too large to handle. Right now, > as pointed out a while back in the thread, the only attempted action taken > in the code is to do a registry lookup if the variable name starts as PERL. > I'll next try making that a last-ditch retrieval, and wiring in a > GetEnvironmentStrings() call + result traversal to see if we can get the > value that way, if we see the ERROR_NOT_ENOUGH_MEMORY case. > > There is precedence in the code already for using GetEnvironmentString(). > win32_clearenv() uses it. >

What about GetEnvironmentValueW? Just curious.

# Fri, 15 Jul 2011 08:56:09 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	Christian Walde <walde.christian [...] googlemail.com>, perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re: [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Fri, 15 Jul 2011 10:55:33 -0500
To:	Eric Brine <ikegami [...] adaelis.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download (untitled) / with headers
text/plain 411b

On Fri, Jul 15, 2011 at 10:17 AM, Eric Brine <ikegami@adaelis.com> wrote: Show quoted text

> > What about GetEnvironmentValueW? Just curious. >

I'm not intimately familiar with the Perl win32 code as a whole, but I think some more extensive overhaul would be needed to use the *W functions (treating their arguments as Unicode) throughout the env-handling pieces. In my mind that goes beyond the scope of this issue. --Phil

# Mon, 18 Jul 2011 20:37:58 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject:	Re[PATCH] [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 18 Jul 2011 22:36:50 -0500
To:	Christian Walde <walde.christian [...] googlemail.com>
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download win32.c
text/x-csrc 111.5k

Message body is not shown because sender requested not to inline it.

Download (untitled) / with headers
text/plain 414b

Download (untitled) / with headers
text/html 434b

The attached patch solves the reported issue. It includes a new testfile to run the win32 runtime environment through the same paces as t/run/runenv.t (but without the forks), and of course to verify the bug fix.

Patch is against blead. I believe the issue exists in older versions of Perl. I did the initial exploration, development and testing with 5.14.1, so I know the fix is effective there also.

--Phil

# Mon, 18 Jul 2011 20:37:59 -0700 The RT System itself - Type patch added

# Mon, 18 Jul 2011 20:40:52 -0700 Phil Monsen <philip.monsen@gmail.com> - Correspondence added [Reply] [Comment]

CC:	perl5-porters [...] perl.org, Christian Walde <walde.christian [...] googlemail.com>
Subject:	[PATCH] [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Mon, 18 Jul 2011 22:40:20 -0500
To:	perlbug-followup [...] perl.org
From:	Philip Monsen <philip.monsen [...] gmail.com>

Download 0001-Fixes-to-allow-win32-Perl-to-properly-handle-PERL5LI.patch
text/plain 12.7k

Message body is not shown because sender requested not to inline it.

Download (untitled) / with headers
text/plain 606b

Download (untitled) / with headers
text/html 868b

On Mon, Jul 18, 2011 at 10:36 PM, Philip Monsen <philip.monsen@gmail.com> wrote:
Show quoted text

The attached patch solves the reported issue. It includes a new testfile to run the win32 runtime environment through the same paces as t/run/runenv.t (but without the forks), and of course to verify the bug fix.

Patch is against blead. I believe the issue exists in older versions of Perl. I did the initial exploration, development and testing with 5.14.1, so I know the fix is effective there also.

It would help if I actually attached the actual patch.

Now done. Sorry about the confusion.

--Phil

# Mon, 18 Jul 2011 20:40:52 -0700 The RT System itself - Type patch added

# Mon, 18 Jul 2011 22:00:04 -0700 Father Chrysostomos <sprout@cpan.org> - Correspondence added [Reply] [Comment]

RT-Send-CC:

perl5-porters [...] perl.org

Download (untitled) / with headers
text/plain 838b

On Mon Jul 18 20:40:52 2011, icerider wrote: Show quoted text

> On Mon, Jul 18, 2011 at 10:36 PM, Philip Monsen > <philip.monsen@gmail.com>wrote: >

> > The attached patch solves the reported issue. It includes a new

> testfile

> > to run the win32 runtime environment through the same paces as > > t/run/runenv.t (but without the forks), and of course to verify the

> bug fix.

> > > > Patch is against blead. I believe the issue exists in older

> versions of

> > Perl. I did the initial exploration, development and testing with

> 5.14.1,

> > so I know the fix is effective there also. > > > >

> It would help if I actually attached the actual patch. > > Now done. Sorry about the confusion.

I can’t really test your patch, but you sound as though you know what you are talking about. :-) So, thank you. I’ve just applied it as 1fcb0052e32fe.

# Mon, 18 Jul 2011 22:00:05 -0700 Father Chrysostomos <sprout@cpan.org> - Status changed from 'open' to 'resolved'

# Tue, 19 Jul 2011 00:23:35 -0700 "Christian Walde" <walde.christian@googlemail.com> - Correspondence added [Reply] [Comment]

CC:	perl5-porters [...] perl.org
Subject:	Re: [PATCH] [perl #87322] Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7
Date:	Tue, 19 Jul 2011 09:22:54 +0200
To:	"Philip Monsen" <philip.monsen [...] gmail.com>
From:	"Christian Walde" <walde.christian [...] googlemail.com>

Download (untitled) / with headers
text/plain 853b

On Tue, 19 Jul 2011 05:40:20 +0200, Philip Monsen <philip.monsen@gmail.com> wrote: Show quoted text

> On Mon, Jul 18, 2011 at 10:36 PM, Philip Monsen <philip.monsen@gmail.com>wrote: >

>> The attached patch solves the reported issue. It includes a new testfile >> to run the win32 runtime environment through the same paces as >> t/run/runenv.t (but without the forks), and of course to verify the bug fix. >> >> Patch is against blead. I believe the issue exists in older versions of >> Perl. I did the initial exploration, development and testing with 5.14.1, >> so I know the fix is effective there also. >> >>

> It would help if I actually attached the actual patch. > > Now done. Sorry about the confusion.

Phil, when i run into you at a YAPC or other, you're getting a few of whatever beverage you prefer. Thanks a lot. :) -- With regards, Christian Walde

# Tue, 19 Jul 2011 00:23:35 -0700 The RT System itself - Type patch added

# Tue, 12 Feb 2013 03:50:56 -0800 Leon Timmermans <fawaka@gmail.com> - Fixed In 5.16.0 added

# Tue, 12 Feb 2013 03:50:56 -0800 Leon Timmermans <fawaka@gmail.com> - Type patch deleted

# Tue, 12 Feb 2013 03:50:57 -0800 Leon Timmermans <fawaka@gmail.com> - Type patch deleted

# Tue, 12 Feb 2013 03:51:40 -0800 Leon Timmermans <fawaka@gmail.com> - PatchStatus Applied added