Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7 #11218

Closed
p5pRT opened this issue Mar 30, 2011 · 31 comments
Closed

Comments

@p5pRT
Copy link

p5pRT commented Mar 30, 2011

Migrated from rt.perl.org#87322 (status was 'resolved')

Searchable as RT87322$

@p5pRT
Copy link
Author

p5pRT commented Mar 30, 2011

From @wchristian

Created by mithaldu@yahoo.de

Short​:

On Windows 7 environment variable values can be arbitrarily large. However the perl executable ignores $ENV{PERL5LIB} if it goes above 32kbyte, despite being able to read it properly. This behavior can be reproduced with this minimal test script (example run outputs included)​:

https://gist.github.com/894331

Long​:

On Wed, 30 Mar 2011 10​:54​:57 +0200, Christian Walde <mithaldu@​yahoo.de> wrote​:

On Wed, 30 Mar 2011 02​:22​:32 +0200, perl@​0ne.us <perl@​0ne.us> wrote​:

I just got this FAIL report today from CPANTesters​:

http​://www.cpantesters.org/cpan/report/0ee8eb99-6c5d-1014-a630-0d8a02da96e1

As you can see, the prereq is correctly recognized by the toolchain
and loaded - it's even in the lengthy @​INC list, BUT when executing the
test script it seems like @​INC disappears somehow? Since I don't know
the details of your setup I have to assume something is broken.

https://gist.github.com/893981

Of note​: The test ( https​://gist.github.com/893981#L225 ) clearly has a whole battery of directories in $ENV{PERL5LIB} ( https​://gist.github.com/893981#L247 ) which the perl executable can obviously see, including the Params​::Classify dir as the very first; since that's a Data​::Dumper output of %ENV. However @​INC remains unchanged by that. ( https://gist.github.com/893981#L579 )

This seems to indicate to me that the perl executable itself is, for whatever reason, flat out ignoring the PERL5LIB and/or failing to inject it into @​INC. Other possibilities include Module​::Build messing things up or the Windows Perl executable exclusively having issues.

Thanks to a hint from Brian Raven i managed to reduce this to a minimal case and compared it across WinXP, Win7 and a Linux box.

Turns out that​:
- on Win XP perl dies if an ENV entry is >28kbyte
- on Linux perl dies if an ENV entry is >128kbyte
- on Win7 perl handles arbitrarily large values in ENV entries, but flat-out ignores any PERL5LIB values over 32kbyte
(all with 5.10, tested with both ActivePerl and Strawberry Perl)

Since this acts the same across both AP and Strawberry, it seems to be a genuine perl core bug.

Perl Info

Flags:
     category=core
     severity=low

Site configuration information for perl 5.10.1:

Configured by 1 at Thu Nov  4 19:37:29 2010.

Summary of my perl5 (revision 5 version 10 subversion 1) configuration:

   Platform:
     osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
     uname='Win32 strawberryperl 5.10.1.4 #1 Thu Nov  4 19:32:28 2010 i386'
     config_args='undef'
     hint=recommended, useposix=true, d_sigaction=undef
     useithreads=define, usemultiplicity=define
     useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
     use64bitint=undef, use64bitall=undef, uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='gcc', ccflags =' -s -O2 -DWIN32 -DHAVE_DES_FCRYPT  -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fno-strict-aliasing -DPERL_MSVCRT_READFIX',
     optimize='-s -O2',
     cppflags='-DWIN32'
     ccversion='', gccversion='3.4.5', gccosandvers=''
     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
     d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=12
     ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='g++', ldflags ='-s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'
     libpth=C:\strawberry\c\lib
     libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32
     perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32
     libc=, so=dll, useshrplib=true, libperl=libperl510.a
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
     cccdlflags=' ', lddlflags='-mdll -s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'

Locally applied patches:



@INC for perl 5.10.1:
     C:/strawberry/perl/lib
     C:/strawberry/perl/site/lib
     C:\strawberry\perl\vendor\lib
     .


Environment for perl 5.10.1:
     CYGWIN=nodosfilewarning
     HOME (unset)
     LANG (unset)
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=c:\cygwin\bin;C:\Program Files (x86)\Git\cmd;C:\Program Files (x86)\PHP\;C:\Perl\site\bin;C:\Perl\bin;C:\Program Files (x86)\ActiveState Komodo IDE 6\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\TortoiseSVN\bin;C:\Program Files (x86)\TortoiseGit\bin;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin
     PERL_BADLANG (unset)
     SHELL (unset)

@p5pRT
Copy link
Author

p5pRT commented Mar 30, 2011

From @jandubois

The problem is the creation of subprocesses when the environment block
is too large​:

"CreateProcessA() fails if the total size of the environment block for
the process exceeds 32,767 characters."

I don't consider this a bug in Perl, it is just a limitation of the OS.

Using the Unicode version of the API (CreateProcessW) would avoid this
particular problem, but could still run into other issues, where target
programs can't deal with these huge environment blocks.

@p5pRT
Copy link
Author

p5pRT commented Mar 30, 2011

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 30, 2011

From @wchristian

Jan, sorry to be blunt, but you missed something while reading​:

This is a problem specific to Windows 7 in a pretty unique manner.

Windows 7 does not have that limit and can create the new process just
fine. I was able to create processes with an $ENV{PERL5LIB} of 300
kilobyte, with the PERL5LIB value accessible inside the perl process via
%ENV, complete and intact.

Whichever part is responsible for @​INC however just throws its hands up
when it sees the ENV size and does absolutely nothing.

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @wchristian

On Wed Mar 30 06​:35​:51 2011, Mithaldu wrote​:

On Windows 7 environment variable values can be arbitrarily large.
However the perl executable ignores $ENV{PERL5LIB} if it goes above
32kbyte, despite being able to read it properly. This behavior can
be reproduced with this minimal test script (example run outputs
included)​:

https://gist.github.com/894331

Nobody seems to have taken an interest, and i sadly lack the knowledge
and skill to poke at it myself. This is however still being an issue and
is causing false negatives when smoking on Win7.

So i have to ask​: Could someone please take a look at whatever piece of
code populates @​INC? :)

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From robertmay@cpan.org

On 11 July 2011 01​:37, Christian Walde via RT <perlbug-followup@​perl.org> wrote​:

On Wed Mar 30 06​:35​:51 2011, Mithaldu wrote​:

On Windows 7 environment variable values can be arbitrarily large.
However the perl executable ignores $ENV{PERL5LIB} if it goes above
32kbyte, despite being able to read it properly. This behavior can
be reproduced with this minimal test script (example run outputs
included)​:

https://gist.github.com/894331

Nobody seems to have taken an interest, and i sadly lack the knowledge
and skill to poke at it myself. This is however still being an issue and
is causing false negatives when smoking on Win7.

So i have to ask​: Could someone please take a look at whatever piece of
code populates @​INC? :)

I did start to have a look at this, and quickly got out of my depth.
It's almost certainly down to limits on the enviroment block size for
the process. See​:

http​://msdn.microsoft.com/en-us/library/ms682653(v=vs.85).aspx
http​://blogs.msdn.com/b/oldnewthing/archive/2010/02/03/9957320.aspx

Which state a limit for the environment block at ~32k bytes. But
it's actually a whole load more complicated than that, depending on
version of the c runtime and whether unicode is the default for the OS
or not (Windows 2000 onwards?) - as on more recent OS the environment
block is always unicode, and calls to Get/SetEnvironmentVariable as
converted to/from Unicode as required. It's also possible for the
process creator to specify the size of the environment block allocated
to a process (See the unicode variant of CreateProcess()).

I'm not sure what perl can do about this, as it has to live with the
ENVIRONMENT block it has ...

Rob.

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @wchristian

On Mon Jul 11 04​:25​:33 2011, ROBERTMAY wrote​:

I'm not sure what perl can do about this, as it has to live with the
ENVIRONMENT block it has ...

I am not sure where i am going wrong in communicating this. :(

The environment block is there, unharmed, accessible and perfectly
fine. The perl interpreter just chooses to ignore it on Windows 7.

On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}`
and see my entire PERL5LIB stuff, even if it's a megabyte of data. At
the same time @​INC is empty, despite the data obviously being available
to the Perl interpreter.

This is not a case of Perl being unable to get the ENV data.

It gets the data without any sort of problem.

It just looks at it, goes "welp" and completely ignores it when it's
supposed to fill in @​INC, despite everything being just fine, present
and accessible.

I'm fairly sure that this is caused by there being some sort of filter
inside the function that populates @​INC which bails out when `length
$ENV{PERL5LIB} > $os_env_size`. Which would be fairly pointless, since
in all cases i've observed the OS itself refuses to create the process
in the first place.

On poking of theorbtwo i've written a test to demonstrate. It is in the attachment.

--
With regards,
Christian Walde

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @wchristian

use strict;
use warnings;
use Test​::More;
for my $add_entries ( 100, 1 ) {
  $ENV{PERL5LIB} = join ";", ("x" x 5000) x $add_entries;
  my ( $inc_entries ) = `perl -e "print scalar \@​INC"`;
  if( $? ) {
  pass "when ENV is too big, the child process dies (this happens only on Win XP or Linux)";
  next;
  }

  my $expected_entries = ( @​INC + $add_entries ) - grep { /dbgp/ && /Komodo/ } @​INC; # the grep is only there so this passes in my IDE

  if( $add_entries == 1 ) {
  is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@​INC (this should work everywhere)";
  }

  if( $add_entries == 100 ) {
  is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@​INC (this fails on Win7)";
  }
}
done_testing;

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @icerider70

On Mon, Jul 11, 2011 at 11​:06 AM, Christian Walde <
walde.christian@​googlemail.com> wrote​:

On Mon Jul 11 04​:25​:33 2011, ROBERTMAY wrote​:

I'm not sure what perl can do about this, as it has to live with the

ENVIRONMENT block it has ...

I am not sure where i am going wrong in communicating this. :(

The environment block is there, unharmed, accessible and perfectly
fine. The perl interpreter just chooses to ignore it on Windows 7.

On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}`
and see my entire PERL5LIB stuff, even if it's a megabyte of data. At
the same time @​INC is empty, despite the data obviously being available
to the Perl interpreter.

This is not a case of Perl being unable to get the ENV data.

It gets the data without any sort of problem.

It just looks at it, goes "welp" and completely ignores it when it's
supposed to fill in @​INC, despite everything being just fine, present
and accessible.

I'm fairly sure that this is caused by there being some sort of filter
inside the function that populates @​INC which bails out when `length
$ENV{PERL5LIB} > $os_env_size`. Which would be fairly pointless, since
in all cases i've observed the OS itself refuses to create the process
in the first place.

On poking of theorbtwo i've written a test to demonstrate. It is in the
attachment.

Looking at your test file, and then at the S_incpush*() functions in perl.c,
I'd say the test is faulty. If I read the code correctly, supplying
non-existent directories in PERL5LIB *should* result in them being ignored
when augmenting @​INC.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @wchristian

On Mon, 11 Jul 2011 19​:34​:11 +0200, Philip Monsen <philip.monsen@​gmail.com> wrote​:

On Mon, Jul 11, 2011 at 11​:06 AM, Christian Walde <
walde.christian@​googlemail.com> wrote​:

On Mon Jul 11 04​:25​:33 2011, ROBERTMAY wrote​:

I'm not sure what perl can do about this, as it has to live with the

ENVIRONMENT block it has ...

I am not sure where i am going wrong in communicating this. :(

The environment block is there, unharmed, accessible and perfectly
fine. The perl interpreter just chooses to ignore it on Windows 7.

On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}`
and see my entire PERL5LIB stuff, even if it's a megabyte of data. At
the same time @​INC is empty, despite the data obviously being available
to the Perl interpreter.

This is not a case of Perl being unable to get the ENV data.

It gets the data without any sort of problem.

It just looks at it, goes "welp" and completely ignores it when it's
supposed to fill in @​INC, despite everything being just fine, present
and accessible.

I'm fairly sure that this is caused by there being some sort of filter
inside the function that populates @​INC which bails out when `length
$ENV{PERL5LIB} > $os_env_size`. Which would be fairly pointless, since
in all cases i've observed the OS itself refuses to create the process
in the first place.

On poking of theorbtwo i've written a test to demonstrate. It is in the
attachment.

Looking at your test file, and then at the S_incpush*() functions in perl.c,
I'd say the test is faulty. If I read the code correctly, supplying
non-existent directories in PERL5LIB *should* result in them being ignored
when augmenting @​INC.

If that sort of filtering were to happen, then this test should fail​:

is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@​INC (this should work everywhere)";

As it doesn't i don't think that filtering does happen.

Furthermore​: The test is a simplified version of things that happened when doing smoke runs on Windows 7 and did actually cause false negatives to be registered in the cpantesters database.

I'm not just making this stuff up, it is a problem that has already exploded nastily in reality and the stuff i communicate is merely a way to condense it down to essentials.

--
With regards,
Christian Walde

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @icerider70

On Mon, Jul 11, 2011 at 1​:23 PM, Christian Walde <
walde.christian@​googlemail.com> wrote​:

On Mon, 11 Jul 2011 19​:34​:11 +0200, Philip Monsen <philip.monsen@​gmail.com>
wrote​:

Looking at your test file, and then at the S_incpush*() functions in
perl.c,
I'd say the test is faulty. If I read the code correctly, supplying
non-existent directories in PERL5LIB *should* result in them being ignored
when augmenting @​INC.

If that sort of filtering were to happen, then this test should fail​:

is $inc_entries, $expected_entries, "when ENV is the right size, all
PERL5LIB entries are added to \@​INC (this should work everywhere)";

After some further staring at perl.c and experimentation on the command
line, I agree that my earlier assertion was dead wrong.

As it doesn't i don't think that filtering does happen.

Furthermore​: The test is a simplified version of things that happened when
doing smoke runs on Windows 7 and did actually cause false negatives to be
registered in the cpantesters database.

I'm not just making this stuff up, it is a problem that has already
exploded nastily in reality and the stuff i communicate is merely a way to
condense it down to essentials.

Understood.

However, your test code, as-is, failed both tests out of the box for me
under 5.12.1 on HP-UX (with the only change being changing the path
separator from ";" to "​:"). This is mostly what led me to my earlier hasty
conclusion. I was able to modify the code to pass both tests on my HP-UX
system. The key there was making sure to compare apples to apples, i.e.
getting the size of @​INC as seen by the subshell-spawned perl, not as
already set in the test script, in the event they're different.

Of course, that proved to just be a red herring for the Win7 issue. In a
64-bit Win7 environment I do see what you're seeing. Total size of the
environment (%ENV key lengths + value lengths) in the subshell is as
expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long, then
@​INC isn't augmented at all.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @icerider70

use strict;
use warnings;
use Test​::More;
my $base_inc_size = `PERL5LIB="" perl -e "print scalar \@​INC"`;
my $base_inc = `PERL5LIB="" perl -e "print \"\@​INC\""`;
for my $add_entries ( 100, 1 ) {
  $ENV{PERL5LIB} = join "​:", ("x" x 5000) x $add_entries;
  unlink "foo.txt";
  !system("perl", "foo.pl") or do {
  pass "when ENV is too big, the child process dies (this happens only on Win XP or Linux)";
  next;
  };
  open(FOO, "<", "foo.txt") or die;
  chomp(my $inc_entries = <FOO>);
  close(FOO);

  my $expected_entries = ( $base_inc_size + $add_entries ) - grep { /dbgp/ && /Komodo/ } (split ' ', $base_inc); # the grep is only there so this passes in my IDE

  if( $add_entries == 1 ) {
  is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@​INC (this should work everywhere)";
  }

  if( $add_entries == 100 ) {
  is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@​INC (this fails on Win7)";
  }
}
done_testing;

@p5pRT
Copy link
Author

p5pRT commented Jul 11, 2011

From @icerider70

On Mon, Jul 11, 2011 at 5​:44 PM, Philip Monsen <philip.monsen@​gmail.com>wrote​:

Of course, that proved to just be a red herring for the Win7 issue. In a
64-bit Win7 environment I do see what you're seeing. Total size of the
environment (%ENV key lengths + value lengths) in the subshell is as
expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long, then
@​INC isn't augmented at all.

Also -- Disregard the attachment in my previous message. It was just code I
was toying with and wasn't meant to be significant for this discussion.

@p5pRT
Copy link
Author

p5pRT commented Jul 12, 2011

From @wchristian

On Tue, 12 Jul 2011 00​:44​:21 +0200, Philip Monsen <philip.monsen@​gmail.com> wrote​:

However, your test code, as-is, failed both tests out of the box for me
under 5.12.1 on HP-UX (with the only change being changing the path
separator from ";" to "​:").

Ah, OS conventions. :(

Of course, that proved to just be a red herring for the Win7 issue. In a
64-bit Win7 environment I do see what you're seeing. Total size of the
environment (%ENV key lengths + value lengths) in the subshell is as
expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long, then
@​INC isn't augmented at all.

Thanks a lot for looking into this! You're the first person to corroborate my findings. :)

--
With regards,
Christian Walde

@p5pRT
Copy link
Author

p5pRT commented Jul 12, 2011

From @icerider70

On Tue, Jul 12, 2011 at 2​:08 AM, Christian Walde <
walde.christian@​googlemail.com> wrote​:

On Tue, 12 Jul 2011 00​:44​:21 +0200, Philip Monsen <philip.monsen@​gmail.com>
wrote​:

In a 64-bit Win7 environment I do see what you're seeing. Total size of
the
environment (%ENV key lengths + value lengths) in the subshell is as
expected, but if the value of PERL5LIB exceeds 32766 (2**15 - 2) long,
then
@​INC isn't augmented at all.

Thanks a lot for looking into this! You're the first person to corroborate
my findings. :)

FWIW, I've confirmed the phenomenon also occurs on 64-bit Win 2008 (under
Strawberry 64-bit). Next step is to do some instrumentation/debugging to
try to zero in on why.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 12, 2011

From @craigberry

On Mon, Jul 11, 2011 at 11​:06 AM, Christian Walde
<walde.christian@​googlemail.com> wrote​:

On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}`
and see my entire PERL5LIB stuff, even if it's a megabyte of data. At
the same time @​INC is empty, despite the data obviously being available
to the Perl interpreter.

I think what's in Perl's %ENV hash is irrelevant; when it loads values
into @​INC, it's not looking at that -- it's fetching a single value
from the actual environment using PerlEnv_getenv(), which is usually a
macro pointing to getenv(), which on Windows appears to be a macro
pointing to win32_getenv(), which is implemented in terms of
GetEnvironmentVariableA, which I believe is documented to only support
32K.

As a workaround, you could try storing these absurdly long values in
the registry. win32_getenv() does support fetching values from there
when the name begins with "PERL".

Sorry for all the caveats and wishy-washy assertions; these are just
guesses based on a couple minutes of code inspection.

@p5pRT
Copy link
Author

p5pRT commented Jul 12, 2011

From zefram@fysh.org

Craig A. Berry wrote​:

GetEnvironmentVariableA, which I believe is documented to only support
32K.

So where is %ENV getting it from?

-zefram

@p5pRT
Copy link
Author

p5pRT commented Jul 12, 2011

From @wchristian

GetEnvironmentVariableA, which I believe is documented to only support
32K.

This is documented here​: http​://msdn.microsoft.com/en-us/library/ms682653(v=vs.85).aspx

Specifically it says​:

Windows Server 2003 and Windows XP/2000​: The maximum size of the environment block for the process is 32,767 characters.
The maximum size changed starting with Windows Vista and Windows Server 2008.

--

I think what's in Perl's %ENV hash is irrelevant; when it loads values
into @​INC, it's not looking at that -- it's fetching a single value
from the actual environment using PerlEnv_getenv(), which is usually a
macro pointing to getenv(), which on Windows appears to be a macro
pointing to win32_getenv(), which is implemented in terms of
GetEnvironmentVariableA
As far as implementation details go, this is interesting information.

The hypothesis would then not be that there is sort of filtering in the INC population, but that the ENV population uses a superior method of data retrieval than the INV population.

The question is​: What is it?

--
With regards,
Christian Walde

@p5pRT
Copy link
Author

p5pRT commented Jul 12, 2011

From @craigberry

On Tue, Jul 12, 2011 at 9​:46 AM, Zefram <zefram@​fysh.org> wrote​:

Craig A. Berry wrote​:

GetEnvironmentVariableA, which I believe is documented to only support
32K.

So where is %ENV getting it from?

Probably directly from the CRT's environ array. There is a lot of
magic here and there to handle %ENV, especially in hv.c.

@p5pRT
Copy link
Author

p5pRT commented Jul 12, 2011

From @icerider70

On Tue, Jul 12, 2011 at 10​:00 AM, Christian Walde <
walde.christian@​googlemail.com> wrote​:

GetEnvironmentVariableA, which I believe is documented to only support

32K.

This is documented here​: http​://msdn.microsoft.com/en-**
us/library/ms682653(v=vs.85).**aspx<http​://msdn.microsoft.com/en-us/library/ms682653%28v=vs.85%29.aspx>

Specifically it says​:

Windows Server 2003 and Windows XP/2000​: The maximum size of the

environment block for the process is 32,767 characters.
The maximum size changed starting with Windows Vista and Windows Server
2008.

After reading this a couple times it seems an approach that may work, since
when populating @​INC we only need PERL5LIB to be readonly, would be to call
GetEnvironmentStrings and parse it manually to get the value of PERL5LIB.
The doc also says​:

  There is no technical limitation on the size of the environment block.

It should be easy to check whether this would work.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 14, 2011

From @icerider70

On Tue, Jul 12, 2011 at 10​:30 AM, Philip Monsen <philip.monsen@​gmail.com>wrote​:

After reading this a couple times it seems an approach that may work, since
when populating @​INC we only need PERL5LIB to be readonly, would be to call
GetEnvironmentStrings and parse it manually to get the value of PERL5LIB.
The doc also says​:

There is no technical limitation on the size of the environment block\.

It should be easy to check whether this would work.

Result so far​: within win32_getenv(), the GetEnvironmentValueA() call
returns 0 when PERL5LIB is set to a ridiculously large value. Subsequent
GetLastError() returns ERROR_NOT_ENOUGH_MEMORY, which implies that it found
the value in the environment (we would get ERROR_ENVVAR_NOT_FOUND if it
weren't in the env) but that it was simply too large to handle. Right now,
as pointed out a while back in the thread, the only attempted action taken
in the code is to do a registry lookup if the variable name starts as PERL.
I'll next try making that a last-ditch retrieval, and wiring in a
GetEnvironmentStrings() call + result traversal to see if we can get the
value that way, if we see the ERROR_NOT_ENOUGH_MEMORY case.

There is precedence in the code already for using GetEnvironmentString().
win32_clearenv() uses it.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 14, 2011

From @icerider70

On Thu, Jul 14, 2011 at 11​:24 AM, Philip Monsen <philip.monsen@​gmail.com>wrote​:

I'll next try making that a last-ditch retrieval, and wiring in a
GetEnvironmentStrings() call + result traversal to see if we can get the
value that way, if we see the ERROR_NOT_ENOUGH_MEMORY case.

This works for 5.14.1, and existing tests in test suite are unaffected.
Patch against blead to come.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 15, 2011

From @ikegami

On Thu, Jul 14, 2011 at 2​:24 PM, Philip Monsen <philip.monsen@​gmail.com>wrote​:

On Tue, Jul 12, 2011 at 10​:30 AM, Philip Monsen <philip.monsen@​gmail.com

wrote​:

After reading this a couple times it seems an approach that may work,
since
when populating @​INC we only need PERL5LIB to be readonly, would be to
call
GetEnvironmentStrings and parse it manually to get the value of PERL5LIB.
The doc also says​:

There is no technical limitation on the size of the environment

block.

It should be easy to check whether this would work.

Result so far​: within win32_getenv(), the GetEnvironmentValueA() call
returns 0 when PERL5LIB is set to a ridiculously large value. Subsequent
GetLastError() returns ERROR_NOT_ENOUGH_MEMORY, which implies that it found
the value in the environment (we would get ERROR_ENVVAR_NOT_FOUND if it
weren't in the env) but that it was simply too large to handle. Right now,
as pointed out a while back in the thread, the only attempted action taken
in the code is to do a registry lookup if the variable name starts as PERL.
I'll next try making that a last-ditch retrieval, and wiring in a
GetEnvironmentStrings() call + result traversal to see if we can get the
value that way, if we see the ERROR_NOT_ENOUGH_MEMORY case.

There is precedence in the code already for using GetEnvironmentString().
win32_clearenv() uses it.

What about GetEnvironmentValueW? Just curious.

@p5pRT
Copy link
Author

p5pRT commented Jul 15, 2011

From @icerider70

On Fri, Jul 15, 2011 at 10​:17 AM, Eric Brine <ikegami@​adaelis.com> wrote​:

What about GetEnvironmentValueW? Just curious.

I'm not intimately familiar with the Perl win32 code as a whole, but I think
some more extensive overhaul would be needed to use the *W functions
(treating their arguments as Unicode) throughout the env-handling pieces.
In my mind that goes beyond the scope of this issue.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2011

From @icerider70

The attached patch solves the reported issue. It includes a new testfile to
run the win32 runtime environment through the same paces as t/run/runenv.t
(but without the forks), and of course to verify the bug fix.

Patch is against blead. I believe the issue exists in older versions of
Perl. I did the initial exploration, development and testing with 5.14.1,
so I know the fix is effective there also.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2011

From @icerider70

win32.c

@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2011

From @icerider70

On Mon, Jul 18, 2011 at 10​:36 PM, Philip Monsen <philip.monsen@​gmail.com>wrote​:

The attached patch solves the reported issue. It includes a new testfile
to run the win32 runtime environment through the same paces as
t/run/runenv.t (but without the forks), and of course to verify the bug fix.

Patch is against blead. I believe the issue exists in older versions of
Perl. I did the initial exploration, development and testing with 5.14.1,
so I know the fix is effective there also.

It would help if I actually attached the actual patch.

Now done. Sorry about the confusion.

--Phil

@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2011

From @icerider70

0001-Fixes-to-allow-win32-Perl-to-properly-handle-PERL5LI.patch
From 1d5be21b9d6b5ab7cd34b4034b9d736067289b99 Mon Sep 17 00:00:00 2001
From: Phil Monsen <philip.monsen@pobox.com>
Date: Mon, 18 Jul 2011 22:16:55 -0500
Subject: [PATCH] Fixes to allow win32 Perl to properly handle PERL5LIB.
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="------------1.7.2.5"

This is a multi-part message in MIME format.
--------------1.7.2.5
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit


On Windows Vista, 7 and 2008, the win32 API call
GetEnvironmentVariableA() does not return environment values
with string length of greater than 32766, even though
such variables are supported in the environment.

This consequently caused @INC not to be populated for
such values of PERL5LIB on those OSes, as reported in
RT #87322.

This commit reworks the code so that GetEnvironmentStrings()
is called if GetEnvironmentVariableA() indicates the requested
value is set in the environmtn.  The old fallback of consulting
the registry for variables beginning with "PERL" is retained, but
as a last-ditch fallback rather than the only recourse.

A new test file, t/win32/runenv.t has been added to validate
that the new behavior is working properly, as well as that
general environment variable handling is in accordance with
expectations, since t/run/runenv.t does not run on Win* platforms.
The new test file is essentially a non-forking clone of
t/run/runenv.t, with modifications to test cases to run properly
on Win* platforms, and with a new test case to test the new behavior.
---
 MANIFEST          |    1 +
 pod/perldelta.pod |   27 ++++++-
 t/win32/runenv.t  |  250 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 win32/win32.c     |   36 +++++++-
 4 files changed, 308 insertions(+), 6 deletions(-)
 create mode 100644 t/win32/runenv.t


--------------1.7.2.5
Content-Type: text/x-patch; name="0001-Fixes-to-allow-win32-Perl-to-properly-handle-PERL5LI.patch"
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment; filename="0001-Fixes-to-allow-win32-Perl-to-properly-handle-PERL5LI.patch"

diff --git a/MANIFEST b/MANIFEST
index 71ba80e..bab622e 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -5175,6 +5175,7 @@ t/uni/tr_sjis.t			See if Unicode tr/// in sjis works
 t/uni/tr_utf8.t			See if Unicode tr/// in utf8 works
 t/uni/upper.t			See if Unicode casing works
 t/uni/write.t			See if Unicode formats work
+t/win32/runenv.t		Test if Win* perl honors its env variables
 t/win32/system.t		See if system works in Win*
 t/win32/system_tests		Test runner for system.t
 t/x2p/s2p.t			See if s2p/psed work
diff --git a/pod/perldelta.pod b/pod/perldelta.pod
index ff141fd..a11a8e7 100644
--- a/pod/perldelta.pod
+++ b/pod/perldelta.pod
@@ -526,6 +526,12 @@ and if calling C<dtrace> actually lets you instrument code. This
 generally requires being run as root, so this test file is primarily
 intended for use by the dtrace subcommittee of p5p.
 
+=item *
+
+F<t/win32/runenv.t> was added to test aspects of Perl's environment
+variable handling on MSWin32 platforms.  Previously, such tests were
+skipped on MSWin32 platforms.
+
 =back
 
 =head1 Platform Support
@@ -569,9 +575,26 @@ and compilation changes or changes in portability/compatibility.  However,
 changes within modules for platforms should generally be listed in the
 L</Modules and Pragmata> section.
 
-=over 4
+=head3 Windows
 
-=item XXX-some-platform
+=over
+
+=item *
+
+On Windows 7, 2008 and Vista, C<@INC> is now always properly populated
+based on the value of PERL5LIB set in the environment.  Previously,
+values of PERL5LIB longer than 32766 bytes were skipped when C<@INC>
+was being populated.  Tests for environment handling were
+also added (see L</Testing> section).  Fixes
+L<RT #87322|https://rt.perl.org/rt3/Public/Bug/Display.html?id=87322>.
+
+=back
+
+=head3 XXX-some-platform
+
+=over
+
+=item *
 
 XXX
 
diff --git a/t/win32/runenv.t b/t/win32/runenv.t
new file mode 100644
index 0000000..2576168
--- /dev/null
+++ b/t/win32/runenv.t
@@ -0,0 +1,250 @@
+#!./perl
+#
+# Tests for Perl run-time environment variable settings
+# Clone of t/run/runenv.t but without the forking, and with cmd.exe-friendly -e syntax.
+#
+# $PERL5OPT, $PERL5LIB, etc.
+
+BEGIN {
+    chdir 't' if -d 't';
+    @INC = '../lib';
+    require Config; import Config;
+    require File::Temp; import File::Temp qw/:POSIX/;
+
+    require Win32;
+    ($::os_id, $::os_major) = ( Win32::GetOSVersion() )[ 4, 1 ];
+    if ($::os_id == 2 and $::os_major == 6) {    # Vista, Server 2008 (incl R2), 7
+	$::tests = 43;
+    }
+    else {
+	$::tests = 42;
+    }
+
+    require './test.pl';
+}
+
+plan tests => $::tests;
+
+my $PERL = $ENV{PERL} || '.\perl';
+my $NL = $/;
+
+delete $ENV{PERLLIB};
+delete $ENV{PERL5LIB};
+delete $ENV{PERL5OPT};
+
+
+# Run perl with specified environment and arguments, return (STDOUT, STDERR)
+sub runperl_and_capture {
+  my ($env, $args) = @_;
+
+  # Clear out old env
+  local %ENV = %ENV;
+  delete $ENV{PERLLIB};
+  delete $ENV{PERL5LIB};
+  delete $ENV{PERL5OPT};
+
+  # Populate with our desired env
+  for my $k (keys %$env) {
+     $ENV{$k} = $env->{$k};
+  }
+
+  # This is slightly expensive, but this is more reliable than
+  # trying to emulate fork(), and we still get STDERR and STDOUT individually.
+  my $stderr_cache = tmpnam();
+  my $stdout = `$PERL @$args 2>$stderr_cache`;
+  my $stderr = '';
+  if (-s $stderr_cache) {
+    open(my $stderr_cache_fh, "<", $stderr_cache)
+      or die "Could not retrieve STDERR output: $!";
+    while ( defined(my $s_line = <$stderr_cache_fh>) ) {
+      $stderr .= $s_line;
+    }
+    close $stderr_cache_fh;
+    unlink $stderr_cache;
+  }
+  
+  return ($stdout, $stderr);
+}
+
+sub try {
+  my ($env, $args, $stdout, $stderr) = @_;
+  my ($actual_stdout, $actual_stderr) = runperl_and_capture($env, $args);
+  local $::Level = $::Level + 1;
+  is ($stdout, $actual_stdout);
+  is ($stderr, $actual_stderr);
+}
+
+#  PERL5OPT    Command-line options (switches).  Switches in
+#                    this variable are taken as if they were on
+#                    every Perl command line.  Only the -[DIMUdmtw]
+#                    switches are allowed.  When running taint
+#                    checks (because the program was running setuid
+#                    or setgid, or the -T switch was used), this
+#                    variable is ignored.  If PERL5OPT begins with
+#                    -T, tainting will be enabled, and any
+#                    subsequent options ignored.
+
+try({PERL5OPT => '-w'}, ['-e', '"print $::x"'],
+    "", 
+    qq(Name "main::x" used only once: possible typo at -e line 1.${NL}Use of uninitialized value \$x in print at -e line 1.${NL}));
+
+try({PERL5OPT => '-Mstrict'}, ['-I..\lib', '-e', '"print $::x"'],
+    "", "");
+
+try({PERL5OPT => '-Mstrict'}, ['-I..\lib', '-e', '"print $x"'],
+    "", 
+    qq(Global symbol "\$x" requires explicit package name at -e line 1.${NL}Execution of -e aborted due to compilation errors.${NL}));
+
+# Fails in 5.6.0
+try({PERL5OPT => '-Mstrict -w'}, ['-I..\lib', '-e', '"print $x"'],
+    "", 
+    qq(Global symbol "\$x" requires explicit package name at -e line 1.${NL}Execution of -e aborted due to compilation errors.${NL}));
+
+# Fails in 5.6.0
+try({PERL5OPT => '-w -Mstrict'}, ['-I..\lib', '-e', '"print $::x"'],
+    "", 
+    <<ERROR
+Name "main::x" used only once: possible typo at -e line 1.
+Use of uninitialized value \$x in print at -e line 1.
+ERROR
+    );
+
+# Fails in 5.6.0
+try({PERL5OPT => '-w -Mstrict'}, ['-I..\lib', '-e', '"print $::x"'],
+    "", 
+    <<ERROR
+Name "main::x" used only once: possible typo at -e line 1.
+Use of uninitialized value \$x in print at -e line 1.
+ERROR
+    );
+
+try({PERL5OPT => '-MExporter'}, ['-I..\lib', '-e0'],
+    "", 
+    "");
+
+# Fails in 5.6.0
+try({PERL5OPT => '-MExporter -MExporter'}, ['-I..\lib', '-e0'],
+    "", 
+    "");
+
+try({PERL5OPT => '-Mstrict -Mwarnings'}, 
+    ['-I..\lib', '-e', '"print \"ok\" if $INC{\"strict.pm\"} and $INC{\"warnings.pm\"}"'],
+    "ok",
+    "");
+
+open my $fh, ">", "Oooof.pm" or die "Can't write Oooof.pm: $!";
+print $fh "package Oooof; 1;\n";
+close $fh;
+END { 1 while unlink "Oooof.pm" }
+
+try({PERL5OPT => '-I. -MOooof'}, 
+    ['-e', '"print \"ok\" if $INC{\"Oooof.pm\"} eq \"Oooof.pm\""'],
+    "ok",
+    "");
+
+try({PERL5OPT => '-w -w'},
+    ['-e', '"print $ENV{PERL5OPT}"'],
+    '-w -w',
+    '');
+
+try({PERL5OPT => '-t'},
+    ['-e', '"print ${^TAINT}"'],
+    '-1',
+    '');
+
+try({PERL5OPT => '-W'},
+    ['-I..\lib','-e', '"local $^W = 0;  no warnings;  print $x"'],
+    '',
+    <<ERROR
+Name "main::x" used only once: possible typo at -e line 1.
+Use of uninitialized value \$x in print at -e line 1.
+ERROR
+);
+
+try({PERLLIB => "foobar$Config{path_sep}42"},
+    ['-e', '"print grep { $_ eq \"foobar\" } @INC"'],
+    'foobar',
+    '');
+
+try({PERLLIB => "foobar$Config{path_sep}42"},
+    ['-e', '"print grep { $_ eq \"42\" } @INC"'],
+    '42',
+    '');
+
+try({PERL5LIB => "foobar$Config{path_sep}42"},
+    ['-e', '"print grep { $_ eq \"foobar\" } @INC"'],
+    'foobar',
+    '');
+
+try({PERL5LIB => "foobar$Config{path_sep}42"},
+    ['-e', '"print grep { $_ eq \"42\" } @INC"'],
+    '42',
+    '');
+
+try({PERL5LIB => "foo",
+     PERLLIB => "bar"},
+    ['-e', '"print grep { $_ eq \"foo\" } @INC"'],
+    'foo',
+    '');
+
+try({PERL5LIB => "foo",
+     PERLLIB => "bar"},
+    ['-e', '"print grep { $_ eq \"bar\" } @INC"'],
+    '',
+    '');
+
+# Tests for S_incpush_use_sep():
+
+my @dump_inc = ('-e', '"print \"$_\n\" foreach @INC"');
+
+my ($out, $err) = runperl_and_capture({}, [@dump_inc]);
+
+is ($err, '', 'No errors when determining @INC');
+
+my @default_inc = split /\n/, $out;
+
+is ($default_inc[-1], '.', '. is last in @INC');
+
+my $sep = $Config{path_sep};
+my @test_cases = (
+	 ['nothing', ''],
+	 ['something', 'zwapp', 'zwapp'],
+	 ['two things', "zwapp${sep}bam", 'zwapp', 'bam'],
+	 ['two things, ::', "zwapp${sep}${sep}bam", 'zwapp', 'bam'],
+	 [': at start', "${sep}zwapp", 'zwapp'],
+	 [': at end', "zwapp${sep}", 'zwapp'],
+	 [':: sandwich ::', "${sep}${sep}zwapp${sep}${sep}", 'zwapp'],
+	 [':', "${sep}"],
+	 ['::', "${sep}${sep}"],
+	 [':::', "${sep}${sep}${sep}"],
+	 ['two things and :', "zwapp${sep}bam${sep}", 'zwapp', 'bam'],
+	 [': and two things', "${sep}zwapp${sep}bam", 'zwapp', 'bam'],
+	 [': two things :', "${sep}zwapp${sep}bam${sep}", 'zwapp', 'bam'],
+	 ['three things', "zwapp${sep}bam${sep}${sep}owww",
+	  'zwapp', 'bam', 'owww'],
+);
+
+# This block added to verify fix for RT #87322
+if ($::os_id == 2 and $::os_major == 6) {    # Vista, Server 2008 (incl R2), 7
+  my @big_perl5lib = ('z' x 16) x 2049;
+    push @testcases, [
+        'enough items so PERL5LIB val is longer than 32k',
+        join($sep, @big_perl5lib), @big_perl5lib,
+    ];
+}
+
+foreach ( @testcases ) {
+  my ($name, $lib, @expect) = @$_;
+  push @expect, @default_inc;
+
+  ($out, $err) = runperl_and_capture({PERL5LIB => $lib}, [@dump_inc]);
+
+  is ($err, '', "No errors when determining \@INC for $name");
+
+  my @inc = split /\n/, $out;
+
+  is (scalar @inc, scalar @expect,
+      "expected number of elements in \@INC for $name");
+
+  is ("@inc", "@expect", "expected elements in \@INC for $name");
+}
diff --git a/win32/win32.c b/win32/win32.c
index cffd2b5..e67a735 100644
--- a/win32/win32.c
+++ b/win32/win32.c
@@ -1693,6 +1693,7 @@ win32_getenv(const char *name)
     dTHX;
     DWORD needlen;
     SV *curitem = NULL;
+    DWORD last_err;
 
     needlen = GetEnvironmentVariableA(name,NULL,0);
     if (needlen != 0) {
@@ -1705,10 +1706,37 @@ win32_getenv(const char *name)
         SvCUR_set(curitem, needlen);
     }
     else {
-	/* allow any environment variables that begin with 'PERL'
-	   to be stored in the registry */
-	if (strncmp(name, "PERL", 4) == 0)
-	    (void)get_regstr(name, &curitem);
+	last_err = GetLastError();
+	if (last_err == ERROR_NOT_ENOUGH_MEMORY) {
+	    /* It appears the variable is in the env, but the Win32 API
+	       doesn't have a canned way of getting it.  So we fall back to
+	       grabbing the whole env and pulling this value out if possible */
+	    char *envv = GetEnvironmentStrings();
+    	    char *cur = envv;
+    	    STRLEN len;
+    	    while (*cur) {
+		char *end = strchr(cur,'=');
+		if (end && end != cur) {
+		    *end = '\0';
+		    if (!strcmp(cur,name)) {
+			curitem = sv_2mortal(newSVpv(end+1,0));
+			*end = '=';
+			break;
+		    }
+	    	    *end = '=';
+	    	    cur = end + strlen(end+1)+2;
+		}
+		else if ((len = strlen(cur)))
+	    	    cur += len+1;
+    	    }
+    	    FreeEnvironmentStrings(envv);
+	}
+	else {
+	    /* last ditch: allow any environment variables that begin with 'PERL'
+	       to be obtained from the registry, if found there */
+	    if (strncmp(name, "PERL", 4) == 0)
+		(void)get_regstr(name, &curitem);
+	}
     }
     if (curitem && SvCUR(curitem))
 	return SvPVX(curitem);

--------------1.7.2.5--


@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2011

From @cpansprout

On Mon Jul 18 20​:40​:52 2011, icerider wrote​:

On Mon, Jul 18, 2011 at 10​:36 PM, Philip Monsen
<philip.monsen@​gmail.com>wrote​:

The attached patch solves the reported issue. It includes a new
testfile
to run the win32 runtime environment through the same paces as
t/run/runenv.t (but without the forks), and of course to verify the
bug fix.

Patch is against blead. I believe the issue exists in older
versions of
Perl. I did the initial exploration, development and testing with
5.14.1,
so I know the fix is effective there also.

It would help if I actually attached the actual patch.

Now done. Sorry about the confusion.

I can’t really test your patch, but you sound as though you know what
you are talking about. :-) So, thank you. I’ve just applied it as
1fcb005.

@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2011

@cpansprout - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Jul 19, 2011
@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2011

From @wchristian

On Tue, 19 Jul 2011 05​:40​:20 +0200, Philip Monsen <philip.monsen@​gmail.com> wrote​:

On Mon, Jul 18, 2011 at 10​:36 PM, Philip Monsen <philip.monsen@​gmail.com>wrote​:

The attached patch solves the reported issue. It includes a new testfile
to run the win32 runtime environment through the same paces as
t/run/runenv.t (but without the forks), and of course to verify the bug fix.

Patch is against blead. I believe the issue exists in older versions of
Perl. I did the initial exploration, development and testing with 5.14.1,
so I know the fix is effective there also.

It would help if I actually attached the actual patch.

Now done. Sorry about the confusion.

Phil, when i run into you at a YAPC or other, you're getting a few of whatever beverage you prefer. Thanks a lot. :)

--
With regards,
Christian Walde

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant