New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Large PERL5LIB ENV values get ignored instead of imported into @INC on Windows 7 #11218
Comments
From @wchristianCreated by mithaldu@yahoo.deShort: On Windows 7 environment variable values can be arbitrarily large. However the perl executable ignores $ENV{PERL5LIB} if it goes above 32kbyte, despite being able to read it properly. This behavior can be reproduced with this minimal test script (example run outputs included): https://gist.github.com/894331 Long: On Wed, 30 Mar 2011 10:54:57 +0200, Christian Walde <mithaldu@yahoo.de> wrote:
Thanks to a hint from Brian Raven i managed to reduce this to a minimal case and compared it across WinXP, Win7 and a Linux box. Turns out that: Since this acts the same across both AP and Strawberry, it seems to be a genuine perl core bug. Perl Info
|
From @janduboisThe problem is the creation of subprocesses when the environment block "CreateProcessA() fails if the total size of the environment block for I don't consider this a bug in Perl, it is just a limitation of the OS. Using the Unicode version of the API (CreateProcessW) would avoid this |
The RT System itself - Status changed from 'new' to 'open' |
From @wchristianJan, sorry to be blunt, but you missed something while reading: This is a problem specific to Windows 7 in a pretty unique manner. Windows 7 does not have that limit and can create the new process just Whichever part is responsible for @INC however just throws its hands up |
From @wchristianOn Wed Mar 30 06:35:51 2011, Mithaldu wrote:
Nobody seems to have taken an interest, and i sadly lack the knowledge So i have to ask: Could someone please take a look at whatever piece of |
From robertmay@cpan.orgOn 11 July 2011 01:37, Christian Walde via RT <perlbug-followup@perl.org> wrote:
I did start to have a look at this, and quickly got out of my depth. http://msdn.microsoft.com/en-us/library/ms682653(v=vs.85).aspx Which state a limit for the environment block at ~32k bytes. But I'm not sure what perl can do about this, as it has to live with the Rob. |
From @wchristianOn Mon Jul 11 04:25:33 2011, ROBERTMAY wrote:
I am not sure where i am going wrong in communicating this. :( The environment block is there, unharmed, accessible and perfectly On Windows 7, inside the perl process, i can do `print $ENV{PERL5LIB}` This is not a case of Perl being unable to get the ENV data. It gets the data without any sort of problem. It just looks at it, goes "welp" and completely ignores it when it's I'm fairly sure that this is caused by there being some sort of filter On poking of theorbtwo i've written a test to demonstrate. It is in the attachment. -- |
From @wchristianuse strict; my $expected_entries = ( @INC + $add_entries ) - grep { /dbgp/ && /Komodo/ } @INC; # the grep is only there so this passes in my IDE if( $add_entries == 1 ) { if( $add_entries == 100 ) { |
From @icerider70On Mon, Jul 11, 2011 at 11:06 AM, Christian Walde <
Looking at your test file, and then at the S_incpush*() functions in perl.c, --Phil |
From @wchristianOn Mon, 11 Jul 2011 19:34:11 +0200, Philip Monsen <philip.monsen@gmail.com> wrote:
If that sort of filtering were to happen, then this test should fail: is $inc_entries, $expected_entries, "when ENV is the right size, all PERL5LIB entries are added to \@INC (this should work everywhere)"; As it doesn't i don't think that filtering does happen. Furthermore: The test is a simplified version of things that happened when doing smoke runs on Windows 7 and did actually cause false negatives to be registered in the cpantesters database. I'm not just making this stuff up, it is a problem that has already exploded nastily in reality and the stuff i communicate is merely a way to condense it down to essentials. -- |
From @icerider70On Mon, Jul 11, 2011 at 1:23 PM, Christian Walde <
After some further staring at perl.c and experimentation on the command As it doesn't i don't think that filtering does happen.
Understood. However, your test code, as-is, failed both tests out of the box for me Of course, that proved to just be a red herring for the Win7 issue. In a --Phil |
From @icerider70use strict; my $expected_entries = ( $base_inc_size + $add_entries ) - grep { /dbgp/ && /Komodo/ } (split ' ', $base_inc); # the grep is only there so this passes in my IDE if( $add_entries == 1 ) { if( $add_entries == 100 ) { |
From @icerider70On Mon, Jul 11, 2011 at 5:44 PM, Philip Monsen <philip.monsen@gmail.com>wrote:
Also -- Disregard the attachment in my previous message. It was just code I |
From @wchristianOn Tue, 12 Jul 2011 00:44:21 +0200, Philip Monsen <philip.monsen@gmail.com> wrote:
Ah, OS conventions. :(
Thanks a lot for looking into this! You're the first person to corroborate my findings. :) -- |
From @icerider70On Tue, Jul 12, 2011 at 2:08 AM, Christian Walde <
FWIW, I've confirmed the phenomenon also occurs on 64-bit Win 2008 (under --Phil |
From @craigberryOn Mon, Jul 11, 2011 at 11:06 AM, Christian Walde
I think what's in Perl's %ENV hash is irrelevant; when it loads values As a workaround, you could try storing these absurdly long values in Sorry for all the caveats and wishy-washy assertions; these are just |
From zefram@fysh.orgCraig A. Berry wrote:
So where is %ENV getting it from? -zefram |
From @wchristian
This is documented here: http://msdn.microsoft.com/en-us/library/ms682653(v=vs.85).aspx Specifically it says:
--
The hypothesis would then not be that there is sort of filtering in the INC population, but that the ENV population uses a superior method of data retrieval than the INV population. The question is: What is it? -- |
From @craigberryOn Tue, Jul 12, 2011 at 9:46 AM, Zefram <zefram@fysh.org> wrote:
Probably directly from the CRT's environ array. There is a lot of |
From @icerider70On Tue, Jul 12, 2011 at 10:00 AM, Christian Walde <
After reading this a couple times it seems an approach that may work, since There is no technical limitation on the size of the environment block. It should be easy to check whether this would work. --Phil |
From @icerider70On Tue, Jul 12, 2011 at 10:30 AM, Philip Monsen <philip.monsen@gmail.com>wrote:
Result so far: within win32_getenv(), the GetEnvironmentValueA() call There is precedence in the code already for using GetEnvironmentString(). --Phil |
From @icerider70On Thu, Jul 14, 2011 at 11:24 AM, Philip Monsen <philip.monsen@gmail.com>wrote:
--Phil |
From @ikegamiOn Thu, Jul 14, 2011 at 2:24 PM, Philip Monsen <philip.monsen@gmail.com>wrote:
What about GetEnvironmentValueW? Just curious. |
From @icerider70On Fri, Jul 15, 2011 at 10:17 AM, Eric Brine <ikegami@adaelis.com> wrote:
I'm not intimately familiar with the Perl win32 code as a whole, but I think --Phil |
From @icerider70The attached patch solves the reported issue. It includes a new testfile to Patch is against blead. I believe the issue exists in older versions of --Phil |
From @icerider70 |
From @icerider70On Mon, Jul 18, 2011 at 10:36 PM, Philip Monsen <philip.monsen@gmail.com>wrote:
It would help if I actually attached the actual patch. Now done. Sorry about the confusion. --Phil |
From @icerider700001-Fixes-to-allow-win32-Perl-to-properly-handle-PERL5LI.patchFrom 1d5be21b9d6b5ab7cd34b4034b9d736067289b99 Mon Sep 17 00:00:00 2001
From: Phil Monsen <philip.monsen@pobox.com>
Date: Mon, 18 Jul 2011 22:16:55 -0500
Subject: [PATCH] Fixes to allow win32 Perl to properly handle PERL5LIB.
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="------------1.7.2.5"
This is a multi-part message in MIME format.
--------------1.7.2.5
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit
On Windows Vista, 7 and 2008, the win32 API call
GetEnvironmentVariableA() does not return environment values
with string length of greater than 32766, even though
such variables are supported in the environment.
This consequently caused @INC not to be populated for
such values of PERL5LIB on those OSes, as reported in
RT #87322.
This commit reworks the code so that GetEnvironmentStrings()
is called if GetEnvironmentVariableA() indicates the requested
value is set in the environmtn. The old fallback of consulting
the registry for variables beginning with "PERL" is retained, but
as a last-ditch fallback rather than the only recourse.
A new test file, t/win32/runenv.t has been added to validate
that the new behavior is working properly, as well as that
general environment variable handling is in accordance with
expectations, since t/run/runenv.t does not run on Win* platforms.
The new test file is essentially a non-forking clone of
t/run/runenv.t, with modifications to test cases to run properly
on Win* platforms, and with a new test case to test the new behavior.
---
MANIFEST | 1 +
pod/perldelta.pod | 27 ++++++-
t/win32/runenv.t | 250 +++++++++++++++++++++++++++++++++++++++++++++++++++++
win32/win32.c | 36 +++++++-
4 files changed, 308 insertions(+), 6 deletions(-)
create mode 100644 t/win32/runenv.t
--------------1.7.2.5
Content-Type: text/x-patch; name="0001-Fixes-to-allow-win32-Perl-to-properly-handle-PERL5LI.patch"
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment; filename="0001-Fixes-to-allow-win32-Perl-to-properly-handle-PERL5LI.patch"
diff --git a/MANIFEST b/MANIFEST
index 71ba80e..bab622e 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -5175,6 +5175,7 @@ t/uni/tr_sjis.t See if Unicode tr/// in sjis works
t/uni/tr_utf8.t See if Unicode tr/// in utf8 works
t/uni/upper.t See if Unicode casing works
t/uni/write.t See if Unicode formats work
+t/win32/runenv.t Test if Win* perl honors its env variables
t/win32/system.t See if system works in Win*
t/win32/system_tests Test runner for system.t
t/x2p/s2p.t See if s2p/psed work
diff --git a/pod/perldelta.pod b/pod/perldelta.pod
index ff141fd..a11a8e7 100644
--- a/pod/perldelta.pod
+++ b/pod/perldelta.pod
@@ -526,6 +526,12 @@ and if calling C<dtrace> actually lets you instrument code. This
generally requires being run as root, so this test file is primarily
intended for use by the dtrace subcommittee of p5p.
+=item *
+
+F<t/win32/runenv.t> was added to test aspects of Perl's environment
+variable handling on MSWin32 platforms. Previously, such tests were
+skipped on MSWin32 platforms.
+
=back
=head1 Platform Support
@@ -569,9 +575,26 @@ and compilation changes or changes in portability/compatibility. However,
changes within modules for platforms should generally be listed in the
L</Modules and Pragmata> section.
-=over 4
+=head3 Windows
-=item XXX-some-platform
+=over
+
+=item *
+
+On Windows 7, 2008 and Vista, C<@INC> is now always properly populated
+based on the value of PERL5LIB set in the environment. Previously,
+values of PERL5LIB longer than 32766 bytes were skipped when C<@INC>
+was being populated. Tests for environment handling were
+also added (see L</Testing> section). Fixes
+L<RT #87322|https://rt.perl.org/rt3/Public/Bug/Display.html?id=87322>.
+
+=back
+
+=head3 XXX-some-platform
+
+=over
+
+=item *
XXX
diff --git a/t/win32/runenv.t b/t/win32/runenv.t
new file mode 100644
index 0000000..2576168
--- /dev/null
+++ b/t/win32/runenv.t
@@ -0,0 +1,250 @@
+#!./perl
+#
+# Tests for Perl run-time environment variable settings
+# Clone of t/run/runenv.t but without the forking, and with cmd.exe-friendly -e syntax.
+#
+# $PERL5OPT, $PERL5LIB, etc.
+
+BEGIN {
+ chdir 't' if -d 't';
+ @INC = '../lib';
+ require Config; import Config;
+ require File::Temp; import File::Temp qw/:POSIX/;
+
+ require Win32;
+ ($::os_id, $::os_major) = ( Win32::GetOSVersion() )[ 4, 1 ];
+ if ($::os_id == 2 and $::os_major == 6) { # Vista, Server 2008 (incl R2), 7
+ $::tests = 43;
+ }
+ else {
+ $::tests = 42;
+ }
+
+ require './test.pl';
+}
+
+plan tests => $::tests;
+
+my $PERL = $ENV{PERL} || '.\perl';
+my $NL = $/;
+
+delete $ENV{PERLLIB};
+delete $ENV{PERL5LIB};
+delete $ENV{PERL5OPT};
+
+
+# Run perl with specified environment and arguments, return (STDOUT, STDERR)
+sub runperl_and_capture {
+ my ($env, $args) = @_;
+
+ # Clear out old env
+ local %ENV = %ENV;
+ delete $ENV{PERLLIB};
+ delete $ENV{PERL5LIB};
+ delete $ENV{PERL5OPT};
+
+ # Populate with our desired env
+ for my $k (keys %$env) {
+ $ENV{$k} = $env->{$k};
+ }
+
+ # This is slightly expensive, but this is more reliable than
+ # trying to emulate fork(), and we still get STDERR and STDOUT individually.
+ my $stderr_cache = tmpnam();
+ my $stdout = `$PERL @$args 2>$stderr_cache`;
+ my $stderr = '';
+ if (-s $stderr_cache) {
+ open(my $stderr_cache_fh, "<", $stderr_cache)
+ or die "Could not retrieve STDERR output: $!";
+ while ( defined(my $s_line = <$stderr_cache_fh>) ) {
+ $stderr .= $s_line;
+ }
+ close $stderr_cache_fh;
+ unlink $stderr_cache;
+ }
+
+ return ($stdout, $stderr);
+}
+
+sub try {
+ my ($env, $args, $stdout, $stderr) = @_;
+ my ($actual_stdout, $actual_stderr) = runperl_and_capture($env, $args);
+ local $::Level = $::Level + 1;
+ is ($stdout, $actual_stdout);
+ is ($stderr, $actual_stderr);
+}
+
+# PERL5OPT Command-line options (switches). Switches in
+# this variable are taken as if they were on
+# every Perl command line. Only the -[DIMUdmtw]
+# switches are allowed. When running taint
+# checks (because the program was running setuid
+# or setgid, or the -T switch was used), this
+# variable is ignored. If PERL5OPT begins with
+# -T, tainting will be enabled, and any
+# subsequent options ignored.
+
+try({PERL5OPT => '-w'}, ['-e', '"print $::x"'],
+ "",
+ qq(Name "main::x" used only once: possible typo at -e line 1.${NL}Use of uninitialized value \$x in print at -e line 1.${NL}));
+
+try({PERL5OPT => '-Mstrict'}, ['-I..\lib', '-e', '"print $::x"'],
+ "", "");
+
+try({PERL5OPT => '-Mstrict'}, ['-I..\lib', '-e', '"print $x"'],
+ "",
+ qq(Global symbol "\$x" requires explicit package name at -e line 1.${NL}Execution of -e aborted due to compilation errors.${NL}));
+
+# Fails in 5.6.0
+try({PERL5OPT => '-Mstrict -w'}, ['-I..\lib', '-e', '"print $x"'],
+ "",
+ qq(Global symbol "\$x" requires explicit package name at -e line 1.${NL}Execution of -e aborted due to compilation errors.${NL}));
+
+# Fails in 5.6.0
+try({PERL5OPT => '-w -Mstrict'}, ['-I..\lib', '-e', '"print $::x"'],
+ "",
+ <<ERROR
+Name "main::x" used only once: possible typo at -e line 1.
+Use of uninitialized value \$x in print at -e line 1.
+ERROR
+ );
+
+# Fails in 5.6.0
+try({PERL5OPT => '-w -Mstrict'}, ['-I..\lib', '-e', '"print $::x"'],
+ "",
+ <<ERROR
+Name "main::x" used only once: possible typo at -e line 1.
+Use of uninitialized value \$x in print at -e line 1.
+ERROR
+ );
+
+try({PERL5OPT => '-MExporter'}, ['-I..\lib', '-e0'],
+ "",
+ "");
+
+# Fails in 5.6.0
+try({PERL5OPT => '-MExporter -MExporter'}, ['-I..\lib', '-e0'],
+ "",
+ "");
+
+try({PERL5OPT => '-Mstrict -Mwarnings'},
+ ['-I..\lib', '-e', '"print \"ok\" if $INC{\"strict.pm\"} and $INC{\"warnings.pm\"}"'],
+ "ok",
+ "");
+
+open my $fh, ">", "Oooof.pm" or die "Can't write Oooof.pm: $!";
+print $fh "package Oooof; 1;\n";
+close $fh;
+END { 1 while unlink "Oooof.pm" }
+
+try({PERL5OPT => '-I. -MOooof'},
+ ['-e', '"print \"ok\" if $INC{\"Oooof.pm\"} eq \"Oooof.pm\""'],
+ "ok",
+ "");
+
+try({PERL5OPT => '-w -w'},
+ ['-e', '"print $ENV{PERL5OPT}"'],
+ '-w -w',
+ '');
+
+try({PERL5OPT => '-t'},
+ ['-e', '"print ${^TAINT}"'],
+ '-1',
+ '');
+
+try({PERL5OPT => '-W'},
+ ['-I..\lib','-e', '"local $^W = 0; no warnings; print $x"'],
+ '',
+ <<ERROR
+Name "main::x" used only once: possible typo at -e line 1.
+Use of uninitialized value \$x in print at -e line 1.
+ERROR
+);
+
+try({PERLLIB => "foobar$Config{path_sep}42"},
+ ['-e', '"print grep { $_ eq \"foobar\" } @INC"'],
+ 'foobar',
+ '');
+
+try({PERLLIB => "foobar$Config{path_sep}42"},
+ ['-e', '"print grep { $_ eq \"42\" } @INC"'],
+ '42',
+ '');
+
+try({PERL5LIB => "foobar$Config{path_sep}42"},
+ ['-e', '"print grep { $_ eq \"foobar\" } @INC"'],
+ 'foobar',
+ '');
+
+try({PERL5LIB => "foobar$Config{path_sep}42"},
+ ['-e', '"print grep { $_ eq \"42\" } @INC"'],
+ '42',
+ '');
+
+try({PERL5LIB => "foo",
+ PERLLIB => "bar"},
+ ['-e', '"print grep { $_ eq \"foo\" } @INC"'],
+ 'foo',
+ '');
+
+try({PERL5LIB => "foo",
+ PERLLIB => "bar"},
+ ['-e', '"print grep { $_ eq \"bar\" } @INC"'],
+ '',
+ '');
+
+# Tests for S_incpush_use_sep():
+
+my @dump_inc = ('-e', '"print \"$_\n\" foreach @INC"');
+
+my ($out, $err) = runperl_and_capture({}, [@dump_inc]);
+
+is ($err, '', 'No errors when determining @INC');
+
+my @default_inc = split /\n/, $out;
+
+is ($default_inc[-1], '.', '. is last in @INC');
+
+my $sep = $Config{path_sep};
+my @test_cases = (
+ ['nothing', ''],
+ ['something', 'zwapp', 'zwapp'],
+ ['two things', "zwapp${sep}bam", 'zwapp', 'bam'],
+ ['two things, ::', "zwapp${sep}${sep}bam", 'zwapp', 'bam'],
+ [': at start', "${sep}zwapp", 'zwapp'],
+ [': at end', "zwapp${sep}", 'zwapp'],
+ [':: sandwich ::', "${sep}${sep}zwapp${sep}${sep}", 'zwapp'],
+ [':', "${sep}"],
+ ['::', "${sep}${sep}"],
+ [':::', "${sep}${sep}${sep}"],
+ ['two things and :', "zwapp${sep}bam${sep}", 'zwapp', 'bam'],
+ [': and two things', "${sep}zwapp${sep}bam", 'zwapp', 'bam'],
+ [': two things :', "${sep}zwapp${sep}bam${sep}", 'zwapp', 'bam'],
+ ['three things', "zwapp${sep}bam${sep}${sep}owww",
+ 'zwapp', 'bam', 'owww'],
+);
+
+# This block added to verify fix for RT #87322
+if ($::os_id == 2 and $::os_major == 6) { # Vista, Server 2008 (incl R2), 7
+ my @big_perl5lib = ('z' x 16) x 2049;
+ push @testcases, [
+ 'enough items so PERL5LIB val is longer than 32k',
+ join($sep, @big_perl5lib), @big_perl5lib,
+ ];
+}
+
+foreach ( @testcases ) {
+ my ($name, $lib, @expect) = @$_;
+ push @expect, @default_inc;
+
+ ($out, $err) = runperl_and_capture({PERL5LIB => $lib}, [@dump_inc]);
+
+ is ($err, '', "No errors when determining \@INC for $name");
+
+ my @inc = split /\n/, $out;
+
+ is (scalar @inc, scalar @expect,
+ "expected number of elements in \@INC for $name");
+
+ is ("@inc", "@expect", "expected elements in \@INC for $name");
+}
diff --git a/win32/win32.c b/win32/win32.c
index cffd2b5..e67a735 100644
--- a/win32/win32.c
+++ b/win32/win32.c
@@ -1693,6 +1693,7 @@ win32_getenv(const char *name)
dTHX;
DWORD needlen;
SV *curitem = NULL;
+ DWORD last_err;
needlen = GetEnvironmentVariableA(name,NULL,0);
if (needlen != 0) {
@@ -1705,10 +1706,37 @@ win32_getenv(const char *name)
SvCUR_set(curitem, needlen);
}
else {
- /* allow any environment variables that begin with 'PERL'
- to be stored in the registry */
- if (strncmp(name, "PERL", 4) == 0)
- (void)get_regstr(name, &curitem);
+ last_err = GetLastError();
+ if (last_err == ERROR_NOT_ENOUGH_MEMORY) {
+ /* It appears the variable is in the env, but the Win32 API
+ doesn't have a canned way of getting it. So we fall back to
+ grabbing the whole env and pulling this value out if possible */
+ char *envv = GetEnvironmentStrings();
+ char *cur = envv;
+ STRLEN len;
+ while (*cur) {
+ char *end = strchr(cur,'=');
+ if (end && end != cur) {
+ *end = '\0';
+ if (!strcmp(cur,name)) {
+ curitem = sv_2mortal(newSVpv(end+1,0));
+ *end = '=';
+ break;
+ }
+ *end = '=';
+ cur = end + strlen(end+1)+2;
+ }
+ else if ((len = strlen(cur)))
+ cur += len+1;
+ }
+ FreeEnvironmentStrings(envv);
+ }
+ else {
+ /* last ditch: allow any environment variables that begin with 'PERL'
+ to be obtained from the registry, if found there */
+ if (strncmp(name, "PERL", 4) == 0)
+ (void)get_regstr(name, &curitem);
+ }
}
if (curitem && SvCUR(curitem))
return SvPVX(curitem);
--------------1.7.2.5--
|
From @cpansproutOn Mon Jul 18 20:40:52 2011, icerider wrote:
I can’t really test your patch, but you sound as though you know what |
@cpansprout - Status changed from 'open' to 'resolved' |
From @wchristianOn Tue, 19 Jul 2011 05:40:20 +0200, Philip Monsen <philip.monsen@gmail.com> wrote:
Phil, when i run into you at a YAPC or other, you're getting a few of whatever beverage you prefer. Thanks a lot. :) -- |
Migrated from rt.perl.org#87322 (status was 'resolved')
Searchable as RT87322$
The text was updated successfully, but these errors were encountered: