Skip Menu |
Report information
Id: 96008
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: LeonT <fawaka [at] gmail.com>
Cc:
AdminCc:

Operating System: Linux
PatchStatus: (no value)
Severity: medium
Type: core
Perl Version: 5.14.1
Fixed In: (no value)



Subject: use and require are affected by the open pragma
Date: Sun, 31 Jul 2011 23:05:07 +0200
To: perlbug [...] perl.org
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 3.2k
This is a bug report for perl from fawaka@gmail.com, generated with the help of perlbug 1.39 running under perl 5.14.1. ----------------------------------------------------------------- use and require are affected by the open pragma, as seen here: % perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl % perl '-Mopen=:encoding(utf8)' -e 'require "x.pl"' utf8 "\xF1" does not map to Unicode at x.pl line 1. ----------------------------------------------------------------- --- Flags: category=core severity=medium --- Site configuration information for perl 5.14.1: Configured by leon at Tue Jun 28 15:59:52 CEST 2011. Summary of my perl5 (revision 5 version 14 subversion 1) configuration: Platform: osname=linux, osvers=2.6.38-8-generic, archname=x86_64-linux uname='linux leon-laptop 2.6.38-8-generic #42-ubuntu smp mon apr 11 03:31:24 utc 2011 x86_64 x86_64 x86_64 gnulinux ' config_args='-de -Dprefix=/home/leon/perl5/perlbrew/perls/perl-5.14.1' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.6.1 20110409 (prerelease)', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /usr/lib/x86_64-linux-gnu /lib64 /usr/lib64 libs=-lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.13' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' Locally applied patches: --- @INC for perl 5.14.1: /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/site_perl/5.14.1/x86_64-linux /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/site_perl/5.14.1 /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/5.14.1/x86_64-linux /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/5.14.1 . --- Environment for perl 5.14.1: HOME=/home/leon LANG=en_US.utf8 LANGUAGE=en_US:en LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/leon/perl5/perlbrew/bin:/home/leon/perl5/perlbrew/perls/perl-5.14.1/bin:/home/leon/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games PERLBREW_HOME=/home/leon/.perlbrew PERLBREW_PATH=/home/leon/perl5/perlbrew/bin:/home/leon/perl5/perlbrew/perls/perl-5.14.1/bin PERLBREW_PERL=perl-5.14.1 PERLBREW_ROOT=/home/leon/perl5/perlbrew PERLBREW_VERSION=0.25 PERL_BADLANG (unset) SHELL=/bin/bash
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Mon, 1 Aug 2011 10:27:12 +0100
To: perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 450b
On Sun, Jul 31, 2011 at 02:05:23PM -0700, Leon Timmermans wrote: Show quoted text
> use and require are affected by the open pragma, as seen here: > > % perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl > % perl '-Mopen=:encoding(utf8)' -e 'require "x.pl"' > utf8 "\xF1" does not map to Unicode at x.pl line 1.
Well, the pragma's docs says that it affects "all I/O" within the lexical scope, so surely it's behaving as per docs? -- Thank God I'm an atheist.....
Download (untitled) / with headers
text/plain 720b
On Mon Aug 01 02:27:45 2011, davem wrote: Show quoted text
> On Sun, Jul 31, 2011 at 02:05:23PM -0700, Leon Timmermans wrote:
> > use and require are affected by the open pragma, as seen here: > > > > % perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl > > % perl '-Mopen=:encoding(utf8)' -e 'require "x.pl"' > > utf8 "\xF1" does not map to Unicode at x.pl line 1.
> > Well, the pragma's docs says that it affects "all I/O" within the lexical > scope, so surely it's behaving as per docs?
I really don't think loading a module is can be reasonably called «I/O within the lexical scope». The encoding with which a module is loaded with should not depend on a pragma set by the first module requiring it. That's just plain evil. Leon
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Mon, 1 Aug 2011 15:36:00 +0100
To: perl5-porters [...] perl.org
From: Nicholas Clark <nick [...] ccl4.org>
Download (untitled) / with headers
text/plain 2.1k
On Mon, Aug 01, 2011 at 10:27:12AM +0100, Dave Mitchell wrote: Show quoted text
> On Sun, Jul 31, 2011 at 02:05:23PM -0700, Leon Timmermans wrote:
> > use and require are affected by the open pragma, as seen here: > > > > % perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl > > % perl '-Mopen=:encoding(utf8)' -e 'require "x.pl"' > > utf8 "\xF1" does not map to Unicode at x.pl line 1.
> > Well, the pragma's docs says that it affects "all I/O" within the lexical > scope, so surely it's behaving as per docs?
$ cat >y.pl require "x.pl"; $ ./perl -Ilib -e 'use open ":encoding(utf8)"; require "y.pl"' $ No warning. perlfunc.pod says: Otherwise, C<require> demands that a library file be included if it hasn't already been included. The file is included via the do-FILE mechanism, which is essentially just a variety of C<eval> with the caveat that lexical variables in the invoking script will be invisible to the included code. Has semantics similar to the following subroutine: and makes no comment about lexical pragmata. The y.pl example above shows that the implementation doesn't propagate lexical pragmata *into* the file that it's requiring. So it looks like the current behaviour is "as if" [and probably actually *is* - not groveled too far] that the open for the (internal) do is being treated as being in the caller's lexical scope, before a new scope is set up to actually process the file. All things above being equal, I'm not sure if I think that this is a bug. *But*: require only loads a file once. Hence if you have two modules which require or use the same dependent module, one of which is use open ":encoding(utf8)" require foo; and the other is no open; require foo; then what you actually get in %foo:: is going to depend on the order that the two snippets of code above happen to get wrong. Which is action at a distance. And feels horribly like it will cause much wailing, gnashing of teeth, pain, suffering, lost hours, and caustic blog posts. And caustic blog posts are bad marketing, even if we are so callous as to disregard the waste of the hours. Or at least, that's how somewhat sleep-deprived me sees it. Nicholas Clark
Download (untitled) / with headers
text/plain 796b
On Mon Aug 01 07:36:28 2011, nicholas wrote: Show quoted text
> So it looks like the current behaviour is "as if" [and probably > actually > *is* - not groveled too far] that the open for the (internal) do is > being treated as being in the caller's lexical scope, before a new > scope > is set up to actually process the file.
That's exactly what happens (see pp_require and check_type_and_open in pp_ctl.c). Show quoted text
> then what you actually get in %foo:: is going to depend on the order > that > the two snippets of code above happen to get wrong. > > Which is action at a distance. And feels horribly like it will cause > much > wailing, gnashing of teeth, pain, suffering, lost hours, and caustic > blog > posts.
Yeah, that is another complication of this. The open pragma seems to be collecting issues :-(. Leon
CC: perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Mon, 1 Aug 2011 16:29:40 -0300
To: Nicholas Clark <nick [...] ccl4.org>
From: Brian Fraser <fraserbn [...] gmail.com>
Download open_noprop.diff
text/x-diff 1.9k

Message body is not shown because sender requested not to inline it.

Download (untitled) / with headers
text/plain 3.9k
On Mon, Aug 1, 2011 at 11:36 AM, Nicholas Clark <nick@ccl4.org> wrote:
Show quoted text
$ cat >y.pl
require "x.pl";
$ ./perl -Ilib -e 'use open ":encoding(utf8)"; require "y.pl"'
$

No warning.

perlfunc.pod says:

   Otherwise, C<require> demands that a library file be included if it
   hasn't already been included.  The file is included via the do-FILE
   mechanism, which is essentially just a variety of C<eval> with the
   caveat that lexical variables in the invoking script will be invisible
   to the included code.  Has semantics similar to the following subroutine:


and makes no comment about lexical pragmata. The y.pl example above shows
that the implementation doesn't propagate lexical pragmata *into* the
file that it's requiring.

So it looks like the current behaviour is "as if" [and probably actually
*is* - not groveled too far] that the open for the (internal) do is
being treated as being in the caller's lexical scope, before a new scope
is set up to actually process the file.

All things above being equal, I'm not sure if I think that this is a bug.
*But*:

require only loads a file once. Hence if you have two modules which require
or use the same dependent module, one of which is

   use open ":encoding(utf8)"
   require foo;

and the other is

   no open;
   require foo;

then what you actually get in %foo:: is going to depend on the order that
the two snippets of code above happen to get wrong.

Which is action at a distance. And feels horribly like it will cause much
wailing, gnashing of teeth, pain, suffering, lost hours, and caustic blog
posts.

And caustic blog posts are bad marketing, even if we are so callous as to
disregard the waste of the hours.

Or at least, that's how somewhat sleep-deprived me sees it.

Nicholas Clark

Well, assuming this is considered a bug, here's a patch to resolve it : ) It merely passes ":raw" to PerlIO_openn instead of NULL - The latter which forces the function to use the layers defined by use open, thus causing this entire mess : )

diff --git a/lib/open.t b/lib/open.t
index 1bf7dca..1d61086 100644
--- a/lib/open.t
+++ b/lib/open.t
@@ -186,18 +186,39 @@ SKIP: {
     eval q[use Encode::Alias;use open ":std", ":locale"];
     is($@, '', 'can use :std and :locale');
 
-    use open IN => ':non-existent';
-    eval {
-    require Symbol; # Anything that exists but we havn't loaded
-    };
-    like($@, qr/Can't locate Symbol|Recursive call/i,
-     "test for an endless loop in PerlIO_find_layer");
+#This test was using dubious behaviour - It relied on the open pragma to propagate
+#it's effect into the layer used for require, which is no longer true.
+#See the next test as for why this was changed.
+
+#    use open IN => ':non-existent';
+#    eval {
+#    require Symbol; # Anything that exists but we havn't loaded
+#    };
+#    like($@, qr/Can't locate Symbol|Recursive call/i,
+#     "test for an endless loop in PerlIO_find_layer");
+
+
+    open my $out_fh, ">:raw", "x.pl";
+    print { $out_fh } "package F;\n # \xF1\n;1;";
+    close $out_fh;
+
+#This require would've thrown "utf8 "\xF1" does not map to Unicode" if require
+#was affected by open.
+    {
+        use open qw( :encoding(utf8) );
+            eval {
+                local @INC = ".";
+                require "x.pl";
+            };
+        is $@, '', "open does not propagate into require()";
+    }
 }
 
 END {
     1 while unlink "utf8";
     1 while unlink "a";
     1 while unlink "b";
+    1 while unlink "x.pl";
 }
 
 # the test cases beyond __DATA__ need to be executed separately
diff --git a/pp_ctl.c b/pp_ctl.c
index d5924d8..7d5e58a 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@@ -3649,7 +3649,7 @@ S_check_type_and_open(pTHX_ SV *name)
     }
 
 #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
-    return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
+    return PerlIO_openn(aTHX_ ":raw", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
 #else
     return PerlIO_open(p, PERL_SCRIPT_MODE);
 #endif

CC: perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Mon, 01 Aug 2011 13:59:16 -0600
To: Nicholas Clark <nick [...] ccl4.org>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 285b
Just because you call a function that does I/O doesn't mean that that function should inherit its caller's I/O layers. That requires dynamic not lexical scoping. do provides a separate lexical scope. Therefore, so does require. A different question is $foo = `cat $file`. --tom
CC: Nicholas Clark <nick [...] ccl4.org>, perl5-porters [...] perl.org, bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Tue, 2 Aug 2011 11:46:22 +0200
To: Brian Fraser <fraserbn [...] gmail.com>
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 550b
On Mon, Aug 1, 2011 at 9:29 PM, Brian Fraser <fraserbn@gmail.com> wrote: Show quoted text
> On Mon, Aug 1, 2011 at 11:36 AM, Nicholas Clark <nick@ccl4.org> wrote: > Well, assuming this is considered a bug, here's a patch to resolve it : ) It > merely passes ":raw" to PerlIO_openn instead of NULL - The latter which > forces the function to use the layers defined by use open, thus causing this > entire mess : )
Perl modules/scripts are text files, not binary files. I don't think ":raw" really is the correct thing to do, though it may work out in the end. Leon
CC: perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Tue, 2 Aug 2011 10:54:51 +0100
To: Brian Fraser <fraserbn [...] gmail.com>
From: Nicholas Clark <nick [...] ccl4.org>
Download (untitled) / with headers
text/plain 1.2k
On Mon, Aug 01, 2011 at 04:29:40PM -0300, Brian Fraser wrote: Show quoted text
> Well, assuming this is considered a bug, here's a patch to resolve it : ) It > merely passes ":raw" to PerlIO_openn instead of NULL - The latter which > forces the function to use the layers defined by use open, thus causing this > entire mess : )
Except I *think* that the call: Show quoted text
> diff --git a/pp_ctl.c b/pp_ctl.c > index d5924d8..7d5e58a 100644 > --- a/pp_ctl.c > +++ b/pp_ctl.c > @@ -3649,7 +3649,7 @@ S_check_type_and_open(pTHX_ SV *name) > } > > #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO) > - return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name); > + return PerlIO_openn(aTHX_ ":raw", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name); > #else > return PerlIO_open(p, PERL_SCRIPT_MODE); > #endif
now opens the file then pushes "raw", and pushing "raw" does binmode() on the file handle, which isn't what's desired on Win32. I'm not sure how to call PerlIO in such a way as to override the open pragma, but also honour the passed in mode. PerlIO is not my favourite part of the core Perl source code, and hasn't been since it was *written*, given that it was initially written without consideration for threads. You would have thought that that mistake would have already been learned. Nicholas Clark
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 527b
On Tue Aug 02 02:55:21 2011, nicholas wrote: Show quoted text
> I'm not sure how to call PerlIO in such a way as to override the open > pragma, > but also honour the passed in mode.
I think we should add a :default pseudo-layer. It should be fairly trivial to implement in core, though I don't think the API allows it do be done in a CPAN module. Show quoted text
> PerlIO is not my favourite part of the core Perl source code
It's nobody's favorite part of the core, which is its main problem. Without getting some love it will stay the way it is now. Leon
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.4k
On Tue Aug 02 02:55:21 2011, nicholas wrote: Show quoted text
> On Mon, Aug 01, 2011 at 04:29:40PM -0300, Brian Fraser wrote: >
> > Well, assuming this is considered a bug, here's a patch to resolve
> it : ) It
> > merely passes ":raw" to PerlIO_openn instead of NULL - The latter
> which
> > forces the function to use the layers defined by use open, thus
> causing this
> > entire mess : )
> > Except I *think* that the call: >
> > diff --git a/pp_ctl.c b/pp_ctl.c > > index d5924d8..7d5e58a 100644 > > --- a/pp_ctl.c > > +++ b/pp_ctl.c > > @@ -3649,7 +3649,7 @@ S_check_type_and_open(pTHX_ SV *name) > > } > > > > #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO) > > - return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0,
> NULL, 1, &name);
> > + return PerlIO_openn(aTHX_ ":raw", PERL_SCRIPT_MODE, -1, 0, 0,
> NULL, 1, &name);
> > #else > > return PerlIO_open(p, PERL_SCRIPT_MODE); > > #endif
> > now opens the file then pushes "raw", and pushing "raw" does binmode() > on the > file handle, which isn't what's desired on Win32. > > I'm not sure how to call PerlIO in such a way as to override the open > pragma, > but also honour the passed in mode.
How about temporarily clearing the hints (after using a SAVE* macro)? Show quoted text
> > PerlIO is not my favourite part of the core Perl source code, and > hasn't been > since it was *written*, given that it was initially written without > consideration for threads. You would have thought that that mistake > would > have already been learned. > > Nicholas Clark >
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.6k
On Tue Aug 02 06:21:58 2011, sprout wrote: Show quoted text
> On Tue Aug 02 02:55:21 2011, nicholas wrote:
> > On Mon, Aug 01, 2011 at 04:29:40PM -0300, Brian Fraser wrote: > >
> > > Well, assuming this is considered a bug, here's a patch to resolve
> > it : ) It
> > > merely passes ":raw" to PerlIO_openn instead of NULL - The latter
> > which
> > > forces the function to use the layers defined by use open, thus
> > causing this
> > > entire mess : )
> > > > Except I *think* that the call: > >
> > > diff --git a/pp_ctl.c b/pp_ctl.c > > > index d5924d8..7d5e58a 100644 > > > --- a/pp_ctl.c > > > +++ b/pp_ctl.c > > > @@ -3649,7 +3649,7 @@ S_check_type_and_open(pTHX_ SV *name) > > > } > > > > > > #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO) > > > - return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0,
> > NULL, 1, &name);
> > > + return PerlIO_openn(aTHX_ ":raw", PERL_SCRIPT_MODE, -1, 0, 0,
> > NULL, 1, &name);
> > > #else > > > return PerlIO_open(p, PERL_SCRIPT_MODE); > > > #endif
> > > > now opens the file then pushes "raw", and pushing "raw" does binmode() > > on the > > file handle, which isn't what's desired on Win32. > > > > I'm not sure how to call PerlIO in such a way as to override the open > > pragma, > > but also honour the passed in mode.
> > How about temporarily clearing the hints (after using a SAVE* macro)?
Or pushing the new scope before opening the file? Show quoted text
>
> > > > PerlIO is not my favourite part of the core Perl source code, and > > hasn't been > > since it was *written*, given that it was initially written without > > consideration for threads. You would have thought that that mistake > > would > > have already been learned. > > > > Nicholas Clark > >
> >
CC: "'Nicholas Clark'" <nick [...] ccl4.org>, <perl5-porters [...] perl.org>, <bugs-bitbucket [...] rt.perl.org>
Subject: RE: [perl #96008] use and require are affected by the open pragma
Date: Tue, 2 Aug 2011 11:36:47 -0700
To: "'Leon Timmermans'" <fawaka [...] gmail.com>, "'Brian Fraser'" <fraserbn [...] gmail.com>
From: "Jan Dubois" <jand [...] activestate.com>
Download (untitled) / with headers
text/plain 1.2k
On Tue, 02 Aug 2011, Leon Timmermans wrote: Show quoted text
> On Mon, Aug 1, 2011 at 9:29 PM, Brian Fraser <fraserbn@gmail.com> wrote:
> > On Mon, Aug 1, 2011 at 11:36 AM, Nicholas Clark <nick@ccl4.org> wrote: > > Well, assuming this is considered a bug, here's a patch to resolve it : ) It > > merely passes ":raw" to PerlIO_openn instead of NULL - The latter which > > forces the function to use the layers defined by use open, thus causing this > > entire mess : )
> > Perl modules/scripts are text files, not binary files. I don't think > ":raw" really is the correct thing to do, though it may work out in > the end.
No, it won't. It will switch the DATA filehandle to :raw mode as well, so you'll end up with "\r\n" line endings on Windows. Perl actually used to read the source in :raw mode on Windows, but (incompletely) turned on text mode when it reached the __DATA__ token. This created quite a number of bugs, all merged into this ticket: https://rt.perl.org/rt3/Public/Bug/Display.html?id=28106 For Perl 5.14 I changed the default to reading source code in text mode, which fixed all those bugs. I'll soon remove the broken code that unsuccessfully tries to switch on text-mode mid-stream because it appears that nobody knows how to fix it (and it is unrelated to the open pragma anyways). Cheers, -Jan
CC: Leon Timmermans <fawaka [...] gmail.com>, Nicholas Clark <nick [...] ccl4.org>, perl5-porters [...] perl.org, bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Thu, 4 Aug 2011 05:11:13 -0300
To: Jan Dubois <jand [...] activestate.com>
From: Brian Fraser <fraserbn [...] gmail.com>
Download (untitled) / with headers
text/plain 1.3k
On Tue, Aug 2, 2011 at 3:36 PM, Jan Dubois <jand@activestate.com> wrote: Show quoted text
> On Tue, 02 Aug 2011, Leon Timmermans wrote:
> > Perl modules/scripts are text files, not binary files. I don't think > > ":raw" really is the correct thing to do, though it may work out in > > the end.
> > No, it won't. It will switch the DATA filehandle to :raw mode as well, > so you'll end up with "\r\n" line endings on Windows. > > Perl actually used to read the source in :raw mode on Windows, but > (incompletely) turned on text mode when it reached the __DATA__ token. > This created quite a number of bugs, all merged into this ticket: > > https://rt.perl.org/rt3/Public/Bug/Display.html?id=28106 > > For Perl 5.14 I changed the default to reading source code in text mode, > which fixed all those bugs. I'll soon remove the broken code that > unsuccessfully tries to switch on text-mode mid-stream because it > appears that nobody knows how to fix it (and it is unrelated to the open > pragma anyways). > > Cheers, > -Jan > > >
Alright, then I think that Father C's suggestion is our winner here. Unfortunately I can't get SAVEHINTS(); to play nice (my ignorance of PerlIO is only surpassed by my inexperience with scope.c), so I can't volunteer myself for this one. But if it helps, from my hamhanded testing, all you need to do is clear PL_curcop->cop_hints of HINT_LEXICAL_IO_(IN|OUT) before calling PerlIO_openn.
CC: perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Sun, 7 Aug 2011 23:42:19 +0200
To: perlbug-followup [...] perl.org
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 494b
On Tue, Aug 2, 2011 at 2:47 PM, Leon Timmermans via RT <perlbug-followup@perl.org> wrote: Show quoted text
> On Tue Aug 02 02:55:21 2011, nicholas wrote:
>> I'm not sure how to call PerlIO in such a way as to override the open >> pragma, >> but also honour the passed in mode.
> > I think we should add a :default pseudo-layer. It should be fairly > trivial to implement in core, though I don't think the API allows it do > be done in a CPAN module.
Attached the patch. It turned out to be quite simple. Leon

Message body is not shown because sender requested not to inline it.

CC: perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Sun, 21 Aug 2011 01:37:14 -0300
To: Leon Timmermans <fawaka [...] gmail.com>
From: Brian Fraser <fraserbn [...] gmail.com>
Download (untitled) / with headers
text/plain 188b
On Sun, Aug 7, 2011 at 6:42 PM, Leon Timmermans <fawaka@gmail.com> wrote: Show quoted text
> Attached the patch. It turned out to be quite simple. > >
With 5.15.2 out of the door, can this get committed?
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 359b
On Sat Aug 20 21:37:43 2011, fraserbn@gmail.com wrote: Show quoted text
> On Sun, Aug 7, 2011 at 6:42 PM, Leon Timmermans <fawaka@gmail.com> wrote: >
> > Attached the patch. It turned out to be quite simple. > > > >
> With 5.15.2 out of the door, can this get committed?
If this adds a new pseudo-layer accessible from Perl space, then it needs documentation, doesn’t it?
CC: perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Sun, 21 Aug 2011 04:52:03 -0300
To: perlbug-followup [...] perl.org
From: Brian Fraser <fraserbn [...] gmail.com>
Download (untitled) / with headers
text/plain 901b
On Sun, Aug 21, 2011 at 3:37 AM, Father Chrysostomos via RT < perlbug-followup@perl.org> wrote: Show quoted text
> > If this adds a new pseudo-layer accessible from Perl space, then it > needs documentation, doesn’t it? >
Would something like this do? diff --git a/lib/PerlIO.pm b/lib/PerlIO.pm index 46e6e44..2c36d72 100644 --- a/lib/PerlIO.pm +++ b/lib/PerlIO.pm @@ -190,6 +190,15 @@ On Win32 platforms this I<experimental> layer uses the nati rather than the unix-like numeric file descriptor layer. Known to be buggy as of perl 5.8.2. +=item :default + +When opening a stream without manually setting a layer, Perl implicitly +uses a combination of several layers, which are dependent on build +options and OS. +As this behavior can be augmented (through the C<open> pragma, or +by modifying C<${^OPEN}>), you can use this layer to ignore that and +use the default instead. + =back =head2 Custom Layers
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Sun, 21 Aug 2011 15:52:53 +0200
To: perlbug-followup [...] perl.org
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 994b
On Sun, Aug 21, 2011 at 8:37 AM, Father Chrysostomos via RT <perlbug-followup@perl.org> wrote: Show quoted text
> On Sat Aug 20 21:37:43 2011, fraserbn@gmail.com wrote:
>> On Sun, Aug 7, 2011 at 6:42 PM, Leon Timmermans <fawaka@gmail.com> wrote: >>
>> > Attached the patch. It turned out to be quite simple. >> > >> >
>> With 5.15.2 out of the door, can this get committed?
> > If this adds a new pseudo-layer accessible from Perl space, then it > needs documentation, doesn’t it?
At least as important, it needs tests. I wrote some, but I bumped into some rather peculiar bugs in PerlIO that prevent it from working as expected. It does exactly the same as ":perlio" or ":crlf" (depending on platform) but the former subtly does the wrong thing: adding a second layer of buffering. In fact, in some cases it's even possible to end up with three(!) layers of buffering. Haven't had time to properly perlbug them though, will do so soon. I think it'd be better to fix that first before applying this. Leon
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.7k
On Thu Aug 04 01:11:45 2011, fraserbn@gmail.com wrote: Show quoted text
> On Tue, Aug 2, 2011 at 3:36 PM, Jan Dubois <jand@activestate.com> wrote: >
> > On Tue, 02 Aug 2011, Leon Timmermans wrote:
> > > Perl modules/scripts are text files, not binary files. I don't think > > > ":raw" really is the correct thing to do, though it may work out in > > > the end.
> > > > No, it won't. It will switch the DATA filehandle to :raw mode as well, > > so you'll end up with "\r\n" line endings on Windows. > > > > Perl actually used to read the source in :raw mode on Windows, but > > (incompletely) turned on text mode when it reached the __DATA__ token. > > This created quite a number of bugs, all merged into this ticket: > > > > https://rt.perl.org/rt3/Public/Bug/Display.html?id=28106 > > > > For Perl 5.14 I changed the default to reading source code in text mode, > > which fixed all those bugs. I'll soon remove the broken code that > > unsuccessfully tries to switch on text-mode mid-stream because it > > appears that nobody knows how to fix it (and it is unrelated to the open > > pragma anyways). > > > > Cheers, > > -Jan > > > > > >
> Alright, then I think that Father C's suggestion is our winner here.
Except it can’t work properly, because SAVEHINTS and PL_hints deal with *compile-time* hints, whereas PerlIO looks at the hints in the already-compiled op tree that is running (in PL_curcop, in other words). PL_curcop->cop_hints can’t be modified, because that won’t be thread-safe. Show quoted text
> > Unfortunately I can't get SAVEHINTS(); to play nice (my ignorance of
PerlIO Show quoted text
> is only surpassed by my inexperience with scope.c), so I can't volunteer > myself for this one. But if it helps, from my hamhanded testing, all you > need to do is clear PL_curcop->cop_hints of HINT_LEXICAL_IO_(IN|OUT)
before Show quoted text
> calling PerlIO_openn.
CC: perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Thu, 22 Sep 2011 11:29:25 -0300
To: perlbug-followup [...] perl.org
From: Brian Fraser <fraserbn [...] gmail.com>
Download (untitled) / with headers
text/plain 2.8k
On Mon, Aug 22, 2011 at 6:34 PM, Father Chrysostomos via RT <perlbug-followup@perl.org> wrote:
Show quoted text
On Thu Aug 04 01:11:45 2011, fraserbn@gmail.com wrote:
> On Tue, Aug 2, 2011 at 3:36 PM, Jan Dubois <jand@activestate.com> wrote:
>
> > On Tue, 02 Aug 2011, Leon Timmermans wrote:
> > > Perl modules/scripts are text files, not binary files. I don't think
> > > ":raw" really is the correct thing to do, though it may work out in
> > > the end.
> >
> > No, it won't. It will switch the DATA filehandle to :raw mode as well,
> > so you'll end up with "\r\n" line endings on Windows.
> >
> > Perl actually used to read the source in :raw mode on Windows, but
> > (incompletely) turned on text mode when it reached the __DATA__ token.
> > This created quite a number of bugs, all merged into this ticket:
> >
> > https://rt.perl.org/rt3/Public/Bug/Display.html?id=28106
> >
> > For Perl 5.14 I changed the default to reading source code in text mode,
> > which fixed all those bugs. I'll soon remove the broken code that
> > unsuccessfully tries to switch on text-mode mid-stream because it
> > appears that nobody knows how to fix it (and it is unrelated to the open
> > pragma anyways).
> >
> > Cheers,
> > -Jan
> >
> >
> >
> Alright, then I think that Father C's suggestion is our winner here.

Except it can’t work properly, because SAVEHINTS and PL_hints deal with
*compile-time* hints, whereas PerlIO looks at the hints in the
already-compiled op tree that is running (in PL_curcop, in other words).

PL_curcop->cop_hints can’t be modified, because that won’t be thread-safe.

>
> Unfortunately I can't get SAVEHINTS(); to play nice (my ignorance of
PerlIO
> is only surpassed by my inexperience with scope.c), so I can't volunteer
> myself for this one. But if it helps, from my hamhanded testing, all you
> need to do is clear PL_curcop->cop_hints of HINT_LEXICAL_IO_(IN|OUT)
before
> calling PerlIO_openn.




Bit of a surprising discovery; Turns out that passing a single ":" for the layers skips the fetch from the context layers:
perl -wE 'use open qw( :encoding(UTF-8) ); open my $fh, "<:", "etc"; say PerlIO::get_layers($fh);'

That will only get the relevant default layers, while removing the colons makes it work as usual -- So we can abuse this (mis)feature to fix the bug:

diff --git a/pp_ctl.c b/pp_ctl.c
index 3a2de27..a9c0afd 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@@ -3692,7 +3692,7 @@ S_check_type_and_open(pTHX_ SV *name)
}

#if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
- return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
+ return PerlIO_openn(aTHX_ ":", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
#else
return PerlIO_open(p, PERL_SCRIPT_MODE);
#endif

(tests in a previous mail)

Is this worth documenting somewhere?
CC: perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Thu, 22 Sep 2011 16:35:40 +0200
To: Brian Fraser <fraserbn [...] gmail.com>
From: Rafael Garcia-Suarez <rgs [...] consttype.org>
On 22 September 2011 16:29, Brian Fraser <fraserbn@gmail.com> wrote: Show quoted text
> Bit of a surprising discovery; Turns out that passing a single ":" for the > layers skips the fetch from the context layers: > perl -wE 'use open qw( :encoding(UTF-8) ); open my $fh, "<:", "etc"; say > PerlIO::get_layers($fh);' > That will only get the relevant default layers, while removing the colons > makes it work as usual -- So we can abuse this (mis)feature to fix the bug: > diff --git a/pp_ctl.c b/pp_ctl.c > index 3a2de27..a9c0afd 100644 > --- a/pp_ctl.c > +++ b/pp_ctl.c > @@ -3692,7 +3692,7 @@ S_check_type_and_open(pTHX_ SV *name) > } > > #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO) > - return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, > &name); > + return PerlIO_openn(aTHX_ ":", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, > &name); > #else > return PerlIO_open(p, PERL_SCRIPT_MODE); > #endif > (tests in a previous mail) > Is this worth documenting somewhere?
I vote yes, because I find this accidental feature actually neat. And doubly so if we use it in the core...
CC: Brian Fraser <fraserbn [...] gmail.com>, perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Thu, 22 Sep 2011 09:09:18 -0600
To: Rafael Garcia-Suarez <rgs [...] consttype.org>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 609b
Rafael Garcia-Suarez <rgs@consttype.org> wrote on Thu, 22 Sep 2011 16:35:40 +0200: Show quoted text
> I vote yes, because I find this accidental feature actually neat. And > doubly so if we use it in the core...
My instinct says that you're right. It is always cool to find new uses for our tools that their inventor never dreamt of. I *REALLY* wish I understood layers better. I can do simple stuff, but not fancy bits, and there are some things for which fanciness is still called for. Is there any other/better documentation of all this than what's in PerlIO(3) and the tiny bits in perlfunc? thanks, --tom
CC: Brian Fraser <fraserbn [...] gmail.com>, perl5-porters [...] perl.org
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Thu, 22 Sep 2011 17:23:13 +0200
To: Tom Christiansen <tchrist [...] perl.com>
From: Rafael Garcia-Suarez <rgs [...] consttype.org>
Download (untitled) / with headers
text/plain 392b
On 22 September 2011 17:09, Tom Christiansen <tchrist@perl.com> wrote: Show quoted text
> I *REALLY* wish I understood layers better.  I can do simple stuff, > but not fancy bits, and there are some things for which fanciness is > still called for.  Is there any other/better documentation of all this > than what's in PerlIO(3) and the tiny bits in perlfunc?
There's the scary low-level docs in perliol.
RT-Send-CC: perl5-porters [...] perl.org, fraserbn [...] gmail.com
Download (untitled) / with headers
text/plain 1.2k
On Thu Sep 22 07:30:00 2011, fraserbn@gmail.com wrote: Show quoted text
> Bit of a surprising discovery; Turns out that passing a single ":" for the > layers skips the fetch from the context layers: > perl -wE 'use open qw( :encoding(UTF-8) ); open my $fh, "<:", "etc"; say > PerlIO::get_layers($fh);' > > That will only get the relevant default layers, while removing the colons > makes it work as usual -- So we can abuse this (mis)feature to fix the
bug: Show quoted text
> > diff --git a/pp_ctl.c b/pp_ctl.c > index 3a2de27..a9c0afd 100644 > --- a/pp_ctl.c > +++ b/pp_ctl.c > @@ -3692,7 +3692,7 @@ S_check_type_and_open(pTHX_ SV *name) > } > > #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO) > - return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, > &name); > + return PerlIO_openn(aTHX_ ":", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, > &name); > #else > return PerlIO_open(p, PERL_SCRIPT_MODE); > #endif
Thank you. Applied as 639dfab029. Show quoted text
> > (tests in a previous mail)
I didn’t read that sentence until I had already written one. It turns out that your test, however, passes even without the bug fix, because the error you were testing for is actually a warning. So I went with my test (commit 6406c527c743), as it was easier than fixing yours. :-)
RT-Send-CC: perl5-porters [...] perl.org, fraserbn [...] gmail.com
Download (untitled) / with headers
text/plain 146b
On Thu Sep 22 07:30:00 2011, fraserbn@gmail.com wrote: Show quoted text
> Is this worth documenting somewhere?
Yes, and I have documented it with commit c0fd9d2.
Subject: Re: [perl #96008] use and require are affected by the open pragma
Date: Fri, 23 Sep 2011 16:12:34 +0200
To: Tom Christiansen <tchrist [...] perl.com>, Perl5 Porters <perl5-porters [...] perl.org>
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 671b
On Thu, Sep 22, 2011 at 5:09 PM, Tom Christiansen <tchrist@perl.com> wrote: Show quoted text
> I *REALLY* wish I understood layers better.  I can do simple stuff, > but not fancy bits, and there are some things for which fanciness is > still called for.  Is there any other/better documentation of likeall this > than what's in PerlIO(3) and the tiny bits in perlfunc?
Barely. perliol defines some internals but that's not useful for anyone who isn't implementing a layer in C (which very few people do, and even fewer do it correctly). Most of it is reading the source and trying things out. That's also why hardly anyone ever uses them, other than the unavoidable :encoding. Leon


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org