Skip Menu |
Report information
Id: 38181
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: nikratio <Nikolaus [at] rath.org>
Cc:
AdminCc:

Operating System: Linux
PatchStatus: (no value)
Severity: low
Type: core
Perl Version: 5.8.7
Fixed In: (no value)



Subject: Default IO Layer is ignored in 3-argument open call
Date: Sat, 7 Jan 2006 18:48:09 +0100 (CET)
To: perlbug [...] perl.org
From: nikratio [...] nokile.rath.org (Nikolaus Rath)
Download (untitled) / with headers
text/plain 4.4k
This is a bug report for perl from Nikolaus@rath.org, generated with the help of perlbug 1.35 running under perl v5.8.7. ----------------------------------------------------------------- Consider the following program: #!/usr/bin/perl -CSDA use strict; use warnings; use utf8; my($fh,$buf); open($fh, "<", "utf8.txt"); #open($fh, "<utf8.txt"); #open($fh, "<:utf8", "utf8.txt"); $buf = <$fh>; print "'$buf'"; print "(is utf8)\n" if utf8::is_utf8($buf); exit; Where utf8.txt contains some utf8 encoded characters. The first open call doesn't work, $buf has no utf8 flag and the output is garbled. The two commented open calls work. Citing perldoc -f open: will open the UTF-8 encoded file containing Unicode characters, see perluniintro. (Note that if layers are specified in the three-arg form then default layers set by the "open" pragma are ignored.) Clearly I didn't specify an IO-layer in my first open call, yet the default layer was ignored. [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl v5.8.7: Configured by Debian Project at Mon Dec 12 13:59:01 UTC 2005. Summary of my perl5 (revision 5 version 8 subversion 7) configuration: Platform: osname=linux, osvers=2.6.10, archname=i486-linux-gnu-thread-multi uname='linux rothera 2.6.10 #1 smp fri may 13 09:24:22 utc 2005 i686 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.7 -Dsitearch=/usr/local/lib/perl/5.8.7 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.7 -Dd_dosuid -des' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu9)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=/lib/libc-2.3.5.so, so=so, useshrplib=true, libperl=libperl.so.5.8.7 gnulibc_version='2.3.5' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib' Locally applied patches: SPRINTF0 - fixes for sprintf formatting issues - CVE-2005-3962 --- @INC for perl v5.8.7: /etc/perl /usr/local/lib/perl/5.8.7 /usr/local/share/perl/5.8.7 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . --- Environment for perl v5.8.7: HOME=/home/nikratio LANG=en_US.UTF-8 LANGUAGE (unset) LC_ADDRESS=de_DE.UTF-8 LC_COLLATE=de_DE.UTF-8 LC_CTYPE=de_DE.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_MEASUREMENT=de_DE.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_MONETARY=de_DE.UTF-8 LC_NAME=de_DE.UTF-8 LC_NUMERIC=de_DE.UTF-8 LC_PAPER=de_DE.UTF-8 LC_TELEPHONE=de_DE.UTF-8 LC_TIME=de_DE.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=~/bin:~/Projekte/Scripts:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11 PERL_BADLANG (unset) PERL_UNICODE=SDAL SHELL=/bin/bash
Subject: Re: [perl #38181] Default IO Layer is ignored in 3-argument open call
Date: Sun, 08 Jan 2006 22:22:25 +0900
To: perl5-porters [...] perl.org
From: SADAHIRO Tomoyuki <bqw10602 [...] nifty.com>
Download (untitled) / with headers
text/plain 1.7k
Show quoted text
> This is a bug report for perl from Nikolaus@rath.org, > generated with the help of perlbug 1.35 running under perl v5.8.7. > > > ----------------------------------------------------------------- > Consider the following program: > > > #!/usr/bin/perl -CSDA > use strict; > use warnings; > use utf8; > my($fh,$buf); > open($fh, "<", "utf8.txt"); > #open($fh, "<utf8.txt"); > #open($fh, "<:utf8", "utf8.txt"); > $buf = <$fh>; > print "'$buf'"; > print "(is utf8)\n" if utf8::is_utf8($buf); > exit; > > Where utf8.txt contains some utf8 encoded characters. The first open > call doesn't work, $buf has no utf8 flag and the output is garbled. The > two commented open calls work. > > Citing perldoc -f open: > > will open the UTF-8 encoded file containing Unicode > characters, see perluniintro. (Note that if layers are > specified in the three-arg form then default layers set > by the "open" pragma are ignored.) > > Clearly I didn't specify an IO-layer in my first open call, yet the > default layer was ignored. >
Thank you for report. But "perl-current" (under development) works even if open($fh, "<", "utf8.txt") as well as other two calls. This problem seems to have been recently resolved: Show quoted text
____________________________________________________________________________ [ 25925] By: rgs on 2005/10/31 14:27:32 Log: Fix [perl #37533] open pragma ignored for all 3-arg opens Also, correct the documentation in open.pm that stated that open() did not override open.pm-given default encodings. Branch: perl ! lib/open.pm perlio.c
____________________________________________________________________________ Regards, SADAHIRO Tomoyuki
Subject: Re: [perl #38181] Default IO Layer is ignored in 3-argument open call
Date: Mon, 09 Jan 2006 01:03:48 +0900
To: perl5-porters [...] perl.org
From: SADAHIRO Tomoyuki <bqw10602 [...] nifty.com>
Download (untitled) / with headers
text/plain 2.8k
Show quoted text
> > #!/usr/bin/perl -CSDA
(snip) Show quoted text
> > open($fh, "<", "utf8.txt");
(snip) Show quoted text
> > Citing perldoc -f open: > > > > will open the UTF-8 encoded file containing Unicode > > characters, see perluniintro. (Note that if layers are > > specified in the three-arg form then default layers set > > by the "open" pragma are ignored.) > >
In this case the open pragma is not loaded but the -C switch is set. Nevertheless the part of perlfunc does not mention -C. Here is a doc patch (against perl@26695) for open.pm and perlfunc.pod P.S. In perlfunc.pod, some C<open> refer to the pragma and others refer to the function, while pod2html makes a link to either. (actually all the tags are linked to the function). In the patch, B<open> is used for the pragma (but I'm not sure B<> is proper). Regards, SADAHIRO Tomoyuki diff -ur perl~/lib/open.pm perl/lib/open.pm --- perl~/lib/open.pm Mon Oct 31 23:56:43 2005 +++ perl/lib/open.pm Sun Jan 08 23:25:39 2006 @@ -169,9 +169,8 @@ "layers" (also known as "disciplines") for all I/O. Any two-argument open(), readpipe() (aka qx//) and similar operators found within the lexical scope of this pragma will use the declared defaults. -Three-argument opens are not affected by this pragma since there you -(can) explicitly specify the layers and are supposed to know what you -are doing. +Even three-argument opens may be affected by this pragma +when they don't specify IO layers in MODE. With the C<IN> subpragma you can declare the default layers of input streams, and with the C<OUT> subpragma you can declare @@ -180,7 +179,7 @@ If you have a legacy encoding, you can use the C<:encoding(...)> tag. -if you want to set your encoding layers based on your +If you want to set your encoding layers based on your locale environment variables, you can use the C<:locale> tag. For example: diff -ur perl~/pod/perlfunc.pod perl/pod/perlfunc.pod --- perl~/pod/perlfunc.pod Mon Jan 02 23:27:04 2006 +++ perl/pod/perlfunc.pod Mon Jan 09 00:21:37 2006 @@ -3103,14 +3103,14 @@ You may use the three-argument form of open to specify IO "layers" (sometimes also referred to as "disciplines") to be applied to the handle that affect how the input and output are processed (see L<open> and -L<PerlIO> for more details). For example +L<Encode::PerlIO> for more details). For example open(FH, "<:utf8", "file") will open the UTF-8 encoded file containing Unicode characters, -see L<perluniintro>. (Note that if layers are specified in the -three-arg form then default layers set by the C<open> pragma are -ignored.) +see L<perluniintro>. Note that if layers are specified in the +three-arg form then default layers stored in ${^OPEN} (see L<perlvar>; +usually set by the B<open> pragma or the switch B<-CioD>) are ignored. Open returns nonzero upon success, the undefined value otherwise. If the C<open> involved a pipe, the return value happens to be the pid of
Subject: Re: [perl #38181] Default IO Layer is ignored in 3-argument open call
Date: Mon, 09 Jan 2006 01:29:30 +0900
To: perl5-porters [...] perl.org
From: SADAHIRO Tomoyuki <bqw10602 [...] nifty.com>
Download (untitled) / with headers
text/plain 2.1k
A reply to myself... Show quoted text
> -L<PerlIO> for more details). For example > +L<Encode::PerlIO> for more details). For example
Sorry this change is wrong. The reference here should be PerlIO.pm (Thus no change is need here). Revised patch is below. P.S. Why does a link to Encode/PerlIO.pm is produced from L<PerlIO> on conversion from POD to HTML? I don't know.... Regards, SADAHIRO Tomoyuki diff -ur perl~/lib/open.pm perl/lib/open.pm --- perl~/lib/open.pm Mon Oct 31 23:56:43 2005 +++ perl/lib/open.pm Mon Jan 09 00:38:52 2006 @@ -169,9 +169,8 @@ "layers" (also known as "disciplines") for all I/O. Any two-argument open(), readpipe() (aka qx//) and similar operators found within the lexical scope of this pragma will use the declared defaults. -Three-argument opens are not affected by this pragma since there you -(can) explicitly specify the layers and are supposed to know what you -are doing. +Even three-argument opens may be affected by this pragma +when they don't specify IO layers in MODE. With the C<IN> subpragma you can declare the default layers of input streams, and with the C<OUT> subpragma you can declare @@ -180,7 +179,7 @@ If you have a legacy encoding, you can use the C<:encoding(...)> tag. -if you want to set your encoding layers based on your +If you want to set your encoding layers based on your locale environment variables, you can use the C<:locale> tag. For example: diff -ur perl~/pod/perlfunc.pod perl/pod/perlfunc.pod --- perl~/pod/perlfunc.pod Mon Jan 02 23:27:04 2006 +++ perl/pod/perlfunc.pod Mon Jan 09 01:19:16 2006 @@ -3108,9 +3108,9 @@ open(FH, "<:utf8", "file") will open the UTF-8 encoded file containing Unicode characters, -see L<perluniintro>. (Note that if layers are specified in the -three-arg form then default layers set by the C<open> pragma are -ignored.) +see L<perluniintro>. Note that if layers are specified in the +three-arg form then default layers stored in ${^OPEN} (see L<perlvar>; +usually set by the B<open> pragma or the switch B<-CioD>) are ignored. Open returns nonzero upon success, the undefined value otherwise. If the C<open> involved a pipe, the return value happens to be the pid of ##### END OF PATCH
CC: perl5-porters [...] perl.org
Subject: Re: [perl #38181] Default IO Layer is ignored in 3-argument open call
Date: Sun, 8 Jan 2006 22:35:18 +0100
To: SADAHIRO Tomoyuki <bqw10602 [...] nifty.com>
From: Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>
Download (untitled) / with headers
text/plain 506b
On 1/8/06, SADAHIRO Tomoyuki <bqw10602@nifty.com> wrote: Show quoted text
> A reply to myself...
> > -L<PerlIO> for more details). For example > > +L<Encode::PerlIO> for more details). For example
> > Sorry this change is wrong. The reference here should be PerlIO.pm > (Thus no change is need here). Revised patch is below.
Thanks, applied as #26739. Show quoted text
> P.S. Why does a link to Encode/PerlIO.pm is produced from L<PerlIO> > on conversion from POD to HTML? I don't know....
A bug ? That should be the PerlIO module...
Download (untitled) / with headers
text/plain 813b
Show quoted text
> This problem seems to have been recently resolved: >
Show quoted text
____________________________________________________________________________
> [ 25925] By: rgs on 2005/10/31 14:27:32 > Log: Fix [perl #37533] open pragma ignored for all 3-arg opens > Also, correct the documentation in open.pm that stated > that open() did not override open.pm-given default encodings. > Branch: perl > ! lib/open.pm perlio.c
Is it possible to somehow work around this problem until the fix reaches my distribution's packages? I'm thinking of using a custom 3-arg open function that explicitly adds the default layers to the layer specification before calling the builtin open. Unfortunately, I don't know to get the list of default layers... Thanks Nikolaus


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org