Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PerlIO::encoding doesn't handle fallback modes correctly? #8614

Closed
p5pRT opened this issue Sep 22, 2006 · 8 comments
Closed

PerlIO::encoding doesn't handle fallback modes correctly? #8614

p5pRT opened this issue Sep 22, 2006 · 8 comments

Comments

@p5pRT
Copy link

p5pRT commented Sep 22, 2006

Migrated from rt.perl.org#40401 (status was 'open')

Searchable as RT40401$

@p5pRT
Copy link
Author

p5pRT commented Sep 22, 2006

From @steve-m-hay

Created by steveh@Mugwump.uk.radan.com

I'm trying to use the FB_HTMLCREF fallback mode with PerlIO​::encoding,
but it isn't doing what I expect.

If I set $PerlIO​::encoding​::fallback to Encode​::FB_HTMLCREF (or,
equivalently, to Encode​::HTMLCREF | Encode​::LEAVE_SRC) and set the
STDOUT encoding layer to 'encoding(windows-1252)' then any output to
STDOUT seems to be repeated, regardless of whether the fallback
mechanism comes into play or not​:

.\perl -MEncode -MPerlIO​::encoding -e "$PerlIO​::encoding​::fallback =
Encode​::FB_HTMLCREF; binmode STDOUT, '​:encoding(windows-1252)'; print
qq[\x{0104}]"
ĄĄ
.\perl -MEncode -MPerlIO​::encoding -e "$PerlIO​::encoding​::fallback =
Encode​::HTMLCREF | Encode​::LEAVE_SRC; binmode STDOUT,
'​:encoding(windows-1252)'; print qq[\x{0104}]"
ĄĄ
.\perl -MEncode -MPerlIO​::encoding -e "$PerlIO​::encoding​::fallback =
Encode​::FB_HTMLCREF; binmode STDOUT, '​:encoding(windows-1252)'; print
qq[abc]"
abcabc
.\perl -MEncode -MPerlIO​::encoding -e "$PerlIO​::encoding​::fallback =
Encode​::HTMLCREF | Encode​::LEAVE_SRC; binmode STDOUT,
'​:encoding(windows-1252)'; print qq[abc]"
abcabc

Using the HTMLCREF bitmask flag without the LEAVE_SRC flag seems to fix
this, but unless I'm missing something the above behaviour is not correct​:

.\perl -MEncode -MPerlIO​::encoding -e "$PerlIO​::encoding​::fallback =
Encode​::HTMLCREF; binmode STDOUT, '​:encoding(windows-1252)'; print
qq[\x{0104}]"
Ą
.\perl -MEncode -MPerlIO​::encoding -e "$PerlIO​::encoding​::fallback =
Encode​::HTMLCREF; binmode STDOUT, '​:encoding(windows-1252)'; print qq[abc]"
abc

Encode​::encode() doesn't have this problem (and leaving out the
LEAVE_SRC flag in that case, of course, causes trouble if the source
string cannot be modified)​:

.\perl -MEncode -e "print encode('windows-1252', qq[\x{0104}],
Encode​::FB_HTMLCREF)"
Ą
.\perl -MEncode -e "print encode('windows-1252', qq[\x{0104}],
Encode​::HTMLCREF | Encode​::LEAVE_SRC)"
Ą
.\perl -MEncode -e "print encode('windows-1252', qq[\x{0104}],
Encode​::HTMLCREF)"
Modification of a read-only value attempted at
C​:/p5p/bleadperl/lib/Encode.pm line 147.

Perl Info

Flags:
     category=library
     severity=medium

Site configuration information for perl 5.9.4:

Configured by steveh at Fri Sep 22 16:42:49 2006.

Summary of my perl5 (revision 5 version 9 subversion 4) configuration:
   Platform:
     osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
     uname=''
     config_args='undef'
     hint=recommended, useposix=true, d_sigaction=undef
     useithreads=define, usemultiplicity=define
     useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
     use64bitint=undef, use64bitall=undef, uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 
-D_CONSOLE -DNO_STRICT -DHAVE_DES_FCRYPT  -DPERL_IMPLICIT_CONTEXT 
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX',
     optimize='-MD -Zi -DNDEBUG -O1',
     cppflags='-DWIN32'
     ccversion='12.00.8804', gccversion='', gccosandvers=''
     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
     d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
     ivtype='long', ivsize=4, nvtype='double', nvsize=8, 
Off_t='__int64', lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf 
-libpath:"c:\perl\lib\CORE"  -machine:x86'
     libpth=C:\PROGRA~1\MICROS~2\VC98\lib
     libs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib 
  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib 
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib 
odbc32.lib odbccp32.lib msvcrt.lib
     perllibs=  oldnames.lib kernel32.lib user32.lib gdi32.lib 
winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib 
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib 
version.lib odbc32.lib odbccp32.lib msvcrt.lib
     libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl59.lib
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
     cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug 
-opt:ref,icf  -libpath:"c:\perl\lib\CORE"  -machine:x86'

Locally applied patches:



@INC for perl 5.9.4:
     C:/perl/lib
     C:/perl/site/lib
     .


Environment for perl 5.9.4:
     HOME (unset)
     LANG (unset)
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
 
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\perl\bin
     PERL_BADLANG (unset)
     SHELL (unset)


-- 


------------------------------------------------
Radan Computational Ltd.

The information contained in this message and any files transmitted with it are confidential and intended for the addressee(s) only. If you have received this message in error or there are any problems, please notify the sender immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Radan Computational Ltd. The recipient(s) of this message should check it and any attached files for viruses: Radan Computational will accept no liability for any damage caused by any virus transmitted by this email.

@p5pRT
Copy link
Author

p5pRT commented May 26, 2013

From @jkeenan

Could someone familiar with PerlIO issues take a crack at this ticket
under Perl 5.18.0?

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented May 26, 2013

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 10, 2018

From user42_kevin@yahoo.com.au

I struck this too in recent Debian i386 packaged 5.26.1,

use strict;
use Encode;
use PerlIO::encoding;
$PerlIO::encoding::fallback = Encode::LEAVE_SRC();
binmode(STDOUT, ":encoding(ASCII)");
print "one\n";
print "two\n";

which prints "one two" three times, whereas I expected just once, which
get by omitting the $PerlIO​::encoding​::fallback line.

I struck this when putting $PerlIO​::encoding​::fallback = FB_PERLQQ() to
get qq output - and that constant includes LEAVE_SRC. The single-bit
PERLQQ() is ok.

I presume some PerlIO​::encoding internals want in-place rather than
LEAVE_SRC.

Perhaps the PerlIO​::encoding docs could say what Encode options are ok.
I imagine that could be DIE_ON_ERR, WARN_ON_ERR, PERLQQ, HTMLCREF, and
XMLCREF. Is a die ok?

@toddr
Copy link
Member

toddr commented Feb 13, 2020

Actually the output there on 5.30 was:

one
one
two
one
two
one
two
one
two

@toddr
Copy link
Member

toddr commented Feb 13, 2020

@steve-m-hay @khwilliamson is this a windows bug? It doesn't seem to be.

@Leont
Copy link
Contributor

Leont commented Feb 15, 2020

Encode​::LEAVE_SRC is never a valid flag value for $fallback. :encoding should either ignore or actively reject this.

@Leont
Copy link
Contributor

Leont commented Mar 4, 2020

This is a duplicate of #7309

@Leont Leont closed this as completed Mar 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants