Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] local PerlIO::encoding::fallback settings + encoding causing infinite loop #15117

Open
p5pRT opened this issue Jan 4, 2016 · 4 comments

Comments

@p5pRT
Copy link

p5pRT commented Jan 4, 2016

Migrated from rt.perl.org#127149 (status was 'open')

Searchable as RT127149$

@p5pRT
Copy link
Author

p5pRT commented Jan 4, 2016

From zhouzhen1@gmail.com

Created by zhouzhen1@gmail.com

I saw the issue when I use File​::Slurper + Data​::Table together in one
script. I then simplified the demo case to be like below. Basically it's
like, you open a file in with some encoding, and set
$PerlIO​::encoding​::fallback in that "local", then out of that "local" scope
you open another file with some encoding, you get an infinite loop when
reading the second file.

use 5.022;

my $file1 = '/tmp/foo';
my $file2 = '/tmp/bar';
system("echo bar > $file2");

{ # what's in File​::Slurper
  my $layer = '​:raw​:encoding(utf-8)';
  local $PerlIO​::encoding​::fallback = 1;
  open my $fh1, ">$layer", $file1 or die $!;
  close $fh1 or die $!;
}

{ # what's in Data​::Table
  open(my $fh2, $file2) or die $!;
  binmode $fh2, "​:encoding(UTF-8)";
  while (my $line = <$fh2>) {
  warn $line;
  }
}

It can also be simplified to like below.

use 5.022;

my $file1 = '/tmp/foo';
my $file2 = '/tmp/bar';
system("echo foo > $file1");
system("echo bar > $file2");

{
  local $PerlIO​::encoding​::fallback = 1;
  open my $fh1, "<​:encoding(utf-8)", $file1 or die $!;
  close $fh1 or die $!;
}

{
  open my $fh2, '<​:encoding(utf-8)', $file2 or die $!;
  while (my $line = <$fh2>) {
  warn $line;
  }
}

Perl Info

Flags:
    category=library
    severity=medium
    module=PerlIO

Site configuration information for perl 5.22.1:

Configured by builduser at Tue Dec 15 10:04:11 CET 2015.

Summary of my perl5 (revision 5 version 22 subversion 1) configuration:

  Platform:
    osname=linux, osvers=4.3.2-1-arch, archname=x86_64-linux-thread-multi
    uname='linux flo-64 4.3.2-1-arch #1 smp preempt fri dec 11 07:19:47 cet
2015 x86_64 gnulinux '
    config_args='-des -Dusethreads -Duseshrplib -Doptimize=-march=x86-64
-mtune=generic -O2 -pipe -fstack-protector-strong -Dprefix=/usr
-Dvendorprefix=/usr -Dprivlib=/usr/share/perl5/core_perl
-Darchlib=/usr/lib/perl5/core_perl -Dsitelib=/usr/share/perl5/site_perl
-Dsitearch=/usr/lib/perl5/site_perl
-Dvendorlib=/usr/share/perl5/vendor_perl
-Dvendorarch=/usr/lib/perl5/vendor_perl -Dscriptdir=/usr/bin/core_perl
-Dsitescript=/usr/bin/site_perl -Dvendorscript=/usr/bin/vendor_perl
-Dinc_version_list=none -Dman1ext=1perl -Dman3ext=3perl
-Dcccdlflags='-fPIC' -Dlddlflags=-shared
-Wl,-O1,--sort-common,--as-needed,-z,relro
-Dldflags=-Wl,-O1,--sort-common,--as-needed,-z,relro'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv
-fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-march=x86-64 -mtune=generic -O2 -pipe
-fstack-protector-strong',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include'
    ccversion='', gccversion='5.3.0', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678,
doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16,
longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-Wl,-O1,--sort-common,--as-needed,-z,relro
-fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib
/usr/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include-fixed /usr/lib
/lib/../lib /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
-lgdbm_compat
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.22'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/usr/lib/perl5/core_perl/CORE'
    cccdlflags='-fPIC', lddlflags='-shared
-Wl,-O1,--sort-common,--as-needed,-z,relro -L/usr/local/lib
-fstack-protector-strong'



@INC for perl 5.22.1:
    /home/zhou/perl5/lib/perl5/x86_64-linux-thread-multi
    /home/zhou/perl5/lib/perl5
    /usr/lib/perl5/site_perl
    /usr/share/perl5/site_perl
    /usr/lib/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib/perl5/core_perl
    /usr/share/perl5/core_perl
    .


Environment for perl 5.22.1:
    HOME=/home/zhou
    LANG=en_US.utf8
    LANGUAGE (unset)
    LC_CTYPE=zh_CN.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)

PATH=/home/zhou/.rakudobrew/bin:/home/zhou/.rakudobrew/bin:/home/zhou/.rakudobrew/moar-nom/install/share/perl6/site/bin:/home/zhou/.rakudobrew/moar-nom/install/bin:/home/zhou/.rakudobrew/moar-nom/panda/bin:/home/zhou/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
    PERL5LIB=/home/zhou/perl5/lib/perl5
    PERL_BADLANG (unset)
    PERL_LOCAL_LIB_ROOT=/home/zhou/perl5
    PERL_MB_OPT=--install_base "/home/zhou/perl5"
    PERL_MM_OPT=INSTALL_BASE=/home/zhou/perl5
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Jan 13, 2016

From @tonycoz

On Mon Jan 04 06​:12​:19 2016, zhouzhen1@​gmail.com wrote​:

I saw the issue when I use File​::Slurper + Data​::Table together in one
script. I then simplified the demo case to be like below. Basically
it's
like, you open a file in with some encoding, and set
$PerlIO​::encoding​::fallback in that "local", then out of that "local"
scope
you open another file with some encoding, you get an infinite loop
when
reading the second file.

The problem is that since PerlIO​::encoding is loaded within the local's block, $PerlIO​::encoding​::fallback is restored to it's original value
at entry to the block - undef.

This can be prevented by explicitly loading PerlIO​::encoding before localizing $PerlIO​::encoding​::fallback

Also, since PerlIO​::encoding is loaded *after* you initialized fallback, it's being reset to the default set in PerlIO/encoding.pm before the value is used.

That said, PerlIO​::encoding shouldn't go into an infinite loop, the attached patch fixes that for this case by ensuring the STOP_AT_PARTIAL flag is set, which is needed for decoding.

Tony

@p5pRT
Copy link
Author

p5pRT commented Jan 13, 2016

From @tonycoz

0001-perl-127149-ensure-the-CHECK-value-has-STOP_AT_PARTI.patch
From 460df6ceb4e243f5e2b68e72c445fc4e1e883897 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 13 Jan 2016 12:11:45 +1100
Subject: [perl #127149] ensure the CHECK value has STOP_AT_PARTIAL

This flag is required for PerlIO::encoding to do its job.

Also document the race between localizing
$PerlIO::encoding::fallback and loading PerlIO::encoding.
---
 ext/PerlIO-encoding/encoding.pm | 7 ++++++-
 ext/PerlIO-encoding/encoding.xs | 5 ++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/ext/PerlIO-encoding/encoding.pm b/ext/PerlIO-encoding/encoding.pm
index 7bae036..df1ded6 100644
--- a/ext/PerlIO-encoding/encoding.pm
+++ b/ext/PerlIO-encoding/encoding.pm
@@ -1,7 +1,7 @@
 package PerlIO::encoding;
 
 use strict;
-our $VERSION = '0.23';
+our $VERSION = '0.24';
 our $DEBUG = 0;
 $DEBUG and warn __PACKAGE__, " called by ", join(", ", caller), "\n";
 
@@ -46,6 +46,11 @@ When the layer is pushed, the current value of C<$PerlIO::encoding::fallback>
 is saved and used as the CHECK argument when calling the Encode methods
 encode() and decode().
 
+Note: localizing C<$PerlIO::encoding::fallback> without
+PerlIO::encoding being loaded and then using the C<:encoding> layer
+will leave C<$PerlIO::encoding::fallback> as C<undef> at the end of
+the scope.
+
 =head1 SEE ALSO
 
 L<open>, L<Encode>, L<perlfunc/binmode>, L<perluniintro>
diff --git a/ext/PerlIO-encoding/encoding.xs b/ext/PerlIO-encoding/encoding.xs
index 3e4645c..599728c 100644
--- a/ext/PerlIO-encoding/encoding.xs
+++ b/ext/PerlIO-encoding/encoding.xs
@@ -6,6 +6,9 @@
 
 #define OUR_DEFAULT_FB	"Encode::PERLQQ"
 
+/* from Encode's encode.h */
+#define  ENCODE_STOP_AT_PARTIAL 0x0800 /* stop at partial explicitly */
+
 #if defined(USE_PERLIO)
 
 /* Define an encoding "layer" in the perliol.h sense.
@@ -163,7 +166,7 @@ PerlIOEncode_pushed(pTHX_ PerlIO * f, const char *mode, SV * arg, PerlIO_funcs *
 	PerlIOBase(f)->flags |= PERLIO_F_UTF8;
     }
 
-    e->chk = newSVsv(get_sv("PerlIO::encoding::fallback", 0));
+    e->chk = newSViv(SvIV(get_sv("PerlIO::encoding::fallback", 0)) | ENCODE_STOP_AT_PARTIAL);
     e->inEncodeCall = 0;
 
     FREETMPS;
-- 
2.1.4

@p5pRT
Copy link
Author

p5pRT commented Jan 13, 2016

The RT System itself - Status changed from 'new' to 'open'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants