PerlIO, STDERR and "encoding" layer problem #12906

p5pRT · 2013-04-08T18:33:21Z

Migrated from rt.perl.org#117537 (status was 'open')

Searchable as RT117537$

p5pRT · 2013-04-08T18:33:21Z

From victor@vsespb.ru

Created by victor@vsespb.ru

There is a ticket for Encode module
https://rt.cpan.org/Ticket/Display.html?id=84493
Problem that (probably Encode code) sometimes hangs in deep recursion,
when encoding layer set for STDERR and there are errors in encoding processing.

DANKOGAI suggest that it's actually PerlIO documentation problem and that should
be documented that PerlIO can't be used with STDERR (only exception
:utf8 layer).

Thus I submit this bug here.

This perldoc http://perldoc.perl.org/PerlIO.html does not mention STDERR.

Also open pragma doc http://perldoc.perl.org/open.html advertises use
of encoding with STDERR with single bytes encodings
"On the other hand, if only output is chosen to be in :encoding(koi8r)
, a :std will cause only the STDOUT and STDERR to be in koi8r "

I find also problem in the fact that carp/die/croak cannot be used now
with single-byte encoding locales.

(othere related topics https://rt.cpan.org/Ticket/Display.html?id=84282
http://www.perlmonks.org/?node_id=1026042 )

Perl Info


Flags:
    category=core
    severity=low

Site configuration information for perl 5.10.1:

Configured by Debian Project at Tue Nov 27 00:14:30 UTC 2012.

Summary of my perl5 (revision 5 version 10 subversion 1) configuration:

  Platform:
    osname=linux, osvers=2.6.42-23-generic,
archname=x86_64-linux-gnu-thread-multi
    uname='linux komainu 2.6.42-23-generic #36-ubuntu smp tue apr 10
20:39:51 utc 2012 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
-Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr
-Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10
-Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl/5.10.1
-Dsitearch=/usr/local/lib/perl/5.10.1 -Dman1dir=/usr/share/man/man1
-Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1
-Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio
-Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib
-Dlibperl=libperl.so.5.10.1 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing
-pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.3', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.11.1.so, so=so, useshrplib=true, libperl=libperl.so.5.10.1
    gnulibc_version='2.11.1'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib
-fstack-protector'

Locally applied patches:



@INC for perl 5.10.1:
    /etc/perl
    /usr/local/lib/perl/5.10.1
    /usr/local/share/perl/5.10.1
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.10
    /usr/share/perl/5.10
    /usr/local/lib/site_perl
    .


Environment for perl 5.10.1:
    HOME=/home/vse
    LANG=ru_RU.utf8
    LANGUAGE=en_US:en
    LC_MESSAGES=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/vse/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/vse/.rvm/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

p5pRT · 2013-04-23T14:02:18Z

From @Leont

On Mon, Apr 8, 2013 at 8:33 PM, Victor Efimov <perlbug-followup@perl.org> wrote:

There is a ticket for Encode module
https://rt.cpan.org/Ticket/Display.html?id=84493
Problem that (probably Encode code) sometimes hangs in deep recursion,
when encoding layer set for STDERR and there are errors in encoding processing.

DANKOGAI suggest that it's actually PerlIO documentation problem and that should
be documented that PerlIO can't be used with STDERR (only exception
:utf8 layer).

Saying "PerlIO can't be used with STDERR" is outright silly, because
you can't use STDERR without it. It makes sense that you're restricted
in what you can do with it, but those restrictions aren't well
defined.

Thus I submit this bug here.

This perldoc http://perldoc.perl.org/PerlIO.html does not mention STDERR.

Also open pragma doc http://perldoc.perl.org/open.html advertises use
of encoding with STDERR with single bytes encodings
"On the other hand, if only output is chosen to be in :encoding(koi8r)
, a :std will cause only the STDOUT and STDERR to be in koi8r "

I find also problem in the fact that carp/die/croak cannot be used now
with single-byte encoding locales.

(othere related topics https://rt.cpan.org/Ticket/Display.html?id=84282
http://www.perlmonks.org/?node_id=1026042 )

How about setting the fallback value (somewhat documented in
PerlIO::encoding and Encode) to FB_QUIET? I'd guess that would solve
your problem.

Leon

p5pRT · 2013-04-23T14:02:19Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2013-04-26T13:38:36Z

From victor@vsespb.ru

Hello.

Seems it helps indeed.

One of examples which hangs (I was able to reproduce only on FreeBSD 9)

perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; croak
"\320\220\320\262\321\202\320\276\320\277\320\260\321\200\320\272"'

With fallback:

perl -MEncode -MCarp -MPerlIO::encoding -e 'use strict;
$PerlIO::encoding::fallback = Encode::FB_QUIET; binmode STDERR,
":encoding(koi8-r)"; croak
"\320\220\320\262\321\202\320\276\320\277\320\260\321\200\320\272"'

does not hang.

Thanks!

I still think it should be documented better.

BTW Possible example of hang in real world application (that's how
actually this happened to me):

When "confess" is used, it prints a stacktrace with function parameters.
And some parameters can contain "random" binary data (user's file
content, non-unicode filenames etc). That's pretty hard to debug if
happen on user's machine - you don't even have a stacktrace and can't
reproduce it ;(

On Tue Apr 23 07:02:18 2013, LeonT wrote:

On Mon, Apr 8, 2013 at 8:33 PM, Victor Efimov <perlbug-
followup@perl.org> wrote:

There is a ticket for Encode module
https://rt.cpan.org/Ticket/Display.html?id=84493
Problem that (probably Encode code) sometimes hangs in deep
recursion,
when encoding layer set for STDERR and there are errors in encoding
processing.

DANKOGAI suggest that it's actually PerlIO documentation problem and
that should
be documented that PerlIO can't be used with STDERR (only exception
:utf8 layer).

Saying "PerlIO can't be used with STDERR" is outright silly, because
you can't use STDERR without it. It makes sense that you're restricted
in what you can do with it, but those restrictions aren't well
defined.

Thus I submit this bug here.

This perldoc http://perldoc.perl.org/PerlIO.html does not mention
STDERR.

Also open pragma doc http://perldoc.perl.org/open.html advertises
use
of encoding with STDERR with single bytes encodings
"On the other hand, if only output is chosen to be in
:encoding(koi8r)
, a :std will cause only the STDOUT and STDERR to be in koi8r "

I find also problem in the fact that carp/die/croak cannot be used
now
with single-byte encoding locales.

(othere related topics
https://rt.cpan.org/Ticket/Display.html?id=84282
http://www.perlmonks.org/?node_id=1026042 )

How about setting the fallback value (somewhat documented in
PerlIO::encoding and Encode) to FB_QUIET? I'd guess that would solve
your problem.

Leon

p5pRT · 2013-05-09T19:16:37Z

From victor@vsespb.ru

Ah, no, seems it didn't help.

Here is another PoC
Hangs on Ubuntu 10.04 perl 5.10.1 and Ubuntu 12.04 server, perl 5.14.2

#!/usr/bin/perl
use Encode;
use PerlIO::encoding;
use strict;
use warnings;
use utf8;

$PerlIO::encoding::fallback = Encode::FB_QUIET;
binmode STDERR, ":encoding(koi8-r)";
my $i = 1;

for my $a1 (qw/а б в й ё/) {
for my $a2 (qw/А /) {
for my $a3 (qw/А Б В Г Д /) {
print STDERR encode("UTF-8", "Автопарк".$a1.$a2.$a3);
++$i;
print "i=$i\n";
}
}
}
__END__

On Fri Apr 26 06:38:36 2013, vsespb wrote:

Hello.

Seems it helps indeed.

One of examples which hangs (I was able to reproduce only on FreeBSD 9)

perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; croak
"\320\220\320\262\321\202\320\276\320\277\320\260\321\200\320\272"'

With fallback:

perl -MEncode -MCarp -MPerlIO::encoding -e 'use strict;
$PerlIO::encoding::fallback = Encode::FB_QUIET; binmode STDERR,
":encoding(koi8-r)"; croak
"\320\220\320\262\321\202\320\276\320\277\320\260\321\200\320\272"'

does not hang.

Thanks!

I still think it should be documented better.

BTW Possible example of hang in real world application (that's how
actually this happened to me):

When "confess" is used, it prints a stacktrace with function parameters.
And some parameters can contain "random" binary data (user's file
content, non-unicode filenames etc). That's pretty hard to debug if
happen on user's machine - you don't even have a stacktrace and can't
reproduce it ;(

On Tue Apr 23 07:02:18 2013, LeonT wrote:

On Mon, Apr 8, 2013 at 8:33 PM, Victor Efimov <perlbug-
followup@perl.org> wrote:

There is a ticket for Encode module
https://rt.cpan.org/Ticket/Display.html?id=84493
Problem that (probably Encode code) sometimes hangs in deep
recursion,
when encoding layer set for STDERR and there are errors in encoding
processing.

DANKOGAI suggest that it's actually PerlIO documentation problem and
that should
be documented that PerlIO can't be used with STDERR (only exception
:utf8 layer).

Saying "PerlIO can't be used with STDERR" is outright silly, because
you can't use STDERR without it. It makes sense that you're restricted
in what you can do with it, but those restrictions aren't well
defined.

Thus I submit this bug here.

This perldoc http://perldoc.perl.org/PerlIO.html does not mention
STDERR.

Also open pragma doc http://perldoc.perl.org/open.html advertises
use
of encoding with STDERR with single bytes encodings
"On the other hand, if only output is chosen to be in
:encoding(koi8r)
, a :std will cause only the STDOUT and STDERR to be in koi8r "

I find also problem in the fact that carp/die/croak cannot be used
now
with single-byte encoding locales.

(othere related topics
https://rt.cpan.org/Ticket/Display.html?id=84282
http://www.perlmonks.org/?node_id=1026042 )

How about setting the fallback value (somewhat documented in
PerlIO::encoding and Encode) to FB_QUIET? I'd guess that would solve
your problem.

Leon

p5pRT · 2013-05-09T20:10:36Z

From @Leont

On Thu, May 9, 2013 at 9:16 PM, Victor Efimov via RT
<perlbug-followup@perl.org> wrote:

Ah, no, seems it didn't help.

Here is another PoC
Hangs on Ubuntu 10.04 perl 5.10.1 and Ubuntu 12.04 server, perl 5.14.2

#!/usr/bin/perl
use Encode;
use PerlIO::encoding;
use strict;
use warnings;
use utf8;

$PerlIO::encoding::fallback = Encode::FB_QUIET;
binmode STDERR, ":encoding(koi8-r)";
my $i = 1;

for my $a1 (qw/а б в й ё/) {
for my $a2 (qw/А /) {
for my $a3 (qw/А Б В Г Д /) {
print STDERR encode("UTF-8", "Автопарк".$a1.$a2.$a3);
++$i;
print "i=$i\n";
}
}
}

That encode there is wrong. It's converting a unicode string into a
byte string, but the unicode filehandle will interpret that as
Latin-1.

Regardless of that this does seem to trigger an infinite loop of some
kind. At first sight it seems PerlIO::encoding isn't robust in the
face of Encode::FB_QUIET.Given FB_QUIET, Encode can return an empty
strings, which :encoding fails to recognize as an error condition.
It's an infinite loop of trying to parse it, but only consuming 0
bytes.

Leon

p5pRT · 2013-05-09T20:15:56Z

From victor@vsespb.ru

I know that encode here is wrong. This is just example how Perl hangs on
_random_ _binary_ data (I didn't find better way to reproduce problem
and to create some PoC).

Also, this hangs with FB_QUIET and without FB_QUIET.

On Thu May 09 13:10:36 2013, LeonT wrote:

On Thu, May 9, 2013 at 9:16 PM, Victor Efimov via RT
<perlbug-followup@perl.org> wrote:

Ah, no, seems it didn't help.

Here is another PoC
Hangs on Ubuntu 10.04 perl 5.10.1 and Ubuntu 12.04 server, perl 5.14.2

#!/usr/bin/perl
use Encode;
use PerlIO::encoding;
use strict;
use warnings;
use utf8;

$PerlIO::encoding::fallback = Encode::FB_QUIET;
binmode STDERR, ":encoding(koi8-r)";
my $i = 1;

for my $a1 (qw/а б в й ё/) {
for my $a2 (qw/А /) {
for my $a3 (qw/А Б В Г Д /) {
print STDERR encode("UTF-8", "Автопарк".$a1.$a2.$a3);
++$i;
print "i=$i\n";
}
}
}

That encode there is wrong. It's converting a unicode string into a
byte string, but the unicode filehandle will interpret that as
Latin-1.

Regardless of that this does seem to trigger an infinite loop of some
kind. At first sight it seems PerlIO::encoding isn't robust in the
face of Encode::FB_QUIET.Given FB_QUIET, Encode can return an empty
strings, which :encoding fails to recognize as an error condition.
It's an infinite loop of trying to parse it, but only consuming 0
bytes.

Leon

p5pRT · 2013-05-09T21:55:33Z

From @Leont

On Thu, May 9, 2013 at 10:15 PM, Victor Efimov via RT
<perlbug-followup@perl.org> wrote:

Also, this hangs with FB_QUIET and without FB_QUIET.

Though it seems for different reasons. On handles other than STDERR it
only hangs for FB_QUIET.

What I think is happening is this: :encoding::Flush calls Encode,
which warns to stderr, which in order to be written to calls Flush.
However, because Flush refuses to recurse, the buffer keeps begin full
and Write loops trying to flush.

I'm not sure how to solve that issue. Making STDERR always FB_QUIET
may make sense, once we've resolved the other issue.

Leon

p5pRT · 2013-05-09T22:10:25Z

From victor@vsespb.ru

Ok, thanks for the information!

On Thu May 09 14:55:33 2013, LeonT wrote:

On Thu, May 9, 2013 at 10:15 PM, Victor Efimov via RT
<perlbug-followup@perl.org> wrote:

Also, this hangs with FB_QUIET and without FB_QUIET.

Though it seems for different reasons. On handles other than STDERR it
only hangs for FB_QUIET.

What I think is happening is this: :encoding::Flush calls Encode,
which warns to stderr, which in order to be written to calls Flush.
However, because Flush refuses to recurse, the buffer keeps begin full
and Write loops trying to flush.

I'm not sure how to solve that issue. Making STDERR always FB_QUIET
may make sense, once we've resolved the other issue.

Leon

p5pRT added Severity Low distro-Linux type-core labels Oct 19, 2019

xenu removed the affects-5.10 label Nov 19, 2021

xenu removed the Severity Low label Dec 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PerlIO, STDERR and "encoding" layer problem #12906

PerlIO, STDERR and "encoding" layer problem #12906

p5pRT commented Apr 8, 2013

p5pRT commented Apr 8, 2013

p5pRT commented Apr 23, 2013

p5pRT commented Apr 23, 2013

p5pRT commented Apr 26, 2013

p5pRT commented May 9, 2013

p5pRT commented May 9, 2013

p5pRT commented May 9, 2013

p5pRT commented May 9, 2013

p5pRT commented May 9, 2013

PerlIO, STDERR and "encoding" layer problem #12906

PerlIO, STDERR and "encoding" layer problem #12906

Comments

p5pRT commented Apr 8, 2013

p5pRT commented Apr 8, 2013

From victor@vsespb.ru

Created by victor@vsespb.ru

p5pRT commented Apr 23, 2013

From @Leont

p5pRT commented Apr 23, 2013

p5pRT commented Apr 26, 2013

From victor@vsespb.ru

p5pRT commented May 9, 2013

From victor@vsespb.ru

p5pRT commented May 9, 2013

From @Leont

p5pRT commented May 9, 2013

From victor@vsespb.ru

p5pRT commented May 9, 2013

From @Leont

p5pRT commented May 9, 2013

From victor@vsespb.ru