From @toddr

Created by @toddr

http​://matrix.cpantesters.org/?dist=IO+1.35_01

I'm doing a review of the IO in blead to see if we can publish the
dual life module to CPAN. I'm finding that Strawberry Perl below 5.18
are failing on t/io_utf8.t

I've simplified the program as much as possible​:

#!./perl

use IO​::File;

unlink "io_utf8";

my $io = IO​::File->new;
$io->open("io_utf8", ">​:utf8");
print $io chr(256);
undef $io;

$io = IO​::File->new;
$io->open("io_utf8", "<​:utf8");
ord(<$io>);

my $i = 5;
my $got = $io->ungetc($i);
print "want $i; got $got\n";

The output looks like this​:

C​:\strawberry\cpan\build\IO-1.35_01-yI3WZZ>perl t/io_utf8.t
want 5; got -1

If I take out the ord line, the program passes. Switching the ord to a
$io->getc doesn't behave the same, so I'm guessing this isn't an EOF
issue and something more magical going on with IO​::File?

Flags:
    category=library
    severity=low
    module=IO

Site configuration information for perl 5.16.3:

Configured by strawberry-perl at Tue Mar 12 13:56:09 2013.

Summary of my perl5 (revision 5 version 16 subversion 3) configuration:

  Platform:
    osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread
    uname='Win32 strawberry-perl 5.16.3.1 #1 Tue Mar 12 13:55:20 2013 i386'
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags =' -s -O2 -DWIN32  -DPERL_TEXTMODE_SCRIPTS
-DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fno-strict-aliasing
-mms-bitfields',
    optimize='-s -O2',
    cppflags='-DWIN32'
    ccversion='', gccversion='4.6.3', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='long
long', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='g++', ldflags ='-s -L"C:\strawberry\perl\lib\CORE"
-L"C:\strawberry\c\lib"'
    libpth=C:\strawberry\c\lib C:\strawberry\c\i686-w64-mingw32\lib
    libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=, so=dll, useshrplib=true, libperl=libperl516.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-mdll -s
-L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'

Locally applied patches:



@INC for perl 5.16.3:
    C:/strawberry/perl/site/lib
    C:/strawberry/perl/vendor/lib
    C:/strawberry/perl/lib
    .


Environment for perl 5.16.3:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin
    PERL5_CPANPLUS_IS_RUNNING=5892
    PERL5_CPAN_IS_RUNNING=5892
    PERL_BADLANG (unset)
    SHELL (unset)

From @toddr

The failing test was introduced by perl #116322.

commit 62ba912
Author​: Karl Williamson <public@​khwilliamson.com>
Date​: Sun Feb 17 14​:59​:39 2013 -0700

  Tests for [perl #116322]

commit 10e621b
Author​: Christian Hansen <chansen@​cpan.org>
Date​: Sun Feb 17 14​:50​:30 2013 -0700

  PATCH​: [perl #116322]​: getc() and ungetc() with unicode failure
 
  ungetc() had no knowledge of UTF-8. This patch adds it.
 
  The committer fleshed out the author's code to make a patch, making
  a few small changes.

From @toddr

Further research​:
I mis-read the ord line. the issue is not ord so much as <$io> which is probably causing a EOF.

On the affected MSWin32 versions of perl, I found that these tests also fail using the IO they ship with. Which means the bug is there regardless. Given that I added a skip to the ord check.

Once I did so, I found that the code is really broken for UTF-8 anyway.

# got '0'
# expected '6269'
Malformed UTF-8 character (unexpected non-continuation byte 0xea, immediately after start byte 0xeb) in ord at t/io_utf8
.t line 38.
# Failed at t/io_utf8.t line 38

Given the failing test fails on affected versions of perl regardless of whether I use the IO they have or the new IO, I'm inclined to add a skip for all of these tests. The perl versions aren't supported so it's not like we're going to patch them, and presumably the new IO "more better" than the old IO for other things on Win32

At which point, all platforms should be passing and I can ship the new IO package to CPAN.

From @bulk88

On Fri Jul 31 15​:58​:20 2015, TODDR wrote​:

Given the failing test fails on affected versions of perl regardless
of whether I use the IO they have or the new IO, I'm inclined to add a
skip for all of these tests. The perl versions aren't supported so
it's not like we're going to patch them, and presumably the new IO
"more better" than the old IO for other things on Win32

At which point, all platforms should be passing and I can ship the new
IO package to CPAN.

I dont know enough to really have an opinion since it is utf8 related, so I am giving an opinion since khw/nobody else responded.

Maybe related link http​://www.perlmonks.org/?node_id=1011831

C ungetc unshifts 1 byte, and doesn't guarantee that you can run ungetc multiple times. I'd croak if you give it above 0x7F or 0xFF, maybe 0xFF can be special cased as EOF if you argue EOF is a character (it isn't IMO). Maybe the "if ((SvIOK_notUV(c) && SvIV(c) < 0) || (SvNOK(c) && SvNV(c) < 0.0)) croak("Negative character number in ungetc()");" can be moved around so that check happens only if the PerlIO object is utf8 flagged, and for a byte PerlIO object pushing a negative/high ascii byte is not a problem.

--
bulk88 ~ bulk88 at hotmail.com

The RT System itself - Status changed from 'new' to 'open'

From @toddr

This is how I think the test should be adjusted so it can stay dual life.

Dual-Life/IO@5d05078

I'll submit it to p5p once I get CPAN testers results in a week.

From @toddr

On Jul 31, 2015, at 7​:09 PM, bulk88 via RT <perlbug-followup@​perl.org> wrote​:

On Fri Jul 31 15​:58​:20 2015, TODDR wrote​:

Given the failing test fails on affected versions of perl regardless
of whether I use the IO they have or the new IO, I'm inclined to add a
skip for all of these tests. The perl versions aren't supported so
it's not like we're going to patch them, and presumably the new IO
"more better" than the old IO for other things on Win32

At which point, all platforms should be passing and I can ship the new
IO package to CPAN.

I dont know enough to really have an opinion since it is utf8 related, so I am giving an opinion since khw/nobody else responded.

Maybe related link http​://www.perlmonks.org/?node_id=1011831

Right. best I can tell, that was the cause of the 2 commits that introduced a fix for the perlmonks issue. It even says in the commit message IO was completely unaware of UTF8 for getc. I suspect that these tests introduced by that fix revealed a windows bug we fixed in 5.18.

Since I'm releasing IO, not perl to CPAN, I'm inclined to ignore it rather than doing something crazy as a work around in the XS that would be unnecessary everywhere else. Especially since it was broken anyway in the old IO modules. I feel a little guilty taking that seemingly apathetic attitude. But for a dual life module, if anyone actually complained about it, I'd advise them to upgrade to a newer SP to fix it. Especially since everything else seems to work fine with the improvements old perl or not.

Todd

From @tonycoz

On Fri Jul 31 21​:50​:51 2015, toddr@​cpanel.net wrote​:

On Jul 31, 2015, at 7​:09 PM, bulk88 via RT <perlbug-
followup@​perl.org> wrote​:

On Fri Jul 31 15​:58​:20 2015, TODDR wrote​:

Given the failing test fails on affected versions of perl regardless
of whether I use the IO they have or the new IO, I'm inclined to add
a
skip for all of these tests. The perl versions aren't supported so
it's not like we're going to patch them, and presumably the new IO
"more better" than the old IO for other things on Win32

At which point, all platforms should be passing and I can ship the
new
IO package to CPAN.

I dont know enough to really have an opinion since it is utf8
related, so I am giving an opinion since khw/nobody else responded.

Maybe related link http​://www.perlmonks.org/?node_id=1011831

Right. best I can tell, that was the cause of the 2 commits that
introduced a fix for the perlmonks issue. It even says in the commit
message IO was completely unaware of UTF8 for getc. I suspect that
these tests introduced by that fix revealed a windows bug we fixed in
5.18.

Since I'm releasing IO, not perl to CPAN, I'm inclined to ignore it
rather than doing something crazy as a work around in the XS that
would be unnecessary everywhere else. Especially since it was broken
anyway in the old IO modules. I feel a little guilty taking that
seemingly apathetic attitude. But for a dual life module, if anyone
actually complained about it, I'd advise them to upgrade to a newer SP
to fix it. Especially since everything else seems to work fine with
the improvements old perl or not.

How did this work out?

cpantesters.org is down, but you should have received the summary reports.

Tony

From @Leont

On Wed Jul 29 22​:59​:45 2015, TODDR wrote​:

http​://matrix.cpantesters.org/?dist=IO+1.35_01

I'm doing a review of the IO in blead to see if we can publish the
dual life module to CPAN. I'm finding that Strawberry Perl below 5.18
are failing on t/io_utf8.t

I've simplified the program as much as possible​:

#!./perl

use IO​::File;

unlink "io_utf8";

my $io = IO​::File->new;
$io->open("io_utf8", ">​:utf8");
print $io chr(256);
undef $io;

$io = IO​::File->new;
$io->open("io_utf8", "<​:utf8");
ord(<$io>);

my $i = 5;
my $got = $io->ungetc($i);
print "want $i; got $got\n";

The output looks like this​:

C​:\strawberry\cpan\build\IO-1.35_01-yI3WZZ>perl t/io_utf8.t
want 5; got -1

If I take out the ord line, the program passes. Switching the ord to a
$io->getc doesn't behave the same, so I'm guessing this isn't an EOF
issue and something more magical going on with IO​::File?

Before 5.18 (or actually ec1da99), unread (the primitive used by ungetc) was quite unreliable on :crlf. This was a bug in core, not in IO. Though I suppose IO could work around such a failure by pushing a "​:pending" layer on top and redoing the ungetc.

Leon

From @toddr

On Fri Apr 08 09​:36​:30 2016, LeonT wrote​:

On Wed Jul 29 22​:59​:45 2015, TODDR wrote​:

http​://matrix.cpantesters.org/?dist=IO+1.35_01

I'm doing a review of the IO in blead to see if we can publish the
dual life module to CPAN. I'm finding that Strawberry Perl below 5.18
are failing on t/io_utf8.t

I've simplified the program as much as possible​:

#!./perl

use IO​::File;

unlink "io_utf8";

my $io = IO​::File->new;
$io->open("io_utf8", ">​:utf8");
print $io chr(256);
undef $io;

$io = IO​::File->new;
$io->open("io_utf8", "<​:utf8");
ord(<$io>);

my $i = 5;
my $got = $io->ungetc($i);
print "want $i; got $got\n";

The output looks like this​:

C​:\strawberry\cpan\build\IO-1.35_01-yI3WZZ>perl t/io_utf8.t
want 5; got -1

If I take out the ord line, the program passes. Switching the ord to
a
$io->getc doesn't behave the same, so I'm guessing this isn't an EOF
issue and something more magical going on with IO​::File?

Before 5.18 (or actually ec1da99), unread (the primitive used by
ungetc) was quite unreliable on :crlf. This was a bug in core, not in
IO. Though I suppose IO could work around such a failure by pushing a
"​:pending" layer on top and redoing the ungetc.

Leon

I've applied a patch to the test and am smoking it on CPAN with updates from blead now​:

http​://matrix.cpantesters.org/?dist=IO+1.36_01

From @toddr

IO 1.39 was released in April 2018. This case can be closed.