Skip Menu |
Report information
Id: 131716
Status: rejected
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: exodist7 [at] gmail.com
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type:
Perl Version: 5.27.1
Fixed In: (no value)



Date: Fri, 7 Jul 2017 13:25:47 -0700
From: Chad Granum <exodist7 [...] gmail.com>
Subject: perl 5.27.1 breaks Test2::Plugin::IOSync
To: perlbug [...] perl.org
Download (untitled) / with headers
text/plain 4.9k
Example reports:


This module passes on perls 5.8.8 -> 5.26 so long as Test-Simple and Test2-Suite are both up to date (there are some cpan failures related to these being out of date, the 3 reports above are not due to old versions)


Strangely 5.8.1 fails with the exact same problems as 5.27.1, maybe a regression?


From my own debugging I see 2 problems, they may be related:

1) writes appear to happen out of order (that is we have multiple clones of a filehandle, and we write to each clone, the on-disk copy does not reflect the order of the print statements to the handles, even thought hey all have autoflush set, are sent entire \n terminated lines, and point at the same on-disk file)

2) PerlIO layers seem to be applied differently than they were before, almost like applying one to STDOUT will also apply it to any pre-existing clones of STDOUT, even if it did not previously (this is a guess).

Please note that there is a module all these load called Test2::Plugin::OpenFixPerlIO  which works around a perlio bug that appears to have been fixed for 5.27.1, but changing things so that the module is not loaded has no effect.

All these problems are related to PerlIO layers, the module itself uses PerlIO layers to do its thing.

I tried to break this down into a minimal repro case, but was not able to do so yet.

-Chad


[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=low
---
Site configuration information for perl 5.27.1:

Configured by exodist at Fri Jul  7 08:37:46 PDT 2017.

Summary of my perl5 (revision 5 version 27 subversion 1) configuration:
___
  Platform:
    osname=linux
    osvers=4.11.7-1-arch
    archname=x86_64-linux-thread-multi
    uname='linux abydos 4.11.7-1-arch #1 smp preempt sat jun 24 09:07:09 cest 2017 x86_64 gnulinux '
    config_args='-de -Dprefix=/home/exodist/perl5/perlbrew/perls/5.27.1 -Dusethreads -Dusedevel -Aeval:scriptdir=/home/exodist/perl5/perlbrew/perls/5.27.1/bin'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'    
    optimize='-O2'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='7.1.1 20170621'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/include-fixed /usr/lib /lib/../lib /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.25.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.25'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    Devel::PatchPerl 1.38

---
@INC for perl 5.27.1:
    /home/exodist/perl5/perlbrew/perls/5.27.1/lib/site_perl/5.27.1/x86_64-linux-thread-multi
    /home/exodist/perl5/perlbrew/perls/5.27.1/lib/site_perl/5.27.1
    /home/exodist/perl5/perlbrew/perls/5.27.1/lib/5.27.1/x86_64-linux-thread-multi
    /home/exodist/perl5/perlbrew/perls/5.27.1/lib/5.27.1

---
Environment for perl 5.27.1:
    HOME=/home/exodist
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/exodist/perl5/perlbrew/bin:/home/exodist/perl5/perlbrew/perls/5.27.1/bin:/home/exodist/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
    PERLBREW_BASHRC_VERSION=0.78
    PERLBREW_HOME=/home/exodist/.perlbrew
    PERLBREW_MANPATH=/home/exodist/perl5/perlbrew/perls/5.27.1/man
    PERLBREW_PATH=/home/exodist/perl5/perlbrew/bin:/home/exodist/perl5/perlbrew/perls/5.27.1/bin
    PERLBREW_PERL=5.27.1
    PERLBREW_ROOT=/home/exodist/perl5/perlbrew
    PERLBREW_VERSION=0.78
    PERL_BADLANG (unset)
    SHELL=/usr/bin/zsh

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 4.4k
On Fri, 07 Jul 2017 20:25:57 GMT, exodist7@gmail.com wrote: Show quoted text
> Example reports: > http://www.cpantesters.org/cpan/report/c71f790c-62cf-11e7-a383- > 3a6add4d7def > http://www.cpantesters.org/cpan/report/c9d1fa8a-62cf-11e7-a09b- > 166edd4d7def > http://www.cpantesters.org/cpan/report/1d0446c4-62ec-11e7-99fd- > 83061003a64b > > > This module passes on perls 5.8.8 -> 5.26 so long as Test-Simple and > Test2-Suite are both up to date (there are some cpan failures related > to > these being out of date, the 3 reports above are not due to old > versions) > > > Strangely 5.8.1 fails with the exact same problems as 5.27.1, maybe a > regression? > > > From my own debugging I see 2 problems, they may be related: > > 1) writes appear to happen out of order (that is we have multiple > clones of > a filehandle, and we write to each clone, the on-disk copy does not > reflect > the order of the print statements to the handles, even thought hey all > have > autoflush set, are sent entire \n terminated lines, and point at the > same > on-disk file) > > 2) PerlIO layers seem to be applied differently than they were before, > almost like applying one to STDOUT will also apply it to any pre- > existing > clones of STDOUT, even if it did not previously (this is a guess). > > Please note that there is a module all these load called > Test2::Plugin::OpenFixPerlIO which works around a perlio bug that > appears > to have been fixed for 5.27.1, but changing things so that the module > is > not loaded has no effect. > > All these problems are related to PerlIO layers, the module itself > uses > PerlIO layers to do its thing. > > I tried to break this down into a minimal repro case, but was not able > to > do so yet. > > -Chad >
Here is the commit to blead which appears to have broken this CPAN module: ##### commit 99b847695211f825df6299aa9da91f9494f741e2 Author: Tony Cook <tony@develop-help.com> AuthorDate: Thu Jun 1 15:11:27 2017 +1000 Commit: Tony Cook <tony@develop-help.com> CommitDate: Wed Jun 7 11:25:59 2017 +1000 [perl #131221] improve duplication of :via handles Previously duplication (as with open ... ">&...") would fail unless the user supplied a GETARG, which wasn't documented, and resulted in an attempt to free and unreferened scalar if supplied. Cloning on thread creation was simply broken. We now handle GETARG correctly, and provide a useful default if it returns nothing. Cloning on thread creation now duplicates the appropriate parts of the parent thread's handle. ##### At the immediately preceding commit: ##### $ ./bin/perl -v | head -2 | tail -1 This is perl 5, version 27, subversion 1 (v5.27.1 (v5.27.0-141-g5d09ee1)) built for x86_64-linux $ ./bin/cpanm --verbose Test2::Plugin::IOSync ... Successfully installed Test2-Plugin-IOSync-0.000008 ##### At the breaking commit: ##### $ ./bin/perl -v | head -2 | tail -1 This is perl 5, version 27, subversion 1 (v5.27.1 (v5.27.0-142-g99b8476)) built for x86_64-linux $ ./bin/cpanm --verbose Test2::Plugin::IOSync ... Test Summary Report ------------------- t/IOEvents.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 t/IOEvents/Base.t (Wstat: 512 Tests: 3 Failed: 2) Failed tests: 2-3 Non-zero exit status: 2 t/IOEvents/STDERR.t (Wstat: 512 Tests: 3 Failed: 2) Failed tests: 2-3 Non-zero exit status: 2 t/IOEvents/STDOUT.t (Wstat: 512 Tests: 3 Failed: 2) Failed tests: 2-3 Non-zero exit status: 2 t/IOMuxer.t (Wstat: 512 Tests: 3 Failed: 2) Failed tests: 2-3 Non-zero exit status: 2 t/IOMuxer/Layer.t (Wstat: 256 Tests: 6 Failed: 1) Failed test: 6 Non-zero exit status: 1 t/IOSync.t (Wstat: 768 Tests: 5 Failed: 3) Failed tests: 3-5 Non-zero exit status: 3 t/OpenFixPerlIO.t (Wstat: 768 Tests: 13 Failed: 3) Failed tests: 1-2, 10 Non-zero exit status: 3 Files=8, Tests=37, 1 wallclock secs ( 0.03 usr 0.01 sys + 0.82 cusr 0.06 csys = 0.92 CPU) Result: FAIL Failed 8/8 test programs. 16/37 subtests failed. Makefile:867: recipe for target 'test_dynamic' failed make: *** [test_dynamic] Error 3 FAIL ##### But note that in the test run above I got failures in 8 test files. The number of files with failures in at least one of the CPANtesters reports was only 3. I haven't kept up with Test2 in the past two years, so I'm unfamiliar with the style of testing done in the failing files and cannot further diagnose the problem. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
CC: Perl5 Porteros <perl5-porters [...] perl.org>
Date: Fri, 7 Jul 2017 18:06:53 -0700
From: Chad Granum <exodist7 [...] gmail.com>
To: Max Maischein <perlbug-followup [...] perl.org>
Subject: Re: [perl #131716] perl 5.27.1 breaks Test2::Plugin::IOSync
Would it be possible to make it not diplicate the via layer when the getarg is missing? I think part of the problem is that I do not want the layers to dup with the handle.

On Jul 7, 2017 4:26 PM, "James E Keenan via RT" <perlbug-followup@perl.org> wrote:
Show quoted text
On Fri, 07 Jul 2017 20:25:57 GMT, exodist7@gmail.com wrote:
> Example reports:
> http://www.cpantesters.org/cpan/report/c71f790c-62cf-11e7-a383-
> 3a6add4d7def
> http://www.cpantesters.org/cpan/report/c9d1fa8a-62cf-11e7-a09b-
> 166edd4d7def
> http://www.cpantesters.org/cpan/report/1d0446c4-62ec-11e7-99fd-
> 83061003a64b
>
>
> This module passes on perls 5.8.8 -> 5.26 so long as Test-Simple and
> Test2-Suite are both up to date (there are some cpan failures related
> to
> these being out of date, the 3 reports above are not due to old
> versions)
>
>
> Strangely 5.8.1 fails with the exact same problems as 5.27.1, maybe a
> regression?
>
>
> From my own debugging I see 2 problems, they may be related:
>
> 1) writes appear to happen out of order (that is we have multiple
> clones of
> a filehandle, and we write to each clone, the on-disk copy does not
> reflect
> the order of the print statements to the handles, even thought hey all
> have
> autoflush set, are sent entire \n terminated lines, and point at the
> same
> on-disk file)
>
> 2) PerlIO layers seem to be applied differently than they were before,
> almost like applying one to STDOUT will also apply it to any pre-
> existing
> clones of STDOUT, even if it did not previously (this is a guess).
>
> Please note that there is a module all these load called
> Test2::Plugin::OpenFixPerlIO  which works around a perlio bug that
> appears
> to have been fixed for 5.27.1, but changing things so that the module
> is
> not loaded has no effect.
>
> All these problems are related to PerlIO layers, the module itself
> uses
> PerlIO layers to do its thing.
>
> I tried to break this down into a minimal repro case, but was not able
> to
> do so yet.
>
> -Chad
>

Here is the commit to blead which appears to have broken this CPAN module:

#####
commit 99b847695211f825df6299aa9da91f9494f741e2
Author:     Tony Cook <tony@develop-help.com>
AuthorDate: Thu Jun 1 15:11:27 2017 +1000
Commit:     Tony Cook <tony@develop-help.com>
CommitDate: Wed Jun 7 11:25:59 2017 +1000

    [perl #131221] improve duplication of :via handles

    Previously duplication (as with open ... ">&...") would fail
    unless the user supplied a GETARG, which wasn't documented, and
    resulted in an attempt to free and unreferened scalar if supplied.

    Cloning on thread creation was simply broken.

    We now handle GETARG correctly, and provide a useful default if it
    returns nothing.

    Cloning on thread creation now duplicates the appropriate parts of the
    parent thread's handle.
#####

At the immediately preceding commit:

#####
$ ./bin/perl -v | head -2 | tail -1
This is perl 5, version 27, subversion 1 (v5.27.1 (v5.27.0-141-g5d09ee1)) built for x86_64-linux
$ ./bin/cpanm --verbose Test2::Plugin::IOSync
...
Successfully installed Test2-Plugin-IOSync-0.000008
#####

At the breaking commit:

#####
$ ./bin/perl -v | head -2 | tail -1
This is perl 5, version 27, subversion 1 (v5.27.1 (v5.27.0-142-g99b8476)) built for x86_64-linux
$ ./bin/cpanm --verbose Test2::Plugin::IOSync
...
Test Summary Report
-------------------
t/IOEvents.t       (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
t/IOEvents/Base.t  (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
t/IOEvents/STDERR.t (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
t/IOEvents/STDOUT.t (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
t/IOMuxer.t        (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
t/IOMuxer/Layer.t  (Wstat: 256 Tests: 6 Failed: 1)
  Failed test:  6
  Non-zero exit status: 1
t/IOSync.t         (Wstat: 768 Tests: 5 Failed: 3)
  Failed tests:  3-5
  Non-zero exit status: 3
t/OpenFixPerlIO.t  (Wstat: 768 Tests: 13 Failed: 3)
  Failed tests:  1-2, 10
  Non-zero exit status: 3
Files=8, Tests=37,  1 wallclock secs ( 0.03 usr  0.01 sys +  0.82 cusr  0.06 csys =  0.92 CPU)
Result: FAIL
Failed 8/8 test programs. 16/37 subtests failed.
Makefile:867: recipe for target 'test_dynamic' failed
make: *** [test_dynamic] Error 3
FAIL
#####

But note that in the test run above I got failures in 8 test files.  The number of files with failures in at least one of the CPANtesters reports was only 3.

I haven't kept up with Test2 in the past two years, so I'm unfamiliar with the style of testing done in the failing files and cannot further diagnose the problem.

Thank you very much.
--
James E Keenan (jkeenan@cpan.org)

---
via perlbug:  queue: perl5 status: new
https://rt.perl.org/Ticket/Display.html?id=131716
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 404b
I have found ways to mitigate all the breakages that do not involve changes to perl itself, and they do not break older versions. If nothing else is done for this task I should be ok. That said, I want to propose that a via layer without a GETARGS should simply not be applied during cloning. Or at the very least make it so that a GETARGS that is present, but returns undef can implement that behavior.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 686b
On Sat, 08 Jul 2017 03:25:53 GMT, exodist7@gmail.com wrote: Show quoted text
> I have found ways to mitigate all the breakages that do not involve > changes to perl itself, and they do not break older versions. If > nothing else is done for this task I should be ok. >
Resolving ticket. Show quoted text
> That said, I want to propose that a via layer without a GETARGS should > simply not be applied during cloning. Or at the very least make it so > that a GETARGS that is present, but returns undef can implement that > behavior.
Chad, if you would like to get discussion going about this, I'd suggest sending email to p5p with fresh discussion about it. Thank you very much. -- James E Keenan (jkeenan@cpan.org)


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org