Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13 #16907

Closed
p5pRT opened this issue Mar 26, 2019 · 43 comments

Comments

@p5pRT
Copy link

p5pRT commented Mar 26, 2019

Migrated from rt.perl.org#133958 (status was 'resolved')

Searchable as RT133958$

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2019

From @jkeenan

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two
separate bug tickets for these failures, since one failure appeared long
before the others. This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13. See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED
54 Non-zero exit status​: 1
  [stdio] -Dcc=clang++ -Duse64bitall
  [stdio] -Dcc=clang++
  [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13, that
of course being one of Carlos Guevara's rigs. These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
  );

# rt.cpan.org 53469 is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err, '$@​ untouched' );
cmp_ok( $!, '==', $Errno, '$! untouched' );
#####

Thank you very much.
Jim Keenan

perl perl perl

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2019

From @khwilliamson

On 3/25/19 6​:45 PM, James E Keenan (via RT) wrote​:

# New Ticket Created by James E Keenan
# Please include the string​: [perl #133958]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two
separate bug tickets for these failures, since one failure appeared long
before the others. This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13. See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED
54 Non-zero exit status​: 1
[stdio] -Dcc=clang++ -Duse64bitall
[stdio] -Dcc=clang++
[stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13, that
of course being one of Carlos Guevara's rigs. These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
);

# rt.cpan.org 53469 is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err, '$@​ untouched' );
cmp_ok( $!, '==', $Errno, '$! untouched' );
#####

Thank you very much.
Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2019

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2019

From @jkeenan

On 3/27/19 11​:15 PM, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM, James E Keenan (via RT) wrote​:

# New Ticket Created by James E Keenan
# Please include the string​: [perl #133958]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two
separate bug tickets for these failures, since one failure appeared long
before the others. This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13. See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED
54 Non-zero exit status​: 1
[stdio] -Dcc=clang++ -Duse64bitall
[stdio] -Dcc=clang++
[stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13, that
of course being one of Carlos Guevara's rigs. These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
);

# rt.cpan.org 53469 is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err, '$@​ untouched' );
cmp_ok( $!, '==', $Errno, '$! untouched' );
#####

Thank you very much.
Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer. How would I do that?

2. Even if I did that, how would that address the problem that we've
only been able to observe this problem during smoke-testing?

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2019

From @khwilliamson

On 3/28/19 6​:21 AM, James E Keenan wrote​:

On 3/27/19 11​:15 PM, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan
# Please include the string​:  [perl #133958]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two
separate bug tickets for these failures, since one failure appeared long
before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED
54 Non-zero exit status​: 1
     [stdio] -Dcc=clang++ -Duse64bitall
     [stdio] -Dcc=clang++
     [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13, that
of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
            );

# rt.cpan.org 53469  is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err,               '$@​ untouched' );
cmp_ok( $!, '==', $Errno,   '$! untouched' );
#####

Thank you very much.
Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer.  How would I do that?

Instructions are in perlhacktips. It's just some Configure options to
clang, which I believe you're already using. Then you run the test suite.

2. Even if I did that, how would that address the problem that we've
only been able to observe this problem during smoke-testing?

There are two likely causes for something showing up only in smoke
testing that I can think of off-hand. One is timing, with lots of
parallel jobs. And the other is that there is a read of uninitialized
memory. Address sanitizer will catch this latter cause. This is
similar to valgrind on Linux. I have run valgrind on the test suite
recently with no errors beyond the usual two in CPAN (and yes there is a
ticket for these open)

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2019

From @jkeenan

On 3/28/19 10​:16 AM, Karl Williamson wrote​:

On 3/28/19 6​:21 AM, James E Keenan wrote​:

On 3/27/19 11​:15 PM, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan
# Please include the string​:  [perl #133958]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two
separate bug tickets for these failures, since one failure appeared
long
before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED
54 Non-zero exit status​: 1
     [stdio] -Dcc=clang++ -Duse64bitall
     [stdio] -Dcc=clang++
     [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13, that
of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
            );

# rt.cpan.org 53469  is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err,               '$@​ untouched' );
cmp_ok( $!, '==', $Errno,   '$! untouched' );
#####

Thank you very much.
Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer.  How would I do that?

Instructions are in perlhacktips.  It's just some Configure options to
clang, which I believe you're already using.  Then you run the test suite.

2. Even if I did that, how would that address the problem that we've
only been able to observe this problem during smoke-testing?

There are two likely causes for something showing up only in smoke
testing that I can think of off-hand.  One is timing, with lots of
parallel jobs.

I should note that in this environment $TEST_JOBS is set to 2. So
(assuming I understand the test code correctly), only handy00.t and
handy01.t really matter. I tried running those two in parallel by​:

$> cd t; ./perl harness ../ext/XS-APItest/t/handy00.t
../ext/XS-APItest/t/handy01.t; cd -

I could not reproduce the problem. But it seems to occur regularly
inside Test-Smoke's harness.

And the other is that there is a read of uninitialized
memory.  Address sanitizer will catch this latter cause.  This is
similar to valgrind on Linux.

I will try to test with valgrind today or tomorrow.

I have run valgrind on the test suite
recently with no errors beyond the usual two in CPAN (and yes there is a
ticket for these open)

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Apr 2, 2019

From @jkeenan

On Thu, 28 Mar 2019 15​:15​:50 GMT, jkeenan@​pobox.com wrote​:

On 3/28/19 10​:16 AM, Karl Williamson wrote​:

On 3/28/19 6​:21 AM, James E Keenan wrote​:

On 3/27/19 11​:15 PM, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan
# Please include the string​:  [perl #133958]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two
separate bug tickets for these failures, since one failure appeared
long
before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED
54 Non-zero exit status​: 1
     [stdio] -Dcc=clang++ -Duse64bitall
     [stdio] -Dcc=clang++
     [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13, that
of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
            );

# rt.cpan.org 53469  is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err,               '$@​ untouched' );
cmp_ok( $!, '==', $Errno,   '$! untouched' );
#####

Thank you very much.
Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer.  How would I do that?

Instructions are in perlhacktips.  It's just some Configure options to
clang, which I believe you're already using.  Then you run the test suite.

2. Even if I did that, how would that address the problem that we've
only been able to observe this problem during smoke-testing?

There are two likely causes for something showing up only in smoke
testing that I can think of off-hand.  One is timing, with lots of
parallel jobs.

I should note that in this environment $TEST_JOBS is set to 2. So
(assuming I understand the test code correctly), only handy00.t and
handy01.t really matter. I tried running those two in parallel by​:

$> cd t; ./perl harness ../ext/XS-APItest/t/handy00.t
../ext/XS-APItest/t/handy01.t; cd -

I could not reproduce the problem. But it seems to occur regularly
inside Test-Smoke's harness.

And the other is that there is a read of uninitialized
memory.  Address sanitizer will catch this latter cause.  This is
similar to valgrind on Linux.

I will try to test with valgrind today or tomorrow.

I mistyped. I meant to say that I would try to build a perl with AddressSanitizer. See below.

I have run valgrind on the test suite
recently with no errors beyond the usual two in CPAN (and yes there is a
ticket for these open)

My efforts to build and test a perl at blead with AddressSanitizer in this FreeBSD-13 were largely unsuccessful.

After struggling with the syntax of the ./Configure invocation, I was eventually able to get 'make' to complete successfully. See attachment with name starting with 'perl_V'.

However, I could not run 'make test_harness' successfully. Normally, in these VMs I have $TEST_JOBS set to 2. However, 'make test_harness' was 'Killed' in the t/re/*.t tests. See attachment starting with 'test_harness'.

I then set TEST_JOBS=1 and ran 'make test' -- so no tests were running in parallel. I got only a little farther on. See attachment starting with 'make_test'.

Now, I concede​: (a) I've never previously attempted to build with AddressSanitizer; (b) This VM may be too small to run a perl built with AddressSanitizer. But Carlos's FreeBSD-13 smoker, while not large, is larger than mine -- and we're still getting the failures on non-threaded builds in t/Legacy/More.t which are the subject of this RT. See, e.g., http​://perl5.test-smoke.org/report/83974.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 2, 2019

From @jkeenan

t/re/no_utf8_pm ................................................ ok
t/re/overload .................................................. ok
t/re/pat ....................................................... FAILED--expected 860 tests, saw 108
t/re/pat_advanced .............................................. ok
t/re/pat_advanced_thr .......................................... ok
t/re/pat_psycho ................................................ # Test process timed out - terminating
FAILED--expected 15 tests, saw 0
t/re/pat_psycho_thr ............................................ Killed
*** Error code 137

Stop.
make​: stopped in /home/jkeenan/gitwork/perl

@p5pRT
Copy link
Author

p5pRT commented Apr 2, 2019

From @jkeenan

Summary of my perl5 (revision 5 version 29 subversion 10) configuration​:
  Commit id​: 8e48a48
  Platform​:
  osname=freebsd
  osvers=13.0-current
  archname=amd64-freebsd-thread-multi
  uname='freebsd perl-reporter-05 13.0-current freebsd 13.0-current r340361 generic amd64 '
  config_args='-des -Dusedevel -Duseithreads -Dcc=clang -Accflags=-fsanitize=address -Aldflags=-fsanitize=address -Alddlflags=-shared -fsanitize=address'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=define
  usemultiplicity=define
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='clang'
  ccflags ='-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fsanitize=address -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2'
  optimize='-O2'
  cppflags='-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fsanitize=address -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion=''
  gccversion='4.2.1 Compatible FreeBSD Clang 6.0.1 (tags/RELEASE_601/final 335540)'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='clang'
  ldflags ='-pthread -Wl,-E -fsanitize=address -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/lib /usr/local/lib /usr/lib/clang/6.0.1/lib /usr/lib
  libs=-lpthread -lgdbm -ldl -lm -lcrypt -lutil
  perllibs=-lpthread -ldl -lm -lcrypt -lutil
  libc=
  so=so
  useshrplib=false
  libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags=' '
  cccdlflags='-DPIC -fPIC'
  lddlflags='-shared -shared -fsanitize=address -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  HAS_TIMES
  MULTIPLICITY
  PERLIO_LAYERS
  PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_IMPLICIT_CONTEXT
  PERL_MALLOC_WRAP
  PERL_OP_PARENT
  PERL_PRESERVE_IVUV
  PERL_USE_DEVEL
  USE_64_BIT_ALL
  USE_64_BIT_INT
  USE_ITHREADS
  USE_LARGE_FILES
  USE_LOCALE
  USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC
  USE_LOCALE_TIME
  USE_PERLIO
  USE_PERL_ATOF
  USE_REENTRANT_API
  USE_THREAD_SAFE_LOCALE
  Built under freebsd
  Compiled at Mar 29 2019 14​:44​:36
  %ENV​:
  PERL2DIR="/home/jkeenan/gitwork/perl2"
  PERL_WORKDIR="/home/jkeenan/gitwork/perl"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.29.10/amd64-freebsd-thread-multi
  /usr/local/lib/perl5/site_perl/5.29.10
  /usr/local/lib/perl5/5.29.10/amd64-freebsd-thread-multi
  /usr/local/lib/perl5/5.29.10

@p5pRT
Copy link
Author

p5pRT commented Apr 2, 2019

From @jkeenan

./miniperl -Ilib make_ext.pl cpan/Archive-Tar/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Attribute-Handlers/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Attribute-Handlers directly
./miniperl -Ilib make_ext.pl cpan/AutoLoader/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/AutoLoader directly
./miniperl -Ilib make_ext.pl cpan/CPAN/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/CPAN-Meta/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/CPAN-Meta-Requirements/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/CPAN-Meta-Requirements directly
./miniperl -Ilib make_ext.pl cpan/CPAN-Meta-YAML/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/CPAN-Meta-YAML directly
./miniperl -Ilib make_ext.pl dist/Carp/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Carp directly
./miniperl -Ilib make_ext.pl cpan/Config-Perl-V/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Config-Perl-V directly
./miniperl -Ilib make_ext.pl dist/Devel-SelfStubber/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Devel-SelfStubber directly
./miniperl -Ilib make_ext.pl cpan/Digest/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Digest directly
./miniperl -Ilib make_ext.pl dist/Dumpvalue/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Dumpvalue directly
./miniperl -Ilib make_ext.pl dist/Env/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Env directly
./miniperl -Ilib make_ext.pl ext/Errno/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Exporter/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Exporter directly
./miniperl -Ilib make_ext.pl dist/ExtUtils-CBuilder/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/ExtUtils-CBuilder directly
./miniperl -Ilib make_ext.pl cpan/ExtUtils-Constant/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/ExtUtils-Constant directly
./miniperl -Ilib make_ext.pl cpan/ExtUtils-Install/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/ExtUtils-Install directly
./miniperl -Ilib make_ext.pl cpan/ExtUtils-MakeMaker/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/ExtUtils-Manifest/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl ext/ExtUtils-Miniperl/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for ext/ExtUtils-Miniperl directly
./miniperl -Ilib make_ext.pl dist/ExtUtils-ParseXS/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/File-Fetch/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/File-Fetch directly
./miniperl -Ilib make_ext.pl ext/File-Find/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for ext/File-Find directly
./miniperl -Ilib make_ext.pl cpan/File-Path/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/File-Path directly
./miniperl -Ilib make_ext.pl cpan/File-Temp/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/File-Temp directly
./miniperl -Ilib make_ext.pl ext/FileCache/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for ext/FileCache directly
./miniperl -Ilib make_ext.pl dist/Filter-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Filter-Simple directly
./miniperl -Ilib make_ext.pl cpan/Getopt-Long/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Getopt-Long directly
./miniperl -Ilib make_ext.pl cpan/HTTP-Tiny/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/I18N-Collate/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/I18N-Collate directly
./miniperl -Ilib make_ext.pl dist/I18N-LangTags/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/I18N-LangTags directly
./miniperl -Ilib make_ext.pl dist/lib/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/IO-Compress/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/IO-Socket-IP/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/IO-Socket-IP directly
./miniperl -Ilib make_ext.pl cpan/IO-Zlib/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/IO-Zlib directly
./miniperl -Ilib make_ext.pl cpan/IPC-Cmd/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/IPC-Cmd directly
./miniperl -Ilib make_ext.pl ext/IPC-Open3/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for ext/IPC-Open3 directly
./miniperl -Ilib make_ext.pl cpan/JSON-PP/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Locale-Maketext/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Locale-Maketext directly
./miniperl -Ilib make_ext.pl cpan/Locale-Maketext-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Locale-Maketext-Simple directly
./miniperl -Ilib make_ext.pl cpan/Math-BigInt/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Math-BigInt directly
./miniperl -Ilib make_ext.pl cpan/Math-BigRat/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Math-BigRat directly
./miniperl -Ilib make_ext.pl cpan/Math-Complex/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Math-Complex directly
./miniperl -Ilib make_ext.pl cpan/Memoize/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Memoize directly
./miniperl -Ilib make_ext.pl dist/Module-CoreList/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/Module-Load/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Module-Load directly
./miniperl -Ilib make_ext.pl cpan/Module-Load-Conditional/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Module-Load-Conditional directly
./miniperl -Ilib make_ext.pl cpan/Module-Loaded/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Module-Loaded directly
./miniperl -Ilib make_ext.pl cpan/Module-Metadata/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/NEXT/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/NEXT directly
./miniperl -Ilib make_ext.pl dist/Net-Ping/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Net-Ping directly
./miniperl -Ilib make_ext.pl cpan/Params-Check/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Params-Check directly
./miniperl -Ilib make_ext.pl cpan/Perl-OSType/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Perl-OSType directly
./miniperl -Ilib make_ext.pl cpan/PerlIO-via-QuotedPrint/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/PerlIO-via-QuotedPrint directly
./miniperl -Ilib make_ext.pl cpan/Pod-Checker/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/Pod-Escapes/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Pod-Escapes directly
./miniperl -Ilib make_ext.pl cpan/Pod-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Pod-Simple directly
./miniperl -Ilib make_ext.pl ext/Pod-Functions/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl ext/Pod-Html/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/Pod-Parser/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/Pod-Perldoc/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/Pod-Usage/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Safe/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Safe directly
./miniperl -Ilib make_ext.pl dist/Search-Dict/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Search-Dict directly
./miniperl -Ilib make_ext.pl dist/SelfLoader/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/SelfLoader directly
./miniperl -Ilib make_ext.pl cpan/Term-ANSIColor/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Term-ANSIColor directly
./miniperl -Ilib make_ext.pl cpan/Term-Cap/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Term-Complete/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Term-Complete directly
./miniperl -Ilib make_ext.pl dist/Term-ReadLine/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Term-ReadLine directly
./miniperl -Ilib make_ext.pl dist/Test/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Test directly
./miniperl -Ilib make_ext.pl cpan/Test-Harness/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/Test-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Test-Simple directly
./miniperl -Ilib make_ext.pl dist/Text-Abbrev/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Text-Abbrev directly
./miniperl -Ilib make_ext.pl cpan/Text-Balanced/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Text-Balanced directly
./miniperl -Ilib make_ext.pl cpan/Text-ParseWords/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Text-ParseWords directly
./miniperl -Ilib make_ext.pl cpan/Text-Tabs/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Text-Tabs directly
./miniperl -Ilib make_ext.pl dist/Thread-Queue/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Thread-Queue directly
./miniperl -Ilib make_ext.pl dist/Thread-Semaphore/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Thread-Semaphore directly
./miniperl -Ilib make_ext.pl dist/Tie-File/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Tie-File directly
./miniperl -Ilib make_ext.pl ext/Tie-Memoize/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for ext/Tie-Memoize directly
./miniperl -Ilib make_ext.pl cpan/Tie-RefHash/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Tie-RefHash directly
./miniperl -Ilib make_ext.pl cpan/Time-Local/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Time-Local directly
./miniperl -Ilib make_ext.pl dist/XSLoader/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/autodie/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/autodie directly
./miniperl -Ilib make_ext.pl dist/autouse/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/autouse directly
./miniperl -Ilib make_ext.pl dist/base/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/bignum/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/bignum directly
./miniperl -Ilib make_ext.pl dist/constant/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/constant directly
./miniperl -Ilib make_ext.pl dist/encoding-warnings/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/encoding-warnings directly
./miniperl -Ilib make_ext.pl cpan/experimental/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/experimental directly
./miniperl -Ilib make_ext.pl dist/if/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/libnet/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/parent/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/parent directly
./miniperl -Ilib make_ext.pl cpan/perlfaq/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/perlfaq directly
./miniperl -Ilib make_ext.pl cpan/podlators/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Manifying 2 pod documents
./miniperl -Ilib make_ext.pl cpan/version/pm_to_blib MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/version directly
./miniperl -Ilib mkppport
ppport.h in cpan/DB_File is up-to-date
ppport.h in cpan/IPC-SysV is up-to-date
ppport.h in cpan/Win32API-File is up-to-date
ppport.h in dist/IO is up-to-date
ppport.h in dist/Storable is up-to-date
./miniperl -Ilib make_ext.pl DynaLoader.o MAKE="make" LIBPERL_A=libperl.a LINKTYPE=static CCCDLFLAGS=
./miniperl -Ilib make_ext.pl lib/auto/Cwd/Cwd.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/PathTools/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Cwd.bs ../../lib/auto/Cwd/Cwd.bs 644

  Making utilities
./miniperl -Ilib lib/unicore/mktables -C lib/unicore -P pod -maketest -makelist -p
lib/unicore/mktables​: Files seem to be ok, not bothering to rebuild. Add '-w' option to force build
./miniperl -Ilib make_ext.pl lib/auto/B/B.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/B/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- B.bs ../../lib/auto/B/B.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Compress/Raw/Bzip2/Bzip2.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Compress-Raw-Bzip2/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Bzip2.bs ../../lib/auto/Compress/Raw/Bzip2/Bzip2.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Compress/Raw/Zlib/Zlib.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Compress-Raw-Zlib/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Zlib.bs ../../lib/auto/Compress/Raw/Zlib/Zlib.bs 644
./miniperl -Ilib make_ext.pl lib/auto/DB_File/DB_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/DB_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- DB_File.bs ../../lib/auto/DB_File/DB_File.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Data/Dumper/Dumper.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/Data-Dumper/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Dumper.bs ../../lib/auto/Data/Dumper/Dumper.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Devel/PPPort/PPPort.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/Devel-PPPort/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- PPPort.bs ../../lib/auto/Devel/PPPort/PPPort.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Devel/Peek/Peek.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/Devel-Peek/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Peek.bs ../../lib/auto/Devel/Peek/Peek.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Digest/MD5/MD5.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Digest-MD5/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- MD5.bs ../../lib/auto/Digest/MD5/MD5.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Digest/SHA/SHA.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Digest-SHA/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- SHA.bs ../../lib/auto/Digest/SHA/SHA.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Encode/Encode.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Encode/Byte/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Byte.bs ../../../lib/auto/Encode/Byte/Byte.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/CN/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- CN.bs ../../../lib/auto/Encode/CN/CN.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/EBCDIC/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- EBCDIC.bs ../../../lib/auto/Encode/EBCDIC/EBCDIC.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/JP/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- JP.bs ../../../lib/auto/Encode/JP/JP.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/KR/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- KR.bs ../../../lib/auto/Encode/KR/KR.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/Symbol/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Symbol.bs ../../../lib/auto/Encode/Symbol/Symbol.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/TW/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- TW.bs ../../../lib/auto/Encode/TW/TW.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/Unicode/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Unicode.bs ../../../lib/auto/Encode/Unicode/Unicode.bs 644
/home/jkeenan/gitwork/perl/cpan/Encode/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Encode.bs ../../lib/auto/Encode/Encode.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Fcntl/Fcntl.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/Fcntl/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Fcntl.bs ../../lib/auto/Fcntl/Fcntl.bs 644
./miniperl -Ilib make_ext.pl lib/auto/File/DosGlob/DosGlob.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/File-DosGlob/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- DosGlob.bs ../../lib/auto/File/DosGlob/DosGlob.bs 644
./miniperl -Ilib make_ext.pl lib/auto/File/Glob/Glob.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/File-Glob/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Glob.bs ../../lib/auto/File/Glob/Glob.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Filter/Util/Call/Call.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Filter-Util-Call/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Call.bs ../../lib/auto/Filter/Util/Call/Call.bs 644
./miniperl -Ilib make_ext.pl lib/auto/GDBM_File/GDBM_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/GDBM_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- GDBM_File.bs ../../lib/auto/GDBM_File/GDBM_File.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Hash/Util/Util.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/Hash-Util/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Util.bs ../../lib/auto/Hash/Util/Util.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Hash/Util/FieldHash/FieldHash.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/Hash-Util-FieldHash/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- FieldHash.bs ../../lib/auto/Hash/Util/FieldHash/FieldHash.bs 644
./miniperl -Ilib make_ext.pl lib/auto/I18N/Langinfo/Langinfo.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/I18N-Langinfo/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Langinfo.bs ../../lib/auto/I18N/Langinfo/Langinfo.bs 644
./miniperl -Ilib make_ext.pl lib/auto/IO/IO.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/IO/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- IO.bs ../../lib/auto/IO/IO.bs 644
./miniperl -Ilib make_ext.pl lib/auto/IPC/SysV/SysV.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/IPC-SysV/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- SysV.bs ../../lib/auto/IPC/SysV/SysV.bs 644
./miniperl -Ilib make_ext.pl lib/auto/List/Util/Util.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Scalar-List-Utils/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Util.bs ../../lib/auto/List/Util/Util.bs 644
./miniperl -Ilib make_ext.pl lib/auto/MIME/Base64/Base64.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/MIME-Base64/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Base64.bs ../../lib/auto/MIME/Base64/Base64.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Math/BigInt/FastCalc/FastCalc.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Math-BigInt-FastCalc/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- FastCalc.bs ../../lib/auto/Math/BigInt/FastCalc/FastCalc.bs 644
./miniperl -Ilib make_ext.pl lib/auto/NDBM_File/NDBM_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/NDBM_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- NDBM_File.bs ../../lib/auto/NDBM_File/NDBM_File.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Opcode/Opcode.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/Opcode/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Opcode.bs ../../lib/auto/Opcode/Opcode.bs 644
./miniperl -Ilib make_ext.pl lib/auto/POSIX/POSIX.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/POSIX/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- POSIX.bs ../../lib/auto/POSIX/POSIX.bs 644
./miniperl -Ilib make_ext.pl lib/auto/PerlIO/encoding/encoding.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/PerlIO-encoding/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- encoding.bs ../../lib/auto/PerlIO/encoding/encoding.bs 644
./miniperl -Ilib make_ext.pl lib/auto/PerlIO/mmap/mmap.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/PerlIO-mmap/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- mmap.bs ../../lib/auto/PerlIO/mmap/mmap.bs 644
./miniperl -Ilib make_ext.pl lib/auto/PerlIO/scalar/scalar.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/PerlIO-scalar/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- scalar.bs ../../lib/auto/PerlIO/scalar/scalar.bs 644
./miniperl -Ilib make_ext.pl lib/auto/PerlIO/via/via.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/PerlIO-via/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- via.bs ../../lib/auto/PerlIO/via/via.bs 644
./miniperl -Ilib make_ext.pl lib/auto/SDBM_File/SDBM_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/SDBM_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- SDBM_File.bs ../../lib/auto/SDBM_File/SDBM_File.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Socket/Socket.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Socket/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Socket.bs ../../lib/auto/Socket/Socket.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Storable/Storable.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/Storable/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Storable.bs ../../lib/auto/Storable/Storable.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Sys/Hostname/Hostname.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/Sys-Hostname/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Hostname.bs ../../lib/auto/Sys/Hostname/Hostname.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Sys/Syslog/Syslog.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Sys-Syslog/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Syslog.bs ../../lib/auto/Sys/Syslog/Syslog.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Tie/Hash/NamedCapture/NamedCapture.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/Tie-Hash-NamedCapture/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- NamedCapture.bs ../../lib/auto/Tie/Hash/NamedCapture/NamedCapture.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Time/HiRes/HiRes.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/Time-HiRes/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- HiRes.bs ../../lib/auto/Time/HiRes/HiRes.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Time/Piece/Piece.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Time-Piece/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Piece.bs ../../lib/auto/Time/Piece/Piece.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Unicode/Collate/Collate.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/cpan/Unicode-Collate/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Collate.bs ../../lib/auto/Unicode/Collate/Collate.bs 644
./miniperl -Ilib make_ext.pl lib/auto/Unicode/Normalize/Normalize.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/Unicode-Normalize/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Normalize.bs ../../lib/auto/Unicode/Normalize/Normalize.bs 644
./miniperl -Ilib make_ext.pl lib/auto/XS/APItest/APItest.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
./miniperl -Ilib make_ext.pl lib/auto/XS/Typemap/Typemap.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/XS-Typemap/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Typemap.bs ../../lib/auto/XS/Typemap/Typemap.bs 644
./miniperl -Ilib make_ext.pl lib/auto/attributes/attributes.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/attributes/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- attributes.bs ../../lib/auto/attributes/attributes.bs 644
./miniperl -Ilib make_ext.pl lib/auto/mro/mro.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/mro/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- mro.bs ../../lib/auto/mro/mro.bs 644
./miniperl -Ilib make_ext.pl lib/auto/re/re.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/ext/re/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- re.bs ../../lib/auto/re/re.bs 644
./miniperl -Ilib make_ext.pl lib/auto/threads/threads.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/threads/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- threads.bs ../../lib/auto/threads/threads.bs 644
./miniperl -Ilib make_ext.pl lib/auto/threads/shared/shared.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic
/home/jkeenan/gitwork/perl/dist/threads-shared/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- shared.bs ../../lib/auto/threads/shared/shared.bs 644
./perl -Ilib -I. -f pod/buildtoc -q
cd t && (rm -f perl; /bin/ln -s ../perl perl)
TESTFILE=harness ./runtests choose
No saved state, selection will be empty
base/cond.t ........................................................ ok
base/if.t .......................................................... ok
base/lex.t ......................................................... ok
base/num.t ......................................................... ok
base/pat.t ......................................................... ok
base/rs.t .......................................................... ok
base/term.t ........................................................ ok
base/translate.t ................................................... ok
base/while.t ....................................................... ok
comp/bproto.t ...................................................... ok
comp/cmdopt.t ...................................................... ok
comp/colon.t ....................................................... ok
comp/decl.t ........................................................ ok
comp/filter_exception.t ............................................ ok
comp/final_line_num.t .............................................. ok
comp/form_scope.t .................................................. ok
comp/fold.t ........................................................ ok
comp/line_debug.t .................................................. ok
comp/hints.t ....................................................... ok
comp/multiline.t ................................................... ok
comp/opsubs.t ...................................................... ok
comp/our.t ......................................................... ok
comp/package.t ..................................................... ok
comp/package_block.t ............................................... ok
comp/parser.t ...................................................... ok
comp/parser_run.t .................................................. ok
comp/redef.t ....................................................... ok
comp/proto.t ....................................................... ok
comp/retainedlines.t ............................................... ok
comp/require.t ..................................................... ok
comp/term.t ........................................................ ok
comp/uproto.t ...................................................... ok
comp/use.t ......................................................... ok
comp/utf.t ......................................................... ok
run/cloexec.t ...................................................... ok
run/dtrace.t ....................................................... skipped​: no dtrace
run/exit.t ......................................................... ok
run/fresh_perl.t ................................................... ok
run/noswitch.t ..................................................... ok
run/runenv.t ....................................................... ok
run/locale.t ....................................................... ok
run/switch0.t ...................................................... ok
run/script.t ....................................................... ok
run/switcha.t ...................................................... ok
run/switchC.t ...................................................... ok
run/switchd-78586.t ................................................ ok
run/switchd.t ...................................................... ok
run/switchDx.t ..................................................... skipped​: DEBUGGING build required
run/switchF.t ...................................................... ok
run/switches.t ..................................................... ok
run/switchF1.t ..................................................... ok
run/switchF2.t ..................................................... ok
run/switchI.t ...................................................... ok
run/switchM.t ...................................................... ok
run/switchn.t ...................................................... ok
run/switchp.t ...................................................... ok
run/switcht.t ...................................................... ok
run/switchx.t ...................................................... ok
cmd/elsif.t ........................................................ ok
cmd/for.t .......................................................... ok
cmd/mod.t .......................................................... ok
cmd/subval.t ....................................................... ok
cmd/switch.t ....................................................... ok
io/argv.t .......................................................... ok
io/binmode.t ....................................................... ok
io/bom.t ........................................................... ok
io/closepid.t ...................................................... ok
io/crlf.t .......................................................... ok
io/data.t .......................................................... ok
io/defout.t ........................................................ ok
io/crlf_through.t .................................................. ok
io/eintr.t ......................................................... skipped​: various portability issues
io/dup.t ........................................................... ok
io/eintr_print.t ................................................... skipped​: various portability issues
io/errno.t ......................................................... ok
io/fflush.t ........................................................ ok
io/fs.t ............................................................ ok
io/errnosig.t ...................................................... ok
io/inplace.t ....................................................... ok
io/iofile.t ........................................................ ok
io/layers.t ........................................................ ok
io/iprefix.t ....................................................... ok
io/nargv.t ......................................................... ok
io/openpid.t ....................................................... ok
io/open.t .......................................................... ok
io/paragraph_mode.t ................................................ ok
io/perlio.t ........................................................ ok
io/perlio_fail.t ................................................... ok
io/perlio_leaks.t .................................................. ok
io/perlio_open.t ................................................... ok
io/pipe.t .......................................................... ok
io/print.t ......................................................... ok
io/pvbm.t .......................................................... ok
io/read.t .......................................................... ok
io/say.t ........................................................... ok
io/sem.t ........................................................... ok
io/semctl.t ........................................................ ok
io/shm.t ........................................................... ok
io/socket.t ........................................................ ok
io/tell.t .......................................................... ok
io/socketpair.t .................................................... ok
io/utf8.t .......................................................... ok
re/alpha_assertions.t .............................................. ok
io/through.t ....................................................... ok
re/charset.t ....................................................... ok
# Test process timed out - terminating
re/fold_grind_8.t ..................................................
All 9 subtests passed
# Test process timed out - terminating
re/fold_grind_a.t ..................................................
All 9 subtests passed
# Test process timed out - terminating
re/fold_grind_aa.t .................................................
All 9 subtests passed
# Test process timed out - terminating
re/fold_grind_d.t ..................................................
All 9 subtests passed
# Test process timed out - terminating
re/fold_grind_l.t ..................................................
All 12 subtests passed
re/fold_grind_T.t .................................................. skipped​: Couldn't find a UTF-8 turkic locale
# Test process timed out - terminating
re/fold_grind_u.t ..................................................
All 9 subtests passed
re/keep_tabs.t ..................................................... ok
re/no_utf8_pm.t .................................................... ok
re/overload.t ...................................................... ok
Killed
*** Error code 137

Stop.
make​: stopped in /home/jkeenan/gitwork/perl

@p5pRT
Copy link
Author

p5pRT commented Apr 3, 2019

From @iabyn

On Tue, Apr 02, 2019 at 02​:31​:16PM -0700, James E Keenan via RT wrote​:

Now, I concede​: (a) I've never previously attempted to build with
AddressSanitizer; (b) This VM may be too small to run a perl built with
AddressSanitizer.

Address Sanitizer needs lots of memory. My 16Gb system struggles to
run 10 tests in parallel when compiled with ASan.

Just t/re/pat_psycho.t consumes about 1.7Gb on a threaded debugging build.

--
31 December 1661​: "I have newly taken a solemne oath about abstaining from
plays". 1 January 1662​: "And after ... we went by coach to the play".
  -- The Diary of Samuel Pepys

@p5pRT
Copy link
Author

p5pRT commented Apr 12, 2019

From @jkeenan

On Tue, 02 Apr 2019 21​:31​:16 GMT, jkeenan wrote​:

On Thu, 28 Mar 2019 15​:15​:50 GMT, jkeenan@​pobox.com wrote​:

On 3/28/19 10​:16 AM, Karl Williamson wrote​:

On 3/28/19 6​:21 AM, James E Keenan wrote​:

On 3/27/19 11​:15 PM, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan
# Please include the string​:  [perl #133958]
# in the subject line of all future correspondence about this
issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing
of the
Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two
separate bug tickets for these failures, since one failure
appeared
long
before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t .........................
FAILED
54 Non-zero exit status​: 1
     [stdio] -Dcc=clang++ -Duse64bitall
     [stdio] -Dcc=clang++
     [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-
13, that
of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that
the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That
commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure
manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
            );

# rt.cpan.org 53469  is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err,               '$@​ untouched' );
cmp_ok( $!, '==', $Errno,   '$! untouched' );
#####

I have since built perl5 blead on FreeBSD-13 (i) unthreaded; (ii) without AddressSanitizer; (iii) $TEST_JOBS=1. In this manner​:

1. In 'make test_harness' the last test in ../cpan/Test-Simple/t/Legacy/More.t FAILs.

#####
../cpan/Test-Simple/t/Legacy/More.t (Wstat​: 256 Tests​: 54 Failed​: 1)
  Failed test​: 54
  Non-zero exit status​: 1
#####

2. However, when, in the same build, I run that file manually, that file PASSes.

#####
$ cd t;./perl harness ../cpan/Test-Simple/t/Legacy/More.t; cd -
../cpan/Test-Simple/t/Legacy/More.t .. ok
All tests successful.
Files=1, Tests=54, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.09 cusr 0.04 csys = 0.15 CPU)
Result​: PASS
#####

The above was built/tested at​:

#####
commit a719c52 (HEAD -> blead, origin/blead, origin/HEAD)
Author​: Tomasz Konojacki <me@​xenu.pl>
AuthorDate​: Tue Apr 9 23​:15​:41 2019 +0200
Commit​: Steve Hay <steve.m.hay@​googlemail.com>
CommitDate​: Wed Apr 10 22​:52​:21 2019 +0100
#####

But when I build *threaded*, the test passes during 'make test_harness'.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 12, 2019

From @khwilliamson

On 4/11/19 7​:57 PM, James E Keenan via RT wrote​:

On Tue, 02 Apr 2019 21​:31​:16 GMT, jkeenan wrote​:

On Thu, 28 Mar 2019 15​:15​:50 GMT, jkeenan@​pobox.com wrote​:

On 3/28/19 10​:16 AM, Karl Williamson wrote​:

On 3/28/19 6​:21 AM, James E Keenan wrote​:

On 3/27/19 11​:15 PM, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan
# Please include the string​:  [perl #133958]
# in the subject line of all future correspondence about this
issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing
of the
Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two
separate bug tickets for these failures, since one failure
appeared
long
before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during
smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.​:

http​://perl5.test-smoke.org/report/82470
http​://perl5.test-smoke.org/report/82946

Test failures​:
~~ ../cpan/Test-Simple/t/Legacy/More.t .........................
FAILED
54 Non-zero exit status​: 1
     [stdio] -Dcc=clang++ -Duse64bitall
     [stdio] -Dcc=clang++
     [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-
13, that
of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most, but not all, non-threaded builds.

iii. Appear to have first reported during a smoke-test in
http​://perl5.test-smoke.org/report/79958, which suggests that
the
failures were introduced in, or shortly before commit
https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That
commit
is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure
manually
-- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

#####
$ tail cpan/Test-Simple/t/Legacy/More.t
            );

# rt.cpan.org 53469  is_deeply with regexes
is_deeply( qr/a/, qr/a/, "same regex" );

# These two tests must remain at the end.
is( $@​, $Err,               '$@​ untouched' );
cmp_ok( $!, '==', $Errno,   '$! untouched' );
#####

I have since built perl5 blead on FreeBSD-13 (i) unthreaded; (ii) without AddressSanitizer; (iii) $TEST_JOBS=1. In this manner​:

1. In 'make test_harness' the last test in ../cpan/Test-Simple/t/Legacy/More.t FAILs.

#####
../cpan/Test-Simple/t/Legacy/More.t (Wstat​: 256 Tests​: 54 Failed​: 1)
Failed test​: 54
Non-zero exit status​: 1
#####

2. However, when, in the same build, I run that file manually, that file PASSes.

#####
$ cd t;./perl harness ../cpan/Test-Simple/t/Legacy/More.t; cd -
../cpan/Test-Simple/t/Legacy/More.t .. ok
All tests successful.
Files=1, Tests=54, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.09 cusr 0.04 csys = 0.15 CPU)
Result​: PASS
#####

The above was built/tested at​:

#####
commit a719c52 (HEAD -> blead, origin/blead, origin/HEAD)
Author​: Tomasz Konojacki <me@​xenu.pl>
AuthorDate​: Tue Apr 9 23​:15​:41 2019 +0200
Commit​: Steve Hay <steve.m.hay@​googlemail.com>
CommitDate​: Wed Apr 10 22​:52​:21 2019 +0100
#####

But when I build *threaded*, the test passes during 'make test_harness'.

One possibility is to bisect this. You don't have that big a range of
commits that it could be IIUC. It should be possible to construct a
bisect that runs a whole 'make test'

@p5pRT
Copy link
Author

p5pRT commented Apr 14, 2019

From @jkeenan

On Fri, 12 Apr 2019 15​:52​:53 GMT, public@​khwilliamson.com wrote​:

One possibility is to bisect this. You don't have that big a range of
commits that it could be IIUC. It should be possible to construct a
bisect that runs a whole 'make test'

We don't have a value for the '--target' switch to
Porting/bisect.pl that would evaluate the overall result of
'make test' or 'make test_harness'. I suppose that we could
create such a target, but that's going to require many hours
of work and more brain power than I currently have.

I did, however, try to bisect manually -- but before I get
to that, let's recall the original problem. We have​:

* A new test failure in a very old, cpan-upstream, test.
* The error only occurs on the development version of one
  operating system.
* The error only occurs on unthreaded builds.
* The error never seems to appear when the test file is run
  by itself.
* The error appears frequently, but nonetheless
  intermittently, when run in test harnesses, such as during
  smoke-testing.

In addition, it appears that when the error does appear, it
looks like this​:

#####
cpan/Test-Simple/t/Legacy/More ...# Failed test '$! untouched'
# at t/Legacy/More.t line 184.
# got​: 12
# expected​: 42
# Looks like you failed 1 test of 54.
FAILED at test 54
#####

Since the error is intermittent, it's quite likely that a
single bisection run will be unable to pinpoint the first
bad commit in a reproducible manner. Hence, it's quite
likely that the best we can do is to identify a short range
of commits in which the error first becomes apparent.

So, over the past few days, I have built and run the test
suite on an unthreaded perl on FreeBSD-13 several dozen
times. I identified all the commits between v5.29.8 and
v5.29.9. To make a long story short, I believe that the
FAIL first appeared in one of these two adjacent commits​:

#####
commit 3b89859
Author​: Karl Williamson <khw@​cpan.org>
AuthorDate​: Sun Mar 17 21​:59​:52 2019 -0600
Commit​: Karl Williamson <khw@​cpan.org>
CommitDate​: Sun Mar 17 22​:17​:28 2019 -0600

regcomp.c​: Use mnemonic for flag parameter

###
commit 6ef7fe5
Author​: Karl Williamson <khw@​cpan.org>
AuthorDate​: Sun Mar 17 22​:11​:04 2019 -0600
Commit​: Karl Williamson <khw@​cpan.org>
CommitDate​: Sun Mar 17 22​:17​:29 2019 -0600

PATCH​: [perl #131551] Too deep regex compilation recursion

This patch, started by Yves Orton, and refined in
consultation with Tony Cook, imposes a maximum depth of
unclosed left parentheses, at which point it croaks. This
is to prevent the segfault in the ticket.

The patch adds a variable that can be set to increase or
decrease this limit at run time (actually regex compilation
time) should this be desired, and hence our pre-determined
limit of 1000 can be changed if necessary.
#####

Why cannot I not say which of these was the first bad
commit? My bisection process first identified
6ef7fe5 as the first bad
commit. Then I said to $self, let's verify this by running
'make test' at the previous commit and this one. I started
to get the failure at the first of the two commits. (I
should note that the two commits entered the repository only
a second apart, so they were probably part of the same 'git
push'.)

I don't understand the internals that were modified in those
two commits, so I'm still at a loss as to explain these
intermittent failures -- especially the fact that they only
occur on non-threaded builds.

The problem persists. See, e.g.,
http​://perl5.test-smoke.org/report/85075.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 14, 2019

From @khwilliamson

On 4/14/19 1​:59 PM, James E Keenan via RT wrote​:

On Fri, 12 Apr 2019 15​:52​:53 GMT, public@​khwilliamson.com wrote​:

One possibility is to bisect this. You don't have that big a range of
commits that it could be IIUC. It should be possible to construct a
bisect that runs a whole 'make test'

We don't have a value for the '--target' switch to
Porting/bisect.pl that would evaluate the overall result of
'make test' or 'make test_harness'. I suppose that we could
create such a target, but that's going to require many hours
of work and more brain power than I currently have.

We could make a shell script that called make test and then returned
appropriately. I could help out devising that should it become necessary.

I did, however, try to bisect manually -- but before I get
to that, let's recall the original problem. We have​:

* A new test failure in a very old, cpan-upstream, test.
* The error only occurs on the development version of one
operating system.
* The error only occurs on unthreaded builds.
* The error never seems to appear when the test file is run
by itself.
* The error appears frequently, but nonetheless
intermittently, when run in test harnesses, such as during
smoke-testing.

In addition, it appears that when the error does appear, it
looks like this​:

#####
cpan/Test-Simple/t/Legacy/More ...# Failed test '$! untouched'
# at t/Legacy/More.t line 184.
# got​: 12
# expected​: 42
# Looks like you failed 1 test of 54.
FAILED at test 54
#####

Since the error is intermittent, it's quite likely that a
single bisection run will be unable to pinpoint the first
bad commit in a reproducible manner. Hence, it's quite
likely that the best we can do is to identify a short range
of commits in which the error first becomes apparent.

So, over the past few days, I have built and run the test
suite on an unthreaded perl on FreeBSD-13 several dozen
times. I identified all the commits between v5.29.8 and
v5.29.9. To make a long story short, I believe that the
FAIL first appeared in one of these two adjacent commits​:

#####
commit 3b89859
Author​: Karl Williamson <khw@​cpan.org>
AuthorDate​: Sun Mar 17 21​:59​:52 2019 -0600
Commit​: Karl Williamson <khw@​cpan.org>
CommitDate​: Sun Mar 17 22​:17​:28 2019 -0600

regcomp.c​: Use mnemonic for flag parameter

###
commit 6ef7fe5
Author​: Karl Williamson <khw@​cpan.org>
AuthorDate​: Sun Mar 17 22​:11​:04 2019 -0600
Commit​: Karl Williamson <khw@​cpan.org>
CommitDate​: Sun Mar 17 22​:17​:29 2019 -0600

PATCH​: [perl #131551] Too deep regex compilation recursion

This patch, started by Yves Orton, and refined in
consultation with Tony Cook, imposes a maximum depth of
unclosed left parentheses, at which point it croaks. This
is to prevent the segfault in the ticket.

The patch adds a variable that can be set to increase or
decrease this limit at run time (actually regex compilation
time) should this be desired, and hence our pre-determined
limit of 1000 can be changed if necessary.
#####

Why cannot I not say which of these was the first bad
commit? My bisection process first identified
6ef7fe5 as the first bad
commit. Then I said to $self, let's verify this by running
'make test' at the previous commit and this one. I started
to get the failure at the first of the two commits. (I
should note that the two commits entered the repository only
a second apart, so they were probably part of the same 'git
push'.)

I don't understand the internals that were modified in those
two commits, so I'm still at a loss as to explain these
intermittent failures -- especially the fact that they only
occur on non-threaded builds.

The problem persists. See, e.g.,
http​://perl5.test-smoke.org/report/85075.

It's much more likely to be 6ef7fe5.
It seems to me that we should revert it, and see if the problem goes
away. If it doesn't revert cleanly, let me know, and I'll set up a
branch for you with it reverted. And if the bug persists revert the
other one. That should tell us if it's really one of these two

Thank you very much.

@p5pRT
Copy link
Author

p5pRT commented Apr 14, 2019

From @jkeenan

On Sun, 14 Apr 2019 20​:22​:59 GMT, public@​khwilliamson.com wrote​:
[snip]

It's much more likely to be 6ef7fe5.
It seems to me that we should revert it, and see if the problem goes
away. If it doesn't revert cleanly, let me know, and I'll set up a
branch for you with it reverted. And if the bug persists revert the
other one. That should tell us if it's really one of these two

The branch idea sounds good, as Carlos's smokers test branches. But, as you anticipated, a simple 'git revert' yields conflicts. See http​://paste.scsys.co.uk/583957. If you let me know how the conflicts should be resolved, I'll push the branch.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 14, 2019

From @jkeenan

On Sun, 14 Apr 2019 20​:22​:59 GMT, public@​khwilliamson.com wrote​:
[snip]

It's much more likely to be 6ef7fe5.
It seems to me that we should revert it, and see if the problem goes
away. If it doesn't revert cleanly, let me know, and I'll set up a
branch for you with it reverted. And if the bug persists revert the
other one. That should tell us if it's really one of these two

Unfortunately, reverting the second commit did not clear up the FAIL​:

#####
smoke-me/jkeenan/rt133958-revert-6ef7fe53
#####
../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ ok
===( 157250;770 1/54 0/? )==========================================
# Failed test '$! untouched'
# at t/Legacy/More.t line 184.
# got​: 12
# expected​: 42
# Looks like you failed 1 test of 54.
../cpan/Test-Simple/t/Legacy/More.t ................................ Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/54 subtests
#####

I will now try to revert only the first of the two commits in question.

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 24, 2019

From @iabyn

On Sun, Apr 14, 2019 at 03​:49​:22PM -0700, James E Keenan via RT wrote​:

Unfortunately, reverting the second commit did not clear up the FAIL​:

#####
smoke-me/jkeenan/rt133958-revert-6ef7fe53
#####
../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ ok
===( 157250;770 1/54 0/? )==========================================
# Failed test '$! untouched'
# at t/Legacy/More.t line 184.
# got​: 12
# expected​: 42
# Looks like you failed 1 test of 54.
../cpan/Test-Simple/t/Legacy/More.t ................................ Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/54 subtests
#####

The test is failing because the process's errno ($!) has not being
restored to its original value at the end of test script. It would be
interesting to see what the error code means. Can you compile and execute
this program on the failing host?

  #include <stdio.h>
  #include <errno.h>

  int main(int argc, char**argv)
  {

  errno = 12;
  perror("this is the error​:");
  return 0;
  }

Failing that, you should be able to find error code 12 in the file
/usr/include/errno.h, or in one of the files '#include'd from there.
For example on linux, I end up at /usr/include/asm-generic/errno-base.h
with the line

  #define ENOMEM 12 /* Out of memory */

It's possible that linux and BSD share the same basic error code numbers,
in which case the smoke may be intermittently failing due to running out
of memory. Which is odd, as that test only takes 16Mb resident on my linux
DEBUGGING build, compared with 34Mb resident for t/re/pat_psycho.t for
example (which doesn't appear to be frequently failing).

--
O Unicef Clearasil!
Gibberish and Drivel!
  -- "Bored of the Rings"

@p5pRT
Copy link
Author

p5pRT commented Apr 24, 2019

From @jkeenan

On Wed, 24 Apr 2019 14​:23​:06 GMT, davem wrote​:

On Sun, Apr 14, 2019 at 03​:49​:22PM -0700, James E Keenan via RT wrote​:

Unfortunately, reverting the second commit did not clear up the FAIL​:

#####
smoke-me/jkeenan/rt133958-revert-6ef7fe53
#####
../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................
ok
===( 157250;770 1/54 0/?
)==========================================
# Failed test '$! untouched'
# at t/Legacy/More.t line 184.
# got​: 12
# expected​: 42
# Looks like you failed 1 test of 54.
../cpan/Test-Simple/t/Legacy/More.t ................................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/54 subtests
#####

The test is failing because the process's errno ($!) has not being
restored to its original value at the end of test script. It would be
interesting to see what the error code means. Can you compile and
execute
this program on the failing host?

#include <stdio.h>
#include <errno.h>

int main(int argc, char**argv)
{

errno = 12;
perror("this is the error​:");
return 0;
}

Failing that, you should be able to find error code 12 in the file
/usr/include/errno.h, or in one of the files '#include'd from there.
For example on linux, I end up at /usr/include/asm-generic/errno-
base.h
with the line

#define ENOMEM 12 /* Out of memory */

It's possible that linux and BSD share the same basic error code
numbers,
in which case the smoke may be intermittently failing due to running
out
of memory. Which is odd, as that test only takes 16Mb resident on my
linux
DEBUGGING build, compared with 34Mb resident for t/re/pat_psycho.t for
example (which doesn't appear to be frequently failing).

On FreeBSD-13​:

#####
[c] $ uname -mrs
FreeBSD 13.0-CURRENT amd64

[c] $ cat 133958-errno.c
#include <stdio.h>
#include <errno.h>

int main(int argc, char**argv)
{

  errno = 12;
  perror("this is the error​:");
  return 0;
}

[c] $ cc --version
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
Target​: x86_64-unknown-freebsd13.0
Thread model​: posix
InstalledDir​: /usr/bin

[c] $ cc 133958-errno.c -o 133958-errno

[c] $ ./133958-errno
this is the error​:: Cannot allocate memory

[c] $ cd /usr/local/include

[include] $ ack '\t(12|42)\t' /usr/include/sys/errno.h
#define ENOMEM 12 /* Cannot allocate memory */
#define ENOPROTOOPT 42 /* Protocol not available */
#####

On Linux​:

#####
$ ack '\t(12|42)\t' /usr/include/asm-generic/errno-base.h
#define ENOMEM 12 /* Out of memory */
#####

So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h.

I wonder if, when cpan/Test-Simple/t/Legacy/More.t, was added back in 2002, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value.

Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2​:

#####
[jkeenan] $ uname -mrs
FreeBSD 11.2-STABLE amd64
[jkeenan] $ ack '\t(12|42)\t' /usr/include/sys/errno.h
#define ENOMEM 12 /* Cannot allocate memory */
#define ENOPROTOOPT 42 /* Protocol not available */
#####
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 25, 2019

From @iabyn

On Wed, Apr 24, 2019 at 08​:10​:19AM -0700, James E Keenan via RT wrote​:

So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h.

I wonder if, when cpan/Test-Simple/t/Legacy/More.t, was added back in 2002, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value.

Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2​:

The error number 42 is irrelevant. It's just an arbitrary but humorous
number that $! is set to at the start of the script, in order to see that
it doesn't get changed during the course of the script (i.e. test that
Test​::More etc localise $! when doing their stuff.)

The fact that its getting changed to ENOMEM is more interesting. This
might indicate a lack of resources on the testing machine, and would also
explain its intermittency.

However, experimenting with 'ulimit -d', as I decrease the available
memory, I see the script usually dying after test 51, never 54.
Which might instead indicate something off with BSD and the setting of
errno.

I'm only seeing this test script fail on one smoker,
  freebsd 13.0-CURRENT (amd64/1 cpu)
and its only been failing since around v5.29.8-61-gb5e20476d6

Test-Simple was upgraded in v5.29.7-102-ga6afdf72cd

@p5pRT
Copy link
Author

p5pRT commented Apr 27, 2019

From @jkeenan

On Thu, 25 Apr 2019 08​:13​:03 GMT, davem wrote​:

On Wed, Apr 24, 2019 at 08​:10​:19AM -0700, James E Keenan via RT wrote​:

So there is no error '42' on Linux -- or at least not in
/usr/include/asm-generic/errno-base.h.

I wonder if, when cpan/Test-Simple/t/Legacy/More.t, was added back in
2002, '42' was considered to be a sufficiently "high" error number as
to be suitable for use in testing as an impossible-to-achieve value.

Note also that error number 42 is not a new one for FreeBSD-13. On
FreeBSD-11.2​:

The error number 42 is irrelevant. It's just an arbitrary but humorous
number that $! is set to at the start of the script, in order to see
that
it doesn't get changed during the course of the script (i.e. test that
Test​::More etc localise $! when doing their stuff.)

The fact that its getting changed to ENOMEM is more interesting. This
might indicate a lack of resources on the testing machine, and would
also
explain its intermittency.

However, experimenting with 'ulimit -d', as I decrease the available
memory, I see the script usually dying after test 51, never 54.
Which might instead indicate something off with BSD and the setting of
errno.

I'm only seeing this test script fail on one smoker,
freebsd 13.0-CURRENT (amd64/1 cpu)
and its only been failing since around v5.29.8-61-gb5e20476d6

Test-Simple was upgraded in v5.29.7-102-ga6afdf72cd

As an experiment, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 29, 2019

From @iabyn

On Sat, Apr 27, 2019 at 06​:02​:20AM -0700, James E Keenan via RT wrote​:

As an experiment, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the
problematic machine when running more More.t? To confirm whether or not
its an issue with limited resources? For that matter, how much VM *is*
available to run the test?

--
The Enterprise's efficient long-range scanners detect a temporal vortex
distortion in good time, allowing it to be safely avoided via a minor
course correction.
  -- Things That Never Happen in "Star Trek" #21

@p5pRT
Copy link
Author

p5pRT commented Apr 29, 2019

From @jkeenan

On Mon, 29 Apr 2019 08​:46​:09 GMT, davem wrote​:

On Sat, Apr 27, 2019 at 06​:02​:20AM -0700, James E Keenan via RT wrote​:

As an experiment, I created a branch in which I reverted that upgrade
to Test-Simple. I didn't really expect it to clear up the anomalous
test failures on non-threaded builds on FreeBSD-13, and it did not.
See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-
revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the
problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or
not
its an issue with limited resources? For that matter, how much VM *is*
available to run the test?

On my VM​:

#####
$ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size'
Memory size​: 512MB
#####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 29, 2019

From @khwilliamson

On 4/29/19 5​:57 AM, James E Keenan via RT wrote​:

On Mon, 29 Apr 2019 08​:46​:09 GMT, davem wrote​:

On Sat, Apr 27, 2019 at 06​:02​:20AM -0700, James E Keenan via RT wrote​:

As an experiment, I created a branch in which I reverted that upgrade
to Test-Simple. I didn't really expect it to clear up the anomalous
test failures on non-threaded builds on FreeBSD-13, and it did not.
See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-
revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the
problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or
not
its an issue with limited resources? For that matter, how much VM *is*
available to run the test?

On my VM​:

#####
$ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size'
Memory size​: 512MB
#####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB.

You'd think 2G would be enough.

Would using $^M help in debugging this situation at all?

@p5pRT
Copy link
Author

p5pRT commented Apr 29, 2019

From @khwilliamson

On 4/29/19 5​:57 AM, James E Keenan via RT wrote​:

On Mon, 29 Apr 2019 08​:46​:09 GMT, davem wrote​:

On Sat, Apr 27, 2019 at 06​:02​:20AM -0700, James E Keenan via RT wrote​:

As an experiment, I created a branch in which I reverted that upgrade
to Test-Simple. I didn't really expect it to clear up the anomalous
test failures on non-threaded builds on FreeBSD-13, and it did not.
See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-
revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the
problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or
not
its an issue with limited resources? For that matter, how much VM *is*
available to run the test?

On my VM​:

#####
$ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size'
Memory size​: 512MB
#####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB.

Another option to rule out that its locale handling, is to add
-Accflags='-DUSE_THREAD_SAFE_LOCALE'
to your Configure options.

The thread safe locale handling is usually only selected on threaded
builds, and is very different from the non-thread safe. It seems
unlikely that the newer handling would work when the old tried and true
doesn't, but it could be a bug in this unstable OS version that gets
bypassed. As this seems pretty easy to try, I think it might be worth a
try.

@p5pRT
Copy link
Author

p5pRT commented May 2, 2019

From @khwilliamson

On 4/29/19 5​:57 AM, James E Keenan via RT wrote​:

On Mon, 29 Apr 2019 08​:46​:09 GMT, davem wrote​:

On Sat, Apr 27, 2019 at 06​:02​:20AM -0700, James E Keenan via RT wrote​:

As an experiment, I created a branch in which I reverted that upgrade
to Test-Simple. I didn't really expect it to clear up the anomalous
test failures on non-threaded builds on FreeBSD-13, and it did not.
See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-
revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the
problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or
not
its an issue with limited resources? For that matter, how much VM *is*
available to run the test?

On my VM​:

#####
$ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size'
Memory size​: 512MB
#####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB.

Tony Cook set me up a VM of FreeBSD 13 with 4096MB. I ran the test
suite several times, and it did not fail for me, but look at this
excerpt from the log

io/through.t ....................................................... ok
Cannot allocate memory
re/fold_grind_a.t .................................................. ok
Cannot allocate memory
re/fold_grind_8.t .................................................. ok

plus several more like that.
Yet all tests passed.

So something is getting errno 12, and printing out the associated
English text, but it doesn't affect the pass/fail of these tests.

That makes it seem likely that its the smaller memory size available in
these other smokers is so small that it can't recover as it does here.
Perhaps someone can shed light on how the recovery is done.

But the question becomes, why is this happening now, and not earlier,
and why only threaded builds? I presume the freebsd smokers of lower
version numbers of the OS have the same amount of vm as this. Is this
correct?

I ran through the test suite with clang asan enabled, and got no errors,
besides undefined behavior. One of the header files on the platform had
code that left shifts a negative integer, and it showed up ubiquitously
until I silenced it. The path to the hdr includes 'machine' as a
directory, so it's quite likely that the behavior it gets is as expected.

I tried to do leak detection, but it says​:

==64995==AddressSanitizer​: detect_leaks is not supported on this platform

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @tonycoz

On Mon, 25 Mar 2019 17​:45​:17 -0700, jkeenan@​pobox.com wrote​:

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two
separate bug tickets for these failures, since one failure appeared long
before the others. This is the first of the two tickets.

Does the attached fix this for you?

I was only able to reproduce the messages from re/fold_grind_a.t, but from the mechanism of the failure (unexpected changes to errno) I'm pretty sure this will solve it for More.t too.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @tonycoz

0001-perl-133958-preserve-errno-on-successful-malloc-real.patch
From 30dd168e6e502186d2de002acaf77323162c97ff Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Fri, 3 May 2019 14:49:50 +1000
Subject: (perl #133958) preserve errno on successful malloc/realloc

In general perl doesn't try to preserve errno (aka $!) since we're
aiming at the same behaviour as for C code - errno is only meaningful
if a function returned an error.

The exception to that is when perl is working without an explicit
request from the perl programmer.

When code is performing assignments, concatenating strings, pushing on
arrays etc, perl is exercising the memory allocation machinery,
calling malloc() and realloc().

It turns out that at least on one platform, realloc() can modify errno
on success.

It appears to be happening when jemalloc (the malloc() implementation
used on FreeBSD) tries to extend a memory arena and fails, leaving the
error number from that failure in errno, from truss:

mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory'

This magic call appears to be a FreeBSD specific mechanism to resize
the anonymous mapping.  On Linux the equivalent seems to be calling
mremap().

In each case for the test code mmap() is successfully called
immediately afterwards:

mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000)

and realloc() succeeds.

glibc() realloc seems to be simpler, AFAICT from reading the code it
only uses mremap() when the memory block is the entire mapping,
ie. for large blocks rather than for memory arenas, and it doesn't
request the same address, so it doesn't fail.

For blocks that are part of arenas, glibc tries to expand in-place
within the current arena (with no extending the arena itself) or falls
back to malloc, so there's no chance for errno to be changed on a
successful realloc().
---
 util.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/util.c b/util.c
index 7276fd901d..27b221837c 100644
--- a/util.c
+++ b/util.c
@@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size)
     dTHX;
 #endif
     Malloc_t ptr;
+    dSAVEDERRNO;
 
 #ifdef USE_MDH
     if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size)
 	Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size);
 #endif
     if (!size) size = 1;	/* malloc(0) is NASTY on our system */
+    SAVE_ERRNO;
 #ifdef PERL_DEBUG_READONLY_COW
     if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE,
 		    MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) {
@@ -154,6 +156,11 @@ Perl_safesysmalloc(MEM_SIZE size)
 #endif
     PERL_ALLOC_CHECK(ptr);
     if (ptr != NULL) {
+        /* malloc() can modify errno() even on success, but since someone
+	   writing perl code doesn't have any control over when perl calls
+	   malloc() we need to hide that.
+	*/
+        RESTORE_ERRNO;
 #ifdef USE_MDH
 	struct perl_memory_debug_header *const header
 	    = (struct perl_memory_debug_header *)ptr;
@@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
 	ptr = safesysmalloc(size);
     }
     else {
+        dSAVE_ERRNO;
 #ifdef USE_MDH
 	where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE);
         if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -276,6 +284,11 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
        might allocate memory/free/move memory, and until we do the fixup, it
        may well be chasing (and writing to) free memory.  */
 	if (ptr != NULL) {
+	    /* realloc() can modify errno() even on success, but since someone
+	       writing perl code doesn't have any control over when perl calls
+	       realloc() we need to hide that.
+	    */
+	    RESTORE_ERRNO;
 #ifdef PERL_TRACK_MEMPOOL
 	    struct perl_memory_debug_header *const header
 		= (struct perl_memory_debug_header *)ptr;
-- 
2.21.0

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @tonycoz

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

On Mon, 25 Mar 2019 17​:45​:17 -0700, jkeenan@​pobox.com wrote​:

We are experiencing massive test failures in our smoke-testing of the
Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two
separate bug tickets for these failures, since one failure appeared
long
before the others. This is the first of the two tickets.

Does the attached fix this for you?

I was only able to reproduce the messages from re/fold_grind_a.t, but
from the mechanism of the failure (unexpected changes to errno) I'm
pretty sure this will solve it for More.t too.

Karl did most of the work in tracking this down.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @hvds

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non-declaration above it, and similarly in the second RESTORE case.

Hugo

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @tonycoz

On Fri, May 03, 2019 at 02​:57​:20AM -0700, Hugo van der Sanden via RT wrote​:

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non-declaration above it, and similarly in the second RESTORE case.

Oops, yep.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @jkeenan

On Fri, 03 May 2019 10​:02​:35 GMT, tonyc wrote​:

On Fri, May 03, 2019 at 02​:57​:20AM -0700, Hugo van der Sanden via RT
wrote​:

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non-
declaration above it, and similarly in the second RESTORE case.

Oops, yep.

Tony

Tony, when you revise your patch to reflect Hugo's comments above, can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @khwilliamson

On 5/3/19 5​:40 AM, James E Keenan via RT wrote​:

On Fri, 03 May 2019 10​:02​:35 GMT, tonyc wrote​:

On Fri, May 03, 2019 at 02​:57​:20AM -0700, Hugo van der Sanden via RT
wrote​:

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non-
declaration above it, and similarly in the second RESTORE case.

Oops, yep.

Tony

Tony, when you revise your patch to reflect Hugo's comments above, can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

I made the changes suggested by Hugo, and pushed the result to
smoke-me/khw-freebsd

I can confirm that this getst rid of the spurious messages in #134076

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @dur-randir

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

Does the attached fix this for you?

Shouldn't there be a new configure probe for a broken malloc, so systems without such behaviour do not suffer performance penalty?

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @jkeenan

On Fri, 03 May 2019 15​:08​:59 GMT, public@​khwilliamson.com wrote​:

On 5/3/19 5​:40 AM, James E Keenan via RT wrote​:

On Fri, 03 May 2019 10​:02​:35 GMT, tonyc wrote​:

On Fri, May 03, 2019 at 02​:57​:20AM -0700, Hugo van der Sanden via RT
wrote​:

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the
non-
declaration above it, and similarly in the second RESTORE case.

Oops, yep.

Tony

Tony, when you revise your patch to reflect Hugo's comments above,
can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

I made the changes suggested by Hugo, and pushed the result to
smoke-me/khw-freebsd

I can confirm that this getst rid of the spurious messages in #134076

Now I'm confused. I've already started smoking the smoke-me/tonyc/133958-realloc-errno-success branch. How does that differ from the smoke-me/khw-freebsd branch?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented May 3, 2019

From @khwilliamson

On 5/3/19 9​:24 AM, James E Keenan via RT wrote​:

On Fri, 03 May 2019 15​:08​:59 GMT, public@​khwilliamson.com wrote​:

On 5/3/19 5​:40 AM, James E Keenan via RT wrote​:

On Fri, 03 May 2019 10​:02​:35 GMT, tonyc wrote​:

On Fri, May 03, 2019 at 02​:57​:20AM -0700, Hugo van der Sanden via RT
wrote​:

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the
non-
declaration above it, and similarly in the second RESTORE case.

Oops, yep.

Tony

Tony, when you revise your patch to reflect Hugo's comments above,
can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

I made the changes suggested by Hugo, and pushed the result to
smoke-me/khw-freebsd

I can confirm that this getst rid of the spurious messages in #134076

Now I'm confused. I've already started smoking the smoke-me/tonyc/133958-realloc-errno-success branch. How does that differ from the smoke-me/khw-freebsd branch?

Thank you very much.

I didn't know Tony had pushed a new branch. There wasn't a reply on the
list, so I thought your request to him had come too late in his day.
But my guess is that his branch and mine identical in functionality, so
continue smoking his.

@p5pRT
Copy link
Author

p5pRT commented May 4, 2019

From @tonycoz

On Fri, 03 May 2019 04​:40​:11 -0700, jkeenan wrote​:

On Fri, 03 May 2019 10​:02​:35 GMT, tonyc wrote​:

On Fri, May 03, 2019 at 02​:57​:20AM -0700, Hugo van der Sanden via RT
wrote​:

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the
non-
declaration above it, and similarly in the second RESTORE case.

Oops, yep.

Tony

Tony, when you revise your patch to reflect Hugo's comments above, can
you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

As you saw, I pushed it last night and this morning here's the patch for the record.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 4, 2019

From @tonycoz

0001-perl-133958-preserve-errno-on-successful-malloc-real.patch
From 373d744e311ee3721ab7a4ee2bd8a33b88a7f387 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Fri, 3 May 2019 14:49:50 +1000
Subject: (perl #133958) preserve errno on successful malloc/realloc

In general perl doesn't try to preserve errno (aka $!) since we're
aiming at the same behaviour as for C code - errno is only meaningful
if a function returned an error.

The exception to that is when perl is working without an explicit
request from the perl programmer.

When code is performing assignments, concatenating strings, pushing on
arrays etc, perl is exercising the memory allocation machinery,
calling malloc() and realloc().

It turns out that at least on one platform, realloc() can modify errno
on success.

It appears to be happening when jemalloc (the malloc() implementation
used on FreeBSD) tries to extend a memory arena and fails, leaving the
error number from that failure in errno, from truss:

mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory'

This magic call appears to be a FreeBSD specific mechanism to resize
the anonymous mapping.  On Linux the equivalent seems to be calling
mremap().

In each case for the test code mmap() is successfully called
immediately afterwards:

mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000)

and realloc() succeeds.

glibc() realloc seems to be simpler, AFAICT from reading the code it
only uses mremap() when the memory block is the entire mapping,
ie. for large blocks rather than for memory arenas, and it doesn't
request the same address, so it doesn't fail.

For blocks that are part of arenas, glibc tries to expand in-place
within the current arena (with no extending the arena itself) or falls
back to malloc, so there's no chance for errno to be changed on a
successful realloc().
---
 util.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/util.c b/util.c
index 7276fd901d..e5320c8b8c 100644
--- a/util.c
+++ b/util.c
@@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size)
     dTHX;
 #endif
     Malloc_t ptr;
+    dSAVEDERRNO;
 
 #ifdef USE_MDH
     if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size)
 	Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size);
 #endif
     if (!size) size = 1;	/* malloc(0) is NASTY on our system */
+    SAVE_ERRNO;
 #ifdef PERL_DEBUG_READONLY_COW
     if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE,
 		    MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) {
@@ -182,6 +184,11 @@ Perl_safesysmalloc(MEM_SIZE size)
 	ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE);
 	DEBUG_m(PerlIO_printf(Perl_debug_log, "0x%" UVxf ": (%05ld) malloc %ld bytes\n",PTR2UV(ptr),(long)PL_an++,(long)size));
 
+        /* malloc() can modify errno() even on success, but since someone
+	   writing perl code doesn't have any control over when perl calls
+	   malloc() we need to hide that.
+	*/
+        RESTORE_ERRNO;
     }
     else {
 #ifdef USE_MDH
@@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
 	ptr = safesysmalloc(size);
     }
     else {
+        dSAVE_ERRNO;
 #ifdef USE_MDH
 	where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE);
         if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -296,6 +304,12 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
 	    maybe_protect_ro(header->prev);
 #endif
 	    ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE);
+
+	    /* realloc() can modify errno() even on success, but since someone
+	       writing perl code doesn't have any control over when perl calls
+	       realloc() we need to hide that.
+	    */
+	    RESTORE_ERRNO;
 	}
 
     /* In particular, must do that fixup above before logging anything via
-- 
2.21.0

@p5pRT
Copy link
Author

p5pRT commented May 4, 2019

From @tonycoz

On Fri, 03 May 2019 08​:20​:06 -0700, randir wrote​:

On Thu, 02 May 2019 22​:11​:19 -0700, tonyc wrote​:

Does the attached fix this for you?

Shouldn't there be a new configure probe for a broken malloc, so
systems without such behaviour do not suffer performance penalty?

Such behaviour is permitted by the C standard.

From the specification for errno (C11)​:

The value of errno may be set to nonzero by a library function call
whether or not there is an error, provided the use of errno is not documented in the
description of the function in this International Standard.

and the standard doesn't document any behaviour for errno for realloc(), malloc(), so the behaviour is permitted.

Similarly POSIX says for errno​:

The setting of errno after a successful call to a function is unspecified unless the description of that function specifies that errno shall not be modified.

and there's nothing in the documentation of realloc() guaranteeing errno is unchanged on success.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 9, 2019

From @tonycoz

On Fri, 03 May 2019 17​:57​:22 -0700, tonyc wrote​:

As you saw, I pushed it last night and this morning here's the patch
for the record.

Applied as 9f30064.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 9, 2019

@tonycoz - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

p5pRT commented May 22, 2019

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Author

p5pRT commented May 22, 2019

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants