Skip Menu |

To: perlbug [...] perl.org
Date: Mon, 25 Mar 2019 20:45:02 -0400
CC: Chad Granum <exodist7 [...] gmail.com>, Carlos Guevara <carlos [...] carlosguevara.com>
Subject: cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: James E Keenan <jkeenan [...] pobox.com>
Download (untitled) / with headers
text/plain 1.7k
We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two separate bug tickets for these failures, since one failure appeared long before the others. This is the first of the two tickets. cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13. See, e.g.: http://perl5.test-smoke.org/report/82470 http://perl5.test-smoke.org/report/82946 Test failures: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status: 1 [stdio] -Dcc=clang++ -Duse64bitall [stdio] -Dcc=clang++ [stdio] -Dcc=clang++ DEBUGGING We only have one regular smoke-testing rig set up for FreeBSD-13, that of course being one of Carlos Guevara's rigs. These failures: i. Do not seem to occur on threaded builds. ii. Seem to occur on most, but not all, non-threaded builds. iii. Appear to have first reported during a smoke-test in http://perl5.test-smoke.org/report/79958, which suggests that the failures were introduced in, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48. That commit is in a deleted branch: smoke-me/khw-vlb. iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing. v. Here's the end of the file which is failing: ##### $ tail cpan/Test-Simple/t/Legacy/More.t ); # rt.cpan.org 53469 is_deeply with regexes is_deeply( qr/a/, qr/a/, "same regex" ); # These two tests must remain at the end. is( $@, $Err, '$@ untouched' ); cmp_ok( $!, '==', $Errno, '$! untouched' ); ##### Thank you very much. Jim Keenan perl perl perl
From: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
To: perl5-porters [...] perl.org
Date: Wed, 27 Mar 2019 21:15:26 -0600
Download (untitled) / with headers
text/plain 2.1k
On 3/25/19 6:45 PM, James E Keenan (via RT) wrote: Show quoted text
> # New Ticket Created by James E Keenan > # Please include the string: [perl #133958] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/Ticket/Display.html?id=133958 > > > > We are experiencing massive test failures in our smoke-testing of the > Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two > separate bug tickets for these failures, since one failure appeared long > before the others. This is the first of the two tickets. > > cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during > smoke-testing of non-threaded builds on FreeBSD-13. See, e.g.: > > http://perl5.test-smoke.org/report/82470 > http://perl5.test-smoke.org/report/82946 > > Test failures: > ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED > 54 Non-zero exit status: 1 > [stdio] -Dcc=clang++ -Duse64bitall > [stdio] -Dcc=clang++ > [stdio] -Dcc=clang++ DEBUGGING > > We only have one regular smoke-testing rig set up for FreeBSD-13, that > of course being one of Carlos Guevara's rigs. These failures: > > i. Do not seem to occur on threaded builds. > > ii. Seem to occur on most, but not all, non-threaded builds. > > iii. Appear to have first reported during a smoke-test in > http://perl5.test-smoke.org/report/79958, which suggests that the > failures were introduced in, or shortly before commit > https://perl5.git.perl.org/perl.git/commitdiff/8021814b48. That commit > is in a deleted branch: smoke-me/khw-vlb. > > iv. Both Carlos and I have attempted to reproduce this failure manually > -- but it only seems to crop up during smoke-testing. > > v. Here's the end of the file which is failing: > > ##### > $ tail cpan/Test-Simple/t/Legacy/More.t > ); > > > # rt.cpan.org 53469 is_deeply with regexes > is_deeply( qr/a/, qr/a/, "same regex" ); > > > # These two tests must remain at the end. > is( $@, $Err, '$@ untouched' ); > cmp_ok( $!, '==', $Errno, '$! untouched' ); > ##### > > Thank you very much. > Jim Keenan > > perl perl perl > >
Have you tried manually running this with address sanitizer?
Date: Thu, 28 Mar 2019 08:21:31 -0400
To: perlbug-followup [...] perl.org
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: James E Keenan <jkeenan [...] pobox.com>
Download (untitled) / with headers
text/plain 2.4k
On 3/27/19 11:15 PM, karl williamson via RT wrote: Show quoted text
> On 3/25/19 6:45 PM, James E Keenan (via RT) wrote:
>> # New Ticket Created by James E Keenan >> # Please include the string: [perl #133958] >> # in the subject line of all future correspondence about this issue. >> # <URL: https://rt.perl.org/Ticket/Display.html?id=133958 > >> >> >> We are experiencing massive test failures in our smoke-testing of the >> Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two >> separate bug tickets for these failures, since one failure appeared long >> before the others. This is the first of the two tickets. >> >> cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during >> smoke-testing of non-threaded builds on FreeBSD-13. See, e.g.: >> >> http://perl5.test-smoke.org/report/82470 >> http://perl5.test-smoke.org/report/82946 >> >> Test failures: >> ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED >> 54 Non-zero exit status: 1 >> [stdio] -Dcc=clang++ -Duse64bitall >> [stdio] -Dcc=clang++ >> [stdio] -Dcc=clang++ DEBUGGING >> >> We only have one regular smoke-testing rig set up for FreeBSD-13, that >> of course being one of Carlos Guevara's rigs. These failures: >> >> i. Do not seem to occur on threaded builds. >> >> ii. Seem to occur on most, but not all, non-threaded builds. >> >> iii. Appear to have first reported during a smoke-test in >> http://perl5.test-smoke.org/report/79958, which suggests that the >> failures were introduced in, or shortly before commit >> https://perl5.git.perl.org/perl.git/commitdiff/8021814b48. That commit >> is in a deleted branch: smoke-me/khw-vlb. >> >> iv. Both Carlos and I have attempted to reproduce this failure manually >> -- but it only seems to crop up during smoke-testing. >> >> v. Here's the end of the file which is failing: >> >> ##### >> $ tail cpan/Test-Simple/t/Legacy/More.t >> ); >> >> >> # rt.cpan.org 53469 is_deeply with regexes >> is_deeply( qr/a/, qr/a/, "same regex" ); >> >> >> # These two tests must remain at the end. >> is( $@, $Err, '$@ untouched' ); >> cmp_ok( $!, '==', $Errno, '$! untouched' ); >> ##### >> >> Thank you very much. >> Jim Keenan >> >> perl perl perl >> >>
> > > Have you tried manually running this with address sanitizer? >
1. I myself have never run address sanitizer. How would I do that? 2. Even if I did that, how would that address the problem that we've only been able to observe this problem during smoke-testing? Thank you very much. Jim Keenan
To: James E Keenan <jkeenan [...] pobox.com>, perlbug-followup [...] perl.org
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Karl Williamson <public [...] khwilliamson.com>
Date: Thu, 28 Mar 2019 08:16:00 -0600
Download (untitled) / with headers
text/plain 3.2k
On 3/28/19 6:21 AM, James E Keenan wrote: Show quoted text
> On 3/27/19 11:15 PM, karl williamson via RT wrote:
>> On 3/25/19 6:45 PM, James E Keenan (via RT) wrote:
>>> # New Ticket Created by  James E Keenan >>> # Please include the string:  [perl #133958] >>> # in the subject line of all future correspondence about this issue. >>> # <URL: https://rt.perl.org/Ticket/Display.html?id=133958 > >>> >>> >>> We are experiencing massive test failures in our smoke-testing of the >>> Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two >>> separate bug tickets for these failures, since one failure appeared long >>> before the others.  This is the first of the two tickets. >>> >>> cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during >>> smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.: >>> >>> http://perl5.test-smoke.org/report/82470 >>> http://perl5.test-smoke.org/report/82946 >>> >>> Test failures: >>> ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED >>> 54 Non-zero exit status: 1 >>>      [stdio] -Dcc=clang++ -Duse64bitall >>>      [stdio] -Dcc=clang++ >>>      [stdio] -Dcc=clang++ DEBUGGING >>> >>> We only have one regular smoke-testing rig set up for FreeBSD-13, that >>> of course being one of Carlos Guevara's rigs.  These failures: >>> >>> i. Do not seem to occur on threaded builds. >>> >>> ii. Seem to occur on most, but not all, non-threaded builds. >>> >>> iii. Appear to have first reported during a smoke-test in >>> http://perl5.test-smoke.org/report/79958, which suggests that the >>> failures were introduced in, or shortly before commit >>> https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit >>> is in a deleted branch: smoke-me/khw-vlb. >>> >>> iv. Both Carlos and I have attempted to reproduce this failure manually >>> -- but it only seems to crop up during smoke-testing. >>> >>> v. Here's the end of the file which is failing: >>> >>> ##### >>> $ tail cpan/Test-Simple/t/Legacy/More.t >>>             ); >>> >>> >>> # rt.cpan.org 53469  is_deeply with regexes >>> is_deeply( qr/a/, qr/a/, "same regex" ); >>> >>> >>> # These two tests must remain at the end. >>> is( $@, $Err,               '$@ untouched' ); >>> cmp_ok( $!, '==', $Errno,   '$! untouched' ); >>> ##### >>> >>> Thank you very much. >>> Jim Keenan >>> >>> perl perl perl >>> >>>
>> >> >> Have you tried manually running this with address sanitizer? >>
> > 1. I myself have never run address sanitizer.  How would I do that?
Instructions are in perlhacktips. It's just some Configure options to clang, which I believe you're already using. Then you run the test suite. Show quoted text
> > 2. Even if I did that, how would that address the problem that we've > only been able to observe this problem during smoke-testing?
There are two likely causes for something showing up only in smoke testing that I can think of off-hand. One is timing, with lots of parallel jobs. And the other is that there is a read of uninitialized memory. Address sanitizer will catch this latter cause. This is similar to valgrind on Linux. I have run valgrind on the test suite recently with no errors beyond the usual two in CPAN (and yes there is a ticket for these open) Show quoted text
> > Thank you very much. > Jim Keenan >
Date: Thu, 28 Mar 2019 11:15:35 -0400
From: James E Keenan <jkeenan [...] pobox.com>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequentfailures in non-threaded builds on FreeBSD-13
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 3.8k
On 3/28/19 10:16 AM, Karl Williamson wrote: Show quoted text
> On 3/28/19 6:21 AM, James E Keenan wrote:
>> On 3/27/19 11:15 PM, karl williamson via RT wrote:
>>> On 3/25/19 6:45 PM, James E Keenan (via RT) wrote:
>>>> # New Ticket Created by  James E Keenan >>>> # Please include the string:  [perl #133958] >>>> # in the subject line of all future correspondence about this issue. >>>> # <URL: https://rt.perl.org/Ticket/Display.html?id=133958 > >>>> >>>> >>>> We are experiencing massive test failures in our smoke-testing of the >>>> Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two >>>> separate bug tickets for these failures, since one failure appeared >>>> long >>>> before the others.  This is the first of the two tickets. >>>> >>>> cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during >>>> smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.: >>>> >>>> http://perl5.test-smoke.org/report/82470 >>>> http://perl5.test-smoke.org/report/82946 >>>> >>>> Test failures: >>>> ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED >>>> 54 Non-zero exit status: 1 >>>>      [stdio] -Dcc=clang++ -Duse64bitall >>>>      [stdio] -Dcc=clang++ >>>>      [stdio] -Dcc=clang++ DEBUGGING >>>> >>>> We only have one regular smoke-testing rig set up for FreeBSD-13, that >>>> of course being one of Carlos Guevara's rigs.  These failures: >>>> >>>> i. Do not seem to occur on threaded builds. >>>> >>>> ii. Seem to occur on most, but not all, non-threaded builds. >>>> >>>> iii. Appear to have first reported during a smoke-test in >>>> http://perl5.test-smoke.org/report/79958, which suggests that the >>>> failures were introduced in, or shortly before commit >>>> https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit >>>> is in a deleted branch: smoke-me/khw-vlb. >>>> >>>> iv. Both Carlos and I have attempted to reproduce this failure manually >>>> -- but it only seems to crop up during smoke-testing. >>>> >>>> v. Here's the end of the file which is failing: >>>> >>>> ##### >>>> $ tail cpan/Test-Simple/t/Legacy/More.t >>>>             ); >>>> >>>> >>>> # rt.cpan.org 53469  is_deeply with regexes >>>> is_deeply( qr/a/, qr/a/, "same regex" ); >>>> >>>> >>>> # These two tests must remain at the end. >>>> is( $@, $Err,               '$@ untouched' ); >>>> cmp_ok( $!, '==', $Errno,   '$! untouched' ); >>>> ##### >>>> >>>> Thank you very much. >>>> Jim Keenan >>>> >>>> perl perl perl >>>> >>>>
>>> >>> >>> Have you tried manually running this with address sanitizer? >>>
>> >> 1. I myself have never run address sanitizer.  How would I do that?
> > Instructions are in perlhacktips.  It's just some Configure options to > clang, which I believe you're already using.  Then you run the test suite.
>> >> 2. Even if I did that, how would that address the problem that we've >> only been able to observe this problem during smoke-testing?
> > There are two likely causes for something showing up only in smoke > testing that I can think of off-hand.  One is timing, with lots of > parallel jobs.
I should note that in this environment $TEST_JOBS is set to 2. So (assuming I understand the test code correctly), only handy00.t and handy01.t really matter. I tried running those two in parallel by: $> cd t; ./perl harness ../ext/XS-APItest/t/handy00.t ../ext/XS-APItest/t/handy01.t; cd - I could not reproduce the problem. But it seems to occur regularly inside Test-Smoke's harness. Show quoted text
> And the other is that there is a read of uninitialized > memory.  Address sanitizer will catch this latter cause.  This is > similar to valgrind on Linux.
I will try to test with valgrind today or tomorrow. Show quoted text
> I have run valgrind on the test suite > recently with no errors beyond the usual two in CPAN (and yes there is a > ticket for these open)
>> >> Thank you very much. >> Jim Keenan >>
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 5.2k
On Thu, 28 Mar 2019 15:15:50 GMT, jkeenan@pobox.com wrote: Show quoted text
> On 3/28/19 10:16 AM, Karl Williamson wrote:
> > On 3/28/19 6:21 AM, James E Keenan wrote:
> >> On 3/27/19 11:15 PM, karl williamson via RT wrote:
> >>> On 3/25/19 6:45 PM, James E Keenan (via RT) wrote:
> >>>> # New Ticket Created by  James E Keenan > >>>> # Please include the string:  [perl #133958] > >>>> # in the subject line of all future correspondence about this issue. > >>>> # <URL: https://rt.perl.org/Ticket/Display.html?id=133958 > > >>>> > >>>> > >>>> We are experiencing massive test failures in our smoke-testing of the > >>>> Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two > >>>> separate bug tickets for these failures, since one failure appeared > >>>> long > >>>> before the others.  This is the first of the two tickets. > >>>> > >>>> cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during > >>>> smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.: > >>>> > >>>> http://perl5.test-smoke.org/report/82470 > >>>> http://perl5.test-smoke.org/report/82946 > >>>> > >>>> Test failures: > >>>> ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED > >>>> 54 Non-zero exit status: 1 > >>>>      [stdio] -Dcc=clang++ -Duse64bitall > >>>>      [stdio] -Dcc=clang++ > >>>>      [stdio] -Dcc=clang++ DEBUGGING > >>>> > >>>> We only have one regular smoke-testing rig set up for FreeBSD-13, that > >>>> of course being one of Carlos Guevara's rigs.  These failures: > >>>> > >>>> i. Do not seem to occur on threaded builds. > >>>> > >>>> ii. Seem to occur on most, but not all, non-threaded builds. > >>>> > >>>> iii. Appear to have first reported during a smoke-test in > >>>> http://perl5.test-smoke.org/report/79958, which suggests that the > >>>> failures were introduced in, or shortly before commit > >>>> https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit > >>>> is in a deleted branch: smoke-me/khw-vlb. > >>>> > >>>> iv. Both Carlos and I have attempted to reproduce this failure manually > >>>> -- but it only seems to crop up during smoke-testing. > >>>> > >>>> v. Here's the end of the file which is failing: > >>>> > >>>> ##### > >>>> $ tail cpan/Test-Simple/t/Legacy/More.t > >>>>             ); > >>>> > >>>> > >>>> # rt.cpan.org 53469  is_deeply with regexes > >>>> is_deeply( qr/a/, qr/a/, "same regex" ); > >>>> > >>>> > >>>> # These two tests must remain at the end. > >>>> is( $@, $Err,               '$@ untouched' ); > >>>> cmp_ok( $!, '==', $Errno,   '$! untouched' ); > >>>> ##### > >>>> > >>>> Thank you very much. > >>>> Jim Keenan > >>>> > >>>> perl perl perl > >>>> > >>>>
> >>> > >>> > >>> Have you tried manually running this with address sanitizer? > >>>
> >> > >> 1. I myself have never run address sanitizer.  How would I do that?
> > > > Instructions are in perlhacktips.  It's just some Configure options to > > clang, which I believe you're already using.  Then you run the test suite.
> >> > >> 2. Even if I did that, how would that address the problem that we've > >> only been able to observe this problem during smoke-testing?
> > > > There are two likely causes for something showing up only in smoke > > testing that I can think of off-hand.  One is timing, with lots of > > parallel jobs.
> > I should note that in this environment $TEST_JOBS is set to 2. So > (assuming I understand the test code correctly), only handy00.t and > handy01.t really matter. I tried running those two in parallel by: > > $> cd t; ./perl harness ../ext/XS-APItest/t/handy00.t > ../ext/XS-APItest/t/handy01.t; cd - > > I could not reproduce the problem. But it seems to occur regularly > inside Test-Smoke's harness. >
> > And the other is that there is a read of uninitialized > > memory.  Address sanitizer will catch this latter cause.  This is > > similar to valgrind on Linux.
> > I will try to test with valgrind today or tomorrow. >
I mistyped. I meant to say that I would try to build a perl with AddressSanitizer. See below. Show quoted text
>
> > I have run valgrind on the test suite > > recently with no errors beyond the usual two in CPAN (and yes there is a > > ticket for these open)
> >>
My efforts to build and test a perl at blead with AddressSanitizer in this FreeBSD-13 were largely unsuccessful. After struggling with the syntax of the ./Configure invocation, I was eventually able to get 'make' to complete successfully. See attachment with name starting with 'perl_V'. However, I could not run 'make test_harness' successfully. Normally, in these VMs I have $TEST_JOBS set to 2. However, 'make test_harness' was 'Killed' in the t/re/*.t tests. See attachment starting with 'test_harness'. I then set TEST_JOBS=1 and ran 'make test' -- so no tests were running in parallel. I got only a little farther on. See attachment starting with 'make_test'. Now, I concede: (a) I've never previously attempted to build with AddressSanitizer; (b) This VM may be too small to run a perl built with AddressSanitizer. But Carlos's FreeBSD-13 smoker, while not large, is larger than mine -- and we're still getting the failures on non-threaded builds in t/Legacy/More.t which are the subject of this RT. See, e.g., http://perl5.test-smoke.org/report/83974. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
Subject: make_test.sanitize.address.tail.txt
t/re/no_utf8_pm ................................................ ok t/re/overload .................................................. ok t/re/pat ....................................................... FAILED--expected 860 tests, saw 108 t/re/pat_advanced .............................................. ok t/re/pat_advanced_thr .......................................... ok t/re/pat_psycho ................................................ # Test process timed out - terminating FAILED--expected 15 tests, saw 0 t/re/pat_psycho_thr ............................................ Killed *** Error code 137 Stop. make: stopped in /home/jkeenan/gitwork/perl
Subject: perl_V.freebsd13.sanitize.address.txt
Summary of my perl5 (revision 5 version 29 subversion 10) configuration: Commit id: 8e48a48cc5006a06110dd20994b419be393e42ce Platform: osname=freebsd osvers=13.0-current archname=amd64-freebsd-thread-multi uname='freebsd perl-reporter-05 13.0-current freebsd 13.0-current r340361 generic amd64 ' config_args='-des -Dusedevel -Duseithreads -Dcc=clang -Accflags=-fsanitize=address -Aldflags=-fsanitize=address -Alddlflags=-shared -fsanitize=address' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='clang' ccflags ='-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fsanitize=address -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2' optimize='-O2' cppflags='-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fsanitize=address -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='' gccversion='4.2.1 Compatible FreeBSD Clang 6.0.1 (tags/RELEASE_601/final 335540)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='clang' ldflags ='-pthread -Wl,-E -fsanitize=address -fstack-protector-strong -L/usr/local/lib' libpth=/usr/lib /usr/local/lib /usr/lib/clang/6.0.1/lib /usr/lib libs=-lpthread -lgdbm -ldl -lm -lcrypt -lutil perllibs=-lpthread -ldl -lm -lcrypt -lutil libc= so=so useshrplib=false libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags=' ' cccdlflags='-DPIC -fPIC' lddlflags='-shared -shared -fsanitize=address -L/usr/local/lib -fstack-protector-strong' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API USE_THREAD_SAFE_LOCALE Built under freebsd Compiled at Mar 29 2019 14:44:36 %ENV: PERL2DIR="/home/jkeenan/gitwork/perl2" PERL_WORKDIR="/home/jkeenan/gitwork/perl" @INC: lib /usr/local/lib/perl5/site_perl/5.29.10/amd64-freebsd-thread-multi /usr/local/lib/perl5/site_perl/5.29.10 /usr/local/lib/perl5/5.29.10/amd64-freebsd-thread-multi /usr/local/lib/perl5/5.29.10
Subject: test_harness.sanitize.address.output.txt

Message body is not shown because it is too large.

Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Dave Mitchell <davem [...] iabyn.com>
To: James E Keenan via RT <perlbug-followup [...] perl.org>
CC: perl5-porters [...] perl.org
Date: Wed, 3 Apr 2019 10:43:08 +0100
Download (untitled) / with headers
text/plain 613b
On Tue, Apr 02, 2019 at 02:31:16PM -0700, James E Keenan via RT wrote: Show quoted text
> Now, I concede: (a) I've never previously attempted to build with > AddressSanitizer; (b) This VM may be too small to run a perl built with > AddressSanitizer.
Address Sanitizer needs lots of memory. My 16Gb system struggles to run 10 tests in parallel when compiled with ASan. Just t/re/pat_psycho.t consumes about 1.7Gb on a threaded debugging build. -- 31 December 1661: "I have newly taken a solemne oath about abstaining from plays". 1 January 1662: "And after ... we went by coach to the play". -- The Diary of Samuel Pepys
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 3.8k
On Tue, 02 Apr 2019 21:31:16 GMT, jkeenan wrote: Show quoted text
> On Thu, 28 Mar 2019 15:15:50 GMT, jkeenan@pobox.com wrote:
> > On 3/28/19 10:16 AM, Karl Williamson wrote:
> > > On 3/28/19 6:21 AM, James E Keenan wrote:
> > >> On 3/27/19 11:15 PM, karl williamson via RT wrote:
> > >>> On 3/25/19 6:45 PM, James E Keenan (via RT) wrote:
> > >>>> # New Ticket Created by  James E Keenan > > >>>> # Please include the string:  [perl #133958] > > >>>> # in the subject line of all future correspondence about this > > >>>> issue. > > >>>> # <URL: https://rt.perl.org/Ticket/Display.html?id=133958 > > > >>>> > > >>>> > > >>>> We are experiencing massive test failures in our smoke-testing > > >>>> of the > > >>>> Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two > > >>>> separate bug tickets for these failures, since one failure > > >>>> appeared > > >>>> long > > >>>> before the others.  This is the first of the two tickets. > > >>>> > > >>>> cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during > > >>>> smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.: > > >>>> > > >>>> http://perl5.test-smoke.org/report/82470 > > >>>> http://perl5.test-smoke.org/report/82946 > > >>>> > > >>>> Test failures: > > >>>> ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... > > >>>> FAILED > > >>>> 54 Non-zero exit status: 1 > > >>>>      [stdio] -Dcc=clang++ -Duse64bitall > > >>>>      [stdio] -Dcc=clang++ > > >>>>      [stdio] -Dcc=clang++ DEBUGGING > > >>>> > > >>>> We only have one regular smoke-testing rig set up for FreeBSD- > > >>>> 13, that > > >>>> of course being one of Carlos Guevara's rigs.  These failures: > > >>>> > > >>>> i. Do not seem to occur on threaded builds. > > >>>> > > >>>> ii. Seem to occur on most, but not all, non-threaded builds. > > >>>> > > >>>> iii. Appear to have first reported during a smoke-test in > > >>>> http://perl5.test-smoke.org/report/79958, which suggests that > > >>>> the > > >>>> failures were introduced in, or shortly before commit > > >>>> https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That > > >>>> commit > > >>>> is in a deleted branch: smoke-me/khw-vlb. > > >>>> > > >>>> iv. Both Carlos and I have attempted to reproduce this failure > > >>>> manually > > >>>> -- but it only seems to crop up during smoke-testing. > > >>>> > > >>>> v. Here's the end of the file which is failing: > > >>>> > > >>>> ##### > > >>>> $ tail cpan/Test-Simple/t/Legacy/More.t > > >>>>             ); > > >>>> > > >>>> > > >>>> # rt.cpan.org 53469  is_deeply with regexes > > >>>> is_deeply( qr/a/, qr/a/, "same regex" ); > > >>>> > > >>>> > > >>>> # These two tests must remain at the end. > > >>>> is( $@, $Err,               '$@ untouched' ); > > >>>> cmp_ok( $!, '==', $Errno,   '$! untouched' ); > > >>>> ##### > > >>>>
I have since built perl5 blead on FreeBSD-13 (i) unthreaded; (ii) without AddressSanitizer; (iii) $TEST_JOBS=1. In this manner: 1. In 'make test_harness' the last test in ../cpan/Test-Simple/t/Legacy/More.t FAILs. ##### ../cpan/Test-Simple/t/Legacy/More.t (Wstat: 256 Tests: 54 Failed: 1) Failed test: 54 Non-zero exit status: 1 ##### 2. However, when, in the same build, I run that file manually, that file PASSes. ##### $ cd t;./perl harness ../cpan/Test-Simple/t/Legacy/More.t; cd - ../cpan/Test-Simple/t/Legacy/More.t .. ok All tests successful. Files=1, Tests=54, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.09 cusr 0.04 csys = 0.15 CPU) Result: PASS ##### The above was built/tested at: ##### commit a719c52dc174d7df309b8c7a4bf318e60fc704e0 (HEAD -> blead, origin/blead, origin/HEAD) Author: Tomasz Konojacki <me@xenu.pl> AuthorDate: Tue Apr 9 23:15:41 2019 +0200 Commit: Steve Hay <steve.m.hay@googlemail.com> CommitDate: Wed Apr 10 22:52:21 2019 +0100 ##### But when I build *threaded*, the test passes during 'make test_harness'. -- James E Keenan (jkeenan@cpan.org)
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Karl Williamson <public [...] khwilliamson.com>
To: perlbug-followup [...] perl.org
CC: perl5-porters [...] perl.org
Date: Fri, 12 Apr 2019 09:52:27 -0600
On 4/11/19 7:57 PM, James E Keenan via RT wrote: Show quoted text
> On Tue, 02 Apr 2019 21:31:16 GMT, jkeenan wrote:
>> On Thu, 28 Mar 2019 15:15:50 GMT, jkeenan@pobox.com wrote:
>>> On 3/28/19 10:16 AM, Karl Williamson wrote:
>>>> On 3/28/19 6:21 AM, James E Keenan wrote:
>>>>> On 3/27/19 11:15 PM, karl williamson via RT wrote:
>>>>>> On 3/25/19 6:45 PM, James E Keenan (via RT) wrote:
>>>>>>> # New Ticket Created by  James E Keenan >>>>>>> # Please include the string:  [perl #133958] >>>>>>> # in the subject line of all future correspondence about this >>>>>>> issue. >>>>>>> # <URL: https://rt.perl.org/Ticket/Display.html?id=133958 > >>>>>>> >>>>>>> >>>>>>> We are experiencing massive test failures in our smoke-testing >>>>>>> of the >>>>>>> Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two >>>>>>> separate bug tickets for these failures, since one failure >>>>>>> appeared >>>>>>> long >>>>>>> before the others.  This is the first of the two tickets. >>>>>>> >>>>>>> cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during >>>>>>> smoke-testing of non-threaded builds on FreeBSD-13.  See, e.g.: >>>>>>> >>>>>>> http://perl5.test-smoke.org/report/82470 >>>>>>> http://perl5.test-smoke.org/report/82946 >>>>>>> >>>>>>> Test failures: >>>>>>> ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... >>>>>>> FAILED >>>>>>> 54 Non-zero exit status: 1 >>>>>>>      [stdio] -Dcc=clang++ -Duse64bitall >>>>>>>      [stdio] -Dcc=clang++ >>>>>>>      [stdio] -Dcc=clang++ DEBUGGING >>>>>>> >>>>>>> We only have one regular smoke-testing rig set up for FreeBSD- >>>>>>> 13, that >>>>>>> of course being one of Carlos Guevara's rigs.  These failures: >>>>>>> >>>>>>> i. Do not seem to occur on threaded builds. >>>>>>> >>>>>>> ii. Seem to occur on most, but not all, non-threaded builds. >>>>>>> >>>>>>> iii. Appear to have first reported during a smoke-test in >>>>>>> http://perl5.test-smoke.org/report/79958, which suggests that >>>>>>> the >>>>>>> failures were introduced in, or shortly before commit >>>>>>> https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That >>>>>>> commit >>>>>>> is in a deleted branch: smoke-me/khw-vlb. >>>>>>> >>>>>>> iv. Both Carlos and I have attempted to reproduce this failure >>>>>>> manually >>>>>>> -- but it only seems to crop up during smoke-testing. >>>>>>> >>>>>>> v. Here's the end of the file which is failing: >>>>>>> >>>>>>> ##### >>>>>>> $ tail cpan/Test-Simple/t/Legacy/More.t >>>>>>>             ); >>>>>>> >>>>>>> >>>>>>> # rt.cpan.org 53469  is_deeply with regexes >>>>>>> is_deeply( qr/a/, qr/a/, "same regex" ); >>>>>>> >>>>>>> >>>>>>> # These two tests must remain at the end. >>>>>>> is( $@, $Err,               '$@ untouched' ); >>>>>>> cmp_ok( $!, '==', $Errno,   '$! untouched' ); >>>>>>> ##### >>>>>>>
> > I have since built perl5 blead on FreeBSD-13 (i) unthreaded; (ii) without AddressSanitizer; (iii) $TEST_JOBS=1. In this manner: > > 1. In 'make test_harness' the last test in ../cpan/Test-Simple/t/Legacy/More.t FAILs. > > ##### > ../cpan/Test-Simple/t/Legacy/More.t (Wstat: 256 Tests: 54 Failed: 1) > Failed test: 54 > Non-zero exit status: 1 > ##### > > 2. However, when, in the same build, I run that file manually, that file PASSes. > > ##### > $ cd t;./perl harness ../cpan/Test-Simple/t/Legacy/More.t; cd - > ../cpan/Test-Simple/t/Legacy/More.t .. ok > All tests successful. > Files=1, Tests=54, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.09 cusr 0.04 csys = 0.15 CPU) > Result: PASS > ##### > > The above was built/tested at: > > ##### > commit a719c52dc174d7df309b8c7a4bf318e60fc704e0 (HEAD -> blead, origin/blead, origin/HEAD) > Author: Tomasz Konojacki <me@xenu.pl> > AuthorDate: Tue Apr 9 23:15:41 2019 +0200 > Commit: Steve Hay <steve.m.hay@googlemail.com> > CommitDate: Wed Apr 10 22:52:21 2019 +0100 > ##### > > But when I build *threaded*, the test passes during 'make test_harness'.
One possibility is to bisect this. You don't have that big a range of commits that it could be IIUC. It should be possible to construct a bisect that runs a whole 'make test' Show quoted text
>
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 4.2k
On Fri, 12 Apr 2019 15:52:53 GMT, public@khwilliamson.com wrote: Show quoted text
> > One possibility is to bisect this. You don't have that big a range of > commits that it could be IIUC. It should be possible to construct a > bisect that runs a whole 'make test'
> >
We don't have a value for the '--target' switch to Porting/bisect.pl that would evaluate the overall result of 'make test' or 'make test_harness'. I suppose that we could create such a target, but that's going to require many hours of work and more brain power than I currently have. I did, however, try to bisect manually -- but before I get to that, let's recall the original problem. We have: * A new test failure in a very old, cpan-upstream, test. * The error only occurs on the development version of one operating system. * The error only occurs on unthreaded builds. * The error never seems to appear when the test file is run by itself. * The error appears frequently, but nonetheless intermittently, when run in test harnesses, such as during smoke-testing. In addition, it appears that when the error does appear, it looks like this: ##### cpan/Test-Simple/t/Legacy/More ...# Failed test '$! untouched' # at t/Legacy/More.t line 184. # got: 12 # expected: 42 # Looks like you failed 1 test of 54. FAILED at test 54 ##### Since the error is intermittent, it's quite likely that a single bisection run will be unable to pinpoint the first bad commit in a reproducible manner. Hence, it's quite likely that the best we can do is to identify a short range of commits in which the error first becomes apparent. So, over the past few days, I have built and run the test suite on an unthreaded perl on FreeBSD-13 several dozen times. I identified all the commits between v5.29.8 and v5.29.9. To make a long story short, I believe that the FAIL first appeared in one of these two adjacent commits: ##### commit 3b89859ad83deaeff7c1ee7911d181ef10236879 Author: Karl Williamson <khw@cpan.org> AuthorDate: Sun Mar 17 21:59:52 2019 -0600 Commit: Karl Williamson <khw@cpan.org> CommitDate: Sun Mar 17 22:17:28 2019 -0600 regcomp.c: Use mnemonic for flag parameter ### commit 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 Author: Karl Williamson <khw@cpan.org> AuthorDate: Sun Mar 17 22:11:04 2019 -0600 Commit: Karl Williamson <khw@cpan.org> CommitDate: Sun Mar 17 22:17:29 2019 -0600 PATCH: [perl #131551] Too deep regex compilation recursion This patch, started by Yves Orton, and refined in consultation with Tony Cook, imposes a maximum depth of unclosed left parentheses, at which point it croaks. This is to prevent the segfault in the ticket. The patch adds a variable that can be set to increase or decrease this limit at run time (actually regex compilation time) should this be desired, and hence our pre-determined limit of 1000 can be changed if necessary. ##### Why cannot I not say which of these was the first bad commit? My bisection process first identified 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 as the first bad commit. Then I said to $self, let's verify this by running 'make test' at the previous commit and this one. I started to get the failure at the first of the two commits. (I should note that the two commits entered the repository only a second apart, so they were probably part of the same 'git push'.) I don't understand the internals that were modified in those two commits, so I'm still at a loss as to explain these intermittent failures -- especially the fact that they only occur on non-threaded builds. The problem persists. See, e.g., http://perl5.test-smoke.org/report/85075. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Karl Williamson <public [...] khwilliamson.com>
To: perlbug-followup [...] perl.org
CC: perl5-porters [...] perl.org
Date: Sun, 14 Apr 2019 14:22:39 -0600
Download (untitled) / with headers
text/plain 4.2k
On 4/14/19 1:59 PM, James E Keenan via RT wrote: Show quoted text
> On Fri, 12 Apr 2019 15:52:53 GMT, public@khwilliamson.com wrote:
>> >> One possibility is to bisect this. You don't have that big a range of >> commits that it could be IIUC. It should be possible to construct a >> bisect that runs a whole 'make test'
>>>
> > We don't have a value for the '--target' switch to > Porting/bisect.pl that would evaluate the overall result of > 'make test' or 'make test_harness'. I suppose that we could > create such a target, but that's going to require many hours > of work and more brain power than I currently have.
We could make a shell script that called make test and then returned appropriately. I could help out devising that should it become necessary. Show quoted text
> > I did, however, try to bisect manually -- but before I get > to that, let's recall the original problem. We have: > > * A new test failure in a very old, cpan-upstream, test. > * The error only occurs on the development version of one > operating system. > * The error only occurs on unthreaded builds. > * The error never seems to appear when the test file is run > by itself. > * The error appears frequently, but nonetheless > intermittently, when run in test harnesses, such as during > smoke-testing. > > In addition, it appears that when the error does appear, it > looks like this: > > ##### > cpan/Test-Simple/t/Legacy/More ...# Failed test '$! untouched' > # at t/Legacy/More.t line 184. > # got: 12 > # expected: 42 > # Looks like you failed 1 test of 54. > FAILED at test 54 > ##### > > Since the error is intermittent, it's quite likely that a > single bisection run will be unable to pinpoint the first > bad commit in a reproducible manner. Hence, it's quite > likely that the best we can do is to identify a short range > of commits in which the error first becomes apparent. > > So, over the past few days, I have built and run the test > suite on an unthreaded perl on FreeBSD-13 several dozen > times. I identified all the commits between v5.29.8 and > v5.29.9. To make a long story short, I believe that the > FAIL first appeared in one of these two adjacent commits: > > ##### > commit 3b89859ad83deaeff7c1ee7911d181ef10236879 > Author: Karl Williamson <khw@cpan.org> > AuthorDate: Sun Mar 17 21:59:52 2019 -0600 > Commit: Karl Williamson <khw@cpan.org> > CommitDate: Sun Mar 17 22:17:28 2019 -0600 > > regcomp.c: Use mnemonic for flag parameter > > ### > commit 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 > Author: Karl Williamson <khw@cpan.org> > AuthorDate: Sun Mar 17 22:11:04 2019 -0600 > Commit: Karl Williamson <khw@cpan.org> > CommitDate: Sun Mar 17 22:17:29 2019 -0600 > > PATCH: [perl #131551] Too deep regex compilation recursion > > This patch, started by Yves Orton, and refined in > consultation with Tony Cook, imposes a maximum depth of > unclosed left parentheses, at which point it croaks. This > is to prevent the segfault in the ticket. > > The patch adds a variable that can be set to increase or > decrease this limit at run time (actually regex compilation > time) should this be desired, and hence our pre-determined > limit of 1000 can be changed if necessary. > ##### > > Why cannot I not say which of these was the first bad > commit? My bisection process first identified > 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 as the first bad > commit. Then I said to $self, let's verify this by running > 'make test' at the previous commit and this one. I started > to get the failure at the first of the two commits. (I > should note that the two commits entered the repository only > a second apart, so they were probably part of the same 'git > push'.) > > I don't understand the internals that were modified in those > two commits, so I'm still at a loss as to explain these > intermittent failures -- especially the fact that they only > occur on non-threaded builds. > > The problem persists. See, e.g., > http://perl5.test-smoke.org/report/85075.
It's much more likely to be 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9. It seems to me that we should revert it, and see if the problem goes away. If it doesn't revert cleanly, let me know, and I'll set up a branch for you with it reverted. And if the bug persists revert the other one. That should tell us if it's really one of these two Show quoted text
> > Thank you very much. >
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 743b
On Sun, 14 Apr 2019 20:22:59 GMT, public@khwilliamson.com wrote: [snip] Show quoted text
> > It's much more likely to be 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9. > It seems to me that we should revert it, and see if the problem goes > away. If it doesn't revert cleanly, let me know, and I'll set up a > branch for you with it reverted. And if the bug persists revert the > other one. That should tell us if it's really one of these two
> >
The branch idea sounds good, as Carlos's smokers test branches. But, as you anticipated, a simple 'git revert' yields conflicts. See http://paste.scsys.co.uk/583957. If you let me know how the conflicts should be resolved, I'll push the branch. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
RT-Send-CC: perl5-porters [...] perl.org
On Sun, 14 Apr 2019 20:22:59 GMT, public@khwilliamson.com wrote: [snip] Show quoted text
> It's much more likely to be 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9. > It seems to me that we should revert it, and see if the problem goes > away. If it doesn't revert cleanly, let me know, and I'll set up a > branch for you with it reverted. And if the bug persists revert the > other one. That should tell us if it's really one of these two
> >
Unfortunately, reverting the second commit did not clear up the FAIL: ##### smoke-me/jkeenan/rt133958-revert-6ef7fe53 ##### ../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ ok ===( 157250;770 1/54 0/? )========================================== # Failed test '$! untouched' # at t/Legacy/More.t line 184. # got: 12 # expected: 42 # Looks like you failed 1 test of 54. ../cpan/Test-Simple/t/Legacy/More.t ................................ Dubious, test returned 1 (wstat 256, 0x100) Failed 1/54 subtests ##### I will now try to revert only the first of the two commits in question. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
To: James E Keenan via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Dave Mitchell <davem [...] iabyn.com>
Date: Wed, 24 Apr 2019 15:22:55 +0100
CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.7k
On Sun, Apr 14, 2019 at 03:49:22PM -0700, James E Keenan via RT wrote: Show quoted text
> Unfortunately, reverting the second commit did not clear up the FAIL: > > ##### > smoke-me/jkeenan/rt133958-revert-6ef7fe53 > ##### > ../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ ok > ===( 157250;770 1/54 0/? )========================================== > # Failed test '$! untouched' > # at t/Legacy/More.t line 184. > # got: 12 > # expected: 42 > # Looks like you failed 1 test of 54. > ../cpan/Test-Simple/t/Legacy/More.t ................................ Dubious, test returned 1 (wstat 256, 0x100) > Failed 1/54 subtests > #####
The test is failing because the process's errno ($!) has not being restored to its original value at the end of test script. It would be interesting to see what the error code means. Can you compile and execute this program on the failing host? #include <stdio.h> #include <errno.h> int main(int argc, char**argv) { errno = 12; perror("this is the error:"); return 0; } Failing that, you should be able to find error code 12 in the file /usr/include/errno.h, or in one of the files '#include'd from there. For example on linux, I end up at /usr/include/asm-generic/errno-base.h with the line #define ENOMEM 12 /* Out of memory */ It's possible that linux and BSD share the same basic error code numbers, in which case the smoke may be intermittently failing due to running out of memory. Which is odd, as that test only takes 16Mb resident on my linux DEBUGGING build, compared with 34Mb resident for t/re/pat_psycho.t for example (which doesn't appear to be frequently failing). -- O Unicef Clearasil! Gibberish and Drivel! -- "Bored of the Rings"
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 3.2k
On Wed, 24 Apr 2019 14:23:06 GMT, davem wrote: Show quoted text
> On Sun, Apr 14, 2019 at 03:49:22PM -0700, James E Keenan via RT wrote:
> > Unfortunately, reverting the second commit did not clear up the FAIL: > > > > ##### > > smoke-me/jkeenan/rt133958-revert-6ef7fe53 > > ##### > > ../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ > > ok > > ===( 157250;770 1/54 0/? > > )========================================== > > # Failed test '$! untouched' > > # at t/Legacy/More.t line 184. > > # got: 12 > > # expected: 42 > > # Looks like you failed 1 test of 54. > > ../cpan/Test-Simple/t/Legacy/More.t ................................ > > Dubious, test returned 1 (wstat 256, 0x100) > > Failed 1/54 subtests > > #####
> > The test is failing because the process's errno ($!) has not being > restored to its original value at the end of test script. It would be > interesting to see what the error code means. Can you compile and > execute > this program on the failing host? > > #include <stdio.h> > #include <errno.h> > > int main(int argc, char**argv) > { > > errno = 12; > perror("this is the error:"); > return 0; > } > > Failing that, you should be able to find error code 12 in the file > /usr/include/errno.h, or in one of the files '#include'd from there. > For example on linux, I end up at /usr/include/asm-generic/errno- > base.h > with the line > > #define ENOMEM 12 /* Out of memory */ > > It's possible that linux and BSD share the same basic error code > numbers, > in which case the smoke may be intermittently failing due to running > out > of memory. Which is odd, as that test only takes 16Mb resident on my > linux > DEBUGGING build, compared with 34Mb resident for t/re/pat_psycho.t for > example (which doesn't appear to be frequently failing).
On FreeBSD-13: ##### [c] $ uname -mrs FreeBSD 13.0-CURRENT amd64 [c] $ cat 133958-errno.c #include <stdio.h> #include <errno.h> int main(int argc, char**argv) { errno = 12; perror("this is the error:"); return 0; } [c] $ cc --version FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) Target: x86_64-unknown-freebsd13.0 Thread model: posix InstalledDir: /usr/bin [c] $ cc 133958-errno.c -o 133958-errno [c] $ ./133958-errno this is the error:: Cannot allocate memory [c] $ cd /usr/local/include [include] $ ack '\t(12|42)\t' /usr/include/sys/errno.h #define ENOMEM 12 /* Cannot allocate memory */ #define ENOPROTOOPT 42 /* Protocol not available */ ##### On Linux: ##### $ ack '\t(12|42)\t' /usr/include/asm-generic/errno-base.h #define ENOMEM 12 /* Out of memory */ ##### So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h. I wonder if, when cpan/Test-Simple/t/Legacy/More.t, was added back in 2002, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value. Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2: ##### [jkeenan] $ uname -mrs FreeBSD 11.2-STABLE amd64 [jkeenan] $ ack '\t(12|42)\t' /usr/include/sys/errno.h #define ENOMEM 12 /* Cannot allocate memory */ #define ENOPROTOOPT 42 /* Protocol not available */ ##### -- James E Keenan (jkeenan@cpan.org)
Date: Thu, 25 Apr 2019 09:12:51 +0100
CC: perl5-porters [...] perl.org
To: James E Keenan via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 1.3k
On Wed, Apr 24, 2019 at 08:10:19AM -0700, James E Keenan via RT wrote: Show quoted text
> So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h. > > I wonder if, when cpan/Test-Simple/t/Legacy/More.t, was added back in 2002, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value. > > Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2:
The error number 42 is irrelevant. It's just an arbitrary but humorous number that $! is set to at the start of the script, in order to see that it doesn't get changed during the course of the script (i.e. test that Test::More etc localise $! when doing their stuff.) The fact that its getting changed to ENOMEM is more interesting. This might indicate a lack of resources on the testing machine, and would also explain its intermittency. However, experimenting with 'ulimit -d', as I decrease the available memory, I see the script usually dying after test 51, never 54. Which might instead indicate something off with BSD and the setting of errno. I'm only seeing this test script fail on one smoker, freebsd 13.0-CURRENT (amd64/1 cpu) and its only been failing since around v5.29.8-61-gb5e20476d6 Test-Simple was upgraded in v5.29.7-102-ga6afdf72cd
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.8k
On Thu, 25 Apr 2019 08:13:03 GMT, davem wrote: Show quoted text
> On Wed, Apr 24, 2019 at 08:10:19AM -0700, James E Keenan via RT wrote:
> > So there is no error '42' on Linux -- or at least not in > > /usr/include/asm-generic/errno-base.h. > > > > I wonder if, when cpan/Test-Simple/t/Legacy/More.t, was added back in > > 2002, '42' was considered to be a sufficiently "high" error number as > > to be suitable for use in testing as an impossible-to-achieve value. > > > > Note also that error number 42 is not a new one for FreeBSD-13. On > > FreeBSD-11.2:
> > The error number 42 is irrelevant. It's just an arbitrary but humorous > number that $! is set to at the start of the script, in order to see > that > it doesn't get changed during the course of the script (i.e. test that > Test::More etc localise $! when doing their stuff.) > > The fact that its getting changed to ENOMEM is more interesting. This > might indicate a lack of resources on the testing machine, and would > also > explain its intermittency. > > However, experimenting with 'ulimit -d', as I decrease the available > memory, I see the script usually dying after test 51, never 54. > Which might instead indicate something off with BSD and the setting of > errno. > > I'm only seeing this test script fail on one smoker, > freebsd 13.0-CURRENT (amd64/1 cpu) > and its only been failing since around v5.29.8-61-gb5e20476d6 > > Test-Simple was upgraded in v5.29.7-102-ga6afdf72cd
As an experiment, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13, and it did not. See http://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-revert-a6afdf72cd. So we can rule out Test-Simple as the cause of the problem. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
Date: Mon, 29 Apr 2019 09:45:51 +0100
CC: perl5-porters [...] perl.org
To: James E Keenan via RT <perlbug-followup [...] perl.org>
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
Download (untitled) / with headers
text/plain 891b
On Sat, Apr 27, 2019 at 06:02:20AM -0700, James E Keenan via RT wrote: Show quoted text
> As an experiment, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13, and it did not. See http://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-revert-a6afdf72cd. > > So we can rule out Test-Simple as the cause of the problem.
Are you able to increase the amount of virtual memory available on the problematic machine when running more More.t? To confirm whether or not its an issue with limited resources? For that matter, how much VM *is* available to run the test? -- The Enterprise's efficient long-range scanners detect a temporal vortex distortion in good time, allowing it to be safely avoided via a minor course correction. -- Things That Never Happen in "Star Trek" #21
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.4k
On Mon, 29 Apr 2019 08:46:09 GMT, davem wrote: Show quoted text
> On Sat, Apr 27, 2019 at 06:02:20AM -0700, James E Keenan via RT wrote:
> > As an experiment, I created a branch in which I reverted that upgrade > > to Test-Simple. I didn't really expect it to clear up the anomalous > > test failures on non-threaded builds on FreeBSD-13, and it did not. > > See http://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- > > revert-a6afdf72cd. > > > > So we can rule out Test-Simple as the cause of the problem.
> > Are you able to increase the amount of virtual memory available on the > problematic machine when running more More.t?
There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it). I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel. Show quoted text
> To confirm whether or > not > its an issue with limited resources? For that matter, how much VM *is* > available to run the test?
On my VM: ##### $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' Memory size: 512MB ##### IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB. -- James E Keenan (jkeenan@cpan.org)
Date: Mon, 29 Apr 2019 15:42:32 -0600
CC: perl5-porters [...] perl.org
To: perlbug-followup [...] perl.org
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Karl Williamson <public [...] khwilliamson.com>
Download (untitled) / with headers
text/plain 1.5k
On 4/29/19 5:57 AM, James E Keenan via RT wrote: Show quoted text
> On Mon, 29 Apr 2019 08:46:09 GMT, davem wrote:
>> On Sat, Apr 27, 2019 at 06:02:20AM -0700, James E Keenan via RT wrote:
>>> As an experiment, I created a branch in which I reverted that upgrade >>> to Test-Simple. I didn't really expect it to clear up the anomalous >>> test failures on non-threaded builds on FreeBSD-13, and it did not. >>> See http://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- >>> revert-a6afdf72cd. >>> >>> So we can rule out Test-Simple as the cause of the problem.
>> >> Are you able to increase the amount of virtual memory available on the >> problematic machine when running more More.t?
> > There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it). > > I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel. > >
>> To confirm whether or >> not >> its an issue with limited resources? For that matter, how much VM *is* >> available to run the test?
> > On my VM: > > ##### > $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' > Memory size: 512MB > ##### > > IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB. >
You'd think 2G would be enough. Would using $^M help in debugging this situation at all?
Date: Mon, 29 Apr 2019 17:48:53 -0600
CC: perl5-porters [...] perl.org
To: perlbug-followup [...] perl.org
From: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
Download (untitled) / with headers
text/plain 1.9k
On 4/29/19 5:57 AM, James E Keenan via RT wrote: Show quoted text
> On Mon, 29 Apr 2019 08:46:09 GMT, davem wrote:
>> On Sat, Apr 27, 2019 at 06:02:20AM -0700, James E Keenan via RT wrote:
>>> As an experiment, I created a branch in which I reverted that upgrade >>> to Test-Simple. I didn't really expect it to clear up the anomalous >>> test failures on non-threaded builds on FreeBSD-13, and it did not. >>> See http://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- >>> revert-a6afdf72cd. >>> >>> So we can rule out Test-Simple as the cause of the problem.
>> >> Are you able to increase the amount of virtual memory available on the >> problematic machine when running more More.t?
> > There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it). > > I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel. > >
>> To confirm whether or >> not >> its an issue with limited resources? For that matter, how much VM *is* >> available to run the test?
> > On my VM: > > ##### > $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' > Memory size: 512MB > ##### > > IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB. >
Another option to rule out that its locale handling, is to add -Accflags='-DUSE_THREAD_SAFE_LOCALE' to your Configure options. The thread safe locale handling is usually only selected on threaded builds, and is very different from the non-thread safe. It seems unlikely that the newer handling would work when the old tried and true doesn't, but it could be a bug in this unstable OS version that gets bypassed. As this seems pretty easy to try, I think it might be worth a try.
CC: perl5-porters [...] perl.org
Date: Wed, 1 May 2019 21:10:18 -0600
From: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
To: perlbug-followup [...] perl.org
Download (untitled) / with headers
text/plain 2.9k
On 4/29/19 5:57 AM, James E Keenan via RT wrote: Show quoted text
> On Mon, 29 Apr 2019 08:46:09 GMT, davem wrote:
>> On Sat, Apr 27, 2019 at 06:02:20AM -0700, James E Keenan via RT wrote:
>>> As an experiment, I created a branch in which I reverted that upgrade >>> to Test-Simple. I didn't really expect it to clear up the anomalous >>> test failures on non-threaded builds on FreeBSD-13, and it did not. >>> See http://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- >>> revert-a6afdf72cd. >>> >>> So we can rule out Test-Simple as the cause of the problem.
>> >> Are you able to increase the amount of virtual memory available on the >> problematic machine when running more More.t?
> > There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it). > > I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel. > >
>> To confirm whether or >> not >> its an issue with limited resources? For that matter, how much VM *is* >> available to run the test?
> > On my VM: > > ##### > $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' > Memory size: 512MB > ##### > > IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB. >
Tony Cook set me up a VM of FreeBSD 13 with 4096MB. I ran the test suite several times, and it did not fail for me, but look at this excerpt from the log io/through.t ....................................................... ok Cannot allocate memory re/fold_grind_a.t .................................................. ok Cannot allocate memory re/fold_grind_8.t .................................................. ok plus several more like that. Yet all tests passed. So something is getting errno 12, and printing out the associated English text, but it doesn't affect the pass/fail of these tests. That makes it seem likely that its the smaller memory size available in these other smokers is so small that it can't recover as it does here. Perhaps someone can shed light on how the recovery is done. But the question becomes, why is this happening now, and not earlier, and why only threaded builds? I presume the freebsd smokers of lower version numbers of the OS have the same amount of vm as this. Is this correct? I ran through the test suite with clang asan enabled, and got no errors, besides undefined behavior. One of the header files on the platform had code that left shifts a negative integer, and it showed up ubiquitously until I silenced it. The path to the hdr includes 'machine' as a directory, so it's quite likely that the behavior it gets is as expected. I tried to do leak detection, but it says: ==64995==AddressSanitizer: detect_leaks is not supported on this platform
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 566b
On Mon, 25 Mar 2019 17:45:17 -0700, jkeenan@pobox.com wrote: Show quoted text
> We are experiencing massive test failures in our smoke-testing of the > Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two > separate bug tickets for these failures, since one failure appeared long > before the others. This is the first of the two tickets.
Does the attached fix this for you? I was only able to reproduce the messages from re/fold_grind_a.t, but from the mechanism of the failure (unexpected changes to errno) I'm pretty sure this will solve it for More.t too. Tony
Subject: 0001-perl-133958-preserve-errno-on-successful-malloc-real.patch
From 30dd168e6e502186d2de002acaf77323162c97ff Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@develop-help.com> Date: Fri, 3 May 2019 14:49:50 +1000 Subject: (perl #133958) preserve errno on successful malloc/realloc In general perl doesn't try to preserve errno (aka $!) since we're aiming at the same behaviour as for C code - errno is only meaningful if a function returned an error. The exception to that is when perl is working without an explicit request from the perl programmer. When code is performing assignments, concatenating strings, pushing on arrays etc, perl is exercising the memory allocation machinery, calling malloc() and realloc(). It turns out that at least on one platform, realloc() can modify errno on success. It appears to be happening when jemalloc (the malloc() implementation used on FreeBSD) tries to extend a memory arena and fails, leaving the error number from that failure in errno, from truss: mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory' This magic call appears to be a FreeBSD specific mechanism to resize the anonymous mapping. On Linux the equivalent seems to be calling mremap(). In each case for the test code mmap() is successfully called immediately afterwards: mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000) and realloc() succeeds. glibc() realloc seems to be simpler, AFAICT from reading the code it only uses mremap() when the memory block is the entire mapping, ie. for large blocks rather than for memory arenas, and it doesn't request the same address, so it doesn't fail. For blocks that are part of arenas, glibc tries to expand in-place within the current arena (with no extending the arena itself) or falls back to malloc, so there's no chance for errno to be changed on a successful realloc(). --- util.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/util.c b/util.c index 7276fd901d..27b221837c 100644 --- a/util.c +++ b/util.c @@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size) dTHX; #endif Malloc_t ptr; + dSAVEDERRNO; #ifdef USE_MDH if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size) Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size); #endif if (!size) size = 1; /* malloc(0) is NASTY on our system */ + SAVE_ERRNO; #ifdef PERL_DEBUG_READONLY_COW if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) { @@ -154,6 +156,11 @@ Perl_safesysmalloc(MEM_SIZE size) #endif PERL_ALLOC_CHECK(ptr); if (ptr != NULL) { + /* malloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + malloc() we need to hide that. + */ + RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr; @@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) ptr = safesysmalloc(size); } else { + dSAVE_ERRNO; #ifdef USE_MDH where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE); if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -276,6 +284,11 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) might allocate memory/free/move memory, and until we do the fixup, it may well be chasing (and writing to) free memory. */ if (ptr != NULL) { + /* realloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + realloc() we need to hide that. + */ + RESTORE_ERRNO; #ifdef PERL_TRACK_MEMPOOL struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr; -- 2.21.0
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 688b
On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote: Show quoted text
> On Mon, 25 Mar 2019 17:45:17 -0700, jkeenan@pobox.com wrote:
> > We are experiencing massive test failures in our smoke-testing of the > > Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two > > separate bug tickets for these failures, since one failure appeared > > long > > before the others. This is the first of the two tickets.
> > Does the attached fix this for you? > > I was only able to reproduce the messages from re/fold_grind_a.t, but > from the mechanism of the failure (unexpected changes to errno) I'm > pretty sure this will solve it for More.t too.
Karl did most of the work in tracking this down. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 330b
On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote: Show quoted text
>+ RESTORE_ERRNO; > #ifdef USE_MDH > struct perl_memory_debug_header *const header > = (struct perl_memory_debug_header *)ptr;
I think you'll need to protect the 'header' declaration from the non-declaration above it, and similarly in the second RESTORE case. Hugo
Date: Fri, 3 May 2019 20:02:27 +1000
To: Hugo van der Sanden via RT <perlbug-followup [...] perl.org>
From: Tony Cook <tony [...] develop-help.com>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 433b
On Fri, May 03, 2019 at 02:57:20AM -0700, Hugo van der Sanden via RT wrote: Show quoted text
> On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
> >+ RESTORE_ERRNO; > > #ifdef USE_MDH > > struct perl_memory_debug_header *const header > > = (struct perl_memory_debug_header *)ptr;
> > I think you'll need to protect the 'header' declaration from the non-declaration above it, and similarly in the second RESTORE case.
Oops, yep. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 737b
On Fri, 03 May 2019 10:02:35 GMT, tonyc wrote: Show quoted text
> On Fri, May 03, 2019 at 02:57:20AM -0700, Hugo van der Sanden via RT > wrote:
> > On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
> > > + RESTORE_ERRNO; > > > #ifdef USE_MDH > > > struct perl_memory_debug_header *const header > > > = (struct perl_memory_debug_header *)ptr;
> > > > I think you'll need to protect the 'header' declaration from the non- > > declaration above it, and similarly in the second RESTORE case.
> > Oops, yep. > > Tony
Tony, when you revise your patch to reflect Hugo's comments above, can you push it to a smoke-me branch? That will be easier for me to test in my FreeBSD-13 VM. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
To: perlbug-followup [...] perl.org
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
From: Karl Williamson <public [...] khwilliamson.com>
Date: Fri, 3 May 2019 09:08:17 -0600
CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 923b
On 5/3/19 5:40 AM, James E Keenan via RT wrote: Show quoted text
> On Fri, 03 May 2019 10:02:35 GMT, tonyc wrote:
>> On Fri, May 03, 2019 at 02:57:20AM -0700, Hugo van der Sanden via RT >> wrote:
>>> On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
>>>> + RESTORE_ERRNO; >>>> #ifdef USE_MDH >>>> struct perl_memory_debug_header *const header >>>> = (struct perl_memory_debug_header *)ptr;
>>> >>> I think you'll need to protect the 'header' declaration from the non- >>> declaration above it, and similarly in the second RESTORE case.
>> >> Oops, yep. >> >> Tony
> > Tony, when you revise your patch to reflect Hugo's comments above, can you push it to a smoke-me branch? > > That will be easier for me to test in my FreeBSD-13 VM. > > Thank you very much. >
I made the changes suggested by Hugo, and pushed the result to smoke-me/khw-freebsd I can confirm that this getst rid of the spurious messages in #134076
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 218b
On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote: Show quoted text
> Does the attached fix this for you?
Shouldn't there be a new configure probe for a broken malloc, so systems without such behaviour do not suffer performance penalty?
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.2k
On Fri, 03 May 2019 15:08:59 GMT, public@khwilliamson.com wrote: Show quoted text
> On 5/3/19 5:40 AM, James E Keenan via RT wrote:
> > On Fri, 03 May 2019 10:02:35 GMT, tonyc wrote:
> >> On Fri, May 03, 2019 at 02:57:20AM -0700, Hugo van der Sanden via RT > >> wrote:
> >>> On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
> >>>> + RESTORE_ERRNO; > >>>> #ifdef USE_MDH > >>>> struct perl_memory_debug_header *const header > >>>> = (struct perl_memory_debug_header *)ptr;
> >>> > >>> I think you'll need to protect the 'header' declaration from the > >>> non- > >>> declaration above it, and similarly in the second RESTORE case.
> >> > >> Oops, yep. > >> > >> Tony
> > > > Tony, when you revise your patch to reflect Hugo's comments above, > > can you push it to a smoke-me branch? > > > > That will be easier for me to test in my FreeBSD-13 VM. > > > > Thank you very much. > >
> > I made the changes suggested by Hugo, and pushed the result to > smoke-me/khw-freebsd > > I can confirm that this getst rid of the spurious messages in #134076
Now I'm confused. I've already started smoking the smoke-me/tonyc/133958-realloc-errno-success branch. How does that differ from the smoke-me/khw-freebsd branch? Thank you very much. -- James E Keenan (jkeenan@cpan.org)
CC: perl5-porters [...] perl.org
Date: Fri, 3 May 2019 11:02:00 -0600
From: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133958] cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13
To: perlbug-followup [...] perl.org
Download (untitled) / with headers
text/plain 1.5k
On 5/3/19 9:24 AM, James E Keenan via RT wrote: Show quoted text
> On Fri, 03 May 2019 15:08:59 GMT, public@khwilliamson.com wrote:
>> On 5/3/19 5:40 AM, James E Keenan via RT wrote:
>>> On Fri, 03 May 2019 10:02:35 GMT, tonyc wrote:
>>>> On Fri, May 03, 2019 at 02:57:20AM -0700, Hugo van der Sanden via RT >>>> wrote:
>>>>> On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
>>>>>> + RESTORE_ERRNO; >>>>>> #ifdef USE_MDH >>>>>> struct perl_memory_debug_header *const header >>>>>> = (struct perl_memory_debug_header *)ptr;
>>>>> >>>>> I think you'll need to protect the 'header' declaration from the >>>>> non- >>>>> declaration above it, and similarly in the second RESTORE case.
>>>> >>>> Oops, yep. >>>> >>>> Tony
>>> >>> Tony, when you revise your patch to reflect Hugo's comments above, >>> can you push it to a smoke-me branch? >>> >>> That will be easier for me to test in my FreeBSD-13 VM. >>> >>> Thank you very much. >>>
>> >> I made the changes suggested by Hugo, and pushed the result to >> smoke-me/khw-freebsd >> >> I can confirm that this getst rid of the spurious messages in #134076
> > Now I'm confused. I've already started smoking the smoke-me/tonyc/133958-realloc-errno-success branch. How does that differ from the smoke-me/khw-freebsd branch? > > Thank you very much. >
I didn't know Tony had pushed a new branch. There wasn't a reply on the list, so I thought your request to him had come too late in his day. But my guess is that his branch and mine identical in functionality, so continue smoking his.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 890b
On Fri, 03 May 2019 04:40:11 -0700, jkeenan wrote: Show quoted text
> On Fri, 03 May 2019 10:02:35 GMT, tonyc wrote:
> > On Fri, May 03, 2019 at 02:57:20AM -0700, Hugo van der Sanden via RT > > wrote:
> > > On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
> > > > + RESTORE_ERRNO; > > > > #ifdef USE_MDH > > > > struct perl_memory_debug_header *const header > > > > = (struct perl_memory_debug_header *)ptr;
> > > > > > I think you'll need to protect the 'header' declaration from the > > > non- > > > declaration above it, and similarly in the second RESTORE case.
> > > > Oops, yep. > > > > Tony
> > Tony, when you revise your patch to reflect Hugo's comments above, can > you push it to a smoke-me branch? > > That will be easier for me to test in my FreeBSD-13 VM. > > Thank you very much.
As you saw, I pushed it last night and this morning here's the patch for the record. Tony
Subject: 0001-perl-133958-preserve-errno-on-successful-malloc-real.patch
From 373d744e311ee3721ab7a4ee2bd8a33b88a7f387 Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@develop-help.com> Date: Fri, 3 May 2019 14:49:50 +1000 Subject: (perl #133958) preserve errno on successful malloc/realloc In general perl doesn't try to preserve errno (aka $!) since we're aiming at the same behaviour as for C code - errno is only meaningful if a function returned an error. The exception to that is when perl is working without an explicit request from the perl programmer. When code is performing assignments, concatenating strings, pushing on arrays etc, perl is exercising the memory allocation machinery, calling malloc() and realloc(). It turns out that at least on one platform, realloc() can modify errno on success. It appears to be happening when jemalloc (the malloc() implementation used on FreeBSD) tries to extend a memory arena and fails, leaving the error number from that failure in errno, from truss: mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory' This magic call appears to be a FreeBSD specific mechanism to resize the anonymous mapping. On Linux the equivalent seems to be calling mremap(). In each case for the test code mmap() is successfully called immediately afterwards: mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000) and realloc() succeeds. glibc() realloc seems to be simpler, AFAICT from reading the code it only uses mremap() when the memory block is the entire mapping, ie. for large blocks rather than for memory arenas, and it doesn't request the same address, so it doesn't fail. For blocks that are part of arenas, glibc tries to expand in-place within the current arena (with no extending the arena itself) or falls back to malloc, so there's no chance for errno to be changed on a successful realloc(). --- util.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/util.c b/util.c index 7276fd901d..e5320c8b8c 100644 --- a/util.c +++ b/util.c @@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size) dTHX; #endif Malloc_t ptr; + dSAVEDERRNO; #ifdef USE_MDH if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size) Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size); #endif if (!size) size = 1; /* malloc(0) is NASTY on our system */ + SAVE_ERRNO; #ifdef PERL_DEBUG_READONLY_COW if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) { @@ -182,6 +184,11 @@ Perl_safesysmalloc(MEM_SIZE size) ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE); DEBUG_m(PerlIO_printf(Perl_debug_log, "0x%" UVxf ": (%05ld) malloc %ld bytes\n",PTR2UV(ptr),(long)PL_an++,(long)size)); + /* malloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + malloc() we need to hide that. + */ + RESTORE_ERRNO; } else { #ifdef USE_MDH @@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) ptr = safesysmalloc(size); } else { + dSAVE_ERRNO; #ifdef USE_MDH where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE); if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -296,6 +304,12 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) maybe_protect_ro(header->prev); #endif ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE); + + /* realloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + realloc() we need to hide that. + */ + RESTORE_ERRNO; } /* In particular, must do that fixup above before logging anything via -- 2.21.0
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 993b
On Fri, 03 May 2019 08:20:06 -0700, randir wrote: Show quoted text
> On Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
> > Does the attached fix this for you?
> > Shouldn't there be a new configure probe for a broken malloc, so > systems without such behaviour do not suffer performance penalty?
Such behaviour is permitted by the C standard. From the specification for errno (C11): The value of errno may be set to nonzero by a library function call whether or not there is an error, provided the use of errno is not documented in the description of the function in this International Standard. and the standard doesn't document any behaviour for errno for realloc(), malloc(), so the behaviour is permitted. Similarly POSIX says for errno: The setting of errno after a successful call to a function is unspecified unless the description of that function specifies that errno shall not be modified. and there's nothing in the documentation of realloc() guaranteeing errno is unchanged on success. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 199b
On Fri, 03 May 2019 17:57:22 -0700, tonyc wrote: Show quoted text
> As you saw, I pushed it last night and this morning here's the patch > for the record.
Applied as 9f300641fc6fb15749f287d7ac7fd6f312b3edd8. Tony
Download (untitled) / with headers
text/plain 313b
Thank you for filing this report. You have helped make Perl better. With the release today of Perl 5.30.0, this and 160 other issues have been resolved. Perl 5.30.0 may be downloaded via: https://metacpan.org/release/XSAWYERX/perl-5.30.0 If you find that the problem persists, feel free to reopen this ticket.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org