Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BBC: 4288c5b broke test in CPAN module XML::Easy #16833

Closed
p5pRT opened this issue Feb 2, 2019 · 11 comments
Closed

BBC: 4288c5b broke test in CPAN module XML::Easy #16833

p5pRT opened this issue Feb 2, 2019 · 11 comments
Labels
BBC Blead Breaks CPAN - changes in blead broke a cpan module(s)

Comments

@p5pRT
Copy link

p5pRT commented Feb 2, 2019

Migrated from rt.perl.org#133816 (status was 'open')

Searchable as RT133816$

@p5pRT
Copy link
Author

p5pRT commented Feb 2, 2019

From @jkeenan

As originally reported in
https://rt.cpan.org/Ticket/Display.html?id=127416, a failure has
appeared during this development cycle in the test suite for CPAN
distribution XML-Easy.

#####
$ ~/bin/perl/dumpjson ZEFRAM.XML-Easy-0.011.log.json
{
  author => "ZEFRAM",
  dist => "XML-Easy",
  distname => "XML-Easy-0.011",
  distversion => 0.011,
  grade => "FAIL",
  prereqs => undef,
  test_output => [
  "Building and testing XML-Easy-0.011",
  "Building XML-Easy",
  "cc
-I/home/jkeenan/var/tad/testing/perl-5.29.4/lib/5.29.4/amd64-freebsd-thread-multi/CORE
-DVERSION=\"0.011\" -DXS_VERSION=\"0.011\" -DPIC -fPIC -c
-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -O2
-pipe -fstack-protector -fno-strict-aliasing -o lib/XML/Easy.o
lib/XML/Easy.c",

"ExtUtils​::Mkbootstrap​::Mkbootstrap('blib/arch/auto/XML/Easy/Easy.bs')",
  "cc -shared -L/usr/local/lib -fstack-protector-strong -o
blib/arch/auto/XML/Easy/Easy.so lib/XML/Easy.o",
  "t/classify.t ........ ok",
  "t/classify_pp.t ..... ok",
  "t/easy_module.t ..... ok",
  "t/easy_module_pp.t .. ok",
  "t/node_basics.t ..... ok",
  "t/node_basics_pp.t .. ok",
  "t/node_object.t ..... ok",
  "t/node_object_pp.t .. ok",
  "t/pod_cvg.t ......... ok",
  "t/pod_cvg_pp.t ...... ok",
  "t/pod_syn.t ......... ok",
  "t/read.t ............ ok",
  "t/read_pp.t ......... ok",
  "WARNING​: Complex regular subexpression recursion limit (65534)
exceeded at t/syntax_main.t line 79, <GEN0> line 1635.",
  "# Looks like your test exited with 25 just after 369.",
  "t/syntax_main.t ..... ",
  "Dubious, test returned 25 (wstat 6400, 0x1900)",
  "Failed 640/1009 subtests ",
  "t/version_synch.t ... ok",
  "t/write.t ........... ok",
  "t/write_pp.t ........ ok",
  "",
  "Test Summary Report",
  "-------------------",
  "t/syntax_main.t (Wstat​: 6400 Tests​: 369 Failed​: 0)",
  " Non-zero exit status​: 25",
  " Parse errors​: Bad plan. You planned 1009 tests but ran 369.",
  "Files=17, Tests=100677, 47 wallclock secs ( 0.46 usr 7.45 sys +
40.20 cusr 6.88 csys = 54.98 CPU)",
  "Result​: FAIL",
  ],
  via => "App​::cpanminus​::reporter 0.17 (1.7044)",
}
#####

I provided a patch (which has not yet been applied). Slaven
subsequently observed that this was probably was related to a change in
blead. Bisection points to​:

#####
HEAD is now at 4288c5b Change REG_INFTY to 2**16-1, instead of 2**15-1
bad - non-zero exit from /tmp/GLUOqiNNx2/bin/perl -I /home/jkeenan/.cpan
-MCPAN​::MyConfig -MCPAN -e
$CPAN​::Config->{build_dir}=q{/tmp/wRUfXf4y2y}; -e install('XML​::Easy');
die unless CPAN​::Shell->expand(Module => 'XML​::Easy')->uptodate;
4288c5b is the first bad commit
commit 4288c5b
Author​: Karl Williamson <khw@​cpan.org>
Date​: Sun Sep 30 12​:10​:17 2018 -0600

Change REG_INFTY to 2**16-1, instead of 2**15-1

This commit doubles the upper limit that unbounded regular expression
quantifiers can match up to. Things like {m,} "+" and "*" now can match
up to U16_MAX times.

We probably should make this a 32 bit value, but doing this doubling was
easy and has fewer potential implications.

See http​://nntp.perl.org/group/perl.perl5.porters/251413 and followups

:040000 040000 5180c35445d2089ef994fcb0634d5e0064197f50
ca323198918692cde3d4d80cca850138f4a1e58b M pod
:100644 100644 ea3a0661d9ffe614167b9e183394e9d89514e1f4
0c5222692138a5614401d0c9b41a1183aee3bc27 M regcomp.h
:040000 040000 fd6ac6b4d41de9a65f73f5ee7c61bbd4b70b9db3
a911dcc50233fe4b31104610c85325128598f631 M t
bisect run success
That took 2991 seconds.
#####

Thank you very much.

@p5pRT
Copy link
Author

p5pRT commented Feb 2, 2019

From @jkeenan

This is perl 5, version 29, subversion 7 (v5.29.7 (v5.29.6-69-gc011174cae)) built for x86_64-linux
(with 93 registered patches, see perl -V for more detail)

Copyright 1987-2018, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http​://www.perl.org/, the Perl Home Page.

@p5pRT
Copy link
Author

p5pRT commented Mar 8, 2019

From @jkeenan

On Sat, 02 Feb 2019 15​:18​:52 GMT, jkeenan@​pobox.com wrote​:

As originally reported in
https://rt.cpan.org/Ticket/Display.html?id=127416, a failure has
appeared during this development cycle in the test suite for CPAN
distribution XML-Easy.

#####
$ ~/bin/perl/dumpjson ZEFRAM.XML-Easy-0.011.log.json
{
author => "ZEFRAM",
dist => "XML-Easy",
distname => "XML-Easy-0.011",
distversion => 0.011,
grade => "FAIL",
prereqs => undef,
test_output => [
"Building and testing XML-Easy-0.011",
"Building XML-Easy",
"cc
-I/home/jkeenan/var/tad/testing/perl-5.29.4/lib/5.29.4/amd64-freebsd-
thread-multi/CORE
-DVERSION=\"0.011\" -DXS_VERSION=\"0.011\" -DPIC -fPIC -c
-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -O2
-pipe -fstack-protector -fno-strict-aliasing -o lib/XML/Easy.o
lib/XML/Easy.c",

"ExtUtils​::Mkbootstrap​::Mkbootstrap('blib/arch/auto/XML/Easy/Easy.bs')",
"cc -shared -L/usr/local/lib -fstack-protector-strong -o
blib/arch/auto/XML/Easy/Easy.so lib/XML/Easy.o",
"t/classify.t ........ ok",
"t/classify_pp.t ..... ok",
"t/easy_module.t ..... ok",
"t/easy_module_pp.t .. ok",
"t/node_basics.t ..... ok",
"t/node_basics_pp.t .. ok",
"t/node_object.t ..... ok",
"t/node_object_pp.t .. ok",
"t/pod_cvg.t ......... ok",
"t/pod_cvg_pp.t ...... ok",
"t/pod_syn.t ......... ok",
"t/read.t ............ ok",
"t/read_pp.t ......... ok",
"WARNING​: Complex regular subexpression recursion limit (65534)
exceeded at t/syntax_main.t line 79, <GEN0> line 1635.",
"# Looks like your test exited with 25 just after 369.",
"t/syntax_main.t ..... ",
"Dubious, test returned 25 (wstat 6400, 0x1900)",
"Failed 640/1009 subtests ",
"t/version_synch.t ... ok",
"t/write.t ........... ok",
"t/write_pp.t ........ ok",
"",
"Test Summary Report",
"-------------------",
"t/syntax_main.t (Wstat​: 6400 Tests​: 369 Failed​: 0)",
" Non-zero exit status​: 25",
" Parse errors​: Bad plan. You planned 1009 tests but ran 369.",
"Files=17, Tests=100677, 47 wallclock secs ( 0.46 usr 7.45 sys
+
40.20 cusr 6.88 csys = 54.98 CPU)",
"Result​: FAIL",
],
via => "App​::cpanminus​::reporter 0.17 (1.7044)",
}
#####

I provided a patch (which has not yet been applied). Slaven
subsequently observed that this was probably was related to a change
in
blead. Bisection points to​:

#####
HEAD is now at 4288c5b Change REG_INFTY to 2**16-1, instead of
2**15-1
bad - non-zero exit from /tmp/GLUOqiNNx2/bin/perl -I
/home/jkeenan/.cpan
-MCPAN​::MyConfig -MCPAN -e
$CPAN​::Config->{build_dir}=q{/tmp/wRUfXf4y2y}; -e
install('XML​::Easy');
die unless CPAN​::Shell->expand(Module => 'XML​::Easy')->uptodate;
4288c5b is the first bad commit
commit 4288c5b
Author​: Karl Williamson <khw@​cpan.org>
Date​: Sun Sep 30 12​:10​:17 2018 -0600

Change REG_INFTY to 2**16-1, instead of 2**15-1

This commit doubles the upper limit that unbounded regular expression
quantifiers can match up to. Things like {m,} "+" and "*" now can
match
up to U16_MAX times.

We probably should make this a 32 bit value, but doing this doubling
was
easy and has fewer potential implications.

See http​://nntp.perl.org/group/perl.perl5.porters/251413 and followups

:040000 040000 5180c35445d2089ef994fcb0634d5e0064197f50
ca323198918692cde3d4d80cca850138f4a1e58b M pod
:100644 100644 ea3a0661d9ffe614167b9e183394e9d89514e1f4
0c5222692138a5614401d0c9b41a1183aee3bc27 M regcomp.h
:040000 040000 fd6ac6b4d41de9a65f73f5ee7c61bbd4b70b9db3
a911dcc50233fe4b31104610c85325128598f631 M t
bisect run success
That took 2991 seconds.
#####

Thank you very much.

Although the test failure reported in this ticket occurred during the 5.29 development cycle, I'm inclined to say that this should *not* be classified as a blocker for perl-5.30 -- at least not until we have other failure reports due to the same commit.

I've supplied a patch to the upstream test file. If it's not the best patch, I'm sure the CPAN maintainer will let us know.

At this point, however, I think the benefit of the change in blead outweighs the (minor) CPAN breakage.

My two cents.

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Mar 8, 2019

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 9, 2019

From @khwilliamson

On 3/8/19 4​:34 PM, James E Keenan via RT wrote​:

On Sat, 02 Feb 2019 15​:18​:52 GMT, jkeenan@​pobox.com wrote​:

As originally reported in
https://rt.cpan.org/Ticket/Display.html?id=127416, a failure has
appeared during this development cycle in the test suite for CPAN
distribution XML-Easy.

#####
$ ~/bin/perl/dumpjson ZEFRAM.XML-Easy-0.011.log.json
{
author => "ZEFRAM",
dist => "XML-Easy",
distname => "XML-Easy-0.011",
distversion => 0.011,
grade => "FAIL",
prereqs => undef,
test_output => [
"Building and testing XML-Easy-0.011",
"Building XML-Easy",
"cc
-I/home/jkeenan/var/tad/testing/perl-5.29.4/lib/5.29.4/amd64-freebsd-
thread-multi/CORE
-DVERSION=\"0.011\" -DXS_VERSION=\"0.011\" -DPIC -fPIC -c
-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -O2
-pipe -fstack-protector -fno-strict-aliasing -o lib/XML/Easy.o
lib/XML/Easy.c",

"ExtUtils​::Mkbootstrap​::Mkbootstrap('blib/arch/auto/XML/Easy/Easy.bs')",
"cc -shared -L/usr/local/lib -fstack-protector-strong -o
blib/arch/auto/XML/Easy/Easy.so lib/XML/Easy.o",
"t/classify.t ........ ok",
"t/classify_pp.t ..... ok",
"t/easy_module.t ..... ok",
"t/easy_module_pp.t .. ok",
"t/node_basics.t ..... ok",
"t/node_basics_pp.t .. ok",
"t/node_object.t ..... ok",
"t/node_object_pp.t .. ok",
"t/pod_cvg.t ......... ok",
"t/pod_cvg_pp.t ...... ok",
"t/pod_syn.t ......... ok",
"t/read.t ............ ok",
"t/read_pp.t ......... ok",
"WARNING​: Complex regular subexpression recursion limit (65534)
exceeded at t/syntax_main.t line 79, <GEN0> line 1635.",
"# Looks like your test exited with 25 just after 369.",
"t/syntax_main.t ..... ",
"Dubious, test returned 25 (wstat 6400, 0x1900)",
"Failed 640/1009 subtests ",
"t/version_synch.t ... ok",
"t/write.t ........... ok",
"t/write_pp.t ........ ok",
"",
"Test Summary Report",
"-------------------",
"t/syntax_main.t (Wstat​: 6400 Tests​: 369 Failed​: 0)",
" Non-zero exit status​: 25",
" Parse errors​: Bad plan. You planned 1009 tests but ran 369.",
"Files=17, Tests=100677, 47 wallclock secs ( 0.46 usr 7.45 sys
+
40.20 cusr 6.88 csys = 54.98 CPU)",
"Result​: FAIL",
],
via => "App​::cpanminus​::reporter 0.17 (1.7044)",
}
#####

I provided a patch (which has not yet been applied). Slaven
subsequently observed that this was probably was related to a change
in
blead. Bisection points to​:

#####
HEAD is now at 4288c5b Change REG_INFTY to 2**16-1, instead of
2**15-1
bad - non-zero exit from /tmp/GLUOqiNNx2/bin/perl -I
/home/jkeenan/.cpan
-MCPAN​::MyConfig -MCPAN -e
$CPAN​::Config->{build_dir}=q{/tmp/wRUfXf4y2y}; -e
install('XML​::Easy');
die unless CPAN​::Shell->expand(Module => 'XML​::Easy')->uptodate;
4288c5b is the first bad commit
commit 4288c5b
Author​: Karl Williamson <khw@​cpan.org>
Date​: Sun Sep 30 12​:10​:17 2018 -0600

Change REG_INFTY to 2**16-1, instead of 2**15-1

This commit doubles the upper limit that unbounded regular expression
quantifiers can match up to. Things like {m,} "+" and "*" now can
match
up to U16_MAX times.

We probably should make this a 32 bit value, but doing this doubling
was
easy and has fewer potential implications.

See http​://nntp.perl.org/group/perl.perl5.porters/251413 and followups

:040000 040000 5180c35445d2089ef994fcb0634d5e0064197f50
ca323198918692cde3d4d80cca850138f4a1e58b M pod
:100644 100644 ea3a0661d9ffe614167b9e183394e9d89514e1f4
0c5222692138a5614401d0c9b41a1183aee3bc27 M regcomp.h
:040000 040000 fd6ac6b4d41de9a65f73f5ee7c61bbd4b70b9db3
a911dcc50233fe4b31104610c85325128598f631 M t
bisect run success
That took 2991 seconds.
#####

Thank you very much.

Although the test failure reported in this ticket occurred during the 5.29 development cycle, I'm inclined to say that this should *not* be classified as a blocker for perl-5.30 -- at least not until we have other failure reports due to the same commit.

I've supplied a patch to the upstream test file. If it's not the best patch, I'm sure the CPAN maintainer will let us know.

At this point, however, I think the benefit of the change in blead outweighs the (minor) CPAN breakage.

My two cents.

Thank you very much.

I added all BBC tickets to be 5.30 blockers to ensure adequate
discussion of them. I don't take a position on any of them. There are
8 open BBC tickets.

@p5pRT
Copy link
Author

p5pRT commented Apr 24, 2019

From @khwilliamson

Removed from blockers list
--
Karl Williamson

@p5pRT
Copy link
Author

p5pRT commented Apr 24, 2019

From @iabyn

On Wed, Apr 24, 2019 at 10​:21​:05AM -0700, Karl Williamson via RT wrote​:

Removed from blockers list

I concur(*).

As as aside, I've just attached an improved patch to the rt.cpan.org
ticket.

(*) "I concur, said William. I agree completely. If you think it's for the
best, then of course do it. Happy to oblige".
  -- good old William the Concurrer.

--
The crew of the Enterprise encounter an alien life form which is
surprisingly neither humanoid nor made from pure energy.
  -- Things That Never Happen in "Star Trek" #22

@jkeenan
Copy link
Contributor

jkeenan commented Jan 31, 2020

From @iabyn

On Wed, Apr 24, 2019 at 10​:21​:05AM -0700, Karl Williamson via RT wrote​:

Removed from blockers list

I concur(*).

As as aside, I've just attached an improved patch to the rt.cpan.org
ticket.

That patch has not yet been applied, nor has a new CPAN release been issued.

@zefram can you take a look?

Thank you very much.
Jim Keenan

@toddr toddr added the BBC Blead Breaks CPAN - changes in blead broke a cpan module(s) label Feb 17, 2020
@toddr
Copy link
Member

toddr commented Feb 17, 2020

--- t/syntax_main.t-	2019-04-24 11:47:08.365496801 +0100
+++ t/syntax_main.t	2019-04-24 12:01:50.459442769 +0100
@@ -35,16 +35,34 @@
 
 # This code checks whether the regexp iteration limit bug (#60034) is
 # present.  The regexp match expression checks for getting the wrong
-# result with a long input, and suffices to diagnose the bug.  However,
-# running that test on a pre-5.10 perl causes the stack to grow large,
+# result with a long input, and suffices to diagnose the bug.
+# for a pattern like /X*/, where X is sub-pattern that can match variable
+# length string, e.g. (ab?), it is currently known that:
+#
+# on < 5.10.0,  the old recursive engine will crash on too long a match;
+# on < 5.29.4,  /X*/ is misinterpreted as /X{0,32767}/
+# on   5.29.4+, /X*/ is misinterpreted as /X{0,65535}/
+#
+# Running that test on a pre-5.10 perl causes the stack to grow large,
 # and if there's a limited stack size then this may overflow it and
 # cause perl to crash.  All pre-5.10 perls have the iteration limit
 # bug, so there's no need to run the proper test on those verions.
 # 5.10 fixed the stack issue, so it's safe to run the proper test there.
-my $have_iterlimit_bug = "$]" < 5.010 || do {
-	local $SIG{__WARN__} = sub { };
-	("a"x40000) !~ /\A(?:X?[a-z])*\z/;
-};
+
+my $iterlimit; # if defined, sets an upper limit for iterations
+
+if ($] < 5.010) {
+    $iterlimit = 0;
+}
+else {
+    local $SIG{__WARN__} = sub { };
+    for my $i (32767, 65535) {
+        if (("a"x($i+1)) !~ /\A(?:X?[a-z])*\z/) {
+            $iterlimit = $i;
+            last;
+        }
+    }
+}
 
 my $data_in = IO::File->new("t/read.data", "r") or die;
 my $line = $data_in->getline;
@@ -75,7 +93,7 @@
 	}
 	SKIP: {
 		skip "perl bug affects long inputs", 2
-			if $have_iterlimit_bug && length($input) >= 32766;
+                    if defined $iterlimit && length($input) >= $iterlimit;
 		is upgraded($input) =~ $recogniser{$prod}, !$syntax_error;
 		is downgraded($input) =~ $recogniser{$prod}, !$syntax_error;
 	}

@toddr
Copy link
Member

toddr commented Feb 17, 2020

If the only thing remaining is a new CPAN release and this isn't a blocker any more, what's the outstanding on this?

@toddr toddr added the Closable? We might be able to close this ticket, but we need to check with the reporter label Feb 17, 2020
@toddr
Copy link
Member

toddr commented Feb 17, 2020

Given the blocker was removed, 5.30 is released, and there is a report to the module in question, I am closing this ticket.

@toddr toddr closed this as completed Feb 17, 2020
@toddr toddr removed the Closable? We might be able to close this ticket, but we need to check with the reporter label Feb 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BBC Blead Breaks CPAN - changes in blead broke a cpan module(s)
Projects
None yet
Development

No branches or pull requests

3 participants