segfault / strange match with recursive subpattern (?0) #15228

p5pRT · 2016-03-14T11:03:36Z

Migrated from rt.perl.org#127705 (status was 'resolved')

Searchable as RT127705$

p5pRT · 2016-03-14T11:03:36Z

From lorenz@math.tu-berlin.de

This is a bug report for perl from lorenz@math.tu-berlin.de,
generated with the help of perlbug 1.40 running under perl 5.23.9.

Since the rework related to #126182, the recursive pattern below causes perlblead to segfault. We observed test failures for our software since March 8. There is also a related (probably) invalid match listed further below.

The pattern is almost the same as on 'perlre', except that I replaced (?-1) by (?0). Changing that to (?1) fixes the segfault but should not really make any difference as far as I can see.

Segfaulting expression:

perl5.23.9 -Dr -e 'print "((5maa-maa)(maa-3maa))" =~ /($(?:[^()]++|(?0))*+$)/;

Compiling REx "($(?:[^()]++|(?0))*+$)"
rarest char ) at 0
rarest char ( at 0
Final program:
1: OPEN1 (3)
3: EXACT <(> (5)
5: SUSPEND (35)
7: CURLYX[0]{0,INFTY} (32)
9: BRANCH (26)
10: SUSPEND (31)
12: PLUS (24)
13: ANYOF[^()][0100-INFINITY] (0)
24: SUCCEED (0)
25: TAIL (30)
26: BRANCH (FAIL)
27: GOSUB0[-26:1] (31)
30: TAIL (31)
31: WHILEM[2/2] (0)
32: NOTHING (33)
33: SUCCEED (0)
34: TAIL (35)
35: EXACT <)> (37)
37: CLOSE1 (39)
39: END (0)
anchored "(" at 0 floating ")" at 1..9223372036854775807 (checking floating) minlen 2
Enabling $` $& $' support (0x7).

EXECUTING...

Matching REx "($(?:[^()]++|(?0))*+$)" against "((5maa-maa)(maa-3maa))"
Intuit: trying to determine minimum start position...
doing 'check' fbm scan, [1..22] gave 10
Found floating substr ")" at offset 10 (rx_origin now 0)...
doing 'other' fbm scan, [0..10] gave 0
Found anchored substr "(" at offset 0 (rx_origin now 0)...
(multiline anchor test skipped)
Intuit: Successfully guessed: match at offset 0
0 <> <((5maa-maa> | 0| 1:OPEN1(3)
0 <> <((5maa-maa> | 0| 3:EXACT <(>(5)
1 <(> <(5maa-maa)> | 0| 5:SUSPEND(35)
1 <(> <(5maa-maa)> | 1| 7:CURLYX[0]{0,INFTY}(32)
1 <(> <(5maa-maa)> | 2| 31:WHILEM[2/2](0)
| 2| whilem: matched 0 out of 0..32767
1 <(> <(5maa-maa)> | 3| 9:BRANCH(26)
1 <(> <(5maa-maa)> | 4| 10:SUSPEND(31)
1 <(> <(5maa-maa)> | 5| 12:PLUS(24)
| 5| ANYOF[^()][0100-INFINITY] can match 0 times out of 2147483647...
| 5| failed...
| 4| failed...
1 <(> <(5maa-maa)> | 3| 26:BRANCH(30)
1 <(> <(5maa-maa)> | 4| 27:GOSUB0[-26:1](31)
1 <(> <(5maa-maa)> | 5| 1:OPEN1(3)
1 <(> <(5maa-maa)> | 5| 3:EXACT <(>(5)
2 <((> <5maa-maa)(> | 5| 5:SUSPEND(35)
2 <((> <5maa-maa)(> | 6| 7:CURLYX[0]{0,INFTY}(32)
2 <((> <5maa-maa)(> | 7| 31:WHILEM[2/2](0)
| 7| whilem: matched 0 out of 0..32767
2 <((> <5maa-maa)(> | 8| 9:BRANCH(26)
2 <((> <5maa-maa)(> | 9| 10:SUSPEND(31)
2 <((> <5maa-maa)(> | 10| 12:PLUS(24)
| 10| ANYOF[^()][0100-INFINITY] can match 8 times out of 2147483647...
| 10| EVAL_AB[before] GOSUB0 ce=62d600 recurse_locinput=0
| 10| EVAL trying tail ... (cur_eval=0)
10 <a-maa> <)(maa-3maa> | 11| 31:WHILEM[2/2](0)
| 11| whilem: matched 1 out of 0..32767
10 <a-maa> <)(maa-3maa> | 12| 9:BRANCH(26)
10 <a-maa> <)(maa-3maa> | 13| 10:SUSPEND(31)
10 <a-maa> <)(maa-3maa> | 14| 12:PLUS(24)
| 14| ANYOF[^()][0100-INFINITY] can match 0 times out of 2147483647...
| 14| failed...
| 13| failed...
10 <a-maa> <)(maa-3maa> | 12| 26:BRANCH(30)
10 <a-maa> <)(maa-3maa> | 13| 27:GOSUB0[-26:1](31)
10 <a-maa> <)(maa-3maa> | 14| 1:OPEN1(3)
10 <a-maa> <)(maa-3maa> | 14| 3:EXACT <(>(5)
| 14| failed...
| 13| EVAL_AB[before] GOSUB0 ce=62da38 recurse_locinput=0
| 12| BRANCH failed...
| 11| whilem: failed, trying continuation...
10 <a-maa> <)(maa-3maa> | 12| 32:NOTHING(33)
10 <a-maa> <)(maa-3maa> | 12| 33:SUCCEED(0)
| 12| subpattern success...
| 10| EVAL_AB[before] GOSUB0 ce=62d600 recurse_locinput=62d2f1
10 <a-maa> <)(maa-3maa> | 9| 31:WHILEM[2/2](0)

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaaffbefc in S_regmatch (reginfo=0x7fffffffd330, startpos=0x62d2f0 "((5maa-maa)(maa-3maa))", prog=0x632100)
at regexec.c:7177
7177 min = ARG1(cur_curlyx->u.curlyx.me);
(gdb) bt
#0 0x00002aaaaaffbefc in S_regmatch (reginfo=0x7fffffffd330, startpos=0x62d2f0 "((5maa-maa)(maa-3maa))",
prog=0x632100) at regexec.c:7177
#1 0x00002aaaaafeccae in S_regtry (reginfo=0x7fffffffd330, startposp=0x7fffffffd198) at regexec.c:3615
#2 0x00002aaaaafeb578 in Perl_regexec_flags (rx=0x6293c8, stringarg=0x62d2f0 "((5maa-maa)(maa-3maa))",
strend=0x62d306 "", strbeg=0x62d2f0 "((5maa-maa)(maa-3maa))", minend=0, sv=0x6292a8, data=0x0, flags=97)
at regexec.c:3329
#3 0x00002aaaaaeb16ec in Perl_pp_match () at pp_hot.c:1816
#4 0x00002aaaaae5721d in Perl_runops_debug () at dump.c:2239
#5 0x00002aaaaad51eca in S_run_body (oldscope=1) at perl.c:2466
#6 0x00002aaaaad513b4 in perl_run (my_perl=0x603010) at perl.c:2389
#7 0x0000000000401062 in main (argc=4, argv=0x7fffffffd7b8, env=0x7fffffffd7e0) at perlmain.c:116

Another odd thing appears if you remove the possessiveness from both matches, I do not think that the result is valid for the pattern. Again, changing to (?1) makes it work, i.e. match the full string.

perl5.23.9 -Dr -e 'print "((5maa-maa)(maa-3maa))" =~ /($(?:[^()]+|(?0))*$)/,"\n";'
Compiling REx "($(?:[^()]+|(?0))*$)"
rarest char ) at 0
rarest char ( at 0
Final program:
1: OPEN1 (3)
3: EXACT <(> (5)
5: CURLYX[0]{0,INFTY} (26)
7: BRANCH (20)
8: PLUS (25)
9: ANYOF[^()][0100-INFINITY] (0)
20: BRANCH (FAIL)
21: GOSUB0[-20:1] (25)
24: TAIL (25)
25: WHILEM[2/2] (0)
26: NOTHING (27)
27: EXACT <)> (29)
29: CLOSE1 (31)
31: END (0)
anchored "(" at 0 floating ")" at 1..9223372036854775807 (checking floating) minlen 2
Enabling $` $& $' support (0x7).

EXECUTING...

Matching REx "($(?:[^()]+|(?0))*$)" against "((5maa-maa)(maa-3maa))"
Intuit: trying to determine minimum start position...
doing 'check' fbm scan, [1..22] gave 10
Found floating substr ")" at offset 10 (rx_origin now 0)...
doing 'other' fbm scan, [0..10] gave 0
Found anchored substr "(" at offset 0 (rx_origin now 0)...
(multiline anchor test skipped)
Intuit: Successfully guessed: match at offset 0
0 <> <((5maa-maa> | 0| 1:OPEN1(3)
0 <> <((5maa-maa> | 0| 3:EXACT <(>(5)
1 <(> <(5maa-maa)> | 0| 5:CURLYX[0]{0,INFTY}(26)
1 <(> <(5maa-maa)> | 1| 25:WHILEM[2/2](0)
| 1| whilem: matched 0 out of 0..32767
1 <(> <(5maa-maa)> | 2| 7:BRANCH(20)
1 <(> <(5maa-maa)> | 3| 8:PLUS(25)
| 3| ANYOF[^()][0100-INFINITY] can match 0 times out of 2147483647...
| 3| failed...
1 <(> <(5maa-maa)> | 2| 20:BRANCH(24)
1 <(> <(5maa-maa)> | 3| 21:GOSUB0[-20:1](25)
1 <(> <(5maa-maa)> | 4| 1:OPEN1(3)
1 <(> <(5maa-maa)> | 4| 3:EXACT <(>(5)
2 <((> <5maa-maa)(> | 4| 5:CURLYX[0]{0,INFTY}(26)
2 <((> <5maa-maa)(> | 5| 25:WHILEM[2/2](0)
| 5| whilem: matched 0 out of 0..32767
2 <((> <5maa-maa)(> | 6| 7:BRANCH(20)
2 <((> <5maa-maa)(> | 7| 8:PLUS(25)
| 7| ANYOF[^()][0100-INFINITY] can match 8 times out of 2147483647...
| 7| EVAL_AB[before] GOSUB0 ce=903668 recurse_locinput=0
| 7| EVAL trying tail ... (cur_eval=0)
10 <a-maa> <)(maa-3maa> | 8| 25:WHILEM[2/2](0)
| 8| whilem: matched 1 out of 0..32767
10 <a-maa> <)(maa-3maa> | 9| 7:BRANCH(20)
10 <a-maa> <)(maa-3maa> | 10| 8:PLUS(25)
| 10| ANYOF[^()][0100-INFINITY] can match 0 times out of 2147483647...
| 10| failed...
10 <a-maa> <)(maa-3maa> | 9| 20:BRANCH(24)
10 <a-maa> <)(maa-3maa> | 10| 21:GOSUB0[-20:1](25)
10 <a-maa> <)(maa-3maa> | 11| 1:OPEN1(3)
10 <a-maa> <)(maa-3maa> | 11| 3:EXACT <(>(5)
| 11| failed...
| 10| EVAL_AB[before] GOSUB0 ce=9039b0 recurse_locinput=0
| 9| BRANCH failed...
| 8| whilem: failed, trying continuation...
10 <a-maa> <)(maa-3maa> | 9| 26:NOTHING(27)
10 <a-maa> <)(maa-3maa> | 9| 27:EXACT <)>(29)
11 <-maa)> <(maa-3maa)> | 9| 29:CLOSE1(31)
11 <-maa)> <(maa-3maa)> | 9| 31:END(0)
| 7| EVAL_AB[before] GOSUB0 ce=903668 recurse_locinput=9033d1
| 3| EVAL_AB[before] GOSUB0 ce=903668 recurse_locinput=0
Match successful!
((5maa-maa)
Freeing REx: "($(?:[^()]+|(?0))*$)"

Flags:
category=core
severity=high

Site configuration information for perl 5.23.9:

Configured by lorenz at Mon Mar 14 10:48:06 CET 2016.

Summary of my perl5 (revision 5 version 23 subversion 9) configuration:
Commit id: eb8fc9f
Platform:
osname=linux, osvers=3.11.10-34-default, archname=x86_64-linux
uname='linux borel 3.11.10-34-default #1 smp wed jan 20 14:13:45 utc 2016 (1e76e80) x86_64 x86_64 x86_64 gnulinux '
config_args='-des -Dprefix=/store/borel/lorenz/prefixes/perlblead -Duseshrplib -DEBUGGING -Doptimize=-O0 -g -pipe -Dextras=XML::LibXML XML::LibXSLT XML::Writer Term::ReadLine::Gnu Term::ReadKey -Dusedevel'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
optimize='-O0 -g -pipe',
cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.8.1 20130909 [gcc-4_8-branch revision 202388]', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib64/gcc/x86_64-suse-linux/4.8/include-fixed /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib /lib64 /usr/lib64 /usr/local/lib64
libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.18.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.18'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/store/borel/lorenz/prefixes/perlblead/lib/5.23.9/x86_64-linux/CORE'
cccdlflags='-fPIC', lddlflags='-shared -O0 -g -pipe -L/usr/local/lib -fstack-protector'

@INC for perl 5.23.9:
/store/borel/lorenz/prefix/lib/perl5
/store/borel/lorenz/prefix/lib/perl5
/usr/site-local/lib/perl5/site_perl
/store/borel/lorenz/prefixes/perlblead/lib/site_perl/5.23.9/x86_64-linux
/store/borel/lorenz/prefixes/perlblead/lib/site_perl/5.23.9
/store/borel/lorenz/prefixes/perlblead/lib/5.23.9/x86_64-linux
/store/borel/lorenz/prefixes/perlblead/lib/5.23.9
.

Environment for perl 5.23.9:
HOME=/homes/combi/lorenz
LANG=en_GB.UTF-8
LANGUAGE (unset)
LC_COLLATE=en_GB.UTF-8
LC_CTYPE=en_GB.UTF-8
LC_MESSAGES=en_GB.UTF-8
LD_LIBRARY_PATH=/store/borel/lorenz/prefix/lib:/store/borel/lorenz/prefix/lib:/usr/lib64/mpi/gcc/openmpi/lib64
LOGDIR (unset)
PATH=/store/borel/lorenz/prefix/bin:/store/borel/lorenz/prefix/bin:/homes/combi/lorenz/.cabal/bin:/homes/combi/lorenz/.local/bin:/store/borel/lorenz/prefix/bin:/store/borel/lorenz/prefix/bin:/homes/combi/lorenz/.cabal/bin:/homes/combi/lorenz/.local/bin:/net/TeXLive/bin/x86_64-linux:/usr/lib64/mpi/gcc/openmpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/site-local/bin:/usr/site-local/share/bin
PERL5LIB=/store/borel/lorenz/prefix/lib/perl5:/store/borel/lorenz/prefix/lib/perl5:/usr/site-local/lib/perl5/site_perl
PERL_BADLANG (unset)
PERL_LOCAL_LIB_ROOT=:/store/borel/lorenz/prefix:/store/borel/lorenz/prefix
SHELL=/bin/bash

p5pRT · 2016-03-14T16:56:46Z

From @demerphq

On 14 March 2016 at 12:03, via RT <perlbug-followup@perl.org> wrote:

# New Ticket Created by
# Please include the string: [perl #127705]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=127705 >

This is a bug report for perl from lorenz@math.tu-berlin.de,
generated with the help of perlbug 1.40 running under perl 5.23.9.

Since the rework related to #126182, the recursive pattern below causes perlblead to segfault. We observed test failures for our software since March 8. There is also a related (probably) invalid match listed further below.

Thank you very much for the report, and sorry for the inconvenience caused.

The pattern is almost the same as on 'perlre', except that I replaced (?-1) by (?0). Changing that to (?1) fixes the segfault but should not really make any difference as far as I can see.

That strongly suggests to me what I did wrong.

I will do the best I can to get to the bottom of this.

Thanks a lot,
Yves

p5pRT · 2016-03-14T16:56:46Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2016-03-15T07:35:58Z

From @demerphq

I believe I have fixed both the issues reported in this ticket. If the
original poster could confirm it would be nice.

Yves

p5pRT · 2016-03-15T07:51:41Z

From @mauke

Am Di 15. Mär 2016, 00:35:58, demerphq schrieb:

I believe I have fixed both the issues reported in this ticket. If the
original poster could confirm it would be nice.

Relevant changes seem to be commit 595de76, ce12e25, and d1c49ad (possibly).

p5pRT · 2016-03-15T12:48:51Z

From lorenz@math.tu-berlin.de

Both reported issues (and the original issue in our code) are fixed with
the latest git version (d603419).

Thanks a lot for the quick response and fix,
Benjamin

p5pRT · 2016-03-15T13:56:32Z

From @demerphq

On 15 March 2016 at 08:51, l.mai@web.de via RT
<perlbug-followup@perl.org> wrote:

Am Di 15. Mär 2016, 00:35:58, demerphq schrieb:

I believe I have fixed both the issues reported in this ticket. If the
original poster could confirm it would be nice.

Relevant changes seem to be commit 595de76, ce12e25, and d1c49ad (possibly).

d1c49ad was helpful in understanding the bug, but was not directly
related to fixing the bug itself.

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2016-03-16T09:28:05Z

From @demerphq

On 15 March 2016 at 10:56, Benjamin Lorenz <lorenz@math.tu-berlin.de> wrote:

Both reported issues (and the original issue in our code) are fixed with
the latest git version (d603419).

Thanks a lot for the quick response and fix,

Thanks for the report, and for running bleading edge perls for testing.

It is very helpful.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2016-03-21T09:23:48Z

@iabyn - Status changed from 'open' to 'pending release'

p5pRT · 2016-05-13T04:05:25Z

From @khwilliamson

Thank you for submitting this report. You have helped make Perl better.

With the release of Perl 5.24.0 on May 9, 2016, this and 149 other issues have been resolved.

Perl 5.24.0 may be downloaded via https://metacpan.org/release/RJBS/perl-5.24.0

p5pRT · 2016-05-13T04:05:26Z

@khwilliamson - Status changed from 'pending release' to 'resolved'

p5pRT closed this as completed May 13, 2016

p5pRT added Severity Low type-regex labels Oct 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

segfault / strange match with recursive subpattern (?0) #15228

segfault / strange match with recursive subpattern (?0) #15228

p5pRT commented Mar 14, 2016

p5pRT commented Mar 14, 2016

p5pRT commented Mar 14, 2016

p5pRT commented Mar 14, 2016

p5pRT commented Mar 15, 2016

p5pRT commented Mar 15, 2016

p5pRT commented Mar 15, 2016

p5pRT commented Mar 15, 2016

p5pRT commented Mar 16, 2016

p5pRT commented Mar 21, 2016

p5pRT commented May 13, 2016

p5pRT commented May 13, 2016

segfault / strange match with recursive subpattern (?0) #15228

segfault / strange match with recursive subpattern (?0) #15228

Comments

p5pRT commented Mar 14, 2016

p5pRT commented Mar 14, 2016

From lorenz@math.tu-berlin.de

p5pRT commented Mar 14, 2016

From @demerphq

p5pRT commented Mar 14, 2016

p5pRT commented Mar 15, 2016

From @demerphq

p5pRT commented Mar 15, 2016

From @mauke

p5pRT commented Mar 15, 2016

From lorenz@math.tu-berlin.de

p5pRT commented Mar 15, 2016

From @demerphq

p5pRT commented Mar 16, 2016

From @demerphq

p5pRT commented Mar 21, 2016

p5pRT commented May 13, 2016

From @khwilliamson

p5pRT commented May 13, 2016