Heap corrupted by regex involving (?| ... ) and ($0) #12096

p5pRT · 2012-05-11T23:02:14Z

Migrated from rt.perl.org#112874 (status was 'resolved')

Searchable as RT112874$

p5pRT · 2012-05-11T23:02:14Z

From @jimav

This is a bug report for perl from james_avera@yahoo.com,
generated with the help of perlbug 1.39 running under perl 5.12.4.

The following Perl program causes a fatal error in glibc indicating
that the heap is probably corrupted. The error messages are written
directly to the terminal (probably /dev/tty), and does not appear on
either stdout or stderr.

#!/usr/bin/perl
use strict; use warnings;

$_ = 'aaa x=${ \$x } bbb a=@a=@{ \@a } ccc';
s{
([\$\@])
(?| # re-use $1, $2 etc in each alternation
([^\w\s\{])
| ( \{
(
(?> \\.
| [^\{\}\\]++
| \{ (?0) \}
)*
)
\}
)
)
}
{replacement}xgs;

Here is what appears on the /dev/tty (captured with 'script'):

*** glibc detected *** /usr/bin/perl: free(): invalid next size
(fast): 0x0000000001198890 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7a6e6)[0x7f6ca7fdf6e6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f6ca7fe39cc]
/usr/lib/libperl.so.5.12(Perl_regexec_flags+0x41c)[0x7f6ca8419e1c]
/usr/lib/libperl.so.5.12(Perl_pp_subst+0x939)[0x7f6ca83b42c9]
/usr/lib/libperl.so.5.12(Perl_runops_standard+0x20)[0x7f6ca83adc10]
/usr/lib/libperl.so.5.12(perl_run+0x35e)[0x7f6ca835782e]
/usr/bin/perl(main+0x149)[0x400d49]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f6ca7f8630d]
/usr/bin/perl[0x400d81]
======= Memory map: ========
00400000-00402000 r-xp 00000000 08:15 1851
/usr/bin/perl
00601000-00602000 r--p 00001000 08:15 1851
/usr/bin/perl
00602000-00603000 rw-p 00002000 08:15 1851
/usr/bin/perl
0116c000-011cf000 rw-p 00000000 00:00 0
[heap]
7f6ca0000000-7f6ca0021000 rw-p 00000000 00:00 0
7f6ca0021000-7f6ca4000000 ---p 00000000 00:00 0
7f6ca6d88000-7f6ca6d9d000 r-xp 00000000 08:15 137088
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f6ca6d9d000-7f6ca6f9c000 ---p 00015000 08:15 137088
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f6ca6f9c000-7f6ca6f9d000 r--p 00014000 08:15 137088
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f6ca6f9d000-7f6ca6f9e000 rw-p 00015000 08:15 137088
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f6ca6f9e000-7f6ca7687000 r--p 00000000 08:15 7656
/usr/lib/locale/locale-archive
7f6ca7687000-7f6ca7690000 r-xp 00000000 08:15 131553
/lib/x86_64-linux-gnu/libcrypt-2.13.so
7f6ca7690000-7f6ca7890000 ---p 00009000 08:15 131553
/lib/x86_64-linux-gnu/libcrypt-2.13.so
7f6ca7890000-7f6ca7891000 r--p 00009000 08:15 131553
/lib/x86_64-linux-gnu/libcrypt-2.13.so
7f6ca7891000-7f6ca7892000 rw-p 0000a000 08:15 131553
/lib/x86_64-linux-gnu/libcrypt-2.13.so
7f6ca7892000-7f6ca78c0000 rw-p 00000000 00:00 0
7f6ca78c0000-7f6ca78d8000 r-xp 00000000 08:15 131576
/lib/x86_64-linux-gnu/libpthread-2.13.so
7f6ca78d8000-7f6ca7ad7000 ---p 00018000 08:15 131576
/lib/x86_64-linux-gnu/libpthread-2.13.so
7f6ca7ad7000-7f6ca7ad8000 r--p 00017000 08:15 131576
/lib/x86_64-linux-gnu/libpthread-2.13.so
7f6ca7ad8000-7f6ca7ad9000 rw-p 00018000 08:15 131576
/lib/x86_64-linux-gnu/libpthread-2.13.so
7f6ca7ad9000-7f6ca7add000 rw-p 00000000 00:00 0
7f6ca7add000-7f6ca7b60000 r-xp 00000000 08:15 131557
/lib/x86_64-linux-gnu/libm-2.13.so
7f6ca7b60000-7f6ca7d5f000 ---p 00083000 08:15 131557
/lib/x86_64-linux-gnu/libm-2.13.so
7f6ca7d5f000-7f6ca7d60000 r--p 00082000 08:15 131557
/lib/x86_64-linux-gnu/libm-2.13.so
7f6ca7d60000-7f6ca7d61000 rw-p 00083000 08:15 131557
/lib/x86_64-linux-gnu/libm-2.13.so
7f6ca7d61000-7f6ca7d63000 r-xp 00000000 08:15 131555
/lib/x86_64-linux-gnu/libdl-2.13.so
7f6ca7d63000-7f6ca7f63000 ---p 00002000 08:15 131555
/lib/x86_64-linux-gnu/libdl-2.13.so
7f6ca7f63000-7f6ca7f64000 r--p 00002000 08:15 131555
/lib/x86_64-linux-gnu/libdl-2.13.so
7f6ca7f64000-7f6ca7f65000 rw-p 00003000 08:15 131555
/lib/x86_64-linux-gnu/libdl-2.13.so
7f6ca7f65000-7f6ca80fc000 r-xp 00000000 08:15 131550
/lib/x86_64-linux-gnu/libc-2.13.so
7f6ca80fc000-7f6ca82fb000 ---p 00197000 08:15 131550
/lib/x86_64-linux-gnu/libc-2.13.so
7f6ca82fb000-7f6ca82ff000 r--p 00196000 08:15 131550
/lib/x86_64-linux-gnu/libc-2.13.so
7f6ca82ff000-7f6ca8300000 rw-p 0019a000 08:15 131550
/lib/x86_64-linux-gnu/libc-2.13.so
7f6ca8300000-7f6ca8306000 rw-p 00000000 00:00 0
7f6ca8306000-7f6ca8472000 r-xp 00000000 08:15 5423
/usr/lib/libperl.so.5.12.4
7f6ca8472000-7f6ca8671000 ---p 0016c000 08:15 5423
/usr/lib/libperl.so.5.12.4
7f6ca8671000-7f6ca8675000 r--p 0016b000 08:15 5423
/usr/lib/libperl.so.5.12.4
7f6ca8675000-7f6ca867a000 rw-p 0016f000 08:15 5423
/usr/lib/libperl.so.5.12.4
7f6ca867a000-7f6ca869b000 r-xp 00000000 08:15 131226
/lib/x86_64-linux-gnu/ld-2.13.so
7f6ca8871000-7f6ca8876000 rw-p 00000000 00:00 0
7f6ca8898000-7f6ca889a000 rw-p 00000000 00:00 0
7f6ca889a000-7f6ca889b000 r--p 00020000 08:15 131226
/lib/x86_64-linux-gnu/ld-2.13.so
7f6ca889b000-7f6ca889d000 rw-p 00021000 08:15 131226
/lib/x86_64-linux-gnu/ld-2.13.so
7fffc1db5000-7fffc1dd6000 rw-p 00000000 00:00 0
[stack]
7fffc1dff000-7fffc1e00000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
Aborted

Flags:
category=core
severity=high

Site configuration information for perl 5.12.4:

Configured by Debian Project at Tue Sep 6 08:08:24 UTC 2011.

Summary of my perl5 (revision 5 version 12 subversion 4) configuration:

Platform:
osname=linux, osvers=2.6.24-28-server,
archname=x86_64-linux-gnu-thread-multi
uname='linux allspice 2.6.24-28-server #1 smp wed aug 18 21:17:51
utc 2010 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
-Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr
-Dprivlib=/usr/share/perl/5.12 -Darchlib=/usr/lib/perl/5.12
-Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl/5.12.4
-Dsitearch=/usr/local/lib/perl/5.12.4 -Dman1dir=/usr/share/man/man1
-Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1
-Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1
-Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm
-Uusesfio -Uusenm -Ui_libutil -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib
-Dlibperl=libperl.so.5.12.4 -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing
-pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.6.1', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib
/usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=, so=so, useshrplib=true, libperl=libperl.so.5.12.4
gnulibc_version='2.13'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib
-fstack-protector'

Locally applied patches:

@INC for perl 5.12.4:
/home/jima/local/share/perl/5.12.4
/home/jima/local/share/perl
/home/jima/lib/perl
/etc/perl
/usr/local/lib/perl/5.12.4
/usr/local/share/perl/5.12.4
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.12
/usr/share/perl/5.12
/usr/local/lib/site_perl
.

Environment for perl 5.12.4:
HOME=/home/jima
LANG=en_US.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH=/home/jima/local/lib
LOGDIR (unset)

PATH=/home/jima/bin:/home/jima/local/bin:/home/jima/jima_tools/linux86_64/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin/X11:/usr/local/bin:/opt/openoffice.org3/program:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/games:.
PERL5LIB=/home/jima/local/share/perl:/home/jima/lib/perl
PERL_BADLANG (unset)
SHELL=/bin/bash

p5pRT · 2012-05-11T23:26:01Z

From @jimav

Sending updated bug demo, as an attachment this time.

If the alternations inside (?| ... ) are swapped, or a 3rd alternation
added, the crash does not happen. See comment in the attached

p5pRT · 2012-05-11T23:26:01Z

From @jimav

bugdemo.pl

p5pRT · 2012-05-11T23:26:02Z

From [Unknown Contact. See original ticket]

Sending updated bug demo, as an attachment this time.

If the alternations inside (?| ... ) are swapped, or a 3rd alternation
added, the crash does not happen. See comment in the attached

p5pRT · 2012-05-11T23:26:02Z

@jimav - Status changed from 'new' to 'open'

p5pRT · 2012-05-13T21:41:17Z

From @iabyn

On Fri, May 11, 2012 at 04:02:15PM -0700, Jim Avera wrote:

The following Perl program causes a fatal error in glibc indicating
that the heap is probably corrupted. The error messages are written
directly to the terminal (probably /dev/tty), and does not appear on
either stdout or stderr.

#!/usr/bin/perl
use strict; use warnings;

$_ = 'aaa x=${ \$x } bbb a=@a=@{ \@a } ccc';
s{
([\$\@])
(?| # re-use $1, $2 etc in each alternation
([^\w\s\{])
| ( \{
(
(?> \\.
| [^\{\}\\]++
| \{ (?0) \}
)*
)
\}
)
)
}
{replacement}xgs;

valgrind shows it crashing on 5.12.4 but not on 5.14.0 or later.

--
Red sky at night - gerroff my land!
Red sky at morning - gerroff my land!
-- old farmers' sayings #14

p5pRT · 2012-05-14T09:18:48Z

From @demerphq

On 13 May 2012 23:40, Dave Mitchell <davem@iabyn.com> wrote:

On Fri, May 11, 2012 at 04:02:15PM -0700, Jim Avera wrote:

The following Perl program causes a fatal error in glibc indicating
that the heap is probably corrupted. The error messages are written
directly to the terminal (probably /dev/tty), and does not appear on
either stdout or stderr.

#!/usr/bin/perl
use strict; use warnings;

$_ = 'aaa x=${ \$x } bbb a=@a=@{ \@a } ccc';
s{
([\$\@])
(?| # re-use $1, $2 etc in each alternation
([^\w\s\{])
| ( \{
(
(?> \\.
| [^\{\}\\]++
| \{ (?0) \}
)*
)
\}
)
)
}
{replacement}xgs;

valgrind shows it crashing on 5.12.4 but not on 5.14.0 or later.

I seem to recall a fix being applied, but I do not remember the
details of who wrote or applied it. Maybe FC?

I think it had to do with miscounting the number of capture buffers.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2012-05-14T10:36:07Z

From @nwc10

On Mon, May 14, 2012 at 11:18:18AM +0200, demerphq wrote:

On 13 May 2012 23:40, Dave Mitchell <davem@iabyn.com> wrote:

On Fri, May 11, 2012 at 04:02:15PM -0700, Jim Avera wrote:

The following Perl program causes a fatal error in glibc indicating
that the heap is probably corrupted. The error messages are written
directly to the terminal (probably /dev/tty), and does not appear on
either stdout or stderr.

#!/usr/bin/perl
use strict; use warnings;

$_ = 'aaa x=${ \$x } bbb a=@a=@{ \@a } ccc';
s{
([\$\@])
(?| # re-use $1, $2 etc in each alternation
([^\w\s\{])
| ( \{
(
(?> \\.
| [^\{\}\\]++
| \{ (?0) \}
)*
)
\}
)
)
}
{replacement}xgs;

valgrind shows it crashing on 5.12.4 but not on 5.14.0 or later.

I seem to recall a fix being applied, but I do not remember the
details of who wrote or applied it. Maybe FC?

I think it had to do with miscounting the number of capture buffers.

Fixed by:

commit fd4be6f
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Feb 25 20:45:08 2011 -0800

[perl #84746] Accessing $2 causes the interpreter to crash

Actually, it doesn't. The original test case was:

#!/usr/bin/perl

my $rx = qr'\$ (?| {(.+?)} | (.+?); | (.+?)(\s) )'x;
my $test = '/home/$USERNAME ';
die unless $test =~ $rx;
print "1: $1\n";
print "2: $2\n" if defined $2;

This crashes even if I put an 'exit' right after the pattern match.

What's happening is that regcomp miscounts the number of capturing
parenthesis pairs (cf. [perl #59734]), so the execution of the regular
expression causes a buffer overflow which overwrites the op_sibling
field of the regcreset op, causing a crash when the op is freed. (The
exact failure may differ between builds, platforms, etc., of course.)

S_reg in regcomp.c keeps a count of the parenthesised groups in a
(?|...) construct, which it updates after each branch, if that branch
has more captures than any previous branch. But it was not updating
the count after the last branch.

So this bug would occur if the last branch had more capturing paren-
theses than any previous branch.

Commit ee91d26, which fixed bug #59734, only solved the problem when
there was just one branch (by updating the count before the loop that
deals with subsequent branches was entered).

This commit changes the code at the end of S_reg to take into account
that RExC_npar (the current paren count) might have been increased by
the last branch.

Since the loop to deal with subsequent branches resets the count
*before* each branch, the code that commit ee91d26 added is no longer
necessary, so this commit removes it.

Inline Patch

diff --git a/regcomp.c b/regcomp.c
index 96a825c..932da1a 100644
--- a/regcomp.c
+++ b/regcomp.c
@@ -7097,12 +7097,6 @@ S_reg(pTHX_ RExC_state_t *pRExC_state, I32 paren, I32 *flagp,U32 depth)
     parse_start = RExC_parse;   /* MJD */
     br = regbranch(pRExC_state, &flags, 1,depth+1);
 
-    if (freeze_paren) {
-        if (RExC_npar > after_freeze)
-            after_freeze = RExC_npar;
-        RExC_npar = freeze_paren;
-    }
-
     /*     branch_len = (paren != 0); */
 
     if (br == NULL)
@@ -7246,7 +7240,7 @@ S_reg(pTHX_ RExC_state_t *pRExC_state, I32 paren, I32 *flagp,U32 depth)
     if (RExC_in_lookbehind) {
 	RExC_in_lookbehind--;
     }
-    if (after_freeze)
+    if (after_freeze > RExC_npar)
         RExC_npar = after_freeze;
     return(ret);
 }
diff --git a/t/re/re_tests b/t/re/re_tests
index 5441437..924434c 100644
--- a/t/re/re_tests
+++ b/t/re/re_tests
@@ -1329,6 +1329,9 @@ X(\w+)(?=\s)|X(\w+)	Xab	y	[$1-$2]	[-ab]
 (?|(?<foo>x)|(?<bar>y))	x	yM	$+{foo}	x	miniperl cannot load Tie::Hash::NamedCapture
 (?|(?<bar>y)|(?<foo>x))	x	yM	$+{foo}	x	miniperl cannot load Tie::Hash::NamedCapture
 (?<bar>)(?|(?<foo>x))	x	yM	$+{foo}	x	miniperl cannot load Tie::Hash::NamedCapture
+# Used to crash, because the last branch was ignored when the parens
+# were counted:
+(?|(b)|()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()()(a))	a	y	$&	a
 
 #Bug #41492
 (?(DEFINE)(?<A>(?&B)+)(?<B>a))(?&A)	a	y	$&	a

git bisect is everyone's friend.

Nicholas Clark

p5pRT · 2012-05-14T12:12:01Z

@iabyn - Status changed from 'open' to 'resolved'

p5pRT closed this as completed May 14, 2012

p5pRT added Severity High distro-Linux hasCoreDump type-core type-regex labels Oct 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Heap corrupted by regex involving (?| ... ) and ($0) #12096

Heap corrupted by regex involving (?| ... ) and ($0) #12096

p5pRT commented May 11, 2012

p5pRT commented May 11, 2012

p5pRT commented May 11, 2012

p5pRT commented May 11, 2012

p5pRT commented May 11, 2012

p5pRT commented May 11, 2012

p5pRT commented May 13, 2012

p5pRT commented May 14, 2012

p5pRT commented May 14, 2012

p5pRT commented May 14, 2012

Heap corrupted by regex involving (?| ... ) and ($0) #12096

Heap corrupted by regex involving (?| ... ) and ($0) #12096

Comments

p5pRT commented May 11, 2012

p5pRT commented May 11, 2012

From @jimav

p5pRT commented May 11, 2012

From @jimav

p5pRT commented May 11, 2012

From @jimav

p5pRT commented May 11, 2012

From [Unknown Contact. See original ticket]

p5pRT commented May 11, 2012

p5pRT commented May 13, 2012

From @iabyn

p5pRT commented May 14, 2012

From @demerphq

p5pRT commented May 14, 2012

From @nwc10

p5pRT commented May 14, 2012