From @dcollinsn

Greetings Porters,

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in DEBUGGING perls without any other symptoms in the normal perl interpreter. The testcase is the file​:

/(?{})|(??{U​:0})/|s|||g

dcollins@​nightshade64​:/usr/local/perl-afl/out$ ~/perldebug/perl -e '/(?{})|(??{U​:0})/|s|||g'
perl​: sv.c​:9875​: Perl_sv_pvn_force_flags​: Assertion `PL_valid_types_PVX[((svtype)((_svpvx)->sv_flags & 0xff)) & 0xf]' failed.
Aborted

On non-debugging perls, this eventually leads to a segfault​:

==11561== Invalid read of size 8
==11561== at 0x6630A5​: Perl_pregfree2 (regcomp.c​:17498)
==11561== by 0x7B2CE7​: Perl_sv_clear (sv.c​:6458)
==11561== by 0x7ADD0F​: Perl_sv_free2 (sv.c​:6885)
==11561== by 0x6E9504​: S_SvREFCNT_dec (inline.h​:166)
==11561== by 0x6E9504​: S_mg_free_struct (mg.c​:549)
==11561== by 0x6E9504​: Perl_mg_free (mg.c​:571)
==11561== by 0x7B2A0F​: Perl_sv_clear (sv.c​:6428)
==11561== by 0x7ADD0F​: Perl_sv_free2 (sv.c​:6885)
==11561== by 0x433874​: S_SvREFCNT_dec (inline.h​:166)
==11561== by 0x433874​: Perl_op_clear (op.c​:896)
==11561== by 0x43412B​: Perl_op_free (op.c​:779)
==11561== by 0x433FCB​: Perl_op_free (op.c​:762)
==11561== by 0x433808​: Perl_op_clear (op.c​:954)
==11561== by 0x43412B​: Perl_op_free (op.c​:779)
==11561== by 0x433FCB​: Perl_op_free (op.c​:762)
==11561== Address 0x28 is not stack'd, malloc'd or (recently) free'd

**GDB**

(gdb) run
Starting program​: /home/dcollins/perldebug/perl -e /\(\?\{\}\)\|\(\?\?\{U​:0\}\)/\|s\|\|\|g
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
perl​: sv.c​:9875​: Perl_sv_pvn_force_flags​: Assertion `PL_valid_types_PVX[((svtype)((_svpvx)->sv_flags & 0xff)) & 0xf]' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6cf4107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff6cf4107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff6cf54e8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff6ced226 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff6ced2d2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00000000009d23dd in Perl_sv_pvn_force_flags (sv=sv@​entry=0x11d8560, lp=lp@​entry=0x7fffffffdfd0, flags=flags@​entry=16384) at sv.c​:9875
#5 0x00000000009dbb3e in Perl_sv_catpvn_flags (dsv=dsv@​entry=0x11d8560, sstr=0x11d2460 "", slen=0, flags=16384) at sv.c​:5361
#6 0x00000000008f2bf3 in Perl_pp_subst () at pp_hot.c​:3133
#7 0x00000000007c90bf in Perl_runops_debug () at dump.c​:2224
#8 0x0000000000532411 in S_run_body (oldscope=1) at perl.c​:2464
#9 perl_run (my_perl=<optimized out>) at perl.c​:2387
#10 0x0000000000428928 in main (argc=3, argv=0x7fffffffe338, env=0x7fffffffe358) at perlmain.c​:116
(gdb) f 4
#4 0x00000000009d23dd in Perl_sv_pvn_force_flags (sv=sv@​entry=0x11d8560, lp=lp@​entry=0x7fffffffdfd0, flags=flags@​entry=16384) at sv.c​:9875
9875 SvGROW(sv, len + 1);
(gdb) info locals
_svpvx = 0x11d8560
s = <optimized out>
len = 6
__PRETTY_FUNCTION__ = "Perl_sv_pvn_force_flags"

**VALGRIND**

==5662== Memcheck, a memory error detector
==5662== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==5662== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==5662== Command​: /home/dcollins/perldebug/perl -e /(?{})|(??{U​:0})/|s|||g
==5662==
perl​: sv.c​:9875​: Perl_sv_pvn_force_flags​: Assertion `PL_valid_types_PVX[((svtype)((_svpvx)->sv_flags & 0xff)) & 0xf]' failed.
==5662==
==5662== Process terminating with default action of signal 6 (SIGABRT)
==5662== at 0x5BDC107​: raise (raise.c​:56)
==5662== by 0x5BDD4E7​: abort (abort.c​:89)
==5662== by 0x5BD5225​: __assert_fail_base (assert.c​:92)
==5662== by 0x5BD52D1​: __assert_fail (assert.c​:101)
==5662== by 0x9D23DC​: Perl_sv_pvn_force_flags (sv.c​:9875)
==5662== by 0x9DBB3D​: Perl_sv_catpvn_flags (sv.c​:5361)
==5662== by 0x8F2BF2​: Perl_pp_subst (pp_hot.c​:3133)
==5662== by 0x7C90BE​: Perl_runops_debug (dump.c​:2224)
==5662== by 0x532410​: S_run_body (perl.c​:2464)
==5662== by 0x532410​: perl_run (perl.c​:2387)
==5662== by 0x428927​: main (perlmain.c​:116)
==5662==
==5662== HEAP SUMMARY​:
==5662== in use at exit​: 131,879 bytes in 671 blocks
==5662== total heap usage​: 858 allocs, 187 frees, 158,569 bytes allocated
==5662==
==5662== LEAK SUMMARY​:
==5662== definitely lost​: 320 bytes in 1 blocks
==5662== indirectly lost​: 2,599 bytes in 38 blocks
==5662== possibly lost​: 0 bytes in 0 blocks
==5662== still reachable​: 128,960 bytes in 632 blocks
==5662== suppressed​: 0 bytes in 0 blocks
==5662== Rerun with --leak-check=full to see details of leaked memory
==5662==
==5662== For counts of detected and suppressed errors, rerun with​: -v
==5662== ERROR SUMMARY​: 0 errors from 0 contexts (suppressed​: 0 from 0)
Aborted

**PERL -V**

dcollins@​nightshade64​:/usr/local/perl-afl$ ./bin/perl -V
Summary of my perl5 (revision 5 version 23 subversion 5) configuration​:
  Commit id​: 7195e5d
  Platform​:
  osname=linux, osvers=3.16.0-4-amd64, archname=x86_64-linux-ld
  uname='linux nightshade64 3.16.0-4-amd64 #1 smp debian 3.16.7-ckt11-1+deb8u4 (2015-09-19) x86_64 gnulinux '
  config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache afl-gcc -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='ccache afl-gcc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-g',
  cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion='', gccversion='5.2.0', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='ccache afl-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/local/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL
  USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
  USE_PERLIO USE_PERL_ATOF
  Built under linux
  Compiled at Oct 22 2015 15​:44​:40
  @​INC​:
  /usr/local/perl-afl/lib/site_perl/5.23.5/x86_64-linux-ld
  /usr/local/perl-afl/lib/site_perl/5.23.5
  /usr/local/perl-afl/lib/5.23.5/x86_64-linux-ld
  /usr/local/perl-afl/lib/5.23.5
  /usr/local/perl-afl/lib/site_perl/5.23.4
  /usr/local/perl-afl/lib/site_perl
  .

From @tonycoz

On Tue Nov 10 19​:04​:55 2015, dcollinsn@​gmail.com wrote​:

After reducing testcases using `afl-tmin` and performing additional
minimization by hand, I have located the following testcase that
triggers an assert fail in DEBUGGING perls without any other symptoms
in the normal perl interpreter. The testcase is the file​:

/(?{})|(??{U​:0})/|s|||g

This can be simplified further to​:

  s/(?{})|(??{U​:0})//g

dcollins@​nightshade64​:/usr/local/perl-afl/out$ ~/perldebug/perl -e
'/(?{})|(??{U​:0})/|s|||g'
perl​: sv.c​:9875​: Perl_sv_pvn_force_flags​: Assertion
`PL_valid_types_PVX[((svtype)((_svpvx)->sv_flags & 0xff)) & 0xf]'
failed.
Aborted

There's two problems in the original​:

1) it looks like #124368 is biting us in a different way in that the regexp being used by pp_subst is the one from the match

2) pp_subst() (in either example) creates a new temp sv (line 3078)​:

  dstr = newSVpvn_flags(orig, s-orig,
  SVs_TEMP | (DO_UTF8(TARG) ? SVf_UTF8 : 0));

then later calls CALLREGEXEC() (line 3129)​:

  } while (CALLREGEXEC(rx, s, strend, orig,
  s == m, /* Yields minend of 0 or 1 */
  TARG, NULL,
  REXEC_NOT_FIRST|REXEC_IGNOREPOS|REXEC_FAIL_ON_UNDERFLOW));

and the pp_nextstate called when running the C<U​:0> code calls FREETMPS, releasing dstr, and so the following (line 3133) tries to work with a freed SV​:

  sv_catpvn_nomg_maybeutf8(dstr, s, strend - s, DO_UTF8(TARG));

and asserts.

Tony

The RT System itself - Status changed from 'new' to 'open'

From @tonycoz

On Wed Dec 02 19​:45​:43 2015, tonyc wrote​:

1) it looks like #124368 is biting us in a different way in that the
regexp being used by pp_subst is the one from the match

Or not, s/// has the same empty regexp behaviour as m//

Tony

From @hvds

I can't reproduce this with blead. Bisect using​:

% Porting/bisect.pl --start=7195e5da55a40d15e29ad80562668bdd6895441f --crash --expect-fail --target=miniperl -e '/(?{})|(??{U​:0})/|s|||g'

..leads to 1dfbe6b which is partway through the huge "revamp context system" from DaveM merged in February (at 9d876b6)​:

It sounds reasonable to me that the context work fixes what appears to be a premature free, so I'm resolving this ticket.

Hugo

@hvds - Status changed from 'open' to 'resolved'