From @dur-randir

Created by @dur-randir

While fuzzing perl v5.27.1-37-g4c95ee9f29 built with afl and run
under libdislocator, I found the following program

for(uc 0..t){0~~pack"UXp>",exp}

to cause an assertion failure. This is a regression in 5.26, bisect points to​:

commit 7cf8d05
Author​: Karl Williamson <khw@​cpan.org>
Date​: Wed Sep 28 15​:05​:17 2016 -0600

  Add details to UTF-8 malformation error messages

GDB info about the crash location is​:

(gdb) bt
#0 __GI_raise (sig=sig@​entry=6) at ../sysdeps/unix/sysv/linux/raise.c​:51
#1 0x00007ffff6cf63fa in __GI_abort () at abort.c​:89
#2 0x00007ffff6cede37 in __assert_fail_base (fmt=<optimized out>,
assertion=assertion@​entry=0x5555559a83ad "expect_len == UTF8SKIP(s)",
  file=file@​entry=0x5555559a7b65 "utf8.c", line=line@​entry=832,
  function=function@​entry=0x5555559b2280 <__PRETTY_FUNCTION__.15262>
"S_unexpected_non_continuation_text") at assert.c​:92
#3 0x00007ffff6cedee2 in __GI___assert_fail (assertion=0x5555559a83ad
"expect_len == UTF8SKIP(s)", file=0x5555559a7b65 "utf8.c", line=832,
  function=0x5555559b2280 <__PRETTY_FUNCTION__.15262>
"S_unexpected_non_continuation_text") at assert.c​:101
#4 0x00005555558857da in S_unexpected_non_continuation_text
(s=0x555555c062d1 "", print_len=2, non_cont_byte_pos=1, expect_len=2)
at utf8.c​:832
#5 0x00005555558871db in Perl_bytes_cmp_utf8 (b=0x555555c05260 "0",
blen=1, u=0x555555c062d2 "", ulen=9) at utf8.c​:1876
#6 0x0000555555780e0a in Perl_sv_eq_flags (sv1=0x555555c0b3b0,
sv2=0x555555c0b428, flags=0) at sv.c​:7887
#7 0x00005555557b63da in Perl_pp_seq () at pp.c​:2383
#8 0x000055555580a669 in S_do_smartmatch (seen_this=0x0,
seen_other=0x0, copied=false) at pp_ctl.c​:5152
#9 0x0000555555803a27 in Perl_pp_smartmatch () at pp_ctl.c​:4670
#10 0x00005555556d5a7d in Perl_runops_debug () at dump.c​:2451
#11 0x00005555555cbb3d in S_run_body (oldscope=1) at perl.c​:2548
#12 0x00005555555cb0bb in perl_run (my_perl=0x555555bed010) at perl.c​:2471
#13 0x0000555555583f3e in main (argc=2, argv=0x7fffffffe1e8,
env=0x7fffffffe200) at perlmain.c​:123
(gdb) p expect_len
$1 = 2
(gdb) p s
$2 = (const U8 * const) 0x555555c062d1 ""
(gdb) p UTF8SKIP(s)
$3 = 1 '\001'

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.27.1:

Configured by root at Sun May 28 01:44:41 MSK 2017.

Summary of my perl5 (revision 5 version 26 subversion 0) configuration:
  Derived from: 4c95ee9f298c2edfc1382d540ff89288790e78b6
  Platform:
    osname=linux
    osvers=4.9.0-3-amd64
    archname=x86_64-linux
    uname='linux dorothy 4.9.0-3-amd64 #1 smp debian 4.9.25-1
(2017-05-02) x86_64 gnulinux '
    config_args='-des -Dusedevel -DDEBUGGING -Dcc=afl-clang-fast
-Doptimize=-O0 -g -ggdb3 -fno-omit-frame-pointer'
    hint=previous
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='afl-clang-fast'
    ccflags ='-DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O0 -g -ggdb3 -fno-omit-frame-pointer'
    cppflags='-DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/rc2)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='afl-clang-fast'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/lib
/usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu
/lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
/usr/local/lib /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/lib
/usr/include/x86_64-linux-gnu /usr/lib
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.24.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.24'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O0 -g -ggdb3 -fno-omit-frame-pointer
-L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    uncommitted-changes


@INC for perl 5.27.1:
    lib
    /usr/local/lib/perl5/site_perl/5.26.0/x86_64-linux
    /usr/local/lib/perl5/site_perl/5.26.0
    /usr/local/lib/perl5/5.26.0/x86_64-linux
    /usr/local/lib/perl5/5.26.0


Environment for perl 5.27.1:
    HOME=/home/afl
    LANG=en_US.UTF-8
    LANGUAGE=en_US:en
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.24.1-dbg/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
    PERLBREW_BASHRC_VERSION=0.78
    PERLBREW_HOME=/home/afl/.perlbrew
    PERLBREW_MANPATH=/home/afl/perlbrew/perls/perl-5.24.1-dbg/man
    PERLBREW_PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.24.1-dbg/bin
    PERLBREW_PERL=perl-5.24.1-dbg
    PERLBREW_ROOT=/home/afl/perlbrew
    PERLBREW_VERSION=0.78
    PERL_BADLANG (unset)
    SHELL=/usr/bin/zshpe

@khwilliamson - Status changed from 'new' to 'open'

From @khwilliamson

Thanks, fixed by
1d5030e

I'm adding my vote for this to go into 5.26.1

--
Karl Williamson

@khwilliamson - Status changed from 'open' to 'pending release'

From @tonycoz

On Sat, 24 Jun 2017 11​:09​:30 -0700, khw wrote​:

Thanks, fixed by
1d5030e

I'm adding my vote for this to go into 5.26.1

The test case​:

for(uc 0..t){0~~pack"UXp>",exp}

here is fragile.

The pack pattern here is​:

- pack as unicode (the error case triggers on exp(5))
- back up one byte
- pack a pointer to the PV of the supplied SV, in big-endian (none is supplied, so the PV of PL_sv_no is used)

So the test case depends on the high-byte of the PV in PL_sv_no being zero, which may not be the case, especially on 32-bit systems.

Changing the pattern to​:

"UXc"

should be less fragile (a zero byte will be packed instead of the high-byte of a pointer.)

I discovered this while testing some other pack changes.

Patch attached.

Tony

From @tonycoz

From 8e61f24b2b152a5e6a8bf22ddbab304ccdb413aa Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@debian9-x32.tony.develop-help.com>
Date: Tue, 8 Aug 2017 11:09:02 +1000
Subject: (perl #131646) make the test less fragile

The original pattern "UXp>" with the $_ that causes the failure, 5,
so we end up packing exp(5) or 148.... with U packs:

 - U (148), producing C2 94, with the UTF8 flag set
 - X - back up a byte,
 - p> - write the address of PL_sv_no's PV in big-ending

The final p> will typically overwrite the 94 with a zero on 64-bit
systems, but with the smaller address space of 32-bit systems that
high-byte is much less likely to be 0, causing the comparison to fail.

Instead just pack a zero byte.
---
 t/lib/warnings/utf8 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/lib/warnings/utf8 b/t/lib/warnings/utf8
index dfc58c12db..a9a6388d31 100644
--- a/t/lib/warnings/utf8
+++ b/t/lib/warnings/utf8
@@ -779,7 +779,7 @@ BEGIN{
 }
 no warnings;
 use warnings 'utf8';
-for(uc 0..t){0~~pack"UXp>",exp}
+for(uc 0..t){0~~pack"UXc",exp}
 EXPECT
 OPTIONS regex
 Malformed UTF-8 character: \\x([[:xdigit:]]{2})\\x([[:xdigit:]]{2}) \(unexpected non-continuation byte 0x\2, immediately after start byte 0x\1; need 2 bytes, got 1\)  in smart match at - line 9.
-- 
2.11.0

From @tonycoz

On Mon, 07 Aug 2017 18​:13​:07 -0700, tonyc wrote​:

So the test case depends on the high-byte of the PV in PL_sv_no being
zero, which may not be the case, especially on 32-bit systems.

Actually, it doesn't depend on the high byte being zero, but it does depend on the high-byte not being a valid continuation byte, which might not be the case.

Tony

From @tonycoz

On Mon, 07 Aug 2017 18​:23​:00 -0700, tonyc wrote​:

On Mon, 07 Aug 2017 18​:13​:07 -0700, tonyc wrote​:

So the test case depends on the high-byte of the PV in PL_sv_no being
zero, which may not be the case, especially on 32-bit systems.

Actually, it doesn't depend on the high byte being zero, but it does
depend on the high-byte not being a valid continuation byte, which
might not be the case.

Patch applied as 9c6b56d with a changed comment to reflect the above.

Tony

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release yesterday of Perl 5.28.0, this and 185 other issues have been
resolved.

Perl 5.28.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.28.0

If you find that the problem persists, feel free to reopen this ticket.

@khwilliamson - Status changed from 'pending release' to 'resolved'