From @dur-randir

Created by @dur-randir

While fuzzing perl v5.29.9-63-g2496d8f3f7 built with afl and run
under libdislocator, I found the program attached to this message to
cause an assertion failure

perl​: toke.c​:4047​: char *S_scan_const(char *)​: Assertion
`isUTF8_CHAR((U8 *) s, (U8 *) send)' failed.

GDB stack trace is following

#0 __GI_raise (sig=sig@​entry=6) at ../sysdeps/unix/sysv/linux/raise.c​:50
#1 0x00007ffff7c25535 in __GI_abort () at abort.c​:79
#2 0x00007ffff7c2540f in __assert_fail_base (fmt=0x7ffff7d87ee0
"%s%s%s​:%u​: %s%sAssertion `%s' failed.\n%n",
  assertion=0x55555593ce38 "isUTF8_CHAR((U8 *) s, (U8 *) send)",
file=0x5555559363c5 "toke.c", line=4047, function=<optimized out>) at
assert.c​:92
#3 0x00007ffff7c330f2 in __GI___assert_fail (assertion=0x55555593ce38
"isUTF8_CHAR((U8 *) s, (U8 *) send)", file=0x5555559363c5 "toke.c",
line=4047,
  function=0x55555595b588 <__PRETTY_FUNCTION__.18838>
"S_scan_const") at assert.c​:101
#4 0x0000555555630613 in S_scan_const (start=0x555555b65030
"è(?#\302\204") at toke.c​:4047
#5 0x0000555555638cba in Perl_yylex () at toke.c​:5087
#6 0x000055555566bb9e in Perl_yyparse (gramtype=258) at perly.c​:340
#7 0x0000555555838e25 in S_doeval_compile (gimme=1 '\001',
outside=0x555555b4e908, seq=4294967258, hh=0x0) at pp_ctl.c​:3502
#8 0x0000555555840bc0 in Perl_pp_entereval () at pp_ctl.c​:4478
#9 0x000055555570b640 in Perl_runops_debug () at dump.c​:2537
#10 0x00005555555ed560 in S_run_body (oldscope=1) at perl.c​:2716
#11 0x00005555555ecade in perl_run (my_perl=0x555555b4c260) at perl.c​:2639
#12 0x00005555555a114e in main (argc=3, argv=0x7fffffffe1b8,
env=0x7fffffffe1d8) at perlmain.c​:127

This is a regression between 5.24 and 5.26, bisect points to

commit 9dfb44e
Author​: Karl Williamson <khw@​cpan.org>
Date​: Sat Dec 3 12​:14​:33 2016 -0700

  toke.c​: Avoid a conversion to/from UTF-8

  If the source file is encoded as UTF-8, we don't have to find its code
  point equivalent when parsing--we can just copy it unchanged. This
  wasn't done before because of the fear the input would be malformed, and
  finding the code point had the side effect of checking for
  well-formedness. The previous commit added wellformedness checking,
  so doing it again here would be redundant.

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.29.9:

Configured by dur-randir at Wed Feb 27 14:51:01 MSK 2019.

Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
  Commit id: c1e47bad34ce1d9c84ed57c9b8978bcbd5a02e98
  Platform:
    osname=darwin
    osvers=13.4.0
    archname=darwin-thread-multi-2level
    uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0:
mon jan 11 18:17:34 pst 2016; root:xnu-2422.115.15~1release_x86_64
x86_64 '
    config_args='-de -Dusedevel -DDEBUGGING -Dusethreads'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3 -g'
    cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=
    so=dylib
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined
dynamic_lookup -L/usr/local/lib -fstack-protector'



@INC for perl 5.29.9:
    lib
    /usr/local/lib/perl5/site_perl/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.29.9
    /usr/local/lib/perl5/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/5.29.9


Environment for perl 5.29.9:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/dur-randir
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin
    PERLBREW_HOME=/Users/dur-randir/.perlbrew
    PERLBREW_MANPATH=/Users/dur-randir/perlbrew/perls/perl-5.22.1/man
    PERLBREW_PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin
    PERLBREW_PERL=perl-5.22.1
    PERLBREW_ROOT=/Users/dur-randir/perlbrew
    PERLBREW_SHELLRC_VERSION=0.84
    PERLBREW_VERSION=0.84
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

From @dur-randir

0021_2

@khwilliamson - Status changed from 'new' to 'open'

From @khwilliamson

As I say in the commit message below, this is a long-standing bug. But it appears to be hard to get the exact circumstances to get it to occur.

As always, thanks for finding this bug.
Fixed by
commit f339d50
Author​: Karl Williamson <khw@​cpan.org>
Date​: Sat Apr 6 12​:38​:56 2019 -0600

  PATCH​: [perl #133992] Assertion failure in scan_const
 
  I haven't done the digging, but this appears to be a failure to include
  UTF-8 processing when 'use utf8' was added to Perl.
 
  The code that was causing this in toke.c had found a qr/(?#... beginning
  of comment in a pattern. It attempted to space up to but not including
  the final character, which is handled later. (In most instances that
  final character is a single-byte ')', but not in this test case. It
  spaced per-byte. The problem is that if the final character is in UTF-8
  and isn't a single byte, it leaves the input position pointing at the
  final byte of that character, which creates malformed UTF-8, which the
  assertion discovered.
 
  The fix is to be cognizant that this is UTF-8 when spacing to the end,
  so that the final position begins at the first byte of it.

--
Karl Williamson

@khwilliamson - Status changed from 'open' to 'pending release'

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

@khwilliamson - Status changed from 'pending release' to 'resolved'