Skip Menu |
Report information
Id: 131642
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: randir <sergey.aleynikov [at] gmail.com>
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: medium
Type: core
Perl Version: 5.27.1
Fixed In: (no value)



To: perlbug [...] perl.org
Subject: regcomp.c:18417: void S_skip_to_be_ignored_text(RExC_state_t *, char **, const _Bool): Assertion `! UTF || UTF8_IS_INVARIANT(**p) || UTF8_IS_START(**p)' failed.
Date: Sat, 24 Jun 2017 12:49:45 +0300
From: Sergey Aleynikov <sergey.aleynikov [...] gmail.com>
Download (untitled) / with headers
text/plain 7.1k
This is a bug report for perl from sergey.aleynikov@gmail.com, generated with the help of perlbug 1.40 running under perl 5.27.1. ----------------------------------------------------------------- [Please describe your issue here] While fuzzing perl v5.27.1-37-g4c95ee9f29 built with afl and run under libdislocator, I found the following program (text also attached to this letter) 00000000 30 2c 73 70 6c 69 74 0d 70 61 63 6b 22 55 75 68 |0,split.pack"Uuh| 00000010 33 22 2c 30 2c 30 2c 27 30 6f 27 |3",0,0,'0o'| to cause an assertion failure, even when run under -c for a syntax check. This is a regression between 5.22 and 5.24, bisect points to: commit 361446f1521a506dd23cbbbe252294f244f360f5 Author: Karl Williamson <khw@cpan.org> Date: Wed Sep 23 14:06:23 2015 -0600 Allow (#...) anywhere white space is under qr//x Wherever you can have white space under /x, you can also have a (#...) comment (even without /x). Prior to this commit, there were several places that allowed the white space but not the comments. This resolves [perl #116639]. GDB info about the crash location is: (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00007fa5483813fa in __GI_abort () at abort.c:89 #2 0x00007fa548378e37 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55e2c166e0b0 "! UTF || UTF8_IS_INVARIANT(**p) || UTF8_IS_START(**p)", file=file@entry=0x55e2c1664f98 "regcomp.c", line=line@entry=18417, function=function@entry=0x55e2c166f9f0 <__PRETTY_FUNCTION__.19589> "S_skip_to_be_ignored_text") at assert.c:92 #3 0x00007fa548378ee2 in __GI___assert_fail (assertion=0x55e2c166e0b0 "! UTF || UTF8_IS_INVARIANT(**p) || UTF8_IS_START(**p)", file=0x55e2c1664f98 "regcomp.c", line=18417, function=0x55e2c166f9f0 <__PRETTY_FUNCTION__.19589> "S_skip_to_be_ignored_text") at assert.c:101 #4 0x000055e2c13d5c8e in S_skip_to_be_ignored_text (pRExC_state=0x7ffcc9ecea20, p=0x7ffcc9ecdd80, force_to_xmod=false) at regcomp.c:18417 #5 0x000055e2c13c14af in S_regatom (pRExC_state=0x7ffcc9ecea20, flagp=0x7ffcc9ece050, depth=4) at regcomp.c:13494 #6 0x000055e2c13b603b in S_regpiece (pRExC_state=0x7ffcc9ecea20, flagp=0x7ffcc9ece17c, depth=3) at regcomp.c:11668 #7 0x000055e2c13b5989 in S_regbranch (pRExC_state=0x7ffcc9ecea20, flagp=0x7ffcc9ece228, first=1, depth=2) at regcomp.c:11593 #8 0x000055e2c13b326c in S_reg (pRExC_state=0x7ffcc9ecea20, paren=0, flagp=0x7ffcc9ece664, depth=1) at regcomp.c:11331 #9 0x000055e2c139b859 in Perl_re_op_compile (patternp=0x0, pat_count=1, expr=0x55e2c1f8a280, eng=0x55e2c1901540 <PL_core_reg_engine>, old_re=0x0, is_bare_re=0x0, orig_rx_flags=2048, pm_flags=2048) at regcomp.c:7100 #10 0x000055e2c12b899d in Perl_pmruntime (o=0x55e2c1f8a150, expr=0x55e2c1f8a280, repl=0x0, flags=2, floor=0) at op.c:5882 #11 0x000055e2c12d67f2 in Perl_ck_split (o=0x55e2c1f7d120) at op.c:11268 #12 0x000055e2c12b4f39 in Perl_op_convert_list (type=155, flags=0, o=0x55e2c1f7d120) at op.c:4889 #13 0x000055e2c136b09e in Perl_yyparse (gramtype=258) at perly.y:889 #14 0x000055e2c1519108 in S_doeval_compile (gimme=1 '\001', outside=0x55e2c1f52340, seq=4294967261, hh=0x0) at pp_ctl.c:3456 #15 0x000055e2c1520b9a in Perl_pp_entereval () at pp_ctl.c:4415 #16 0x000055e2c13f4a7d in Perl_runops_debug () at dump.c:2451 #17 0x000055e2c12eab3d in S_run_body (oldscope=1) at perl.c:2548 #18 0x000055e2c12ea0bb in perl_run (my_perl=0x55e2c1f50010) at perl.c:2471 #19 0x000055e2c12a2f3e in main (argc=4, argv=0x7ffcc9ecf7f8, env=0x7ffcc9ecf820) at perlmain.c:123 (gdb) f 4 #4 0x000055e2c13d5c8e in S_skip_to_be_ignored_text (pRExC_state=0x7ffcc9ecea20, p=0x7ffcc9ecdd80, force_to_xmod=false) at regcomp.c:18417 18417 assert( ! UTF || UTF8_IS_INVARIANT(**p) || UTF8_IS_START(**p)); (gdb) p **p $1 = -92 '\244' [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=medium --- Site configuration information for perl 5.27.1: Configured by root at Sun May 28 01:44:41 MSK 2017. Summary of my perl5 (revision 5 version 26 subversion 0) configuration: Derived from: 4c95ee9f298c2edfc1382d540ff89288790e78b6 Platform: osname=linux osvers=4.9.0-3-amd64 archname=x86_64-linux uname='linux dorothy 4.9.0-3-amd64 #1 smp debian 4.9.25-1 (2017-05-02) x86_64 gnulinux ' config_args='-des -Dusedevel -DDEBUGGING -Dcc=afl-clang-fast -Doptimize=-O0 -g -ggdb3 -fno-omit-frame-pointer' hint=previous useposix=true d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='afl-clang-fast' ccflags ='-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2' optimize='-O0 -g -ggdb3 -fno-omit-frame-pointer' cppflags='-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='' gccversion='4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/rc2)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='afl-clang-fast' ldflags =' -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/lib /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/local/lib /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/lib /usr/include/x86_64-linux-gnu /usr/lib libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.24.so so=so useshrplib=false libperl=libperl.a gnulibc_version='2.24' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,-E' cccdlflags='-fPIC' lddlflags='-shared -O0 -g -ggdb3 -fno-omit-frame-pointer -L/usr/local/lib -fstack-protector-strong' Locally applied patches: uncommitted-changes --- @INC for perl 5.27.1: lib /usr/local/lib/perl5/site_perl/5.26.0/x86_64-linux /usr/local/lib/perl5/site_perl/5.26.0 /usr/local/lib/perl5/5.26.0/x86_64-linux /usr/local/lib/perl5/5.26.0 --- Environment for perl 5.27.1: HOME=/home/afl LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.24.1-dbg/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games PERLBREW_BASHRC_VERSION=0.78 PERLBREW_HOME=/home/afl/.perlbrew PERLBREW_MANPATH=/home/afl/perlbrew/perls/perl-5.24.1-dbg/man PERLBREW_PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.24.1-dbg/bin PERLBREW_PERL=perl-5.24.1-dbg PERLBREW_ROOT=/home/afl/perlbrew PERLBREW_VERSION=0.78 PERL_BADLANG (unset) SHELL=/usr/bin/zsh
Download 0091
application/octet-stream 27b

Message body not shown because it is not plain text.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 228b
I looked at this and the assertion that's failing is valid. The problem is that pack is returning malformed UTF-8, which you can see if you add -Dr to the command line options. So I'm unsure how to proceed. -- Karl Williamson
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 752b
It seems to me that the right fix to this is to forbid pack from returning malformed UTF-8. When I change it to do that, various tests in our suite fail. All these look to be deliberate attempts to generate malformed UTF-8, and testing how this is handled, and they use pack to do that generating. So, it's been known that you can use pack for this, and people have taken advantage of it. No /cpan tests rely on this. But I think we've gotten wise over the years about the perils of malformed UTF-8, and I think this is one that should be fixed. I don't know if it is too late in the 5.28 development cycle to do so, however. But another way of looking at it, it is just in time to fix bugs that would otherwise occur in 5.28 -- Karl Williamson
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 982b
On Sun, 01 Apr 2018 17:13:52 -0700, khw wrote: Show quoted text
> It seems to me that the right fix to this is to forbid pack from > returning malformed UTF-8. When I change it to do that, various tests > in our suite fail. All these look to be deliberate attempts to > generate malformed UTF-8, and testing how this is handled, and they > use pack to do that generating. > > So, it's been known that you can use pack for this, and people have > taken advantage of it. No /cpan tests rely on this. But I think > we've gotten wise over the years about the perils of malformed UTF-8, > and I think this is one that should be fixed.
I too agree that it should be fixed. Show quoted text
> > I don't know if it is too late in the 5.28 development cycle to do so, > however. But another way of looking at it, it is just in time to fix > bugs that would otherwise occur in 5.28
I’m a little wary of doing it this close to a stable release if it’s possible people are relying on it. -- Father Chrysostomos
From: Sawyer X <xsawyerx [...] gmail.com>
Date: Mon, 2 Apr 2018 22:28:26 +0300
To: perlbug-followup [...] perl.org
CC: perl5-porters [...] perl.org
Subject: Re: [perl #131642] regcomp.c:18417: void S_skip_to_be_ignored_text(RExC_state_t *, char **, const _Bool): Assertion `! UTF || UTF8_IS_INVARIANT(**p) || UTF8_IS_START(**p)' failed.
On 04/02/2018 04:16 AM, Father Chrysostomos via RT wrote: Show quoted text
> On Sun, 01 Apr 2018 17:13:52 -0700, khw wrote:
>> It seems to me that the right fix to this is to forbid pack from >> returning malformed UTF-8. When I change it to do that, various tests >> in our suite fail. All these look to be deliberate attempts to >> generate malformed UTF-8, and testing how this is handled, and they >> use pack to do that generating. >> >> So, it's been known that you can use pack for this, and people have >> taken advantage of it. No /cpan tests rely on this. But I think >> we've gotten wise over the years about the perils of malformed UTF-8, >> and I think this is one that should be fixed.
> I too agree that it should be fixed. >
>> I don't know if it is too late in the 5.28 development cycle to do so, >> however. But another way of looking at it, it is just in time to fix >> bugs that would otherwise occur in 5.28
> I’m a little wary of doing it this close to a stable release if it’s possible people are relying on it.
Agreed. I would rather we save it for 5.29.
RT-Send-CC: perl5-porters [...] perl.org, perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.4k
On Mon, 02 Apr 2018 12:28:46 -0700, xsawyerx@gmail.com wrote: Show quoted text
> > > On 04/02/2018 04:16 AM, Father Chrysostomos via RT wrote:
> > On Sun, 01 Apr 2018 17:13:52 -0700, khw wrote:
> >> It seems to me that the right fix to this is to forbid pack from > >> returning malformed UTF-8. When I change it to do that, various > >> tests > >> in our suite fail. All these look to be deliberate attempts to > >> generate malformed UTF-8, and testing how this is handled, and they > >> use pack to do that generating. > >> > >> So, it's been known that you can use pack for this, and people have > >> taken advantage of it. No /cpan tests rely on this. But I think > >> we've gotten wise over the years about the perils of malformed UTF- > >> 8, > >> and I think this is one that should be fixed.
> > I too agree that it should be fixed. > >
> >> I don't know if it is too late in the 5.28 development cycle to do > >> so, > >> however. But another way of looking at it, it is just in time to > >> fix > >> bugs that would otherwise occur in 5.28
> > I’m a little wary of doing it this close to a stable release if it’s > > possible people are relying on it.
> > Agreed. I would rather we save it for 5.29.
I'm not sure what the best approach is. Attached is a patch that just croaks when the returned SV is UTF-8 and malformed. Another option would be to just turn off the UTF-8 flag. Should this use be deprecated instead? -- Karl Williamson
Subject: 0001-Trial-patch-for-131642.patch
From d8f351f05faf4badc3b80de38390cfac14e56418 Mon Sep 17 00:00:00 2001 From: Karl Williamson <khw@cpan.org> Date: Sun, 1 Jul 2018 22:39:47 -0600 Subject: [PATCH] Trial patch for #131642 --- pp_pack.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/pp_pack.c b/pp_pack.c index 5e9cc64301..1eed6c4d79 100644 --- a/pp_pack.c +++ b/pp_pack.c @@ -3147,6 +3147,15 @@ PP(pp_pack) packlist(cat, pat, patend, MARK, SP + 1); + if (SvUTF8(cat)) { + STRLEN result_len; + const char * result = SvPV_nomg(cat, result_len); + + if (! is_utf8_string((U8 *) result, result_len)) { + Perl_croak(aTHX_ "Malformed UTF-8 string returned from pack"); + } + } + SvSETMAGIC(cat); SP = ORIGMARK; PUSHs(cat); -- 2.17.1
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 554b
Fixed by commit fd879d933c2a2ee22ac6e2462acc016aa033854d Author: Karl Williamson <khw@cpan.org> Date: Sun Jul 1 22:39:47 2018 -0600 PATCH: [perl #131642] pack returning malformed UTF-8 This patch causes pack to die rather than return malformed UTF-8. This protects the rest of the core from unexpectedly getting malformed inputs. We'll see if this breaks cpan, with people relying on pcak to create malformed UTF-8. But my current inclination is they should be doing something else to generate it. -- Karl Williamson
Download (untitled) / with headers
text/plain 313b
Thank you for filing this report. You have helped make Perl better. With the release today of Perl 5.30.0, this and 160 other issues have been resolved. Perl 5.30.0 may be downloaded via: https://metacpan.org/release/XSAWYERX/perl-5.30.0 If you find that the problem persists, feel free to reopen this ticket.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org